Shamus McGillicuddy, not to be confused with the McGillicuddy Serious Party of New Zealand, recently wrote at Searchnetworking.com about Don Lester, a senior network engineer with Wenatchee Valley Medical Center in Wenatchee, Washington.

Lester used the application response time and network traffic analysis modules of the NetQoS Performance Center to diagnose problems with performance of a medical records application, which “affects patient care because doctors won’t be able to do things they might normally want to do to help patients.”

The article shows our products at their best, and it would be a bit shameless of us to quote the bits we really like.

Shameless and fun!

“I popped up Reporter-Analyzer first because it's one of the quickest ways for me to look at what's going to a location," he said. "I was able to see that there was a whole bunch of traffic going to one of the locations from the workstation patching server."

Through some quick investigation, Lester was able to learn that the medical center's PC technicians were supposed to push out a patch in the middle of the night using their ability to turn on machines remotely for update. But something had gone wrong, and the PCs had never been turned on. Instead, the patching server waited until morning, when users started turning on their computers. The computers started pulling patches, which slowed down the WAN link at the remote location.

The problem at the second location was completely unrelated.

"There were no signs of anything," Lester said. "The link didn't have any significant traffic at all."

He pulled up another NetQoS tool -- SuperAgent, which analyzes TCP transaction -- and saw a high level of retransmission delay occurring at the second WAN link.

"So there was a problem with that circuit," he said. "We had to work with a third party who managed the circuit because it's not something we have a lot of eyes into. We told them to fix it. And we were able to use the same tool to tell when, in fact, they had fixed it and if it was as good as it was before the malfunction…

It kinda gives you a warm, squishy feeling in your heart, doesn’t it?

…As well as maintaining critical application performance, Lester's NetQoS tools control network costs.

"It isn't unusual for us to be negotiating a [WAN] contract based on a tier of service that is defined by the average amount of bandwidth consumed over a given period of time," he said. "There have been times when the vendor in question will inadvertently overstate our usage. When that happens, we will produce usage graphs over whatever time period is appropriate, detailing our actual usage -- and ultimately reclassify our tier level and negotiate a lower price."

Lester said a more common issue that comes up is dealing with application vendors. Often, when there is a performance problem with an application, the vendors will take a number of standard corrective actions to try to solve the problem. If none of these actions works, the application vendor will often blame the network and claim that the medical center needs more bandwidth.

"We are able to tell rather easily using NetQoS tools whether or not we really have a bandwidth issue and can then share that information in a concise graphical format," Lester said. "This not only saves us the cost of unnecessarily upgrading circuits, but it also results in better service to the end user since this re-engages the vendor. The troubleshooting escalates, and the root cause is isolated and repaired."

I think that the article illustrates one of the reasons that NetQoS focuses on performance rather than fault – that is, performance data is more valuable than fault data. You could use a quick and dirty program to check on fault and availability, but for more difficult performance problems, you need more sophisticated monitoring tools.

(I just got this idea for a comedy sketch in which you have a doctor that only checks for availability – all patients are either “alive” or “dead.”)

We contacted Don Lester for his thoughts. He said that:

“I have a lot more than two applications I worry about, but there are two that are more important than all the others. All in all though, it does detail what everyone should hope to have in their environment. Simply the ability to troubleshoot or analyze without having to guess or act on instinct or hunches. It is a lot easier to do this kind of work with solid, factual information than to sit around pinging everything and hoping you stumble onto a smoking gun.”