Case Study—Network ”X-Ray vision” reveals some surprises
About the company
This case study describes some of the actual networking challenges encountered by a private medical practice of board-certified physicians who specialize in the field of radiology. The company’s goal is to elevate the level of care for patients through advanced diagnostic imaging. The purpose of the network is to transfer large medical images quickly across a number of MPLS-VPN connections.
The U.S. based radiology company described here is a small and highly dedicated team of radiologists and technical staff providing medical imaging services for local area hospitals. The group has a diverse collection of state-of-the-art equipment including nuclear medicine devices such as MRI, PET and SPECT scanners as well as photoacoustic and thermal imaging devices.
Patients are also scanned at their local hospital locations and after processing and initial physician evaluations are completed, the resulting digital images are often sent using an MPLS-VPN network carried over Ethernet to other local hospitals for additional physician analysis and diagnosis.
According to the company’s IT Manager, the network itself isn’t very complicated, but it faces challenges a typical enterprise network would never see. Besides stringent government requirements for patient privacy, network speed and reliability take on a new meaning when delays or failures can sometimes affect a patient’s chances for survival. To add to the challenge, nearly 90% of the network’s traffic consists of very large file transfers between hospitals and even to a physician’s home office.
WAN Appliances Didn’t Help
It would appear that dedicated Ethernet links would ensure high speed and full control of available bandwidth. Yet performance related issues did persist. Using WAN optimization appliances seemed to be an obvious solution based on industry performance claims attributed to conventional networks. However these devices rely primarily on compression and de-duplication to achieve good results. The company’s images were already highly compressed and because the transmissions were never repeated, the WAN appliances had nothing to optimize, essentially providing a near zero gain in performance despite the high cost of installing appliances at every remote location.
A New Software Solution
There were also some unknown issues with network performance that the team could not resolve with the WAN optimizers. They needed the ability to see what was actually happening inside their network devices—not unlike the detailed scans they capture daily of the human body.
Through his connections in the local IT community, the IT manager discovered LiveAction software from ActionPacked Networks. He was intrigued with its rich visualizations, realtime situational awareness, and the ability to configure Cisco routers on-the-fly, so he obtained a trial version for his laptop.
Within a few minutes after installation, LiveAction generated a topology view of the network showing live flows from beginning to end across multiple devices and interfaces. Clicking on different network elements revealed successively deeper and richer levels of information. For the first time he could literally see what was happening on and inside his network. And what he saw was surprising—almost disturbing.
A Taste of Their Own Medicine
Two well known methods for enhancing performance are increasing bandwidth and prioritizing traffic using quality of service (QoS) policies. Increasing bandwidth was not an option for the company since they were already using a high performance MPLS managed service. Because of this, they felt no need to implement QoS. Their assumption was that, aside from transferring large medical images, the remaining traffic on the network was of such low volume and low priority that it would have a negligible effect on performance. Also, the closed network of interconnected hospitals made the network appear immune from outside congestion and interference.
However, just as an MRI or PET scan can reveal unknown human anomalies, LiveAction uncovered several issues that would have been impossible to predict and see using conventional monitoring methods. In one case, the IT manager was summoned to work in the early morning hours by an urgent call stating that the network had slowed to a near shutdown. Using LiveAction, he quickly discovered what was choking the network and preventing file transfers from taking place. The culprit was an ad-hoc peer-to-peer network using his MPLS-VPN network for transport. While this was permitted for normal low volume traffic, the network clients were downloading extremely large files and were apparenty using the network for unauthorized, non-medical reasons.
The IT manager was able to resolve this very quickly using two methods that were unavailable before LiveAction. First he used LiveAction to identify the offending devices and applications and he could literally watch this traffic from source to destination. Then, using LiveAction, he applied a new QoS policy to the network routers. With a few adjustments he was able to create and apply the new policy that essentially blocked the rogue applications and ultimately shaped the traffic for proper prioritization to ensure maximum performance while still permitting other legitimate traffic to traverse the network. Then, using LiveAction’s QoS monitoring and NetFlow visualizations, he was able to confirm that the policies were indeed working and the network was healthy and operating at peak performance.
In addition to healing its network, the company now has a robust and highly visual QoS solution with the flexibility to reprioritize traffic on the spot whenever requirements change at different times of the day or night.