With essentials such as the EHR and billing systems leading the way, healthcare organizations are ever more reliant on their data and computing horsepower for ongoing operations. A server failure or inaccessible data can bring things to a halt until services are restored.
That being the case, there are a number of things a healthcare organization should do in order to both mitigate possible instances of downtime and to recover from data loss. Ralph Wynn of Melville, N.Y.-based data protection firm FalconStor, offered five recommendations for dealing with crashed servers, downtime and data loss.
Know your systems. Patient records, charting systems and imaging systems all need to be analyzed and viewed as parts of a whole. Some parts will naturally be more crucial than others, and a successful recovery plan will acknowledge that. "The first thing you have to understand is what the rank and order [of systems] is, what is most critical to business," and then have a plan for how to bring those pieces online, said Wynn. Taking a hard look at each component of the IT infrastructure and knowing what matters most is one of the first solid steps to take. "Without coming up with some type of business analysis plan, you're not even prepared to know what would happen" should downtime occur, explained Wynn.
Have a plan for recovery timeline. How many applications does a system use? What are the most crucial things that need to be online as soon as possible? Wynn said that after a thorough system audit, the next step is to "look at a timetable as to how readily available an application needs to be." Just knowing which systems are important is not enough. "You need to sit down and write out the steps for getting data moved over," he said. By having a framework in place dictating what is going to be recovered and when – and how long the overall process will take – it takes a large amount of the guesswork out of a recovery. An upshot of this is that an organization will know how much to pony up in advance, as a timeline addresses a lot of the cost factors. "Procuring the secondary site, bandwidth for moving data over daily, staffing," Wynn listed as examples. "Costs [can] become inordinate."
Avoid points of failure. It's not uncommon for there to be bottlenecks in a system's recovery process. If the actual IT system is robust, make sure the people behind it are too, advised Wynn. In many organizations, there is often only one person who knows passwords to various routers, holds administrative privileges in certain systems or even knows where specific backup tapes are located, he continued. As much as possible, Wynn said it is important to avoid a scenario in which an organization's uptime is dependent upon "one person who holds all the keys to the kingdom." Even in a smaller care center, if only one person does IT and isn't onsite, their absence can severely hinder a recovery effort.
Understand the true nature of "recovery." Rebooting a system that has failed entails a lot more than just flipping a switch and powering on new or restored hardware. Technology advances at a rapid pace, and the chances are good that a fail-safe system or newer hardware brought in to replace old failed units will pose operability problems. "Do I have the same equipment?" asked Wynn. "How do I put everything back on, especially if it's a different setup than the [equipment] that went down? What's the quickest way to get it operating?" Think, for instance, of a mission-critical backup tape drive failing and a new one replacing it. Does it talk to the systems it's backing up? Is it compatible with older tape stock? "This drive supports three different types of tape, oh, it turns out that the firmware is old," said Wynn, outlining some of the things that could go wrong.
Virtualize. Virtualization is the process of running an application remotely on a computer, and accessing it on any device that can connect to that computer. Think of it like cloud storage, but for actual software. Wynn said that virtualization enables a healthcare organization to quickly migrate its software resources without hardware being a problem. He noted that with virtualization, he can "quickly take everything that's running on that server and move it to a virtual machine [with] no loss of productivity." In addition to being able to bounce back quickly, virtualization makes testing much cheaper. Instead of paying a team to travel to an offsite testing location, an organization can set up a virtualized copy of its systems and test that with no worry about what will happen to their main systems. From a disaster recovery perspective, Wynn said, the key element of virtualization is "removing dependency on the hardware."
