YP Solutions, located in Metairie, Louisiana, was located just outside New Orleans and less than 1 mile from the now infamous 17th Street Canal. YP Solutions provided services to Yellow Page directory publishers with the largest share of the company’s revenue coming from converting paper based directories to online web based directories. To deliver this service, YP employed web developers, graphic designers, copy writers and search engine optimization specialists. This team of 40 professionals worked in the company’s primary office in Metairie, and YP’s revenue stream was tightly tied to the team’s daily production.
Business Continuity Challenge
New Orleans area businesses faced the sternest test of Business Continuity Planning in the form of Hurricane Katrina. Most, especially those without a cloud disaster recovery plan, failed.
Even nine months before Katrina, in October 2004, National Geographic Magazine published “Gone with the Water,” an eerily accurate but purely hypothetical story about the risk faced by New Orleans. The story detailed what might happen in the city if a major storm were to strike:
“Thousands drowned in the murky brew that was soon contaminated by sewage and industrial waste. Thousands more who survived the flood later perished from dehydration and disease as they waited to be rescued. It took two months to pump the city dry, and by then the Big Easy was buried under a blanket of putrid sediment, a million people were homeless…”
Sadly, the article foretold real events and accurately the depicted of the impact on the lives of residents.
Although the lessons for Business Continuity Planning have not been as widely covered in the media, YP Solutions use of Global Data Vault’s Advanced Data Protection solution can serve as a model to guide businesses in their Business Continuity Plan development.
Let’s examine the history which led to YP’s excellent state of readiness.
Hurricane Ivan was the strongest hurricane of the 2004 Atlantic hurricane season. Ivan reached Category 5 strength and became the sixth most intense Atlantic hurricane on record. At its peak in the Gulf of Mexico, Ivan was the size of the state of Texas.
At 2 a.m. on September 16, 2004, Ivan struck Orange Beach, Alabama, leaving New Orleans unscathed, but in the days leading up to landfall, with New Orleans in the crosshairs, YP Solutions made the decision to evacuate the city. Their evacuation was haphazard, disruptive, and costly; management immediately decided to develop a better Business Continuity Plan.
Building the Solution
For YP Solutions to achieve continuous operation during and after a disaster, the following information system components were deemed critical:
- Microsoft SQL Server based database
- Microsoft Exchange based email
- Production desktops including Adobe Photoshop, InDesign, Dreamweaver, and certain proprietary applications
YP Solutions Information Technology and Production management worked with Global Data Vault to develop a cloud disaster recovery plan which would encompass all the platform systems and functionality needed for normal daily operations.
Global Data Vault used images created with VMware Converter to build replica copies of each of the production servers as well as templates for production desktops. These were installed and hosted inside Global Data Vault’s VMware Infrastructure. A replication solution was then prepared. This would keep data synchronized between YP Solutions and GDV’s data center during normal business operations.
In the final days of implementation, a plan was made to deliver YP’s production data to GDV’s Dallas data center by physically transporting a RAID array containing the data. On August 1, the schedule was set for Labor Day Weekend – one week after Katrina. But when Katrina entered the Gulf of Mexico and took aim on New Orleans, the schedule was accelerated. As the RAID array left YP’s office, it got wet from the first drops of rain preceding the storm.
Late on the night of August 28, 2005, the data arrived in Dallas. As Katrina moved inland on the 29th the data was installed in Dallas, all while YP’s staff dispersed across the United States to the safety of friends’ and relatives’ homes. By the time the data was installed in GDV’s data center, an immediate disaster declaration for the company went into effect.
Four simple steps were required to essentially re-locate all systems operations to the GDV data center:
- Public IP addresses were assigned to the SQL and Exchange Servers
- Internal and external references, such as DNS and MX records were repointed from the primary IP addresses to those on the Advanced Data Protection platform
- The template production desktops were cloned and connected to GDV’s Terminal Server
- And the production staff were given Virtual Private Network (VPN) logon credentials
On completing these steps, YP Solutions was ready to resume normal operations and provide services to its customers across the country. The company had become “virtual” overnight.
This map prepared by NOAA, shows the depth of the flooding in New Orleans. Much of this water stood in the city for two weeks or more. City services were completely suspended. There were no police, no fire department, no electricity, no running water, and no sewerage service. All businesses were closed. Residents were not allowed to return to their homes our businesses for three months.
On the Ground
These images from the days following the storm convey the degree of the damage – but hardly explain the wide ranging scope:
The entire city of New Orleans and over 1 million people were evacuated for 3 months. YP Solutions was able to carry on normal operations while most businesses in the city struggled greatly – and many simply failed. In November 2005, Local Matters acquired YP Solutions. The transaction had been planned well before Hurricane Katrina. Jobs were saved and the economic value of the business was preserved.
Few disasters reach this scale, but YP Solutions’ excellent preparedness and Global Data Vault’s Advanced Data Protection solution serve as an excellent model for Business Continuity Planning for any organization which relies on Information Technology. To learn more about Global Data Vault’s Advanced Data Protection product, see Global Data Vault Advanced Data Protection or contact GDV at Advanced Data Protection Information Request.