text:
Planning in Other Industries (10)
How Any Organization Can Survive Through a DisasterWritten by Judy K. Bell
Many organizations believe that if they protect their data processing systems, stock extra supplies, train employees on basic first-aid, and try to minimize their hazards, they have done all they can do to be prepared for a disaster. The rest is up to mother nature.
In reality, every organization can do a great deal to ensure its survival regardless of whether it is a business, school, or public agency, but the planning must be completed before the disaster strikes. Depending on the size of the organization, this effort can be as simple as assembling in advance key information about the location, determining what resources will be needed, inventorying what back-up communications are available, and determining how damage information will be assessed.
A critical ingredient in any planning effort is identifying a way for people to get in touch with their families through pre-established contact points. Until they know their loved ones are safe and cared for, they will not be able to contribute to the restoration effort.
In large organizations, efforts must be linked together at every level during the planning to ensure consistency. Otherwise, several groups will be relying on the same resources, causing chaos during the disaster. There are five distinct organizational levels which must coordinate their individual plans. The type of information required at each level is uniquely different. For ease of illustration, I will use the terms associated in the business environment.
The first level of planning is at the company level. Here, policy issues must be documented such as how employees are going to be paid, what food and water supplies will be stocked, and how the company will set up shelters. Protection of the company officers must be decided, as well as provisions made for officer succession. Emergency response planning must be identified, determining how they will access damages and injuries, as well as restore the business. Protection of vital records, and determining the need for mutual aid agreements are just two more of the many decisions which need to be made before the disaster.
Second, every department must document their plans. What do they plan to initially do if the disaster strikes during normal work hours? What will they need to do differently if it occurs out-of-hours? What steps do they need to take in advance to protect their employees? What procedures do they need to define to identify the extent of damage and to activate emergency communications links? What arrangements do they need to make with their suppliers?
Third, emergency centers need to be identified where key people will receive the reports of damage and decide how to best redeploy their resources. They must be stocked with the supplies they will need to function, and the roles of all who will report there need to be specified. During the real disaster, others may be required to perform functions which they are not familiar with because the designated person is injured or unavailable.
The fourth level of planning occurs at each location. Emergency response teams need to be trained. They must know how to conduct search and rescue operations, and how to perform first-aid. The public agencies have made it clear that they will not be able to respond to all of the emergencies which will arise during a disaster, so the degree of skills the emergency response team possesses could be the difference between life and death for many people.
Finally, every work group must know what they are to do, who they are to contact, how they will get in touch with their family members, where the supplies are which they need, and many, many, more specific pieces of information. If planning has been performed at the other levels of an organization, but it has not been translated into the information pertinent to every work group, then it will be worthless.
Judy K. Bell is Executive Director of Disaster Survival Planning (tm).
This article adapted from Vol. 2 No. 4, p. 31.
Disaster Recovery in the Movie Industry: MCA Inc./Universal Studios PlanWritten by Margo Young and Richard Newman
Part 1
Like many major companies, Universal Studios spent over a year defining their vital business functions and making adequate preparations to protect them. This company's business functions are different from most; they consist of not only movies and television programs, but a theme park and flourishing tourism business.
Because of the nature of its business, MCA INC./Universal Studios has many unique consideratons in designing a disaster recovery plan. the Studio, in fact, is a separate city unto itseld, with its own fire department, security force, construction crews, transportation systems, restaurants and hotels. All planning resources must be carefully coordinated in order to serve most efficiently the employee and guest population, estimated between 50,000-100,000 at any given time on the 420 acre site.
Their current disaster recovery plan was two years in development and is now in a three-year implementation process that will be constantly updated with annual drills and the continual training of new team members. It was developed through Corporate Risk Management and then turned over to the Senior Vice President and General Manager of Universal City Studios for implementation.
MCA's plan accounts for the importance of offsite backup facilities. Film, tape and critical records are backed up and stored out of state in underground vaults. Furthermore, the plan includes a separate emergency radio network system, and their telecommunications is working on a corporate-wide 800 number. Software packages are backed up and stored offsite for resumption. Contingency planning software has been purchased to record and store all inventory of personnel information and critical records in a free-standing portable computer for use in the E.O.C.
In addition to recognizing the importance of asset protection and business resumption, Universal Studios stresses life safety as its highest priority. Employee safety considerations are comprehensive, as all response team members receive training in first-aid, C.P.R., fire extinguisher use, search and rescue and other emergency skills. In the event of an emergency, the studio would be able to maintain approximately 72 hours of food stuffs and bottled water for stranded employees.
The concern for personal safety is not limited to employees, but extends out to the general public. The Studio currently employs 12 registered nurses for the Studio and Tour operations. Training is conducted on a regular basis and all tram drivers and tour guides are encouraged to obtain C.P.R. training. C.P.R. training is also available to individual employees on a regular basis.
The studio also has abundant and diverse transportation resources: four wheel drive vehicles, maxi-vans and crew cabs for personnel, heavy equipment for removal of debris, water trucks for water storage, fuel trucks for fuel storage, generator vehicles, and honey-wagons.
Another rather inadvertent asset that MCA has, simply by virtue of the industry, is the diversity of languages spoken by the employees.
Tour Operations reports that 30 to 40 tour guides speak more than one language. These tour guides represent interpreters for five languages.
MCA INC./Universal Studios is constantly updating and testing its plan.
The employees, as well as the surrounding community (including the local fire stations and other rescue forces) are involved in occasional mock diasters, which can be quite realistic and convincing with the special effects and stunt people.
The test begins with the initial disaster and concludes, not when the rescue operations have been demonstrated, but only after the "victims" have been transported to the hospital.
Part 2: The recovery plan unfolds
As one of the biggest names in the movie industry, part of MCA Inc./Universal Studios' business is to thrill its audiences with catastrophes of epic proportions on the screen. However, while it is easy to call it a day after concocting an inferno for next summer's blockbuster release, it may not be so simple to resume normal business activities the day after a real-life disaster. MCA, like several other organizations discovering that proactive planning is the key element of corporate survival following a disaster, is currently in the process of implementing a new plan for the Studio. Part 1 took a cursory look at some elements of MCA's disaster recovery plan. Part 2 will provide a more in-depth examination of the plan and some of its prevention measures.
Should any disaster strike, an advisory panel of the Senior Corporate Vice Presidents and CEOs from each of MCA's critical divisions would lead the recovery effort. Dan Slusser, Senior Vice President and General Manager, oversees the plan as a whole, directing and coordinating the recovery effort of the entire organization. The recovery plan itself is then divided into two major functioning units. The data processing plan, handled by George Brenner, Vice President and Director of Corporate Information Services, focuses on business resumption for all data processing-related aspects of the organization. The protection and recovery of the physical facilities, as well as coordination of the corporate-wide recovery, is handled by Paul Holehouse, Director of Corporate Emergency Preparedness.
Another facet of MCA's implementation process involves the construction of a new facility that will serve as the primary meeting location for employees and management following a crisis. Plans are underway to replace the current Emergency Operations Center this coming October with a new 8,000 square foot facility. In addition to all of the equipment that is now in the temporary facility (which includes property-wide utility maps, an uninterrupted power source, a backup emergency generator, incident status boards and portable radios), the new one will have a state-of-the-art training center and applications for the critical information. The mobile unit, or secondary location, is situated in the parking lot area. Should the prime location be immobilized, this site, sufficiently equipped with essential supplies and materials, can be utilized. The unit consists of tents and tarps, which would be more likely than a building to survive what is the biggest threat to MCA, an earthquake. Backup data centers are also located in this area.
In preparing for a disaster, Universal Studios is divided into three teams: the studio itself, the tour, and the amiptheater. Within each of these teams are eight divisions to address the following areas:
- first aid
- food and water
- resources (including transportation)
- damage assessment (including risk management and the actual recovery effort)
- security team
- communications team
- emergency operations team
Each division, which on average consists of 12 people, receives the same training in each of the three departments. Thus, if a partial disaster should occur and only one department was impaired, people from the other teams could easily step in and assist in the recovery process. Furthermore, each of these individual teams tie in to one central, comprehensive corporate plan. While each unit is specifically trained to focus on its own unique hazard area, they can also all function together as one unified group for a large-scale catastrophe.
MCA is also secured outside of the organization by an agreement with their critical vendors, approximately 50 in number, guaranteeing that MCA will be supplied with computers and computer paper, plywood, food, water, and other essentials that may be needed during a disaster.
MCA's corporate headquarters includes over 200 buildings that are divided into assessment and mitigation inspections. In addition to retrofitting old equipment, another aspect of the new plan entails stricter earthquake mitigation procedures for all new construction and modelling. For example, all cabinets must be anchored to the walls, sway bar protection for tape and film storage racks is mandatory, and all large-plate glass areas are coated with mylar to protect from broken glass.
MCA also provides its personnel with easily accessible phone numbers to contact in order to either prevent or alleviate the potential devastations of a disaster. The 5,000+ phones on the property are all equipped with the following information:
- building address and location
- a direct line to the fire department and paramedics, which are located on MCA's property
- a security number
- a safety hotline, which directs all calls around the clock to a team that will investigate any questionable event or situation on the premises
Although the studio covers over 400 acres of land and 200 buildings, the recipient of any phone call can quickly pinpoint the caller's origin on a digital display. In addition to these numbers, MCA has an active rolodex file of all contact numbers for critical service areas.
Like most other businesses, Universal Studios incorporates offsite storage into the overall disaster plan to complement the recovery process. Their facility, inconspicuously located a safe distance from the data center, is a reinforced underground site. While it predominantly serves to store duplicate data, bonded courier service is also available from the offsite location as scheduled and on an emergency basis 24 hours/day, every day of the year. The facility will also handle the movement of tape files to and from their designated disaster recovery site.
A weekly system backup that is taken Sunday evenings is sent offsite early Monday morning, and on-site nightly application data backups are taken and also sent offsite the following morning.
MCA Inc./Universal Studios' disaster recovery plan is making great strides as the implementation phase progresses at a steady pace. Although the plan should be completed within the next two years, the people at MCA realize that implementation is merely the first phase of an ongoing process that involves updating the plan, equipment, and facilities, as well as frequently testing and maintaining the plan to ensure that their organization could survive any disaster.
"The most critical aspect of the entire plan," adds Paul Holehouse, "is the management commitment to continual employee Team Training and to motivate employee preparedness at home as well as at work." And as the creators of many of your favorite disasters on the big screen, MCA knows--while a disaster scenario can provide added drama and thrill to movies and television, it can also result in the downfall of a business that is not prepared.
Part 3
Storage of Emergency On-Site Supplies at MCA/Universal Studios
Once the disaster has occurred and the dust starts to clear, an organized plan must take over to provide the needed emergency supplies to assist the disaster teams with search and rescue, food and water distribution, and the construction supplies necessary to secure critical areas and barricade condemned buildings. Priority of resources will change depending on the time of day the disaster occurs. Flashlights, batteries, portable toilets, lumber, plywood, plastic sheeting, food, water, raingear, blankets, and pre-packaged hygiene kits will be just the start of possibly a three to five day shelter management plan. (A year after the San Francisco earthquake, there are still shelter camps for displaced persons in the Bay area!)
As part of MCA's Corporate Disaster Plan, a reserve of emergency items, such as food, water, and temporary housing, is available based on a given percentage of the maximum occupancy of the entire facility. The management at MCA has to consider visitors and guests as well as employees and, consequently, must anticipate the potential need for an undetermined amount of extra emergency supplies. To account for the added numbers, MCA has devised a plan that will first utilize those resources available on a daily basis. Once those are depleted, people can tap into supplies furnished in an emergency on-site facility, which is to be used exclusively for disaster scenarios.
In case of a large-scale disaster that may debilitate vital resources and facilities, MCA has a 40 foot Matson trailer that they have stocked with emergency supplies. It is specially designed to sustain severe shaking damage that an earthquake would probably incur. Internally, Universal personnel has battened down all equipment with earthquake bracing, straps, and other securing devices to ensure that everything will stay intact if the facility is moved or shaken. This trailer, which has been remodeled by Universal's prop-makers and special effects crew to accommodate the particular needs at the Studio, can be moved to a different location by crane or forklift.
After purchasing the trailer, the Art Department designed an accessible storage facility that clearly displays all the equipment to be used for distribution. This was done by constructing the interior of the facility with plexiglass and wire baskets that are all clearly identifiable for immediate access.
In the assessment of needs, MCA staff determined that it would not be necessary to store food in this facility since it is currently stored in several locations to stock the various eating establishments on the property. Should a disaster strike, all perishable foods would be distributed first. Once this supply was exhausted, the food and water team members would allocate and distribute the stored food. Water is stored in 55 gallon drums on wooden pallets for easy distribution both in the storage facility and in multiple locations on the property.
Communication equipment inside the storage facility includes bullhorns, beepers, a full emergency radio network with remote suitcase repeaters, portable radios, and cellular telephones. The facility also houses a police and fire scanner and will function as a ham radio operators dispatch center.
The storage facility will primarily be used for distribution of equipment to disaster response teams and is equipped with a backup generator with enough fuel for the first 24 hours. In addition to the main disaster supply storage facility, smaller storage areas are strategically placed around the facility with team members instructed on how to use and distribute the contents within.
Additional supplies in the emergency facility include employee coveralls, boots, breathing apparatus, medical supplies, extrication equipment, a vault containing duplicates of keys to critical vehicles needed in the event of a disaster, emergency lighting, generators, emergency radio network equipment, building utility plans, and manuals identifying the operation of all equipment housed in the facility.
Universal Studios Hollywood recently hosted the annual conference for the American Society of Amusement Parks Security & Safety and, as part of the agenda, conference attendees toured the supply trailer as shown above. The facility has been toured by the local and state Fire Department and local disaster preparedness organizations. The emergency supply storage facility is on the Universal Studios Hollywood tram route, a location that is both easily accessible and also free from overhead obstructions or other barriers that may inhibit immediate entrance to the trailer. All tour guides have incorporated a brief description of the Corporate Disaster Plan as part of the tour experience.
As most people are now well aware, a disaster plan is a continuing process. The development of new technologies and lessons learned from disasters worldwide are continually incorporated into MCA/Universal's expanding disaster recovery plan. The best defense in the event of a major disaster is the education of the public and employees in emergency preparedness. The Studio hopes that this supply storage facility and its contents will show both the staff at Universal as well as the general public a positive visible approach in the overall effort involved in creating and implementing a disaster recovery plan.
Margo Young and Richard Newman are staff writers for the Disaster Recovery Journal.
This article adapted from Vol. 3 No.2, p. 29; No. 3, p. 16; No. 4, p. 50.
Saving the BooksWritten by John Morris
The same disaster can, and does, strike a business office, a museum, an architect's office from time to time. The lowest level collects the water and suddenly comes the realization that you have 50 or 100 file cabinets loaded with important records and legal papers, all swimming in dirty water; something has to be done quickly, but what?
These water emergencies are the business of the drying contractor. He flies to the scene, assembles a work force, and hustles everything into cold storage. Freezing stabilizes the damage, prevents mold from forming, and buys time to get the drying operation set up. This may take days or weeks, or even longer. In Los Angeles, after the fire of April 29, 1986, 700,000 books went into the freezer warehouse and stayed there for 2 1/2 years. This was the background for the drying job that set new standards, using a new process that takes half the time and produces a better result.
Late in January, 1989 a load of books was delivered to the Los Angeles Public Library processing center, marking the end of the largest book drying job ever undertaken. In 4 months 560,000 books water-damaged in the 1986 fires were processed in a sophisticated vacuum freeze-drying operation at Saugus, California. Commencing in late September, 1988, 4 days every week a load of books went from the freezer warehouses to Saugus, and 4 loads of dried books went from Saugus back to the library. The driers ran day and night, and the job was finished well within the 6 months time stipulated in the contract.
The 560,000 books represented 80 per cent of the total number of books wet in the 1986 fires. The major share of the lot went to Document Reporcessors of San Francisco, while Airdex of Houston contracted for the remaining 20 per cent. The wet books had been blast-frozen immediately after the fires and placed in warehouses normally used for shrimp and other perishable food items. When the library was finally able to locate processing and storage space, the Los Angeles City Council set aside $2,800,000.00 for the restoration of the books.
For several years vacuum freeze-drying has been considered the preferred method of restoring wet books that have been kept frozen to stabilize damage and prevent mold from forming following a fire or water emergency. The ice-covered books are placed in a vacuum chamber and subjected to vacuum and vapor form (sublimation) and is collected on chilled panels in the form of ice.
Eric Lundquist, founder and president of Document Reprocessors and his associate, Robert Ritchie, have lately refined this conventional drying routine with advances in both technique and equipment and have given the name THERMALINE to the new process. It shortens the typical drying cycle for heavily water-damaged materials, and eliminated the need for refreezing water vapor on cold surfaces within the vacuum chamber. What formerly required 10-15 days can now be done in a 4-8 day period. The new method also provides a mechanism for straightening boards and text blocks during the drying cycle, minimizing the distortion previously encountered during the drying, and useful also for reprocessing books which were dried previously, but are still in need of straightening.
Other refinements provide for enhanced distribution of low heat uniformly to the books in the vacuum chamber. Lundquist has found that when the inside of a frozen book is warmed only slightly above freezing temperature it tends to return easier and in a shorter time to its original alinement, and with less stress to the materials. The effect achieved is somewhat like the benefits of annelaing in working with metals, and is referred to by this term by Lundquist and Ritchie. This method is one of several in the process of patenting.
Having bid on a contract for drying 700,000 books, more or less, in 6 months, Document Reprocessors leased an industrial building in Saugus, 33 miles from central Los Angeles. They installed 5 vacuum chambers, 3 of which are semi-trailer mounted cylindrical tanks, 8 1/2 ft. across and 45 ft. long. The other two are smaller stationary units.
A typical journey for the books began at 7 AM with the loading of palletized books into a refrigerated truck at the freezer warehouse. There were 50,000 boxes of books to be moved eventually, each box containing 15 books, more or less. A single truck would carry 8,000 to 10,000 books. Within an hour after leaving the warehouse the truck was unloaded at Saugus. The books were unpacked, categorized according to the degree of wetness, and prepared for the vacuum chambers. Workers at tables placed books of uniform size spine-down in aluminum tray, 10-15 books in a tray 18 inches long and easily handled. They then placed rigid aluminum plates between books at close intervals, and around the lot wound two or more wraps of elastic 'bungie cord' under tension. Each tray was then loaded on a metal cart roughly 6ft. by 4ft. and 6 tiers high, and the carts were then positioned in the vacuum chambers.
Inside the chambers workers connected the warm water circulating system to metal tubing secured to the underside of the metal shelves of each cart. The vacuum pumps were activated to reduce the internal pressure. The warm water began circulating, and the heat was conducted from tubing to shelves, from shelves to trays, and from trays through the aluminum plates into the damp books. The water temperature was monitored and controlled to limit the book temperature to approzimately 100 F. As the books dry, shrinkage occurs, usually 10 per cent to 20 per cent, and the binding of elastic cord contracts, applying constant pressure toward one end of the tray, combining with the rigid aluminum plates to help straighten any distorted covers or text blocks.
When the drying cycle was complete the racks were removed from the vacuum chambers, the books inspected for humidity (7 per cent maximum) and packed in new boxes for the return trip to the library. Boxes were placed on pallets, shrink wrapped for protection during shipping and handling, and loaded into the truck. When unloaded at the library's processing center in Los Angeles they would be put through cleaning, evaluation and inventory, then be moved along to the temporary location of the Central Library.
Eric Lundquist and Robert Ritchie are graduate mechanical engineers. Mr. Lungquist founded Document Reprocessors in 1979. He had seen the need for improved drying methods when he worked as an insurance claims adjuster. His first major contribution to the science of book drying was probably designing a vacuum chamber on wheels and making it available wherever there was a major water disaster involving books, fine arts, documents or business records. Document Reprocessors acted as consultants to the library staff during the emergency removal of wet books after the fire disaster of April 29, 1986 at the Los Angeles Central Library.
On behalf of himself and Robert Ritchie, Eric Lundquist has applied for patent rights to be assigned to Document Reprocessors for several innovations of the THERMALINE process in the drying of wet books and materials, specifically claiming the following:
1. A method of applying uniform heat and controling the temperature throughout the pressure/time cycle for drying wet books and materials.
2. A method of providing nearly constant compression force on wet books, such that, as shrinkage occurs during the drying cycle, the cover boards, text block and other materials are provided a rigid and straight surface to conform.
3. A method to straighten books that, when previously dried, exhibited severe distortion (but) can be recycled in the described equipment and restored to a near original straightness.
4. A method which is suitable for a very high rate, mass drying of books. The fundamental equipment and process beyond one tray is identical. Only the capacity of the heat sources and vacuum pump(s) needs enlarging.
With their vacuum chambers on wheels Document Reprocessors have been able to move promptly to any part of North America to respond to water emergencies. In 1985 they went to Dalhousie University in Halifax, Nova Scotia to dry 90,000 books wet in a fire in the Law Library. In November of that year, when a disastrous flood left Roanoke, Virginia under 14 ft. of water they were able to save millions of documents and business records. And when an arson fire struck the records of the Supreme Court of New Jersey, and the drying had to be done on site, Eric Lundquist took his Arctic 550 freeze drier across the country in a 747 aircraft and dried the documents. In addition to the western base at Saugus, California he has an eastern base at Middlesex, N.Y. and two seagoing driers.
References on salvage of wet books and materials:
An Ounce of Prevention - A Handbook on Disaster Contingency Planning for Archives, Libraries and Record Center, Toronto, John Barton and Johanna Weilheiser, 1985;
Salvage of Water-damaged Books, Documents, Micrographic and Magnetic Media, San Francisco, Eric G. Lundquist, 1986;
Salvage of Water-damaged Materials, Washington D.C., Library of Congress, Peter Waters, 1975;
Planning Manual for Disaster Control in Scottish Libraries and Record Offices, Edinburgh, National Library of Scotland, Hazel Anderson and John McIntyre, 1985.
Written by John Morris, Professional Services in Loss Control.
This article adapted from Vol. 2 No. 4, p. 12.
Disaster Recovery Planning in Electric Utilities: A Message for the Planning ConsultantWritten by Samuel Mullen
Many readers may be intimately familiar with the aspects of creating plans for data processing in banking and insurance industries. This article will introduce the types of plans which are familiar to contingency planners in electric utilities. It will also describe what utilities expect from outside consultants.
Planners are aware of the variety of contingency and disaster recovery plans which must be considered by various industries. We know from experience that fundamental decisions must be made with respect to the types of plans needed, what details must be included, and who should be involved. Electric utility planners must also make these decisions, and help to coordinate the many departments and sections that may be called upon to respond.
Utilities must continually find ways to maintain a demandingly high level of service reliability. The lights must stay on. Hospitals and medical centers must be restored from an electrical outage so the medical staff can continue to care for patients. Downed wires must be isolated and de-energized quickly to minimize danger to people. Nuclear power plant incidents must be dealt with swiftly, and with technical accuracy. Customers must receive information about electrical emergencies as quickly as it is humanly possible to assess damage. High customer electrical loads must be met on demand in the heat of summer and the cold of winter, or the utility and customers may suffer a blackout that can take several hours to restore. In a few short hours, a hurricane can force to the ground an electrical system that took nearly a century to build. For reasons like these, electric utilities must develop and maintain disaster plans.
Utilities can either develop these plans on their own, or use a consultant for the task. In some cases, the utility will employ a full-time planner to oversee the development of plans, maintain them, and hold regular tests to help ensure their usefulness. In many cases, however, utilities, need quite a few people involved, since there can be several plans installed throughout the company. Several plans cross departmental and sectional boundaries and must therefore be closely coordinated. For example, the data processing plan might require action from people who are also listed in another plan. Since all companies have limited personnel resources, the chances are favorable that several groups will be named in more than one plan.
What types of plans might you find in an electric utility? This will vary from company to company, but some common plans are listed below:
- storm plans--plans that determine how the electrical distribution system will be restored after a severe storm.
- nuclear plans--used for nuclear power plant emergencies.
- public relations plans--procedures to deal with an inquisitive media (newspapers, radio, TV coverage).
- telecommunications plans--restoring communications when they are interrupted.
- systems restoration plans--plans for recovering from a blackout of the electrical system, particularly the high-voltage transmission grid.
- materials management plans--emergency procedures to assist the organization with procurement of materials and services.
- power plants--plans to address emergencies in non-nuclear power plants, e.g., coal, oil and gas fired generation.
- data processing--DP emergency plans that include migration to a backup site to run mainframe applications.
- facilities management plans--plans for migrating to an alternate site, given the normal site has been destroyed or made unusable.
Other plans may also be found, depending on the planning activity going on inside the utility. Every planner knows (or should know) that the need for plans must be 'sold' to management. This process can often take years, and the subsequent development of plans can also take a lot of time. For this reason, the planner must stage tests and exercises often, not only to keep the emergency organization 'well-oiled,' but to use results to point up the need for further plan development.
When a consultant looks to assist a utility with plan development, he or she should be reasonably familiar with utility operation before coming in. Utilities are closely regulated. Managers and supervisors are often people with technical backgrounds, such as engineers. With the recent downsizing of many firms, managers are strapped for available time. Organizations have been flattened to the point where many managers are doing work previously done by subordinates. While most utilities are financially stable, it can take time for consultants to get paid. Budgeting has been decentralized in some companies, but after budgets are submitted, they are often cut, and consulting services may be the first to go. Utility budgets reflect cost-cutting attempts that are taking place inside many organizations. This is done so that their energy prices remain competitive.
Don't be surprised to find that the utility does all its planning in-house. Utility people pride themselves on knowing how a power system operates and knowing what to do when something goes wrong. Very often, they aren't willing to go into a lot of trouble to educate you, as a consultant. Before you come in, review, if you can, the various systems that utilities use to serve their customers. When it comes time to discuss the details of what they need from a consultant, this may help you gain credibility, and perhaps, a contract.
When you do come in, you may find yourself sitting across from a small group, whose combined utility experience might be close to a century. Tell them precisely what you can do a cannot do for them. As in other industries, utility people are proud of the number of years they have been in the business. They also respect people with considerable expertise in other areas, but plan to provide them with proof. Have a list of past utility clients ready for review. You might want to request that the utility jost provide you with a brief written description of what the utility is looking for prior to your meeting. This should save everyone some time.
Utility people are like everyone else when it comes to demonstrations. If you have a colorful, well-planned presentation, the people will appreciate it. The technical types will want details, and if you're selling software and services, a demo disk may be requested.
Remember that utilities, like most everyone else, have been "bitten" by consultants that have promised them the world and not delivered. By the same token, utilities have a great deal of respect for consultants who deliver what they promise. As a consultant, make sure you know what the client wants, then deliver whatever is promised.
Samuel Mullen is Restoration Planning Coordinator for Atlantic Electric, an electric utility serving southern New Jersey.
This article adapted from Vol. 3 No. 2, p. 26.
Planning For Recovery In a Manufacturing EnvironmentWritten by Saul J. Swartout and Pat Moore
We all know that a well designed and practiced contingency plan is the best insurance against financial peril for any corporation or institution with a future. As we prepare for natural and man-made disasters we must understand that industrial and manufacturing losses have very diverse cause profiles ranging from hurricanes, tornadoes, flooding, fires and explosions, to hazardous contamination, collapse of storage racks, vehicular impact, vandalism and malicious mischief etc.
Hazard analysis, loss control and fire prevention in the industrial setting need to be carefully designed to address not only the facility, but the specific processes, equipment and hazards of each particular type of operation. The National Fire Protection Fire Analysis & Research Division latest statistics tell us that:
"Fires in industrial and manufacturing facilities account for 11.8% of non-residential structure fire, 29% of associated direct property damage, 18.4 % of associated civilian deaths, and 22.5% of associated civilian injuries.
Within the industrial and manufacturing facility category, the leading properties are metal or metal products manufacturers (21% of industrial and manufacturing fires, 11% of property damage), wood, furniture, paper, or printing products manufacturers (20% of industrial and manufacturing fires, 11% of property damage), and agricultural farm products facilities (13% of industrial and manufacturing fires, 6% of property damage)."
Quite often there is an assumption that corporate insurance programs cover the majority of any financial loss that might occur. While this is often the case for small and/or simple losses, it very rarely is the case when large losses occur that disrupt normal business operations. As such, it is important to understand what is the true nature of cost of risk to an organization. While the term cost of risk is often used by today's risk management world, it is often less understood by disaster recovery or business recovery personnel or by their senior management. In a broad sense cost of risk is a way of measuring a company's degree of risk by examining several of its worst possible loss scenarios. Once identified, these scenarios should be communicated to senior management so they too can begin to see and support the value of risk management and disaster recovery planning coordinating efforts. Failure to support these efforts can directly affect the company's bottom line.
A business impact analysis is a proven method of determining this cost of risk by identifying the interdependency of manufacturing operations as well as the relationship between manufacturing and other business functions. The analysis should also identify recovery time frames and priorities, potential sources of severe business interruption and cost-effective recovery strategies.
Once this collective information is gathered, analyzed, and the results presented to senior management, agreement should be reached on the recovery strategy and plan development should begin. It will be important to integrate a planning tool with the planning process to facilitate not only plan development, but for maintainability, flexibility and viability of the plan. This automation will also result in a significant decrease in the amount of funds and personnel resources to keep the plan current on an ongoing basis, and should be able to provide instantaneous computerized reporting information at the time of a disaster.
In an industrial or manufacturing environment, a disaster recovery, business resumption and business continuity plan must include:
- An Emergency Response Plan - addressing fire brigade, evacuation, health and safety issues and environmental concerns.
- A Facility and Equipment Restoration Plan - addressing damage assessment, restoration /replacement recommendations, emergency procedures, corrosion control, cleanup, salvage and reconstruction.
- A Product Fulfillment Plan - addressing meeting customers orders with replacement or alternative products.
- A Crisis Management Plan - to expedite the decision making process during recovery and restoration, as well as addressing public affairs and stockholder issues.
Critical priority issues such as restoring manufacturing capability, recovering processes and equipment, replacing product to meet customer demand, meeting contract deadlines, maintaining specialized workforce skills, controlling environmental and security issues, adhering to regulatory compliance schedules, and insuring positive public image must all be addressed in the planning process.
Also important to business continuity are the identification and protection of customer and vendor relationships and special supplier partnerships, as well as production management and process control.
In addition, it is important to consider other strategic corporate business continuity issues beyond just getting the plant back in operation.
These could include continuing to get product to market, producing excess capacity versus just-in-time inventory, buying replacement product externally for resale, or the possibility of shifting product from other markets to protect your best market.
Recovery of the physical plant depends not only on the degree of structural damage, but the presence of any routine or non-routine contamination resulting from the cause of loss, such as the by-products of a fire.
For example, the most common routine contaminant is hydrogen chloride, generated by de-hydrochlorination of PVC plastic exposed to heat.
As stated by BMS CAT's, Dave McDaniel, "In a fire PVC converts 60% by weight to hydrogen chloride gas, which in the presence of water, forms hydrochloric acid. If you put an acid and a metal together, metal salts will form, causing all forms of corrosion.
Electronics in a manufacturing environment are the most susceptible to damage due to corrosion because of the thin metalization paths on the circuit boards, and irreversible damage can occur within days."
A thorough site assessment should be performed as quickly as possible to determine if there is corrosion, and what the levels are.
Emergency procedures should include the removal of the contaminants through proper and specific cleaning protocols, and humidity control.
Non-routine contaminants could include polychlorinated byphenyls (PCBs) asbestos, lead, cadmium, mercury, other heavy metals, and any combination of the combustibles and reactives consumed in the fire, explosion or loss scenario. It may also be necessary, depending upon what was in the facility, to identify those items that would require lab packing, which is the containerization and removal of like hazard classes of material such as all flammable liquids, and all corrosives.
Profiling and disposal will then be necessary. Depending upon the experience level, knowledge and certification of your internal fire brigade, it will be important to pre-qualify and identify your external emergency response resources in your plan.
With regard to your manufacturing equipment, you must give consideration to such factors as a robot operated production line.
The lead time involved in getting replacement items, your vendor required cleaning protocols for recertification, is restoration more cost-effective than replacement.
If your plant operation is driven by information systems or applications, or by a combination of people with support from information systems? Will your plant come to a complete standstill if you lose IS?
Vital records recovery is another critical planning area. Your plan must identify and address the archival records, inventory and retrieval systems, legal retention schedules and protection of such vital records as product specifications and/or formulations, including Material Safety Data Sheets, equipment design schematics, equipment operating and repair reference manuals, facility blueprints, contracts, compliance documentation for EPA, OSHA, DOT, etc., plant management and reporting data, and insurance documentation.
You must give consideration to the possibility of losing this vital information in the fire or explosion, or not having access to it because the building for example, is contaminated.
If you lose this data, how will you continue your operations, and will your insurance coverages refinance the creation of this data?
Most insurance coverage will cover the cost of the physical media format, such as blank tapes, and it will pay the cost to transcribe from the previous generation (hopefully safe in offsite storage) to the blank tapes. Most coverage stops here.
Some coverages will include the extra expense you incur to recreate or regenerate the data lost, but you must specifically request and purchase this coverage.
This example is only one of many which illustrates why it is important to coordinate insurance coverages and disaster recovery and business continuity planning.
It is also important within the planning process to understand what is normally covered within a property insurance program and what is not normally covered. The basic principle upon which property policies respond to losses follows a chain rule. This rule is as follows:
In the event of:
1) A Discreet Event of Physical Loss or Damage, ie; leakage from a tank that takes place over an extended period of time such as spillage from filling the tank up, is not a discreet event, but if the tank splits a seam and ruptures - that would be because it has a defined beginning and an end.
2) To Insured Property
3) From a Risk of Loss
4) Where No Exclusion Applies
5) Which Causes an Interruption of Business Operations.
Then the policy covers:
6) The Defined Loss
7) For the Defined Indemnity Period
In short, special attention needs to be paid to the type or cause of loss, the property involved, and the extent to which profit and continuing fixed charges (business interruption) is covered.
While the degree of coverage varies by policy, even under so called All-Risk policies, all covered losses need to be fortuitive in nature.
As such, losses caused by corrosion, deterioration, rust, wear and tear, inherent vice and the like are not covered losses.
For example, there would be no coverage for stock that is susceptible to light damage that is left in open storage and it discolors or experiences changes in its other characteristics.
Many plastics, pharmaceuticals, as well as various fluids fit into this category.
Along the same lines, loss attributable to manufacturing or processing operations which damage the materials while in process are also not covered.
Additional exclusions often include: loss caused by war, radioactive contamination, faulty workmanship, and contamination.
Additionally, most policies do not cover loss of use of the property unless there is physical damage.
For instance, loss of access / egress to property caused by flood, a government agency ordinance, damage to transportation systems usually does not constitute a loss.
Many types of property are traditionally excluded from property insurance contracts.
Some of them can be covered under specialty contracts; however, these specialty policies often do not cover the potential revenue produced by the property.
Examples of property usually excluded include: precious metals, accounts, currency, valuable papers and fine arts, land and water.
While the amount of uncovered loss may be minor under the above categories, the extent to which it can impact revenues may be significant.
Most insurance contracts, if endorsed for business interruption coverage, cover the loss of profit and fixed charges only for the period of time needed to replace the damaged property.
Therefore, the amount of loss created by the loss of revenue from cancelled contracts or contract postponement can be multiples of what an insurance contract actually covers.
There are numerous other items that can create financial loss that are not covered. They include: loss of employees, loss of management time, loss of key suppliers, loss of market share, loss of reputation, etc.
Depending on the size, complexity, and specific loss characteristics, the above referenced non-covered costs are estimated to be from 1 to 50 times greater than the covered costs.
True protection requires a thorough understanding of risk exposure, implementation of loss prevention and loss mitigation measures, proper insurance coverage, facilitating a business impact analysis, developing and implementing recovery and business continuity strategies, creating team action plans, testing and maintenance of the program, senior management support and funding for all of the above.
Saul J. Swartout is Director, Arkwright Disaster Recovery Services, and Manager, Arkwright Boiler Machinery Services in Malvern, Pa. Pat Moore is Vice President - Business Continuity Education for Strohl Systems, in King of Prussia, Pa.
Library Disaster Recovery PlanningWritten by Jean Uidenich
In the aftermath of the April 29, 1986 and September 3, 1986 Los Angeles Public Library fires, many library owners, operators, and librarians were asking themselves what they would do if disaster struck their facilities leaving them with thousands of water-soaked books, documents, and files. This same question was asked after fires at the Klein and Temple University Law Libraries in 1972, the National Military Personnel Records Center in 1973, the University of Toronto Fleming Library in 1977, the San Diego Aerospace Museum and Library in 1978, and the Dalhousie University Law Library in 1985.
Fires aren't the only catastrophes that can strike library collections. The Florence, Italy flood in 1966, the Corning Museum and the New York and Pennsylvania Library floods in1972; the Cornell University Library and Northwestern University Library floods in 1976; the broken water pipe at Stanford University in 1981; the damaged fire hydrant in 1986at Pepper dine University Library, and the earthquake that occurred near UC Santa Barbara in 1978 are all examples of catastrophic events that have seriously affected library operations.
Many lessons were learned from these disasters, the most important being the need for libraries to have written disaster plans that also address recovery measures. The success of any salvage operation was discovered to be directly dependent upon preparations made in advance of the incident.
The disasters experienced by libraries and museums have led to the development of successful salvage and recoveryprocedures and techniques. At first it was thought that heating books in an oven would evaporate the water away butthis process was quickly abandoned. Today, the most successful results have been achieved by air drying, vacuum freeze drying, and vacuum thermal drying. However, there is no single best method. Each situation must be evaluated individually depending upon the degree of damage and type of materials involved.
The combination of good preplanning and proper salvage techniques will give libraries the opportunity to recover. The following two cases are studies of the extremes.
On November 12, 1971, the Irvington branch library in Fremont, California experienced a fire. Frantic efforts were made to salvage wet collections but there were numerous delays in obtaining the authorizations to act. The books were finally judged beyond recovery and bulldozed into a sanitary landfill.
On the other hand, a book, The Merchant's Almanac, rested for100 years in deep water in the wreck of the Bertrand at the bottom of the Missouri River until it was eventually recovered by the Smithsonian Institute and triumphantly restored by vacuum drying. These two incidents emphasize the need for expert assistance which can contribute significantly to recovery success. Most libraries do not have an experienced book salvage expert on staff. Therefore, the preplan should contain a list of names and telephone numbers of salvage experts to call upon in an emergency.
A successful recovery plan should address five action phases:
After a building is declared safe to enter, the first step in the recovery process is to assess the type and degree of damage.
It is crucial at this point that librarians or bibliographers familiar with the library collections assisting the damage assessment. It is also important to be aware of insurance company requirements.
For example, the insurance carrier may want to appraise the damages and direct the salvage process.
The preplan should establish priorities for materials to be saved first and those to be discarded, taking into consideration the intellectual value versus the artifactual value. Photographs and notes should be taken during the damage appraisal phase.
Vital records and very valuable books and materials should be located. Then, priorities and plans for salvaging can be made.
If priorities cannot be established during the initial appraisal process, it is best to take the conservative route, earmark the questionable materials for salvage, and make the final decision when time is not critical.
Mold growth can be expected to begin within 48 hours unless the environment of the flooded area is stabilized.
Consequently, every effort should be made to reduce high temperatures and to provide ventilation. Generally the following considerations should be kept in mind:
- Damp books in temperatures above 70 degrees F. and humidity above 70% will be subject to mold growth.
- Undisturbed archival files will not be so quickly attacked by mold.
- Very wet books, or those still submerged in water, will not develop mold.
If temperature and humidity are a problem, then steps must betaken to control mold growth. During warm weather, temperatures can be reduced by turning on air conditioning. In cool weather, heat inside the affected areas should be turned off.
If mold growth becomes a problem, then is may be necessary to utilize fungicidal fogging. However, this should never be undertaken without proper professional supervision. These measures must comply with federal, state and local hazardous substance regulations.
Determining mitigation strategies is often most difficult. After establishing priorities and deciding what books are to be salvaged, it is necessary to decide to what degree the materials should be salvaged and to select what methods will be used. It is helpful to formulate a general plan regarding salvage. If time is short, it may be necessary to transport all materials to freezers in order to buy time for more rational decision making at a later date.
Experience has shown that freezing water-damaged materials at temperatures below zero degrees Fahrenheit, preferably-20degrees F., will stabilize mold growth and facilitate salvage efforts. Although freezing does not remedy mold damages, it does not harm the materials further. Evidence has shown that wet material can be held in the frozen state for a long as six years without further deterioration.
The three most common salvage methods today are air drying, vacuum thermal drying and vacuum freeze drying. The determination of which method to use depends upon many factors, including the amount of damaged material, the extent of damage, the type of material, the type of paper and print, bound versus unbound, and other variables. This is where the expertise of a professional book conservator is important.
Implementation can require the most planning. A preplan is of enormous benefit during this phase. Quick action must betaken after the mitigation strategy phase. This is especially important to the success of the total salvage operation. Great volumes of water-damaged materials may have to be removed in as little time as possible.
It is essential that people be selected and designated in advance to supervise salvage operations. These people should be given the authority to make on-the-spot decisions without obtaining the approval of management who may not be available or have the expertise to make a technical decision.
Depending upon the extent of the loss, large numbers of people may have to be assembled to begin book removal. Assistance can come in the form of community volunteers, as was the case for the Los Angeles Public Library which used1,500 volunteers, or local temporary help can be hired. In either case, an extraordinary amount of organization is required.
Once the salvage processes have begun it is necessary to continuously evaluate the results. If the materials have been frozen, there is more time to make clear decisions regarding the salvage methods (i.e., vacuum thermal versus vacuum freeze drying). Sample numbers of books can be dried and carefully examined to determine which method works best. The salvage and restoration operation may be completed in several days depending upon the amount of damaged material or it may endure for months. Stanford University's Meyer Library flood in 1978 wetted approximately 52,000 books. Salvaging and restoration took over six months but only 34books were finally discarded.
The tremendous task of reshelving will have to be considered. Inventory may have to be taken while the books are still frozen in order to establish the order of salvage.
Preplanning will plan a significant role in the degree of success that is achieved. The job won't be easy but preplanning will help to put order and direction into a situation that could otherwise develop into a major disaster.
This article was written by Jean Uidenich of M&M Protection Consultants in Los Angeles, California.
This article adapted from Vol. 2 No. 3, p. 22.
Measuring the Pulse of Disaster Recovery at Bethesda Memorial HospitalWritten by Staff Writers
As computers become increasingly integrated into the day-to-day operations of hospitals, senior managers are asking that contingency plans be in place to ensure that the required computer capacity be continuously available -no matter what happens.
The emphasis on computer disaster recovery planning in the health care industry is no accident. It follows the strategic integration of computers into virtually every area of hospital operations. The American Hospital Association is considering standards for computer contingency planning in health care.
The process of ensuring continuous computer operations is as complicated an issue as has ever faced hospital management.
Consider how Bethesda Memorial Hospital, a progressive 362-bed acute care hospital in Boynton Beach, FL, is approaching the question of computer operations contingency planning. The hospital is a little more than halfway through implementing a comprehensive disaster recovery strategy designed to keep its extensive information systems intact in virtually any situation.
Absolutely Indispensable
'We have installed computer systems and integrated them so deeply within the hospital, that rather than being supplementary to the functioning of the hospital, they are absolutely indispensable,' says Charles W. Stewart, Vice President of Information Systems. 'Eventually, you realize you can't function very well without the computer resource.'
As with most hospitals, Bethesda Memorial awoke to the need for a disaster recovery plan when its external auditors identified the exposure. Ernst & Young, its New York-based auditors, noted that the more deeply Bethesda Memorial integrated computer systems in its ancillary departments, the more it pointed to a major weakness in the event of a major long-term outage.
In its 1989 report to management, the auditors urged the hospital to address the situation by beefing up the physical security of the computer room and implementing and periodically testing a formal computer disaster recovery plan.
The first step was relatively easy. Bethesda Memorial rebuilt its data center and installed a limited access system complete with a tracking mechanism to keep a log of who is in the data center. The enlarged data center is equipped with a multiple zone Halon fire extinguishing system and other alarms.
The second step, the disaster recovery plan, was much more complicated. The first task in protecting any asset is to describe that asset. The biggest challenge for Stewart and his staff was the security audit to specifically define the investment in information systems at Bethesda Memorial and how to prioritize them in the context of their recovery in the case of disaster.
To get a handle on this task, the hospital found that The Living Disaster Recovery Planning System (LDRPS) from Strohl Systems, Tampa, FL, allowed the staff to establish recovery plans quickly for the entire hospital to ensure continued operations in the event of a disaster or disruption.
Computers at Three Levels
Data processing at Bethesda Memorial is integrated into the operations of the hospital on three levels. Primary processing of hospital orders and financial systems is based on an IBM 3090 15OS running the MVS operating system. The system primarily supports a Hospital Information System database developed by Shared Medical Systems (SMS), Malvern, PA. The data base covers the full spectrum of clinical and financial applications. The systems are processed in an online, real-time environment supported by 200 terminals and 73 printers distributed throughout the hospital.
A middle tier of specialized minicomputers supports specific hospital departments. Digital Equipment Corporation VAX mini-computers located in the data center strategically support operations in the Laboratory, Pharmacy, Radiology, Pathology, and Nuclear Medicine. A Data General MV4000 minicomputer supports the Medical Records department to process a sophisticated Diagnosis Related Group (DRG) coding system.
Various departments have also come to rely on personal computers networked with each other. The hospital has five such Local Area Networks in place. The Executive Network connects executives with each other and the mainframe. The Public Relations Network supports desktop publishing as well as a public physicians referral application. The Nursing Network provides full word processing support to each of the nursing administration offices as well as a nursing staff interface to the mainframe. The hospital telephone system is administered by a Communications Network that provides, among other features, a telephone answering service to client physicians. The Physician Staff Office Network provides word processing, committee reporting, and appointment monitoring in support of staff physicians.
As dependent as Bethesda Memorial is on SMS for its application software, it is not surprising that the hospital approached the vendor for assistance in the area of disaster recovery. Bob Johnson, SMS Manager of Support & Professional Services, notes that SMS is not in the contingency planning business.
'This was a client need that we wanted to satisfy. We do know our applications and the hospital's operating environment,' he says. In response, SMS combined its own expertise with that of Stewart's staff and an independent specialist in contingency planning.
The hospital conducted an audit of all its information resources, finishing it in October. One critical step was to determine which systems were primary--critical to the operations of the hospital--and which were supplementary. Of course, the primary systems had to be recovered first. The decisions were not always obvious.
'We were surprised in some instances,' Stewart recalls. 'Some applications, although not determined as primary systems, were part of a critical path.'
An analysis revealed that these applications had to be recovered in order that a dependent primary application be recovered.
At this stage of planning, Bethesda Memorial also specified the outage window at 36 hours. Every organization has to determine at what point it will declare an emergency and transfer computer operations to a point outside the organization. For the hospital, the outage window is 36 hours. If the hospital expects an outage to last more than 36 hours, it will affect the off-site plan immediately.
The hospital also used the planning process to establish standard off-site storage requirements for programs and data as well as identifying the human resources required to recover data processing.
As expected, Bethesda Memorial encountered a number of problems. One of the most significant was that all of the hospital's computer terminals were hardwired to the mainframe through an IBM Series I front-end processor. Because there was no remote terminal controller, there was no way to get the data out of the hospital. The hopital is now installing the remote terminals and controllers to give it online, off-site processing capabilities.
Software security becomes more complicated with remote processing functions. In response, the hospital's auditors have recommended that the data center install a comprehensive software security system like IBM's RACF or Computer Associate's Top Secret. The hospital is also negotiating a contract for a hot-site computer center it can occupy in case of disaster.
Not Just Data Processing
LDRPS manages the recovery of not only data processing functions, but other hospital functions as well.
Bethesda Memorial uses the system to handle risk management at various ancillary departments.
'[Our plan] does an outstanding job in automating both data center recovery planning as well as end user departments and, therefore, is a fully functional corporate recovery planning system,' Stewart notes.
Other business units or departments within Bethesda Memorial use LDRPS to generate a department-specific disaster recovery plan. Such plans are much easier to keep up-to-date.
All the individual plans are automatically rolled up into one master disaster recovery plan.
'These individual plans are supplemental to the data center's security plan but are just as important to the functioning of the hospital,' he says.
The PC-based plan system allows hospitals to:
- Centralize and consolidate information concerning operations, system, and health care resources in one location.
- Establish central source of information for all data center equipment and processing, including an evaluation of equipment and capacity usage.
- Facilitate the update of the planning data base as changes occur.
- Audit all operations.
The hospital's plan consists of four integrated components: planning, action, project management, and recovery administration management. Static data related to disaster recovery is maintained in the planning component, dynamic data in the action component. The project management component summarizes information from the action module to produce project management charts. The Recovery Administration Management component provides planning and control utilities for the disaster recovery coordinator.
If a tropical storm--the most likely of disasters that could befall Bethesda Memorial--hit the Boynton Beach community, the facility that would be called upon to provide emergency health care services to victims will not be a victim itself.
'We have confidence in the disaster recovery system we are implementing,' Stewart says. 'Bethesda Memorial is halfway there. Thanks to the self-guided planning process, we have accomplished the most difficult half of the process: documenting procedures, saving the data, and establishing a set of work plans for everyone required in the recovery process.'
The major part of the process left to be done is the testing of the plan with mock disaster drills. Exercising the system on a regular basis ensures that the plan is sound and that necessary updates are implemented. With its disaster recovery system in place, Bethesda Memorial Hospital can ensure the people of Boynton Beach that the hospital is well protected in case of disaster--no matter what happens.
This article adapted from Vol. 3 No. 2, p. 42.
This was no special effect... Disaster Planning Works!Written by Paul Holehouse
At 7:15 p.m. on the evening of November 6, 1990, a security guard on duty in the dispatch office of Universal Studios received a telephone call that would set the stage for a real-life drill of the MCA Corporate Disaster Plan. With hosts of politicians watching election returns just up the hill at the Universal Sheraton, a structure fire was reported on Brownstone Street in the back lot facades of the Studio. A call was placed to Los Angeles County Fire Department Station 60 on Universal's own lot and the 911 system was activated.
Inside the 420 acre property, the fire and intense heat twisted 12-inch steel beams and marched methodically along the back lot, consuming building facades, vehicles, camera equipment, and anything in its path. Rows of classic cars in pristine condition fed the flames. Millions of dollars worth of movie props and lights along with period costumes rich in motion picture history went up in smoke.
Courthouse Square, the site where Michael Fox went "Back to the Future," suffered extensive damage; only the clock tower could be saved. Brownstone Street, which played host to "Dick Tracy," was now just a memory. Streets where Robert Redford and Paul Newman conducted their elaborate "Sting" were destroyed in a shroud of smoke.
King Kong, host to more than four million tourists a year, was barely spared from the blaze, which climbed the hill leading to the Tour Center Complex. Disaster Response Team members established a fire line just below Battlestar Galactica using garden hoses, studio water tanker trucks, fire extinguishers, or anything they could find in an effort to protect that attraction and the garage which houses the fleet of quarter-million dollar trams that lead tourists through the back lot daily.
Fire department personnel, accompanied by Emergency Response Team Members of the Emergency Services Department, responded to the disaster armed with property maps identifying water sources, fire suppression equipment, utility shutoffs, keys for access gates, etc. A fire department incident command post was established just below the fire line in an effort to coordinate firefighting equipment.
When I arrived on the scene, I opened the emergency supply storage and stationed a team member at the door to distribute supplies to firefighters and employee volunteers. I then headed for the Emergency Operations Center (E.O.C.) to set up an incident command. Power that had been lost through burned transformers was restored when Resource Team members installed the backup generators to the E.O.C. Once we were out of the dark, the plan progressed at full speed ahead.
Disaster Recovery Teams
Universal Studios is divided into several teams to handle different areas in the event of a disaster (see Part II, Jul/Aug/Sept, 1990). The decision to have these teams translated into a smoother recovery process and contributed to Universal's successful performance.
Communications Team
Members of the Communications Team were dispatched to Conan Theatre perched atop the Universal Studios Hollywood complex to establish the repeater for the Disaster Plan radio network. This repeater would enable radios to be distributed to team members. Dan Slusser, Senior Vice President and General Manager of Universal City Studios, found out firsthand what all the emergency equipment was capable of doing under fire. Dan, who also serves as Director of the E.O.C, mobilized a task force of key executives responsible for film vault recovery, transportation, and Studio operations. Security and public relations deployed their personnel in a support effort to the Incident Command Center located in parking lot "W."
Resource Team
Resource Team members, equipped with radios, flashlights, and protective clothing, began to distribute equipment to the ever-growing army of concerned returning employees. Patrols were established to keep flying embers from straying into as yet unaffected facades. The fire extinguisher and fire hose training of over one thousand employees over the past two years paid off.
Emergency Operations Center (E.O.C.) Team
E.O.C. Team members began monitoring the radio communication to track the arrival of more than 100 pieces of firefighting equipment. The fire's path was traced on large wall maps in the E.O.C. and executives monitored the news media from televisions powered by the backup generator.
Food and Water Team
The Food and Water Team arrived and provided sandwiches and water to firefighters positioned around the property. Coffee brewed on backup power was more than welcome as the recovery process continued into the early hours of the following day. Mobile hot kitchens were established by the American Red Cross to feed tired and hungry emergency personnel.
Emergency Response Team
An Emergency Response Team member was dispatched to the Fire Department command center, armed with a radio to establish communication with the E.O.C. Other Emergency Response Team members, equipped with a 150 gallon capacity pumper, were deployed to patrol the property and identify any developing hot spots while monitoring the County Fire frequency.
Late in the evening when the fire had been contained, members of the media were escorted into the Emergency Operations Center to receive statements from MCA Executives brought into the E.O.C. Conference Room.
An estimated four acres of history valued at more than 25 million dollars had been lost in flames. However, only one injury--a minor burn--was sustained by a firefighter. The orchestrated efforts of the Disaster Plan and firefighting units had saved valuable property and limited injuries despite the 50 m.p.h. gusts of wind.
The Disaster Plan at Universal Studios had been tested and passed the test. The plan continued into the next week of cleanup, using the gloves, dust masks, goggles, and hard hats that were all on hand in the specially designed emergency supply storage facility (see previous article).
Debriefing meetings were conducted in the E.O.C. with all team members to discuss how operations had been conducted and could be improved.
All plans look good in a manual, and paper table-top drills help, but to see the training pay dividends by allowing the company business to open on time the next day...that makes all the effort worthwhile. The management support, long before the disaster, and personal involvement of top executives made the difference.
Paul Holehouse is the Director of Corporate Emergency Preparedness for MCA Inc. He has over 15 years in the loss prevention field, coordinating safety, fire prevention, disaster planning and environmental compliance.
This article adapted from Vol. 4 No. 1, p. 34.
Automate Your Manufacturing... But Beware of Disasters!Written by Raja K. Iyer, Ph.D., CDRP
Manufacturing strategy is widely recognized as an integral part of a firm's overall corporate strategy to gain and retain competitive advantages.
Advances in computer- and communications-based technologies have contributed to an explosive growth in automated approaches to the implementation of manufacturing strategies within organizations.
The tools of advanced technologies, properly and effectively utilized, have harnessed a wide range of benefits for firms, including reduced costs, increased productivity, greater flexibility, higher quality, etc., enabling firms to improve their competitive position in a number of ways.
These benefits and the resultant competitive advantages can be directly traced to increased reliance upon information and communications in real-time, which have effectively removed time and distance as barrier to competition.
However, these computer- and communications-based manufacturing solutions have also created, or have the potential to create, significant risks due to disasters. Consider the following scenarios:
- Company A, a major manufacturer of snack foods, has all of its four large warehouses completely automated with bar-code technology for inventory control and robotics-based warehouse handling. One of its large warehouses in the east coast suffers a major disaster due to a once-in-a-100-years snow and ice storm, which has knocked out all power to the warehouse and has prevented loading, unloading, and driving of its trucks due to icy roads!
- Company B, a major light equipment manufacturer with full CIM implementation and just-in-time and flexible manufacturing systems, is in the path of a major hurricane of the Hurricane Andrew variety!
- Company C, a major mail-order company, has its fully automated warehouse in Connecticut but all its mail-order operations in the World Trade Center building in New York City!
- Company D, a small manufacturing firm with CAD, CAM, MRP II, and FMS systems fully integrated into a CIM/ CIE concept, suffers extensive damage from a tornado that has ripped its roof apart and flooded the facilities due to torrential rain!
While these are imaginary scenarios, disasters of these variety are indeed real. While most businesses protect themselves with business interruption insurance, can these companies actually survive such disasters?
Can they recover from the loss of data and property effectively and in reasonable time to meet their customer orders and demands? Can they continue to be competitive in the market place, or are they likely to lose customer confidence slowly and painfully in the future?
While daily backup and offsite storage are essential procedures for firms dependent on automation of their manufacturing operations, can these firms recover and resume operations effectively and efficiently within a specified time period?
Recovering Technologies Only means Recovering Facilitators
There can be no doubt that data processing and communications technologies are merely facilitators of efficiency and effectiveness in transacting business; after all, businesses thrived even before the days of the computers and the telephones.
What is important to recognize, however, is that these advances in technologies have certainly resulted in changing the fundamental ways in which business is conducted in modern times. Advances in computers and communications technologies have led to changes in business processes and business functions to such an extent that these changes have also resulted in significant transformations in organizational, structural, strategic and competitive environments faced by organizations today.
Despite such marvels of computers and communications technologies in conducting the 'normal course of business,' recovery and resumption of these technologies during a disaster merely means one has recovered the facilitator, but what about the facilitated? That is, the business processes and the business functions which are facilitated by these technologies?
In disaster recovery and contingency planning, it is imperative that we keep the business process/business function recovery and resumption in focus at all times. For instance, company A with its automated warehouses, Company B with fully automated manufacturing, the mail-order Company C with its operations interrupted by bombing, Company D devastated by a tornado, and other small firms as well as medium and large companies may pride themselves on their ability to recover data from their offsite storage facilities. However, what good is the data if there are no concomitant plans to recover manufacturing and business functions, such as making the deliveries from the warehouses, taking mail-orders, and serving customers from an alternate site. In most, if not all, cases, data happens to be static while business processes / business functions are dynamic, implying that data recovery gets an organization back to the point when the disaster struck, whereas recovering business functions alone can take the organization forward. In disaster recovery and contingency planning, then, one must not only consider technologies (the facilitators) as key areas of concern for recovery and resumption, but also the business processes and business functions (the facilitated). Disaster recovery and business resumption plans can gain significantly from this view of technologies as facilitators of business processes/business functions.
The Need for Disaster Recovery and Business Continuity Planning: The Case of the Manufacturing Automation
Increasingly, manufacturing automation is not just seen as a strategy for gaining competitive advantages but also for the very survival of businesses. As a matter of fact, the concepts of computer integrated manufacturing (CIM) and computer integrated enterprise (CIE) have led to unprecedented revolution in the manufacturing sector of developed economies on a global scale, perhaps even comparable to the earlier industrial revolution. The CIM and the CIE concepts have fostered a growing and still-emerging focus on both upstream and downstream aspects of manufacturing. Upstream applications include design engineering, CAD/CAM, robotics, and flexible manufacturing systems. Downstream manufacturing considerations utilizing CIM concepts include MRP II, KANBAN, Just-in-Time or JIT, and shop floor scheduling and control. Increasingly, the CIM concept is not only being viewed as a means to solve problems which inhibit excellence in manufacturing but also as a foundation upon which to build a computer integrated enterprise (CIE) which can provide both intra-organizational and inter-organizational integration of information and production systems.
One of the key ingredients in the CIE concept which is most relevant to disaster recovery and business continuity planning is that intra-organizational and inter-organizational integration include both information and functionalities. This means that CIE seeks to integrate not only "islands of information automation" but also "islands of functionality automation." This implies that if and when a disaster strikes a manufacturing organization that is part of a CIE, an unprepared firm is likely to suffer loss of information as well as the ability to function. While automation of information flow can be restored with traditional backup and recovery procedures, albeit to a limited extent, restoration of manufacturing functionalities, especially in a CIE environment, needs significant attention from the personnel, property, and information perspectives.
Even with the available disaster recovery and business continuity planning methodologies and disaster recovery vendor services, disaster recovery and business continuity planning for manufacturing functionalities suffers from a lack of much needed prescriptions for mitigating risks from natural and man-made disasters. After all, recovering computer and communications systems in a remote "hotsite" alone is not enough to keep the product pipeline flowing. This shortcoming in corporate-level disaster recovery and business continuity planning can be especially detrimental in a CIE environment where inter-organizational integration leads to interdependence of firms along the value-added chain (see Figure 1 p. 59) within and across industries. In short, firms no longer have disasters in isolation; other firms, both upstream and downstream along the value-added chain, are likely to suffer from a disaster, as presented earlier in the hypothetical situations.
This means that any firm along the value-added chain may be required to have disaster recovery and contingency plans not only under the scenario that the firm itself suffers a disaster, but also under the likelihood that its suppliers and/or customers along the value-added chain has a disaster, thereby resulting in business interruptions along the value-added chain.
Disaster Recovery and Business Continuity Planning in the CIE Environment: A Value-Added Chain Approach
Consider firms that are integrated in a closed-loop manner into a CIE along the value-added chain in an industry, as shown in Figure 1.
In this simplified value-added chain representation of a CIE, if one or more firms located in one node suffers from a disaster, firms in other nodes, both upstream and downstream, are likely to suffer the consequences of the same disaster. For instance, if a firm in the warehousing/distribution node is shut down due to a major, prolonged snowstorm, its impact is likely to be felt by the customer/market node downstream as well as the production, production scheduling, and other nodes upstream.
While some of the impacts are tangible and easily identifiable and quantifiable, most impacts are intangible and difficult to quantify.
Examples of intangible impacts due to disasters include loss of customer confidence and satisfaction and concomitant loss of customers and market share. In an integrated CIE environment the risks of such intangible losses are even greater. Such losses would include loss of supplier confidence, resulting in the loss of suppliers-based competitive advantages; loss of competitive ability to thwart substitute products and services; and eventual erosion of barriers to entry into the industry due to exposure to unmitigated tangible and intangible losses from disaster.
Disaster Recovery and Resumption for Manufacturing
It is clear from our discussions thus far that disaster recovery and business resumption for manufacturing firms take on added dimensions. First, the recovery of data and communications alone is insufficient because these are simply facilitators of the business processes and business functions.
Second, when firms are nodes in a value-added network of a computer integrated enterprise (CIE), disaster recovery must consider at least two dimensions: a firm's own primary recovery plan when a firm has a disaster of its own, and a secondary disaster recovery plan when another firm in the value-added CIE network has a disaster along with ripple effects along the CIE network.
Finally, any firm, be it manufacturing or service-oriented, must recognize that the recovery of data and communications is only static, that is the recovery and resumption of technologies merely takes the firm back to the state when the disaster struck; further recovery and resumption involves business processes and business functions which are facilitated by the technologies.
It is imperative that firms, which have already automated or planning to automate most, if not all, of their manufacturing operations, consider the impact of disasters on their operations.
Ample challenges exist for contingency planners and vendors alike to develop and implement specific disaster recovery and business resumption options as firms worldwide advance toward the goals of agile manufacturing and global competitive advantages through continued manufacturing strategy automation.
In this article we have presented a discussion of a notable, recent phenomenon in our manufacturing organizations, namely the automation of most of the functions and processes in a manufacturing organization.
While such automation facilitates agile manufacturing and global competitiveness through the disappearance of time and distance barriers to competition, increased dependence on advanced computer and communication technologies also creates significant problems and challenges in the context of disaster recovery and business continuation planning for these types of organizations. The challenges multiply further when these firms are part of a value-added network in a computer-integrated enterprise environment.
We also suggested frameworks within which these problems and challenges may be analyzed; these frameworks included a value-added CIE network, along with a table discussing the impact of natural and man-made disasters on business functions and processes in manufacturing organizations in the CIE network.
We then proposed alternative disaster recovery and business recovery strategies for the various technologies (the facilitators) and the business processes / business functions (the facilitated). We believe that our proposals in this article constitute a starting point for a careful examination of disaster recovery and business resumption planning considerations in automated manufacturing environments.
Raja K. Iyer, Ph.D., CDRP is associate professor of information systems and management sciences in the College of Business Administration at the University of Texas at Arlington.
This article adapted from V8#3.
Contingency Planning for the Healthcare IndustryWritten by M.J. (Doc) Trujillo-Fernandez
Information processing activities is not just a recordkeeping aspect of an organization. Resources gathered via the automated information processing system is the most valuable asset a healthcare institution may have.
Every data center is subject to certain interruptions of service. They may involve an equipment failure that results in a few hours of downtime, a fire that destroys the data center itself while your operations is still conducting business as usual, or a major disaster that affects the entire community. A Disaster Recovery Plan is designed to reduce the consequences to an acceptable level, should you lose your data processing capabilities. It is important to realize that your plan must respond to the full range of potential disasters, up to major regional catastrophies. In addition, the Disaster Recovery Plan, as a protective measure, must be comprehensive and yet flexible enough to cover the entire data processing operation or any part of it.
The development of a Disaster Recovery Plan must be treated as one of your most important organizational projects. It requires the commitment and direct involvement of senior management. The project leader, or Disaster Recovery Consultant, is usually someone who is familiar with security or operations. The individual designated should develop a project plan, complete with estimates of cost and time involvement in the total effort to produce a workable Disaster Recovery Plan. A project of this nature involves the efforts of many people and a considerable amount of time. However, because it is critical to the survival of your organization after a single catastrophic event, it deserves the full support of top management.
How much could a disaster cost your healthcare institution if you did not have aviable recovery plan? In addition to the immediate costs that result from personal injury or death and destruction of your facility, hardware and software, a data processing interruption would at a minimum affect:
- Ancillary departments
- Patient billings
- Patient information
- Laboratory tests
- Cash receipts
- Accounts receivables
- Inventory records
- Collection records
- Payroll/Personnel files
- Financial statements
- Purchasing
Business Interruption and Extra Expense Insurance can offset some of these losses initially. Mutual assistance may mitigate some of the effects at the outset. However, you would surely reach the limits of your insurance policies and far outstrip emergency backup capabilities in very short order. The immediate financial losses could very well be exacerbated by legal ramifications and consequential effects of your business interruption. In fact, an article in the June 7, 1982, issue of Computer World points out: 'fewer than 7% of all companies that experience computer damage to their DP operations are in business five years after the loss, according to a widely circulated insurance industry report'.
Records have shown that it can happen to us, today, tomorrow or five years from now. As we stand back and truly assess what we stand to lose if one or more contingencies occur, we have embarked on identifying and recognizing the inevitable risk.
Loss due to computer frauds were estimated to be 30 billion for 1985. This figure does not even include losses due to:
- Hardware failure
- Fire and water damage
- Power outage
- Sabotage
The key to preventing loss is a well organized structure. It has been estalished, that an organization with the least amount of structure will sustain the highest amount of loss.
The attitude 'It won't happen to us' is just not good enough in today's world. Perhaps you are located in an area that is not subject to floods, earthquakes, winter storms or hurricanes. But every Hospital can be seriously damaged by fire, explosion, plane crash or sabotage by a disenchanted employee. Certain situations may deny access to your facility. These may include a hostage situation or a toxic spill that requires evacuation of the single area for days or weeks. Of course, the probability of any single one of these events is small, but the impact on your business would be catastrophic. Therefore, a workable plan to deal with them and a reliable recovery plan represent prudent management.
An extended service outage may cause your hospital to:
- Turn away patients because the EDP dependent functions cannot be adequately supported.
- Operate at a considerably slowed pace.
- Have a cash flow problem.
- Provide inaccurate as well as longer than usual billing.
News media reports may erode public confidence to the point that future growth would be impossible. Service bureau agreements and other contracts probably have clauses that absolve you of all blame for the disaster. However, you may encounter serious reluctance when you try to sign a new contract with the firm that suffered a major loss because of your disaster.
The examples given in Table 1 are but a few of the potential exposures. It is important to identify the specific risks that apply to the facility and assess the potential dollar losses that are associated with each one. Such risk assessment is very helpful in evaluating possible strategies, because the total cost of the recovery plan for any given year should not exceed the total potential losses for that same period. Risk assessment also leads to the selection of specific strategies that could be applied following different types of emergencies.
To understand that scope of the problem, a project leader must classify the types of emergencies that could affect the data center according to the following table:
Class 1 Few hours (power failure, illness or injury)
Class 2 More serious but less than 72 hours (hardware failure or minor fire)
Class 3 More than 72 hours and affects only data center (explosion, major fire or sabotage)
Class 4 More than 72 hours and affects data center and client operations facilities (toxic spills, strikes or severe power outage)
Class 5 Major disaster affecting entire community or region (flood, earthquake or winter storm)
For a Disaster Recovery Plan to be successful it must be tailored to your local conditions. The process of establishing realistic objectives based on an assessment of the risks involved, selecting appropriate strategies, and assigning qualified persons to each planning task is fundamental to your success. A control center must be identified which can be quickly activated and which will be equipped with enough telephones for local and long distance communications to the Disaster Recovery facility. In any kind of an emergency operation, it is logical to address the most critical problem first. Thus, it is important to decide in advance the priorities of the various applications. The plan must address who is to perform specific duties during the recovery period. These people must be selected very carefully, alternates identified, and plans should be documented to train and test those individuals in the performance of their duties.
In the healthcare industry, significant financial losses and critical patient billing information could accumulate very rapidly if the computer goes down. Emergency or mutual assistance agreements are designed to take advantage of other facilities excess capacity for relatively short periods of time. These agreements depend upon the other facility being compatible and having extra capacity at a time when it is useful to you. Even when they could be utilized, these agreements provide only a stop gap means to process your most critical application for a short period of time. Just think of the thousands of dollars that would be a direct loss if you could not process daily cash receipts or patient billings for two weeks.
Some of the strategies that may be addressed could be applied to events in more than one of the classes listed above. Some of them would be employed in concert with others. The strategies include reverting to manual or degraded service, procuring off-the-shelf or warehouse replacements, employing mutual support agreements, moving to a company-owned backup facility and activating the plan to use the selected recovery facility. It is important to calculate the uninsured costs associated with each of these strategies and compare them to the potential losses associated with each of the potential disasters.
Once the Disaster Recovery Plan has been organized, it is essential to the recovery effort to review and update the information on a regular basis. Information that is out-of-date with present operations will be of little or no use in an emergency situation. The plan must be updated on a scheduled basis so that names and telephone numbers for notification of key staff members are current. The plan must also be updated on a scheduled basis so that new functions and or organizational changes are included and the plan can direct the recreation of operations to support the DP department's objectives and use needs.
A practical analysis utilizing the information covered should clearly indicate to you the need for a Disaster Recovery Plan. While we all hope never to have to utilize a disaster recovery plan, we must recognize that foresight and preparedness may be the key to your organization's survival in the future.
M.J. (Doc) Trujillo-Fernandez is Manager of Contingency Planning Services, CBA Inc.
This article adapted from Vol. 2 No. 3, p. 13.