Special Report: Overcoming Harvey
In the wake of Hurricane Harvey’s August devastation of the Texas Gulf Coast, and in particular, the city of Houston, we asked how system integrators met the immediate business challenges of getting customers back on line against the equally pressing personal challenges of its workers who were affected individually by the hurricane. Here is one of those stories.
Wood’s response to the storm used the following process model, with pre-prepared business continuity plans and processes.
- Response (1st responders)
- Recovery (2nd responders)
- Restoration (3rd responders).
Some aspects of the planning pre-dated Hurricane Harvey. Business Continuity plans were already in place with a framework to convene planning, response, and recovery teams from business unit leadership. Pre-event planning meetings were scheduled and held the week before the storm was forecast to make landfall on the Gulf Coast.
The business continuity plans have a pre-defined structure and protocol to name an incident team leader, who delegated areas of responsibility to the management team for specific operations areas of concern, and established a standard agenda for planning communications during and following the incident.
The incident team managed potential impact assessment and data analysis, used past storm response experience, and established connection to information sources such as NOAA, USGS, local municipality and flood control districts, weather forecasting models, GIS data, industry impact forecasts from the U.S Department of Homeland Security’s InfraGard and Homeland Security Information Network. Analysis of forecasting data adopted an impact model following weather forecast similitude to Allison (a 2001 storm with stalling, or recharging rain delivering inundating, extended bands of rain), and the 2016 Tax Day Flood (which dropped 17 inches of rain on Houston). Lessons learned from those storms were factored into potential consequence considerations and response planning.
In the planning phase, we assessed potential impact to our employees and how to monitor their security, health, and safety during and after the storm. Second, we assessed how to protect our facilities and the assets contained within facility staging areas (both Wood and customer-owned). Third, we assessed potential disruption to our customers and their facilities, and what support contingencies needed to be planned for impacts before and after the event.
Employee and customer contact lists were assigned to leaders on the incident leadership team for personal contact during and following the storm. They followed protocol established before the storm for frequency and methods of communication with contacts, and intra-incident team communications and status monitoring.
Potential for facility impacts resulted in the decision to protect our staging facilities from water as appropriate for siting, elevation, and geography. To secure computing infrastructure and staged control equipment, we followed pre-established Staging Security Guidelines for backups and performed graceful full power-down of all development systems and staged equipment. Then we elevated equipment and wrapped it in heavy duty plastic sheeting to protect it from potential water damage. Finally, facilities were secured by hard lock-down in preparation for extended loss of power, access controls, and security systems.
Daily coordination meetings took place via Skype, and interim update communications used email, text messaging groups, and telephone calls. Several members of the incident response team were impacted personally by Harvey, requiring re-assignment of some responsibilities, or alternate communications means or methods.
Team members reported known impacts and urgent needs, including high-water rescue for several employees and even some incident team members. The team compiled daily reports on the status of the employees and facilities. Damage assessment surveys were conducted as soon as buildings were safety accessible to decide whether to re-open and re-establish business operations.
As the floodwaters rose, the incident team continued to maintain contact and update the team on impact changes and escalation. In some cases, where employees had to be evacuated, the incident team networked with other employees nearby to meet the evacuees at rescue points and provide transportation, shelter, or other assistance.
The company community service organization established a program called "Hope After Harvey." This group collected information on impacted employees, and set up a needs-to-resources information exchange, dispatch operation, and online fund to collect donations for assistance. The needs-to-resources exchange provided supplies, tools, labor, and emergency funds.
Many of our customers were impacted, including those with flooded corporate offices in the Energy Corridor. We offered temporary office space where available to facilities that were shut down, and all project resources temporarily were re-allocated to other facilities.
We had ongoing control system modernization projects. These included testing and cutovers in a facility not affected by the storm and prioritized by the Department of Energy as "critical" to sustaining supplies while much Gulf Coast refining production was interrupted.
We were able to defer testing and cutover operations to help the facility reduce operational risks and sustain production during the outage. In many cases, projects were deferred or re-planned due to the lost schedule.
Most of our customers demonstrated phenomenal resilience throughout this event. They got plants back up and operating even though administration, maintenance, and support buildings were in various states of recovery, dry out, demolition or repair. Many of the production assets weathered the storm better than habitable buildings.
In the first few weeks after the flood, volunteer teams and donated material were mobilized to assist employees with salvage, muck-out, and removal of damaged contents, flooring, sheetrock, insulation, etc. Initial facility safety inspections determined we could return to our offices having only lost three business days to closure.
Our incident recovery teams conducted secondary assessment and inspection of the equipment that had been protected prior to the storm. They unpacked, put equipment back in place, and powered systems back up. We were happy it only took one work day to secure our facilities, and only a few hours to restore systems to operation, ready for employees who had access to safe routes to return to work.
Restoration and rebuilding
We continue to support our employees and customers, including employee donation of vacation time to assist fellow employees who need more time off to recover from the storm. The incident team conducted workshops to capture response lessons learned. Those will be incorporated in updated business continuity plans.
Our ability to prepare, respond, adapt, and recover was a real victory, validating that our planning, potential impact assessments, and preparedness were successful this time. As with Hurricanes Allison, Rita and Ike, and the Memorial Day and Tax Day floods, we continue to learn, improve, and refine our decision making and preparation for the next one.
Learn from the past, prepare for the worst, hope and pray for the best. But always have a plan: dedicated leaders and a caring workforce who can work together to make wise, timely decisions and act on those decisions when needed. These elements are the key to enduring and overcoming catastrophic events.