Keeping HAL Cool

Texaco's data center in Houston, Texas, serves several critical corporate needs, including research and development, analysis of field surveys and the "7 x forever" support of the global data center. The central energy building also supports executive offices and other business support areas.


Texaco's data center in Houston, Texas, serves several critical corporate needs, including research and development, analysis of field surveys and the "7 x forever" support of the global data center. The central energy building also supports executive offices and other business support areas.

In 1995, with the pending addition of 60,000 square feet of office space at the data-center headquarters, Texaco facility engineers felt nervous that the existing setup would not be able to accommodate the additional load. In a recent interview, Glenn Hanson, facility operations supervisor, recalled how-new to the post-he had noticed that, "we weren't getting the cooling [we needed]."

In fact, the 30-year-old plant had undergone a number of capacity expansions that resulted in a state of complex and unpredictable infrastructure control. A departure from legacy water-cooled mainframes to higher-power-density air-cooled midrange server environments posed new challenges to the existing infrastructure.

HVAC tops data-center concerns

The fact is, such challenges are endemic to operating mission-critical facilities. "Cooling is the biggest problem facing data-center designers today," Dr. Roger Schmidt, head of air-cooled mainframe development at Armonk, N.Y.-based IBM stated at a recent presentation, adding that campus-wide cooling systems must evolve as data-systems designers migrate to state-of-the-art technology. Specifically, central chiller plants must adapt to nodes popping up in the infrastructure web.

In a perfect world, designers would always keep a sense of global perspective when designing cooling subsystems. Unfortunately, data-center hardware designers rarely understand the full impact of their decisions on existing cooling systems.

Subsystems at the Texaco facility were no different. As it was, problems abounded in the existing infrastructure, and Hansen found many faults in the addled and outdated cooling system. For example, "The return [chilled-water] pressure was so much higher than the supply pressure," he recalls. "We were having to run more and more equipment just to get the heat transfer we needed."

Wild fluctuations in chilled-water pressure, temperature and flow caused constant complaints about space temperature (see Figure 1, right) Flawed pump-control strategies eroded control valves. A destructive hydraulic water hammer blew the cast-iron impeller casing apart on a 2,000-gpm centrifugal pump. Frantic operators attempted to control critical space- and process-cooling loads with manual isolation valves, wasting capacity, energy and manpower. In addition, excessive chilled water returning from remote buildings wreaked havoc with central-plant capacity control. Temperature and humidity instability occurred as central-station air-handler coils dehumidified the data center while steam generators in computer-room air handlers operated continuously to keep the humidity levels up to setpoint. Moreover, these massive inefficiencies made utility costs difficult to budget. (See Figure 2, this page, for an overview of the original faulty chiller plant.)

Driving with the brakes on

There were further challenges in simply attempting to rectify the problems with the existing systems. For example, maintenance shutdowns and disruptions to critical processes were not allowed-and are still not allowed-according to Texaco's corporate mandates. Sweeping changes to the chiller plant didn't seem possible without taking the data center and all the support structure down for a period of weeks.

Texaco's engineers faced a dangerous lack of both temperature and humidity control despite the fact that their chilled-water plant was pumping over 10,560 gallons per minute (gpm). It was as if they were pressing the accelerator all the way down while driving with the brakes on (see Table 1, this page). How could the company update the system to accommodate both new air-cooled servers and an additional 60,000 square feet without having to shut the entire chiller plant down?

Hanson recognized that prior engineering efforts lacked global perspective; he was managing a series of tactical improvements that were causing adverse conditions when operated concurrently. One system loosely tacked onto another led to fragmented temperature-control concepts and incompatible chilled-water pumping strategies for each building. Further, specifications for chilled-water coils were incompatible with chiller selections, and system design flaws precluded an effective operator-training program. Erratic temperatures and flows wasted available capacity and redundancy.

Jack Dale Associates, Baltimore, was retained to benchmark the campus systems and solve their dilemma. The biggest challenge seemed to be to develop a set of design documents that would allow Texaco to implement the engineered solutions without any disruptions to the global data center or to research and development areas. Led by the motto that "a problem well defined will seek its own solution," Hanson and the consultants began seeking the guiding principles for producing the construction drawings and specifications.

Hydraulic decoupling

Prior to developing design concepts, existing systems were extensively tested and evaluated. A holistic system review identified various interactions confirming the operators' experiences and explaining failure modes of recurring disruptions. In-the-field and over-the-phone interviews with the operators were conducted. Hanson preferred keeping facility operators involved in the design process because, in his words, "They're the one's that are going to make it or break it."

The consulting engineers outlined improvements and procedures that boosted chiller-plant efficiency and are credited with saving Texaco at least $100,000 per year in utility costs (see Table 2, below). The course of action primarily dealt with existing operational problems. However, the design also offered "hooks" for expansion of the chilled-water reserve-and reduced energy consumption.

"Once we put in the bidirectional hydraulic decoupling bridges, that pretty much set the demand," Hanson explains (see Figure 2, page 64 ). "Plus, reducing the pumping horsepower throughout the complex [by 950 hp] and then putting in the variable-frequency drives ended up so that whatever the demand was is what the central chilled-water system distributed, no more and no less."

The design and construction phases of the project were completed quickly-with no facility shutdown-and at a reasonable cost of less than $600,000. This was significantly better than the estimated cost of $2 million and two-week shutdown of the entire 5,000-ton chiller plant that had been proposed by another engineering firm.

Five years after the project is complete, Texaco still relies on the redesigned system. With a satisfied grin, Hanson comments, "Aside from the fact that we've been able to train our operators much faster than before ... the beauty of this system is that it basically runs itself."

No comments
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
Each year, a panel of Control Engineering editors and industry expert judges select the System Integrator of the Year Award winners.
Control Engineering Leaders Under 40 identifies and gives recognition to young engineers who...
Learn more about methods used to ensure that the integration between the safety system and the process control...
Adding industrial toughness and reliability to Ethernet eGuide
Technological advances like multiple-in-multiple-out (MIMO) transmitting and receiving
Virtualization advice: 4 ways splitting servers can help manufacturing; Efficient motion controls; Fill the brain drain; Learn from the HART Plant of the Year
Two sides to process safety: Combining human and technical factors in your program; Preparing HMI graphics for migrations; Mechatronics and safety; Engineers' Choice Awards
Detecting security breaches: Forensic invenstigations depend on knowing your networks inside and out; Wireless workers; Opening robotic control; Product exclusive: Robust encoders
The Ask Control Engineering blog covers all aspects of automation, including motors, drives, sensors, motion control, machine control, and embedded systems.
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
News and comments from Control Engineering process industries editor, Peter Welander.
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
This is a blog from the trenches – written by engineers who are implementing and upgrading control systems every day across every industry.
Anthony Baker is a fictitious aggregation of experts from Callisto Integration, providing manufacturing consulting and systems integration.
Integrator Guide

Integrator Guide

Search the online Automation Integrator Guide

Create New Listing

Visit the System Integrators page to view past winners of Control Engineering's System Integrator of the Year Award and learn how to enter the competition. You will also find more information on system integrators and Control System Integrators Association.

Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Control Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.