Data center discussion
CSE: What problems arise due to a lack of a standard for data center energy efficiency or Power Utilization Equivalence (PUE)?
Keith Lane: There have been many unsubstantiated claims of very low PUE levels from data center operators and salespeople. PUE is a good matrix to identify energy efficiency, but there must be a standard to make sure that you are comparing the same thing. Also, the PUE levels are going to be dependent on the climate. There has been some discussion in the industry about providing a coefficient based on climate and location to provide more meaning to PUE. In some way it would be a method of comparing efficiency of data centers around the world.
Very low PUE numbers are suspect. By the time you take the UPS losses, transformer losses, I square R losses of the cables, and a few lights, you will be over 1.1 PUE before any mechanical or plumbing systems are brought into the equations. Additionally, engineers and owners should ask whether PUE claims are based on theoretical evaluation, actual best case scenario, average, or peak information.
Steve Berry: I agree. The main problem is an inability to make like-for-like comparisons. Other issues include setting performance aspirations too high or low, and concentrating effort on “playing the PUE game” rather than pursuing good, efficient design.
Peter Babigian: With a new spotlight on carbon footprint, the Green Grid makes a very important first step toward a calculation intended to help data center managers understand the level of efficiency at which facilities are operating. There are several areas in which the methods for computing this efficiency must be further refined. At the simple level, PUE is computed by dividing total facility power by IT equipment power. The areas in which further refinements to PUE are required and underway include the following:
Geographic location: Better PUE numbers are more easily achieved in temperate areas than in areas with an extreme hot or cold climate.
Distinctions between facility and IT equipment: From internal server fans to cooling elements (such as pumps, refrigeration, blowers, and heat exchangers within the IT equipment itself), it is not yet clear how we can distinguish between IT equipment and cooling equipment. In addition, the categorization of certain components, such as facility equipment versus IT equipment, skews the PUE calculation.
Dynamic nature: Simply dividing the total power by the IT power would produce a result, but there is no formal definition for measurement frequency or for averaging requirements. If measurements are taken on a day of extreme hot weather and low IT equipment usage, the results would be far different than if the measurements were taken on a day of temperate weather and high equipment use. In reality, PUE is not constant. Rather, it is always changing and computations should be adjusted to reflect this.
Compute cycles: Present calculations do not account for the number of compute cycles or the actual work produced by energy that is expended.
Data centers are encouraged to compare PUE numbers and discuss the measures they have taken to increase the operating efficiency of the facility. In addition to the variables above, the PUE calculation can be skewed by, for example, running a data center with higher server inlet and outlet temperatures or leaving IT equipment running that isn't necessarily processing data. There is not yet a comprehensive enough standard to support PUE comparison between different facilities and business entities. The new attention from the corporate energy czars on PUE results and metrics is contributing to an overall environmental improvement. But until the above factors are incorporated into the evolving standard, we all need to realize that PUE is better for demonstrating individual site characteristics and improvements than for comparing ourselves with others.
Michael Ryan: What is great about this is that even though the methods are still being refined by the Green Grid, PUE has everyone looking at their data center operations differently than before. It has really brought a focus on how to reduce energy usage in the data center. Even the simplest PUE calculation can help you identify areas where your data center energy efficiency can be improved.
CSE: For a recent new construction or retrofit project, describe some MEP measures that significantly improved PUE.
Babigian: Occupancy sensors and high-efficiency lighting are two ways to improve PUE. Turning lights out in a large facility can save a great deal of money. Zoning sensors into pod areas ensures that large amounts of lighting are not needlessly burning. Other effective measures include:
Hot-aisle containment systems to reduce cold and hot air from mixing, along with blocking/blanking of raised floor openings
Installation of cabling overhead rather than under the floor to allow unrestricted airflow
Optimization of equipment layout using computational fluid dynamics (CFD) analysis to ensure that efficient airflow is achieved
When data center size and contiguous adjacent office space are coupled with a good balance, data center heat output can be used to warm office space.
Lane: I'd add:
“Just right” sizing of electrical distribution equipment: Oversized electrical equipment and additional redundancy can provide for a system that is loaded to less than 50%. Transformers and UPS systems operate at lower efficiency at a level less than 50% loaded. More losses equal less efficiency and higher PUE.
Modular build-out of the electrical systems: Modular build-outs can ensure higher percent loading of the electrical equipment and a more efficient electrical system.
Outside air economizers with “free” cooling: Air handling units with evaporative coils are a very efficient method of cooling data centers and other critical environments. In cooler climates, the system can operate in this mode without a chiller a majority of the time.
Increase cold aisle temperatures: Increasing the allowable cold aisle temperatures to the high 70s F or low 80s F may allow the system to operate without a chiller. This can reduce the initial cost of the data center, and the size of the standby generators, while lowering PUE.
Berry: Factors that have had the most significant impact on PUE in recent projects include the following:
Relaxation of environmental criteria in data halls and electrical plant rooms
Optimizing chilled water temperature to minimize system energy consumption (chiller power, chilled water pumping power, and computer room air conditioner (CRAC) unit fan power)
The use of water-cooled chillers with water-side free cooling
The use of actively controlled variable volume fans in CRAC units.
Ryan: Most of the larger gains in PUE are typically found in the mechanical systems:
Outside air and airside economization should be used as much as is feasible to control server heat loads.
Extend room relative humidity percent ranges as wide as possible and base them on equipment manufacturers' specs.
Specify all air handling equipment with variable speed drives to allow them to ramp back when room loads are less than design conditions.
Chilled water loops should be variable in all aspects using variable speed pumping systems, chillers, and cooling towers.
The chilled water loop temps should be as high as possible to increase economization use and include temperature resets based on load conditions.
In the electrical systems, typically it is the UPS where the largest efficiency gains can be made. The UPS should be designed such that it operates efficiently across the full range of expected operation. A data center with redundant UPS system can run at 30% to 40% use. This can be a relatively low efficiency point on UPS systems, so proper system design and equipment selection can greatly reduce the losses that occur in the UPS. Lastly, a comprehensive monitoring and control system that integrates both building management systems and environmental monitoring systems into one package is a necessary tool for tracking and keeping a low PUE.
CSE: What methods of fire protection design do you prefer for data centers and laboratories with sensitive equipment, and why?
Ryan: It really depends on what is in the space. If the equipment is high-dollar and poses a high risk to a company's revenue or operations when not functioning, then a clean agent system such as FM200 is likely merited. We typically deploy only a double interlocking pre-action system or standard water-based sprinkler systems.
Berry: A multilayered approach is preferred—early warning fire detection (typically aspirated systems) as well as conventional smoke detection. Early detection allows for manual intervention to locate problems before the fire occurs. Water mist is preferred for data center applications with conventional sprinklers in laboratory areas. Water mist provides the advantage of minimizing water damage risk compared to conventional sprinklers. When compared to gas systems, water mist also provides the advantage of avoiding pressure venting requirements that tend to be large and pose problems of integrating into the building fabric.
Babigian: In this situation, it is imperative for the client's goals and expectations to be clear. We typically design pre-action sprinkler systems, and occasionally we recommend gaseous suppression systems. Sometimes, clients think that by using a gaseous system, they can avoid having any sprinkler piping in the data center. This is generally not the case, as the state or city codes typically mandate sprinkler systems. If clients understand how low the likelihood is of water accidentally entering the room, many opt out of adding gaseous systems.
Lane: In a wet system, maintenance personnel can potentially knock off a head and cause significant damage and cost. Normally a water sprinkler system is not recommended in a data center, but the local authorities may require it. A flooding fire extinguishing system or clean agent system is the preferred method for the critical environment of a data center. These agents are nonconductive and can be discharged with occupants in the room. In some cases, NFPA 75 allows a data center to continue to operate when a gaseous system is discharged.
CSE: If you could get owners or engineers to do one thing differently to prevent errors, waste, or higher cost, what would you recommend?
Lane: Work together as a team to provide a good basis of design (BOD) document at the very beginning of the project. The BOD spells out the design intent, the system requirements, and the interaction between the MEP systems. Getting this hammered out in the beginning will save significant time and money during the design, development, and construction document phase of the project.
Berry: Be very clear in understanding what your operational requirements are and communicate this clearly to the team.
Ryan: I would recommend fully understanding your data center requirements and availability needs. Make it as modular as possible to allow the data center to easily grow and change over time. Right size the infrastructure for your initial needs and provide avenues for simple “plug-and-play” capacity growth over time. Many data center projects are needlessly built to the extremes. Typically, only a small portion of the data center equipment is critical, not the entire data center. However, people still build data centers as if they are a homogenous critical environment.
Babigian: Give us a little more time! Many projects today must be on the fast track, with demands for pre-purchase specifications and the procurement of systems and equipment before the design is even complete. We would greatly benefit from more upfront coordination with the end-user and facilities personnel. However, the tendency to compress the project schedule may be more of a cultural issue.
Today, engineering and construction are examined with a modular “LEGO block” mentality, which is the idea that all parts are interchangeable. If this was true, all parts of a system would be inherently inefficient as their design would require the balancing of many different performance requirements. This has been worsened by the constant technological changes that we deal with daily. The upfront coordination would increase the opportunity for the design team, facilities personnel, and end users to determine where flexibility and interchangeability are required in the system. Less compressed project schedules would allow more opportunity to review alternative designs, perform better cost analysis, and accommodate changes in requirements or field conditions. This would allow for more decisions to be based on what is most beneficial for the user in the long term, rather than on short-term schedule impact.
CSE: What engineering tools and resources do you find most useful for design (computational fluid dynamics, BIM, ASHRAE guidelines, Green Grid papers, etc.)?
Babigian: ASHRAE is a good starting point, and CFD is very useful for fine-tuning a design and optimizing the layout. BIM is useful in building design, and a great tool for visualization and identifying conflicts.
Berry: Commercially available CFD software written specifically for data centers is particularly useful when analyzing white space issues. However, commercial software is not available for many important engineering issues related to data centers—issues such as the transient thermal response of computer rooms when cooling is interrupted, and the transient effect this has on chilled water systems when some elements are backed by UPS and others are not. We have found it necessary to develop, test, and verify bespoke in-house software in order to model it and design better systems.
Ryan: That is a tough one because right now the data center is evolving very quickly. ASHRAE is always a good resource when it comes to the chilled water plans. ASHRAE TC9.9 and the data center series ASHRAE is creating are also very good. Green Grid is doing great work around data center systems, efficiency, and metrics. As for tools, we have also started to lean heavily on BIM during our design process as it helps tremendously in the trade-coordination phase of the project. Interestingly enough, CFD is becoming less of a requirement as the aisles become completely contained and the air flows become predictable.
Lane: We have used 3-D and BIM extensively on our mission critical projects. In complicated MEP systems, 3-D modeling and BIM will allow the engineers to determine collisions, routing, and sizing issues long before the problems are found in the field. There is no doubt that there is significant value in providing 3-D and BIM on complex mission critical projects. With 3-D/BIM, there will be fewer problems during construction, less requests for information, and a reduced number of change orders resulting in a total lower cost to the owner.
CSE: What technology trends for standby power and power conditioning systems are proving to be practical, affordable, and reliable in the field?
Lane: One trend we are seeing more of is modular/containerized systems. We have been involved in the design of containerized data centers. These systems offer several advantages including:
Minimum upfront cost
Optimum use of outside air, which can reduce PUE
Potential use of multiple server technologies.
I do not think the modular/containerized data centers will take over the conventional brick-and-mortar data center, but these systems do have their place in the critical environment systems of the future.
Berry: An arrangement we see more often is a centralized standby generation plant with local controllers installed in separate fire-rated rooms that mitigates against any single point of failures. The centralized standby generation plant can be operated at its maximum efficiency by the use of a load-sharing system. Centralized power conditioning systems should be configured using a distributed electrical system where the load is distributed across all the conditioning units. This enables each power conditioning unit to be operated at maximum efficiency.
Ryan: Standby power has not changed much, but the equipment has become more sophisticated over the years in the area of monitoring. With the UPS, we are starting to use flywheels as energy storage over batteries. They are less expensive to maintain and take up less space than batteries. The biggest changes in UPS over the past few years have been the steadily increasing efficiencies and more modular approaches to deploying capacity, all of which are great for the data center.
Read more about data center efficiency in Bill Kosik's "Data Center Power Use Effectiveness (PUE): Understanding the Contributing Factors"
Peter Babigian, LEEP AP, RCDD Principal
Steve Berry, Associate Director
Keith Lane, PE, RCDD, LEED AP Partner/Principal
Michael Ryan, PE Senior Staff Engineer Global Lab & Data Center Design Services Org.