Data center design
In the information age, data centers are one of the most critical components of a facility. If the data center isn’t reliable, business can’t be done. Experts provide insights on cooling and power issues, cloud computing, and energy efficiency.
- Kevin V. Dickens, PE, LEED BD+C, Mission critical design principal, Jacobs Engineering, St. Louis
- Terrence J. Gillick, President, Primary Integration Solutions Inc., Charlotte, N.C.
- Bill Kosik, PE, CEM, BEMP, LEED AP BD+C , Principal data center energy technologist, HP Technology Services, Chicago
- Keith Lane, PE, RCDD, NTS, RTPM, LC, LEED AP BD+C, President/CEO, Lane Coburn & Associates LLC, Bothell, Wash.
- David E. Wesemann, PE, LEED AP, ATD, President, Spectrum Engineers Inc., Salt Lake City
CSE: What challenges does a data center project pose that are unique from other structures?
Kevin V. Dickens: Energy optimization. By definition these structures are energy intensive, but responsible mechanical design demands that we do all we can to minimize the power required to support the IT base load. Hence, the focus on the power usage effectiveness (PUE) metric in all of its manifestations. The follow-on challenge with PUE is simply being secure enough to create and use your own version of the metric (based on the information available to you), and not getting caught up in “PUE envy” and the pointless comparisons and competition. The only thing that really matters is a PUE tool that works in your particular application.
Terrence J. Gillick: While the enclosure is straightforward and sturdy, the mechanical and electrical systems are unique in their capabilities and sophistication. In particular, relative to the footprint of the building, the electrical density is very high, and the mechanical systems include precision air conditioning systems that are significantly more complex than traditional HVAC systems. For a commissioning agent, one of the larger challenges is to validate the building automation and control systems and electrical power management systems to ensure that the sequences of operations defined by the engineer of record are properly implemented and, in turn, to validate that the facility will operate as intended.
Bill Kosik: First, I think the time frame is more like one to three years or less. The industry continues to move very rapidly. It is not cloud computing itself that is driving change, it is what is causing the need for cloud computing—the Internet of Things, as it is called. Everything from your car’s diagnostic system to the temperature in your refrigerator is in the cloud. This will continue to drive a greater and greater need for data centers.
Keith Lane: Data centers are very unique facilities. The sheer amount of power and critical nature of the loads being served require significant expertise. Uninterruptable power supplies (UPS), large standby generators, fuel supplies, large conductors, medium-voltage services, large transformers, various voltages, harmonic distortion, metering, PUE, and energy efficiency all must be considered in the design of data center facilities. Because of the unique nature of the electrical load profile, the heating of underground electrical duct banks must be evaluated. This involves 3-D modeling of the underground feeders as well as a comprehensive failure mode analysis and Neher-McGrath heating calculations. The initial cost of building a data center is tremendous. The long-term costs associated with running a data center include the electrical and water services, which are very significant and must be considered during the design process. The electrical and mechanical engineer must work collaboratively to ensure the most reliable and cost-effective systems are designed and implemented. Enough design time must be built into the schedule to ensure value engineering ideas are fully vetted. Additionally, comprehensive commissioning of the data center should be provided by a third party to ensure all components of the mechanical, electrical, plumbing (MEP), and fire protection system work independently and as a system prior to actually serving critical loads.
David E. Wesemann: Capacity of the utility supply, in particular electrical, is much greater than other buildings. Many times the electrical utility is not prepared for the large loads a data center poses on the electrical grid and many months, or even years, are required for planning and preparation. There are three key items:
- Redundancy and reliability requirements: Data centers normally require a much higher level of reliability that is provided with redundant components and paths.
- Security: Data center facilities, including the infrastructure equipment, have a higher level of security risk associated with them.
- Maintenance: Data center infrastructure components and paths generally cannot be taken out of service for maintenance. Designs must accommodate this.
CSE: When working in mission critical facilities, what’s the most difficult challenge you face?
Gillick: Working in an occupied mission-critical facility—for example, commissioning an upgrade to existing MEP systems—poses the greatest challenge for us in protecting the critical load for patient care, social media, revenue generation, retail sales, or back-office operations. In planning commissioning for these facilities, we define every step of the process in great detail to avoid impacting ongoing operations.
Dickens: After energy, the second challenge is to provide appropriate redundancy strategies based on the facility’s demands, budget, and operating model. It isn’t as simple as working toward the Uptime Institute’s Tier II or Tier III certification. It really needs to be right-sized for the project. The business drivers for a cloud provider, a banking institution, or a command and control facility for the Dept. of Defense are vastly different. The redundancy strategy to support that mission, be it physical duplication, virtualization via software, or mirrored facilities, is just as unique. If you simply “double up” then you are being intellectually dishonest and letting your customer down.
Wesemann: Meeting the reliability and capacity requirements with a limited budget. Many times owners/end users have high expectations for reliability without the budget to support those expectations. Usually this can be overcome by educating the owner/end users with a cost/risk analysis and discussion.
CSE: How will cloud computing affect data centers in the next three to five years?
Dickens: For those of us in facilities, I think the primary impact will simply be more work. But as far as the facilities themselves, the mix in the cabinets may change, but we will still be responding to higher load densities and the need for modularity within the data center. Whatever the cloud is today, it will be something different in the future; that’s the nature of the information age. But the parallel evolution on the facilities side will continue to be seen in expanded environmental parameters and a slow trek to direct fluid cooling.
Wesemann: We can only speculate. Cloud computing will still require data centers, which will still require engineers to design them. There may be some centralization of data centers as cloud computing picks up users; however, as more companies offer cloud computing, they may migrate away from large central data centers to more, smaller data centers. I believe there will always be specialized businesses and services that have unique and critical applications that will want to be kept “in house” rather than rely on remote sites and the limited bandwidth and reliability of the telecomm service providers to communicate between users and applications.
Gillick: Cloud computing servers and storage equipment will require an exponential increase in raised floor space to support the continued growth of cloud computing over the course of the next three to five years. Data center owners, planners, and IT planners are now challenged with determining how much additional space they will need to allocate, lease, and/or build to support cloud computing requirements.
CSE: Please describe a recent new building project you’ve worked on—share problems you’ve encountered, how you’ve solved them, and engineering aspects of the project you’re especially proud of.
Wesemann: One issue we have on new projects is the estimation of IT/server equipment load for the sizing of mechanical and electrical equipment. On one particular project, a data center user requested 300 W/sq ft for IT load alone, not including the power for the mechanical cooling. After much debate and with some reservation, the user agreed to allow the design team to use 150 W/sq ft as the basis of design, with expansion capabilities to 300 W/sq ft. Modular mechanical and electrical systems were used for the initial load with provisions to easily add components if and when needed. The data center has been in operation for about five years and based on the last report, the IT load has never increased beyond W/sq ft.
Gillick: We recently provided commissioning for the first phase of a multiphase greenfield data center construction project for a global company. The first phase comprised 200,000 sq ft of Tier III data center space with a large electrical service. Overall challenges included communicating effectively and succinctly the requirements and goals of the commissioning program, effectively scheduling commissioning activities 18 months in advance, and developing a plan to concurrently commission mechanical, electrical, and control systems. Two critical scheduling milestone goals were established: one, that the building automation system be installed and commissioned prior to functional testing; and two, that a factory witness test of the control systems and graphics be required prior to their deployment at the site. Successful completion of these milestones enabled our team to validate and use all of the trending features of the building automation system (BAS) and the power quality and monitoring system as a reporting tool during functional commissioning and contributed to meeting the aggressive project schedule. Additionally, we provided full lifecycle commissioning services beginning at schematic design and continuing for one year beyond beneficial occupancy. The process required early integration of the building automation and energy management systems and power monitoring and control systems—in particular, an extensive point count.
Lane: Our firm has worked closely with Silent Aire Technology for more than five years enhancing the design of modular data center deployments around the country. There are numerous challenges and numerous benefits to the design, construction, and deployment of modular data centers. Modular data centers are designed and built as a complete system. The entire mechanical and electrical system is built around the client’s IT infrastructure needs and requirements. Modular offsite construction, as opposed to the conventional brick and mortar data center, delivers the following main benefits: speed, performance, and cost-containment. Building offsite in a controlled, safe, and environmentally friendly space may quite often allow for much quicker deployment. In addition to the time savings in building the mechanical, electrical, and structural components, all components and systems are tested in the factory before shipping to the site. This saves time and money during the final integrated systems testing (IST) before hand-over to the client in the field. Optimal performance is more quickly achieved from a modular data center versus a solution built from scratch. The design specifications are tested and verified before the unit ships, thereby delivering immediate quality assurance. There are several advantages to building offsite, but the main reasons are cost control and speed to market. Delays related to weather, site conditions, unreliable or inconsistent labor forces, and labor inefficiencies are greatly reduced or eliminated in a warehouse/prefabrication environment. This leads to reduction in cost and schedule. Additionally, estimating the total cost of the project is typically more accurate. In the end, this represents less risk to the end user. Modular data centers that are factory built can come with either a UL or ETL safety certification label that certifies they have been factory tested and meet the required electrical safety codes and requirements. This safety certification allows the modular data centers to be classified as equipment and not modular buildings, which may often circumvent permitting and inspection requirements that would normally be demanded by the authority having jurisdiction (AHJ) in a brick and mortar build, thus allowing for aggressive and expedient deployments. Other benefits to the modular data centers being classified as equipment are: they can be depreciated as equipment, and opportunities for leasing or financing of the modular data center exist as well.