Still more on ‘cutting the copper’
One evening, an easy method to scale back on the amount of copper we were installing dawned upon us—a concept that was certainly not revolutionary, but simply logical.
By Joe Guentert
For about 15 years now, I’ve been hanging out with an outstanding group of consulting engineering firms, contractors, and owners who focus solely on the design, construction, and operation of large data centers.
Somewhere around 1997, when load growth in data centers so suddenly accelerated, we began to realize how much copper we were installing. One evening, over some Scotch, an easy method to scale back dawned upon us—a concept that was certainly not revolutionary, but simply logical.
We realized that if we just changed our infrastructure electrical design from distributing power at 480 V to distributing instead at 600 V, we could push 25% more power into the facility through the same size conductors and duct banks.
Conversely, we also realized that we could deliver the same amount of power into the facility through smaller cables, smaller circuit breakers, and lighter switchgear bus—by making just that one simple change.
The optimal end result turned out to be a combination of both of the above methods, and there was as much cost reduction in switchgear and breakers as there was from reduction in duct bank ampacities. What would have otherwise become boatloads of pricey 5000-amp drawout circuit breakers and 5000 A switchgear bus at 480 V, could remain as 3000 A and 4000 A breakers and bus, but instead at 600 V. The savings in installed cost was tremendous.
Everyone on the team was a bit apprehensive at first that changing from 480 V to 600 V could stretch cable insulation and switchgear and breakers to their dielectric limits and cause insulation failures—but it all worked great, without any such problems. We then designed and built our next dozen or so large data centers in this fashion, using 600-V switchgear, 600-V distribution transformer secondary windings, 600-V generators, 600-V UPS, and chillers and pump motors and CRAH units.
The concept was great while it lasted—until loads grew dramatically again, and we found ourselves bumping up against the ceiling once more, and having to install too much copper.
In one respect, the overall direction in the data center industry seems to be going backward lately. The recent pronounced trend toward operating servers on the data floors at 230 Vac, supplied from 400 V WYE/230 V distribution, is pushing the gains back in the opposite direction. With duct bank heating de-rating factors applied, a large trunk feeder operating at 400 V will require about 60% more tons of copper than a feeder operating at 600 V, to transmit the equivalent block of power from one place to another.
Moreover, since the 400 Y/230 V distribution arrangement requires at least a full size neutral (and sometimes, an OVERSIZE neutral), the 400-V option typically will require DOUBLE the tonnage of copper, compared to the 600-V three-wire trunk feeders we’ve been using.
Beyond that, the lower operating voltages of substation transformers and generators is now generally causing significant increases in short-circuit duties throughout these systems, and increasing arc-flash hazard everywhere in the facility.
A 2500 kVA substation transformer with a 400-V WYE secondary, having a standard IEEE/ANSI impedance in the 5.75% range can supply a heckuva lot of secondary short-circuit current, especially when operating momentarily in parallel with a 2.5M VA 400-V generator. That tends to result in larger fault duties than anyone would want, all the way to the distribution inside the computer rooms—and, all the way down to the rack level.
So now, in many cases, we’re back not only to 5000-amp breakers and bus, but to 5000 A 4-pole breakers, and a full neutral bus in the switchgear (I got a call from a major switchgear manufacturer a while back, asking “…do you have any idea why you’d want to have an 8000-amp neutral bus in lineups of 400/230-V switchgear, and do you have any idea how we would BUILD that…??”).
The objective of using 400 V WYE distribution in a data center is clear, in allowing the elimination of one entire level of transformations. And, the particular level of transformation that is eliminated by using the approach happens to also eliminate one of the greatest efficiency “bottlenecks” in a large data center. But, there are obviously trades-offs to be made in changing to distribution at 400 V.
In my opinion, it’s time to re-visit the loadenter unit substation concept we’ve been talking about for so many weeks, and consider now taking the 400, 480, and 600 V distribution voltages up to much higher voltages.
There are a lot of smart folks looking at this at this very moment. A number of new developments are coming together at the same time, including recent innovations in medium-voltage switchgear design and medium-voltage UPS systems.
These lead me to believe that infrastructure distribution at a nominal voltage like 4160 V will become the “New 480 V” in power distribution in large data centers, in the not-too-distant future. Even nominal system voltages that haven’t been very common in a while—say, for example, 2400 V—could return to popularity.
Here’s an interesting presentation, entitled “Medium Voltage Electrical Systems For Data Centers” from Michael Mosman, PE, CTO of CCG Facilities Integration, that nicely frames the discussion.
I expect that the liquid-filled HDC substation transformers we’ve been discussing for so many weeks will also figure prominently in this “next-generation” of large data centers, soon to come. More about this next week.