Data center design-build upgrade is easier with process controls
Streamlined with a new process control system, M.C. Dean designs redundancy and security into a data center facility with 6 control panels (2 PLC and 4 remote I/O), 2 redundant automation stations, 16 remote I/O racks with more than 1,800 hard I/O points, 5 operator workstations, 2 server racks, 10 network cabinets, and 45 network switches for collecting more than 35,000 soft I/O points.
A project to replace an out-of-date data center began in 2012; M.C. Dean focused on building the new LEED Gold facility to support a 6 MW data floor, which would eventually to be expanded to 10 MW. The project included 6 control panels (2 PLC and 4 remote I/O), 2 redundant automation stations, 16 remote I/O racks with more than 1,800 hard I/O points, 5 operator workstations, 2 server racks, 10 network cabinets, and 45 network switches for collecting more than 35,000 soft I/O points.
The largest issue with the current data center was its age. Over the years, network and power cables had become entangled under the data center floor, making it extremely difficult to troubleshoot connection problems. The heating, ventilation, air conditioning (HVAC) system was being pushed beyond its design limits, and the uninterruptible power supply (UPS) system was so old it lacked the backup power needed to keep the facility online during system outages.
The problem threatened to shut down the facility at any moment for days at a time. Just one day of potential IT service outage costs the company $25 million in lost productivity. Building the new data center presented a number of challenges and design requirements, including integration of facility systems into one operator platform, meeting Uptime Tier III certification, and integrating physical and virtual security into the design.
Three main disciplines contributed to the project. The electrical portion was done by M.C. Dean and included the electrical switchgear, protective relays, electrical metering, lighting, and the fire alarm system. M.C. Dean also was responsible for the controls portion that included mechanical process control, supervisory control and data acquisition (SCADA) interface, designing and building the control panels, and designing the network infrastructure. Southland Industries was responsible for the mechanical portion of the project, including designing the mechanical process, sizing, and selecting equipment, such as chillers, pumps, and air handling units, with the industrial instrumentation to control the equipment. Southland was also responsible for providing the building automation system, which served the noncritical HVAC spaces in the facility.
Seamlessly integrating all facility systems into one workstation was the most important design requirement for the project. The customer was operating the facility with four workstations representing six systems to evaluate utility plant status. Having all the data exist in separate workstations and software packages made it hard to troubleshoot problems and track maintenance activities. For the new facility, the customer sought an integrated solution combining all separate hardware and software packages in a central workstation.
Three main systems needed integrating. The mechanical system consisted of several drives, chillers, pumps, and transmitters that represented more than 1,800 hard I/O points. There were 72 variable speed drives (VFDs), which needed to use hard I/O for control and Profinet for monitoring. There were also several mechanical skid systems, such as reverse osmosis, rainwater harvesting, refrigerant detection, and chemical treatment. These skid systems had soft and hard points that needed to come back to the central system for monitoring and control.
The electrical devices included 120 relays, 130 electrical meters, and more than 550 miscellaneous electrical devices, including power distribution units (PDUs), lighting inverters, UPS units, battery chargers, and smart circuit breakers. All of these devices communicate over soft I/O with many protocols to choose from.
The last big hurdle was the building automation system (BAS), designed to serve all noncritical spaces of the facility, including the office building and the facility access center. Because this system was provided by the mechanical contractor and was based on a different software and hardware package, we needed to figure out how to integrate the graphics on both systems into one operator workstation while still retaining the same overall "look and feel."
M.C. Dean had a lot of previous experience with the process control system in mission-critical process plants but never in a critical operating environment such as a data center. After reviewing the project with Siemens, and after comparing several different distributed control systems, we decided that control system was the best fit to integrate with the automation stations chosen for control over the mechanical equipment. At this point, we were unsure about how we would collect all information from the soft I/O points due to the complexity and number of soft communication devices.
With more than 550 devices spread over more than 20 device types, we needed to be careful in our software selection to make the integration as fast and efficient as possible. We found a control system with two add-ins designed for integrating communication protocols into the operating system (OS) tag servers.
The first add-in provides the ability to integrate hardware devices that use different protocols into the control system. Some of these protocols include DNP3, Modbus, and IEC-60870. Because we needed to integrate devices from the electrical and mechanical systems, this worked out as the solution to get them to co-exist.
The second add-in works in much the same way as the first, except that it includes a driver for the IEC-61850 protocol. This was imperative because we planned to use this protocol to communicate with the electrical meters and protection relays, which represented the majority of the hardware devices throughout the data center. Both add-ins included the control system framework to communicate to the soft I/O devices. They helped to bring together the idea of "seamless integration" because the operators would no longer have to go to separate workstations or use separate programs to operate the facility.
To use the add-ins in the most efficient manner, we used a database automation (DBA) tool to automatically generate the OS database with the display hierarchy, required variables, alarm signals, and faceplates. Using the database tool, we created one device "type" for each unique piece of hardware and then replicated that type to save engineering time. So Instead of creating 550 individual devices and attaching tags to each one of them, we created 20 device types and then numbered them incrementally within their type to represent all the devices.
This saved us a substantial amount of engineering time that would have otherwise been lost to mindless data entry for each device. It eliminated potential data entry errors because the tags are linked with the operator faceplates so that if a change is made to a device type, that change will be reflected in every instance of that device in the system.
The same concept increased engineering efficiency when working with the hard I/O points through the use of a bulk engineering tool that uses Microsoft Excel spreadsheets to format the individual I/O points for import into the control system. We first created our control module types in the control system software editor. We then organized all of our individual I/O points in the Excel template file including rack, slot, and point locations; which control module type they belonged to; any interconnections they would have; and other relevant information that the control system would need. We then imported both items into engineering software, at which point the control system would auto-generate a control sheet for each device that was assigned in the Excel spreadsheet.
With around 1,200 devices to integrate, the engineering software reduced the amount of engineering time required to manually create the symbol table, address IO, and make the interconnections between sheets. These tools helped us stay ahead of the already tight construction schedule and also assured us that the number of errors we would find during commissioning would be much smaller than if we had to do all programming via manual data entry.
Another design challenge we faced was vertically integrating all of the different project teams within M.C. Dean. Typically, projects are horizontal where each "team" finishes its part and passes it on to the next phase's team and so on. At the end of the project when it's time to bring everything together for commissioning, problems arise due to lack of process overlap, no common understanding of the problems or solutions, and things ultimately get worked out in a reactionary fashion. We were determined as a company not to let this happen and to work closely both within our own firm and with the other disciplines to ensure that when it came time for commissioning, there would be no holes left in the design and everything would come together as smoothly as possible.
Normally, our electrical design team chooses to use certain protection relays for all electrical designs. It is something they have become comfortable with and typically what is specified on most jobs. The difference in this project is that the M.C. Dean controls team had to design and integrate the network infrastructure to monitor and control all of these relays that the electrical design team was responsible for choosing. This was the first hurdle in the vertical integration approach that we would have to get over.
The electrical design team began to look at other protection relay manufacturers as possible alternatives to the standard relay used. Another line of relays was examined as a possible alternate solution.
We asked each manufacturer to design a test for the project that we could test in our office. After reviewing the designs, we came up with a pre-test analysis of the similarities and differences between each manufacturer's solutions. Each manufacturer then built a rack that contained all of the relays needed to replicate one of the main-tie-tie-main lineups we would have on site. One rack consisted of a redundant star topology that is "tried and true," while the other was an IEC-61850 relay platform. Representatives from each manufacturer joined us while we simultaneously tested and compared racks side by side over a 3-day period. We compared many different components, including network speed, ease of use, security, redundancy, and overall functionality.
One experiment we performed was to determine the speed at which the relay could process commands sent from an HMI and then send back a response to that same HMI. We used a global positioning system (GPS) time clock to timestamp the messages sent between the relay and HMI. The one relay averaged about a 3.8-second delay while the other relay averaged only a 0.7-second delay. The faster relay has the flexibility to allow end users to determine the scan time at which they wanted to process logic within the relay. This is very similar to a PLC having different scan times based on time process PID loops and other critical information. The other relay did not have any options to adjust the scan time of the logic, determined to be the bottleneck in the test.
At the end of the day, the chosen technology was more cost-effective, cut down on the physical infrastructure needed to implement the design, simplified the programming effort within the controller, and provided a faster, more powerful relay. If we had not stepped back and considered a different solution, we would have wasted materials and engineering time for a less-optimized offering.