Safe acceleration: Automating the world’s largest and fastest machine
The Large Hadron Collider (LHC) at the European Organization for Nuclear Research (CERN) is built in a circular tunnel, 27 km in circumference and 50 to 150 meters underground, extending from Lake Geneva to the French Jura. When the accelerator becomes operational, two proton beams will be fired in opposite directions and brought to collision in detector chambers. Scientists estimate there will be around 600 million collisions per second, providing a correspondingly enormous amount of data to help answer essential questions in physics.
Some of the demanding technical requirements include:
- 9,600 magnets to guide the proton beams, including more than 1,200 superconducting dipole magnets each 14.2 meters in length;
- The magnets are first cooled to -193 °C using gaseous helium. The temperature goes down progressively by using over 10,000 tons of liquid nitrogen, and then to -271 °C with almost 60 tons of liquid helium;
- Of the four main detectors in the LHC, the 46 meter ATLAS is the greatest apparatus of its kind.
- It is used as a multi-detector, among other things, to detect the mysterious Higgs boson and dark matter particles;
- A modular service vehicle, called the Train Inspection Monorail (TIM) can travel through the entire accelerator tunnel on a monorail;
- Hundreds of Simatic S7-300 and S7-400 controllers, including 36 powering interlock controllers (PIC) with functions for the power supply, ensure high availability and reliability of all critical systems; and
- The supervisory system PVSS (process visualization and control system) provides visualization and monitoring of most of the control systems in the LHC. The annual data traffic of about 15 petabytes (15 million GB) would fill more than 1.7 million double-sided DVDs.
It was CERN’s desire to use commercially available controllers and processors for the automation of the systems. However, the controllers had to demonstrate their reliability in a series of rigorous tests, in a way that has hardly ever been performed and continues to be performed for any other application.
Protection against hackers
“Redundant installations such as the Simatic S7-400H fault-tolerant type of controllers may offer a high degree of operational safety. But given that most controllers, field devices and even actuators are now directly connected to Ethernet, who can guarantee that no one will take over the controller, crash it and compromise its security?” asks Dr. Stefan Lüders from the computer security team of the IT department at CERN.
The team led by Dr. Lüders developed a special test bench to examine the vulnerability of controllers, SCADA systems, and other Ethernet-connected devices to cyber attacks. They were concerned not only about hackers with criminal intent trying to gain access from the outside, but also viruses and worms that can be introduced through a variety of channels—including USB sticks and compact flash cards. In contrast to the usual patches that can be installed in an office environment, controllers cannot be easily updated daily with the latest antivirus protection, even if it is available.
As part of the validation of controllers, 31 devices from seven manufacturers were systematically tested for penetration resistance with the vulnerability scanners Nessus and Netwox. In addition to interference through overload (such as denial of service attacks), the tests also included provoked attacks on vulnerabilities in operating systems by infiltration of malicious software and malicious manipulation of TCP/IP-based protocols. About one third of the tested devices failed.
Those test results led to a “very productive interaction with Siemens” and ultimately made “Simatic controllers significantly more secure over the years; now they meet the stringent requirements at CERN,” Dr. Lüders says.
Protection against proton bombardment
The field devices of the Simatic ET 200M distributed I/O were tested for their resilience and bombarded with protons for this purpose. This durability test targets the I/O cards in particular.
Siemens engineers maintained the CERN mean time between failures (MTBF) by exchanging the optocoupler on the cards.
Another example is the robust control of the 1,400 helium supply valves. The enclosure of the Sipart PS2 electro pneumatic positioner located on the valves only contains passive electronics, which are resistant to radiation. The active electronics are installed in cabinet drawers in parallel service tunnel or alcoves. The operational safety of the entire LHC is guaranteed by a system of powering interlock controllers (PIC), consisting of a total of 36 Simatic S7-300 controllers.
The PIC ensures that all safety conditions are met prior to the powering and during the operation of the magnets.
When critical events or failures occur, the proton beam can be quickly switched off within a few milliseconds. The reliability was demonstrated during the initial startup of the LHC in September 2008.
Redundant detector safety
A detector safety system (DSS) is responsible for the immediate status monitoring of the detectors and the protection of essential detector equipment. It consists of a controller-based front-end for safety-critical tasks and a SCADA back-end for configuration and monitoring. Two fault-tolerant Simatic S7-400 controllers executing the same process code in constant synchronization operate in the front-end, independent of the backend.
If a problem occurs, the more functional controller automatically assumes exclusive operation until the other has been updated.
The operator of the back-end computer runs PVSS (process visualization and control system) from ETM, a subsidiary of Siemens AG. It determines the information to be collected and analyzed and the pre-defined safety measures for the frontend controllers. The code executed at the controller end is identical across the DSS installation. This code is entirely data-driven, with data taken from the PVSS configuration database. This enables the DSS to be adapted to the form, implementation, and evaluation of the experiments at any time.
World’s longest refrigerator
One of the biggest challenges in the automation of the LHC, however, was the safe cooling—or cryogenics—of the superconducting magnets guiding and accelerating the two beams. In this “longest refrigerator in the world,” 16 Simatic S7-400s each control about 250 closed loops and 500 alarms and interlocks within a cycle of less than 500 ms.
The 15,000 radiation-tolerant sensors and actuators in the direct vicinity of the magnets are accessed via Profibus or WorldFIP fieldbuses, communicating over a few kilometers of optical fiber.
Each pair of controllers is flanked by eight front-end industrial PCs interfacing the WorldFIP bus lines.
There are also a total of 52 remote switch boxes for 5,000 cryogenic instruments at the eight injection points of the LHC. At point 4, the two opposing proton beams are accelerated ultimately to 7 TeV (Tera electron volt) by the superconducting RF resonators.
In each sector, one of the two S7-400 controllers is assigned to a 2,460 metre arc section in the LHC.
The other controls the cryogenics in each of the 270 meter straight sections located near the injection areas.
1 million monitoring channels
The PVSS software was standardized CERNwide and advised for all SCADA functions in 2002. According to Dr. Enrique Blanco, from the Industrial Controls and Electronics group, PVSS was selected from a total of more than a hundred competitor solutions.
To help illustrate the gigantic control proportions involved in this project, consider that each of the four large detectors contains more than one million I/O monitoring channels. In addition, there is a variety of sub-detectors that can also operate as standalone systems if needed. This requires a highly distributed architecture of hundreds of computers.
Train Inspection Monorail
The TIM is a driverless inspection vehicle that travels through the entire accelerator tunnel suspended from an overhead monorail. It was developed for use when entering the tunnel would be too dangerous, for example, during tests, commissioning, or cryogenic cool-down of the magnets.
“We must ensure that the vehicle will always stop immediately if it encounters people still in the tunnel, or an unexpected obstacle,” says Keith Kershaw, head of the handling technology team.
TIM is equipped with a laser scanner for this purpose, controlled by a fail-safe Simatic S7-300, which triggers the emergency stop through the Profisafe protocol. IWLAN (Industrial Wireless LAN) components are used for the communication between the modules.
Michael Babb is editor of Control Engineering Europe. Nicolas Mader of Siemens AG Industry Automation Switzerland, and Dipl.-Ing. (Univ.) Karsten Schneider of Siemens AG Industry Automation, Nuremberg, contributed to this article.