GM’s Net Keeps Production Moving
When designing its global command and control network, General Motors executives' first priority was supporting production operations.
General Motors recently implemented a sweeping revamp of its global manufacturing information technology (IT) infrastructure. Understandably proud of the new system, Marc Brailov, GM information systems & Services news relations manager, characterizes it as “a plant IT strategy transformation.” GM has built on the fundamentals of manufacturing to create a bottom-up control network that honors the hierarchy of its plants and the needs of its people.
“Over the past three or four years, we have been working hard to improve the flexibility and agility of our manufacturing processes,” says Kirk Gutmann, global information officer for manufacturing and quality, who was chief architect of and the driving force behind the Plant IT Strategy Transformation project. “We’ve been working out a plan to align our systems, controls, and IT strategy behind that to provide flexibility, agility, and scalability as we continue to build facilities in smaller markets across the globe.”
GM’s earlier plant-network architecture was based on proprietary local area networks (LANs). Interfacing to the IT world was typically done controller by controller strictly for sharing specific-vehicle option information and for communicating non-conformance breakdown situations up to a host for fault annuciation. As the strategy evolved, interfaces shifted to support EtherNet/IP for device-to-device communications.
GM’s Global Command and Control Center communicates through three regional command and control centers to 185 manufacturing plants around the world.
“What we try to do is to break it down into a hierarchy,” Gutmann continues. “We start with the station, then the zone, and we hook things together through the network. Then we ramp that up to the plant level.” Every plant is self-contained, he says, and they can do their own monitoring and troubleshooting, and work together to solve those issues. “Then we forward [information] on to a command center that can help the plant, and also work during off-hours when we have issues with that particular plant,” he adds.
Ken Knight, executive director of vehicle manufacturing engineering for automation and controls, says, the strategy “drives flexible uses of automation around the world, helping us support the ability to build products from one facility to another. In doing so, we use common equipment sets for each of our facilities. We try to have solutions that are common between sites, that allow us to deploy technology and automation quickly and very cost effectively.”
The hub of the system, which GM operates with help from technology services provider EDS, is the General Motors Global Command and Control Center (GC3) in Pontiac, MI. From there, high-speed data links reach out to three regional command and control centers (RC3) strategically located to serve local production plants around the world.
An RC3 in Rüsselsheim, Germany, serves facilities in Europe. A similar regional facility in Sao Paulo, Brazil, serves South America, Africa and the Middle East. The third facility, located along with the GC3 in Pontiac, MI, serves all of North America and Asia Pacific.
Each regional facility connects directly to local command and control centers (C3s) on site at each of the company’s 185 manufacturing plants. These C3s reach down to the production lines, even into individual workcells.
What makes this system stand out is not its hierarchical structure, but the psychology behind it. “The ultimate purpose,” says Knight, “is to support operations at the workcell level.”
The difference this focus makes becomes clear when one looks at how the system’s architects designate levels in the hierarchy. “Global Command and Control Center” is a grand title that sounds really important, but that’s not what GM calls Level 1. Instead, Level 1 resides in the individual manufacturing plants. Level 2 is the three RC3s, and the GC3 is at the bottom — Level 3.
“We recognize that our primary objective is to produce cars,” says Gutmann. “Everything we do on the IT side is directly aimed toward that goal.”
“The architecture for the plant floor level is based on a PLC,” says Knight, “which is our brains for the majority of the command and control activity done in our manufacturing sites.”
GM is also a very large consumer of robotics, using robots for everything from holding weld guns to painting cars. So that’s another key part of the architecture.
At the Lansing, MI, assembly plant, robotic systems team with humans to work more effectively than eithr could alone. Here, having a robot carry a human inside a vehicle avoids awkward and stressful crawling around to install the interior. Source: General Motors
“We manage the programming of those devices as well as the interfacing of those robots to other devices in our architecture,” says Knight. “There are things like weld controllers that we consider key in our architecture, and we manage that interface and the programming of those devices.”
In a typical North American facility, GM would have a high level of automation that would do everything from welding of metal bodies to painting of vehicles, and all the conveyance and material handling between those processes. Automation would also be used for specific quality control related issues such as repeatablility, reliability or ergonomic assists for workers.
Name of the game is support
While level 2 and 3 centers carry the term “command and control” in their names, their primary function is really support. Recognizing that all the real action happens on individual production lines, the GC3 and RC3 centers are set up and function as technical support centers for production. Their activity is mainly event-driven. If there is a problem that threatens production throughput, they swing into action, providing whatever it takes to avoid slowdowns or shutdowns when possible, and getting systems up and running quickly when it’s not.
“Most conditions are solved locally,” Knight points out. “Very seldom is there a system failure that is solved from a remote location. If there are high-level problems primarily in the backbone of the information grid, they can be solved from the command center.”
Gutmann elaborates: “As soon as we see an alarm, we immediately call down to the area to see what’s going on. If it’s something they can handle, or if they’re just taking a piece of equipment off line for adjustment or during a break, that’s fine. If we start seeing something that’s more problematic, we’re going to scramble a team to get down on the shop floor, help people with that piece of equipment, and use some of our technicians as well as people with factory floor experience to troubleshoot it and get that piece of equipment back on line.”
GM’s production operates in a high-mix, high-volume mode. With so many options available, automobile consumers are increasingly designing their own vehicles as special orders. Simultaneously, GM’s production lines build these vehicles in volumes that mass-production pioneers could only dream about. It is only possible because the company’s IT networks are integrated from top to bottom, and capable of tracking each vehicle from the time it is ordered to the time it reaches the customer.
Purpose-built automated systems position the heavy vehicle body onto the chassis precisely. Source: General Motors
Each vehicle is built to a recipe based on the options ordered. As the vehicle moves through the fabrication and assembly process, a “cyberspace ghost” tracks along with it. When a radio is installed, for example, the ghost is augmented with information about radio make, model, and serial number, who installed it, and results of any tests run on it. This data capture takes place for every other action taken to build the vehicle.
Using the ghost, a vehicle can be tracked backwards from the time it leaves the factory, through final test and assembly to see the physical components leaving as part of a completed vehicle from the final assembly plant in, say, Lansing, MI, started out as individual subassemblies from suppliers scattered over the Earth. Similarly, the vehicles’ information ghosts combine data characterizing those components from the scattered facilities that made them.
GM’s network is an intranet of global scale, with connectivity into every individual workcell, gathering information about details as small as the final torque achieved by each airgun tightening each bolt. All this data is funneled into massive enterprise-level databases.
If, five years from now, an issue arises about a certain bolt in a certain vehicle, GM will be able to find out who installed it, which airgun the installer used, what the airgun’s torque setting was, whether it achieved that set torque, and who inspected it.
While this information system is impressive, archiving data against possible future criticism is not what Gutmann’s global IT infrastructure is for. Rather, it provides tight communication linking production systems, where technical problems can cripple throughput, to support people who can resolve them.
“Lost production costs $120 per second,” points out David Scott, process information officer for GM’s North American Information Systems & Services, “and lost time can never be recovered.”
“We have a common architecture that we use from a control standpoint,” Knight says. “The architecture drives key components: processors, robotics, conveyors, mechanical and electrical designs, and control software and hardware.”
Gutmann says the first thing GM did was standardize the technology. “Then, we started to focus on the process behind operating that technology, and the integration of that technology with the factory floor,” he says.
Structure, repeatability, good people
Gutmann explains: “We’ve standardized the way we address our IP devices in a controls network. We’ve standardized the way that we prioritize that information on the plant floor and how we monitor the networks. So all the processes behind troubleshooting, monitoring, and addition and removal of equipment have all been standardized, and people across the globe have been trained with the same methodology.
“We engineer our networks just like we would engineer a new engine line, with the same kind of discipline, same kind of structure, and the same kind of repeatability. There are lessons we learn along the way. As we make changes, we want to make sure that our next plant and our next upgrade reflects those lessons learned.”
The network infrastructure is, of course, just enabling technology. What provides the value is the personnel. With few exceptions, the people at the RC3s and GC3 have been recruited from production positions. “We’ve tried to reskill and train our people to adopt this new technology and, more importantly, we’ve tried to get people to understand what their jobs are,” Gutmann says.
Typically, about 85% of GM employees in the IT space have worked in manufacturing, or in the controls areas within manufacturing, says Gutmann. “I think that does a couple of things for you. One is that you understand the importance of time: if there is an issue, it’s all hands on deck until it’s resolved. I also think you learn a little bit about fail-safe. You also learn a lot about robustness, and designing things in place so that you minimize the outages to the environment,” says.
Gutmann summarizes: “As we’ve looked at applying some of the new [IT] technologies to the plant floor, I think one of the keys is not losing the fundamentals of manufacturing. This is where a lot of people make a lot of mistakes. If you look at the design of a factory environment, it’s important that you look at it from from a hierarchy: span of control of people, span of control of an area of the plant, of the automation, or, if it’s a manual line, of the people themselves, and even the skilled trades and the people who have to work in that environment. Standardized work with standardized systems and processes, with employees working together as teams via cross-functional rotation of employees between functional groups are all critical to creating the kind of environment you want to have.”
C.G. Masi is a senior editor with Control Engineering. Contact him at firstname.lastname@example.org .
Production alarm! Diagnosis and response at General Motors
DBLQ%% says Kerk Gutmann, global information officer for manufacturing and quality at General Motors.
Wait a minute! “… gets a call from the C3 …” Don’t production people have to ask for help?
“No,” says Gutmann, who was chief architect of and the driving force behind GM’s Plant IT Strategy Transformation Project. “Engineers at the C3s constantly monitor production activity. At the Lansing facility, for example, a bank of flat panel displays show summaries of production activity in real time. Software can identify and flag any anomaly instantly. If the data doesn’t return to normal within 5 minutes, the system alerts a human.
At that point, the engineer has several options. The first thing to do is to click on the HMI display to open up another screen with a more detailed view of the situation. This usually happens long before the 5-minute warning. Depending on the situation’s severity, the engineer can continue monitoring activity, or take action.
The first action is to contact the people on the production line. Most issues can be resolved quickly by the people at the point of failure, and many don’t actually affect production.
Suppose, for example, data coming into C3 indicates that an airgun is not torquing bolts tight enough. There are several possibilities: operator not holding the gun on long enough, low air pressure, accidental resetting of the gun’s clutch, mechanical failure of the tool, and sensor failure. The appropriate action depends on the actual cause.
Easiest to rectify, of course, is human error. Someone — most appropriately the line supervisor — needs to tell the operator that saving a fraction of a second isn’t worth mis-torquing a bolt.
Low air pressure means diagnosing the problem with the air supply. Is there a leak? Perhaps a hose got kinked. Has the compressor motor failed? These are issues that C3 will be able to see before anyone on the production line. Any general system failure will affect several stations, for example, and show up on the C3 HMI as multiple point failures.
The line supervisor or assembler is in the best position to pick up on a clutch mis-setting or tool failure — provided they know there’s something wrong. “I thought it felt odd,” is a not-uncommon response.
Those problems call for immediate action. A failed tool, for example, calls for shutting the line until the tool can be replaced. GM customers will not be happy about rattles or squeaks in their new vehicle because it was assembled with a faulty tool.
A failed sensor, on the other hand, is another matter. If the tool is actually doing its job, and only reporting bad data, the assembler can keep going while a replacement is brought in. C3 engineers can annotate database entries to indicate that the readings were bad, not the work.
How would you know it was the data, not the work? The assembler, who has used that tool to tighten hundreds, or even thousands, of fasteners correctly, can tell if there’s nothing wrong. “It just feels right.”
The moral of this story is that solving industrial production problems calls for a team effort. Someone with a view of the entire production line needs to be part of that team. That someone is the C3 engineer.