System integration project answers
System integration projects for automation devices and systems can be complex. Those involved, internally or externally, may benefit from advice given by representatives of two of the 2017 System Integrator of the Year winners. The representatives discussed virtualization, project management, IT and OT integration, thin clients, code conversion, and mobile devices, among other topics.
Answering questions jointly for Panacea Technologies were Will Aja, vice president of customer operations, and Abhijit Jog, vice president of projects.
Shawn Campion is president and CEO of Integro Technologies.
Jeff Miller is director of project management, Interstates Control Systems Inc.
[This online version of this article contains more information than the print version and links to prior roundtable coverage, below.]
Roundtable, Part 3
CFE Media: What is a leading trend in last few automation integration projects?
Aja/Jog: The biggest trend we have seen in recent years has been virtualization of the computer infrastructure and the increasing use of consumer devices such as tablets and cell phones for process interaction and alarm annunciation. The resulting convergence has required integrators to be experts in both the information technology (IT) and operations technology (OT) portions of a project. Clients want automation design and configuration as they always have, but now they are asking for network design and implementation, cyber-security, building automation system (BAS) integration, and other ancillary services together as part of one project or contract. Panacea recognized this trend several years ago and made investments to respond accordingly.
Campion: The last few integration projects we have been reassessing established machine vision inspection and imaging techniques and incorporating deep learning (artificial Intelligence), neural networks, field programmable gate array (FPGA) and graphics processing unit (GPU) processors, and hyper-spectral imaging into some new and established solutions. The combination of deep learning with hyper imaging has permitted us to explore solutions that were previously unable to be solved using traditional image analysis because of the number of complex dynamic variables set for defect classification. The increased image data density of the hyper-spectral data cubes in conjunction with a more intuitive training technique leads to a excellent end user application solutions.
Miller: One of the big changes we are seeing is larger and larger legacy control-system replacements in existing plants that cannot afford to be down for more than just a few days. System integrators (SIs) are being pushed to find ways to drastically reduce the amount of time needed to do a full rip and replace on the control system. This takes tremendous attention to detail and lots of pre-shutdown planning.
CFE Media: What technology-related advice would you offer as a result of these projects?
Aja/Jog: It is becoming vital for growth as a company, as well as measured success on projects, to make investments in new technologies and diverse skill sets. It is no longer sufficient to have a staff of engineers that are hyper-focused on process, automation and supervisory control and data acquisition (SCADA) system design. It’s important to have networking, database, security, and virtualization experts as part of a high functioning and diverse team. Controllers and physical human-machine interfaces (HMIs) are becoming more of a commodity every year, and growth in the software, IT, and cybersecurity space is becoming the most important factor for successful projects.
Campion: Using and applying deep learning/neural network still requires an inspection specification and clear definitions of what constitutes a good, bad, or marginal product/assembly from the customer. The new image processing and classification techniques can completely reverse the standard machine vision integration progression and stages. For example, large inline production image databases across multiple shifts, batches, and/or SKUs are required to be captured prior to the start of system training and pre-classified by the customer’s quality department. The image databases are critical to execution and construction of an algorithmic model.
Miller: When available the SI should use the hardware conversion kits that control vendors are now offering. They don’t solve all the issues but certainly can drastically reduce the amount of time needed for rewiring. The SI also needs to consider whether or not to use vendor-provided code conversion tools. We have found that for most process plants we work in the conversion tools are not adequate, and in most cases we will code the system from scratch using code generation tools we have developed for our own use.
CFE Media: What workflow or project management advice can you offer?
Miller: For these types of retrofits it is critical that the project manager (PM) and team follow very strict documentation guidelines as team members spend time on-site looking for potential issues. It is critical for engineering staff to spend time on-site to validate if hardware kits will work effectively. They also need to look for potential wiring issues that may have been introduced over the years as changes were made and not documented in the drawing package. With a shutdown of only a few days one of these surprises can really affect the ability to be successful.
Proactive risk management is another key to success. You need to spend more time thinking of what risks could affect the project and build plans to mitigate those risks. Another huge success for our retrofit teams has been to switch to more of an agile project delivery methodology. Tasks are defined down to the level of eight hours or less, and each task includes a description as to what "done" really looks like. This allows team members to jump on any task that is fully-designed and planned and once executed move on to the next task very quickly.
CFE Media: What system integration advice can you offer?
Miller: One of the biggest things that will affect an SI’s ability to be successful is whether or not they put the time in at the plant site up front looking for the potential issues and working through a plan to address them. When you are retrofitting a 4,000 plus I/O plant in three to four days you have to plan for contingencies and execute flawlessly.
CFE Media: What standards can help a project?
Aja/Jog: Standards are required for every aspect of the project, whether it is for graphics, controller programming, documentation, and other areas. Development of robust standards is critical to project success. Good standards are the foundation and scaffolding on which the entire project is built. End users need to pay increased attention to the quality of these standards and the ability of these standards to handle their future needs. We have seen many instances where implementation of bad standards has resulted in plants being crippled for their entire lives; production is reduced, changes become difficult, and routine maintenance takes more time.
Campion: As a system integrator we do not build a "standard" original equipment manufacturer (OEM) machine. Obtaining a complete mechanical and electrical specification prior to the start of the project eliminates potential re-work/design due to differences in standards between the system integrator and the end user. The differences may seem minor but can cause extensive engineering time expenditure or change orders contingent when the standards are provided by the customer. Adopting customer standards also can lead to faster system adoption/acceptance rates by operators, technicians, and maintenance staff.
Miller: As with most projects it is very important to set a standard for all team members to follow, which will mean that team members can switch from one task to the next seamlessly. This is especially important when running an agile development methodology.
CFE Media: Please describe elements of a recent project that were particularly successful.
Aja/Jog: We have recently been executing several projects that focus heavily on thin client deployment as opposed to physical HMI deployment. Whether for a new process line or as part of a migration effort implementing thin clients, using a thin-client management software platform has greatly improved commissioning times and has reduced the amount of time and equipment needed for testing. This creates an immediate positive budget impact, but also carries several long-term benefits by decoupling physical HMIs from the system.
The life cycle of most major HMI platforms seems to be getting shorter as new hardware releases happen more frequently. Having a server client architecture deployed on thin clients helps shield clients from these constant updates and greatly increases the life cycle of systems.
Miller: One of our more recent retrofits was in the flour milling industry, and the plant had more than 4,000 input/output (I/O) connections that we needed to complete in less than a week. We put a tremendous amount of planning into the work that needed to happen during the shutdown, and the client was extremely pleased when the plant came back up in the time frame promised. We had more than 20 people on site and had built kits for each area that needed panel work done. This kept people working on their own work and out of each other’s ways.
CFE Media: What was a particularly challenging aspect to the system integration project, and how was it resolved?
Aja/Jog: A common theme to our projects has been working more closely with the IT department. As the IT and OT spaces begin to overlap, we have found difficulty working with IT groups to set standards and architectures that won’t conflict with critical manufacturing processes. Setting standards for things like virtual machine (VM) snapshot schedules, remote access, and operating system (OS) patch management can sometimes be a very tough process with IT departments reluctant to make changes or embrace change in general. This has led to a couple instances where the control infrastructure was crippled by VM snapshots. The way VM snapshots work is by pausing and then accelerating the clock of the virtual machine, which can affect our historian data. In another instance, automatic OS patch deployment and the resulting rebooting of the server resulted in the backup and primary server being offline at the same time, causing a loss of access to certain systems. The consequences have ranged from mild operational inconveniences to loss of data, which carries severe impacts in the regulated industries.
Most situations were rectified because of an investment in IT resources several years ago which acted as a mediator in these situations, but in some cases a change wasn’t made until a major downtime event was experienced. As the automation and IT space continue to merge I expect there to still be a few rough spots along the way.
Miller: With any retrofit of this size, it’s common to find wiring issues and especially instrumentation that has not worked for years. This takes everyone working together to troubleshoot what needs to be changed to make the system work. This system was no different. Wires had been routed through panels more like junction boxes and other wiring was nothing like the schematics provided. The team had to quickly react and get into troubleshooting mode. They did so, very well.
– Edited by Mark T. Hoske, content manager, Control Engineering, CFE Media, firstname.lastname@example.org.
Consumer devices help with industrial projects.
Legacy upgrades need to happen quickly.
Software, cybersecurity, and validation are critically important.
What information here will help with your next system integration project?
See 2017 System Integrator of the Year roundtable parts 1 and 2 linked below.
Read more about these and other winners of the System Integrator of the Year award at www.controleng.com/SIY.
Use the Global System Integrator Database to match engineering specialties, vendor experience, and other attributes to your project needs.
www.controleng.com keywords: system integration, project management