9 ways to avoid crisis upgrades to automation IT
Here’s how to avoid crisis upgrades related to unsupported automation IT legacy systems that may be without documentation or anyone who remembers anything about them.
We recently had to fix a number of 12-year-old routers that were used for building automation, utility data collection, and alert notifications to maintenance and quality staff. This system fell between the cracks of automation IT and business IT support. It was standard business IT hardware, but the applications were not standard business applications and some were integrated with the site’s automation and utility systems. It was not a production critical system but was a critical support system. The system was in sad shape with intermittent memory problems and corroded connections because of the router’s locations outside of a controlled IT environment. When the system finally failed, it became a situation of, "If you touch it, then you own it."
No router documentation
No one wanted to touch the system; the interface was an RS-232 connection using a command line interface that no one understood. None of the IT staff even had a system that had an RS-232 interface. There was no documentation on the router configuration, and no one who worked on it 12 years ago could be found to help. There were no working backups of the configuration, no installation documentation, and an untested and unworkable disaster recovery plan. It was originally installed by the automation group, but no one remembered why they were involved or who was even in that group at the time. Because an automation professional was involved in the original installation, this became a case of "because you touched the IT, now you own the IT." After multiple attempts to fix the old routers, they were finally replaced with four new switches that support a web interface, with new installation documentation, and with a tested disaster recovery plan.
In retrospect, the automation group was the best one equipped to handle the situation. Automation groups regularly handle systems that are 10, 20, or even 30 years old. Automation groups regularly have to maintain internal documentation on MULES (Mature Unsupported Legacy Execution Systems). Automation groups often have experience in text-based and direct-connect interfaces that were common in the 1980s through the early 2000s. Automation groups often have to deal with production and production support critical systems and coordinate with the production, quality, and maintenance groups to schedule support activities.
Similar situations where automation IT assets are difficult to support will become more likely in the future. Systems developed in the mid-1980s and later are heavily based on commercial hardware and software, which will no longer be supported by the business IT functions. The business IT group will focus on hardware and software that is usually no older than 10 years, while automation IT will have to support systems much older. As older production and production support systems roll off business IT support, they will roll onto automation IT support.
9 ways to avoid crisis upgrades
There are some simple tasks the automation IT group can do to handle replacements and upgrades in a less "crisis-driven" manner.
1. Ensure that every system is identified and labeled (with a permanent marker, please; pencil lines disappear in 10 years or less).
2. Include on the label which department is responsible for support, and the cycle time for checking that status of the equipment.
3. Don’t assume the departments remain the same over 10 years, so be prepared to update the labels.
4. Perform an annual review of all owned devices, making sure the installation, disaster recovery, and backup documentation is available.
5. If you don’t have a formal tracking system, just set up an Outlook calendar with the review dates and other information on each review event.
6. Plan for the long-term in information management.
7. Don’t rely on the same servers being available in 5 to 10 years.
8. Don’t assume that floppy disk, CD, or even DVD readers will be readily available in 10 to 20 years.
9. Assume that the end users of the system will know nothing about the system; to them it is plumbing, hidden behind walls and only interesting if it breaks.
If you will become the group that supports MULES, it is best to be prepared for a completely different IT environment. We can’t predict the future environment, but if history is any indication, it will be wildly different from today’s web-based, Ethernet everywhere, server-based environment.
– Dennis Brandl is president of BR&L Consulting in Cary, N.C. His firm focuses on manufacturing IT. Edited by Mark T. Hoske, content manager, Control Engineering, firstname.lastname@example.org.
This posted version contains more information than the print / digital edition issue of Control Engineering.
At www.controleng.com, search Brandl for more on related topics.
See other articles for 2015 at www.controleng.com/archive.
See other Manufacturing IT articles.
See the Information Control newsletter.