Flexible Batch Proven in First Week
Mead Johnson Nutritionals (Evansville, Ind.), a world leader in adult and infant nutritional products, sought to become flexible and quick in adapting new products at its 32 oz. processing center. Engineers there needed to replace an obsolete control system with technology capable of supporting global enterprise integration and reduce process variability caused by manual interventions.
Mead Johnson Nutritionals (Evansville, Ind.), a world leader in adult and infant nutritional products, sought to become flexible and quick in adapting new products at its 32 oz. processing center. Engineers there needed to replace an obsolete control system with technology capable of supporting global enterprise integration and reduce process variability caused by manual interventions. They also were given a two week window to remove the old system, install the new system, perform all testing, and return to production status.
Mead Johnson’s 32 oz. process center produces 15 liquid formulated nutritional food products for two very different markets. The first is liquid infant formulas including Enfamil, ProSobee, and LactoFree. The second is liquid adult nutritional formulas such as Boost, Isocal, and Sustacal.
Formulated food products, especially infant formulas, require consistency because the product may be the sole source of nutrition for the consumer. Major food components, including proteins, vitamins, minerals, fats, and carbohydrates, are precisely blended to produce a nutritional product in accordance with Food and Drug Administration guidelines and regulations.
Producing multiple products in the processing center requires piping and equipment usage flexibility to deliver product to the 32 oz. ready-to-use-container packaging area, as well as other packaging areas.
How it began
Mead Johnson’s 32 oz. process center was constructed in the early 1970’s. The original control system included panel-board analog instrumentation, chart recorders, and hard-wired relays providing motor control and sequencer logic. In 1981 part of the original control system was replaced with a Foxboro Micro-Blendtrol and several PLCs (programmable logic controllers), but operator interface remained primarily pushbuttons, switches, lights, and panel board instrumentation.
To maintain blending consistency, a custom communication protocol was developed to support integrating the Micro-Blendtrol and the PLCs.
A custom built, stand-alone programmable controller to perform clean-in-place (CIP) activities was also installed.
Process valves shared by the control system and the CIP system were installed with double wiring, making maintenance and interlocking a nightmare.
Over the years, functionality changed, and devices were added and removed. As is often the case, drawings and documentation did not keep pace with the changes, making troubleshooting and maintenance challenging. Adding to the confusion, much of the wiring from the original control system remained in place, "just in case it was needed." Sound familiar?
In the mid 1990s, a PC with data acquisition software was connected to the main PLC. This interface provided operators with rudimentary graphical views of the process, but provided no control interface. "In retrospect, introducing a PC interface was a good first step and paved the way for operator participation of what a new control system should look and feel like," explains Bill Smith, control system maintenance facilitator at Mead Johnson.
Discovering what existed
The project team by its makeup was committed to executing this project differently. The team consisted of a process engineer, a project manager, a maintenance facilitator, a production facilitator, operators, maintenance technicians, and the system integrator would collaboratively design, implement, test, cut-over, and start-up the new system. The system knowledge gained using this project approach would yield long-term benefits at the "hands-on" organizational level.
Previous projects had surveyed operators about what they wanted, and needed, but often missed why, resulting in compromises and missed expectations. This project was going to determine what needed to be done and help people understand why. Requirements and expectations were going to be defined, documented, and delivered.
Goals of the engineering phase (phase 1) were to establish a solid knowledge base of current control system functionality and develop a clear understanding of expectations for the new system. Deliverables of the engineering phase included:
Updating and/or developing electrical drawings to reflect the current control system "as-built" condition;
Development of detailed electrical engineering, design and installation documentation;
Creation of a detailed scope of work and functional description documentation package; and
Assembling for parallel construction a request-for-quote package and submiting it to a variety of system integrators and electrical contractors.
In preparing the request for project funding, hardware and software research, along with vendor visits, was conducted to evaluate available technologies. Early findings revealed advanced batch software was emerging, but the project team would need "a-leap-of-faith" if a solution was to be installed that would meet undefined future needs.
Recognizing the new system needed to incorporate the Microsoft Windows NT operating system, PCs (personal computers) would be part of the solution.
A decision was also made to integrate process control and CIP into the same hardware and software platform. Integrating CIP into the control system was consistent with ANSI/ISA-S88.01 batch models, and would simplify design, implementation, and testing of interlocks as well as ease the coordination and management of process units, and process cells.
The project team was aware that efforts were underway to implement SAP’s R/3 enterprise management system throughout Mead Johnson, and the 32 oz. process center would need capabilities to integrate with the R/3 system in the future. The project team concluded that applications and platforms supporting OPC (OLE for process control) client/server communications would be an appropriate method to implement and manage enterprise integration with process centers.
Products selected for this project included: Allen-Bradley PLCs (Mayfield Heights, O.) for base level control; Intellution (Norwood, Mass.) FIX and Visual Batch software for control interface and recipe management; Total Control Products remote terminals (Melrose Park, Ill.); Intercolor remote graphic terminals (Duluth, Ga.); and Cape Software (Houston, Tex.) for nonintrusive process simulations.
In October 1997′ approval was received for a 3-phase project (engineering, system integration, and construction) to upgrade the 32 oz. process centers control system to meet the following requirements:
Deliver a solution that eliminated barriers to new product adaptability;
Deliver a solution that minimized manual intervention contributing to process variability;
Deliver a solution that eliminated the risk associated with obsolete control system components;
Be sure 32-oz. process center was not out of production for more than two weeks;
Ensure the new control system was ready for production on July 24, 1998, without excuses; and
Keep the project within budget.
Mead Johnson’s project team took great care in selecting the engineering firm for phase 1. For the project to be successful, the team was convinced the correct engineering firm needed technical competence, flexibility, and empathy with the schedule; the winner also needed to be void of bureaucracy and stubbornness. "We were looking for a firm who could partner with our people, understood our needs, and could make the most out of our existing documentation," explains Richard Noel, senior process engineer.
Phase 1 activities were awarded to Malisko Engineering (St. Louis, Mo.) with effort commencing immediately and completing in December when the request for quote package was completed.
Mead Johnson’s project team was joined by Malisko engineers and phase 1 tasks underwent detail planning to co-exist with production schedules. The team willingly accepted additional assignments and engaged in phase 1 activities. "Enthusiasm was so good, that individuals began to worry about details outside their control __ a healthy situation, but one that required frequent reminders to remain focused and work efficiently and accurately, for everything to come together in the end," explains Gregg Easley, 32-oz. operations facilitator.
Even though phase 1 consumed about 15% of the project’s funds and 20% of the available schedule, it laid the foundation for success via the knowledge shared, documentation developed, detailed schedule produced, and team enthusiasm. When asked if members should develop this much detail, and consume this much time in a future project, the entire project team unanimously replied "Yes!"
Al Kroupa, project manager confides, "Enthusiasm, cooperation, and quality work were produced from the beginning. I knew the assembled team could and would do what was necessary to deliver the project on time."
Shortly after welcoming 1998, the project teams first order of business was to evaluate competitive bids, select a system integrator, and complete phase 2 and 3 within the 27 remaining weeks.
Malisko Engineering’s bid was selected and hardware was shipped to its St. Louis office.
Mead Johnson’s process engineer, operators, and technicians began making weekly trips to St. Louis to assist in every aspect of system development. Working around production schedules, days off, and other logistical issues, operators continued participating in reviews and testing. Technicians assembled control system hardware, constructed networks, and installed software. No one observed; everyone was engaged.
Hardware and network communications were tested beyond any previous project. For example, in addition to making sure remote operator interface panels communicated every command, every response, and every message was verified from end-to-end several times and under various system operating conditions.
Relying on detailed documentation developed during phase 1, the control system application software began to develop one process unit and one process cell at a time, closely adhering to ANSI/ISA-S88.01 models.
To obtain operator feedback on usability likes and dislikes, a process unit base level software, including database, graphic screens, and process simulation was rapidly created. Testing began as soon as the base level control was ready. Changes were immediately communicated to engineers working on other process units so consistency and improvements would be universally applied.
Throughout the project, software logic was tested during the day, corrected and made ready for retesting the following day. Operators and engineers repeated this cycle until everyone was satisfied the software met the defined scope and would produce a quality product.
Phase 1’s design approach used a top-down iterative process, making extensive use of flow charting, with each iteration adding detail. The documentation produced was understood by all team members and was used during testing to verify the implemented system operated as planned through all conditional branches.
Phase 2 progressed using a bottom-up implementation and testing style. Unit base level control was developed and tested first. Completion of testing at one level released the process unit or process cell for the next layer of implementation, and testing. Once a unit or cell was completed, unit coordination testing was conducted. By the time the recipe layer of software was completed, lower software levels had experienced many iterations of use (see Software architecture).
As promoted by ANSI/ISA-S88.01 proponents, the team proved development of small, reusable, and stand-alone phase logic modules to be efficient. It made no difference which recipes used which modules; the modules always performed the same.
Early establishment of an interlocking philosophy placed safety interlocks at the base (PLC) level and process interlocks at the phase logic (Visual Batch) level.
Safety interlocks were defined as interlocks necessary to protect people and equipment, regardless of how the process was being operated. Process interlocks were defined as conditions necessary to ensure product quality.
Similar to needing safety and process interlocks, quality products can only be produced when the entire control system is available. With control distributed among several PLCs and with redundant batch management servers, a watch-dog-timer (WDT) routine was necessary to monitor system processor health and inform of system failures. Each phase logic module monitors the WDT and branches to failure logic when a system failure is detected.
A daily challenge the team faced was what became known as "featuritus." As software capabilities were learned, request to make the solution "just-a-little-better" began to cross the project manager’s desk. This is when the experience of core team membership paid dividends. Each opportunity was evaluated. If it was not a must-have, or did not offer a positive impact to the project schedule, it was documented and targeted for implementation after startup. Changes remained minimal, and the installed control system, very closely mimicked the design developed in phase 1.
Moment of truth
July arrived and it was time to move the entire system from St. Louis to Evansville and begin installation. But before installation could begin, demolition of the old system was necessary. With a two week window to remove the old, install the new, test the installation, and conduct water and product trials, there was no time for wasted efforts.
Because of how the old control system had evolved, the PLCs in 32 oz. processing center contained logic that effected other processing areas. Removing these PLCs would have adverse effect on bulk product receiving and two other processing centers. Accurate, quick, and detailed cut-over planning needed to eliminate the reliance of one process area on another.
Weeks before shut-down, detailed planning meetings were held. Participants from all process centers were invited to assist in developing a detailed cut-over plan that identified every task, established task priorities, assigned each task to an individual, and defined what day and shift each task was to be completed. When finished, the detailed schedule was made available for the entire site to view. Four days were allowed to install new PLCs in bulk receiving and the two process centers and release the areas to production…as it turned out, four days was enough.
Selecting an electrical contractor for construction (phase 3) was an important milestone the project team faced. Contractor selection criteria included: adequate and experienced staff; willingness to help identify activities that could be performed prior to shut-down; commitment to completing within the two week window; and whose employees were willing to ignore job titles and integrate with the project teams.
Working around the clock, teams executed the cut-over plan, completed water and product trials, and had 32-oz. processing center ready for production one day earlier than planned.
Flexibility of the new system was immediately challenged, and the challenge was met. A new product recipe was developed and added to 32 oz. processing centers recipe library in a matter of hours, a process reported to require more than a week using the old system.
Mead Johnson declared this project a success because it met defined goals, and proved it’s possible to immerse a multi-discipline project team to quickly, accurately, and economically implement a process automation solution.
One improvement the project team recommends is development of a more efficient method for documenting detailed designs. This project used detailed flow charts, however the team believes sequential function charts may be a more efficient design tool (see sidebar story for related information).
One thing the project team would not change was involving many stakeholders in details of every project phase. "It was total project immersion that helped fulfill our commitment to develop knowledgeable people at the hands-on organizational level," says Al Kroupa.
Mead Johnson’s project team stopped to pat one another on the back, but only for a moment. Already new productivity challenges have arisen and the solution implemented in 32 oz. processing center is poised to take advantage of every opportunity.
Flexible automation at a glance
Engineers at Mead Johnson used collaborative techniques to solve problems common to many control system projects.
Project team challenges include:
Obsolete hardware with custom interfaces;
Process variability because of manual intervention;
Poor and missing documentation;
Install project in two week window; and
Control system software difficult to change and test.
To overcome these challenges Mead Johnson’s project team decided to:
Use ISA-S88 models;
Select a system integrator partner;
Form a multidiscipline project team;
Use non-intrusive process simulation; and
Have operators conduct software testing.
Source: Control Engineering
Coming soon, batch industries next big benefit
Between 1988 and 1996, a community of batch experts produced the first document describing models and terminology to explain complex batch processes without knowledge of or prejudice to vendors’ products.
Before receiving official standards status, ANSI/ISA-S88.01 Batch Control Part 1: Models and Terminology was heralded, and used to design flexible batch processes. Several vendors began developing products based on the concepts revealed in draft releases of S88.01.
Many users do not appreciate how S88.01 describes the physical and procedural elements necessary to achieve a flexible batch solution without explaining what these elements should look like.
This lack of detail, especially for recipes, was not an oversight. The S88 committee wanted time to fully examine and document requirements for defining and constructing batch recipes in a separate document, leading to the birth of dS88.02 Batch Control Part 2: Data Structures and Guidelines for Languages.
ISA dS88.02 is focused on data that must move into, out-of, and within a batch process cell. Lynn Craig, S88 committee chairman, explains, "S88.02 doesn’t define the interface between systems, but how to depict a master recipe and how to transmit data necessary to define a master recipe or a batch schedule, for example. The methodology for data movement described in dS88.02 relies on features provided by relational databases. For example, intent is not to move the recipe, but to allow movement of data necessary to recreate a recipe. Recipes defined in S88.01 contain five categories of information: procedure, formula, equipment requirements, header, and a category named ‘other information.’, with formula further divided into process inputs, process outputs, and parameters. All data types have been addressed in the dS88.02 standard. When the standard is finished, many of the questions left unanswered in S88.01 concerning recipe categories, will be addressed."
Technical report ISA-TR88.03, issued in 1996, discussed three possible methods of depicting recipes; table, sequential function chart (SFC), and Gantt chart formats.
According to Mr. Craig, the dS88.02 committee determined none of the three formats fully addressed batch control recipe needs. The standard needed formats providing clear separation of recipe procedures and equipment procedures. The committee agreed equipment procedural elements, such as equip-ment phases, could be described using traditional SFC format. However, recipe procedures have unique require-ments not fully solvable within IEC-60848’s (formerly IEC-848) function chart standard.
In IEC-60848 compliant function charts, an active step is terminated when the conditional expression (transition) following the step (state) becomes true. At that instant, a token held by the active step is passed to the step following the transition. This instantaneous and irrever-sible transition from active to inactive is undesirable in most batch processes. For example, S88.01 permits one operation to be active on a unit at a time. Opera-tions are constructed of multiple phases with no limit on the number of phases that may be simultaneously active. Thus, a recipe procedure should be aware of active operations, but may not be aware of all active phases within those operations. Since phases may be executing exception logic, the need for orderly completion is essential and should not be prematurely interrupted by a recipe transition condition becoming true.
To provide consistent depiction of recipe procedures, ISA-dS88.02 introduces Procedure Function Charts (PFCs) combining the strengths and avoiding most of the weaknesses of SFCs and Gantt charts. Based on IEC-60848 function chart step/transition methods (and adding intuitive visual features provided by Gantt charts) PFCs can show what needs to be done.
Development of PFCs maintains separation between recipe procedural elements and equipment procedural elements introduced in S88.01. Mr. Craig explains, "The difficulty in applying SFCs is, recipes instruct what to run, but execution occurs elsewhere. Once an equipment procedural element starts, it needs to execute independently and respond to a recipe transition condition consistent with its own logic. This often creates delays between when a recipe transition becomes true and when an equipment procedural element actually completes."
When dS88.02 is completed (targeted for early 1999) the batch industry will have a nonproprietary batch recipe depiction and data transfer standard for on-site recipes residing in enterprise systems and control systems.
The "meat" of dS88.02 is contained in three sections.
Section 4 defines data models and discusses how these models relate to the reference models introduced in S88.01. These data models form the framework for uniform modeling and description of batch automation components. (In database terms, section 4 specifies a standard set of objects and defines their relationships.) The S88 committee is quick to point out that models presented in this section are not intended to specify standardized system architecture for batch control system, but exist to present a common reference model.
Section 5 explains data structures and their associated exchange mechanisms, including how recipe data is to be managed to support recreating a recipe. Contained in section 5 is information to support exchange of master recipes; to describe capabilities and specifications of process cells; to allow exchange of batch histories; and to exchange batch schedules.
Section 6 presents language guidelines that specify how recipes should look and act. This section describes table, SFC, and Gantt chart formats, and introduces PFCs.
Most batch management vendors claim product conformance to S88. Many have chosen sequential function charts to depict recipe execution, but have deviated in some way from the IEC-60848 SFC standard to overcome the problems explained above. Not surprising, each vendor has "solved" the problems in its own manner. "When dS88.02 is released, many of the differences between recipe management products claiming S88 conformance should disappear," predicts Mr. Craig.
We can only hope Mr. Craig is correct. ISA-S88.01 has proven beneficial by allowing batch users and batch solution providers to "sing from the same sheet of music:" dS88.02 provides hope that everyone may sing "in tune."