Project: Biopharmaceutical filtration automation

07/11/2005


February 1, 2006

Project Status Summary:
1. Control software development code complete 100%
2. HMI software code complete 100%
3. HMI external application configuration 100%
4. MF beta testing 100%
5. UF beta testing 100%
6. FAT 100%

Factory Acceptance Testing
The factory acceptance test (FAT) was completed on Friday, which marks the end of the project until commissioning at the client site in May. The client team at the FAT consisted of the process engineers, automation engineers, validation personnel and project managers. In the test week, the team performed the following:

  1. 1. I/O testing with approved validation protocols. The testing repeated and spot-checked our point-to-point test that was performed before the client team arrived. This test helped wring out discrepancies between the protocols and the field devices. One major discrepancy was that of the instrument ranges– the protocols referenced the process range rather than the instrument span range (4-20mA). This confusion led to both re-work of the documents and reconfiguring of a few instrument ranges. This was another case where our software methodology and control modules came in handy – changing the range in the software is simply done at the control module faceplate, not directly in the code. In total, the I/O checkout took nearly six days to complete and document – four days before the client arrived and two days with them.

  2. Network and ARS Testing. We performed two tests for network integrity and data acquisition. One test proved that our system could effectively communicate with a 3rd party PLC over the network. This was done to simulate the communication between our system and others that will be in the actual plant, such as the WFI loop and CIP skid. We also tested the logging of batch events to the archiving and reporting system (ARS) using a 3rd party PC. These tests strictly tested the software, but the client deemed the FAT as the best location to perform them.

  3. Alarm, Interlocks and Status Verifications. The hard-wired alarms were tested to verify the safety devices were in proper working order, along with a spot check of software alarms and interlocks. The alarms and interlocks were all tested previously during our development testing on our simulated system. The lead designer of the unit status indicators was on site, so it was deemed appropriate to test the unit status indicators for proper functionality. We were able to execute small test recipes that sequenced through the various states, such as Post Use, Post CIP, or Sanitized, in order to prove proper functionality.

  4. Functional Testing. The process team did run some process functions using the equipment modules. They concentrated on the pressure and flow control for each skid and verified proper control strategies for each. Process recipes were not tested and will have to be further developed on site.

As is the case with most FATs, it is difficult to complete all of the testing desired in the amount of time allotted. Construction constraints usually dictate that the skids must be shipped in a timely manner and therefore, the FAT usually gets cut short. Many of the tests outlined in the FAT protocols will now be performed for the first time at the client site during commissioning. Overall, the FAT was deemed successful. Our software and computers were boxed up at the end of the week and will be shipped with the equipment to the client site. Our few remaining punch list items will be address during commissioning.

What's Next?
After we ship out the turnover package next week, the project is put on hold until commissioning at the client site, which is slated for May and June. Until then, we move on to new projects, yet reserve the time slot for the upcoming field work.


January 24, 2006

Project Status Summary:
1. Control software development code complete 100%
2. HMI software code complete 100%
3. HMI external application configuration 100%
4. MF beta testing 100%
5. UF beta testing 100%

Factory Acceptance Testing
We started the factory acceptance test (FAT) this week at the skid vendor’s shop. The first goal of the testing is to prove communication between the software and control devices, which is generally documented with a loop check qualification. We began this task on Wednesday and completed it over the weekend. As mentioned previously, this is an important task that should not be overlooked in planning for an FAT, especially when the software and the instrumentation are supplied by different vendors, as was the case in our project. With the loop checks complete, more functional testing of the skid will continue this week. The functional testing is an engineering exercise to prove that the equipment supplied does in fact meet the design requirements and allows the client to accept the equipment as built. The difficult part of this analysis, however, is replicating process conditions at the vendor’s shop, which usually has limited utilities for use. For example, during this particular FAT, clean steam is not available for steam in place operations, so any testing of this functionality will be with simulated process variables.

After the point-to-point verification, we then test the hard-wired alarms and safety interlocks. It is important to first prove that safety interlocks are in place and functional before any further testing. After the safety tests, we will proceed to loop tuning and then the more integrated process testing. The equipment modules will be run individually for both normal and abnormal conditions (such as hold and abort) and also sequenced together in mock recipes.

Concurrent with our testing, the client’s process team will also be performing physical verifications and document reviews. Their tests will include a P&ID walk down, surface finish analysis, spray ball coverage tests on the vessels, slope verification, panel inspection and turnover package document review.

Documentation
All of the tests performed are documented in a cGMP manner. We provided the FAT testing protocols for this project, which included the framework for our point-to-point test, alarm tests, and functional verifications. The documents are written with a more free form approach then a validation protocol such as an operational qualification. In FAT documents, the tests required and results are documented, but the instructions are less regimented then pre-approved protocols. The experienced process and automation engineer will need to determine the best method for each test, depending on available utilities and operating conditions. They will also need to determine whether results meet expected design criteria with or without equipment modifications. The completed FAT documents will become the basis of further testing during commissioning at the client site.

What's Next?
The factory acceptance test at the equipment vendor’s shop will continue this week.


January 11, 2006

Project Status Summary:
1. Control software development code complete 100%
2. HMI software code complete 100%
3. HMI external application configuration 100%
4. MF beta testing 100%
5. UF beta testing 100%

Customer Acceptance Testing
We completed the customer acceptance testing just before the holidays and have now packed up the system for the factory acceptance test (FAT). For the final piece of testing at our office, the customer’s automation engineer reviewed the code structure, documentation and the testing procedure we employed. The data format was also reviewed for incorporation into their plant wide system. Since the customer visit, we have updated the documents and the code per the punch list items. In total, we had 24 punch list items to address: 60 % were graphics related and 40 % were sequence or code modifications. Upon completion of these items, we shipped all of the relevant hardware to the skid vendor for incorporation into the electrical panels. These panels need to be completed prior to the equipment FAT, so we can properly and safely perform wet testing. When finished, each skid will have its own PLC and operator workstation, complete with full keyboard and pointing device for ease of use.

Factory Acceptance Testing
The factory acceptance test at the skid vendor’s shop is scheduled to begin January 18th. The first two days of the FAT are slated for I/O check out on the skids. This is an obligatory, yet often overlooked task during FAT scheduling discussions. In most cases, the FAT is scheduled as a one-week activity, but depending on the amount of I/O, it may take 2 full days to finalize the I/O check out. For this project, we will begin the I/O testing on the 18th, prior to the customer arriving. This will allow us time to properly document and test the I/O and prep the system for subsequent wet testing, which may include initial loop tuning and safety checks. The customer will arrive on the following Monday and will then have a full week of testing time. The testing will include both mechanical and operation tests on the skids. The mechanical tests will include a drawing walk down, slope verification, surface finish verification, dimensional check, and spray ball coverage test of the tanks. The operational testing will use the software to test equipment module functionality and alarms. If time permits, we will also begin running recipes, but recipes are not normally tested at an FAT.

What's Next?
The factory acceptance test at the equipment vendor’s shop is scheduled to begin next week.


December 19, 2005

Project Status Summary:
1. Control software development code complete 100%
2. HMI software code complete 100%
3. HMI external application configuration 100%
4. MF beta testing 90%
5. UF beta testing 90%

Customer Acceptance Testing
We completed the majority of the customer acceptance testing this week, which is a significant milestone for the project. The customer testing is always one of the most exciting, yet stressful, events for any automation project. For weeks now, we have been plugging away at the implementation based on evolving requirements and our running dialogue with the customer. Acceptance testing is the point where the end users get to see the system come alive for the first time– and see what all the automation specifications, theories, and hard work ultimately yielded.

Two different client groups are performing the testing. Last week, the process engineers performed functional testing on the system, along with verifying the look and feel of the graphics and recipe editor. This week, the customer’s automation engineer will perform more “under the hood” testing and will also work with us on the plan for integrating the data to their central servers. The process team spent three days at our office to complete their tests and it is expected the automation engineer will spend another two at our office next week. The five-day total is rather typical for a system of this size and scope.

Tests Performed
The first part of any FAT (factory acceptance test) is the initial training of the engineers so they can familiarize themselves with the actual implementation. Although the faceplates for the equipment modules and control modules are rather intuitive, it takes a bit of hands on use to understand all of the capabilities and features consolidated in each. After gaining familiarity with the system, we tested each equipment module for proper functionality. The equipment modules were run both individually and concurrently, mimicking process conditions as much as possible. The testing is enhanced with the use of our simulator, which simulates the analog values in a response representative to the process conditions. The simulation is key for proper testing, for it allows the process to play out as it might normally do in the field, and allow the testers to see actual conditions unfold. For example, there may be significant or subtle differences in processing when a tank drains at its normal rate, versus simply driving the analog level value from 100% to 0%. The simulator is yet another tool employed to help ensure success at the field start-up and prevent code changes in the future. The extensive equipment module testing was performed over a two-day period and uncovered some minor changes to the sequence of operations, which is expected for any acceptance testing. For the scope of the project (78 equipment modules), the amount of sequence changes was remarkably low– less than five!

The graphics and interfaces were also examined during testing and were accepted with minimal changes. This is an area where customer preferences come into play much more than any specification or document. Most end users are very particular about the look and feel of the piping diagrams, the color schemes, the fixed text and dynamic text on their system. Our graphics, dynamos and general layout have been evolving over the years to where now we have a rather fine tuned package. We have developed graphical interfaces that maximize the information with minimal screen footprint. It is also critical not to clutter process screens. The use of features such as tags on/off and displaying of device mode states by exception (only display if not in auto) help keep the screens clean and easy to read.

Documentation
Another key aspect to acceptance testing is the documentation review. During our in-house development testing, each tested module is signed and dated per cGMP (current good manufacturing practices) documentation requirements and compiled into a completed test document. During the visit, it is essential that the customer reviews and accepts the testing documentation that has been generated; it is their record that each line of code has been tested and acceptably documented. Some customers prefer to perform some of the testing themselves and sign off on the documentation, or in this case, the functional testing acted as a spot check on our work.

What's Next?
The customer’s automation engineer will visit our San Francisco office to finish acceptance testing; meanwhile some system components will be shipped to the equipment fabricator for incorporation into the filter skid electrical enclosures. We now begin system clean-up per the acceptance testing punch list before we hit the road for the factory test at the equipment fabricator’s shop.

Blogger Holiday
This blog will not be updated again until the week of January 9, when work resumes on the project following the holiday season.


December 12, 2005

Project Status Summary:
1. Control software development code complete 100%
2. HMI software code complete 100%
3. HMI external application configuration 100%
4. MF beta testing 90%
5. UF beta testing 90%

Historical Data
Current automation systems are capable of generating large volumes of data that can be captured, stored, and analyzed. Even a small system like the one used for this project can generate many gigabytes of data over a few months of operation. Collecting and storing this data has become increasingly inexpensive as hard drive capacity increases and prices decrease.

For this project, our system will tie into an existing plant continuous historian--GE Proficy Historian for continuous data. Event based data such as alarms, system messages, and operator actions data will be collected in a Microsoft SQL Server database. Continuous data will be presented in standard iFix HMI trend displays. Presentation of event-based data is a bit more challenging due to the depth and richness of the data as compared to continuous data. Consider that each value stored in a continuous database includes a tag, parameter, value, and timestamp whereas an alarm event may include tag, parameter, alarm type, alarm priority, plant area, active time, cleared time, acknowledge time, value at active, alarm setpoint when active, time in alarm, min/max/avg. while in alarm, acknowledged by username, and workstation where acknowledged.

Alarm History
iFix includes an alarm ODBC service that logs alarm events to an ODBC compliant database, such as SQL Server. For each event that occurs in the alarming subsystem, a record is created in the database. For example, when an alarm becomes active, a new record is added. When an operator acknowledges the alarm, a separate record is added to the database--unrelated to the active event. The same is true when the alarm clears as another unrelated record is created. We have developed SQL stored procedures that combine all three separate events into a single record in a table of our own design. The result is that alarm data are significantly compressed (three events combined into one) and each record contains all pertinent information about an alarm. On batch systems, the alarm is also related to a batch or phase-simplifying batch analysis and reporting.

Operator Logs
Our custom user interface applications for control modules, equipment modules, and operator prompts include functionality for logging actions performed by an operator. For example, when an operator opens a valve from a control module or equipment module faceplate, an event is logged that includes the timestamp, module tag, module description, parameter, username, workstation, old value, new value, and a user-configurable custom message associated with the parameter. When operators are prompted to perform an action or enter data as part of an automated sequence, the same type of data is logged.

The data can be logged either to a text file or to a message queue using Microsoft Message Queue (MSMQ). We use MSMQ for asynchronous queued data transport across the network. Application programming with MSMQ is extremely simple and provides outstanding functionality for reliably moving data across the network. MSMQ is a service provided with the Windows XP operating system. Message queuing automatically queues up messages for later delivery if a destination computer is not reachable. When the target computer becomes available, the queued messages are automatically forwarded. The target computer for our application is the SQL Server used for event based historical logging. We have developed an application that runs as a Windows service for removing messages from a message queue and moving the information into a SQL Server database. Together, these applications provide for the reliable, asynchronous, and disconnected transfer of data across the network and into a historical relational database without requiring the source application to hold a direct database connection. Logging data with this design is extremely robust and will never interrupt plant operations because a database server is not accessible.

What's Next?
The customer will visit our San Francisco office to review the automation application. A few testing activities will occur during the customer review. As always, there may be a few more customer requested modifications once they have a chance to view and run the application against the simulated process.


December 7, 2005

Project Status Summary:
1. Control software development code complete 100%
2. HMI software code complete 100%
3. HMI external application configuration 98%
4. MF beta testing 75%
5. UF beta testing 75%

Equipment-centric data access
The TagEvents application, as described previously, retrieves tag-specific historical event based data from a SQL Server database. The interface for this application is a context (right-click) menu on each faceplate. The user right-clicks a faceplate to view the context menu and then selects the 'Event History' menu item. Selecting this menu item opens a Web page that accepts a Tag parameter that is used to filter data from the backend SQL Server database.

We consider the HMI an excellent location from which to access equipment-based enterprise data. By selecting a pump, valve, or other instrument, the operator has selected a filter. The design of our faceplates (user-configurable context menu) allows for the retrieval of a wide variety of data as long as it can be accessed by the control system's module tag. Information such as maintenance history, equipment specs, vendor information, or any other equipment-based information can be brought directly to HMI users without code changes to the HMI code.

We are also providing a more full-function data presentation application we call Log Viewer. This application presents SQL Server data in a standard data grid. The left side of the application is a user-configurable tree of nodes where each node represents a filter against the viewed data source. For example, an engineer may setup a set of equipment-based nodes such as MF Skid, UF Skid, and Harvest Tank. When an operator clicks the MF Skid filter in the tree, the data being viewed (alarms or operator actions) are filtered to show only data for devices around the MF Skid. The image below filters all DVC control module types (discrete valves). The filters are entirely user configurable from the user interface, which is also a Web-based application developed in ASP.NET. The application can also be accessed from any desktop Internet Explorer browser.


November 30, 2005

Project Status Summary:
1. Control software development code complete 100%
2. HMI software code complete 100%
3. HMI external application configuration 98%
4. MF beta testing 50%
5. UF beta testing 40%

External Applications
One of the most difficult aspects of an automation project these days happens outside of the traditional core of an automation project– process control and user interface. Customers expect systems to capture operator actions, and provide electronic work instructions, continuous data and alarm historians, shift reports, use reports, batch reports, or batch recipe management. While these functions are not necessary for successfulplant operation, users depend upon system-generated data to verify that product is manufactured under the designed process conditions of time, temperature, pressure, flow, pH, dissolved oxygen, and other parameters. Exceptional conditions must be easily identified in the captured data with the meansto determine cause and effect of such exceptions. In the realm of FDA regulated industry, bad or missing data may result in a batch that cannot be brought to market.

Integrating these applications into the HMI environment for operator access can be very difficult. The fundamental minimum requirements we have developed for integrating an application into the HMI are as follows:

  1. External applications must not permit access to the file system. Applications that provide a File/Open, File/Save, or other file-browsing dialog introduce a potential system security loophole. To prevent this loophole, an application should not provide such features or it should be manageable through the same security mechanism employed by the HMI application.

  2. External applications should run without obscuring the alarm banner. The HMI is designed so the operator may always view active alarm activity. Rapid response to an alarm may be critical to personnel safety, product quality, or equipment safety and should never be obscured by an external application such as a batch report.

  3. External applications that can be hosted in the HMI are more suitable than those that must run as a stand-alone executable. This may be done either through a Web interface or through a programmatic interface such as ActiveX. This type of application can be hosted as a standard graphic display. We have a single graphic used for Web-based applications. The graphic uses Internet Explorer ActiveX control which, when given a URL, can display a Web page without the IE toolbar. As a graphic, the Web page fits nicely between the HMI toolbar and the alarm banner. Our toolbar also has a pair of Previous/Next buttons that function just like the IE Forward/Back buttons. All HMI graphics participate in the page history (Previous/Next) function but stand-alone executables cannot participate in this navigation scheme.

  4. An external application that is not hosted in a graphic must not leave multiple orphaned windows open behind the HMI. Either the application must remain above the HMI at all times or opening the application will use an existing instance that may be hidden instead of creating a new instance and leaving a previous instance orphaned.

To meet customer expectations and essential HMI security requirements, we have found that Web-based technology is the easiest means of integrating external functionality into the HMI. A tangential benefit of using Web technology is that the same applications are available outside of the HMI for supervisors, engineers, and plant management.

We have created our own browser application, called CascadeBrowserX, to meet requirement number 4. This application is a single VB6 form with an Internet Explorer control for viewing Web pages. What makes CascadeBrowserX special is that it has no toolbar, no menu, always stays above the HMI, and an open instance is always reused to eliminate the chance of an orphaned window. The application is an ActiveX EXE application that accepts command line arguments including a URL, window height and width (in percent of screen height and width), horizontal and vertical positioning (in percent of screen height and width), title bar text, and allows or prevents window resizing.

We will be integrating a number of applications into the HMI using a variety of methods. TagEvents is a data retrieval Web application that runs in a windowed CascadeBrowserX. Our LogViewer application is a more functional Web data access application hosted in an HMI graphic display. The BatchEM recipe editor application is a stand-alone external executable that will never open more than a single instance.

What's Next?
Testing continues. I'll provide more information about these three applications and how they interface with system data.


November 22, 2005

Project Status Summary:
1. Control software development code complete 100%
2. HMI software code complete 100%
3. HMI external application configuration 95%
4. MF beta testing 65%
5. UF beta testing 25%

Testing
Testing is the primary project activity currently as we work to get the software fully functional before the customer checkout December 13. The development system is set up in San Francisco. One tester is working locally and another resource is testing software from our Tinley Park, IL, office over a terminal server. As issues arise in the microfiltration software, they are corrected and verified, and fixes are distributed to other modules with similar functionality– including the ultrafiltration (UF) software. This makes UF testing much easier.

External Applications
The project includes our own batch recipe execution system– BatchEM. The BatchEM recipe editor is accessible from the HMI. We also provide a Web-based application called LogViewer that simplifies access to a back end SQL Server database. Our TagEvents Web application is a scaled down LogViewer that can receive a module tag so the page only shows data specific to a particular piece of equipment. With applications that retrieve data from SQL Server, we have to make sure that the desired data gets put into SQL Server in the first place. More information on this technology will be presented in upcoming blog entries.


November 9, 2005

Project Status Summary:
1. MF software design 100%
2. MF software code complete 100%
3. UF software design 100%
4. UF software code complete 100%
5. HMI software code complete 100%

We have reached initial code completion and are well into beta testing. As normally happens, testing begins slowly and accelerates. This acceleration in testing happens for a few reasons including the following:

  • Testers require ramp up time to become proficient in the application and to understand what to expect;

  • Simulation normally needs tweaking to adequately test the application;

  • Number of code defects decrease over time as they are found and fixed; and

  • A code defect found in one module is fixed in all modules that use a similar construct. This eliminates defects in modules yet to be tested.

HMI Displays
The project now includes the following 7 HMI displays:

  • System Overview

  • Harvest Vessel

  • Microfiltration Skid

  • Filtrate Hold Vessel

  • Ultrafiltration (UF) Vessel

  • UF Skid

  • Peer-to-Peer Signals

Each of the processing units and the basic process flow are shown on a single overview display. Pressure, temperature, and volume are displayed for each tank. The current status for each unit is also displayed with a countdown timer showing how much time remains before the current status expires.

The peer-to-peer display provides status information for messages that are passed between PLC processors for control purposes. This display will prove very helpful when troubleshooting the system. The communications between processors—critical to system operation—would otherwise be a mystery that only a programmer could decipher. With this display, savvy operators, supervisors, and engineers can perform basic diagnostics without asking a control engineer to dig into the running code.

The top area of the display includes a navigation toolbar and the bottom area includes a small active alarm grid. Individual graphic displays reside between these two sections. As users navigate between graphic displays, the toolbar and alarm sections do not change. In fact there is actually a single master HMI graphic with a toolbar on top, alarm banner on bottom, and nothing in the middle. Graphic displays are layered above the master graphic. This design allows us to provide a global alarm and navigation system that can be reused between projects.

The toolbar button labeled Graphics is a drop-down that lists the available graphics on this workstation. The list of graphics shown in this dropdown is defined in an external XML text file that is read by the master graphic when the HMI is loaded. To add another graphic display to the menu requires only the addition of an entry in the text file. The XML entry includes the menu text, the file name of the graphic display to open when the menu item is selected, and security level required by users to access the menu item. We use the same configuration file to define menu items for the Trends button and the Apps button. The Apps button allows the user to access external applications from the HMI. For example, the Apps dropdown may include menu items for RSLogix programming software, RSLinx communications software, and SQL Server Enterprise Manager. Applications such as these will normally be limited to users with advanced security privileges.

What's Next?
Next up, testing, testing, and more testing. Also, there are a few other external applications we provide with our system for managing access to data archives, which are being set up this week and should be completed by my next update.


November 2, 2005

Project Status Summary:
1. MF software design 100%
2. MF software code complete 100%
3. UF software design 100%
4. UF software code complete 100%
5. HMI software code complete 75%

We reached code completion for the ultrafiltration (UF) skid equipment modules. The UF graphic displays are all that remain to reach 100% code completion for the project. We have begun internal alpha testing of the harvest tank and micro-filtration skid in parallel with completing UF graphics.

Another interesting change developed this past week. The customer informed the skid builder that a ControlNet card was needed in the UF PLC for a few remote points on a transfer panel. Luckily, the customer copied us on his e-mail to the skid builder. I say luckily because we may not have been notified otherwise. This is a relatively small change to the I/O scan, but we prefer to make the change in our labs than in the field.

Customer Acceptance
We are quickly approaching completion of the development phase of the project. We have scheduled the Customer Acceptance Test for December 13– 16, 2005 in our San Francisco office. We expect to be complete with all development, all internal testing, and all formalized development testing before that date. When the customer arrives, they may choose to execute a formal test or informally review and test the software. We prefer that a customer do their best to explore the software and even try to break it – if they can. It is difficult to overemphasize how much easier it is to change software in our development labs than at a customer site. End users should always review vendor software before it is shipped.

I recommend executing a formal test followed by some type of stress testing. Stress testing could be nearly anything including cycling HMI power unexpectedly to see how to restore the system and to see if the controls are still running in the controller. Do recipes resume where they left off? Do the phases go to a hold state? Try the click test where a user simply clicks on HMI hot spots as quickly as possible to see if the system locks up when a user works quickly– the system should never lock up. Try logging out when a dialog box is displayed. Try entering strings in integer fields and visa versa. Try changing screens rapidly. Try starting then immediately stopping a sequence. Click the same button repeatedly or click and hold a button that would normallybe clicked momentarily. Try anything that seems out of the ordinary. The system should allow this behavior without adverse or unexpected results.

I also recommend that operators see the software before it leaves the lab. We have received many good suggestions from operators related to ease of use and practical use features that we welcome such suggestions–before the software reaches the field.

What's Next?
Project code completion is expected next week. The HMI displays are the final piece of the development puzzle so I'll spend some time reviewing the graphic displays. Initial internal testing– alpha testing – will be performed over the next few weeks. This should expose initial coding errors and potentially a few design issues as we compare a simulated process to the documented design.


October 27, 2005

Project Status Summary:
1. MF (micro-filtration) software design 100%
2. MF software code complete 100%
3. UF (ultra-filtration) software design 100%
4. UF software code complete 95%
5. HMI software code complete 60%

Though we are making good progress in all areas of the project, with three of the phases complete, there still remains more work with the UF equipment modules. The customer has requested changes that require two additional CIP pressure-hold equipment modules for the TFF (tangential flow) skids. These are relatively simple equipment modules.

The HMI is set up and we now have a complete MF system with application software, process simulation, and graphic displays. We have not begun MF skid development testing, but project personnel continue moving forward to reach code completion for the project. The recent change in the design freeze (from October 15 to October 26) will move everything out another two weeks, including testing.

The simulation is relatively simple to write for this sort of process. See the August 9, 2005, entry for a discussion of how we perform the process simulation. What I find interesting about this custom simulation application is the short time required to develop the application. We have resources that are very capable at writing custom applications in VB6, VB.Net, ASP.Net, and XML/XSLT. The applications we write for our own internal purposes differ greatly from the applications we write for our customers. Any application that gets delivered to a customer site is developed against GAMP (good automated manufacturing practice; www/ispe.org ) guidelines. We always assume that customer-delivered applications must be bullet-proof and we work very hard to make them so. However, we don't code to the same standards for applications and tools developed for internal use only. Quick turnaround is normally a high priority for such tools in order to meet project schedule objectives. Nonetheless, these internally used applications still function as desired and help make project development and execution much more efficient than if we did not have such tools. Also, tools developed for internal purposes often become the seeds for customer-delivered applications that receive the full scrutiny and quality control outlined by GAMP. Our simulation application is one such tool.

What's Next?
We are coding the final UF equipment modules including and the UF graphic displays. These will likely extend beyond this week and be completed sometime next week. Of course, there may be a few more change requests from the customer this week. Next week we will likely be one week away from code completion and development testing.


October 19, 2005

Project Status Summary:
1. MF software design--100%
2. MF software code complete--100%
3. UF software design--100%
4. UF software code complete--85%
5. HMI software code complete--40%

HMI Setup
We are still setting up the HMI system that includes two operator interface terminals and one rack mount SCADA server. The server was received from Dell with one drive on two partitions. This makes software development and distribution a bit more difficult to manage and it complicates system management for the customer. At project's end we provide the customer with software we have developed in a single setup program using Wise for Windows Installer. Normally, we would provide a single installation file for the workstations and the server. However, the setup location will default to a single hard drive letter. Because users often just step through setup program files with little care, we would create a separate setup program if the default drive should be D: instead of C: To do this, the drive had to be wiped clean and reformatted as a single partition. We then had to reinstall the operating system and all other software before setting up the SCADA server.

I have to admit, we were a bit surprised to find the multiple partitions. It was common to receive a machine with two partitions under Windows NT 4.0 because it did not support a boot partition greater than 4 GB without using third party software or up to 7.8 GB by physically moving the drive, extending the partition, and moving back to the original machine. Windows 2000 and 2003 no longer have this limitation.

Design Change Requests
We received requested modifications from the customer last week for additional CIP pulsing options for all units. These changes will require an additional three days to incorporate into the design and program. We planned for UF software to be 100% code complete this week but these changes will push out code completion to the following week. Although we identified last Friday as the design freeze date, we learned that the customer has not finished reviewing the MF design document and has yet to begin reviewing the UF design document. We will continue toward software completion adding these changes we received before the design freeze.

Change requests received after the design freeze will be evaluated. If change requests are relatively minor, we will likely incorporate them during the testing phase with no impact to project pricing or schedule. More significant changes will require that we provide the customer with a quotation and schedule impact analysis.

What's Next?
By next week, the developers should begin informal software testing (alpha and beta testing) using HMI graphics and a fully simulated process. HMI development won't be completed yet, but there should be process graphics available to begin testing.


October 4, 2005

The UF (ultrafiltration) skid instrumentation includes flow meters that transmit a pulse signal where each pulse corresponds to a volume of liquid that has passed through the measuring element. We had previously assumed that these signals would not be used by the application, as we have never used this type of signal in biotech applications. That was a bad assumption.

The customer is requesting a totalizer control module that counts pulses. We already have a totalizer module that integrates a rate signal over time. We developed the new totalizer over the past week. It was not a significant effort since much of the existing totalizer code was reused. Also, because our control modules are largely a collection of reusable subroutines, new modules can be quickly created as long as the functionality can be achieved with existing subroutines. The USC module explained in last week's post was one that required new functionality. We are code complete with the new USC and doing some internal testing. We'll test it more extensively when we finally get the HMI set up and we can start running equipment modules from a user interface.

Speaking of the HMI– we finally have all the hardware for the entire HMI system. The SCADA server has also just arrived (maybe a small celebration is in order). Our developers are deep into UF equipment module development, so HMI setup will wait until they are code complete. The UF equipment modules are currently 30% code complete and should be 100% in two more weeks.

We have not received customer approval of the MF (microfiltration) design– something that makes us increasingly uncomfortable over time. We have incorporated the few comments previously received. I suspect we simply need to remind the customer and the approval will be received shortly thereafter.

The UF design was submitted two weeks ago– something I neglected to mention when it happened. We have, as yet, received no comments on this design. The comments on the MF skid design were few and we expect even fewer for the UF design because they are so similar. We are pushing to get comments or approval of the UF design within two weeks to keep the project moving forward.

The design of cross PLC communications was modified this past week. Originally, the system design included a ControlNet network specifically for PLC-to-PLC communications. The information passed between controllers includes signals for module interlocks, sequence permissives, and handshaking between units when material is passed from one unit to another. Because of the obvious safety issues involved with this type of communication, ControlNet was selected over Ethernet because of the deterministic nature of ControlNet communications. However, the customer has notified us that the ControlNet network will be replaced by a dedicated Ethernet network due to system design and installation constraints outside of our control.

What's Next?
In addition to normal project updates, I'll provide a few more details of the nature of a deterministic scan, how this differs in control systems, and how it can be achieved through application implementation if not provided by the system vendor.


September 27, 2005

The microfiltration (MF) system PLC control code– IO, control modules, interlocks, and equipment modules – is “code complete.” If you’re not familiar with this term, it is because we at Cascade Controls have developed our own terminology to describe different levels of completion. Too often when someone says they are 'done,' there is still more work to do. To avoid this type of communication error, we developed and defined our own terms for qualifying degrees of done-ness. These are borrowed terms that we have defined to match our normal project execution methodology.

  • Code Complete– First pass at writing an application where a programmer believes all defined functionality has been included. Nothing is tested at this point.

  • Alpha Tested– The code has been minimally and informally tested by the programmer. The programmer is comfortable enough with the code to have it reviewed by others and rigorously tested.

  • Beta Tested– The code has been tested according to written test procedures by someone other than the programmer. This is normally done not only to test the code, but to test the test protocols.
    Once Beta Testing is complete and the code is known to pass all test procedures, it is made available for formal testing either by the customer or under supervision by the customer. This stage is called by different names based upon the customer's nomenclature including Factory Acceptance Test (FAT), Customer Acceptance Test (CAT), or Development Test (DT).

Code Complete– Sort Of
Now that I've said the MF skid is Code Complete, I must provide qualification. Our USC (Unit Status Controller– see blog entry from August 30, 2005) control module does not quite meet the customer’s requirements, so we are working on the necessary changes. The function required by this USC includes an In-Use state that is automatically set whenever a recipe is running. When the recipe ends, the USCwill automatically shed to another status depending upon the how the recipe ended. If the recipe is aborted, the USC will shed to the state as it existed just prior to the start of the recipe. If the recipe is completed properly, the USC must automatically shed to a state specifically defined for each USC and recipe.

Also, our standard USC puts no constraint upon transition between states. The new USC must support state transition logic that explicitly defines the states allowed from any given state. For HMI purposes, each state will be paired with a state-enable bit that is set by logic so the user interface can know what states are allowed from any given state.

The final feature change allows a user to immediately expire a state timer without changing the timer target.

What's Next?
The design, documentation, and implementation of these modifications will likely require two more weeks of work. Meanwhile, other team members are aggressively coding the ultrafiltration (UF) skid equipment modules. Also, we await approval of the design for the MF skid. One of the key reviewers was recently on vacation so the review period is extended a little longer than initially planned. Approval (or more changes) should be received by the end of this week.


September 19, 2005

This week our work revolves around the plant's operational philosophy and operator access to modules from the HMI. Should an operator be allowed to run an equipment module from the HMI? If the equipment module is in auto mode, can an operator change operational parameters? Should modules be locked into auto mode by recipes or should operators be allowed to put modules in manual mode while a recipe is operating? If an operator changes an equipment module to manual mode, should the recipe automatically transition to a hold state?

These operational questions arise with every project and get different answers from nearly every customer. In fact, we even get different answers for one customer at different facilities. To address the variety of operational philosophies previously encountered, we have tried to design our modules with the flexibility to meet any reasonable operational requirement. These features include:

  1. Module parameters can only be changed at the HMI from module faceplates. The faceplates include functions to secure parameter access and to log changes. Limiting parameter changes to a single user interface component ensures consistent security control and data historization

  2. Faceplate parameter changes are logged with the module tag, parameter name, module description, module type, change message, timestamp in local time and UTC, workstation name, user name, user's security group, old value, and new value. Logged actions are written to a local text file or to a SQL database via Microsoft Message Queue

  3. All modules have a ModeLock parameter. When a module is mode locked, a user is not allowed to change a module's mode. Modules may be mode locked by a recipe to prevent operator interaction with many functions. Recipes that set the mode lock parameter should reset the parameter when finished

  4. All parameters are individually secured. Each parameter is assigned a security level. A user must have an access level that is greater than or equal to the security level of the parameter. Parameter security levels are maintained in an XML configuration file

On this project we have encountered a new twist that is not easily covered by the features described above. The customer has asked that recipes auto lock all equipment modules and when in auto mode, operator parameters should not be changeable by operators. Each equipment module is designed with a set of parameters designed to be used by operators and, therefore, called 'Operator Parameters'. Since operators at the HMI may operate an equipment module for non-recipe use, these parameters provide the flexibility to take advantage of well-tested sequences to meet abnormal plant operational needs. These parameters are always available from an equipment module faceplate. This approach will significantly limit the capability of an operator to intervene during recipe execution short of aborting the entire recipe—something all other customers have found too constraining.

These issues of operator access have long lasting and wide reaching impact on the daily operation of the plant, plant efficiency, and the frequency and magnitude of operator errors. Despite the ardent supporters of one philosophy or another, the best approach is one that is considered carefully by personnel from operations, process engineering, maintenance, automation and any other discipline involved with daily plant operation.

Details of the UF design will be discussed next week.


September 13, 2005

There are a few new developments for the project this week. First, we finally received most of the HMI hardware but still await the main server that will function as the iFix SCADA node. This server delivers data to the client HMI displays. We'll wait until we can get the server before developing the HMI applications.

We also received an updated I/O list from the skid vendor. There are a number of inconsistencies between the I/O list and the P&IDs. Maintaining consistency between drawings and I/O lists is frequently a difficult task when managed as separate data sources.There are software products designed to prevent this sort of problem, although they can be a bit pricey and often have a significant learning curve. On larger projects, keeping these two data sources synchronized can be very time consuming. We've developed our own internal tool to populate AutoCAD drawing objects with textual information from an external ODBC database. It does not have the extensive list of functionality found in commercial products, but it meets our needs and saves significant time keeping the two data sources synchronized.

Initial customer review of the MF design specification requires that the CIP drain sequence operation manipulate a couple more discrete valves. The customer has also requested a pre-CIP Pressure Hold Test sequence that will generally be used before cleaning operations. This will require an additional 4 equipment modules for the project and modifications to the design documents. The UF design document scheduled for submittal at the end of last week is rescheduled for early next week, after we update the design to include these modifications.

Meanwhile, equipment module code development continues with additional resources added to the project. We currently have one engineer completing the software design while three engineers are developing code for equipment modules whose design is already finished. One new resource will be coding MF sequences in our Tinley Park, IL office for Safe State, Membrane Drain, and Integrity Test equipment modules. Another resource will fly from Tinley Park to our San Francisco office to work closely with the design engineer to code MF CIP equipment modules. The CIP sequences are likely to be the largest sequences but will be very similar structurally for both the MF and UF skids. The engineer responsible for the design will begin coding equipment modules between duties answering customer questions and modifications to the design based upon customer feedback.

Next week, I'll discuss the plant's operational philosophy and operator access to modules from the HMI.


September 7, 2005

As I’ve been saying for the past few weeks, no HMI hardware has arrived this week either, so we continue to develop equipment module logic in parallel with the software design.

The automation design on this project is divided into two main documents grouped by major process system– Microfiltration (MF) and Ultrafiltration (UF). The UF system design specification will be submitted later this week. The MF system design document is roughly 150 pages and includes design details for the follow control modules.

Equipment System

Control Modules


Harvest Vessel 22

22

Harvest Vessel Temperature Control

10

MF Skid Feed

11

MF Skid Retentate

6

Shared Equipment

17

CIP Distribution

4

Misc. Support

12

PLC Peer to Peer

47

The last two control module groups may require a little explanation. The Misc. Support modules include basic application support functionality. This includes 5 DI (discrete input) modules to monitor communications with other PLC processors on the network. If communication to another PLC is lost, the local DI module will generate an alarm. The Unit Status modules described in last week's post are included in this group as well as modules that manage the BatchEM recipe management interface similar to what is commonly known as PLI (Phase Logic Interface) with off-the-shelf recipe management products.

The PLC Peer-to-Peer modules provide monitor and control functionality across PLC processors. For example, when sanitizing the harvest tank, the piping is opened back to a fermentor, which is controlled on another PLC and the code is being developed by another integrator. Therefore, a module is required to read the transfer line trap temperature so the harvest sanitization sequence can verify achieving and maintenance of the proper temperature during the sequence. Other modules provide handshaking across PLC processors for the transfer of a batch across control hardware boundaries. Still other modules provide access to arbitrated shared equipment across PLC processors.

Experienced PLC programmers are familiar with writing code for communications across PLC processors. DCS systems have historically reduced or eliminated the need for this type of coding by automatically resolving tag references globally across the entire system including controllers. Systems that provide automatic cross controller communications simply by referencing a tag/attribute value significantly reduce initial engineering, the volume of custom code, and long-term code maintenance. The 47 modules for PLC peer-to-peer communications for the MF system are not necessary in other systems.

The MF design document includes detailed sequencing design for the following 25 equipment modules.

  1. Harvest Vessel Agitator Control

  2. Harvest Vessel CIP Airblow/Drain Sequence

  3. Harvest Vessel Drain Sequence

  4. Harvest Vessel CIP Flush Sequence

  5. Harvest Vessel Inactivation Sequence

  6. Harvest Vessel Level Control

  7. Harvest Vessel Post-SIP Pressure Monitor

  8. Harvest Vessel Pre-SIP Pressure Hold Test Sequence

  9. Harvest Vessel Pressure Control

  10. Harvest Vessel Post-SIP Pressure Hold Test Sequence

  11. Harvest Vessel Purified Water Drop Control

  12. Harvest Vessel Receive Sequence

  13. Harvest Vessel Safe State Sequence

  14. Harvest Vessel SIP Sequence

  15. Harvest Vessel Temperature Control

  16. MF Skid CIP Airblow/Drain Sequence

  17. MF Skid Flush Sequence

  18. MF Skid Feed Flow Control

  19. MF Skid Filtrate Flow/Pressure Control

  20. MF Skid CIP Flush Sequence

  21. MF Skid Integrity Test Sequence

  22. MF Skid Membrane Drain Control

  23. MF Skid Concentration/Diafiltration Processing Sequence

  24. MF Skid Retentate Pressure Control

  25. MF Skid Safe State Sequence

All of these equipment modules are available for individual operation from the HMI. They may also be operated and sequenced through our BatchEM recipe management application. Providing access to individual phases outside of a recipe is functionality we have found to be missing in most batch applications. This design provides the maximum benefits of recipe management without eliminating the flexibility to operate the plant under exceptional conditions, which are frequently not so exceptional after all.

The UF design will be completed next week. Two weeks are scheduled for the customer to review and provide feedback on the MF design.


August 30, 2005

No HMI hardware has arrived this week either, so development of the user interface must remain on hold.

However, work continues on equipment module design in parallel with equipment module development. We expect to complete the first pass design on all equipment modules by next week. The design will then be given to the customer for review. It is normal for the customer to request a few changes with their review, but we expect these changes to be functionally minimal. Again, we have been developing equipment module code without an approved design document to stay aligned with the project schedule.

Speaking of schedule, we have received a modification to the project schedule. Originally, we planned to complete our software by the end of September for factory acceptance testing (FAT) at the skid vendor's site in mid-October. The newly scheduled FAT date is in mid-December. However, we plan to keep our schedule unchanged. We don't expect to need the extra time for developing the application software.

There have also been recent developments around the design for equipment status software. Biotech applications require exacting standards for cleanliness. Equipment internals must be cleaned and sterilized after each batch. A new batch cannot be started unless there is a high degree of confidence that the process is free of foreign material and microorganisms. Equipment that is cleaned is often marked 'CLEAN' with a physical sign hung prominently for all to see. Manufacturing instructions often instruct the operator to verify a piece of equipment is marked CLEAN or STERILE or some other such status before proceeding with the instructions. If not properly marked, an operator will clean or steam the equipment before further processing.

Many in the industry now look to the automation system to track the processing status for a piece of equipment and to provide automatic time-based expiration of various states. This functionality has been added to the project scope. We implement this functionality through a control module called a unit status controller (USC) that provides a logical status for any piece of equipment that may have a unique and independent status. The end user has defined status for 4 distinct states:

  • In Use—after start of a production recipe until recipe is finished;

  • Post Use—after a production recipe has finished until cleaning recipes have finished successfully;

  • Post CIP Hold—after CIP or SIP recipe is finished successfully or until a timer expires. Upon timer expiration, the status is switched back to Post Use; and

  • Post SIP Hold—after SIP recipe is finished successfully and the previous status was Post CIP Hold. This status also has a timer that, when expired, automatically drops the status back to Post CIP Hold unless that timer has expired and the status is then dropped to Post Use.

This is a simplified version of the state logic, but it explains the primary functionality. The code will be developed as a state machine within the control module. States may be set by sequences—equipment modules or recipes—or by an operator from the HMI with sufficient security access. Expiration timers and the transition logic will be contained within the USC control module logic.

Until the HMI hardware arrives, we’ll continue to work on equipment modules and develop a first draft of the equipment module design document for submittal to the client.


August 30, 2005

No HMI hardware has arrived this week either, so development of the user interface must remain on hold.

However, work continues on equipment module design in parallel with equipment module development. We expect to complete the first pass design on all equipment modules by next week. The design will then be given to the customer for review. It is normal for the customer to request a few changes with their review, but we expect these changes to be functionally minimal. Again, we have been developing equipment module code without an approved design document to stay aligned with the project schedule.

Speaking of schedule, we have received a modification to the project schedule. Originally, we planned to complete our software by the end of September for factory acceptance testing (FAT) at the skid vendor's site in mid-October. The newly scheduled FAT date is in mid-December. However, we plan to keep our schedule unchanged. We don't expect to need the extra time for developing the application software.

There have also been recent developments around the design for equipment status software. Biotech applications require exacting standards for cleanliness. Equipment internals must be cleaned and sterilized after each batch. A new batch cannot be started unless there is a high degree of confidence that the process is free of foreign material and microorganisms. Equipment that is cleaned is often marked 'CLEAN' with a physical sign hung prominently for all to see. Manufacturing instructions often instruct the operator to verify a piece of equipment is marked CLEAN or STERILE or some other such status before proceeding with the instructions. If not properly marked, an operator will clean or steam the equipment before further processing.

Many in the industry now look to the automation system to track the processing status for a piece of equipment and to provide automatic time-based expiration of various states. This functionality has been added to the project scope. We implement this functionality through a control module called a unit status controller (USC) that provides a logical status for any piece of equipment that may have a unique and independent status. The end user has defined status for 4 distinct states:

  • In Use—after start of a production recipe until recipe is finished;

  • Post Use—after a production recipe has finished until cleaning recipes have finished successfully;

  • Post CIP Hold—after CIP or SIP recipe is finished successfully or until a timer expires. Upon timer expiration, the status is switched back to Post Use; and

  • Post SIP Hold—after SIP recipe is finished successfully and the previous status was Post CIP Hold. This status also has a timer that, when expired, automatically drops the status back to Post CIP Hold unless that timer has expired and the status is then dropped to Post Use.

This is a simplified version of the state logic, but it explains the primary functionality. The code will be developed as a state machine within the control module. States may be set by sequences—equipment modules or recipes—or by an operator from the HMI with sufficient security access. Expiration timers and the transition logic will be contained within the USC control module logic.

Until the HMI hardware arrives, we’ll continue to work on equipment modules and develop a first draft of the equipment module design document for submittal to the client.


August 25, 2005

The end user has ordered the HMI hardware, but the equipment has not yet arrived so graphic development is again delayed. As described in last week's entry, static displays were developed without the PLC controller to develop against. Once the HMI hardware is available, the HMI software will be set up to communicate with the PLC and the graphic displays will be checked out against the running logic.

Again, to keep the project moving forward, we continue working on areas that would normally occur after the basic graphic development is complete. We are currently designing and developing automatic sequences for the equipment modules. Having a working simulated process with a graphical HMI interface is extremely helpful when developing equipment modules because we can run the sequence from the HMI and observe the results on graphic displays. Since equipment modules simply manipulate control modules or other equipment modules, observing a sequence from a process graphic is infinitely easier than observing the sequence from within the programming code.

The real meat of the control application is in the equipment modules and the current design includes 51 equipment modules (roughly twice what was expected at the start of the project). Each equipment module is designed as a state machine similar to the S88 phase logic state diagram. What sets our equipment module state diagram apart from the S88 phase state diagram is that any number of states may be defined for one equipment module and state transition logic is also uniquely defined. This may seem like a very loose structure, yet it still provides a consistent, standard structure despite the number of states and state names.

We have developed a single standard equipment module HMI faceplate interface that works with all equipment modules. When the faceplate loads, it interrogates the referenced equipment module and dynamically adds state command buttons— dynamically discovering the appropriate state label to apply to each button. The equipment module includes state enabled flags maintained by state transition logic so that the faceplate may enable/disable state buttons as determined by equipment module internal logic.

Currently we have completed the design for about half of the 51 equipment modules. Equipment module code has been developed for the following 7 harvest tank equipment modules:
1. Agitation Control
2. Pressure Control
3. Temperature Control
4. Vessel Drain
5. Purified Water Drop Control
6. Tank Level Control
7. Pre-SIP Pressure Hold Test

Designing, developing, and testing equipment modules will consume the majority of the project between now and the scheduled date for code completion at the end of September.

What's Next?
Next week and through the month of September we will be focused on equipment module development. Additional HMI work will start when the HMI equipment becomes available.


August 16, 2005

We expected to be working on the HMI this past week. Unfortunately, the workstations on which we were to develop have not arrived because the customer and skid builder are still working out the details of the hardware that will be used as workstations. While our San Francisco office is doing most of the development, our home office in Tinley Park, IL, has much more company owned hardware on which we can develop. So to keep the project moving forward, Tinley Park resources have developed static graphic displays. Static displays include the basic process equipment layout, piping connections, and device dynamos that are configured for specific device tags but have not been verified because they are not developed against a PLC controller with the proper control module software. We currently plan to provide seven graphic displays:

1. Recovery overview;
2. Harvest vessel;
3. Micro filtration skid;
4. Ultra filtration vessel;
5. Ultra filtration skid;
6. Filtrate hold vessel; and
7. Control system network status.

Using graphic building blocks from our internally developed iFix graphics library (tanks, pipes, and iFix dynamos) we have created individual graphic files for all but the control system network status display.

The diamond shaped satellites near the various devices are part of a single grouped object that references the control module that interfaces with the device. For example, a valve includes the valve itself, a mode satellite (auto or manual mode), an interlock satellite, and a batch hold status satellite. To add a valve to the graphic, a user drags a valve from the library and drops it on the graphic at the desired location. After dropping the valve on the graphic, a dialog pops up where the user may type in the appropriate control module tag. After clicking OK on the dynamo form, all of the dynamic portions of the valve are linked via OPC to the proper parameters in the PLC.


The iFix dynamo feature has helped to reduce graphic development time at least by 50% when compared to editing each dynamic property of each graphic device.

Currently, the PLC code is loaded into a development ControlLogix PLC in San Francisco while the graphic files are created on workstations in Tinley Park, IL. The graphics in Tinley Park are not connected to the PLC controller in San Francisco but we may see if we can do this to allow us to debug the graphics until the workstations arrive.

As we move forward in the project, receiving the workstations will be essential to moving forward. Developing complex sequences and recipes is extremely difficult and slow without the aid of a fully functional HMI system. As for exactly what happens next, that’s hard to say without the workstations. Check back with us next week to see how things turn out.


August 9, 2005

Over the past week we have been involved with process simulations to allow developers to run sequences, test alarm conditions, execute recipes, and generate operational data that can be used for development of data analysis applications for this project. Historically, we have developed simulations inside the control system. Recently, however, we have developed a simple simulation application in VB.Net that uses OPC to manipulate module parameters based upon a simulation calculation defined by the user. The application reads user-defined OPC inputs, performs a simulation calculation, and writes to OPC outputs based upon the calculation results. Calculations are written in VB Script and the entire simulation is saved to disk as an XML file. However the simulation is written, the purpose is not to verify proper I/O connectivity, but verify software functionality of control modules, interlocks, graphics, equipment sequences, phases, and recipes.

We always parameterize simulations that allow us to speed up or slow down the response. For this process, a typical batch may take two days—much too long for effective testing. We'll tune our simulation so a batch can be filtered in a few minutes or as long as a couple of hours.

Liquid flows are the easiest of analog processes to simulate. A flow controller PID output can be used to represent the flow transmitter signal after proper range scaling between the two values. Personally, I like to add a simple non-linear relationship to make the simulation a bit more life-like. The value is calculated as the SIN of the PID output rescaled between 0 and PI/2. The result is a value between 0 and 1 that follows a non-linear path.

Simulate a discrete valve by adding a time delay after a change of state from the valve control module output. The time delay should be easily configurable to permit simulating valve failures or slow acting valves. It may be tempting to have a single time constant for all valves, but eventually you'll want different values for different valves, therefore I recommend on having a unique time delay for each individual valve.

Temperature simulations are also relatively simple to simulate by calculating a temperature increment for each simulation scan. The increment is a number to add or subtract from the current temperature to create the new temperature value. On each scan, the temperature is increased or decreased by an increment. The increment should generally be the combination of multiple increments, each representing a process condition that influences temperature. For example, nearly every temperature simulation we include an ambient effect that will eventually bring the temperature to ambient conditions if no other driving force is active. The increment can be calculated as follows:

Da = Ka x (Tc– Ta)

Where:

  • Da is an incremental change to the process temperature caused by ambient conditions;

  • Ka is the relative ambient driving force (a parameter to be tuned by trial and error to get a realistic ambient effect relative to other temperature effects);

  • Tc is the current process temperature; and

  • Ta is the ambient temperature which can be a static value of say 20 °C.

The same calculation will be used for any other process condition that affects the process temperature such as jacket heating/cooling with steam or glycol. Each process effect will have a unique Ka and Ta that can be tuned to impart the desired response.

Next week we’ll address HMI aspects of this project.


August 2, 2005

A good interlock design is one of the most crucial parts of any project as it provides protection of a company's most valuable assets--personnel, product and plant equipment. There are a few rules that we follow to make the interlock implementation easy to design, implement, and troubleshoot.

Rule 1 – Interlocks are implemented at the control module layer between control modules. Since interlocking logic is designed to prevent unsafe equipment operation, it makes sense to implement them on the software that performs equipment control.

Rule 2 – Device I/O points are monitored and manipulated exclusively through control modules. Equipment modules, phases, or other logic will never write directly to or read directly from an I/O point. Other code modules will interface with process equipment through the interfaces provided by control modules. For example, if a phase is to open a valve, it does so by commanding the appropriate valve control module to open-- if the valve is currently used by an operator or interlocked close, the phase is unable to open the valve. Interlock integrity is essential to safe plant operations.

Rule 3 – Interlocking logic is based upon control module states, not the actual I/O point. This is important for providing plant operations and maintenance personnel with the tools necessary to operate the plant safely when process instrumentation is not functioning properly. For example, a failed valvelimit switch may falsely indicate a valve to be closed when it is actually open. This state may interlock other valves or motors. By providing a switch failure override within a valve control module, users can override the failed switch and continue normal operations if interlocks are based upon a control module logical state determined by limit switch position or a user override. This design distributes override functionality inside each control module (at the source) instead of adding it to the interlock logic (often after the fact).

Interlock logic is notorious for confusing operators because it is designed to prevent or permit equipment operation; yet this logic is seldom displayed to operators. To make interlocks easier to understand, we have modularized interlocks and developed a faceplate interface so they are accessible by operators. Each control module type includes an input permissive parameter whose value must be True to permit operation of the control module. An interlock control module performs the interlocking logic and generates a single bit that is passed to the device control module's permissive input. For example, an interlock control module has been created for feed pump PMP-012-02. The interlock control module tag is PMP-012-02-I. The “-I” suffix identifies the module as containing interlock logic for control module PMP-012-02--the feed pump variable speed motor controller.


Anyone can find the interlocking logic for a device if all they need to look for is the device tag followed by “-I”. But how does this help the operator? The key to helping the operator is an easily accessible display (the control module faceplate) and a way to translate the logic into descriptive text. For an operator, it is not important to know the details of the complete logic but the conditions that are preventing equipment operation. Conceptually any interlock looks something like this diagram.


Since any condition could prevent equipment operation, we present each condition to the operator with a text description that is backlit either in green (this condition is OK) or in red (this one prevents equipment operation).

When a device is interlocked, an icon is displayed next to the device on the graphic.
The faceplate opens when an operator clicks on this icon, thereby making it obvious what prevents equipment operation.
Next week we’ll cover simulating the process.

 

 

 



July 26, 2005

The foundation of our automation design is device control managed by control modules. The following device control modules are to be delivered with this project. Not listed are other control modules we will implement for simple functions that do not interface directly with I/O devices.

Acronym

Description

AI

Analog Indicator

DB

Deadband Controller

DC

Discrete Output Controller

DI

Discrete Indicator

DIJ

Discrete Jumper Indicator

DMC

Discrete Motor Controller

DVC

Discrete Valve Controller

ILK

Interlock Indicator

MAN

Manual Loader

PID

Proportional, Integral, Derivative Controller

PIDB

Batch PID Controller

TOT

Totalizer

VMC

Variable Speed Motor Controller

One control module not used for direct device control is the Acquire/Release Controller (ARC) that is used for the arbitration of shared equipment. Batch automation systems typically provide arbitration functionality for shared equipment but we have found that arbitration is often necessary when operating a facility without batch recipe management. Our solution is to push device arbitration into the controller assigning an ARC module to every shared resource. For this project, a valve manifold between the harvest tank and a filtration skid will have an ARC module. Both the harvest tank and the filtration skid must manipulate the valve manifold independently of each other. Sequence code for the harvest tank that manipulates the valve manifold must first request and acquire the manifold. If the manifold is currently acquired by another resource (for example, the filter skid), the ARC module ignores the harvest tank requests. Once the ARC is released by the filter skid, a request by the harvest tank will change the ownership to the harvest tank preventing acquisition by other users.

 

 

Another benefit of this type of arbitration is that it allows for the creation of owners (such as an operator) outside the batching system. We provide an ARC faceplate accessible to the operator by clicking on a graphic object that displays the current owner. The faceplate displays the current resource owner, and a drop down for manually requesting the resource. One of the options in the drop down is 'Operator.' If an operator acquires a resource—for maintenance or other manual intervention—automatic sequences will not be allowed to manipulate the resource until released by the user. The “Request” and “Release” faceplate controls are secured such that operators may request a resource but only a supervisor may release a resource
ARC operator faceplate We have implemented each control module type so that the code is never duplicated for any control module type or for any subset of a control module. For example, this project includes 106 valves to be controlled by the DVC control module. We have only one copy of the DVC code that is used by all 106 instances. The data for each instance is moved into working registers used by the DVC code, which moves the results back to the instance registers. This design significantly reduces the amount of code we must write, maintain and validate. In general, anytime we find ourselves writing duplicate code we will turn that code into some type of subroutine that can be called whenever needed.

Multiple modules use some of the same functions. For example, AI, DB, PID, PIDB, TOT, and VMC use analog input processing (rescaling, low cutoff, alarming, low pass filter, etc.). Therefore, the code for analog input processing is a subroutine called by each of these control modules, which ensures that all have the same functionality.
The next post will discuss the interlocking design strategy for the project.


July 20, 2005

The design process for this project follows a bottom up strategy from field devices to recipes in this order.
1. I/O
2. Control modules and interlocks
3. Equipment modules
4. Phases
5. Recipes

The skid vendor has provided P&IDs and electrical diagrams. From these drawings, we have entered I/O information into our project database. One record in this database includes tag, description, P&ID drawing, field panel, I/O type, controller node, I/O slot, card channel, alarm area, and a few other fields that provide wiring details.

Managing the I/O information is an area where we must interface with other project disciplines. Because we keep our own database, we must be informed of changes to plant equipment design with impact to the instrumentation. This requires that we always be vigilant and monitor project communications or updates to documentation. For a small project such as this, it is not difficult to do. For a recent large project, we would receive I/O changes on a monthly basis even while software was being tested and validated.

On our project, we have a skid that feeds a tank that feeds another skid. Different PLC controllers control the two skids. The question then is: To which of the two controllers should the tank I/O be wired? The correct method is to wire the I/O to the PLC that contains the device control for that I/O. This may require that the tank I/O be split between the two PLCs, since the tank inputs are controlled by the upstream PLC and the outputs are controlled by the downstream PLC. This is important for a few reasons. First, if the control for a valve is in one controller and the I/O in another, the device requires a network connection for simple device operation. Should the network connection break between the two panels, the device can no longer be operated. A single controller should always be designed to manage basic device level control without requiring anything outside the controller and its I/O subsystem. Another reason is the value of modularity in the equipment. Wiring designs based upon geographic location often result in a system that is very difficult to maintain because a single field panel may include wiring for a variety of equipment. Although it may be more expensive to wire a point where it functionally belongs rather than wiring it to the nearest panel, this extra cost is more than offset by the cost of managing a bad I/O mapping for the life of the plant.

Separating the I/O cards based upon equipment boundaries will also simplify copying the wiring design. For example, the first 4 I/O cards (AI, AO, DI, DO) may be set aside for the skid I/O and the next 4 cards set aside for the tank I/O. This is not the most efficient use of the PLC hardware, but is much easier to manage and maintain over the long run and much easier to copy should a new tank or new skid be added to the plant.

The next post will cover the design of control modules.


July 14, 2005

The project has begun. Sales presentations are ancient history, purchase orders issued, technical information exchanged, kickoff meetings are over, and schedules are set. In short, the honeymoon is over. What remains are the details of bringing an automation system to life.

The entries for this blog will be a sort of journal that chronicles the activities of the project as it happens. But more than a sterile rehash of daily or weekly events, it will include opinion, explanation, and pontification about the project and its progress.

The content of this blog will follow these general guidelines:
1) The customer will not be mentioned by name, although they are aware of the blog.

2) The blog is focused primarily upon the development of automation software. Hardware, instrumentation, installation, or other aspects of the project concerning the physical equipment will not be significant topics.

The Project
Project deliverables will include software design documents, formal test procedure documents, and the actual code. Although each of these is delivered individually throughout the course of the project, all will be assembled at the end of the project and delivered together in a Turn-Over Package (TOP).

Design
The first deliverable—the design document—requires that we first design the automation application. The design process includes reviewing customer requirements, customer conversations, and internal discussion to determine the best means to achieve the desired results.

As the design effort proceeds, we will capture the design concepts in a database application we have developed internally using a Microsoft SQL Server back end database with a Microsoft Access front end for entering the information. Not only will this database be used for developing the design, but it also generates documentation and code used in both the PLC and HMI software. This database will be mentioned often in future postings because it is an essential tool for implementing the project.

For the record, we create a separate back end SQL database for each new project we undertake.

What's Next?
Next week’s post will deal with the beginnings of the detailed design.





No comments
The Engineers' Choice Awards highlight some of the best new control, instrumentation and automation products as chosen by...
The System Integrator Giants program lists the top 100 system integrators among companies listed in CFE Media's Global System Integrator Database.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
This eGuide illustrates solutions, applications and benefits of machine vision systems.
Learn how to increase device reliability in harsh environments and decrease unplanned system downtime.
This eGuide contains a series of articles and videos that considers theoretical and practical; immediate needs and a look into the future.
Additive manufacturing benefits; HMI and sensor tips; System integrator advice; Innovations from the industry
Robotic safety, collaboration, standards; DCS migration tips; IT/OT convergence; 2017 Control Engineering Salary and Career Survey
Integrated mobility; Artificial intelligence; Predictive motion control; Sensors and control system inputs; Asset Management; Cybersecurity
Featured articles highlight technologies that enable the Industrial Internet of Things, IIoT-related products and strategies to get data more easily to the user.
This article collection contains several articles on how automation and controls are helping human-machine interface (HMI) hardware and software advance.
This digital report will explore several aspects of how IIoT will transform manufacturing in the coming years.

Find and connect with the most suitable service provider for your unique application. Start searching the Global System Integrator Database Now!

Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Mobility as the means to offshore innovation; Preventing another Deepwater Horizon; ROVs as subsea robots; SCADA and the radio spectrum
Future of oil and gas projects; Reservoir models; The importance of SCADA to oil and gas
Automation Engineer; Wood Group
System Integrator; Cross Integrated Systems Group
Jose S. Vasquez, Jr.
Fire & Life Safety Engineer; Technip USA Inc.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me