Project Success Requires Vision and Attention to Detail

Many companies talk about needing flexibility, agility, repeatability, and tight integration in highly automated processes, but far fewer actually achieve such lofty goals. One exception is Genentech's (South San Francisco, Calif.) new pharmaceutical manufacturing campus located in Vacaville, California.

By Dave Harrold, CONTROL ENGINEERING May 1, 2000
  • Process and advanced control

  • Batch control

  • Control theory

  • Hierarchical control

  • Information systems

  • Productivity, management

Electronic signatures gain acceptance

Many companies talk about needing flexibility, agility, repeatability, and tight integration in highly automated processes, but far fewer actually achieve such lofty goals. One exception is Genentech’s (South San Francisco, Calif.) new pharmaceutical manufacturing campus located in Vacaville, California.

Early in 1994, Genentech began planning for additional biotechnology production capacity to accommodate several new products including Herceptin (Trastuzumab), a product for treating metastic breast cancer. (Herceptin received U.S. Food and Drug Administration (FDA) approval in third-quarter 1998.)

Biotechnology pharmaceutical products begin in a cell culture laboratory where engineered cells are taken from a cell bank and grown in successively larger fermenters until enough cells are available to seed a production-sized fermenter (12,000 liter, 3,170 gal.). Once production begins, the cells are carefully ‘fed’ nutrients and buffers to produce the required protein. Feeding cells too quickly produces unwanted proteins. Feeding too slowly ‘starves’ the cells.

After a fermentation process is completed, the product protein is harvested, cells are separated from product protein and recovery and purification is accomplished using large-scale filtration and chromatography.

Being able to produce a variety of biotechnology products in the same facility requires use of different nutrients, buffers, fermenters, filters, chromatographs, holding tanks, etc. Flexibility to transfer product from anywhere to anywhere is achieved using a series of instrumented transfer panels.

No less important than production automation and control is ensuring the cleanliness of equipment and transfer piping, thus CIP (clean in place) and SIP (steam in place) equipment is an integral part of the overall automation and control design.

‘It has been a real benefit to have the cleaning and preparation of equipment and production of intermediates integrally controlled by the automation system,’ says Scott Zimmerman, manager of purification operation manufacturing.

In short, Genentech’s Vacaville facility is a multi-product, multi-stream process, and that equates to complex pharmaceutical product batching operations.

Adding to the complexity, pharmaceutical producers are governed by FDA regulations designed to verify that a specific process and its systems will consistently produce a product meeting predetermined specifications and quality attributes. This ‘validation,’ required at the highest levels for pharmaceutical and food and beverage producers, is really the goal of every industry.

Understanding problems

Often project goals appear obvious, but ensuring project success requires understanding and documenting the business problems each stakeholder expects the project to solve (see CE , Aug. ’99, p. 34).

Genentech’s design team, led by associate director of automation, Hans Koening-Bastiaan, documented stakeholder expectations to include:

  • Build a flexible, multi-product cell culture batch manufacturing plant;

  • Automate in such a way to facilitate throughput and easily introduce new products;

  • Verify automation systems provide repeatable operational results; and

  • Integrate systems to collect process data necessary to minimize production cycle times.

‘We knew we needed to have a very modular design,’ says Mr. Koening-Bastiaan, ‘so early in the project we contracted Lynn Craig, president of Manufacturing Automation Associates (Medford, N.J.) and chairman of the ISA S88 committee, and Robert Brokamp, manager of automation engineering with Jacobs Engineering’s (Cincinnati, O.), to become an integral part of our design team and help ensure stakeholder goals could and would be delivered.’

Genentech’s products are extremely complex to produce. For example, one batch of final product requires as many as 80 intermediate products, not including the equipment cleaning and sterilization control logic. To meet stakeholder expectations, Genentech’s design required identification, segregation, and creation of:

  • Equipment recipes;

  • Intermediate product recipes; and

  • Final product recipes.

In any well-equipped ‘kitchen,’ available equipment to create a recipe includes many pots, pans, ovens, stovetops, mixers, etc. In Genentech’s biotech kitchen each piece of physical equipment has a corresponding equipment recipe (software model) that contains the equipment’s operational logic and status, such as how long a piece of equipment can go unused before it needs to be re-cleaned. Ensuring each piece of equipment remains a stand-alone entity allows equipment to be used in any sequence to accommodate existing products, as well as any future biotechnology product Genentech’s ‘chefs’ might invent.

As is the case in preparing most recipes, intermediate product recipes are frequently necessary. For example, Genentech final product recipes require intermediate recipes to create media, buffers, CIP wash solutions, etc. Holding to the design philosophy of three recipe types, Genentech is able to support creation of new intermediate recipes without impacting existing intermediate recipes.

When each intermediate product is created, production data-i.e., lot number, amounts and source of raw materials, mixing times, etc.-are collected and retained in the lot historian database for as long as the intermediate product exists. When a final product recipe uses a portion of the intermediate product, a copy of the intermediate product data is attached to the final product recipe to fulfill material genealogy tracking regulations.

200 steps

With over 200 equipment and intermediate recipes involved in producing a single final product recipe, maintaining product control is essential.

‘A recipe controlled process allows us to achieve much higher process consistency from batch to batch. There are no surprises,’ explains Mr. Zimmerman.

Information collected by the final product recipe during its journey includes:

  • Equipment the final product passes through;

  • How long the product resides in each piece of equipment;

  • What equipment recipes were executed;

  • Quantities and sequences of intermediate product additions along with a copy of the intermediate products data;

  • Operator observation and interaction comments; and

  • Electronic signatures used to verify actions were completed per approved procedures.

Proof for FDA

This kind of information helps achieve two goals. First, it ensures sufficient proof the final product meets predetermined specifications and quality attributes required by FDA regulations. Second, it ensures stakeholder goals of providing repeatable operational results and minimizing production cycle times are consistently achieved.

Genentech’s design team had repeatedly witnessed the importance of providing the implementation team with sufficient understanding and documentation to ensure that how the solution was implemented did not compromise designed-in flexibility.

That meant documenting everything, a time-consuming task but necessary for two reasons. First, creating a detailed design ensured designers thoroughly identified and understood every piece of equipment and how it worked. Second, the detailed design documents would become the testing and validation qualification specifications for the control system implementation. The project team recognized it would be much less expensive, embarrassing, and stressful to find and correct design errors on paper than to wait until the hardware and software were installed in the facility.

Creation of a detail design document from a preliminary design requires conducting repetitive design passes that produce increasingly greater linkage details between the physical and procedural elements of the process (see CE , Apr. ’99, p. 89). To understand how physical and procedural elements work together to solve the control and automation equation, consider the following example.

A chef is asked to prepare a turkey with dressing. Preparation of the dressing requires use of measuring cups, knives, bowls, etc. (equipment recipes) and several intermediate recipes (i.e., melt, add and saute, mix). When the intermediate recipes are completed, the dressing is placed inside the prepared turkey (another intermediate recipe) and the entire thing is cooked in an oven (another equipment recipe) until done.

When multiple recipes (procedures) share a common resource, that resource must manage
and communicate its current status to avoid recipe conflicts and product contamination.

Detailing the design

In the Genentech kitchen, detail design required identifying all physical elements of the equation. At the lowest level of S88 models, control modules (i.e., pumps, on/off valves, control loops, agitators) are identified and designed to include the software logic necessary to be self-sufficient entities. For example, an on/off control valve module includes discrete inputs and outputs, operator interface, alarms, status feedback, interlocks, clean/dirty status, etc.-all the elements necessary for a self-contained, self-sufficient entity.

Adopting and holding firm to a self-contained, self-sufficient design philosophy permits higher level programming logic to acquire, for example, only a ‘clean’ on/off valve, or to make a request the on/off valve open or close. The on/off valve complies with request consistent with its present status and internal logic and notifies the requesting program when the request is or is not satisfied.

S88 control modules may be grouped together to form equipment modules for such things as jacket services, batching stations, transfer stations, header controls, etc. Again, each module is a self-sufficient entity. Moving further upward in the physical hierarchy, control and equipment modules are grouped together to form units, such as fermenters, filters, and holding tanks, and units are grouped to form process areas.

On the procedural side of the equation, Genentech’s multi-product, multi-stream process requirements constitutes complex batching. To maximize equipment usage and throughput while minimizing production cycle times, recipes must be executed concurrently, thus the need to provide product, intermediate, and equipment recipes (see Concurrent recipe diagram).

To successfully use a concurrent recipe philosophy requires special attention to how exclusive acquisition, use, and release of equipment entities by different recipes is conducted and managed.

For example, the Make product – step B recipe (green), Make buffer intermediate recipe (blue), Purge equipment recipe (yellow), and Regen cip equipment recipe (yellow) share Buffer hold tank #1 as a common piece of equipment. Being able to validate and ensure a product recipe’s genealogy meets specifications and attributes requires the contents of Buffer hold tank #1 to contain a single buffer solution lot number. That means Buffer hold tank #1’s equipment status must include:

  • Available and dirty;

  • Available and clean;

  • Available and clean but expired;

  • Available and sterile;

  • Available and buffer batched;

  • Buffer batched and expired;

  • Processing; and

  • Out of service.

When Make product – step A recipe is ready to advance to step B , the recipe logic will only acquire Buffer hold tank #1 when the status is Available and buffer batched . Similarly, intermediate recipe Make buffer’s logic can acquire Buffer hold tank #1 when the status is Available and clean. Equipment recipe Purge’s logic acquires Buffer hold tank #1 only when the status is Available and dirty or Available and clean but expired, and so forth. Adopting a philosophy of having each control and equipment module (i.e., Buffer tank #1) manage and maintain its own status avoids recipe conflicts and ensures product integrity.

By the time Genentech had reached the detailed design phase, the FDA had finalized regulations for use of electronic records and signatures. Genentech’s project team viewed this as an opportunity to improve record keeping by integrating operator technician interactions with the control system to strengthen quality assurance functions.

One of the keys in the validation process is proving a piece of equipment, a string of software code, or other component is predictable-that it does the same thing over and over. People aren’t always predictable; that’s why validation of manually executed procedures is so difficult and under constant scrutiny. But when well-developed procedures are combined with well-trained operators and a reliable, traceable, integrated electronic signature system, repeatable performance is consistently achievable.

‘Because our manual actions are under control of the batchmanager, they are consistently executed, confirmed, and electronically signed,’ says Eric Dolan, senior technician. ‘Technicians realize and appreciate the advantage of electronic signatures following completed actions,’ adds Doug Irvine, lead bioprocess technician.

Getting it done

Genentech’s automation vision required an ‘open,’ yet securable control system. After significant evaluation, Genentech chose Siemens Moore Process Solutions (Spring House, Pa.) APACS+ DCS (distributed control system).

Security was achieved by placing a firewall between Genentech’s corporate network and the DCS network where 45 unit controllers wired to over 6,000 I/O points, eight distributed graphic servers, shadowed batch management servers, and 24 human machine interface (HMI) nodes are hosted. Along with traditional corporate IT services, a few view only HMIs and a reporting/historian server are located on the corporate network.

Project size and complexity dictated stringent adherence to a formalized change management system to facilitate communication and coordination among the 40 or so engineers working simultaneously on the project. By the end of commissioning over 2,000 tests had been completed on the DCS hardware and software. Along the way Moore’s project team was able to incorporate enhancements that have since found their way into APACS+ standard batch product. Significant among these enhancements are:

  • Configuration tools that permit automatic configuration of unit relative graphic displays with unit data appropriately mapped and automatic generation of documentation and test protocols;

  • Advanced recipe sorting and filtering capabilities based on a recipe’s class, type, and status;

  • Advanced equipment handling techniques based on equipment status;

  • Change management improvements especially in the area of integration of electronic signatures; and

  • Report storage, accessibility, and generation modifications for continuous and batch event data.

Ensuring the project remained on schedule and that over 2,000 changes were properly defined, authorized, implemented, and tested required centrally locating the entire project execution team along with modification and enhancement of the procedures, policies, and systems used.

Among the procedures and policies changed was a move away from an ‘hours expended’ to an ‘earned value’ system, which tracks the progress of work completed.

On projects of this size and complexity, recognizing outstanding individual and/or team performance often gets lost in the day-to-day grind. Genentech’s management recognized the quality of work produced by the implementation team often can wane following long hours and numerous changes. To help ensure continuously high quality work and team spirit, Genentech implemented a discretionary bonus system paid directly to project team members for outstanding performance during the ‘heat of the battle.’

Moore’s APACS+ system includes a library of ready-to-use software modules, but the complexity of this project resulted in developing additional modules, some that became part of the standard product and a few developed specifically to meet Genentech’s requirements.

As different parts of the project progressed at different paces, the project team’s appreciation for the benefits of reusable software modules steadily increased and resulted in significant savings, especially in areas of testing and validation.

Like any control and automation project, what’s implemented and tested at the factory requires ‘tweaking’ once the control system is installed and operational. To facilitate timely and accurate on-site software changes two project team members were identified early in the project, remained intimately involved with the development and testing of the software, and relocated to California when the system shipped to assist Genentech with changes and enhancements.

At press time, Genentech’s final FDA approval is pending, but that doesn’t mean nothing is happening.

Efforts are underway for the control system to collect and share process data, alarms, etc., with an Oracle database. Using a dedicated report server, users set-up queries of the Oracle database to retrieve up-to-the-minute product life cycle reports using standard Web browser technology.

Information automation

Laura Wright, senior manager of manufacturing quality assurance explains, ‘Before a product can be approved and released, each production exception must be reviewed, commented, and electronically signed. For example, any alarm conditions occurring during production are treated as exceptions and must be reviewed, a determination made as to the effect each alarm has on product quality, comments entered into the system and attached to the products batch record, and everything must be electronically signed. This DCS is making it a lot easier to review by exception.’

Automating information in this way achieves two critical benefits. First, every action, every comment, every step of producing a product is electronically collected and associated with a specific batch. Poor handwriting, incorrect date and/or time entries, are eliminated. Second, having everything stored electronically permits data mining to compare batches, shifts, seasonal patterns, etc. making it easier to identify recurring problems and conduct root cause analysis to keep equipment and personnel performing at peak ability.

Genentech’s Vacaville, California, facility is an excellent example of how worthwhile automation projects can be when a vision is established and everyone works together to achieve or exceed the vision.

Electronic signatures gain acceptance

We use PINs (personal identification numbers) and ATMs (automatic teller machines) to move money around. We use e-mail, fax, phones, and other electronic means to put together a ‘deal,’ but when it comes time to finalize the contract, real pens and wet ink signatures must be attached in the presence of witnesses.

Until 1997 wet ink signatures were also required by regulatory agencies, such as the U.S. Food and Drug Administration (FDA), to validate correct procedures were followed when producing food and beverage and pharmaceutical consumer products. The widespread use of automation systems combined with hard-copy procedures and wet ink signatures made assembly of regulatory compliance documentation unwieldy.

Today’s encryption technology makes it practical to replace wet ink with digital electronic signatures in a host of industries.

A digital signature is not a signature in the handwritten sense of the word, but it serves the same purpose. In fact 26 states already have laws making digital signatures legally binding, thus further opening up the electronic commerce market being driven by the Internet. Fortunately the FDA’s definition of electronic signatures is aligned with the commercial world’s definition for electronic signatures. The FDA’s definition is, ‘A computer data compilation of any symbol or series of symbols executed, adopted, or authorized by an individual to be the legally binding equivalent of the individual’s handwritten signature.’

As everyone becomes more comfortable with digital signatures, their use will spread into all aspects of our lives including verification of who did what and when in most manufacturing processes. For example, management of change is a widely recognized term in chemical, pharmaceutical, hydrocarbon, and similar other industries, but remains mostly a paper trail. When an operator changes an alarm value or bypass’s an interlock condition many industries are covered by regulations that require such changes be authorized and documented, but few control and automation systems provide comprehensive toolsets to assist online management of change activities. Most control systems provide some form of password protected logon capabilities, but many control rooms either don’t use this capability, or have a single logon for an entire shift of people. The latter means that every change is logged to a single user, regardless of whom actually initiated the action.

FDA regulations addressing the use of electronic signatures in the food and beverage industry require every change be followed by the user’s password. It is not permissible to enter several changes followed by a single password entry. Each password must be unique to an individual employee. The company must be able to prove another employee has never used the password.

Digital electronic signatures being developed for electronic commerce, and already part of most secure Internet Web sites, use complex algorithms to create two ‘key’ numbers-a private and a public key. Think of these keys like comparing signatures at the bank. When opening a bank account, you left a copy of your signature on file. When you write a check, the bank can compare the signature on the bottom of the check (private key) with the signature on file (public key). But digital signatures are much harder to ‘forge’ than real signatures. The numbers making up a digital signature can be any length, but experts agree they should be at least 1,024 bits long. Similar to the bank example, you might use your private key to electronically sign a sell order with your broker. When your broker receives your purchase agreement they decode it using your public key. If your public key works, the purchase agreement is validated as coming from you and the transaction is completed.

Digital electronic signatures are fast becoming an acceptable method of ensuring your business transactions are in fact your business transactions. As health, environmental, and safety regulations continue to tighten, and as more companies move toward contract manufacturing, the need to ensure that product production can be tracked and verified will gain importance. Sophisticated digital electronic signature technologies will be part of the solution.