Project: Baton Rouge Wastewater Pump Station SCADA System – 2005-07-11

To read the blog for 2007 click here.

December 18, 2006

With the holiday season upon us, we are once again reminded that we have so much to be grateful for: our health, our families, and our friends, and we at QDS Systems would like to take this opportunity to wish all of our readers, coworkers, and associates a joyful holiday season and a very successful and prosperous new year. Our next entry will be in the New Year when we will be renewed, hopeful, and excited at the prospects of a brighter tomorrow.

The system integration blog update will take a brief hiatus from its regular weekly updates during the upcoming holidays. Watch for its return early in January.

December 11, 2006

This week our Engineering Manager met with the Prime Consultant for the City and visited the data center where the servers are housed. The two discussed system capabilities and the possibility of expanding the project.

As we await news from the City and there is little or no movement on the project, I would like to take this opportunity to recap the achievements to date on the project:

• Two servers are running SCADA software and a third is running a database application.

• 13 stations have been converted to cellular modem communication using Modbus.

• Graphics displays have been created for the 13 stations.

• Sample reports and sample help screens have been submitted for review.

• A sample ‘representative’ site has been set up demonstrating the graphics for the remaining stations and has been submitted for review.

December 4, 2006

This week has been another slow one on the project. The engineering staff continues to perform periodic graphic improvements and improvements in trending. Alarm notification is still an issue at this time with the hardware not functioning properly and our engineering staff resorting to calling for backup from the engineering manager. Hopefully, we will have better news next week.

November 27, 2006

Engineering staff continues to perform periodic work on items associated with our internal graphics review; however, due to the short work week last week and the uncertain direction the project is headed, there is little else to report. We are still awaiting a decision, which we hope will be to continue and even expand QDS’s role in the project.

November 20, 2006

This week our engineering staff continued working on the resolution of issues concerning our internal graphics review. Due to an unrelated incident at one of the upgraded stations, we are in the process of checking the city’s laptop computer, which is used to download to the RTUs for proper operation.

All of us at QDS also want to wish our readers a Happy Thanksgiving!

November 13, 2006

We are still awaiting the city’s requirements for reporting and alarm notification from the prime consultant. We have converted the 13 booster stations; submitted more screens for review including reporting, help, and screens for the next group of stations to be converted; are working on alarm notification functionality; and monitoring and working to perfect the stations already converted.

Until decisions are made by the city as to exactly how they want to proceed, this project will remain inactive.

November 6, 2006

The engineering staff worked on the issues noticed during our internal graphics review. A few minor issues needed to be corrected. While working on the corrections, an addressing issue concerning 4 output points per station was discovered. This is being corrected by our engineering staff. We are still waiting for the City’s review.

In other developments, the USB style audio device came in this week and it will be installed as soon as time permits.

October 30, 2006

After correcting the driver issue for the alarm notification software, tech support remembered a call from another customer concerning the need for a sound card on a server similar to the one we are using for the same purpose. Our purchasing department ordered a USB style audio device which should come in next week. We will try it when it arrives.

In addition, QDS met with selected members of the City’s maintenance personnel this week to discuss some options for converting the remaining 38 stations. Nothing firm has been established at this time, but it appears that they may be looking for viable alternatives to help get the project jump-started.

We started another internal review this week to try to improve the graphical displays. Some minor issues were observed and are being corrected at this time.

October 23, 2006

This week our engineering staff re-installed the hardware required for the alarm notification software. There is a driver issue, however, that we are working with tech support to resolve.

At our last meeting with the City, the subject of how to handle the stalled Phase II contract was discussed. City maintenance personnel pointed out that the Phase II plans did not accurately reflect current wiring at the stations. In some cases, the Phase II plans (completed by an electrical sub-consultant no longer under contract) did not reflect what was desired to be added and eliminated at the site. QDS had visited a typical station several months earlier with City maintenance personnel and verified these were valid issues. There was also a discussion whether the contract documents required the entire station to be re-documented by the Phase II contractor, as current documentation of wiring was inadequate for troubleshooting.

As a follow-up on this issue, another meeting has been scheduled this week with the City and engineering consultants, this time to view a pumping station to compare the Phase II plans with actual wiring and hardware installed at that station.

October 16, 2006

It seems we have a new DPW director, a new Chief Engineer, a new overall project manager /engineering consultant. And most of the key people in the WWT department have changed positions! A review of the project to date and underlying assumptions has begun. This week our engineering manager met with the City and consultants in a higher level meeting during which parties were brought up to speed regarding the history of the project to date. There was discussion of contractual issues, mainly how to re-invigorate the stalled Phase II contract. Key to the discussion was the question of whether to re-bid the contract, redefine and incorporate the work into existing contracts, or even to cancel all contracts and start over.

October 9, 2006

Gaining access to the remote server location proved easier than expected. First, my name was placed on a list by a responsible person with QDS. Then, I had to go to the site and fill out some paperwork. From there, I had to have my hand scanned. Then, I was issued a card. The card initiates a hand scanner. When the scanner reads my hand correctly, access is gained for two seconds. You then find yourself in a man trap. You have to repeat the process to get out of the trap and into the center.

All of that is to say that when I gained access, and installed the alarm notification hardware and software, it did not work. I lost my basic connectivity to the server and could no longer communicate. I removed the newly installed equipment and returned it to the shop. When I installed it there, it worked without problem. I learned from this that it is always better to thoroughly test the equipment at the shop before installing it in the field. I will try to reinstall it at the server location again this week.

October 2, 2006

This week we ordered a modem and voice-over-IP (VOIP) device to test the alarm notification software. Once the equipment arrives, it will be set up at the remote server location. In order to gain access to the location, I must first gain security clearance. If all goes well, I should be able to let you know how that goes next week.

We have yet to hear any feedback concerning our sample screens submitted last week.

September 25, 2006

This past week we officially submitted preliminary sample screens for the remaining stations to the city’s prime consultant who forwarded them to the city for review. That review is now pending.

This week our engineering staff started setting up the specified software package that will send alarm notifications to city maintenance personnel. This is specified as ‘a robust, redundant system using well-proven methods,’ allowing us to notify personnel via voice messages to their phone or cell phone, and via text messages and e-mail. Bidirectional communications will allow operators to acknowledge alarms via voice or text message.

September 19, 2006

As a former Boy Scout myself , I can attest to the fact that unexpected things happen — in business and in life. I am Mark Medley, the replacement engineer talked about in last week’s blog. About a month ago, I suddenly experienced some chest pain that sent me to the hospital. Two stents and 4 weeks later, I am happy to be able to praise the wonders of modern medical technology. And this after missing only a few days of work.

For the past several months it has been my job to write this weekly blog. It did not come naturally, either, and I needed to be groomed for the task. I will continue to write it until the project is completed; however, Stan Prutz, our Engineering Manager, may write it occasionally as he has something to say or if I am otherwise indisposed. For those of you who don’t know who QDS is, QDS Systems, Inc. is a systems integration company with two decades of experience in the municipal water and waste water market. The company supplies everything from a single packaged drive to a full-blown SCADA system complete with engineering, design, production, installation or anything in between. The company uses a distributed project management style that allows different engineers to own portions of the projects. This ensures that experienced resources are available — even when an unexpected illness such as mine arises.

As for the project….

The ball is truly in the City’s court with this project at this time. Sample screens, sample help, and sample reports are built and awaiting the City’s and prime consultant’s input. Periodic monitoring of the data transfer rates and the latest bills from our provider continue to indicate that all is well with the system.

September 12, 2006

It would have been hard to make contingency plans for the number of unexpected turns this project has taken. Add three more to the list.

Our replacement project engineer, on the job for about 3 months now and just up to speed, was diagnosed recently with two 99% and one 50% heart artery blockages. The doctors diagnosed and corrected one 99% blockage with our engineer being off the project for only a week. He is out this week for a stent to correct the second blockage–a procedure that we trust will go as smoothly as the first. The 50% blockage is being corrected with medication.

Changes in the Department of Public Works are our current major hurdle. A new DPW director and staff engineer, both familiar with building HVAC automation systems but not municipal wastewater SCADA and control systems, have decided to test their favored building automation system at a pump station site. This is being considered as an alternative to the pump controller recently selected in Phase I that the City has executed a contract on.

During a recent several-hour meeting with ourselves and the City’s staff engineer, the prime consultant tactfully attempted to explain why this is risky and not in the City’s best interest. While it has been explained that such an alternative could not come close to meeting the original specifications, the DPW director and his staff engineer have called into question whether all the functionality of the original specifications is necessary. Of course, we are considerably along in our contract, having earned in the neighborhood of 50% of the contract value so far. If the City should change direction at this point, they will be out a considerable amount of money!

And the third wrinkle: The Phase II contractor (supplier and installer of the control panels) who had been awarded a low-bid contract almost 2 years ago but was never given a notice to proceed, has been denied a change order to cover their cost changes since that time (which are considerable, particularly due to last year’s hurricanes Katrina and Rita). City purchasing says this portion of the contract must be rebid. Estimated time for the process: 9 months.

The prime consultant has recommended instead that the scope be changed to replacement of the interiors of existing panels, and that a change order be initiated to our contract for this work–a solution that would most quickly get this project to completion.

Not even a former Eagle scout could have anticipated all this…..

September 6, 2006

Engineering staff has continued working on the sample submittal for the remaining 38 stations , as well as the help screens and a reporting format . However, we are limited by the fact that we are still waiting on the city’s reporting requirements from the prime consultants.

We have had two internal reviews of the graphics and tentative internal approval to look into a W eb – based reporting format accessible directly from the control screens.

We are continuing the periodic monitoring of data usage on all of the modems. At this time , it appears that we have good control over all of the data usage.

August 28

Our engineering staff should complete the sample submittal of the remaining 38 stations by tomorrow ( 8/29 ) for review.

Continued monitoring indicates that we are well within the allowable usage for all modems.

We still await the city’s requirements for alarm notification and reporting procedures from the prime consultants.

August 21, 2006

In the last few weeks our engineering staff completed the conversion of the last 5 of the 13 sites that only needed communications upgrades. All conversions went as expected.

Template screens for the remaining 38 stations are being designed and a sample will be available for viewing by the end of the week.

We are still waiting on the prime consultant to provide information concerning the city’s requirements for alarm notification and reporting.

We received yet another high bill from the modem provider indicating that one of the modems transferred an excessive amount of data — even though all of the modems have the same utilization and none of the others used over the proper amount. We have requested a meeting with the provider to try and get to the bottom of this issue. Periodic monitoring continues with expected results, that we are within our proper usage.

July 31, 2006

Our engineering staff successfully converted 2 more stations this past week with no unanticipated issues arising from those conversions. Preparations were made to perform the conversion of the 5 remaining stations this week, 2 on Tuesday and the remaining 3 on Thursday.

Monitoring of the data usage on the modems continues with no surprises. Out of the 6 new modem accounts set up , only one needed tech support. That issue stemmed from the accounts being improperly set up at the beginning : the provider set up two modems using a duplicate ESN and IP address.

We are still waiting on the prime consultant to provide information concerning the city’s requirements for alarm notification and reporting. Today we will make a written request to the prime consultant for that information , because we are otherwise nearing completion and need it to fully complete this phase of the project.

July 24, 2006

Although our engineering staff intended to convert 3 stations this week, multiple problems with the first conversion prevented this from occurring. An issue of the pumps not running because no lead pump was identified was resolved by a programming scheme to identify the lead pump. Then there was an issue with the RTU not accepting the MODBUS configuration. The unit had to be replaced and the old unit must be sent to the manufacturer for repair. We will attempt to convert the remaining 2 units from last week early this week.

Six new modem accounts were opened last week for the conversions of 6 more pumping stations. The modem setups are proceeding even smoother and with less technical support than in previous times.

The periodic monitoring of the existing modems still indicates that the transfer rate is acceptable.

July 24, 2006

Although our engineering staff intended to convert 3 stations this week, multiple problems with the first conversion prevented this from occurring. An issue of the pumps not running because no lead pump was identified was resolved by a programming scheme to identify the lead pump. Then there was an issue with the RTU not accepting the MODBUS configuration. The unit had to be replaced and the old unit must be sent to the manufacturer for repair. We will attempt to convert the remaining 2 units from last week early this week.

Six new modem accounts were opened last week for the conversions of 6 more pumping stations. The modem setups are proceeding even smoother and with less technical support than in previous times.

The periodic monitoring of the existing modems still indicates that the transfer rate is acceptable.

July 18, 2006

A fourth and fifth conversion—including the write tests—were completed this week. The conversions were not trouble-free, however, prompting a revision in the procedure to ensure that the site can be returned to its pre-conversion programming. This week our engineering staff will attempt to complete three more conversions.

Periodic monitoring of the data transfer reported by the cell service indicates that the high data-transfer issue reported last week has been resolved for now. Continued monitoring will ensure an efficient data transfer rate.

July 12, 2006

Our engineering staff successfully completed a third conversion of an existing RTU to Modbus last week , as well as completed commissioning of last week ‘ s second RTU. Write capability was tested before leaving the sites using a wireless wide-area network (WWAN ) enabled notebook. This week we plan a more aggressive schedule and will attempt two RTU Modbus conversions.

This previous month we had an unexpectedly high amount of data transfer charged to the cell account for the first pump station we commissioned. This amounted to 46 megs of overage (23 times the normal amount), for which we were charged a whopping $670 in overages. As we had previously taken necessary steps to limit the amount of data transferred to the 2 meg contracted limit and tested this, we are unclear at this time how such an overage could have occurred. We are currently monitoring the data transfer reported by the cell service provider every several days to determine if their usage rates match with our own measures.

We still await the prime consultant regarding the City’s requirements for alarm notification and reporting.

July 3, 2006

Setup of the cell modems progressed better than it has in the past, requiring only 3 engineering hours to configure 6 modems. Verizon low – level technical support was able to resolve communications issues. We got the name and direct number of their technician for future reference.

Engineering staff performed the field conversion of the second pumping station to the new SCADA system last week. This will be 100% complete with another brief site visit. We plan to convert one station per week until the 13 stations where existing RTUs will be re-used are complete. Our goal is to convert stations in one site visit, minimizing the time to complete the job. A third pumping station conversion is planned for later this week.

We are still awaiting the prime consultant regarding the City’s requirements for alarm notification and reporting.

June 27, 2006

Received Static IP address for 6 more modems last week from Verizon.
Historically, these have been troublesome to get from them, often requiring their Level 4 support to intervene. We will need to test each to verify their functionality.

Performed a test of the writes for PS 514 last week with myself and a representative of the city present. Started and stopped all of the pumps from a laptop by first logging onto the server, and then by going to the website. This success should mean that we are very close to replicating the system to more sites.

The prime consultant sent us a recap of the meeting with the city last week but no further guidance on the reporting and alarming as we had been expecting. My hope is that this delay means that either the city has decided to write up their own requirements for these items or that the prime consultant has taken more time to get the city’s actual requirements from them before making assumptions as to what those requirements might be.

June 19, 2006

Met with city maintenance personnel and prime consultant to resolve some issues concerning alarm notification and reporting. Not much if anything was resolved at the meeting. The city personnel seemed unclear exactly what their needs were, while the prime consultant seemed bent on pushing its own expectations rather than finding out what the city’s needs were.

City and prime consultant both seemed satisfied with graphic design of the screens, with the exception of one minor problem with the resolution of a single symbol.

Began preparation for a second station to go on-line. OPC server has been configured and tested using a second modem and RTU representing a second station. Hope to finish all preparations early this week and field conversion by weeks’ end.

June 13, 2006

The engineering staff is working this week on OPC server redundancy issues and enabling SCADA write control now that security access levels have been established. Later this week we will again meet with the prime consultant and city maintenance personnel to conduct a second review and iron out issues related to configuration of alarms and reports.

Alarm notification within the existing SCADA system is presently configured such that critical station alarms are sent via text messaging to a shared cell phone. Operators can remotely acknowledge alarms. Alarm notifications are also sent when the alarms return to normal. Critical alarms are defined as station power failure or the failure of multiple pumps. A redundant third-party application including traditional voice dialer boards and telephone lines is employed in this scheme.

The prime consultant’s original plans were to duplicate what was used in the old SCADA system, using the same third-party application. The new SCADA system graphical software has an integrated ability to send e-mail messages. We are testing this to determine if messaging in this way can provide as reliable a notification system, which would represent a more integrated, simpler, lower maintenance solution.

June 6, 2006

As the new project engineer on this job I have spent the last couple of weeks getting up to speed on the scope of the job and all of it’s requirements. At the same time I have been familiarizing myself with the software and hardware for the project. There are some items which need to be completed before we move on to the next phase.

The past week has seen much progress toward the item on the first graphics review punch list. Our focus is on perfecting the graphics screens, control functions, and redundancy aspects of the system, as mentioned in the punch list, before replicating the system to other pump stations.

Some of the items in need of attention require input from the City and therefore a meeting has been set up for this week to discuss reporting issues, call lists, maintenance logs and input devices.

May 31, 2006

Last week the Engineer called a meeting to discuss our price escalation request. While this topic was the main subject of discussion, City maintenance personnel were on hand and numerous other matters were discussed. There appeared to be general consensus that our price adjustment requests were reasonable and justified. These requests will be lumped together with price adjustment requests from those involved in other phases to submit to the City Council for approval.

The first graphics review produced a long list of items needing to be addressed, many well outside the scope of graphics. Since the City accepted our alternate proposal to their original request, there have been some scope changes versus the original bid documents. This has resulted in it not being completely clear to the Engineer what is and isn’t included in our current scope of work. Recall the unusual situation of our receiving a notice to proceed on this project about 3 months prior to the Engineer (prime consultant) receiving a contract.

The Engineer has asked for a second graphics review meeting in the next few weeks, assuming a re-review of all the items from the first graphics meeting would be complete by that time. That won’t be the case, but we do need to have another meeting to continue to proceed in that direction.

May 23, 2006

It’s taking some time for our new project manager to get his hands around all the software involved in this project. Some detail cleanup work needs to be completed at the first pump station this week in preparation for changes being replicated to the other sites.

The City is moving glacially slow at getting the PLC/HMI contract issued. Glacially slow also describes the process of resolving change order issues and getting a notice to proceed to the contractor that is building and installing the control panels. The wheels of government turn slowly….

May 17, 2006

The prime consultant is just getting around to the matter of our price adjustment request from several months earlier. Their lead systems engineer has asked for supporting documentation (much of which we had previously supplied)in preparation for meeting with us next week.

Our project team conducted a half-day project review meeting, including the new and old project manager. Our new project manager continues to study the plans, the specifications and our project documentation so far to become fully aware of all job requirements.

May 10, 2006

We have a new project engineer on board that we are working to get up to speed on this project. Fortunately, he has considerable experience in this area, and expectations are he will be up to speed in short order.

We are working toward incorporating write access (the ability to override booster station automatic control from the SCADA system) and approval-meeting-requested changes into the initial station to prove these prior to replicating to the 12 other booster stations. Write access has been disabled pending direction from the City regarding its security requirements. It was decided that only three people would have write access, for which a supervisor login password will be required.

The prime consultant’s project manager is having some difficulty keeping all parties focused on completion of this project. Numerous entities’ approvals are required, including the City’s purchasing department, the City Council, Department of Public Works management, as well as sewer operations management.

Still no contract in place with a vendor to provide the PLC and local HMI hardware.

We understand that the prime consultant has reviewed our written request for price adjustments to the contract, and that we will receive a call shortly from their lead systems engineer to discuss this.

May 2, 2006

This week we found ourselves back at the first pump station, where we had placed an RTU, providing maintenance services when the station wouldn’t run after a rain storm. It has proven to be one of the costs of delaying the
project: as long as the project is still active, we are expected to direct system troubleshooting.

In this case, the station wasn’t starting because of a bug in the original program. The bug prevented either pump from starting if both pumps happened to be initially called in the same scan. This bug was corrected.

All the pumps weren’t running at the same speed (which is necessary because they all pump into a common header). Equalizing speeds required a recalibration of the signal follower circuit cards in the three analog variable frequency drives.

Regulation of suction pressure by the variable speed drives was erratic. Speed control is programmatically adjusted in the RTU with speed increment and decrement commands, dependent upon set vs. actual suction pressure. Examination indicated that tuning was required to increase the rate at which speed changes were made, but in smaller steps. This resulted in steady suction pressure regulation.

It was noted by the City maintenance personnel that while two of the three pumps at this station are allowed to alternately run, two always start. This was the way the program was originally designed. Because this is a booster pump station, it starts based upon incoming suction pressure. A change was made to allow the second pump to start at an adjustable higher suction setpoint, after an adjustable time delay. This allows one pump to alternately be called under lower flow conditions.

This station only runs during significant rain events. Fortunately, there was a heavy rain that day and the station was running, which allowed us to correct these issues. Most of these changes will also need to be made to the 12 other booster stations.

April 25, 2006

There are many things that can occur to disrupt a long-term project. One of them is personnel changes. During preceding years, the primary consultant lost its lead engineer and the end user lost its chief engineer. It’s our turn now to break in a new project engineer. Fortunately, our former project engineer is available on a part-time basis to ease the transition, we have a ready replacement, and our project management system has a structured, distributed design.

We still have not been given the official word on who was selected as a vendor to provide the PLC and local HMI hardware, although we believe the City exerted its influence to stay with the same vendor they were using previously, despite some known technical limitations.

April 17, 2006

Last Friday we conducted a second graphics review specifically with the primary engineering consultant. Although he was in India and I in Baton Rouge, we were able to have a teleconference and each access the SCADA system via our Web browsers. Unlike the review with the City earlier in the week, there was a lot of feedback, both verbally and then confirmed in writing. This review went considerably beyond the issue of graphics, also including the desired configuration for alarms, reports, redundancy and final documentation.

This week we will look into adjusting plans based upon the review. A number of the suggestions need to be implemented at the first station prior to roll-out to the other 12, so we can be certain we are not faced with considerable rework.

April 12, 2006

This Monday we conducted the planned graphics review meeting with the City and its prime consultant. This meeting was to confirm acceptance of the SCADA screens and layout for the first booster station, prior to our proceeding with cutover of the remaining 12 stations.

It was reported that the existing SCADA system has stopped sending alarm messages completely, and that the City did not want to invest any time or money into the old system. This has given some renewed motivation to moving the booster stations onto the new SCADA system as soon as possible.

Largely, the City was happy with the graphics as presented. A suggestion was made by an operator to add the setpoint ranges to the screens next to the actual values. The pump station number needs to be displayed on all screens.

One of the prime consultant’s engineers was participating in the meeting via a conference call. It was determined it would be best to arrange a separate meeting to address his concerns.

April 5, 2006

Some progress–April 10th has been set for a graphics review meeting. This is to confirm acceptance of the SCADA screens and layout for the first booster station, which is required before we proceed with cutover of the remaining 12 stations.

The Phase II contract has some issues that were pointed out earlier by the end user. Essentially, a number of items they say they originally asked for in the design didn’t make it into the Phase II contract bid documents. We reminded them again last week that these issues remain to be resolved, and they will need to take action on their end to resolve them–we can’t do it for them.

Our concern is that we can do a hundred percent fabulous job providing what was requested, yet still have a less-than-satisfied end user.

March 28, 2006

Still no word on PLC and local HMI vendor selection. We think they have decided but haven’t made it public yet.

Still no word on scheduling a graphics review meeting to confirm acceptance of the SCADA screens and layout for the first booster station. This is required before we proceed with cutover of the remaining 12 stations.

Still no word on a Phase II contractor notice to proceed. This is the portion of the project that includes building the control panels and installing them. My conversation today with the general contractor indicates that it has gotten very difficult for him, too, to get qualified people to man jobs. This situation started after hurricanes Katrina and Rita. It is expected to continue for years, due to the resulting huge demand for contract labor in South Louisiana and Mississippi. With higher costs to complete the job, an electrical subcontractor facing the same labor issues, and uncertainty regarding the desired layout of the panels, this phase may be headed for re-engineering and a rebid.

So, we are largely at a standstill waiting for others.

We completed the previously mentioned budgetary review of our portion of this project. The current projection from the engineer is that the project completion date will be delayed 1 year. We have requested adjustments to our contract for labor rate increases, extended mobilization, additional hardware and services as well as testing of the new PLC/HMI combination.

March 20, 2006

The previous week was one of the critical weeks for the project. We had a meeting with the engineering contractors to discuss our future plans. From our discussion with the engineers, it is clear that the selection of the hardware for the 38 RTU stations will be decided soon. Also, we put together a summary of the work that we had done until that point and requested an approval for the graphics so that we could proceed with the remaining 12 booster stations.

The most important concern for us at this point is to provide a working and an efficient system to the end users as soon as possible. It has been more than 6 months since we got the notice to proceed. Due to the frequent delays in decision making, the cost of the proposed system has gone higher. We have spent a lot of time on research and testing. We are sure that we can make the SCADA system work efficiently. It depends on how quickly the other two phases are planned and put into action.

On the technical front, we tested the redundancy of our OPC server using VPN connection between the two redundant servers. All our servers have dual Network Interface Cards. Initially we had connected the second network card of both the servers on a different switch to provide redundancy. The first network card of both the OPC servers were connected to different routers to provide different static IPs. However, we ran into some networking issues. We finally decided to use only one network connection and VPN both the OPC servers to provide reliable redundant system. We shall make those changes to the actual servers this week. We shall let you know how it goes.

March 6, 2006

Last week we discussed cell communications bandwidth usage, promising to give you some comparative data. Our comparison is based upon monitoring the system for the past 8-9 days.

After the changes made to aggregate data into 1 polled data block vs. 5 smaller data blocks, data usage went down about 80-85%. This shall save us a lot of bandwidth and will allow us to poll the sites more frequently. Along with the reduced bandwidth, the other advantage we saw was increased data accuracy. We had trouble reading a few blocks of data on an irregular basis. Our analysis over the past few days has shown that not a single data packet was lost after we made the changes.

We worked with the city engineer last week to get the city’s Nextel phone to recieve alarm messages. The tests showed that the messages were sent to the Nextel phone reliably in the form of e-mail messages. The city’s maintenance personnel have been instructed to take timely action upon receiving the alarm messages.

Our discussion with the city officials last week seems to indicate that the PLC selection for the 38 pump stations will be done soon. The engineers and contractors met with the PLC manufacturers last week to decide on the hardware. But, as always, we can just be optimistic and hopeful.

February 28, 2006

As mentioned in the previous blog, the data usage over the cellular network had to be limited using bit packing and sequential addressing methods. The changes were made to the RTU last week and our monitoring has revealed some really positive results. We shall be able to get the exact numbers after careful monitoring for about a week, but the rough estimates show that the bandwidth usage has gone down at least by 30%. This is a significant improvement considering the fact that the cell service provider charges a lot of money on any overages on the account. We should have some numerical data to share next week.

More than two months of monitoring, troubleshooting and modifications has brought us to a point to where we feel confident that the system works properly. Last week, we made electronic submittals for the graphics review of the software. We sent all the necessary details to access the site to the two engineers associated with the approval process. It’s an important test for us, and if we can cross this milestone, our journey through the remaining 12 booster station installations will be relatively smooth.

This week we will work on fine tuning the alarming capabilities and make sure the city engineer gets the alarm messages in time on his cell phone to take timely action. More on this next week…

February 21, 2006

Monitoring of bandwidth usage during the past week indicates our current polling scheme is not very data efficient. We need to stay within 2 megabytes per month data usage per site or data transmission charges will soar vs. the allocated budget. Considering the 27 words of data to be transferred, one might be led to believe updates every few minutes would be possible. The reality is that to send and receive a Modbus message, there is considerable overhead. To reduce the impact of the overhead, bits are preferably packed into words, and words are kept sequential within the RTU memory map so that one Modbus function call can read a series of registers in one request. Lacking a bit packing function, we have used a sequence of if/then statements to pack bits into words. The original RTU programs were not written sequentially (and could not be written sequentially due to controller addressing requirements). Moving values around inside the controller to force data into a sequential block will allow the OPC server to read all values in one read.

The primary consultant announced last week that they intended to have a PLC and HMI recommended and selected within the next two weeks. Seeing will be believing—past history indicates this projection is probably overly optimistic.

February 16, 2006

Only the very patient should pursue municipal work.

When will our project get back on schedule? When can we venture to accurately re-schedule? If you look back, you will see that on September 6th, 2005 we first reported that the City of Baton Rouge had decided it needed to rebid the contract that supplied the PLC and HMI hardware to be used at the remote pump stations in this SCADA project. Despite the lack of feedback from the primary consultant, we are moving forward with amending the project schedule under the assumption that a new PLC and HMI hardware vendor can be selected and contracted with by the end of April.

A complete budgetary review is also being performed. It is suspected that the continuing delays together with price increases over the past year are having a negative impact on the job profitability, due to no fault of our own. More about this after we finish the review.

Last week’s RTU analog output problem turned out to be self-inflicted. The problem originated when our field personnel switched the original 12V dc supply with a new battery-backed 12V dc power supply and charger system. The existing RTU vendor’s 12V dc power source had a requirement to have the negative supply side grounded. Based upon my conversation with one of the RTU manufacturer’s engineers, this point is not well known by the manufacturer, either. It took one of our personnel who had been involved with the original RTU installation 10 years ago to isolate the problem. Such is the value of experience.

February 6, 2006

Last week, we spent some time for some trouble shooting at the test pump station. The pump station was having trouble sending 4-20 mA speed reference signal from the PLC to the VFDs (variable frequency drives). The PLC was bypassed and the current signal was sent from the main control panel directly to the 3 variable frequency drives. The VFDs ran fine if external source gave the current signal. Our analysis showed that the controller had some trouble and it could not generate the required current signal. To test the controller, it was isolated and it gave correct current values at its output card when tested. We are in touch with the PLC manufacturer to help us troubleshoot the problem. We should be able to get a solution to it soon.

We are seeing some immediate advantages of the SCADA system. The problem described above was determined by our regular monitoring of the system since installation. Problems like failing of pumps due to controller or wiring errors can be eliminated in the earlier stages and hence expensive equipments like drives and pumps can be protected from damage. Moreover, timely intervention by the management personnel can help prevent production downtime in any industrial environment.

Our public access period finished last week. We did not get messages of any major troubles accessing the site. This has definitely boosted our confidence in the system that we designed. We are sure that the customers shall be happy to manage their systems more efficiently and in a timely fashion from any corner of the world.

February 1, 2006

SCADA on wheels, anyone? We mentioned implementing SCADA on a Sprint PC Phone last week. For those of you who would prefer to monitor and control your facilities from the convenience of your car (even while cruising down the Interstate, with a friend’s assistance) we’ve got you covered.

Our implementation includes a vehicle-mounted ruggedized PC with a special auto-specific power supply that can withstand the low voltage of engine cranks and automatically shutdown Windows XP and the computer after a programmable time delay or when battery voltage drops below a minimum. This is mounted under the rear seat of my Suburban. In the front console is mounted a pop-out 800×400 VGA touchscreen. A USB hub is mounted in the center arm rest. A Gyration gyroscopic mouse is mounted in the console change tray and an RF keyboard tucks neatly between the seat and console.

Internet connectivity is provided via a Wi-Fi station adapter for when I am parked at my home, office or at many hotels. A Verizon nationwide Broadband wireless PC card gives full broadband access in many markets with 70-110K performance throughout Verizon’s digital coverage areas.

SCADA access is full, using either a client or Web browser. Our intention is to demonstrate the capability to clients, as we see this technology is convenient and a productivity enhancer for application in many maintenance vehicles, including water and wastewater systems.

As a side benefit, such a system is quite convenient for access to the office on the go. Need to search a knowledge base or download a file while you are at a customer site? Go to your car and cruise the Internet, or download a file to a USB key. Need to get something you left at your office? No problem, we log onto Windows terminal services and our complete desktop is available from our vehicle. With our busy startup schedule it has been convenient for these weblog entries. The last few have been written and submitted on the go driving down the Interstate.

January 24, 2006

One week left to the public access period. We will assume no news is good news and that no one has had any serious problems in accessing the SCADA site. We worked with the city’s IT department last week to open the necessary ports for the engineers and maintenance staff to be able to access the SCADA site from their Web browser. The approval had to undergo a systematic hierarchical process with justifications about the need of doing it and the security issues involved with it. It was a difficult situation for us to get approvals at some levels but patience paid off. The city officials can now successfully access the site from their office computers.

Another important milestone is the ability to access the SCADA site from a Sprint PC phone. The device is an Intel Arm Processor based PC phone with Windows Mobile 5.0 Operating System. We successfully installed the Active X control on the phone to access the SCADA site. The device connects to the Internet using the wireless LAN connection if available or with the cellular connection from Sprint. The success of this connectivity proves our point of providing real time data to the maintenance personnel anytime, anywhere. We also installed Indusoft Web Studio on the phone, which means that we can run a client server application on the phone. This will permit the development of local HMI applications for any control loop at the plant.

January 18, 2006

As we mentioned in our previous log, our SCADA website is open to public access. To access the site, Internet Explorer version 5.5 or above is required, however IE 6.0 is recommended for errorless operations. The following settings are needed on your IE web browser. Open Internet Explorer and go to Tools->Internet Options. Click on the tab called Security. Click on the button Custom Level. The page showing the security settings shall open up. Make sure the following 2 options are set to enable: 1. Run ActiveX controls and plugins and 2. Script ActiveX controls marked for scripting.

Also , TCP ports 80 and 1234 need to be opened on your gateway. If you have any troubles making these changes, please contact your system administrator.

Once again, the site can be accessed by copying the following link in your Internet Explorer browser address bar : https://69.2.42.180

On the prompt for username and password, please put username as ‘ guest ‘ and password as ‘ qds ‘

The site is open for public access till the January 30th. All your feedback, positive and negative , can be directed to sprutz@qdssystems.com

January 11, 2006

Since the first SCADA pump station has been up and stable for the past few weeks, we are ready to begin a public access period. Here are the instructions for accessing the site:

In your Internet Explorer web browser type the URL 69.2.42.180

The first time the site is accessed a dialog box will appear asking for your permission to download a Microsoft ActiveX control named ‘Indusoft Symbol Control’. You must allow the control to download to access the site. If your Internet Explorer security settings are set to block ActiveX controls, you will need to contact your system administrator.

A dialog box will appear; enter the User Name as guest and the Password as qds

Since the layout is subject to customer approval, you will see only the first of the 51 sites active (Pump Station 514). While the system is capable of remote pump run and speed control, this capability is disabled for public login. This pump station is not equipped with a rain gauge, so that screen is inactive.

Comments regarding your experience with the site are welcomed. These can be directed to sprutz@qdssystems.com If you are not able to access the site, we would appreciate a short note describing your browser version, operating system and network configuration. We can provide limited assistance.

The public access period will end January 30th.

January 3, 2006

We left the SCADA system working over the holiday weekend for testing. The system is doing what it is suppose to do , but there is a lot of fine tuning still needed. We requested feedback from quite a few of our employees about the functioning of the Web site and the ease of use; this week we will incorporate those suggestions into our system.

December 19, 2005

Some fruits of our labor are finally beginning to appear. The engineering staff did well to get the SCADA servers installed and running at the data center this past week. As I write this, the first of the upgraded existing RTUs is being installed at a pump station. The SCADA interface is taking form, though definitely still in the beta phase. We expect to have news next week regarding a public access period to the site.

January 3, 2006

We left the SCADA system working over the holiday weekend for testing. The system is doing what it is suppose to do , but there is a lot of fine tuning still needed. We requested feedback from quite a few of our employees about the functioning of the Web site and the ease of use; this week we will incorporate those suggestions into our system.

December 27, 2005

We successfully installed the first RTU and our Web SCADA system is up and running. It is in the testing phase and we will be monitoring the system this week. Hopefully, after the test is successful, we can allow you access to the site to gain feedback on some of the improvements needed.

Wishing all our readers a very Happy New Year.

December 19, 2005

Some fruits of our labor are finally beginning to appear. The engineering staff did well to get the SCADA servers installed and running at the data center this past week. As I write this, the first of the upgraded existing RTUs is being installed at a pump station. The SCADA interface is taking form, though definitely still in the beta phase. We expect to have news next week regarding a public access period to the site.

December 13, 2005

With the holidays approaching, the heat and zeal to get work done faster has increased. We are finalizing multiple projects , and a major breakthrough last week is helping keep our spirits high for the holiday season –we successfully established Modbus communications with the existing RTU. Our engineers , with the help of the RTU manufacturer’s tech support, got the communications up and running on the proposed Airlink Raven CDMA modem.

We are planning to install the first RTU in the next two weeks. Our servers are ready and set for collecting the data. We will finalize the HMI for that particular pump station and try to get the system running by the end of the year.

December 6, 2005

Well, we got an answer sooner than we expected regarding the primary consultant’s evaluation vs. our own on the RTU and local HMI. Turns out that the primary consultant isn’t going to evaluate our solution at all. The primary consultant and end user’s lack of prior experience with our proposed solution seems to be the main reason. My point of view is, what’s the harm in looking at something new? After all, the end user was willing. This circumstance reminds me of the old cartoon of the salesman trying to sell Gatling guns to a colonel leading a battle with musket-armed troops who tells the salesman: ‘I don’t have time to look at your contraption, I’ve got a battle to fight!’ The good news in all this is that the primary consultant is in agreement with our need to test whatever RTU and HMI is chosen, and that this will be something we are compensated for.

After the selection process has been completed and contract issued, we will look into the possibility of doing a comparison between the selected hardware and our picks.

We have also encountered some problems with our updated Modbus firmware from the existing RTU manufacturer. It seems the RTU manufacturer’s engineering group responsible for getting this together for us never got the word we were doing a firmware upgrade for Modbus connectivity. Our old program was checked out on the controller by the RTU manufacturer, but they never checked to see that the Modbus functionality was working. Well, it’s not…more on this next week.

November 29, 2005

With a short work week last week, we focused on moving things along with Phase I of the project. We made a request to the primary engineering consultant to add a vendor to the Phase I RTU and HMI solicitation. In our September meeting, we agreed with the client and consultant that an Ethernet vs. Modbus architecture would be preferred. This decision is pointing us toward some considerably different hardware and software.

We have experience building systems using every PLC and specialty pump controller product the city has ever considered. Based on our experience, we chose to do our own RTU/HMI evaluation and tested the most promising products with our system, so as not to find ourselves without a solution that both we and the client could live with. The testing demonstrated our chosen solution offers considerably more functionality for the customer than specified, at about the same price level. It will be interesting to see how the primary consultant’s evaluation compares to our own.

November 22, 2005

The city of Baton Rouge has committed to a Phase I RTU and HMI re-request for proposals late in November with a due date the end of December. It appears possible, then, that a decision will be made sometime in January and a contract completed by March for Phase I, and a notice to proceed issued on Phase II (the installation and panel building contract) about the same time. It appears last week’s prediction of the 2nd Quarter 2006 to get restarted is on target.

We set up the phone/PDA communications on Sprint and, apparently are working with the right person, as our phone/PDA device running Windows Mobile 5.0 was here the next day! We have begun testing this as a tool for SCADA remote access. Currently, it looks promising and is very cool. More on this later.

Materials were ordered to build up a replacement radio cabinet (scope change), but we’re still on target for a complete SCADA system operational test (to one pump station) before the end of the year.

As a special Christmas techie treat, since this is a Web-based SCADA system, we will plan to make the system accessible to YOU for a few weeks, beginning sometime in December, before it goes into service for the customer. It will be beta at that point, but it will be a good test of the system to see how it holds up under general public access. Check future entries in this blog for sign-on instructions.

November 16, 2005

The existing RTU manufacturer shipped the completed Modbus firmware revisions, clearing the way for us to get a sample pump station completely up and running on the new SCADA system in the next few weeks.

Because the firmware was a week late, it did not arrive prior to our project engineer leaving on a scheduled 2-week vacation, which means that other jobs take priority now.

Meanwhile, the city of Baton Rouge purchasing department has suggested a ‘streamlined’ approach for the re-request for proposals of Phase I, however purchasing still has not provided any dates to the primary consultant as to when this process can begin and be completed by, thereby hampering our project scheduling.

With the holiday season approaching, we are lowering our expectations for a speedy completion date. Getting things restarted in the second quarter 2006 is looking more likely each week.

November 8, 2005

The prime consultant is meeting this Wednesday with city purchasing for approval to advertise the Phase I RTU and HMI request for proposals. It will need to be advertised for at least three consecutive weeks. The soonest this can be evaluated and awarded is probably 3 months from now. They are planning a future meeting to discuss the details of changes requested by the city that will affect the Phase II installing contractor.

The existing RTU manufacturer is likely to have in our hand this week the completed Modbus firmware revisions, allowing us to get a sample pump station completely up and running on the new SCADA system next week. Hopefully this RTU to SCADA communication checkout will get done before our project engineer is scheduled to leave on a two-week vacation!

The phone/PDA device running Windows Mobile 5.0 we ordered for testing has never arrived – we need to follow up with carrier on that one. We did, however, prove we can display a simple HTML page from our recommended HMI on a common Verizon cell phone. Recent ‘Wall Street Journal’ accounts of the state of the cell phone industry indicate providers are running toward providing Web services, Wi-Fi, music, TV and other functionality in your cell phone. We are convinced it will only be a few years before maintenance personnel’s use of a cell phone to maintain their systems becomes common.

November 2, 2005

The prime consultant is routing a final draft of the Phase I RTU and HMI request for proposal document to city purchasing for approval. They are also working on terms of a change order to cover post-bid cost increases incurred by the Phase II installing contractor. These increases occurred due to the delay of issuance of a notice to proceed, now extending beyond a year from the initial bid date.

A number of SCADA server configuration tasks were completed. These included review and purge of unused RTU database tags and software licensing setup. Routers and a VPN were setup to simulate the IP connection between the remote data center collocated SCADA servers and our office, which is required for remote maintenance of the system.

The existing RTU manufacturer will complete the Modbus firmware revisions this week, allowing us to begin final RTU communication checkout for existing RTUs to the SCADA system next week.

October 27, 2005

The prime consultant on this project has completed a draft of a Phase I RTU and HMI request for proposal document that is being circulated with the city of Baton Rouge. This consultant also had a meeting with the Phase II installing contractor and the city, mainly to provide them a status update on the project.

We are in the process of testing a phone/PDA device running Windows Mobile 5.0 for use by maintenance personnel in receiving after-hours alarm notifications. The city currently uses a Nextel phone to get email alarm notifications. We believe we can duplicate the existing email alarm functionality plus give them the ability to have the same access to screens they would see on the local HMI. Monthly account cost is projected to be somewhat higher, but close to the cost of their existing service.

Update of the existing RTUs to Modbus-compatible firmware has been proceeding slowly, due to the spare controller being lost somewhere in shipment back to the manufacturer. Also, we have discovered that there have been undisclosed software and hardware modifications done to the first booster station to be tested. This will add to our cost since we need to replace a missing radio and battery charger cabinet (will require change order). Software changes added to this station complicate the original design which had standardized programs for the 13 booster stations into two programs – one for the 2-pump and one for the 3-pump stations.

October 19, 2005

Our last meeting with the prime consultant and city was on 9/23. In that meeting it was agreed that employing standard Ethernet communications was the preferred architecture for the project. In hopes that it would speed things along, I gave our engineers approval after that meeting to consider alternative architectures employing Ethernet HMIs and controllers with an alternative direct Ethernet cell radio. We considered solutions available in the same price range as the RTU and HMI that was to be provided on the previous (now expired) contract. We tested the most promising alternative. The tested alternative also included a cost-effective 0-55 degree C Ethernet switch/router/firewall.

The capabilities of the tested architecture/hardware/software (not specifically mentioned in the original spec but desirable) include:

1. Elimination of most relays and all analog isolators:
a. Digital I/O is individually isolated (necessary to work with separate CPTs in motor starters and VFDs unless relays are added);
b. Analog inputs (differential) are optically isolated so that station 4-wire drive speed feedback and flow inputs stay isolated from one another;
c. Analog outputs are optically isolated so drive speed reference outputs stay isolated from one another; and
d. The 2-wire analog input (differential) from bubbler pressure transmitter powered by panel 24V dc power supply, is isolated from all 4-wire inputs

2. Centralized configuration management for the RTU and HMI (the city has had definite problems with this in past), which makes it easy for maintenance personnel to maintain the system. a. RTU and HMI can be reprogrammed over the cell network from one of the SCADA servers (by QDS Systems or the city using remote terminal server administration, from any authorized Internet-connected computer); b. RTU and HMI have common IP addresses at all sites; only the modem address varies. Router/firewall/switch has common configuration at all sites. This is important in that it allows any of these parts to be changed without needing to use a notebook computer to set an IP address (in other words, they need only one spare of each type of device and we pre-configure all the spares with the default IP address). Because Verizon takes awhile to setup cell accounts, a spare modem and Verizon account is recommended. If a modem needs to be changed, the city can change the modem, call us, and we will (from our desktop while they are on the phone) change the IP address in the OPC servers to point to the new modem’s IP address; c. HMI contains configuration of RTU. To replace an RTU controller head, replace head (5 minutes) and push download button on HMI (password protectable). We can also, from our desktops, download to the RTU from the central server; d. HMI application resides on a compact flash card. To change out an HMI, replace unit (5 minutes) and move compact flash card to new unit. If flash card has been corrupted, use supplied backup flash card. We can also, from our desktops, download to an HMI from the central server; e. Setpoint changes have often been lost when controllers are replaced. This can be virtually eliminated by the HMI automatically updating its copy of the RTU program on a daily basis, including setpoints. To ensure central configuration stays current without incurring excessive cell traffic, we recommend that RTU configurations be transmitted from the local HMIs to the central server on a monthly basis; and f. HMI has the same look and feel as the central HMI, reducing the operator and maintenance personnel learning curve.

Since the selection of a supplier for this equipment is still undecided, we will reveal results of our testing and specific vendors in a later blog posting.

Are there other possible solutions? Certainly. Is our proposed solution a stronger technical and maintenance solution than the previously approved Modbus SCADA RTU and proprietary HMI? We believe so. It is low risk for us as we have experience using this hardware and software successfully, and have fully tested it for use in the city’s particular configuration. Importantly, it is compatible with what we are providing in the other phases without adding cost to those phases.

We spent considerable engineering time checking out the previously approved Modbus SCADA RTU and proprietary HMI alternative before we bid the Phase III SCADA and Phase IV communications contracts to verify we could meet performance requirements and to factor in how much time it would take us to develop on those platforms. We have again done this for the above suggested configuration at our cost. We have already done twice the normal checkout for free.

Of course, additional system configurations will need to be tested and approved by us to guarantee system performance. We will need to charge for this testing. As the prime engineering consultant also has to account for each hour, we trust they will understand that we may also need to charge additionally for our engineering on Phase III and Phase IV if we estimate the engineering development time will be longer for an alternative platform. We knew none of this extra cost was going to be very palatable, which prompted us to test and offer an alternative system. The proposed system meets and exceeds the original specifications and, because of past experience, we know we can configure it without incurring additional engineering costs, allowing us to provide a ‘no additional costs’ guarantee.

October 11, 2005

This past week focused on existing RTUs. It began with the cityof Baton Rouge calling us to assist with two of the RTU units they were not able to replace successfully at two stations. Several problems surfaced. Configuration management was lacking. Each RTU requires about 10 files to be restored to fully reconfigure it. The backup CD had many versions of each station’s configuration files that needed to be picked through to determine the most recent. While later versions of the programming software support a batch upload/download of these files, this was not the case with the earlier versions the city had employed. Resulting backup files were given various similar, but different, names which needed to be changed manually to a common name to be compatible with the batch upload/download structure. The original comments were missing from the application programs. We discovered that the 2.1 programming software’s editor was too small to handle the existing programs. We also discovered that RTUs had undergone a file structure change some years ago so that the city’s 2.42 version would not work with their RTU’s firmware. Thankfully, we had a copy of version 2.3 that did work.

In the next few weeks we will begin updating all firmware in these controllers for Modbus support. Given what we learned this week, the programs will have to be restructured somewhat for the manufacturer’s firmware changes. We have decided to take the manufacturer up on their offer to convert the first one for us. This firmware update will allow us to use the latest version of the programming software.

No word yet from the primary consultant regarding our offer to create a revised set of drawings, nor a release date for a new request for proposals on Phase I.

October 4, 2005

Some of the items on last week’s meeting agenda with the city of Baton Rouge indicated they did not believe the plans and specifications adequately addressed their maintenance personnel’s needs at the individual pump stations. This
issue centered on wiring documentation for the complete pump station and elimination of excess relays from earlier installations. There was disagreement between our and the primary consultant’s interpretation of the specifications regarding the obligation of the contractor to document the complete station’s connections, including new and existing wiring.

It was the city’s belief that the plan documents did not adequately describe what currently existed at most stations. Preparation of plans and specs was contracted to an electrical sub-consultant who was not present at that meeting. We agreed to visit a typical station and trace wires to see what was installed, and compare this to the plans.

So, this week we inspected a typical pump station employing three variable frequency and two fixed speed pumps. An integrated control panel with micro-controller and bubbler systems originally regulated the station. When telemetry was first added 10 years ago, a second control panel was installed with a combination RTU/controller. The original micro-controller in the bubbler panel was removed and replaced with an HMI connected back to the RTU/controller. All interposing relays and inter-wiring from the original micro-controller remained.

We located copies of the manufacturer’s original drawings for the bubbler panel, micro-controller, RTU/controller, and variable speed drives, using these to trace thru actual field wiring.

Missing from the plans is detail on how devices are physically wired at present. Also, the need to do demo work to eliminate the multitude of relays is not noted. Our tracing showed almost all the 50+ relays installed could be eliminated and that more than half of those existing are no longer used. Differences between the actual RTU I/O count and that indicated in the
plans could result in the contractor asking for additional compensation. We offered to create a set of revised drawings for the inspected station for use as a template that the electrical sub-consultant could use to revise the others.

September 27, 2005

Last week, chance meetings with maintenance personnel reviewing the
Phase II drawings indicated that there were numerous scope and technical
issues to be resolved. Left unaddressed, we could find ourselves in a
position that no matter how well we performed on the contract, the city
would be unhappy.

To address this problem, we helped the city develop a punch list of their concerns.
Discussion of the rebid of Phase I and this list became the agenda for our
meeting with city maintenance personnel and the primary engineering consultant.

It was agreed that employing standard Ethernet communications is the
preferred architecture. The primary consultant expressed a strong
preference for RTU and local HMI units to be selected from among the major PLC
manufacturers, for reasons of reliability and support. Our experience has been that a
large installed base is the most important barometer of future hardware support.

We have found that the biggest problem for our customers with local HMI
systems is their short design life–typically 5 to 10 years is the average HMI life span before
a complete software redesign is required for the next generation of hardware and/or software.
For local HMI devices, it is now possible to buy
commoditized WinCE-based touchscreen displays that will run WinCE versions
of graphical interface programs from several major HMI software vendors.
Using this platform, the local and main SCADA HMI systems can be designed to
maintain the same look and feel, lowering the learning curve for operators.

September 19, 2005

Later this week we will meet with the city maintenance personnel and primary engineering consultant to discuss how to proceed with the Phase I RTU (remote telemetry unit) and HMI (human machine interface) contract. Because this portion of the contract expired, it must be rebid. A request for proposals process will again be employed to award this contract.

We are investigating the possibility of using standard Ethernet communications to each site where new RTUs are being installed, retaining the originally planned Modbus communications over TCP for existing RTUs. Our ideal solution for this customer is to use an Ethernet version cell modem and industrial router with NAT (network address translation) and port forwarding. For the RTU, we see a direct Ethernet interface type controller with terminal block type I/O device that can be flexibly oriented to match the client’s existing wiring. For the HMI we have had good experience with the use of standardized Windows CE 6-inch color touch screens. We will propose Indusoft graphical operator interface software for Win CE, the same as used for the main SCADA system interfaces. Using this architecture, there are multiple vendors with compatible hardware and software, assuring the overall SCADA system is not obsolete by the end of life of any one vendor’s product.

A standard Ethernet communications architecture allows the possibility of write-by-exception from field RTU units without additional hardware or software being added to the current design (specified as a polled master/slave system). The advantages of this approach are faster update times while staying within the cellular provider’s cost-effective 2MB monthly bandwidth program.

SCADA Servers and Modbus firmware for existing RTUs has been placed on order. This week we are working on finalizing the required application software orders.

September 13, 2005

Business is starting to settle back down after hurricane Katrina. The planned meeting with the primary engineering firm and the City of Baton Rouge maintenance personnel has been delayed until next week due to vacation schedules at the city.

Between now and then, we’re looking at retaining 13 of the existing RTUs, requiring firmware upgrades to support the Modbus protocol. RTU capabilities with the new firmware were confirmed with the manufacturer last week. Updated EEPROMS are now being ordered along with the server computers, software, and cell modems, allowing us to move forward with a full SCADA implementation at 13 of the most critical stations. This change of schedule prevents Phase I delays from affecting our software development critical path.

September 7, 2005

Hurricane Katrina has passed—the worst storm ever to hit the Gulf South. New Orleans looked like it would be spared until several levees gave way. Now over 80% of the city is covered with up to 20 feet of water. A million people are displaced, which is expected to continue for a minimum of six months to a year.

A large start-up we were working on in the Mobile area was, surprisingly, not postponed due to the hurricane. This was completed in the three days immediately following the hurricane. Getting there from Baton Rouge was an obstacle course. Very little power in Mobile. No hotel rooms were available—we stayed in the client’s home. Only a few restaurants were open (people were never so thankful for a McDonald’s hamburger). Gas lines 50 cars long in front of gas stations that weren’t even open—it took most of the morning to get gas Wednesday. Although our start-up conversions were completed and tested, the client was not able to restart the process line due to low gas pressure in the area.

The exodus from the New Orleans area has increased our Baton Rouge population by an estimated 50% overnight. Many people have taken in friends and relatives. We have become FEMA headquarters for the rebuilding of the New Orleans area. Almost everyone has been affected in some way. Nothing is business as usual anymore.

We have made a business decision to give priority attention to repairs of water, wastewater, and other critical infrastructure when we are asked to become involved. We will delay current customer project completions when these emergency requests occur, which I believe they will understand under the circumstances.

We spoke briefly with the primary engineering firm about having a meeting with them and the City of Baton Rouge maintenance personnel in the next week, as the Phase 1 contract for the RTU and HMI hardware has expired without resolution. And the city purchasing agent has ruled it cannot be extended without a new request for proposals. We are estimating this process will further delay the project overall completion by 60 to 90 days.

I don’t know what project planner could have anticipated the obstacles and delays we have encountered this past week.

August 29, 2005

A difficult week in Louisiana. As I am writing this, a Category 5 hurricane is passing by and we are without utility power. This appears to be the worst storm in more than 25 years to hit the gulf coast. Some of our customers in the New Orleans area in for a rough few weeks—levee breaks are letting the Mississippi River flow into neighborhoods bringing 12 feet of water.

We spent most of this past week preparing for a large start-up in the Mobile area that is likely to be postponed due to the hurricane.

As is the case when a major hurricane appears in the gulf, business life tends to slow as other concerns loom. Therefore, we are not expecting to make much progress on the project this week as people focus on cleanup. A good project planner in this area of the country allows a few weeks for lost time due to hurricanes. For the moment, we are grateful for UPS backup systems.

August 25, 2005

Completed initial end-to-end testing of trial system. This included Control Microsystems ScadaPack Modbus RTU communicating via an Airlink Raven CDMA cell modem over Verizon’s cell infrastructure and the Internet to a host running Software Toolbox’s TOPS Modbus OPC Power Server, Indusoft’s Web Studio HMI, and a networked Microsoft SQL server.

While there is still no word on release of the Phase II contract’s notice to proceed, the good news is that the city of Baton Rouge appears to have worked out details of a contract extension with the overall engineering consultant, which should be completed soon. We are working now to set a date to meet with them and the city’s maintenance management to develop an agreement about this extension. Getting this issue resolved will definitely help pick up the pace of the project.

August 17, 2005

I spent most of the past week meeting with the city’s appointed project engineer. During our initial meeting it was noted that a study for restructuring of the city engineering department has been completed. There will soon be someone permanently appointed as the city engineer (the current city engineer is serving as the interim).

A site visit was made with the city project engineer to NTG (Network Technologies Group), the owner and operator of two local data centers. These facilities are designed with industry-leading physical and network security protection. Their primary data center includes FM-200 based smoke detection and fire suppression systems, redundant cooling controls, redundant UPS and redundant diesel generation systems. This data center is supplied via four Tier 1 Internet providers with fiber backbone, filtered by a redundant Cisco firewall capable of being virtually partitioned. Physical security includes 7×24 on-site monitoring and security personnel, magnetic card readers, biometric hand scanners and man-trap security zones. The base cost for collocation of the SCADA servers at the data center is approximately $500 monthly, including 40GB of monthly bandwidth.

One of the primary advantages offered by the data center is that it provides a clean, protected environment vs. the city’s wastewater central control room. This is important, as most past failures have been related to residue build-up on floppy, CD, tape drives and power supplies. Hard disks have also failed due to computers being accidentally bumped by maintenance and janitorial personnel.

The data center location guarantees QDS Systems 7×24 access, which is important as we are providing a three-year maintenance contract for the collocated equipment. The data center’s provision of redundant power, 7×24 monitoring, firewalls and off-site data backups eliminates the need for the purchase, configuration and maintenance of individual UPS systems, tapes, tape drives, network switches and firewalls.

A site visit was also made with the project engineer to another QDS Systems customer, a $1 billion+ producer of bread, buns and specialty baked goods. Over the past 5 years this company switched its mission-critical sales and shipping operations to a data center hosted implementation of SAP’s enterprise resource planning system. This company recently faced a sharp upturn in itsr business when a major competitor began closing marginal plants as part of a Chapter 11 turnaround. This required major rescheduling of production and shipping at 40 plants as well as managing them carefully to run close to full capacity. The 30-year veteran manager we visited with did not believe his company could have handled the current level of demand had it not made the investment in a company-wide ERP system.

The day completed with a tour of our facility, followed by an overview of major project issues and how we planned to proceed over the next few weeks.

August 9, 2005

Checked in with the general contractor for Phase II. The major national control panel builder who originally bid (and later backed out), has now come back with a much improved price. The price difference between their new price and their original price can be justified due to the past year’s price escalation. A change order will have to be submitted and approved by the city council, so it will likely be September before a notice to proceed can be issued for Phase II.

Given the Phase II delays, we are focusing our work on the SCADA master, which is the critical path item in our schedule.

Had a long conversation with a staff engineer for the city regarding the project. It appears he will be given some responsibility for this project. Because he is in the same suite of offices as the city engineer, he has access to inside consultation that we lack. We set a date to meet later this week to review our progress, make a visit to the local data center, and hopefully get a few decisions made to keep us moving forward on the SCADA master.

The SCADA master planned is a three-computer redundant configuration. Computer hardware includes a Dell data center with redundant power supplies, network connections, and mirrored RAID SCSI drives. Two of the servers are responsible for gathering field data from the remote RTUs using Software Toolbox’s Modbus OPC power (TOP) server. Each field RTU is equipped with an Airlink Raven CDMA cell modem. These modems are each assigned a static IP address on the Verizon wireless network. The TOPS server encapsulates Modbus messages within TCP/IP packets and sends them via the Internet to Verizon. The Airlink Modbus capability dis-assembles packets and sends them out the modem serial port to the RTU. The process is reversed to complete the RTU poll. This IP-based design benefits from improved throughput versus traditional sequential polling schemes, allowing up to 16 Modbus RTUs to be polled simultaneously.

OPC server redundancy is managed using TOP server manager. This software resides on each server, automatically switching primary and secondary OPC servers in the event of a communications or computer failure. The Web-based graphical operator interface of Indusoft’s 6.0 Web Studio is also resident on these two computers. Web Studio’s redundancy design allows the primary or secondary Web Studio servers to log historical data to a third SQL server computer. If the SQL server has an outage, Web Studio writes historical data to its local hard disk. Once the SQL server has been restored, locally written data is automatically backfilled into the SQL database so that a continuous history is maintained.

For client side redundancy, Web Studio’s Active-X control automatically switches client Web browsers to the secondary server should the primary become unavailable.

August 2, 2005

A method of easily converting the existing SCADA system’s 8400 I/O tag database was determined. Sample screens and database were created for the engineering review of a typical pump station. No surprises so far; converted easier than expected. Computer hardware submittals completed.

No word back on our requests for information. Will try redirecting to the next in command City engineer.

A pump station electrician met with us this past week and expressed concern about ease of maintenance and obtaining a coordinated set of electrical drawings for all new and existing station wiring. Much of this is within the scope of Phase II so largely out of our control, but we will place this on the project kickoff meeting agenda. The kickoff meeting is still on hold pending the selection of a control panel vendor and issuance of a notice to proceed to the Phase II general contractor.

July 25, 2005

The city’s non-renewal of the overall engineering consultant’s contract is looming as an obstacle to the engineering schedule. The city engineer’s prior time commitments are leaving little time available for him to serve in this capacity. This is resulting in our requests for information going unanswered.

The engineering team completed a site visit to the city’s existing SCADA system. The physical layout of the hardware and communication networks was examined as well as the current distribution of software functions among computers, which varied considerably from what was indicated in the request for proposal documents.

Although the current system employs one of the most popular graphical interface packages used in this industry, Indusoft 6.0 Web Studio was recommended for this application. It has the ability to duplicate the existing system’s functionality while publishing real-time dynamic and animated graphic screens, trends, alarms and reports to standard Web
browsers natively (without additional configuration). It can also be configured for complete data logging and Web interface redundancy, proven in earlier large scale applications.

A major design goal is to move away from the present system of central control room-centric data gathering toward a system where it is possible to grant full or partial SCADA access to any employee via their desktop PC, a cellular-enabled mobile workstation or a Web-enabled cell phone. This includes text critical alarm transmissions directly to cell phones and
pagers as well as automatic distribution of reports via email in Microsoft Excel format.

July 18, 2005

An initial comprehensive project plan for all phases employing QDS Systems project management system was completed this week. Project managers employ a tightly integrated Web-based implementation of Microsoft Project Server
coupled to a Tenrox professional time and billing system. This system allows project managers to specify activities and tasks, task dependencies, resource and personnel assignments in Microsoft Project. Tasks and assignments are automatically distributed to personnel via Tenrox. Personnel update actual work completed daily in Tenrox, which automatically updates the Microsoft Project plan actual progress.

The Phase II general contractor called a meeting with the City engineer. A proposal was offered requesting additional funds to cover the shortfall (discussed in the previous blog installment). The City engineer is making application for approval of a change order to go before the City Council.

Renewal of the city’s contract with the project overall engineering consultant has been placed on hold, requiring QDS Systems to directly interface with the city engineer. The written notice issued last week to the engineering consultant will need to be redirected.

July 13, 2005

Last week we received signed contracts for Phases III and IV. This week we received a notice to proceed from the city of Baton Rouge, which establishes the project start date as July 11th, 2005. The notice stipulates we are allowed 270 days to complete Phase III software and integration and 150 days to complete the Phase IV communication system. The Phase II control panel assembly and installation has been separately contracted for by the city with a contract completion time of 300 days from the notice to proceed.

Phases II, III and IV are largely interdependent. They must start together. Phase III will complete a month or more after Phase II is completed, as equipment must be physically installed before it can be commissioned and field witness tested. Phase II was bid approximately one year ago, so the notice to proceed for this phase was necessarily delayed until the Phase III and IV suppliers were selected, contracted and issued a notice to proceed.

With the project now underway, two significant challenges have developed:

1. Prior to the bid, the general contractor’s electrical subcontractor had received a price proposal for the control panel fabrication from a leading water industry supplier. This supplier later discovered that they had not included all panels and materials required by the specifications and, due to the year’s delay in issuance of a notice to proceed, were within their rights to withdraw their bid. This situation, coupled with material price increases during the past year, has left the general contractor with an estimated $200,000 price problem. A notice to proceed has not been requested pending resolution of this issue.

2. It is apparent from this scenario that our completion of Phase III within the 270-day contract completion time is not within our control, given our dependence on Phase II’s completion. Add to this our contract time clock begins ticking on July 11th while the Phase II contractor appears to still be weeks away from a notice to proceed.

A written notice of this situation has been provided to the city’s engineering consultant and a meeting has been requested of all parties to discuss methods of resolving these issues.

Stan Prutz, P. E.
QDS Systems, Inc.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.