Short timeline for automation infrastructure

It took six months to build a complete supervisory control and data acquisition (SCADA) system for an oil pipeline along with hardware upgrades, a new network and a control room built from scratch. It was a daunting task for ARB Midstream, but with help from system integrator Industrial Networking Solutions (INS), the project was a big success despite the tight schedule.

ARB Midstream, based in Denver, is an independent, growth-oriented company that provides midstream and marketing solutions for crude oil, refined products and liquefied petroleum gas (LPG). INS is an industrial operations technology (OT)-focused system integrator based in Richardson, Texas. The company distributes products, offers technical support and provides information technology (IT) services for wired and wireless machine networking applications.

When ARB purchased the Texoma crude oil pipeline system in Texas and Oklahoma, the agreement called for ARB to begin operating the pipeline in six months. The system had 37 sites, including 25 truck injection stations and eight truck terminals. It included more than 900 miles of active pipeline, and 950,000 barrels of storage.

“In the process of acquiring these assets, we knew we would need to build a control room and have a proper SCADA system,” said Jerod Blocker, OT manager for ARB Midstream (see Figure 1). “We needed to build a SCADA system in six months — which we were able to accomplish using a blend of different newer technologies, including cloud computing, software-defined WAN and edge computing.”

Industrial Networking Solutions believes its architecture for this project sets a standard for future SCADA projects in the Industrial Internet of Things (IIoT) market. The software-defined wide-area network (SD-WAN) connects networks over large geographic distances. INS also delivered a cloud-hosted SCADA solution with management, visibility, control, reporting, edge computing, message queuing telemetry transport (MQTT) and store-and-forward capabilities.

“This application uses best-in-class vendors,” said Mo Moore, manager of software services for INS. “We have Ignition by Inductive Automation for SCADA, Cradlepoint for cellular communications and software-defined networking, Cisco for managed networks, Moxa industrial computers, Cirrus Link Solutions for MQTT and Amazon Web Services for cloud-hosted services with redundancy.”

“We wanted cloud computing,” Blocker said. “We didn’t want to build a traditional infrastructure. We chose to spend money on the software and on the edge compute. And we wanted a local OPC UA server with the ability to leverage multiple protocols.”

With cloud computing, ARB is paying less than it would have for co-location. The cloud also enabled fast creation of the needed infrastructure

Out with the old

The operating agreement was for six months, from October 2018 to March 2019. As soon as the transaction was finalized, work began on reversing the direction of the pipeline. For the project overall, some key hurdles to clear were wetter-than-usual weather, and less-than-thorough documentation provided by the selling company. Work progressed, however, and ARB took over operation of its first site on February 23, 2019. By March 29, 2019, all sites were being operated by ARB — one day prior to the six-month deadline (see Figure 2).

“ARB approached INS to look at replacing some older VSAT equipment and maybe even some older cellular equipment at some of its remote locations,” said Dave Brewington, services manager for INS. “We also had the requirement to come up with a secure, reliable and scalable network that would allow ARB to grow and to acquire additional remote assets. Cradlepoint’s NetCloud Perimeter allowed us to create an abstracted overlay network built on cellular technology. That allowed us to have fully secure and encrypted end-to-end communications between remote assets and the cloud.”

The overlay network uses the 172 IP address range on top of the SD-WAN’s IPv6, creating a large pool of IP addresses that won’t conflict with each other. It also simplifies network management and eliminates the need for people to memorize all the local IP addresses. The new network is also carrier-agnostic, enabling it to use a mixture of copper, fiber, cellular and satellite for connectivity. The system chooses the best connection.

MQTT all the way

“Another key success factor to the project was the implementation of the MQTT Sparkplug B protocol,” Moore said. “The publish/subscribe model really cut down on cellular bandwidth and utilization of the data plans while providing an extremely fast and responsive human-machine interface (HMI) environment, which really impressed ARB. So, the retirement of the VSAT solutions and going to the MQTT model over cellular was a much better approach, and it really helped ARB implement a secure, robust, cost-effective and scalable infrastructure.”

MQTT uses little bandwidth, and it reports by exception, which can be a big improvement over traditional polling. “On the first day, we saw a 35% difference in data-plane utilization, because we weren’t sending up tags that we didn’t need to send up,” Moore said.

On this project, a weak cellular network meant even basic communications were having trouble getting through. The data from MQTT seamlessly made it all the way up, very quickly. “With Cirrus Link’s MQTT setup, the lightweight protocol to get the data to the cloud, or to your control center, is amazing. It really is a huge, huge benefit,” Moore said.

Quick control

Creating the operations control center was another big test for INS and ARB. But the team pulled it off, and ARB is very happy with the result. “One thing I really love about this system is that we started this control center from scratch — and we built it and had it operational in three and a half months, which is pretty incredible,” said Tom Charboneau, control center manager for ARB (see Figure 3).

Figure 4: INS and ARB agreed early in the process on graphics, colors, and more — saving valuable time later. Courtesy: Inductive Automation[/caption]

With Ignition’s unlimited licensing, there were no extra costs for increasing the number of tags, users or devices. “With this SCADA system, the costing structure was not a tag-based structure as traditional SCADA systems are,” Blocker said. “That was definitely a big, big decider in going with this solution. And the technology it has with the edge computing and MQTT and all the different bells and whistles was icing on the cake, especially for the price.” The system has more than 50,000 tags, with ARB adding to that as more systems come online.

“We now have a fully scalable SCADA system that’s redundant, secure and has the ability to expand with our future growth,” Blocker said. Moore pointed out additional benefits for the future. “ARB now has data collection for machine learning and deep analytics that will really help their business process moving forward,” he said.

Moore said, “The diverse technologies that were implemented as part of this project — from the SD-WAN, to edge products using MQTT, to SCADA in the cloud — really are the gold standard for companies running remote assets or even plant-to-plant technologies in manufacturing. It’s really where people should be going in the future.”

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our CFE Media editorial team and getting the recognition you and your company deserve. Click here to start this process.