Prototyping peer-to-peer applications for the industrial Internet of things

Programming peer-to-peer (machine-to-machine) architecture is challenging for the broad application space of the industrial Internet of things. A unified programming environment provides an intuitive method for publishing and subscribing to data of interest over the network.

By Robert A. Dolin and Bernd Gauweiler April 23, 2014

The industrial Internet of things, IIoT, is a very broad application space. The envisioned applications can be divided into three main architectures: A client/server architecture where a human, acting through a device (such as a mobile phone), interacts with the Internet, a machine-to-machine (M2M) client/server architecture where remote devices feed data to enterprise systems for analysis and decision support (such as a vehicle tracking system), and a peer-to-peer (machines-to-machines) architecture where the devices autonomously cooperate and make local decisions (only on exception or periodically connecting to an enterprise system to post alarms or significant data). These peer-to-peer applications often perform local control of a machine or process in response to local data. This programmatically challenging class of applications can be difficult to prototype.

The various protocols that make up the suite of Internet protocols are well suited to the client/server architectures, as nearly all Internet traffic today follows this architectural model. An area of our interest [1], [2] is applying a peer-to-peer architecture and its required services above the UDP transport layer of the Internet Protocol (IP). The Python environment has an easy and intuitive method for expressing distributed functions that publish data and subscribe to data of interest over the network. The environment hides the aspect of communication from the application-making the peer-to-peer, distributed nature of the application implicit. The application program only needs be concerned with the data it manipulates-much as a noncommunicating application. Diverse applications that are compliant to this architecture can be prototyped in very little time.

I. Python environment

A simple library of classes turns a Python script into an IIoT device, and elevates a locally operating control application to a networked, IP-based solution. The library is designed for ease of use, centered on a single, main class. This application class and its subclasses are generally self-provisioning by computing useful default settings at initialization time. Application programming interfaces (APIs) are available to fine-tune the provisioning parameters to meet a specific application’s demands where necessary. The design cycle is Python-centric and does not generally require any other expert tool. Information between the nodes on the network is exchanged via a publisher/subscriber data model where the unit of exchange is a datapoint. Datapoints are values with semantics. (See more details below.)

The Python script defines datapoints to interface to the network: Input datapoints obtain data received from the network, such as a temperature setpoint. Output datapoints communicate data to the network, such as the current thermostat position. Input and output datapoints implement datapoint types, supporting different physical or abstract data, such as temperature values or alarm conditions.

Additional provisioning data is provided with properties. A property is implemented as an input datapoint with persistent value storage. Properties expand on simple datapoints to include specific semantics: the upper alarm temperature threshold, the temperature alarm hysteresis, or similar descriptions.

Input and output datapoints and properties are generally grouped into profiles. Profiles are a collection of datapoints that form the network visible interface for a service, combined with well-defined behavior. In the example discussed, the service is a carbon dioxide sensor. The role of the service is to publish the amount of carbon dioxide present in the air. The frequency of this publication is determined by the properties embedded within the profile. The Python library allows implementation of a complete profile, including all related datapoints and properties, through a block object within one simple line of code:

co2Sensor= application.block(co2Sensor())

Definitions of datapoint, property and profile can support and promote solutions featuring devices supplied by multiple vendors. Building on existing standards [6] and a rich collection of type and profile definitions [7], compatible devices can recognize each other and exchange data across different suppliers or hardware solutions. 

II. A generic data model

Peer-to-peer control networks must handle data that arrives in a nondeterministic schedule, representing a diversity of physical, logical, or abstract entities. For example, one data item might represent a temperature value in Celsius, encoded in an IEEE 754 double-precision floating point value in big-endian byte order. Another data item might represent the number of beans in a bag using a natural number ranging 0 to 999, encoded in an unsigned 16-bit little-endian scalar, while other data might represent the complex definition of a lighting scene through a suitable structure of data, or a postal address in human-readable alphanumeric form using an UTF-16 unicode.

Within data representing a known entity, such as a temperature value, and using a known encoding, such as an IEEE 754 double-precision floating point value in big-endian format, control networks must also recognize semantics: a room temperature setpoint might have a valid value range between 15 and 28 Celsius, and is applied to a control algorithm that drives a thermostatic radiator valve, while another temperature value, encoded in the same manner, might represent a fire detector’s hysteresis, or define the lower valid value for the room temperature setpoint.

Data must then be conditioned such that the local application can operate with the data. For example, the application may need to rotate the byte order of the 8-byte floating point value upon receipt and interpret the result as a double float value.

When the algorithm produces output data, this data also may need conditioning. For example, a current temperature value might be reported. This might be exchanged over the network as a big-endian 16-bit unsigned integer value, implementing a fixed-point variable ranging from 0 to 655.35 with a resolution of 0.01 C. This programming model conditions this data such that the application is presented with data in its natural form; a floating-point value in this example. Range and resolution limitations are automatically applied by this model, and transcoding between the application presentation and the network presentation of this data is handled transparently.

Peers in the control network must agree on the entity described by each data item, its encoding, and its semantics. Additional aspects (such as explicit range limitations, well-defined default values, or fallback behavior in the event of a fault) are also commonly agreed upon between peers.

To reach such an agreement, peers in a control network either carry all metadata required to enable receivers to fully understand any data they might encounter, or ensure that data only reaches receivers with a previously established contract and level of trust.

This poses a significant difficulty to the device developer and the person integrating multiple control devices into one distributed, peer-to-peer control network. The device developer generally needs in-depth knowledge in the processes, protocols, and procedures involved to write code that correctly conditions output data, and correctly identifies, classifies, and interprets incoming data. The person integrating multiple devices into one such network needs to understand each device’s attributes and capabilities to combine only those devices that "speak the same language." For example, a generic switch may be able to control a generic lamp, but an industrial washing machine may require a more complex "on/off" signal than what the generic switch can provide.

This programming model includes methods and services that automatically evaluate and apply metadata to data interpretation and data conditioning processes, interfacing the application with appropriately conditioned data only. These methods greatly improve on existing technology by providing a transparent comprehensive data conditioning service.

Figure 1 illustrates the data conditioning services and network data flow provided by this programming model.

III. Data addressing

A peer-to-peer network consists of a number of independently operating devices, generally sharing data on a need-to-know basis. For example, a thermostatic radiator valve may need to receive the current room temperature setpoint from some other device. The same thermostatic radiator may not need to know when or why the setpoint changed, and it will not normally need to know about any control operation on the ceiling lights or the garden sprinkler.

That is, a new temperature setpoint value might be communicated from an input device to all thermostatic radiator valves in the same room. At the same time, other rooms’ thermostatic radiator valves may receive other setpoint values from other sources, while the garden sprinkler is controlled as a logically separate unit on the same network.

To facilitate peer-to-peer communication in this manner, devices transmitting data (data publishers) need to know how to reach the applicable targets (data subscribers), and may need to know whether and how to accept or reject incoming data. This mutual knowledge includes knowledge of source and/or destination addresses, information to select the correct datapoint from multiple datapoints defined on the receiver node, routing details, and transport properties (such as service types, the time allowed to detect an outage or enable retransmission, and similar control data).

This programming model includes abstract services to facilitate contract offers and acceptances in the terms of the application script, concealing the required low-level details within the implementation of this model. This simplifies development of distributed control applications. By supporting intrinsic contract management services, this programming model also greatly reduces the need for, and in many cases eliminates the need for, dedicated integration steps when forming a network from distributed applications.

Figure 2 shows a simple control network featuring four distinct devices, each contributing one application. The figure illustrates the set of low-level addressing details required to establish and maintain such a contract, and the realm of application expertise.

IV. Device addressing

Peer-to-peer control networks implement distributed algorithms based on contracts and established trust regarding the data and its semantics, and the addressing and transport properties so that peer A can reach peer B over the network.

However, each participating device must also obtain its own unique identity: the physical device A must assume the logical identity A, and the physical device B must assume the logical identity B, to enable data flow from A to B.

In some systems, the logical identity is the same as the physical identity: devices address each other through unique physical aspects such as unique MAC-ID addresses. In most networks, the abstraction of logical addressing is preferred, as it allows for device replacement without informing the other nodes of the replacement physical address, is more efficient, and supports network topology composed of multiple links.

Systems using logical addressing must obtain and apply a suitable and unambiguous logical address to each device. In either system, methods are required for devices to address each other using the chosen form of addressing.

This programming model includes services for automatic acquisition and maintenance of unambiguous logical device address data, generally not requiring a dedicated central machine or human involvement. This simplifies installation of devices; the integration of these services with this programming model enables the creation of such devices with little effort by the developer.

V. Programming model

The Python programming language [3] provides a rapid development environment, supporting object-oriented development in an interpreted language with integrated bytecode compilation and optional compilation to native binary code [4], [5].

Use of Python for creation of applications for distributed peer-to-peer control networks enables the use of popular and very fast-to-market application development for an application domain, currently dominated by traditional programming languages and strategies, which as a consequence, generally requires much longer development cycles.

The programming model requires only the Python programming language, as the need for proprietary tools and languages is eliminated, while providing a method, through standard Python language features, to continue to use existing code.

VI. Contract fulfillment

Implementing an application for an open, multi-vendor peer-to-peer control network requires the declaration of the application’s interface, and the integration of the application’s central control algorithms with this interface.

The interface defines how a given application communicates with its peers on the network. Such an interface generally consists of one or more implementations of profile(s), chosen from a library of profiles defined in an open, industry standard. [6]

In traditional technologies, implementing interfaces consists of a largely manual and knowledge-intensive process of providing the correct definitions using an interface definition tool and language. Below, the example continues, looking at the definition and implementation of a hypothetical application’s interface of the simple standard carbon dioxide sensor profile. This simple standard profile combines one output datapoint for the current level of carbon dioxide with three mandatory datapoints for provisioning the frequency and the conditions by which the carbon dioxide level is sent to the subscriber nodes.

Figure 4 illustrates how a particular profile is implemented based on the classes outlined in Figure 3, using the standard carbon dioxide sensor profile SFPTco2Sensor [7] as an example.

The Python programming model includes all standard profiles, datapoint and property types, known collectively as resources. In all, over 800 standard, predefined, resources [6] are available to the Python application. Additional, application-specific resources can be defined within the same Python programming model. Applications import those Python resources when necessary to keep resource consumption by unnecessarily loaded resources at a minimum.

Figure 3 visualizes an example of several resource types which an application using this framework can instantiate. Typical applications use the Application class’ block() factory method to implement the desired object. The Application class is a singleton class provided by this framework, which provides all top-level functions and services.

The following eight lines of Python code implement a complete and fully functioning application, including one correct implementation of the standard carbon dioxide sensor profile. Not included is the application algorithm, which would sample a physical CO2 sensor device through peripheral inputs and report the current values through the output datapoint(s), subject to thresholds and timing constraints expressed in the configured property values. The programmed attribute defined in this code example defines a mandatory key for the application, which is used to identify the application in a network.

from izot.device.application import Application
from izot.resources.profiles.co2Sensor import co2Sensor

app = Application()
app.programId = "9F:FF:FF:00:00:00:01"
co2Sensor = app.block(co2Sensor())

app.start()
while True:
    app.service()

Figure 4 also illustrates the operation of the block factory, using the example of the same standard carbon dioxide sensor profile. Because the protocol stack’s API is single-threaded, the Application object allows asynchronous threads to signal events or request activity in several ways:

The control network protocol stack signals the availability of events through an asynchronous event that the Application class captures and transforms into a service signal. This allows the synchronous processing of events raised by the protocol stack. These events include notification of newly arrived network data, or notification of completion events (failure or success of transactions initiated earlier).

In applications that use concurrent processing, one main processing context (such as a thread) instantiates the Application class provided with this model. Other processing threads of the same application may call essentially single-threaded API. In this case, the API call along with all its arguments is entered into a protected function queue by the thread which made this API call, and the service signal is raised. Enqueuing of API calls in this manner is handled automatically by the Application class’s methods and is transparent to the application and application developer.

When the stack indicates the availability of new data for a given datapoint, this raw application data is collected from the stack, formatted, and forwarded to the application in the manner outlined in Figure 1.

Completion event notifications trigger corresponding events.

The service method empties the function queue by executing all enqueued API calls in the main processing thread.

This programming model supports applications with concurrent processing in different threads for interaction with datapoint objects and their values. Each datapoint object supports the standard Python with statement. While code executes within a datapoint object’s with statement, the object is locked against access by other threads. When the with clause terminates, the object is unlocked and added to a protected to-do queue. The service signal also is raised at this time.

The service method processes this to-do list by applying the appropriate action to each enqueued datapoint object, unless the object is locked. Worker threads can explicitly lock a datapoint object beyond the with clause, although this is not generally recommended to prevent the risk of deadlock situations.

In another step executed by the service method, the interoperable self-installation protocol (ISI) [2], used to organize and maintain the network, is serviced when necessary.

The service routine also refreshes selected dynamic datapoint properties on a time-guarded round-robin basis. The time-guarded round-robin algorithm processes one datapoint after another in the order in which they were created, but does not spend longer than a configurable, typically small, amount of time doing so in each service call. When the next service call occurs, this processing continues where the previous cycle left off. When all datapoints have been processed in this manner, processing continues with the first datapoint.

Some attributes associated with datapoints may change as a result of contract establishment or maintenance, or other network management operations performed by another party. For example, a network management tool may connect one input datapoint and one output datapoint. This procedure manages a number of low-level attributes (some shown in Figure 2). The local application may not be aware of these changes. The local application does not generally require knowledge of these changes, but some aspects are frequently inspected by some applications.

To report whether a datapoint is connected, or bound, to at least one other, this service relays the current state of the is_bound datapoint property from the stack to the corresponding datapoint object, which exposes it through its Boolean is_bound property. For example, an application may use this to enable a timer to monitor the timely arrival of updated input data, and may disable this timer for an input datapoint that is not currently connected.

VII. Data interpretation and conditioning

For transport across the network, data is generally presented in a form suitable for a given technology. For example, XML-based solutions exchange data as human-readable textual information using a suitable encoding, such as UTF-8. Other systems support binary data exchange for efficiency, where data is formatted according to rules defined by the network technology. For example, one network technology may require that all numeric data consisting of more than one byte be exchanged using little-endian byte ordering.

Specific data types often stipulate additional rules. For example, one network data type may be defined as a temperature value in units of 0.01 Celsius, or as a duration, in units of 0.1 s, both delivered using an unsigned 16-bit integer quantity.

Once a receiver has applied the network-specific rules, such as byte and bit ordering, the resulting raw application data is subject to those additional rules, which are generally supplied as part of metadata, or with the contract, which governs reception of this particular data item. Other types implement superficial value range limitations to ensure a match between the network data and the associated physical entity. For example, a temperature value represented by an IEEE 754 double-precision floating point value may be restricted such that the temperature value cannot fall below zero Kelvin.

Using traditional methods, the burden of transforming raw application data into algebraic data useful in computation and other algorithms through application of scaling factors, offsets, and range limitations, and its reversal, is left to the developer.

The Python programming model for distributed peer-to-peer control has built-in access to such metadata, and automatically applies the required transformations when presenting incoming data to the application, and when conditioning outgoing data for transmission onto the network.

Figure 1 illustrates how this programming framework provides such services as a fully transparent service to the application.

In this Python programming model, the application developer does not need to be concerned with data conditioning duties, as the programming model automatically applies the appropriate rules. This makes the technology easier to learn and more approachable, compared to traditional methods, and significantly reduces development time, time invested in staff training, and risk of error in an application.

VIII. Device addressing

Most control networks support an external entity to allocate and assign unique logical device addresses. This model optionally supports such a central arbitrator, but defaults to an implementation of the interoperable self-installation protocol detailed in [2].

The ISI protocol, and the support for it built into this programming model, automatically assigns and maintains unique device addresses to each participating device.

Traditional programming models require substantial code to take advantage of the services offered by a central arbitrator. One example is code that implements a client to the Dynamic Host Configuration protocol (DHCP).

In this programming model, no code is required to accomplish the same. The core ISI services are automatically and fully transparently handled by the Application class and its underlying service providers. Those include starting, stopping, and periodically servicing the ISI engine; reliable maintenance of persistent data storage for nonvolatile, modifiable data required by the ISI implementation; and abstraction of low-level protocol details into the realm of application expertise (Figure 2). 

IX. Contract establishment

To exchange data in a useful fashion within a peer-to-peer network, data exchange is generally based on the transmission of metadata, or a previously agreed contract, or a combination thereof.

In this model, the ISI protocol [2] is used to create, manage, and remove these contracts. While the ISI protocol provides a standard mechanism to do so, in-depth knowledge of the application and the underlying networking technology is generally required. Using a traditional programming model, application developers must supply the definition of a contract offered to others, must define which contracts offered by peers might be acceptable, and must implement the contract and the mapping between the data connection contract and the local application in fairly low-level terms.

With this programming model, application developers can focus on expressions of the contract and contract acceptance terms using an abstraction provided. This reduces required levels of training, development time, and margin of error. Developers using this model are not required to provide low-level, protocol-specific references to datapoint indices or numeric data type identifiers. Instead, developers can express the intended contract in the language of the application itself, for example, by referencing a set of datapoints implemented in the application. Such a declaration can take as little as three lines of formatted source code.

To accept a contract offered by a peer, applications must determine whether the offered contract is acceptable, then instruct the ISI protocol how the contract might be applied to the local datapoints. In traditional methods, this is communicated using low-level aspects of the underlying communications protocol, and thus requires in-depth knowledge of those aspects.

In this model, rules of application of a contract to local datapoint items are expressed using a more abstract model, allowing the application developer to express those rules in the language of the application itself. For example, an application can reference a set of datapoints implemented in the application instead of low-level implementation details.

Similar to profiles, the ISI protocol supports a concept of assemblies. An assembly is an application-specific way of grouping datapoint objects for the purpose of establishing and maintaining a data exchange contract. Any data exchange contract negotiated with the ISI protocol references one or more whole assemblies; as far as these contract services are concerned, an ISI assembly is an atomic entity. In legacy implementations, details of contracts and assemblies are communicated through a series of events, each requesting one low-level detail at a time, such as the global datapoint index for a particular member of a particular assembly. While this implementation is very resource efficient when used on severely resource-limited devices, it is difficult to understand, implement, and maintain.

This Python programming model supplies an abstraction to the ISI assembly and contract services, governed by the Assembly and Enrollment classes. These classes allow the application developer to express assemblies and details of contracts (called enrollment in ISI) through the objects already available to the application. The application developer can, therefore, express those details in the familiar language without the need of exposure to low-level protocol, resource, and datapoint properties (although such low-level access is supported as an option).

The Assembly class maintains a list of lists of datapoint objects, provided when constructing the assembly object.

Assuming four datapoint objects A1 through A4, the following describes the example assembly with the Assembly object’s constructor:

assembly = izot.isi.Assembly(((A1,), (A2, A3), (A4,)))

The code example uses a normalized form of the assembly description, that is, lists (or other appropriate containers) are used to group objects even if they contain just one item. When specifying the assembly, the application developer does not need to provide the normalized form, as single objects don’t need to be wrapped into containers. The following code example yields the exact same assembly shape:

assembly =izot.isi.Assembly(
   (
      A1,
      (A2, A3),
      A4
   )
)

Trivial assemblies containing one datapoint can be created by referencing the datapoint alone. However, examining the assembly property always yields a normalized description:

trivial = izot.isi.Assembly( A )
print(trivial.assembly)# prints ( (A, ),)

X. Beyond a polymorphic API

In traditional programming languages such as the family of C programming languages, data typing is generally strict. For example, when an application interface requires a numeric datatype identifier in a 1…65535 integer value range, such an integer value must be provided, or a value must be provided that can be converted into an integer.

The Python programming language implements a different approach, known as duck typing. In duck typing, data types are not generally validated or converted to an expected type. Instead, data is accepted so long as it can satisfy the required operation. In practical terms, this generally means that data is acceptable so long as it can behave like the expected data (for example, an integer), even though it need not be of that type (an integer).

This allows for intrinsically polymorphic programming. For example, a function implemented in ANSI-C might expect a numeric index into an array to address one of the array’s elements. This index is generally required to be supplied as (unsigned) integer data. In duck typing, any data type may be used for the argument of the equivalent function, so long as the data provided can unambiguously address one element within the array.

This Python programming model supports the duck typing principles but employs run time type identification techniques to supply an additional layer of abstraction and freedom, greatly reducing the time and effort required to learn the model.

For example, the ISI protocol requires a numeric datatype identifier for the definition of a data connection contract. In the C programming model, an (unsigned) integer data type identifier is required to satisfy this requirement. Under the rule of duck typing principles, any data type which can produce such a type identifier when needed is supported. In this programming model, the application can supply the numeric (unsigned) integer data type identifier, or an object capable of producing such a type identifier, such as a data type or datapoint object.

The application generally deals with datapoint objects and, during the initialization phase, with data type objects. The programming model detects when these types are used, and obtains the required information from these objects such that they may be used in the place of a numeric type identifier. The following code excerpt demonstrates the implementation of the expansion on the polymorphic Python API, where the relevant portions are highlighted in bold typeface. The code snippet shown implements the type_id property and the related setter (__set_type_id function) for the Python ISI enrollment object.

def __set_type_id(self, v):
 if self.__created_from_raw:
    raise AttributeError(
                 ‘This Enrollment object is read-only’
              )
 if isinstance(v, interface.Datapoint):
self.__type_id =         v.get_data_item()._key
             elif isinstance(v, base.DataType):
            self.__type_id = v._key
    else:                                
            self.__type_id = self._in_range(‘Datapoint type id’, v, 0, 255)

type_id = property(
                lambda self: self.__type_id,
                __set_type_id,
               None, """Primary datapoint type ID or zero for any.

The primary datapoint is type key, or zero if not specified. You may also assign a datapoint object to this property. The property also accepts a Datapoint or DataType object when assigned, but only maintains and returns the numeric type identifier.
           """) 

XI. Free download for Linux computer

A fully operational version of the Python environment described is available for download at https://iiot.echelon.com/get-started. The download, in source code format, is provided free of charge for academic and commercial use. To use the environment, Linux computers are needed. (We used the Raspberry Pi family, but Beaglebone Black, members of the family of Plug Computers, and potentially others are similarly good candidates.) 

– Robert A. Dolin is Echelon’s chief technology officer and system architect. Bernd Gauweiler is a senior software engineer at Echelon Corp. Edited by Mark T. Hoske, content manager, CFE Media, Control Engineering, mhoske@cfemedia.com.

Key concepts

  • Industrial Internet of things has a broad application space.
  • Peer-to-peer (machine-to-machine or M2M) architecture is a challenging class of applications to program.
  • A unified programming environment provides an intuitive method for expressing distributed functions that publish data and subscribe to data of interest over the network. 

Consider this

How much more could industrial Internet of things applications developers accomplish with a library that allows implementation of a complete profile with one line of code?

ONLINE extra

This is a Control Engineering May issue Digital Edition Exclusive, including additional programming examples and information above and more information and references below.

Robert A. Dolin, Echelon’s Chief Technology Officer and system architect, has worked for the company since 1989. He is the principal or co-inventor of 15 Echelon patents and is the author of several papers on device networking and the Internet. He has a BS degree in electrical engineering and computer science from the University of California at Berkeley.

Bernd Gauweiler, a senior software engineer at Echelon Corp., has a strong background in industrial control and automation. For over 20 years, he has focused on programming languages and tools aimed at programming embedded systems and distributed applications, with an emphasis on interoperable multivendor systems. He holds an engineer’s degree (Dipl-Ing (FH), equivalent to a BS degree) in electric engineering and industrial automation from the University of Applied Sciences in Kaiserslautern, Germany.

References

[1] Dolin, Jr, "Programming Language Structures For Use In A Network For Communicating, Sensing And Controlling Information." U.S. Patent 5,490,276, issued Feb. 6, 1996

[2] Gauweiler et al, "Simple Installation Of Devices On A Network." U.S. Patent 8,374,104, issued Feb 12, 2013

[3] van Rossum et al, the Python programming language, https://python.org

[4] Bradshaw, Behnel, Seljebotn et al, the Cython open-source project, https://cython.org

[5] Rigo et al, the PyPy open source project, https://pypy.org

[6] LonMark International, standard resource documentation, https://types.lonmark.org 

[7] LonMark International, CO2 Sensor, https://www.lonmark.org/technical_resources/guidelines/docs/profiles/1070_10.pdf