| Internet-Draft | iiot-framework | January 2022 |
| Makhijani & Dong | Expires 20 July 2022 | [Page] |
Industry control networks host a diverse set of non-internet protocols supporting Industrial-IoT and legacy device connections. These networks are physically separated from the enterprise networks and have been slow to adopt the virtualization technologies. Virtualization is necessary to remove the boundaries between the enterprise and process control networks. This document specifies a framework for the converged industrial network. Specifically, it focuses on the virtual PLC scenario. It covers transition technologies required for the convergence of industrial devices with the enterprise application endpoints.¶
This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.¶
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.¶
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."¶
This Internet-Draft will expire on 20 July 2022.¶
Copyright (c) 2022 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
There is a little cross-over between the network technologies used in the Operational Technology (OT) and Information Technology (IT) environments as the architecture of industrial networks has evolved independently from the IT networks. While industrial networks focussed on deterministic communication, safety, and reliability of process control, they have trailed behind in the adoption of virtualization capabilities. Virtualization is necessary for the industry automation to enable compute and data-intensive services at scale. Moreover, it is also a necessary tool for the convergence of OT and IT systems.¶
Although the virtualization of SCADA and other systems (HMI, MES, Historian, etc.) has happened, the low-level PLCs have remained on the factory floor. Virtualization of PLC is a topic of great interest for fully automated and remote operations in the manufacturing and similar process control industry because it allows direct control of low-level processes from the applications.¶
This memo studies the overall network requirements for support of virtualized PLCs and identifies their characteristics. The benefits include:¶
Enabling PLC virtualization imposes a set of challenging requirements. Broadly they can be viewed from¶
This document presents the baseline industrial architecture in Section 3 and demonstrates that the current hierarchical architecture poses difficulty for the adoption of PLC virtualization in industrial networks Section 4. Section 5 further develops approaches to the support for virtualized PLCs.¶
A distributed conceptual model that could potentially address the above limitations is presented for discussion as a converged industrial-network architecture Section 6. Finally, a summary of requirements is extracted in Section 7.¶
This document discusses those requirements and proposes a path to converged industrial networking.¶
The industrial control networks are interconnection of equipments used for the operation, control or monitoring of machines in the industry environment. It involves different level of communications - between field bus devices, digital controllers and software applications¶
Mechanisms that enable machine to machine communication by use of technologies that enable automatic control and operation of industrial devices and processes leading to minimizing human intervention.¶
Todo¶
Todo¶
Industrial computers/servers for the control of manufacturing processes such as assembly lines.¶
Software System to control industrial processes and collect and manage data.¶
Systems of sensors and controllers that are distributed throughout a plant.¶
Systems that connect production equipment across the factory floor, or multiple plants or sites.¶
Operational Technology field devices include valves, transmitters, switches and actuators etc.¶
The term introduced in this document to represent a converged view of OT and IT networks. Virtualized PLC (vPLC):¶
A software component of PLC, in which the control part of factory devices is decoupled from the I/O component. With vPLCs, the I/O stays local to the machines (sensors, actuators, and drives), while the controller logic lives as a software service implemented over RT- hypervisors.¶
The physical network architecture for process control and operations centric networks as shown in Figure 1 is rigidly hierarchical. Note that the figure is over-simplified and in general, each level will have additional hierarchies to extend networks for scale. For example, a PLC that is controlling a group of fieldbus devices may be controlled by another PLC controller which runs ProfiNet protocol. In such cases protocol translation gateways are needed. Between these gateways there may exist a set of intermediate network switches to extend the range (physical distance) and scale (number of devices) of connectivity on the factory floor. Similarly, system integrators also need a variety of translation gateways to extract and integrate data from field devices as an input to MIS, HMI and other enterprise applications.¶
The hierarchical architecture comprises of security oriented zones that are combined together to represent ICA-95 model (or Purdue model see Appendix A) in which each zone further contains well-defined levels. The communication across the zone tends to get complex as each zone runs over a different technology. Among the three zones (Manufacturing, IDMZ and Enterprise), the enterprise zone network is all IP while manufacturing and IDMZ network on factory floor are a combination of IP and Industrial protocols. IP-based routers are used in manufacturing-zone when there is need to extend the network across different cells on factory floors. There are a large number of IP based firewalls and gateways that perform translation in IDMZ but are required to look at transport payload to determine the industrial protocols and corresponding matching rules.¶
Higher layer applications directly interact with PLCs for send and receive commands and data from the field devices. The data generated by sensors is transmitted by PLCs to industry control systems (SCADA, HMI, MES). Both DCS and SCADA systems collect data from process instruments and respond with commands to the actuators. They control several process control loop instances simultaneously to handle complex processes. Originally, such systems were built using proprietary hardware and software, operating in its own zone without additional connectivity. Operators had to work from centralized control room because these systems did not support remote access. The best practices for data delivery to other systems was in the form of reports, which caused significant time lag.¶
+-+-+-+-+-+-+
^ | Data Apps | External business logic network
: +-+-+-+-+-+-+ (L5)
: || ||
v +-+-+ +-+-+
|IDS| |FW | Translation gateways and firewalls
^ +-+-+ +-+-+ -----+ (L4)
: | |
v +-+-+-+-+-+-+ +-+-+-+-+--+
| vendor A | |vendor B | Interconnection of
| controller| |controller| controllers
^ +-+-+-+-+-+-+ +-+-+-+-+-+- (L2-L3)
: | |
: +-+-+-+-+ +-+-++-+
: | Net X | | Net Y|
v | PLCs | | PLCs |--+ Device-controllers
^ +-+-+-+-+ +-+-+--+ | (L1)
: | | |
: +--+ +--+ +--+ |
v | | | | | | + Field level devices
+--+ +--+ +--+ (L0)
I/O devices (sensors, actuators) generate a large volume of data and also accept process control commands. For an application to handle a variety of low-level protocol translations can be extremely challenging, therefore, solutions such as OPC-UA [OPC_ARCH] or common messaging broker mechanisms MQTT [MQTT_SPEC] are deployed. While OPC-UA is a common representation of data collected from different I/O devices; MQTT is messaging service designed for systems with low resources. Both are higher-level protocols that are generally transmitted over TCP as shown in Figure 2 below.¶
+--+ +--+ +---+
(I/O) -->| | ---->| |-------> | |
+--+ +--+ +---+
PLC OPC server/ OPC clients/
(pub/sub) MQTT broker MQTT subscriber
As is evident from the ICA-95 model (described in Appendix A), the business applications are centralized in the enterprise networks. In this network architecture, with PLCs virtualized, they may be colocated at the edge of manufacturing zone, with the supervisory control systems, or in Enterprise zone along with business applications.¶
Alternatively, virtualized PLCs enable new capabilities such as utilizing cloud to edge-aware network architectures by flexible placement of applications and allocation of resources to different components in industry control systems. Virtualzed PLCs serving from the edges can meet latency constraints to close the control loop for process automation.¶
A physical PLC is generally associated with a few I/O devices and is directly connected. The I/O modules are not required to authenticate or perform any verification on connection. As virtualized PLC may be on the other side of the network, the I/O device requires an authentication mechanism to connect to the PLC. This is necessary to maintain reliability and safety of the system and prevent unauthenticated PLC to interact with the software.¶
Virtualiaton allows consolidation of compute, storage, and network resources, as well as independence from custom hardware. The magnitude by which compute capability is improved allows a single virtualized controller to handle more complex and faster scan cycles.¶
Note: A scan cycle is generally the time taken to read the inputs, execute the program (e.g. ladder logic), and update the outputs. The actual scan time is affected by the processing speed of the PLC, the size of the program, the type of instructions used in the program. Therefore, in virtualized PLCs general-purpose processor speed and the amount of memory available is much higher than most physical PLCs.¶
Then, the performance of the network to handle communicaton delays, packet formation, processing and forwarding overheads become critical to overall system performance. Additionally, use of edge-compute platforms is expected for both consolidating resource consumption and lowering the operational costs.¶
Another difference between physical and virtualized PLC is that with virtualization of PLC, a single controller can communicate with a different group of I/O devices over one or more non-internet protocols such as Modbus, Profibus, CANbus, Profinet [SURV], etc. Each of the protocols specifies its packet format.¶
The benefit is a reduction in the number of controllers, but the requirement challenge is to provide a standard communication format for different I/O devices. Since it is not feasible to communicate packets in native (Fieldbus protocol) format due to address scale limitations (field bus devices have limited address space up to 256 devices), a network element or an end-device is required to perform some format translation.¶
As an example, a factory floor is composed of different cell sites. A set of PLCs controls I/O modules or machines in each cell. Traditionally, when there is an inter-connection requirement between two or more cells, the protocol translation is carried out between the cells. With physical PLC, the translation would be done at the controller, but with virtualized, it may require translation capability on the network element connecting to I/O devices or the devices themselves. Moreover, additional deployments are not integrated with the existing network, creating a new network.¶
The fieldbus devices are serial buses and identify PLC as a device with a specific bus address. In the large scale network and in the application layer this much information is insufficient. It may be required for virtualized PLC to support dual addresses, one exposed for the I/O module and other for IT applications.¶
Moreover, as an exmaple, it is no longer sufficient to indicate basic address 0x14; it may require to specify 'device 0x14, of cell - C1 and factory floor, F1, PLC bus address 0x1 in communication path. The reachability to a specific I/O module should have complete information from virtualized PLC.¶
The fundamental paradigm of security as expressed in ICA-95 architecture changes with virtualized PLC since the PLCs are now moved away from the local manufacturing zone. The zone based security design considerations can employ either or both of the approaches:¶
When zone-based secuity rules are leveraged, location-specific security policies may be more coarse grained. For example, an ingress firewall rule will be required to verify and authenticate that the source site is permitted to send traffic for specified destination.¶
This section conceptualizes a fully virtualized industrial control system in which all the components - network, compute, storage, and applications run on virtual platforms to better understand the gaps and requirements.¶
Virtualization enables the separation of software from hardware. It is the foundation for disaggregating different system components such as operating systems, management tools, service logic, and data.¶
In the case of industrial control systems, by virtualizing SCADA, MES, and HMI, etc. [VPLC_CONV], these softwarized systems are run on commodity hardware or general-purpose CPUs. Main benefit of virtualizing supervisory and control systems is that the overall cost can be controlled since specialized hardware is not required, while operators can perform software upgrades more frequently. Such virtualized software can be placed anywhere, often close to the source of data it needs to process. This, in turn, leads to leveraging edge compute networking for multi-site integration.¶
While applications and services are beginning to get disaggregated, PLCs' virtualization is very early stage. Conceptually, a virtual PLC means that the controller functions are separated from the I/O modules of the devices.¶
Similar to SCADA, MES, HMIs, virtualized PLC may be located anywhere in industry control architecture. However, expanding beyond a factory cite, requires special security considerations discuss in Section 4.5. Adding new virtualization capabilities may require and overall redesign of the network infrastructure which may not desirable in all the cases specifically, from the perspective of maintaining same level of security, reliability and safety requirements.¶
Therefore, we envision that the following different approaches are possible:¶
This is the basic approach with minimal change and minimal impact. A PLC software is virtualized and runs on a commodity hardware supporting legacy interfaces to I/O modules. This type of change is isolated to a specific PLC functionality and the only benefit is use of commodity hardware. Potentially, there is a one to one replacement of physical to software PLC.¶
In this approach PLC is virtualized and can run on a commodity hardware as above; additionally providing a clear separation between hardware and software components by relying on protocol translation gateways as part of the network edges that connect to I/O modules conceptually represented in Figure 3.¶
.-,,-. fieldbus
+-+ .-( )-. IP _______ i/f
| | ---->( network )------->[_______] ------> |==|
+-+ '-( ).-' I/O I/O device
virtualized '-.-' gateway
PLC
Depending on the compute capabilities of the hardware, different instances of virtualized PLC may run simultaneously or a group of PLCs are bundled together in a single instance of virtualized PLC. Furthermore, a virtualized PLC maybe hosted on the same hardware along with SCADA or ICS components.¶
There are two new concepts that will need to be formalized: - The I/O translation gateways are a new component in ICA architecture. These are interface translators on an edge network element devices that perform conversion of network side PDU to device side PDU. - identification of the virtualized PLC as discussed in Section 4.4.¶
The incremental benefit beyond the use of commodity hardware is the ability of encapsulating complex logic in a single instance. A clean separation between PLC logic from I/O module allows changes to PLC logic and I/O devices independently. Since the location of virtualized PLC is with in manufacturing zone there is no impact on the security design.¶
This is the eventual goal to support virtualized PLC in a location independent manner. All benefits considered in Section 5.2.2 apply with an advantage of leveraging third-party edge-compute infrastructure as a tenant.¶
However, security zones are impacted as discussed in Section 4.5.¶
Since a virtualized PLC now looks like an IT-centric software component with OT-specific capabilities, the industrial network framework should evolve accordingly to handle virtual entities in the network. This is referred to as a converged network architecture and its conceptual model with significant functions, components and interfaces are discussed in this section.¶
The foundational concepts of converged industry network architecture has three design principles (i) Ability to virtualize end-points, (ii) Disaggregation of I/O Devices, and (iii) Converged industry-network fabric to support communication between virtualized endpoints and I/O modules.¶
Figure 4 represents a converged network fabric (bottom-left) that enables the transfer of data between software system components (top-left) and the physical devices (bottom-right). The fabric is a shared network infrastructure that allows all the characteristics required in the industrial control networks, such as deterministic, low-latency, and real-time communications.¶
In Figure 4, "B. factory site" represents one or more cell (locations) of the physical devices. Each cell group belongs to one physical site, and there can be multiple such sites. A cell site may be the smallest network in this fabric.¶
"A. Software components" emulate different OT and IT functions hosted in a cloud-like environment which is distributed, i.e., components may be located at various sites or at the edge to support low-latency, deterministic applications. Both field and software components are connected over a converged network (shown as "C. <network"> in Figure 4), which presents a unified view of the network infrastructure interconnecting software and field components.¶
The converged fabric is composed of 3 types of network elements with specific roles.¶
The application or service nodes that get virtualized and softwarized maybe instantiated anywhere in the Industry network independent of the I/O module placement. They are placed in cloud or at the edge or the factory floor itself depending on the usage and type of application. From the communication perspective, these nodes following the state of the art in IT could use technologies and protocol such as IPv6 addresses [RFC8200], and service chains [RFC8300] for steering between different service nodes. Their interface to network will have specific transport requirements (when not transmitted as overlay).¶
Network nodes that form the converged fabric and understand the traffic flows between I/O modules and applications. CIFRs are all expected to run a uniform suite of protocols. In a much simplified view the fabric maybe a core IPv4 or IPv6 based forwarding plane, regardless whether overlay technologies (such as VPN, VxLAN etc.) are used). Potentially interconnected over different physical media technologies (commodity or TSN) Ethernet.¶
CIFRs also perform functions for WAN interfaces and multi site interconnections. The routers performing this function will be at the upper edge of the physical network.¶
These gateways are the lower-level edges of the converged fabric.They are very similar to the existing PLC gateways that are purpose-built for translation between fieldbus/fieldbus or fieldbus/IT protocols. In contrast to traditional gateways CIIGs could be stateless and optionally provide more secure control to I/O device.¶
In the existing deployments, components such as HMI, Historian, MES, and SCADA systems run on dedicated hardware. Virtualizing these system components can be consolidated on a single general-purpose hardware platform, reducing the number of hardware devices and improving the security of data exchanges among these systems.¶
A virtualized PLC decouples controller logic from the I/O component. It allows integration of supervisory and control software components as part of the execution environment by leveraging mature IP-based technologies.¶
Although an exploratory work [VPLC_CONV] and [VPLC_IIC] propose I/O field-buses to be replaced with the high-speed, deterministic media (such as Ethernet). The legacy systems (such as serial fieldbus interfaces) will continue to exist in the foreseeable future. Thus, the architecture must support the communication between the field-bus I/O and PLCs, even when the PLCs are virtualized. This implies that in some cases the fabric will still need protocol translation gateways on cell sites, but they need to be close to the the I/O modules i.e. at the edge of the converged fabric.¶
Component virtualization enables co-location of different service functions on the same hypervisor or entirely at a different location regardless of what security zone they belong to. The constraints aware path chain can be set using [RFC7665]. Moreover, it provides multiple service function chain to support different applications. This type of architecture along with NFV [ETSI_GS_NFV_003] can be extremely resource efficient.¶
Several sensors emit time-series data, that can add to the bandwidth consumption to the information going to the cloud. Deploying big-data application closer on the edge and scale them on-demand provides a sophisticated tool to disaggregate processing of sensory data and summarize for the cloud-enterprise applications.¶
................................................................. . A. <CIN - software components> . . +----------------------+ . . | Application Nodes | . . +-+--------------------+ | . . | SCADA/MES Nodes | | . . +-+--------------------+ | | . . | virtual PLCs | | | . . | +-+ +-+ +-+ +-+ | | | . . | |N| |2| |1| |0| | | | . . | +-+ +-+ +-+ +-+ | | | . . | +----------------+ | | | . . | | O/S Layer | | | | . . | +----------------+ | | | B. <factory site> . . | | |RT hyperV | | - + +----------------------+ . . | +----|-|--|------+ |-+ | Cell Group N | . . +--------|-|--|--------+ +-+--------------------+ | . . | | | | Cell Group 2 | | . . | | | +-+--------------------+ | | . . v v v | Cell Group 1 | | | . . +----------------------+ | | Fieldbus Devices| | | | . . | CIG-Router 1 |<-x->| +(@)-(@)-(@)-(@)--+ | | | . . +-+--------v--|-------+ | | | | | | | | | . . | CIF- Router X | | | +----------------+ | |-+ . . +-+-------------v-----+ |<--yy->| | I/O Modules | | | . . | CIF-Router 1 | | | | +----------------+ |-+ . . | +-----------------+ |<---zz-->| | . . | |C o n v e r g e d| | |_+ + ---------------------+ . . | | F a b r i c | |_+ . . | +-----------------+ | . . +---------------------+ . . C. <network> . .................................................................
A manufacturing facility can be located at more than one site and each site is further divided into cells. Further the machinery, actuators, sensors are associated with the cell connected to the PLCs. These controllers run different protocols such as Ethernet, RT-Ethernet, Modbus, ProfiNet etc.¶
In a vPLC supported environment, the I/O cards are responsible for media access conversion from in and out of the converged fabric. Even the support for legacy PLCs is similar to vPLCs, with the role reduced to only translation function.¶
Converged fabric shown as "B." in Figure 4 is central to the architecture. The connectivity is largely Ethernet based (except I/O device interfaces), potentially running IP protocols on the switches and routers in the network.¶
Since this is a logical fabric, the connectivity is local on a factory floor and can be extended to multiple sites. Interconnecting different sites will use WAN functions. The fabric breaks the hierarchical structure and topology can now be designed as fat-tree (or leaf-spine) network which provides overall more number and multiple deterministic paths between two end points.¶
A key characteristic of legacy Industry networks is that they do not require frequent changes and therefore, topology changes are not dynamic. The fabric could potentially use a combination of software-defined connectivity with IP routing protocols. The routing protocols will maintain the infrastructure reachability among the network nodes and software-defined solutions will manage flow of traffic in a deterministic manner addressing the low-latency and deterministic data delivery of certain type of flows.¶
Device functions and operation does not change. The requirements here are related to how that are reached, identified and discovered in the network.¶
Further motivation and analysis for adapting to OT/IT asymmetric address formats is covered in [I-D.draft-km-industrial-internet-requirements].¶
This document requires no actions from IANA.¶
The architecture at the very least must adhere to the security guidance provided by ICS-95.¶
The International Society of Automation (ICA) has developed a model [ISA95] to describe automated interfaces between enterprise and control systems. In this widely deployed hierarchical model, five levels are defined and they follow a strict ordering of interfaces across the levels. At the lowest level 0, are the physical devices while enterprise applications are at level 5. In between these two levels, there are several supervisory, management, and intermediate data collection applications that provide information to¶
| +-------------------------------+ Enterprise | L5 | Enterprise applications | Security +-- +-------------------------------+ Zone | +-------------------------------+ | L4 | Gateways, servers (ops, mgmt) | IDMZ +-- +-------------------------------+ | +-------------------------------+ | L3 | Supervisory controls | Industry | +-------------------------------+ Security | L1 | Device control | Zone | +-------------------------------+ | L0 |Sensors, Actuators, Robots, etc| (cells or zones) +-- +-------------------------------+
The ICA-95 architecture recommends hierarchy, thereby a separation between factory devices and applications through three different security zones called Manufacturing, DMZ and enterprise zones as shown in Figure 5 as below:¶
The IT applications reside in enterprise networks and perform tasks necessary for business operations such as inventory control, supply-chain logistics, schedule and capacity planning. They need to collect data from the OT systems in order to make those decisions.¶
The OT and IT networks were designed to prevent direct communication between them. The IDMZ serves as an information sharing layer between the IT and OT (L4 and L3) systems. This indicates that additional security rules, inspection and protection of device identity and access is necessary when transiting from L3 to L4.¶
Consists of Levels 0 through 3 site wide production system. Operations at level 3 (L3) Support site-wide view of the production system. They also provide data to L4. Area supervisory control (L2) performs operation and control over a zone or smaller area in a production floor. Each area has specific set of tasks or operations to perform. Basic control at level 1 (L1) is for the actual control of the equipment. The L1 components such include PLCs; they send commands to L0 equipments to perform tasks (e.g. start motor, alter pressure level, or reduce motor speed). Finally, actual process takes place at level 0 (L0). At this level for the process equipments performing actual operations are performed. This include equipment and devices such as motors, pressure valves, temperature, speed, etc sensors, etc.¶
The devices or controllers at level 1 are the ones of specific interest for virtualization and the corresponding challenges are covered in later section.¶
The paradigms of networking in OT are quite different than IP based best-effort networking protocols. Yet, IETF protocols are extensively used in OT applications. Often, it is not possible to get contributors directly from the OT sectors, then it would make more sense to coordinate with well-established consortia where OT scenarios and requirements are is discussed may be utilized. Two well established foundations are IIC [IIC] and OPC-UA [OPC]. For example, a [IIC_TALK] provided overview of IIC activities.¶
Industrial IoT Consortium (IIC) provides use cases, scenarios, and best-practice frameworks to solve specific problems and solution pain points. It is a rich resources of case studies and demonstrations of different test beds. The IIC itself is not involved in standards development, but may help in formalizing requirements, further insights into solutions developed in IETF, and potentially help adoption of those solutions.¶
Open Platform Communications-Unified Architecture (OPC-UA) provides interoperability across different hardware platforms using a standard data model. It standardizes various information models, corresponding client-server architecture and defines necessary access mechanisms to those information models. The OPC-UA is an abstraction layer to provide common interface to different data look-up and event notifications. A number of information models are provided by OPC-UA can be found here [OPC_INFO]. Foe example, OPC has a specification on PLCs. It abstracts PLC specific protocols (such as Modbus, Profibus, etc.) into a standardized interface allowing HMI/SCADA systems to interface with a middleware that converts generic-OPC read/write requests into device-specific requests and vice-versa.¶
Note: OPC-UA information model similar to YANG?¶
IETF solutions will focus on leveraging or extending IETF technologies for IT and OT integration which is at the infrastructure or communication layer. Thus, providing protocols that could potentially benefit higher-level OPC-UA work.¶
Both IIC and OPC could provide guidance to the lower level work.¶