OT Networking Reference Architecture usa.siemens.com/industrial-networks © 2023 by Siemens AG, Berlin and Munich Version 2, 03 -2023 1 Introduction 3 2 Understanding the current state 4 2.1 The trends of today and tomorrow 4 2.1.1 Reliably interconnecting systems 4 2.1.2 Being ready for future changes 4 2.1.3 Insuring system safety and availability with cybersecurity 4 2.2 The Undesigned Network or the status quo of IACS networks 5 2.2.1 The Evolved Network 5 2.2.2 The IT-OT Entangled Flat Network 7 3 Components of an IT / OT Network 8 3.1 Hardware 8 3.1.1 Switches 8 3.1.2 Routers 8 3.1.3 Firewalls 9 3.1.4 Industrial Wireless Local Area Network (IWLAN) 10 3.2 Network Software 10 3.2.1 Network Management System (NMS) 10 3.2.2 Network Services Software 10 3.3 Cybersecurity Software 11 3.3.1 Secure Remote Access 11 3.3.2 SIEM 11 3.3.3 Intrusion Detection System/ Intrusion Prevention System (IDS/IPS) 11 3.3.4 Zero Trust Concept 11 4 Categorizing Future-Ready OT Networks 12 4.1 Small OT Networks – Connecting Isolated Systems 14 4.1.1 Key Considerations 14 4.2 Medium Networks – Connecting Distributed Systems 16 4.2.1 Key Considerations 17 4.3 Large Networks – Connecting Multiple Facilities 18 4.3.1 Key Considerations 18 5. Key Steps 21 Resources 22 Figure list Figure 1 – The Evolved Network 6 Figure 2 – IT-OT Entangled Flat Networ k 7 Figure 3 – Small Network Reference Architecture 14 Figure 4 – Medium Network Reference Architecture 16 Figure 5 – Large Network Reference Architecture 18 Table of Contents 2 The OT Networking Reference Architecture – Table of Contents Version 2, 03 -2023 Operations Technology (OT) networks require different structures and skills to properly design, deploy, maintain, protect, and improve. These skills differ from standard Information Technology (IT) Networks due to the requirements of the OT environment. These differences include how the data is used, expected lifecycles of the equipment, acceptable latencies in data transfer, and operational reliability and availability requirements of an operating plant, and many more. This document provides an overview of the steps needed to implement a cybersecure plant-wide OT network. When discussing OT networking, it is important to understand the current state of the systems and the desired goals. With this assessment, we can design a more complete solution that meets the needs with hardware, software, and services while considering current and future roles and training for personnel. Commonly overlooked factors during network improvements are the potential future requirements of the plant and broader enterprise. This is why we advocate building a scalable network which considers not just the current needs but also creates a flexible solution that can easily adapt to future trends. By building future-ready networks, we are prepared to embrace the latest advancements in technology, enabling easier adoption of new manufacturing concepts, and at a lower total cost of ownership. The purpose of this document is to inform on OT networking best practices and reference OT network architectures. Delving into specific technologies and how to implement them is left to consultants, technical manuals, and further training. 1. Introduction 3 The OT Networking Reference Architecture – Introduction Version 2, 03 -2023 A critical step in being able to design a solution that meets the current needs and considers the trends of tomorrow is to understand the current state of the network. Every network begins by having a need for one system to communicate with another. In the past, this was accomplished by implementing proprietary and physically complicated wiring schemes. The systems of today and of the future meet this need with a network based on an open standard called Ethernet. By utilizing Ethernet, every vendor can enable their devices to communicate with another vendor’s devices. It is on top of Ethernet that common industrial protocols like PROFINET, Modbus TCP/IP, EtherNet/IP and EtherCAT function. The use of Ethernet creates a universal platform for automation. The architectures proposed in this document support all standard Ethernet-based automation protocols. 2.1 The trends of today and tomorrow Most trends in the networking space of Industry 4.0 can be summarized in the following three categories. 2.1.1 Reliably Interconnecting Systems The days of isolated systems are coming to an end due to the cost advantages of being able to automate a greater number of processes. Gathering data from field level and utilizing it for better planning is but one example of how manufacturing is changing from this explosion of data. This shift in our expectations can only be addressed by having a plant-wide network capable of transferring all the data generated by our systems. When these automated processes become part of our day-to-day operation, reliability becomes critical. We must design our OT network to be reliable and resilient. 2.1.2 Being Ready for Future Changes Industry 4.0, Industrial Internet of Things (IIoT), and Edge computing have been driving immense changes in our OT networks. The pace at which we will need to embrace these changes has never been this fast, and yet, it will never be this slow in the future. We must therefore design our networks not just for the current needs, but to be flexible and ready for the inevitable change. 2.1.3 Ensuring System Safety and Availability with Cybersecurity Cybersecurity has become synonymous with safety in the world of increasingly connected digital assets. Now that life-safety systems are increasingly operating on Ethernet networks, the availability and reliability of our network has become mission-critical to our manufacturing plants. Due to increasing portions of our operations becoming digital, we increase the costs incurred when these digital systems fail. This creates a situation that makes cybersecurity attacks more profitable for bad actors. It is therefore imperative to become educated on the risks that exist. Many standards and models have been created to provide best-practices in terms of cybersecurity. IEC 62443 is an internationally recognized series of standards that provide a framework to design and operate cyber-secure Industrial Automation and Control Systems (IACS). Another prevalent OT cybersecurity standard is NIST SP 800-82 which is used more often in critical infrastructure industries (e.g., electrical power distribution, water/wastewater, oil and gas pipelines), but is also being adopted in more typical manufacturing operations (e.g., chemicals, automotive, food & beverage). Both standards have been in development, in various forms, for about two decades and are now very similar in their basic recommendations. This document will rely most directly on IEC 62443 as it has a more general applicability and greater adoption internationally. The architecture drawings included will make references to the Purdue model as it provides a model that segments devices and equipment into hierarchical functions and layers. This document discusses some of the technical controls for securing an OT or Industrial Control System (ICS) network. However, it is extremely important to develop and implement a dedicated OT Cybersecurity Program to be able to fully assess, design, implement, maintain, and continuously improve a secure OT network. While the design and implementation of a cybersecurity program is beyond the scope of this document, the IEC 62443 standard series contains a section (Part 2-1) which describes requirements for OT cybersecurity programs. 2. Understanding the current state 4 The OT Networking Reference Architecture – Understanding the Current State Version 2, 03 -2023 If we think of how safety programs developed over the 1990s and 2000s, we realize that without a continuous improvement program and a well-structured organization to carry out the program’s policies and procedures, a safety program would quickly become an afterthought and the safety posture of your plant would quickly decline. The same is sure to happen with new cybersecurity initiatives if there is no OT-specific cybersecurity program with dedicated resources and well-defined management responsibilities. 2.2 The undesigned network or the status quo of IACS networks Many legacy OT networks are struggling to keep up with the demands of digitalization, including the multiple connections to business-level systems, outside service providers, machinery or packaged OEMs systems, and the large number of Ethernet devices which have been added over the last three to five years. This is often because these networks were not designed for this level of connectedness, and often, they were never actually “designed” at all. They often grew haphazardly as required. Over the years and decades, automation engineers or production personnel added switches and connections to get the data where it was needed but rarely had time or budget to take a step back and improve the overall architecture. This led to two common, but problematic situations – the Evolved network and the Flat network. 2.2.1 The Evolved OT Network Many companies have Ethernet networks that grew organically over time, from mere replacements for the serial connection between a PLC and an HMI to interconnecting multiple cells or areas and then to providing data for higher level systems. Then the I/O and instrumentation layer began to incorporate Ethernet as an easy and standardized approach to replace multiple proprietary field busses. Eventually, these networks morphed into a vast web of interconnected instruments, drives, controllers, HMIs, SCADA, remote access, databases, recipe systems, batch managers, Manufacturing Execution Systems (MES), Enterprise Resource Planning (ERP) systems, and various cloud-based functions (e.g. machine learning, data analytics, data warehouses, etc.). These networks (as depicted in Figure 1) are characterized by a mix of multiple architectures (line, star, ring) and typically have little to no redundancy, often with multiple single points of failure which could bring down large portions of the network. These networks also often contain unmanaged switches which limit diagnostics and security capabilities. Additionally, evolved OT networks may be physically separate (also known as air-gapped) from the business network but are more typically connected via a single firewall controlled by IT. Documentation is also often lacking and typically out of date. QUICK SIDE NOTE on the myth of security via “Air Gapped” Networks: There really is no such thing as security by air gap in industry. Any control system requires maintenance, updates, and probably expansion. Any time a USB flash drive or laptop from a non-dedicated source is connected to the network or a device on the network it represents a breach of the air gap. Due to the lack of documentation and the freeform nature of these networks, they are difficult to troubleshoot and require significant additional time and effort to expand or modify. They often exhibit intermittent connectivity or data throughput issues with no efficient methods to detect the exact location or cause of the issues. 5 The OT Networking Reference Architecture – Understanding the Current State Version 2, 03 -2023 Core Distribution Access Enterprise Network Production Backbone Production Network Production Cells Internet External Firewall HRP Switch Switch Switch Switch Cell 1-1-1 Cell 2-1-1 Cell 2-1-2 Cell 3-1-1 Cell 3-1-2 Switch Switch IT Core Direct Connection to Business Network May or May Not be an IT/OT Firewall Ring Switch Switch Switch Switch Tree Firew Line 3 Backbone Main OT Switch Switch Switch WAN WAN Router Servers Main Control Room Control Room Switch Operator Stations Operator Station Operator Station Operator Station Operator Station Typical Evolved Network Figure 1 – The IT-OT Evolved Network 2.2.2 The IT-OT Entangled Flat Network Another path to Ethernet on the plant floor is the one that is an extension of the business network into the Operational Technology (OT) space. This is where IT either desired to maintain control over all Ethernet networks in the enterprise or was tasked with helping their OT colleagues create and maintain an Ethernet network. Thesecnetworks are known as “flat” networks because there is no protection scheme between production systems and even the rest of the corporate network. These flat networks may be well designed but use IT-style line or star topologies which can be more efficient from a cabling and port count perspective but lack the resiliency of a ring style. Because it is one big network, the OT devices are often intertwined with the IT devices. A flat network presents challenges to both the OT and IT users as problems on one type of equipment can directly affect the other types. Additionally, IT typically enforces IT-centric policies and procedures across the full network. This is because there is no segregation which would allow the OT areas to be managed differently. 6 The OT Networking Reference Architecture – Understanding the Current State Version 2, 03 -2023 Flat networks can cause production equipment slowdowns and outages due to non-OT traffic (e.g. video or large file transfers), network reconfigurations, or firmware updates. Broadcast storms are more common on large non-segmented networks and can act like a Denial-of-Service (DoS) attack. It is also easier to inadvertently create a loop in a large flat network which could then also cause communication issues even taking down parts of the network until resolved. Identifying, isolating, and resolving issues is an immense challenge as the lack of segmentation forces troubleshooting efforts to consider the entire system rather than smaller sections. Another underlying issue with IT managed OT networks can be the service level agreements (or lack thereof) between OT and IT. If a switch goes down or a network problem is suspected during non-office hours there may be no quick route to troubleshooting and repair of the network. As production relies more and more on Ethernet based networks, it is imperative that support structures be created which meet the near instant response times required on the production floor. Figure 2. - IT-OT Entangled Flat Network Core Distribution Access Enterprise Network Production Backbone Production Network Production Cells Internet External Firewall HRP Switch Switch Switch Switch Cell 1-1-1 Cell 2-1-1 Cell 2-1-2 Cell 3-1-1 Cell 3-1-2 Switch Switch IT Core Direct Connection to Business Network No IT/OT Firewall Ring Switch Switch Switch Switch Line 3 Backbone Main OT Switch ERP Station ANDON Board Switch Switch WAN WAN Router Servers Main Control Room Control Room Switch Operator Stations IT Workstation ERP/Email/Web File Servers Manufacturing Office Manufacturing Office Switch ANDON Board IT Workstation ERP/Email/Web IT Workstation ERP/Email/Web Operator Station Operator Station Operator Station Operator Station IT-OT Entangled Evolved Network ERP Station ANDON Board 7 The OT Networking Reference Architecture – Understanding the Current State Version 2, 03 -2023 The OT Networking Reference Architecture – Components of an IT/OT Network When designing an OT network, it is critical to select components that are meant to handle the environmental conditions as well as the application requirements. As an example, IT equipment is usually placed in climate controlled and dust free environments; their fans would fail quickly should be they be subjected to the hot and dusty OT environments. This unexpected downtime can easily cost more than the savings of using components that weren’t designed to operate in OT environments. The selection of components can be simplified by having industry specific certifications like Class 1 Div. 2 for explosive environments found in the oil industry. It is also recommended to look for third party certification of the manufacturer’s claims. Designing networks for hazardous environments is beyond the scope of this document; please consult a trained and experienced control systems engineer if you have needs for hazardous area networking systems. 3.1 Hardware The functions of networking components commonly found in plant-wide OT networks are outlined below along with the recommended Siemens offering. 3.1.1 Switches There are two main categories of switches, unmanaged and managed. Unmanaged switches offer simple connection of Ethernet devices with little or no configuration options, but this simple operation comes with significant opportunity cost. Their unmanaged nature means network diagnostics, security, and network redundancy features are unavailable. Unmanaged switches are not recommended in a plant-wide network for the reasons above. Managed switches allow the connection and networking of ethernet devices in a manner that can be configured and monitored to fit the needs of the network in a secure manner. Noteworthy management functions include redundancy mechanisms (RSTP, ring networks, etc.), security functions (disabling ports, MAC or IP filtering, and user management), network segmentation through VLANs, and some managed switches even support Network Address Translation (NAT) to allow integration of devices into differing IP network structures. In addition to management, monitoring functions allow users to access diagnostic information that assist in troubleshooting and network optimization. Managed switches are recommended to be used in a plant-wide network. All switches shown in the figures below are managed. The recommended Siemens switches are the SCALANCE XC-200, XC-300, XR-300, XM-400 (layer 3 capable), and XR-500 (layer 3 capable). Click here for information on SCALANCE X 3.1.2 Routers Routers move network traffic between different Internet Protocol (IP) subnets and Virtual Local Area Networks (VLANs). Network segmentation (using VLANs and/or subnetting) is highly recommended in plant-wide networks as it creates different security cells that can only be accessed with the help of a router or firewall. Routers enable communication across different subnets and VLANs when one area of the network needs to communicate with a separate segment of the network. For example, a server may need access to many different production lines that are segmented – in this case a router will be needed to traverse these boundaries. Routers are recommended and generally required in the upper layers of the plant-wide network. Routers are placed in the production backbone in the figures below. The recommended Siemens Layer 3 switches with routing capabilities are the SCALANCE XM-400 and XR-500. Click here for information on SCALANCE X 3. Components of an IT/ OT Network 8 Version 2, 03 -2023 3.1.3 Firewalls Firewalls are utilized for restricting communication and access to various areas of the network. A firewall restricts access by following a set of user-defined rules which describe the allowed communication between specific devices (IP Addresses) or segments (VLANs or subnets) in the network. Most firewalls can also act as routers allowing communication between network segments while restricting which specific IP addresses can communicate to each other and which protocols may be used. Another commonly used function is Network Address Translation (NAT). The most common area firewalls will be found are between the internet and the company network. This is to protect the company network from threats existing on the internet. In a plant-wide network, firewalls are recommended between the OT and IT network divide and between the various cells / lines and the backbone for zone/area protection. Recommended types of firewalls are as follows. 3.1.3.1. MAC Firewall (Layer 2) In areas where additional protection is needed within an IP subnet – Layer 2 firewalls can be utilized to restrict communication between devices based on MAC addresses. These devices will typically be found within the Production Cell section in the below figures. In more recent times, Layer 2 or MAC firewalls have been mostly superseded by Layer 3 firewalls due to easier management over time. If a device is replaced it may be set with the same IP address but will likely have a different MAC address so the Layer 2 firewall rules would have to be revised. A Layer 3 firewall relies on IP addresses, so it’s rules would not need revision if a device were replaced with a different MAC address but the same IP address. Many firewalls can be both Layer 2 and Layer 3 firewalls at the same time, it is a matter of configuration. 3.1.3.2. Stateful Inspection Firewall (Layer 3) These devices examine each packet to determine if they meet predefined criteria for permitted communication, which are defined by rules based on IP addresses and TCP/IP ports. This type of firewall will often serve as a zone/area firewall and is located between the Production Cell and Production Backbone areas in the figures below. Siemens firewalls are capable of both Layer 2 and Layer 3 functionality. The recommended Siemens firewalls are the SCALANCE S615, SCALANCE SC-600, and SCALANCE M-800 (the M-800 series are Mobile devices meaning they have wireless interfaces in addition to the wired connections). Click here for information about Siemens Firewalls 3.1.3.3. Next Generation Firewall (NGFW) This type of firewall expands on the functionality of Stateful Inspection Firewall with Deep Packet Inspection (DPI). It may include additional security tools such as an Intrusion Detection System / Intrusion Prevention System (IDS/IPS), malware filtering and antivirus. The defining feature, DPI, goes beyond the information in the packet header and evaluates the packet payload. An example of this advantage is that a PLC can be protected from program downloads from an HMI, but still allow the HMI to read and write data in the PLC. NGFW firewalls are recommended between the company network and internet and recommended for use between the IT and OT networks, which are labeled as IT/OT firewalls in the figure below. For Next Generation Firewalls, Siemens has partnered with leaders like Fortinet, Palo Alto and CheckPoint to offer their best-in-class software solutions with Siemens’ hardware. Click here for information about Next Generation Firewalls The OT Networking Reference Architecture – Components of an IT/OT Network 9 Version 2, 03 -2023 3.2 Network Software 3.2.1 Network Management System (NMS) Network management systems are utilized for centralized monitoring, management, and configuration in networks of all sizes. Commonly found in enterprise (IT) networks, the NMS provides several key functions in industrial networks including – fault, configuration, inventory, performance, security, firmware, and backup management. The centralized nature of an NMS system limits the need of individual device management and saves a substantial amount of time in industrial network administration. The recommended Siemens NMS solution is SINEC NMS – industrial NMS for OT networks. Click here for information about SINEC NMS 3.2.2 Network Services Software The recommended Siemens solution for Network Services is SINEC INS, which includes all the services mentioned below in a single software platform. Click here for information about SINEC INS 3.2.2.1. Syslog Server Syslog is a standard networking protocol for centralized message logging. Syslog servers collect status, fault, and other messages from distributed devices for evaluation. 3.2.2.2 Network Time Protocol (NTP) Server NTP is a networking protocol for time synchronization between different devices to within a few milliseconds. The NTP server provides the reference time that various end point devices in an industrial network, such as PLCs, industrial PCs, or network switches, will synchronize with. Synchronized time allows more effective troubleshooting and logging. 3.2.2.3 Dynamic Host Configuration Protocol (DHCP) Server DHCP is a network management protocol for assigning IP addresses within a network from a DHCP server. The DHCP server assigns unique addresses in a predefined range, which helps prevent duplicate and misconfigured IP addresses. 3.2.2.4 Remote Authentication Dial-in User Service (RADIUS) Server RADIUS is a networking protocol that provides centralized authentication, authorization, and accounting (AAA). The server manages the users and access levels granted to the users. 3.1.4 Industrial Wireless Local Area Network (IWLAN) Industrial wireless networks differ from wireless seen in the IT space. Industrial wireless systems experience more demanding physical environments and more challenging communication requirements. For this reason, specialized equipment has been designed for the requirements in the OT environment such as deterministic communication mechanisms, fast roaming, and features that provide expanded operability in control networks. In figures below, IWLAN is shown as connected to one of the production cells and can be connected in many other areas of the OT network. This document does not focus on the architecture required for IWLAN. Please refer to the additional resources section when considering IWLAN. The recommended Siemens solution for industrial wireless are the SCALANCE W family of radios. Click here for information about SCALANCE W The OT Networking Reference Architecture – Components of an IT/OT Network 10 Version 2, 03 -2023 3.2.2.5 SSH File Transfer Protocol (SFTP) Server SFTP is a networking protocol that provides file access, transfer, and management in a network. The SFTP server is commonly used in industrial networks for storing and managing device configuration and backup files. 3.2.2.6 Domain Name Server (DNS) Server DNS servers are the phone books of the internet by providing IP addresses for human readable domain names (e.g. www.google.com -> 142.251.32.46) 3.3 Cybersecurity Software 3.3.1 Secure Remote Access Secure remote access systems are utilized to allow secured access to industrial network components using technologies such as Virtual Private Networks (VPN). Remote access is typically needed by the original equipment manufacturer (OEM), service providers, and the end user’s engineering team for quickly responding to system issues affecting production and operation. The remote access software authorizes and grants access to various network end points based on user credentials and preconfigured communication relationships. The recommended Siemens solution is SINEMA Remote Connect – secure remote access for OT networks. Click here for information about SINEMA RC 3.3.2 SIEM Security Information and Event Management systems monitor log data (typically in a Syslog server) and look for specific Indicators of Compromise (IoCs), send alerts when suspicious activity is detected, and in some versions may provide playbooks and integrations to helpdesk ticketing systems to help organize the response to an incident. The OT Networking Reference Architecture – Components of an IT/OT Network 3.3.3 Intrusion detection System/ Intrusion Prevention System (IDS/IPS) An IDS is a software and/or device that monitors a network for malicious activity or policy violations. A detection is reported to the network administrator or to a Security Information and Event Management (SIEM) system. IDS systems are typically passive and only notify of an intrusion on the network. An IPS is an expansion in functionality of an IDS. The “prevention” occurs by responding to a “detected” network threat by blocking or dropping the packets from the network. For IDS/IPS, Siemens has partnered with leaders like Nozomi, Claroty, SecureNOK, Palo Alto and Fortinet to offer their best-in-class solution running on Siemens hardware. Click here for information about Next Generation Firewalls 3.3.4 Zero Trust Concept Zero Trust is one of the latest buzzwords that tries to condense several cybersecurity technologies into a single concept. The overarching idea is that perimeter defenses (firewalls, VPN, IDS, IPS, etc.) are not enough. There is too great a chance that an external threat will still be able to get into the network and there are many internal threats that are just as dangerous as the external threats. Thus, there should be no trust of a device or user just because they are inside the perimeter defenses. This means that every device and user should be authenticated for every connection initiated on the network. This extends to the protocols and applications that talk between HMI and PLC, engineering laptop and PLC, or MES and data historian. Zero Trust is a difficult concept to fully apply to an ICS as many devices are older and have no certificate authentication capability or do not have enough on-board processing power or memory to implement additional security protocols. However, Siemens current SCADAs, HMIs, PLCs, switches, and security devices are capable of being configured for use in a Zero Trust environment. Click here for information about the SCALANCE LPE and Zscaler Zero Trust 11 Version 2, 03 -2023 With the infinite possibilities that exist in networks, assessing the state of a network can be a difficult task. The IEC62443 standards provide two different classification models: maturity levels and security levels. Maturity levels describe the efforts put towards having documented systems and repeatable procedures and are used to assess the organizational maturity of an OT cybersecurity program. Security levels (SL) describe the technical risks and the level adversarial capability and motivation faced. The SLs allow the current state of a network and future target states to be unambiguously defined which then enables a detailed gap analysis and the identification of necessary steps to prioritize and mitigate identified risks. It is important to note that no cybersecurity system is perfect and that there will always be some level of risk that must be tolerated. However, the decision to determine what level of risk is acceptable to the business is reserved for the highest level of management. The goal of this document is to provide an overview of the technical controls one should incorporate when designing a future-ready OT network. We will be summarizing the requirements of different maturity and security levels for the following three network sizes: 1. Small – where connecting systems is the highest priority 2. Medium – where building a scalable platform is the highest priority 3. Large – where protecting against advanced threats is the highest priority Every network will inherently go through phases and transformations throughout its lifetime. It is for this reason that we must always begin with proper planning. We want to design a scalable system that can adapt to the changes without requiring a complete redesign. The core of our network will be used to connect multiple production cells, which will typically increase in number over time. By following the fundamentals outlined here for the foundational network, expanding the network in the future will proceed in an efficient manner. 4. Categorizing Future-Ready OT Networks The OT Networking Reference Architecture – Categorizing Future-Ready OT Networks Given the explosion of connected devices and experts forecasting that the Industrial Internet of Things (IIoT) market will grow at 24% annually through 2023 – it’s a good bet that industrial networks will need to be capable of processing and transmitting large amounts of data from many connected devices. Preparing for the future of industrial networking will be essential to remaining competitive. 12 13 © 203bySieSmnSsimin 13 Industrial Network Health Checks typically include: • Assessment of the current status of the network • Benchmark testing to identify potential network issues like: - Packet collisions - Problematic network architecture - Sub-optimal device configuration • Creation of a network asset inventory • A detailed report with recommendations to achieve a reliable network ready for future challenges Receive a free consultation by e-mailing us at siemensci.us@siemens.com HOW FIT IS YOUR COMMUNICATION NETWORK? Consider an Industrial Network Health Check! Version 2, 03 -2023 The OT Networking Reference Architecture – Categorizing Future-Ready OT Networks Figure 3. - Small Network Reference Architecture Servers Core Distribution Access Enterprise Network Production Backbone Production Network Production Cells WAN Internet External Firewall WAN Router HMI and SCADA Historian and Reports Network Services Secure Remote Access Network Management SIEM and IDS/IPS Industrial DMZ Redundant Connections Industrial WLAN Layer 3 VRRP Ring Firewall Firewall Switch Switch Cell 1-1 Cell 1-2 Cell 1-3 Line NAT Firewall Firewall Switch Tree / Star Router Router Redundant Connections Core Note: Optional, but recommended – redundant routers and fi rewall IT / OT Firewalls IT / OT Firewalls Switch 4.1 Small OT Networks – Connecting Isolated Systems New and smaller facilities should begin with the reference architecture below as a baseline that will properly apply fundamentals of OT networking. This will make for efficient expansion of the industrial network as it grows and requires more capacity. A small-size network can be classified using the following criteria: • Less than 50 end point devices • Network disruption above production cells will not result in substantial financial impact • A network spanning up to a few lines (composed of production cells) or a small sized facility 4.1.1 Key Considerations for Small OT Networks 1) Topology • Ring Networks o A ring network is found in the Cell 1-1. A ring network is recommended in automation and OT networks to add resiliency via connection redundancy while maintaining quick recovery times in the millisecond range. o The most common failure mode is a disruption in physical connection, which makes ring redundancy a best-practice and an efficient method of increasing resiliency for industrial networks 14 Version 2, 03 -2023 • Star Networks o A star topology network is found in the Production Cell 1-3. A star network is typically less efficient than a ring network and does not offer redundancy. This introduces many single points of failure into the network, which include each cable, each end device, and the network devices aggregating the connections. • Line Networks o A line topology network is found in Production Cell 1-2. A line topology network offers the lowest possible level of resiliency in a network due to interdependence on the upper-level devices in the line topology. For example, if the first device in the line fails – all other devices downstream will lose connection. It is not recommended to utilize a line topology when uptime is critical. 2) Redundancy Several layers of redundancy are possible in an OT network that will improve resiliency and tolerance of failure in the network. Redundancy is not necessary in all situations but is highly recommended in critical connections that would impact operation if lost. • Ring redundancy, highlighted in the “Topology” section above, is a form of connection redundancy and is commonly used in industrial networks to mitigate loss of physical connections between devices. • Virtual Router Redundancy Protocol (VRRP) can be configured for redundant communication between separate subnets or VLANs, such as the connections seen between Production Cell 1-1 and the routers in the Production Backbone routers. In the event of a failure in one router or physical connection, the remaining functional connection will be detected and utilized as the communication path. VRRP is configured in the redundant router pair – such as the firewalls in Production Cell 1-1, the routers in the Production Backbone, and the redundant IT/OT firewalls between the Production Backbone and Enterprise Network. 3) Recommended Software • Secure Remote Access software is recommended for small networks to enable access by OEMs, service providers, or engineers to the systems when not physically present. • NTP server software is recommended to synchronize time across the devices for efficient troubleshooting and transparency in log timestamps. 4) Additional Security Considerations • End Point Hardening is a critical first step in making sure the overall system is secure. Changing default passwords, keeping up with security updates, and following security recommendations in the user manuals must be done. • Segmentation of the network sets the foundation for creating cells and allowing only the necessary communication between different components. Segmentation is accomplished using VLANs and subnetting. These require routers between different parts of the network. Segmentation can be augmented by firewalls to restrict communication to the specific IPs and Ports needed for the application. • Firewalls should be placed at strategic entry and exit points to protect assets from external and internal threats. It is highly recommended to have a firewall between the IT network and OT network as the IT network is usually connected to the internet. This protects not only the OT network but also protects the IT network from threats that could come from the OT network. The OT Networking Reference Architecture – Categorizing Future-Ready OT Networks • A Demilitarized Zone (DMZ) exists to provide for a controlled set of services that are allowed to communicate from the OT side to the IT side and vice versa. There should not be any communication from the field devices into the IT side and vice versa. • In the diagram above, Secure Remote Access can be configured to allow a remote user to only be able to access a single cell. This can greatly reduce the risk of allowing third-party users access to the OT network as they are only capable of communicating with a small section of the network or even just a single IP address. • Documentation and backups are necessary for troubleshooting any issues with the network. Network drawings, Endpoint information such as IP addresses, MAC addresses, firmware versions, and procedures should be created. 15 Version 2, 03 -2023 4.2 Medium Networks – Connecting Distributed Systems As operations grow and evolve over time, the scale, capacity, and services required of the OT network will correspondingly increase. A medium sized network can be classified using the following criteria: • More than 50 end point devices • Critical operation where network disruption will result in substantial financial impact • A network spanning multiple lines (composed of production cells) or a medium-size facility The OT Networking Reference Architecture – Categorizing Future-Ready OT Networks Core Distribution Access Enterprise Network Aggregation / Backbone Production Backbone Production Network Production Cells WAN Internet External Firewall Servers HMI and SCADA Historian and Reports Network Services Secure Remote Access Network Management SIEM and IDS/IPS WAN Router Servers IT Industrial DMZ Redundant Connections Industrial WLAN Layer 3 VRRP Layer 3 NAT / Firewall Ring Ring Firewall Firewall Firewall Firewall Switch Switch Switch Switch Cell 1-1 Cell 1-2 Cell 1-3 Cell 1-4 Line Firewall Firewall Switch Tree / Star Router Router Redundant Connections Core IT / OT Firewalls IT / OT Firewalls Ring Ring Switch Switch Switch Switch Switch Switch Industrial DMZ Figure 4 - Medium Network Reference Architecture The above medium reference network architecture will build on the key concepts of the small reference network architecture. Each of the key considerations listed below are in addition or an expansion of the considerations covered in the small network architecture 16 Version 2, 03 -2023 4.2.1 Key Considerations for Medium Networks 1) Topology • Ring Networks o As the complexity and importance of the network continues to grow – the presence of ring topologies should be more prevalent to further improve the resiliency of communication. This can now be seen in Cell 1-2 in addition to Cell 1-1. o The