Detecting Cyber Risks Before They Lead to Downtime

Three foundational best practices can help ensure you detect and mitigate operational as well as cybersecurity risks before they lead to downtime.

Protection Background Technology Security 524882074 701x502 (1)

Production lines are the crown jewels for manufacturers, the core systems for creating value and revenue. Downtime, especially unplanned, is anathema. According to Forbes, unplanned downtime costs industrial manufacturers as much as $50 billion a year, and shutdowns can eat up one percent to 10 percent of available production time.

Decades of industrial digitalization have improved efficiency, reduced errors and cut costs – but also introduced greater cyber risk that, if undetected, can add to unplanned downtime or worse, in the case of a malicious attack. Ransomware attacks make headlines, but the real threats to downtime are more mundane: Asset or network misconfigurations that are detected late, undesired process changes, resource usage spikes, new connectivity and other anomalies are far more likely to threaten productivity than outside cyberattacks. They are also well within the control of your operational technology (OT) engineering and IT team.

Three foundational best practices can help ensure you detect and mitigate operational as well as cybersecurity risks before they lead to downtime:

  • Maintaining an accurate inventory of all connected devices.
  • Network segmentation to prevent unwanted access and reduce the blast radius in case of cyber incidents.
  • Non-intrusive network monitoring to detect any threat before it affects uptime

Inventory all Connected Assets

Cybersecurity starts with maintaining an accurate inventory of all connected assets, including where they are and with what they’re communicating. For manufacturers, that means managing OT/industrial control system (ICS) devices as well as any IT and Internet of Things (IoT) devices connecting to the OT network.

Unfortunately, the discovery approaches that work for IT don’t always work for sensitive OT and IoT assets given safety rules, vendor interoperability issues, industrial process requirements and other considerations. Many OT and IoT assets cannot accommodate an agent, or specialized software component installed on the device to perform security-related actions and cannot be scanned by standard IT security tools.

Then there’s a larger issue: Whereas much of IT includes planned obsolescence, OT assets have long refresh cycles - up to 30 years. Until recently they weren’t connected, so they weren’t built with security or even integrity in mind.

Adding security (such as agents) later can be difficult. The “security as an afterthought” albatross has been hard to shake. Some leading security tool vendors are finally implementing “secure by design” principles to newer technology, but as Vedere Labs underscored with its OT:ICEFALL vulnerability disclosure, that is still the exception. To complicate matters, known issues in OT assets associated with “insecure by design” practices are not always assigned a Common Vulnerabilities and Exposures number (CVE, or publicly disclosed security flaw), so they remain less visible and actionable than they should be.

Despite these challenges, OT and IoT can be inventoried using passive techniques to discover, classify and assess them for compliance with your risk policies. This includes agentless techniques, including non-intrusive network monitoring leveraging deep packet inspection (DPI) that understands hundreds of industrial protocols to identify assets, extract configuration details and detect anomalies in connectivity and behavior.

Network Segmentation

Digital transformation is underpinned by the convergence of OT and IT systems. Modern manufacturing wouldn’t be possible without flexible production methodologies that rely on hundreds of digital systems, sensors and interconnectivity between IT and OT networks. Connected programmable logic controllers (PLCs) that control valves, pumps and other equipment manage every process. The use of IoT devices to collect real-time data on production processes has exploded. This data flows into IT or even cloud services to enable better scheduling, forecasting and overall performance against metrics.

Convergence doesn’t mean IT and OT systems are collapsed into a single, flat network. Rather, information is shared to allow them to interoperate. The challenge is how to securely connect IT and OT assets and systems that need to communicate while preventing those that don’t from doing so. Oftentimes, unwanted communication links go unchecked, and vulnerabilities hide in plain sight, based on the assumption that OT and IT are separated when they are not. Such assumptions increase the chance that malware on one network may spread and impact other networks. In manufacturing especially, the answer is segmentation. 

While in IT the common approach is to patch, this may not be possible in production environments. Patching OT devices is notoriously difficult, due to their mission-critical nature. Systems need to be stopped and restarted to load patches, which means downtime, and some processes and equipment, such as a blast furnace, need to be shut down slowly for safety reasons, increasing the costs.

Instead of patching, vulnerable devices must often be dynamically segmented from other parts of the network until they can be remediated. Segmentation projects are notoriously complex, but they needn’t be. Today, you can leverage software that visually maps traffic flows and identifies what should and shouldn’t be communicating based on how assets are classified and how you want them to interact. With a traffic matrix, you can reverse engineer your segmentation policies based on simulations that illustrate the impact of various changes without causing disruption.

Non-Intrusive Network Monitoring

To avoid costly downtime, threats to operational continuity must be detected and investigated as early as possible. That can be accomplished by scanning connected devices for configuration changes and vulnerabilities. However, unlike traditional IT, OT assets cannot be continuously scanned in the same manner and many risks will remain unnoticed. Instead, a system designed for manufacturing environments must have the ability to passively monitor the network infrastructure to locate assets and detect behavior changes and anomalies. That requires understanding dozens of industrial protocols and continuously monitoring the communications and checking against a database of OT/ICS-specific Indicators of Compromise (IOCs, or evidence of a breach) and CVEs.

The bane of many monitoring systems is they produce a flood of information about potential harm, not all of it urgent. To be useful, critical alerts must be prioritized based on operational or cybersecurity risk so the right team can respond.

For example, OT engineers need to quickly spot undesired process values, incorrect measurements or when a critical device fails so they can resolve issues more quickly. Likewise, IT/Security teams need visibility into suspicious user behavior and unauthorized network connections – and, of course, they must be notified as soon as an actual cyberattack is detected.

Automate the Fundamentals to Minimize Downtime

With industrial environments increasingly dependent on digital systems for production, foundational cybersecurity is essential. OT/ICS and even IOT assets present challenges because typically they cannot be patched or scanned – and may even be insecure by design. These challenges can be surmounted by:

  • Holistically managing your asset inventory.
  • Segmenting networks to prevent unwanted access.
  • Monitoring the environment to detect threats as early as possible.
  • Prioritizing and, where possible, automating responses to avoid costly downtime.

All cybersecurity begins with knowing what’s on your network. Beyond that, manufacturers must know what cyber and operational risks exist across sites and be able to detect and prioritize response to threats before they lead to downtime. Each of these activities – including incident response – can largely be automated without risk to OT systems.

Given the cyber skills gap, explosion of specialized assets and evolving threat landscape, automating cybersecurity operations is not just desirable, but imperative.

More in Facility