Industrial Cybersecurity Addresses Crucial Vulnerabilities

Lesley Carhart talks about industrial cybersecurity and risks to critical infrastructure.

Most cybersecurity focuses on laptops, servers, data breaches, personal protection, and high-tech topics like machine learning and 5G. But a lot of critical infrastructure, like power plants, hospital systems, and industrial equipment, is running on old tech. When modern cyber threats collide with this old technology, it leaves critical infrastructure vulnerable. Industrial cybersecurity operates to protect these systems when failure – or even shutting down for a little bit to deal with the threat – costs lives.

See Critical Infrastructure Risks with Lesley Carhart for a complete transcript of the Easy Prey podcast episode.

Lesley Carhart is a cybersecurity leader who focuses specifically on industrial cybersecurity. That includes things like trains, power plants, aircraft, and manufacturing equipment like cranes and drill presses – those are all run by computers these days, so they can all get hacked. Lesley has been working in this area for nearly two decades. Currently they are the Technical Director of Incident Response at Dragos doing incident response and digital forensics when things like this get hacked.

Finding a Career in Industrial Cybersecurity

As opposed to IT cybersecurity, this field is called OT cybersecurity. OT stands for operational technology. It’s a growing field because systems are more and more connected to each other and to networks, and they’re all old and vulnerable.

Lesley started out as a hacker in the 1990s. They started their career as a web developer, but then got hit by the dot-com bust. Left with no other options, they joined the Air Force. When they said they could do things with computers, the Air Force offered them a role soldering circuit boards for airplanes. Lesley went on to get a degree in avionics and do airplane maintenance. It wasn’t what they wanted to do with the rest of their life, but it gave them exposure to critical safety systems and things that are computers but don’t look like it.

After leaving the Air Force, Lesley got another degree in network engineering and went into cybersecurity. They started at a manufacturing company and worked their way up to incident response lead. After ten years there, an Air Force colleague reached out. He was doing a startup specifically for critical infrastructure cybersecurity and asked if Lesley wanted in. They did.

The company was Dragos, and Lesley has been there ever since. Their day job includes doing digital forensics and incident response investigations on the weird industrial and legacy computing equipment that runs everything we rely on in society. When something happens, they go wherever it is in the world and figure out what happened. There’s probably less than 100 people in the world who do what they do.

Traditional Cybersecurity versus Industrial Cybersecurity

There are two major differences between “traditional” IT cybersecurity and industrial and critical infrastructure cybersecurity. One of these is a way of thinking about security, attacks, and responding to incidents. The other is about the tech itself.

The Different Cybersecurity Mindsets

IT cybersecurity often focuses on hacker-type stuff. That’s not the focus as much in industrial cybersecurity. Yes, you want to find out what the attackers are doing and how to stop them. But the real priority is keeping the infrastructure functioning. The power has to stay on, the water has to stay safe, and systems that keep people alive have to stay running. You may notice that there are new configurations on the system or somebody has hacked the domain controller, but it may not have any bearing on whether or not the water is safe. In industrial cybersecurity, you have to prioritize preventing disaster.

You have to stop prioritizing hacker-y stuff. The priority there is keeping people alive and keeping the water safe to drink and keeping the power on.
Lesley Carhart

These critical infrastructure systems are systems of systems. They are full of safety controls, redundancies, and human operators, because they’re made to be safe. To protect them, you have to think about how attackers could defeat the controls. It’s not necessarily going to be that they hacked this or that. It’s more likely to be that they took out the safety controls or changed what the operator was seeing to make them do something different.

Industrial cybersecurity is less concerned with fancy tech tricks than in how attackers can get around safety controls.

In IT cybersecurity, it’s not uncommon to deal with attacks by shutting systems down until you can secure everything. But you can’t do that with critical infrastructure. When these things are keeping the power on or keeping people alive, you can’t just shut them down. You have to focus on not causing a worse impact while you respond to the threat. It’s a completely different set of priorities.

The Difference in Tech

The other main difference between IT and OT cybersecurity is the technology itself. Critical infrastructure has a lot of very old technology that many security people don’t have experience with. Most of the people coming into the field have never worked with Windows 95 or Windows 2000. But those operating systems are still running critical parts of society.

Industrial cybersecurity requires doing security work with very old computers doing very important things. And just pulling them out and replacing them isn’t an option in many cases. These devices have been certified to work together safely as they are. Replacing it with a Windows 11 machine is going to cause massive consequences. You have to have other solutions. Doing this work requires you to do forensics, containment, investigation, and threat hunting on decades-old devices against modern threats.

Security Considerations for Old Technology

When you’re looking at devices that are sometimes decades old, you have to think about different things. The two main considerations in industrial cybersecurity are knowing what’s in your architecture and how it’s laid out, and also what your specific vulnerabilities are. You won’t necessarily be able to fix issues in traditional ways, but you have to have that information.

Once you know what you do have and where you’re vulnerable, you can start to put in mitigations. Mitigations are often things like levels of detection. Passive detection like old-school network packet-based detection is a good one. It doesn’t touch anything, break anything, or require agents that old devices don’t support. Remote access control is also a big one since network connectivity and remote access is popular even though many of these old devices aren’t intended for that use.

Air gapping and leaving critical systems disconnected from networks isn’t really an option anymore. That train has left the station. There’s too much value in connecting systems to get real time telemetry and data on functioning down for the nanosecond. Billing, smart meters, remote detection sensors, and more all require connectivity. And these days, people want remote access to as much as they can. Office workers are getting return-to-office mandates now, but the power company doesn’t want that. Centralized remote technicians are more affordable. Lesley only sees a couple air-gapped systems every year, and most of those are extremely old, government, or nuclear.

Why Upgrading Isn’t the Solution

If you bought an IBM system in the 1970s or 1980s, it came with everything. Screens, terminals, mainframes, it was all there and certified in the factory to work together. Critical infrastructure and industrial equipment is like that even today. When you set up a power plant, everything – PLCs, computer systems, engineering workstations, network infrastructure, all of it – all comes from the vendor, and it is tested for months and certified to all work together safely. But they’re only certified in the condition that they came out of the factory. And when “safety” means human lives, you don’t want to upgrade without a plan and extensive testing.

Upgrades are a big deal. First, they’re massively expensive. Some industries, like oil and gas, have more money and can afford to upgrade more often. On the other end of the spectrum, municipal water may not be able to afford to upgrade more than every twenty years or when things break catastrophically. Upgrades also require major downtime. You can’t take things down more than once or twice a year. And in the case of municipal water, you have to have something to cover the gap. You can’t just tell residents you’re turning off their water for two weeks while you upgrade your systems! Even industries with a lot of resources aren’t upgrading often because of that.

There are good reasons why critical infrastructure technology gets so old. In most industries, upgrades happen every five to ten years. So industrial cybersecurity doesn’t lean on upgrades and updates to keep things secure. Industrial environments are in the lower to mid stages of maturity right now. They’re starting to implement programs, build better architecture, and add defense controls. But globally, there’s a long way to go.

Who Targets Critical Infrastructure

There are a handful of different types of cases that Lesley sees. One is criminal things like ransomware. Criminals are attacking everyone, not necessarily targeting them specifically. But they like industrial targets because they’re often more vulnerable and they are very visible when something goes wrong. Often the attacks don’t affect the actual running of the infrastructure, but the devices that monitor it and make sure it’s running safely. When those things go down, it will probably keep running safely for a while, but it’s a risk because the company no longer knows for sure. Especially when the industry is something like chemicals or hot metal, they sometimes have to shut down the operation because they can’t tell if it’s functioning safely or not.

The second category is insiders. Often that’s not malicious – although there are exceptions. The engineers who work there know how to break things really badly if they’re disgruntled. But most of the time, it’s accidental. Issues happen because someone is bored, or using unapproved systems or software trying to do their jobs, or even just trying to stream a movie during a slow night shift. But even though it’s not intentional, it can cause big problems.

The third type is geopolitical stuff – nation-states, state terrorists, and big cartels. It’s largely industrial espionage and sabotage. These environments often have proprietary data, manufacturing procedures, and similar data attackers can steal and exploit. Sabotage is more concerning. It’s been part of warfare for as long as humans have been around, but now we can do it via computer, and it can be both cheaper and more effective. Every country with the capacity is building footholds and doing recon on everybody they may want to target in the future.

The Political Aspects of Industrial Cybersecurity

A lot of what Lesley finds in terms of geopolitical actors in critical infrastructure systems isn’t malicious … yet. When Lesley finds a state actor in critical infrastructure systems, they’re often doing a lot of things. They’re building in backdoors so they can get in later, looking around to understand how the unique and complicated systems work, and setting things up so they can quickly bring down systems and cause huge problems if they need to in the future.

Industrial cybersecurity requires hunting down adversaries who have been lurking in systems for years.

The fact that many of these actors could quickly cut power, poison the water supply, and other horrific things keeps Lesley up at night. But because they’re not making moves right now, the issue doesn’t get much budget or attention. But everybody wants to have those footholds in case they want to use them in the future.

[Adversaries] might stay in an environment for twenty years before there’s a geopolitical reason to do something.
Lesley Carhart

Since they’re not doing anything malicious right now, they’re especially hard to detect. And detecting is so much harder in older systems with no EDR and maybe not even any PowerShell logging. Sometimes they’re detected when a company gets acquired and the new company has an industrial cybersecurity team and brings in a consultant to check the systems. But maturity is improving, too. Companies are starting to build detection and add monitoring. Sometimes they find really old compromises. Lesley has been brought in for incidents that are six years old, and sometimes older, that were just recently discovered because the company finally put in some detection. Everybody is getting to the state of just starting to see how bad things are.

Dealing with Compromises

Industrial cybersecurity is a very different space to work in. It’s hard to deal with compromises when you can’t just turn off and replace all the hardware. You have to start with an incident response plan. Who you’re going to call, how you make decisions, and what matters are all different in OT. Containment and recovery are different, too.

The SANS Institute has a whitepaper, “The Five ICS Cybersecurity Critical Controls,” that Lesley highly recommends. It has five simple things you can do to start. Things like secure architecture, remote access control, having an incident response plan, basic vulnerability management, and actual detection on the network can start to tackle the problem. They get complex as an organization gets more mature. And it’s a long-term thing, not a quick fix. But those five steps are a good start.

If you want to secure older systems, you have to start with the fundamentals. What’s your network map? What computers do you have? Just start at the beginning and do a little bit. Don’t get overwhelmed trying to do everything at once. Add layers of deterrence to keep adversaries out and detection to spot them before they do something. They can stay in a system for decades before there’s a geopolitical reason to act, so you might have a lot of time to detect.

Just start somewhere, start with the basics, and don’t get overwhelmed by trying to jump into everything at once.
Lesley Carhart

The U.S. Department of Energy has a framework called C2M2, which is cybersecurity maturity modeling. It will let you rate where you’re at in a self-assessment similar to an Excel sheet. Rate yourself in terms of your industrial cybersecurity capabilities, and it will show you where you are in maturity in general and where you compare to other organizations. It’s a good place to start if you don’t know where to begin.

Getting Into Industrial Cybersecurity

Lesley does a lot of mentorship, and she tells young people thinking about getting into cybersecurity that the market is really bad right now. But legacy and OT spaces are good to get into because nobody learns how to do that in college. Computer science programs don’t have time to cover old tech. Everything from banking to industrial environments uses legacy tech, and when people retire or leave, there’s a lot of need for new staff. It’s challenging to hire people for these roles.

For the most part, you don’t need a lot of experience to get started. Often these roles are looking for juniors. It’s ideal if you have some experience in a relevant environment. It doesn’t have to be in a cybersecurity capacity. Working in a shipping facility or on a farm with ag tech is also great because it helps you understand what’s important in these environments and get the right mindset for industrial cybersecurity. Learning how to use old computers is also really helpful.

Unfortunately, younger generations don’t necessarily know how to build computers or set up networks because everything works out of the box. That’s not your fault, but you need that knowledge to do this job. It’s not hard to get your hands on old tech and play around with it. Even getting virtual machines will help. Lesley has run into people coming out of cybersecurity degree programs who don’t understand how packets work. That’s concerning – and they can’t do this job. Be cautious, study, and understand what you’re missing in the foundations.

Interesting Stories from Industrial Cybersecurity

The scariest incidents are ones you never hear about in the news. And there are lots of reasons critical infrastructure providers don’t talk about incidents when nothing is publicly visible. Lesley has found adversary groups in water treatment facilities, traffic light controls, hot metal smelters, chemical plants, biopharma, and more. There’s a lot of scary stuff with peoples gathering capability to do something in the future.

The funniest story they can tell is the time they were called in because a power plant turned itself on in the middle of the night and people were concerned it might signal an attacker. After a day of frantic forensics and everybody freaking out, Lesley finally found the culprit. There was an extra touchscreen computer in a shed that everyone had forgotten about it. Bugs landed on it all the time, and that night they had just happened to hit buttons in the right sequence to turn on the plant.

Almost everything that fails in an industrial context is still maintenance or human error, often from equipment failing. That’s expected in industrial environments. Belts break, sensors die, even parts of computer equipment fail over time. And sometimes it’s things like copper theft or nearby construction accidentally cutting a cable. There are a lot of reasons things fail. But there’s a growing small percentage of intentional attacks, and a lot of preparing for future ones.

Lesley Carhart is terminally online on most social platforms. You can find all the links – including Reddit, BlueSky, Mastodon, and Instagram – on their blog, tisiphone.net. They do a lot of educational content, as well as free mentorship that you can sign up for on their blog.