Building a Bridge Across the Divide: The Intersection of IT and OT Cybersecurity with Mike Holcomb

John Vecchi:

Hello, everyone. You’re listening to the IoT Security Podcast live on Phosphorus Radio. I’m your host, John Vecchi. Today I’d like to introduce a very special guest who I’m sure our listeners know and follow. He’s a fellow of Cybersecurity and the ICS/OT cybersecurity global lead for Fluor, one of the world’s largest engineering procurement and construction companies.

As he says in his profile, his current role gives him the opportunity to work in securing some of the world’s largest ICS/OT environments, from power plants and commuter rail to manufacturing facilities and refineries, all of it. He has his master’s degree in ICS/OT Cybersecurity from the SANS Technology Institute, and he holds and maintains about every ICS/OT certification acronym you can think of. Welcome, Mike Holcomb, to the podcast today. We’re so happy to have you on, man.

Mike Holcomb:

Hi, John. Yeah, no, thanks for having me. I appreciate the invite.

John Vecchi:

This has been great. We’ve been working to get you on. Obviously we’ve had this podcast going for a bit, and we’ve spent a lot of time, Mike, talking around OT/ICS, that whole area. Obviously we cover a lot of other industries, and it’s a mixture. Although the podcast is the IoT Security Podcast, we talk a lot about the OT industrial side and industrial internet of things that … We tend to call it extended internet of things only because, as you know, there are so many of these devices and they span the enterprise IoT devices all the way through to the OT industrial, and even IO MT, medical devices and healthcare. Just about in any of those environments, you can have about all of these devices that are mixed together, right?

Mike Holcomb:

For sure.

John Vecchi:

So it’s really fantastic to have you here, and there are so many things I want to get to today. But I have to start with just your journey. I think we always like to just hear how practitioners and experts and influencers like yourself get to acquiring all the knowledge to focus in this area. As you know, there’s just not enough of you and practitioners out there that focus on this.

And so, it’s always fascinating to just hear your journey. But yours is particularly interesting because you talk about this a lot as the way you started. Probably, I’m guessing, if you look back to where you started, you wouldn’t imagine you would be where you are today, and having the followers and all the influence you have.

So for our listeners … And you could probably spend the whole podcast talking about that journey, but can you give us a little bit of an overview of that and how it all came about and how you ended up where you are here now?

Mike Holcomb:

So I’ve been a computer nerd/cybersecurity nerd since day one. I actually grew up with my best friend. He was the traditional hacker, Kitty. He would do things like crack games so you could copy them or write cheats for games so you get unlimited gold or lives, that type of thing.

And so, I was always really fascinated with that, and he was super nice guy. I always wondered, though, what could somebody with evil intent do with that type of knowledge or those types of skill? That just always stuck with me. And so, I’ve just always been fascinated with cybersecurity really from the get-go. Over years, I’ve worked in pretty much IT the entire time. So I come from the IT side, and I talk about my journey to OT started with Stuxnet in 2010 and Stuxnet was announced. This was where the Americans and the Israelis had teamed up to target with code these uranium centrifuges that we had in Iran, in the Tans. Technological marvel. As more details came out about it, I was just amazed.

So I always talk about Stuxnet was really the first thing that made me get serious about OT, which is probably true. But the more I think about it, there were two other things to share real quick. One was Hackers came out in 1995. For those of us probably over 30 or so that have at least seen the movie once, that it’s an ICS/OT cybersecurity movie.

John Vecchi:

It is.

Mike Holcomb:

It’s all about taking over systems and making things happen in the world. Some of it’s a little stretch like setting off the sprinklers in the school. But being able to take it over at that time a TV station and be able to turn the show that we’re watching to something else, it was very, very possible, and lots of other things in the movie.

So I think that probably was also starting to grease the wheels. If I go back even further, while I was actually going to school, I worked in a bowling alley. I was the one guy, they’re like, “Oh, you know computers.” So, “Hey, we’ve got this Windows 95 machine. We need you to come install this software.” The software basically tied in with all the systems that did all the money, but also turned on the different lanes. So you’re turning on the bowling lanes, the scorers, all things that are moving in the real world and told to do so by a computer. So not as much control, but it still gets me down that road.

I actually spent about a year working in the back, so I was a wrench monkey as they called me, where I was sitting in the back and doing things like changing motors and belts and solenoids and everything electrical and physical. So I think all of that had really geared me up towards being in ICS and OT cyber today.

John Vecchi:

Yeah, and I mean, boy, even back in the bowling alley days, I mean there you go, cyber-physical systems. I mean that was the dawn of we call that cyber-physical today. That’s an amazing journey.

Like you said, you came from the IT side. I always find it interesting to just hear your experience. I mean from the IT side and then moving over to the OT side with operators, what was the biggest difference? I mean what was the most difference to you when you started moving from the IT? I know that we talk about the convergence, and there’s all of that today, but what was the most startling thing in moving over to the OT side from all those years in IT?

Mike Holcomb:

I think the biggest thing for me was because I started making that transition and being at Fluor and learning about control systems in the real world and talking with engineers. We have some of the best engineers in the world that I get to work with on a daily basis, and we talked a lot. I asked a lot of questions.

Probably a year, year and a half goes by, and then all great in practice, read a lot of books, watch a lot of videos, talk with a lot of people. Then actually went on site for my first project. It was actually a very large power plant. It was a hybrid plant that is powered based off on natural gas, but it’s one of the largest power plants of that kind in the western hemisphere.

I step on site and … There are a lot of stories I could tell, but it was just the fact that … And part of the plant was operational, so it was starting to generate electricity. Just being there and having electricity flowing in the air and through you. It was just this really amazing experience.

Then to have the plant manager walk me through not only the data center and the control room, but where the turbines. Then you walk into this four-story building. You walk into the top and you just walk in and this entire building is taken up with the generator. It’s just this massive monstrosity of engineering genius.

John Vecchi:

Wow.

Mike Holcomb:

To sit there and look at this thing was just amazing. I find that very true in a lot of our projects because a lot of our projects, I get involved in the early design stages. So we’re talking about things that are going to be built in four or five or six years down the road. And so, it’s really interesting to be planning cybersecurity around these things that still in my mind are very small and little and then, in all reality, they turn out to be very, very large. But, yeah, just being on site really is what made it hit home for me.

John Vecchi:

Yeah, I can imagine. Is that common, by the way, to have that advanced planning today for those types of facilities? Gosh, I mean you look at … Obviously a lot of these facilities are very old-

Mike Holcomb:

Yup, for sure.

John Vecchi:

… the devices are old, the technology’s old. But it’s encouraging to hear that there’s planning happening that far in advance from a OT cybersecurity and, I guess, even IT/OT converged security that far out. Is that relatively common, do you think?

Mike Holcomb:

It’s a lot more common these days. If we were having this conversation a couple of years ago, it wouldn’t necessarily be the case. For myself, I think with the clients that we work with in my day job at Fluor prior to the Colonial Pipeline breach, not every client wanted to have that conversation. Most did, and we work with some of the largest companies in the world, with some of the best cybersecurity programs in ICS, like a Shell or a Saudi Aramco, and we get to work with them, which is really amazing.

There’s other customers that, “Oh, we got it.” Okay, that’s great if you do. I really question whether they were doing anything about security or not. But that was their choice.

But when Colonial Pipeline happened, I think that really opened up a lot of people’s eyes when you’re talking about owners and operators, the people that own these facilities and run facilities, that they really started to understand not only are we a target, but there’s a real-world impact, whether … We always talk about the number one concern that we’re always worried about in ICS or OT cyber is safety, making sure everybody goes home safely at the end of the day.

We also worry about what could happen to the environment in which that facility resides. Then we want the facility to stay up and running. So we want that power plant to continue to generate electricity 24/7.

When Colonial Pipeline hits, prior to Colonial Pipeline, all we used to talk about were state adversaries coming into your environment. It’s Russia, it’s China, it’s the United States coming into the environment to steal your data, steal information about how that plan operates, and then potentially position themselves to where they could make something happen.

When Colonial Pipeline hit, it was the United States’ largest gasoline pipeline going down, not from a state adversary but a ransomware group. They were just sending out emails basically to every email address they had of everybody in the world trying to get somebody to click on a link or open up an attachment and infect their systems. Because IT burned down to the ground, then it had that impact into OT where they had to shut down the pipeline. There was that impact, that real-world impact. That’s where it did open up a lot of eyes out there.

John Vecchi:

Yeah. You were saying, which is fascinating, that literally in the past couple of years, the needle has moved quite dramatically. As you said, if you had the discussion even as currently as a couple of years ago, it’d be very, very different. Was something like Colonial Pipeline … Was that a turning point? In other words, was that one of the impact events that had a big impact on moving that needle? If not, what were some others over the past couple years in your eyes that have really moved that needle?

Mike Holcomb:

Yeah. Colonial Pipeline to me was the one. Being a long-time IT cyber professional, that to me, I liken it to Target, the Target breach, because for me there’s still a lot of companies out there that weren’t taking cybersecurity seriously from an IT perspective. Then all of a sudden the target breach happened. Everybody had to replace their debit cards or their credit cards, and they’re like, “Oh, we should probably do something about cyber back at the office.”

John Vecchi:

Yeah, yeah.

Mike Holcomb:

I think that’s very true, and a lot of folks can probably relate to that in the IT world. Colonial Pipeline was the target event for ICS/OT. Prior to that, there were other events. Probably Stuxnet was definitely a big turning point for many reasons. One, it was a state adversary targeting another state adversary. That was the first known case using a cyber weapon. Has it happened before that? Probably. Not that us mere mortals are ever going to know. But that’s the first known case.

Then at that point also is a big turning point because you had other states, adversaries looking at, “Well, what happened to the Americans and the Israelis when they did this? Nothing.” So Russia, China, these other countries are like, “We can do this too and nothing’s going to happen.”

John Vecchi:

Right.

Mike Holcomb:

At the same time, they saw Stuxnet and realized, “Wow, that’s an incredible piece of work. We’re probably really way behind the game. We have to step up,” which they have, right, over the years.

John Vecchi:

Yup.

Mike Holcomb:

So Stuxnet definitely was a big one, but that was more, I think, for folks like us already in IT, in the general community. It wasn’t more for the general public. The other one that … 2017 was when the Triton or the Trisis incident happened. This was one where everybody in the industry were, “Oh, this is the one that’s going to wake everybody up. This is the one that’s going to make them take ICS/OT cyber- “

John Vecchi:

Seriously. Right, yeah.

Mike Holcomb:

This was it because this is where the Russian state had gone in and they had taken complete control over a petrochemical facility in the Middle East. They had pointed out that they had taken over the safety instrumented systems or what they call SIS in the plant. The SIS is the fail-safe backup. So if there’s ever a fault condition detected in the environment that could cause damage to the plant or could cause a safety issue, somebody could get hurt or killed, and so it would shut down the plant nicely, and then things can be resolved or fixed before something bad happens. So they had taken over control over the SIS.

Rob Lee, for those who are not familiar with Rob Lee, he’s one of the co-founders and the CEO of Dragos. Dragos is the 800-pound gorilla-

John Vecchi:

Dragos, yeah.

Mike Holcomb:

… of incident response and other things like threat intel and network incident detection in the ICS/OT world. Rob was involved in that case with others, but he always focuses on the fact that the only reason an attacker would take control over the SIS is to kill somebody, period, the end.

John Vecchi:

Damage, yeah.

Mike Holcomb:

He said, in that facility, I think there were over 12 ways that they could impact what we call the process, how the systems control things that move in the world, that they can impact the process to cause something like an explosion that would kill somebody. So we thought that would’ve woken everybody up, but it honestly didn’t have much traction.

It took Colonial Pipeline happening where, again, it wasn’t a state adversary. It was just a ransomware group. And there was a real-world impact that people could see, especially here in the United States.

So I’m in Greenville, South Carolina. If I went to the gas station when the Colonial Pipeline was down, then there was no gas at the gas station. I could go around the corner, there’s another gas station. They had gas. Super long line, right?

John Vecchi:

Yeah.

Mike Holcomb:

Prices were a little elevated. Not the end of the world. But it really was a big impact that people saw all up and down the eastern seaboard, that all because of a ransomware attack gone awry, that all of a sudden we’re stretched thin on gas on the entire east coast of the United States. I think that said a lot and that impressed upon many people that it’s time to at least start thinking about taking ICS and OT cybersecurity seriously.

John Vecchi:

Yeah. Well, that’s exactly right. Just the other day, Mike, you posted, which was so interesting, and I spend a lot of time talking about this timeline as well, the timeline of just the types of attacks we’ve had since Stuxnet. If you just look at literally the past year, maybe you go out to, like you said, maybe a couple years, but, man, you take the past 12 to 18 months and you see the level of activities and the sophistication of attacks. You laid that timeline out from … And we’ve talked about some of them now, the Stuxnets and then the Industroyer series. You have Industroyer1, you have Industroyer2, you talked about Triton. You’ve got Pipedream, which I call a Swiss army knife of just almost push-button simplicity to go attack these types of devices.

Of course, I wrote a blog recently about FrostyGoop, you listed that, which now is targeting Modbus protocols and nothing as far-reaching implications. Then Fuxnet, of course. That one’s interesting because they’re tacking the gateway. They’re now looking at some of the things I spend a lot of time talking about, which is default credentials and underlying just vulnerabilities and default configuration supports and protocols wide open.

Just the underlying configurations on these devices is nothing … It’s just as the device was intended to be used, I guess you could just say, but they’re these sophisticated attacks and, as you mentioned, these living off the land approach. It’s not that there’s some even a fundamental vulnerability. They’re attacking the device in the way … Leveraging in a bad way oftentimes the way these devices are meant to be used.

Mike Holcomb:

Yeah, [inaudible 00:20:09].

John Vecchi:

That’s very eye-opening. You just see that timeline. As you said, it’s just … Boy, just take the last year. This has been a dramatic shift. We know, and you’ve mentioned this, it’s not just nation states. It’s a host of threat actors. It’s ransomware gangs. It’s hacktivists. The Pennsylvania Water facility. They wanted to make a point, I guess you could say, right?

Mike Holcomb:

Right.

John Vecchi:

So when you look at all of these … I think it was a very informative post to just show this. We know these devices are vulnerable. They’re often many … Like I said, talk about 1994. That’s where you feel you are when you look at the fundamental hygiene of a lot of these devices today. When you look at all of that in your practice and all that you spend your time focusing on, what is the best way to get that message across to operators who, as you often say, are just … They’re overstretched. They don’t even have budget to do this. They’re small. They don’t have teams. What is your most effective way to get … And in addition to posts like the one you put out, but to get that message to these operators?

Mike Holcomb:

Yeah, there’s a couple things. It is for people to be aware. Yeah, most people don’t realize the vast majority of critical infrastructure in ICS/OT environments. When we say critical infrastructure, we’re just talking about the ICS or OT environments that provide things that we need in our daily lives, like clean water or power.

And so, I always liken power to finance in the IT world, that they’ve been doing cybersecurity for quite a while. They’ve typically got the budget. They have teams. We don’t worry about them too often. It’s everybody else like water. There are a lot of small mom-and-pop water suppliers, water, wastewater, that they don’t have somebody thinking of OT cybersecurity, let alone OT. They might not even have an IT cybersecurity or an IT person on staff.

So that gives you this idea of, oh, when we see … You mentioned a lot of the water plants that were getting hit because they had devices directly exposed to the internet with default passwords, right?

John Vecchi:

Mm-hmm.

Mike Holcomb:

Yeah. You can start to understand why they had these assets connected to the internet with default passwords. They just don’t have the resources, let alone that knowledge and experience.

John Vecchi:

For those operators, how do you get that message to them? I think it takes time. But they may see it in many cases, but it’s very difficult for them to do much about it, just mostly because they just don’t have the skills. They don’t have the skillset or the people or the resources and oftentimes, as you said, even budget. There’s no particularly any budget many times that says this is specifically for ICS/OT security, IoT security. It gets the second chair in many cases, right?

Mike Holcomb:

Yeah, I think you have to realize that we’re way behind the curve, and even though … This is another thing that Rob Lee really was focused on talking about this year. You can tell I always listen carefully to what he’s sharing, and others as well. But he talked about if you look at an entity like, let’s say, a manufacturer, 95% of their budget goes to IT cybersecurity and 5% goes to OT cyber, even though if you think about it, what that plant manufactures, that’s actually how the company makes money. And so, it’s really this strong disconnect.

John Vecchi:

It makes the money. It makes the bottom line, yeah.

Mike Holcomb:

So when I do work with folks in my day job, I do little consulting on the side, mostly with manufacturing because that’s what we have here in the Carolinas. I think the biggest thing I can do to have an impact is really, believe it or not, to do a briefing, to have a sit down with everybody that I can get to listen to me and talk about, “Here’s the general state of cybersecurity. A little bit in IT, but here’s the general state of cybersecurity in OT.” Then you can drill it down even further to, “Here’s what the cybersecurity landscape looks like in environments just like yours. What are the risks? What are the impacts?”

Oh, you’re in manufacturing? Did you realize that there were almost like 900 ransomware attacks that were successful in manufacturing environments over the last year and that the average downtime was 10 days? How much is that going to cost your organization versus you don’t actually have to spend a bajillion dollars to secure your plant?

Defense is very doable in ICS/OT. These environments are much smaller than traditional IT environments. They’re static. And so, defense is much more doable. Another Rob Lee quote. So he always talks about defense is doable in ICS/OT, and it really is, but people just aren’t investing the time and the focus in securing it. Again, that’s definitely changing, and it’s been changing since Colonial Pipeline, which is great to see.

John Vecchi:

Yeah, there’s no question. I think it was just the other day, that was one of the things you were talking about is like I know, it’s understandable oftentimes for a lot of smaller utilities or critical infrastructure providers and organizations, they don’t have the teams. But as you were saying, sometimes just focusing on the basics. You talked about a lot of those. I mean let’s hit some of those. You’re talking about the architecture, your overall network architecture, some simple things you can do there. Obviously we spend a lot of time talking about the visibility, the asset inventory. How do you even know where it is? There are so many of these types of devices, right?

Mike Holcomb:

Right.

John Vecchi:

You also mentioned things like incident response and your backup and your recovery program. Then, of course, secure remote access, which is becoming more prevalent. I’m not sure if you’re seeing it. We’re seeing more focus on that. As you said, just you’ve got …

Mike Holcomb:

For sure.

John Vecchi:

You always have vendors and others that have access to these devices. Oftentimes that’s a requirement for the device. But you also have hosts of different workers and maybe third parties that are accessing these devices. You have things like secure remote access. I mean so it’s a great list, right?

Mike Holcomb:

Yeah.

John Vecchi:

Do you see those smaller organizations … And, look, what do you think about even bigger organizations might not even be doing all of those things? Is that fair to say?

Mike Holcomb:

Yeah, no, it’s very true, because you can see everything … Small, large, everything in between. So many people, they forget about or they just don’t know about just the basic fundamentals. It’s the same in IT. Before you look for the latest shiny thing or want to talk about AI and zero trust everywhere, it’s do you have a firewall? Is it configured correctly? Just the basic fundamentals can go a long way in reducing your risk.

If you can nail those fundamentals, then as you have time and money and the ability to focus on other things, great. You hit that maturity level and you’re ready for the next step. Awesome. But so many environments of every size, or they’re trying to look for either the silver bullet or they’re focused on the latest shiny thing, and they’re missing all those basic fundamentals like you talked about, right?

John Vecchi:

Yeah.

Mike Holcomb:

I always start with secure network architecture because that’s going to have the most impact to reducing risk, and it’s going to cost you the least amount of anything you’re going to invest in. So why wouldn’t you?

John Vecchi:

Yeah.

Mike Holcomb:

That type of idea. And so, yeah. And building from there. You mentioned the asset inventory, making sure that you know what you have in the environment so you can protect against it. You also need that information when you do things like vulnerability management in ICS/OT, because you’re not going to sit there with something like Nessus and be able to scan the entire network for vulnerabilities, right?

John Vecchi:

Correct, yup.

Mike Holcomb:

You need to understand, “Oh, I have this system,” maybe like a PLC or a programmable logic controller, and that’s a custom … Essentially if you want to … It’s a computer that is connected to things in the real world that makes them move. I always liken it to a thermostat, right?

John Vecchi:

Yup.

Mike Holcomb:

Your thermostat, you set it to a certain temperature. If it gets too warm, the computer, the thermostat will send an electrical signal to your air conditioning unit to turn it on. Then the temperature gets good, you turn it off. And so, when you have these PLCs, do you know what version of firmware it’s running, right?

John Vecchi:

Yeah.

Mike Holcomb:

What vendor is it? What model? So you can use that to look up any known vulnerabilities. Though, honestly, the biggest gap I always see there … And this is another one of those, it doesn’t matter whether it’s a small or large environment, is if they don’t know they have that device, or if they do, do they know where the spare device is? Because what if something happens to that device and they physically need to replace it?

John Vecchi:

Yeah.

Mike Holcomb:

I would say most environments, believe it or not, they don’t even know if they have a spare.

John Vecchi:

Correct.

Mike Holcomb:

If they do, they don’t know where it is. They’re relying on, “Oh, we can just go to our supplier and get a new one within four hours.”

John Vecchi:

Yeah.

Mike Holcomb:

What if something happens to the supplier? What if there’s a run on these types of devices, which really could happen? Yeah. So there’s a couple … Some of those are some of the areas that … Again, it’s nailing those fundamentals before you start worrying about other things.

John Vecchi:

Yeah. I mean talking about the devices a little bit too, I mean you mentioned PLCs. I mean I think there are so many and it’s so … That PLC, when you mentioned where is the backup, where is the replacement, they oftentimes don’t even know where the main one is or how many they have. If they have it, like you said, what model and series is it? What’s the firmer version? What’s the general risk assessment of that thing? Is it running to … Is the key switch positioned in program or remote? You know what I mean?

Mike Holcomb:

Right.

John Vecchi:

There are all of these kinds of things they don’t know. I think what we see, Mike, is just so many … I call them OT-adjacent IoT devices. So obviously in the OT, we think in terms of the PLCs, as you mentioned, the HMIs, the RTUs, the SCADA systems. You have robotics. You have all these types of very OT, classic OT, but you also have these adjacent devices and you have sensors, you have gateways, you’ve got HVACs, you have built door control, access controllers, and cameras and ruggedized printers. To an attacker, that oftentimes can be very low-hanging fruit, right?

Mike Holcomb:

Very much so.

John Vecchi:

And so, do you see that as a challenge … I mean obviously visibility is a challenge, and even not only knowing that you have the device, but, as you said, what is the device? What model is it? Is it a Siemens S7 PLC or is it a Schneider? I don’t know. There’s all these different times. But then you have all the other devices that sometimes get overlooked. How do you see all the various devices? Do you worry about all of them, or do you worry about certain devices a bit more? What’s your take on that?

Mike Holcomb:

Yeah, no, that’s a great question. If you look at an OT environment, like if I go into a manufacturing plant, and a couple of things that people aren’t probably aware of … And I was just talking about this at a conference, because I was talking a little bit more about ICS/OT pen testing, and I appreciate you mentioning it, from the attacker perspective. I’m always thinking from the attacker perspective, what are they going to see? What do I have to protect against?

One thing that probably people don’t realize was you go into an OT environment, whether it’s a power or a manufacturing facility or petrochemical, and the list goes on, you have a lot of Windows servers.

John Vecchi:

Oh man, yeah.

Mike Holcomb:

You’re like, “Oh, here’s our active directory domain controllers.” I’m like, “Oh, wait, you have those in OT?” Right?

John Vecchi:

Yeah.

Mike Holcomb:

It’s like, “Oh, here’s all of our Windows servers that run all the software for running the plant.” It’s like, oh, okay. So if you have an attacker get into the OT environment that’s already well-versed in how to take over Windows systems, they’ll just run rampant through it.

John Vecchi:

Oh, 100%. Yeah.

Mike Holcomb:

Then there’s also the danger if ransomware starts probably on the IT side and it gets right through the DMZ you have between your IT network and your OT network. So it moves from the back office into the plant environment. It just takes off a lot of times like wildfire as well.

John Vecchi:

Yeah, yeah.

Mike Holcomb:

Yeah. So not only do you have these Windows systems … And my favorite is what we call a data historian that some of your listeners might be familiar with. Basically it’s just a Windows server usually running Microsoft SQL Server, probably has some in patch, sits … Probably there’s one in the DMZ, could be wide open from the IT network because it’s used to store data about how the facility is running. If I have a manufacturing plant, I need to know how many widgets did we manufacture today so we can coordinate things like shipping and logistics? We’d also like to get paid, I’m sure. And so, we need that data sitting somewhere.

And so, those types of systems are really easy for attackers to move across and move laterally and pivot through because they’ve just done that forever. But then you also get into an OT environment with those more OT types of assets like PLCs. We also mentioned HMI. HMI are the human machine interface. It’s just a GUI that shows you what the process looks like, and then people can interact and get alarms. They can go in and maybe shut down something if they need to.

John Vecchi:

Yeah.

Mike Holcomb:

And so, you find these different systems. This goes back to one of your earlier points is that it doesn’t matter whether these systems are 30 years old or they’re ones being implemented today, that they’re still based off of technologies that were made 30-plus years ago.

John Vecchi:

Right, yup.

Mike Holcomb:

So you look at something … You mentioned FrostyGoop. FrostyGoop was discovered by Rob’s company, Dragos. FrostyGoop was used by the Russians to turn off the heating for 600 apartment buildings-

John Vecchi:

Ukrainians. Yeah, amazing.

Mike Holcomb:

… in the middle of winter.

John Vecchi:

Winter, yeah.

Mike Holcomb:

Sub-zero temperatures for two days. They were allowed to do that because two things. One was they actually were able to download the firmware on the PLCs, which was interesting. So it probably goes back to your do they have the switch in run motor?

John Vecchi:

Right.

Mike Holcomb:

They probably did not in that case. Then they used the software, which I actually did this for my ICS Village talk at DEFCON this year, was … And with Rob’s permission, I recreated FrostyGoop-

John Vecchi:

That’s awesome.

Mike Holcomb:

… in 15 minutes or less using ChatGPT.

John Vecchi:

Jeez.

Mike Holcomb:

And so, here you have … You can say it’s a state adversary tool, but really-

John Vecchi:

It’s pretty simple.

Mike Holcomb:

… it’s just that simple because that PLC uses Modbus. So we talk about Modbus being the most commonly used OT protocol out there, and it was invented probably 30-plus years ago. I’ll have to check the date.

John Vecchi:

Mm-hmm.

Mike Holcomb:

It’s unauthenticated, right?

John Vecchi:

Yeah.

Mike Holcomb:

It never asks you for a username. It never asks you for a password. It just says, “Oh, you want to read information? Great, here you go,” “Oh, you want to update?” It’s like SNMP from the IT world. So it allows you to read information that they have in the database, which is always values. You can read them, you can write them, you can change it. And so, essentially you can do whatever you want with that system as long as you understand you have to take the time to reverse engineer and understand the process.

So when you look at all these numbers, you have to understand, what do those numbers mean? So that way if you make a change, it has that intended consequence or impact that you wanted. That’s where attackers get hung up, because they can get into your IT network.

We mentioned there’s Windows systems everywhere in OT as well. So they can take over all those systems as well. Then they use the data off those systems to reverse engineer what’s really going on in the plant with all these OT systems. Then they can impact, they can make those changes to really achieve their mission goals, to have that impact. But they have to figure that out.

That’s what takes them the time. So if we were doing things like network security monitoring while they’re sitting there trying to figure things out, we should be catching them. The problem is, and this is another thing Rob does a great job of highlighting, is that less than 5% of OT environments around the world are doing network security monitoring today.

John Vecchi:

Right, right.

Mike Holcomb:

I would argue probably half of those probably aren’t doing a good job at it. That’s very concerning on many fronts, but yeah. If we were looking, we would catch the attackers and we would be able to stop them before they could do something in OT that would have that impact, that could kill somebody, or at least hurt somebody or do harm to the environment, let alone bring production to its knees.

John Vecchi:

Yeah, that’s a really good point. What are you seeing as the appetite … I mean obviously at Phosphorus … Gartner calls it cyber-physical system protection platforms. It’s a new category and it’s all in there.

Mike Holcomb:

Thanks, Gartner.

John Vecchi:

Yeah, it’s yet another category in name. But I mean we spend a lot of time focusing on a different approach, obviously doing something which has historically been considered an allergic reaction, which is actively polling. So unlike, as you mentioned, the traditional Nessus, and those can be very harmful. They’re not really made for OT. They’re made for IT systems. They’re not really made to … I call it waterboarding if they go and talk to those devices that way. It’s truly not meant for the-

Mike Holcomb:

That’s a good way to put it.

John Vecchi:

Yeah, and now looking at how can we safely, actively poll and speak directly to the device in its native protocol? So now we can find it, we can assess it, we can understand, as you said, what its model and series and make and what its assessment is. Is the key switch position in program as opposed to run, things like that?

Mike Holcomb:

Sure.

John Vecchi:

So we’re seeing maybe a little bit of more open mindedness to say, okay, there might be a way to do this. But actually one of the things we focus on is actually now looking at the fundamental hygiene of these devices and saying, “Look, we cannot … ” Like I say, there’s the now, next, never bucket, and we know there are never buckets in PLCs. We may never even ever think about actually patching those devices.

But there’s a lot of different devices. So we think of … If many of these devices can safely be addressed from a hardening perspective, let’s rotate the credentials. Let’s get rid of those default credentials. Let’s shut extraneous services off. If telnet’s wide open, let’s shut that off, and where we can, we can patch.

What kind of change in appetite are you seeing in certain organizations to actually address some of these fundamental hygiene issues, or maybe even just back to actively polling to discover … Get a richer depth of visibility? Is there more of an appetite in your mind for that now or is that needle moving?

Mike Holcomb:

There is for sure, and we’re in a better position today as technology evolves and as mindset and knowledge evolves.

John Vecchi:

Yeah.

Mike Holcomb:

Now I have to stipulate that I’ll never suggest anybody does active scanning in their environment, but there’s a big asterisk there. I think everybody has to make the decision that’s right for their environment with the knowledge that they have of their environment and what works for them.

Yes, you used to say 30 years ago, you couldn’t walk into a plan and run a port scanner without taking old PLCs, where they would crash to where you literally physically had to replace them.

John Vecchi:

Correct.

Mike Holcomb:

That’s true. They weren’t designed to understand garbage traffic. They only understood a few commands. If they got garbage traffic, they just-

John Vecchi:

Die.

Mike Holcomb:

… cried.

John Vecchi:

Yup.

Mike Holcomb:

But I also had a CISO for a large ICS program, and this is outside of my day job, but we talked about, “We have a PLC. If you scan that with Nmap, it would leave a crater in the ground three miles wide.” And he was not kidding. It was like, “Wow.” So it just depends, right?

John Vecchi:

Mm-hmm.

Mike Holcomb:

In 2024, more and more systems. It’s Windows. Am I worried about scanning those Windows systems even with something like Tenable?

John Vecchi:

Yeah, no.

Mike Holcomb:

Probably, probably not. Probably. Again, you have to make the decision for your environment. I can’t tell you you can’t. But it’s Windows, great. You see hardware these days, the vendors have done a really good job of updating the hardware to understand, “Oh, if you get garbage commands from over TCP/IP, just ignore it. Don’t go all crazy. Just ignore it. Just drop it.”

John Vecchi:

Yeah.

Mike Holcomb:

So you do get … Just like you were mentioning, are we still probably doing full active scanning in an environment? No, even though attackers could be, right?

John Vecchi:

Yeah.

Mike Holcomb:

But we also know nobody’s looking for the attackers in the environment most of the time. So they could be there scanning away, you’re just never going to know it. But that you do see active polling as this happy medium where people, it’s like, oh, okay, you can go out … And you mentioned like a Siemens PLC. Siemens PLCs, it might sound strange. They come with SNMP-enabled by default, with a default community string. So do you pull for SNMP? You could. Do you ping every possible IP address on your network? You could …

Again, you have to know what’s safe for your environment, but most environments today, yeah, you can do those basic things to start to understand what’s in the environment, and then you can build from there. Like you mentioned, looking to see, is that PLC is the key in run mode or not. So that way if it’s in run mode, we see it as mostly secure. So maybe at least an attacker couldn’t do something like downgrade the firmware, which we’re talking about-

John Vecchi:

FrostyGoop.

Mike Holcomb:

… that happened in the FrostyGoop instance. Yeah.

John Vecchi:

That’s a really good point, and-

Mike Holcomb:

Definitely. Yeah, so there’s definitely a growing appetite to do more.

John Vecchi:

Mm-hmm. And that’s good. I mean-

Mike Holcomb:

Not going crazy yet, but definitely growing.

John Vecchi:

Right. It’s slow, but sure. It’s slow, but sure. But I mean I think it’s good to hear you from your perspective and all the different facilities that you have your hands at. You’re seeing that needle moving a little bit, right?

Mike Holcomb:

For sure.

John Vecchi:

I mean, like we said, that certainly the attackers are doing whatever they want to do, and there’s no question they’re focusing on these devices. Anything we can do to move that needle a little bit is a really good thing. I know that’s what you preach a lot of the time, right?

Mike Holcomb:

True. Yeah, I do. Yes.

John Vecchi:

So, Mike, as we wrap up, I mean, again, this has been amazing. We’d love to stay in touch with you and have you on again.

Mike Holcomb:

Yeah, definitely.

John Vecchi:

But as we look at wrapping up today … And you mentioned there are some from-the-trenches stories. I mean is there any kind of specific experience story that you’ve experienced that could lead to learnings that would be beneficial for our audience to hear from you?

Mike Holcomb:

Sure, yeah. To me, the biggest one, it’s not super exciting from … I don’t know. I always think of almost like a James Bond movie. I think that’s … A lot of people think of ICS/OT.

John Vecchi:

Mm-hmm.

Mike Holcomb:

It sets a true example of some of, really, the biggest problems we have today. And so, I actually got a call from … This is one of the best things about starting posting on LinkedIn a year and a half ago, because I get to meet people from all over the world and get to talk with them.

I had a CISO for one of the United States largest control system environments, and this was outside of my day job. We don’t work in this sector. But he had called me up, he had been on the job about six months. So he was responsible for IT and OT, and had called me up and had talked about that they have a SOC and their SOC watched IT as well as OT. If the SOC got OT alerts, they send them to the OT team to investigate. I’m like, “Okay, makes sense.”

He said, “But we’re sending over these OT alerts and we’re not hearing back.” I was like, “What do you mean you’re not hearing you back?” He’s like, “We just don’t hear anything.” I was like, “Well, have you asked them what’s going on?” and he said, “No,” which I thought was a little strange, but I get it. Super nice guy, brand new on the job, didn’t know how to, I think, work with the OT folks. And so, that’s one big problem that we have is really it really comes down to IT people and OT people being able to work together.

John Vecchi:

Yeah.

Mike Holcomb:

In this case, yeah, they actually did go to the OT folks and said, “Why aren’t we hearing back from you?” and said, “Well, we’re getting these alerts and it has these IP addresses, but we don’t know what these systems are. So how are we supposed to investigate or respond?”

John Vecchi:

Right. Right.

Mike Holcomb:

It’s those really two big things that are wrapped up. One is … And, again, he was brand new to the job, but he, I think, realized very quickly, like we mentioned earlier, they didn’t have a lot of budget or focus on OT cybersecurity. Again, one of the country’s largest ICS OT environments by far.

John Vecchi:

Wow. That’s so interesting. Yeah.

Mike Holcomb:

Yeah. And it goes back to it doesn’t matter if you come from the IT side of the house or the OT side of the house. It’s the same house that we’re trying to protect.

John Vecchi:

Correct, yeah.

Mike Holcomb:

We’re all in this together. We have to work together, because if … My girlfriend, she’s a … My partner is a therapist, and I always think of IT and OT as a relationship, whether you’re married or you have a partner. If you’re not communicating, like in their case, or if you’re just always at each other’s throats, nothing’s getting done and the attackers are winning, right?

John Vecchi:

Yeah.

Mike Holcomb:

We have to work together. Otherwise, the attackers are just going to continue to win.

John Vecchi:

Yeah, exactly.

Mike Holcomb:

It’s as easy as that.

John Vecchi:

Tremendously important point. Simple but so important. We see it time and time again, these are very disparate teams. They don’t particularly understand each other. Oftentimes they’re in silos. They’re not talking, they’re not communicating. It’s a great example of we need to bridge that gap. We need to make sure they’re talking and collaborating with each other.

Mike Holcomb:

Exactly.

John Vecchi:

That’s just a huge point. It’s a wonderful point. Thanks so much, Mike. So, Mike, look, I know our listeners, a lot of them, follow you already. For those who haven’t yet or haven’t found you, or want to reach out to you, as you said, I know you’ve got a variety of ways people can find you.

Mike Holcomb:

Sure.

John Vecchi:

Help our listeners find you. Where can they go to catch up with you?

Mike Holcomb:

Sure. Yeah, no, I appreciate that. The big one is LinkedIn. I’ve been there about a year and a half. So Mike Holcomb on LinkedIn. Just look for the bald guy with the blue background. That’s me. Feel free to message me, ping me. I get a ton of messages there, so it might take me a day or two to get back. But I do try to get back to everybody.

I do have a YouTube channel. You can just search Mike Holcomb, it’ll pull up. It’s actually called utilsec, which is my little side company, my little LLC. I thought it was cute. So utilsec. I actually have there … A lot of people find me through the 25-hour course I have on an introduction or getting started in ICS/OT cybersecurity. Now I’m just putting out the 10-hour follow-up course, which is really just OSINT and ICS/OT. So I had a lot of fun with that. So I have a lot of the content on YouTube. Then if anybody wants to, they can feel free to email me at [email protected] as well.

John Vecchi:

Wonderful. Well, you heard it, everybody, right from him. Please reach out. Mike, it’s been just amazing. Thanks so much for joining us today, man.

Mike Holcomb:

Yeah. Thanks, John.

John Vecchi:

We’d love to have you back.

Mike Holcomb:

Yeah, it was fun. Thank you.

John Vecchi:

We’d love to have you back, and we’ll do that again.

Mike Holcomb:

Anytime.

John Vecchi:

Remember, everybody, the IoT Security Podcast is brought to you by Phosphorus, leading provider of proactive cyber-physical system security and remediation for the extended internet of things. Thanks again to our great guest, Mike Holcomb. Until we meet again, everybody. I’m John Vecchi. We’ll see you next time on Phosphorus Radio.