Guarding the Gateways: Tackling IoT Vulnerabilities in Critical Systems with Joel Goins

Episode Transcript

John Vecchi:

Well, hello everybody. I’m John Vecchi. You’re listening to the IoT Security Podcast live on Phosphorus Radio. It’s so great you’re joining us today. We talk about a lot of different types of environments and industries on this podcast covering what we call the Extended Internet of Things, and we’ve discussed healthcare. We’ve discussed a host of environments and industries.

But as you know, our listeners, we’ve focused a lot on the manufacturing side, the OT side, the industrial control system side, and we are incredibly lucky today to have a very special guest. Today we’re joined by Joel Goins. Joel is the senior manager of global operations technology security at Johnson Controls.

Joel, we are so excited to have you. Thanks so much for joining us today.

Joel Goins:

It’s good to see you.

John Vecchi:

Exactly. We talk a lot when we talk OT and the manufacturing side and industrial control systems and all of that is we talk a lot about, and I know that you’ll relate to this Joel, that’s a small community. I mean, the number of security experts in this space is really small, and they’re precious commodity.

For our listeners, tell us a little bit about, I love to just ask, how did you get here? How did you arrive on the OT security side? Tell us a little bit about your journey there, if you would, and so our listeners can understand how you got to where you’re from here.

Joel Goins:

Sure.

Well, long story short, the majority of my career is in manufacturing, so about 20 years on the shop floor, running engineering groups, maintenance groups, operations groups with multiple companies across the US. At one point in my career, I was getting a little bit older. I thought, “I’m going to get a change.” This whole six, seven days a week, 10, 12 hour days was getting old.

And so I took about a three-month sabbatical and just reevaluated and thought, where do I want to go? And so I thought, I’ve never done oil and gas. I could apply some of my PLC skills from manufacturing to oil and gas. Got into their SCADA systems, did that for about five or six years, and then migrated into the security side a little bit and really focusing on how do we secure oil and gas deployments across our footprint that I was in.

To give you an example, and the reason security was so important that the company I worked for at that time, roughly half of their power… This was in St. Louis, Missouri, so half of their electricity generation came from natural gas from the company I was working for. If you can imagine if something were to have happened and somebody got into the system and closed what we call the main line valve, we could have darkened half of St. Louis. Tulsa, Oklahoma was another one where we provided half of the natural gas for their power generation.

That really drove home being from Oklahoma, and I was actually living in Tulsa, I’ve got family in Tulsa, I thought maybe I better get on top of this, right? Mom, dad, grandma, and grandpa won’t appreciate the lights go out and grandson or their son didn’t do anything about it.

That company, they were acquired. They were forcing to move or reload. I couldn’t reload at that time, and so I started looking for something else. Where can I apply my manufacturing skill set? Where can I apply my skill set towards PLC and OT and automation? And then also, where can I also loop in cybersecurity side? Quickly realized that there’s a niche in the OT and IoT industry or area of cybersecurity that, as you mentioned, there’s not a lot of people that can fill that niche.

I’ll tell you today, I can go hire somebody in security. I can go hire somebody in OT, but it’s very hard to teach a native security person OT or IoT, right? And so when I’ve hired in the past, I’ve hired industrial control people and taught them security. They really appreciate it, and I think the world’s better off for it.

I was looking for a place where I could call a home that I could bring my skill sets from manufacturing, from operational technology, from security and put it all together. And so I found a global organization that was spinning up a group just to do just that. I was brought on to lead that effort and quickly got into the OT side of things and realized there was a gap in the IoT side of things. We addressed the IoT from a security perspective, so now I feel like we’re in really good shape from an OT and IoT perspective. We have the tools, we have the resources, we have the policies. We’re doing constant audits to determine where are gaps that we have not uncovered yet, right? In the beginning, it was the low-hanging fruit.

But that’s how I got where I am today.

John Vecchi:

Wow, that’s amazing. We’re going to talk about, like you mentioned, the IoT side. I like to call that OT-adjacent IoT devices and cyber-physical systems, right? In many cases we focus a lot on, we’ll talk about that, the OT side. You mentioned PLCs, HMI, SCADA equipment, the RTUs, all the equipment and cyber-physical systems that you’d think immediately fit in that OT, ICS bucket.

But we often forget, right Joel, that there are a lot of OT adjacent IoT systems that are part of that same environment, right? If you forget about those, you may be addressing a big piece of your attack surface, but completely missing another piece that’s, like you said, could be potentially the lowest hanging fruit.

Is that safe to say?

Joel Goins:

Yeah, you’re completely right and the fact that OT and IoT are really converging and both of those are converging with IT. That’s where you get into edge compute, right? It’s a little Linux-based box that’s made wired or wireless in an environment. It’s collecting signals from a PLC. To me, that is the perfect use case for protecting both OT and IoT. I need to protect that box because that box may be set up to receive signals only from a PLC, but there’s nothing to say that somebody couldn’t hack that box and reverse engineer it and start sending signals to a PLC.

And then we run into problems because now we’re putting humans at risk, the operators of those machines.

John Vecchi:

Yeah, yeah. Of course, like I say, if you’ve been following any of the news of late, and you don’t need to be an expert in IoT, OT security and cyber-physical systems. But I think most people would agree that if you look at the news of late, the amount of activity and stories about our critical infrastructure, and it’s continues to be just more and more prevalent. I think people are understanding that this stuff is at risk. It’s a host of things with a lot of different threat actors.

Lately we’ve heard all the… You started with Volt Typhoon. That’s the Chinese actors attacking critical infrastructure, and then that then moved to Salt Typhoon, which is now hitting our critical ISP infrastructure and some of that. You look at the Iranians attacking the Pennsylvania water facility, and now of late the American water facility targeting cyber-physical systems.

You mentioned the servers, right? You’ve got like Fuxnet, which is the Ukrainian side, targeting the default credentials on those gateways that are incredibly… They’re connected to everything. In many cases, you can access all those PLCs through the gateway, whether it’s FrostyGoop and all the attention on critical infrastructure.

I mean from your past when you were really in the utility side, the oil and gas where that’s designated as critical infrastructure, I would maybe argue some of your current employer, some of your infrastructure might be designated critical infrastructure.

But as you think about all of those activities and where you are sitting, how do you digest that? How does that affect just your daily approach to what you’re trying to accomplish with your current group and its focus?

Joel Goins:

I think you hit the nail on the head. I think critical infrastructure and just what we typically classify as critical infrastructure is changing as we as companies and individual people become more reliant on technology. Everything that we do now seems to go through a data center, and I believe we’ll see very shortly where we’ve got electricity, critical infrastructure. Natural gas, critical infrastructure. Water, critical infrastructure. Soon I think we’re going to see fiber become considered critical infrastructure. I think we’ll see the protection around fiber, just the amplifier buildings that are usually along railways.

I think we’ll see some more regulations around those and data centers, right? I was talking to Chris right there at lunch a few weeks ago. He made the comment is, “If AI gets away from you, how do you shut down AI in a data center? How do you do it?” Well, you shut off the AC. You air conditioning. They’re going to burn themselves out.

And so that’s the thing where I’m at today that is one of our large product lines is protecting, controlling the environment of data centers. Making sure that our equipment is secure, making sure that we are monitoring, keeping everything patched, addressing whatever vulnerabilities that may exist as soon as they come out.

John Vecchi:

You mentioned data centers, which is phenomenal. I mean, most people don’t understand or maybe just know particularly that not only, yes, data centers are critical infrastructure today. Every time you talk AI, whether it’s some of the big cloud guys or it’s the data center where everything exists and those cyber-physical systems can make those data centers unbelievably vulnerable, right?

You mentioned things like HVAC controller, which is controlling the cooling PDUs and power distribution units. All of these kinds of things can be attacked and can have an incredible negative, nefarious outcome on those types of critical infrastructure, right?

Joel Goins:

Right. I mean, even down to the backup generator outside. A lot of people don’t give those any thought, but they’re more than just a motor or an engine, a diesel-powered or natural gas-powered engine. They have controls in them. I’m not say it wouldn’t take a lot, but they’re not as sophisticated as other parts of a data center, so for somebody to get into and flip a switch and not allow or put a generator into a shutdown mode so it didn’t come on and then go attack the power at the data center.

There’s a lot of redundancy in data centers, but it’s almost like a denial of service attack, right? You shut one down, the other one can’t keep up. Systems go down. People can’t get to what they need to get to on the internet. In so much of business these days, the applications that we use are our SaaS solutions that are hosted in the cloud, right? At the end of the day, a cloud’s in the data center so we have to protect those.

John Vecchi:

Yeah, yeah, and of course, whenever you think in terms of protecting that, we can’t ignore the state of these devices, right?

Let’s talk a little bit about it. This is what you live with every day. I mean, you come in to say your position where you’re building a group to focus on OT and industrial control systems, but also that OT adjacent IoT landscape. There are billions of these devices worldwide. I’d say there’s well over 60, 65 billion of them worldwide that’s growing at an incredible pace. And so you think in terms of any environment, there’s so many of these devices.

Of course, the state of them is another mess, right? Those listening to this podcast hear me talk about it all the time. A lot of the attacks I just mentioned, whether it’s things like FrostyGoop and Fuxnet or the Iranians targeting our critical infrastructure, they’re often targeting these underlying critical hygiene issues that just seem amazing. Like I said, we’re back in the early ’90s every day we live in this space of IoT, OT security, right? Credentials are default. These deployed mostly with default credentials. Hosts of ports and protocols and services are, again, these are default configurations. They’re deployed with these things wide open, making them horribly insecure firmwares could be eight, nine, 10. Sometimes it could be way older than that with critical CVSs, CVSs and CVEs. The state of these things are pretty terrible.

Joel Goins:

That’s why we picked Phosphorus, right? At the end of the day, like I said earlier, I knew there was a gap. What I didn’t know was how bad or how big that gap was.

And so for us, the first part of our discovery is let’s go find out what we have, right? Know the assets. Know your footprint. That’s step one for us. And then once we’ve understood, okay, how many devices do we have in the environment, what is the state of the device? Is the firmware 20 years old? Does the person responsible for that location, do they even know it’s there?

That was something that was shocking to some folks when we ran the Phosphorous scan is like, I didn’t even know I had these devices in my environment. Why? Well, I started two years ago and there was bad record keeping prior to that. And so when we ran the Phosphorous and started assigning these devices to their physical locations, a lot of people were… I’m not going to say freaked out. They were a little concerned with how many devices they didn’t know existed. But we were able to use the Phosphorous platform to get them to see those devices, to look at the state of the devices, whether it was a default credentials or [inaudible 00:15:49] the firmware was very old.

Now they had a tool, a single pane of glass to go fix these things instead of somebody walking into a building and say, “Oh, I now own 100 IP cameras that I have to touch one by one to go fix the password or upgrade the firmware. I could do this with a couple of clicks and Phosphorous will take care of it for me this weekend. I’ll come in Monday and things will be where they need to be.” That was one of the really big benefits is how do we identify the devices, understand the state, and how do we fix them?

Because we all know the companies are trying to shrink their workforce. Everybody’s trying to save money. You can’t go and throw a thousand devices at a person and say, “Hey, by the way, these are now your responsibility. I’m sorry we didn’t know about them, but they’re yours.” And then go, “By the way, we’re firing the guy in the cube next to you,” right? Good luck, and so we’ve addressed that, the shrinkage of the workforce, the reduction in cost by using a tool like Phosphorous to help us get that single pane of glass so we can understand what we have, understand the state, and then go fix it.

John Vecchi:

Yeah. I mean, right there you just hit on so many massive challenges, I think. As I spend a lot of time talking to practitioners and defenders like yourself and other OT operators, whether it’s across manufacturing, transportation, critical infrastructure, utilities, oil and gas is these issues, which is one of the biggest is just the visibility, right?

I mean, if you think in terms of oil and gas and utilities, electrical, and solar and all of these guys, you have the DOE saying, “Hey, by the end of this year we’re going to start requiring you to have some basic inventory of your IOT, OT assets.” You think, put yourself in those practitioners’ shoes. And the first thing they say is, I don’t even know where most of it is, and it’s all over. I’ve got sites all over. Some of them are incredibly remote. This equipment’s very old. It’s been proliferating for years and years. I mean, the visibility is such an issue.

Because you came from that side as well, why do you think that is? Is it because of, again, the historical allergic approach and reaction to, say, traditional active scanning? Is it just proliferated too fast? We weren’t focusing on it? But what do you think is the reason with this visibility gap we have?

Joel Goins:

I think when you go back to oil and gas, a lot of, and you got a well-head out in the middle of a field and it’s been out there for 30 years. The technology out there might not have been updated, but even up until, well, even currently, a lot of the communication to those RTUs in the field is not end-to-end ethernet IP. It may be ethernet IP to a Moxa or something sitting there, but then it’s serial to the RTU.

That’s a big problem, right? You’re only going to get so much back from the serial device when you’re talking to an over serial. You can pull it, hopefully you get some information back, but that requires some sort of polling engine to go do that. But from just even using a traditional scanner is going to see up to the Moxa and not anything further.

That is the problem, right? It’s the problem with how are you communicating with that end device? Like I said, most of it’s serial. A lot of it’s old. There’s some hesitation, I’m sure, in the market to go spend more capital to put a newer RTU in there. Don’t fix it if it’s not broken, and that’s the perception of it. I could tell you horror stories of cell modems out there in an unlocked cabinet in the middle of a field tied back with it to a company’s network over an APN, right? The APN from Verizon or AT&T if that cell modem’s running the ACP here in the environment, right?

And so that’s a bit concerning when I think about that is how do you address that? It’s really up to those asset owners to say, “You know what? This is a gap and we’re going to have to go fix it or else we’re going to be in trouble,” right? Colonial Pipeline, they learned a heck of a lesson a few years back. Secure your network. And so the US government at least, and with the TSA, they’re starting to put regulations now, stricter regulations on critical.

Things are getting better. What I think is missing yet is still the audit side of things. I believe once TSA comes in and starts mandating audits for critical infrastructure and they start asking those hard questions, do you have a complete asset inventory of everything in your environment, once that starts happening, then people will wake up like, yeah, we’ve got to do something now. As soon as it hits an audit, it has to be turned into a government or regulating body. That’s where people start. That’s when the checkbook opens up because otherwise it’s fines.

John Vecchi:

Yeah. Do you see that change? I mean just even throughout your career in this and just so focused on the OTI, CS side, do you see that needle moving over the years of, I’ve got to get my head around this? I need visibility. I need to understand my risk posture of these things because at the end of the day, really there is an air-gapped environments so much anymore. These things are all somehow network connected.

Is that changing, do you think?

Joel Goins:

I believe it is. In critical infrastructure, oil, gas, electricity, water, it’s changing, right? It’s a top-down change. On the manufacturing side, if they’re an unregulated business, that’s where it gets tough, right? If you talk about pharmaceuticals, where it’s regulating have to report X, Y, and Z. there’s more controls there. But it’s the unregulated manufacturing that is going to be the laggard on this.

It’s the audits, right? At the end of the day, it’s the audits that drive the spend. And so whether it’s a becomes a regulated manufacturing company, or they hire a third party to do an audit because of some other reason… Or even the SEC, right? The SEC in the US now is starting to ask those questions. You have to report. Now, if a company is breached and there’s a material impact to the business. It has to be reported. At the end of the day, that can affect the stock price. And so now you’re driving that visibility and the need to do that up to the board level, and the board starts asking the questions. What do we have to do to make sure that we never have to report a material weakness or issue to the SEC?

John Vecchi:

Yeah. Again, you’re right. I think the hope is that this is now elevating to the board level specifically around these kind of cyber-physical systems.

Are you seeing more CISOs, even CIOs given I call them, and these are endpoints. They don’t have a keyboard. I get it. You can’t put your favorite agent on them, but at the end of the day, these are computers. They’re endpoints. Some of them are just as powerful as laptop. They’re Linux based. You can exploit them just like most any other computers.

Are you seeing more CIOs, CISOs in your observation understanding I’ve got to get my head around these, this attack surface which outnumbers traditional endpoints by an order of magnitude, right?

Joel Goins:

Yeah, we are seeing that, and I think a lot of it has to do with the availability of the information, so there’s some websites now that a lot of CISOs and CIOs are following that are outside their traditional CIO today, right?

It’s what are those other vulnerability websites that are publishing these stories on X, Y, or Z happening? They’re seeing it. I’m seeing it trickle down so that we have a big push right now to get the assets that we’ve discovered in Phosphorus into our CMDB because we want that visibility. It’s no longer reporting on just do I have antivirus on a workstation. Are all my workstations windows patched? It’s what else is in the environment that could be become an attack surface that they don’t know about? And then are we tracking those? Are there metrics around that? What’s the SLA to get whatever IoT device it is patched, right? We’re having those discussions today and moving forward.

I would say probably in the next six to 12 months, we’re going to see a big shift in the standard security focus from desktops and servers and all the agents and things like that to what else. And then what else is in the environment that’s IoT-based that has an antenna hanging off of it? That’s a big concern, right? Is that turned on right? It’s connected to our network over wire, but is it also broadcasting an SSID because some vendor came and put it in your environment and you didn’t know about it? But that’s how they’re remotely getting into it.

Nobody on the, we’ll say, the corporate IT side or corporate network side is aware that that’s happened. So we’re finding those today and we’re going after those and we’re removing those and we’re trying to understand why. What was the roadblock that kept, let’s say, plant X from putting that machine on the network that a vendor had to go and put their own solution in?

A lot of vendor discussions are happening right now within the org to eliminate that, to make it easier and more secure for us to do, or for vendors to come in to do remote maintenance on machines.

John Vecchi:

Yeah, 100%. I mean, that’s like you said, the wireless, it’s unbelievable.

Again, I talked about it, these things. If you look at CISA and their advisory, the good thing about that is they are focusing again on some of those fundamental hygiene issues, saying. My gosh, if your password is 1111 on your PLC, you probably should change that. You should shut off all your… Shut off Telnet or shut off the things you don’t need. I mean, again, if it’s a device like you said, has Bluetooth, Wi-Fi, FTP, HTTP, HTTPS, Bluetooth, and just shut those services off if you don’t need them, right?

Of course your firmware you should patch, but you mentioned patch. I know you know that in this environment and in the OT security side that can be a bad word, right?

Joel Goins:

It is a very bad word.

John Vecchi:

Yes. It’s a very bad word, and I’m seeing a lot of movement in that where we’re starting to get over that and people are understanding you actually can. It might not be firmware right away. As you know, you may never patch particularly some of firmware on certain types of devices, but you can certainly address things like passwords. You can certainly shut off certain services if your keyswitch position on your PLC is a programmer. You can do some of those things.

But you also, as I mentioned, have all these other devices where you do particularly want to address passwords and ports and protocols as well as firmware. How do you even address that with some of your peers who probably said, “Joel, you’re crazy. What are you talking about, we’re going to actually go patch this up?” I mean, is that a real scenario, I imagine.

Joel Goins:

It’s a daily conversation in a lot of aspects.

I mean, there’s this multifaceted. You buy a machine and it’s 10 years old and the PLC is 10 years old. The firmware has never been patched. The password’s wrong or it’s got default creds on there, the keys in remote, there’s some things we can fix. And you mentioned it, right? Flip the key to run mode, go fix the default creds on it. But the vendors, sometimes they don’t even want to go and help you because you bought the machine 10 years ago. The guy that wrote the code for that is retired. They have no idea if going from firmware version 20 to 30 if the code base is still going to work, and they don’t have the talent. They really don’t care to get into it and go reprogram it.

We started looking at doing something like micro-segmentation around that machine, right? Understanding what does it need to talk to? Phosphorus helps us there because when Phosphorus does this polling, we can see the ports that are open. We can say, “Hey, do you need port 80?” Okay, no, you don’t need it. Well, let’s kill that, right? Do you need port 22? Well, we SSH into it. We have to ask the question, right? Are you SSHing into that or you’re remoting into that PLC because it’s a country away, or is it just downstairs and it’s easier for you to do it at your desk? We to have those conversations around what maybe you need to go down and plug in and do this, and probably safer that way to go down there and plug in.

The other thing you touched on was even though we can’t patch in some cases, whether it’s an internal decision or the equipment’s so old that the company that made it’s out of business and you’re afraid to touch it or even reboot it, it’s knowing what assets you have and then doing a risk analysis on an asset. At least you now know where your risk stands, not just from a cybersecurity standpoint, but it could be also at a business risk that, hey, if this machine dies or does not want to start the next day, there is nobody out there to support it. A lot of people haven’t done that type of analysis.

To me that’s more of a top-line revenue risk, I can’t produce product versus a bottom-line security risk. I had to take this thing down and it’s costing me money, or I’ve got to go… They completely break the PLC and I’ve got to start from scratch.

It’s identifying those risk and then letting the powers that be, the decision makers on that particular location decide, okay, we understand the risk. How are we going to mitigate this risk and get it on the risk register so somebody’s aware of it.

John Vecchi:

Yeah.

I often hear a lot, and I think one of the issues in this space around OT security and ICS is that because of the pushback and the bad experiences from historical experiences with some of the legacy technologies that might take down your stuff or patching, as you said, if you shut stuff down that affects our bottom line, it tends to just have a throw out the baby with the bath water effect and since I’m not want to do anything, right?

I think oftentimes when I talk, I mentioned that you may not ever want to patch certain devices and cyber-physical systems to get that, but there are a lot of other things you can do. Number one, you should know what it is and where it is and it’s risk posture, and you may want to do some things that you can do. We can safely update default credentials, or we can safely turn off certain services and ports and protocols. We could even maybe monitor those devices.

And so there are things you can do, but we tend to have that mindset of, because patching is it, I shouldn’t just do anything, right? Do you see that mindset as well, probably with some of your peers?

Joel Goins:

Yeah.

John Vecchi:

You said it’s a daily conversation, right?

Joel Goins:

It is. Yeah, we had one PLC that was on the network and said, “Look.” We play hardball a little bit, right? If you can’t patch it, you need to take it off the network. Then you hear all the, well, but I, but I, but I do X, Y, and Z, And then you start them thinking about, right? Get the asset owner thinking about, okay, well, how do I address these concerns? Because if I sit there and say, “Well, just take it off the network,” that’s the solution doesn’t work for them.

But I can’t solve everything. I need that brain power, the brain trust, if you will, of I’m sitting in Oklahoma. This plant’s in Texas. I’ve never been there, and if I just give them an ultimatum, they’ll come back in most cases with a good idea that satisfies the risk and reducing that risk. And so it’s going back to and knowing what do we have across the globe? That’s what makes Phosphorus invaluable for me, is I build everything there.

John Vecchi:

It starts with that, right? I mean, again, and we hear it again, you can’t really protect or secure things you don’t know. But in this space with these devices and the number of devices, and I say it’s all the IoT things that are intermixed with all of your OT and ICS stuff, and it’s just…

If you don’t know where it is, it’s hard to even begin to think about what you need to do about it. It just seems like that visibility gap has been such an incredibly large challenge in these environments, I think, for so long.

Joel Goins:

It has been. Even conversations when we first spun up Phosphorus talking about I’ve got a UPS with default creds, and I would tell somebody in IT, “Hey, you need to go fix those creds.” They’re like, “Well, so what?” I’m like, “Well, do you know what you can do with default creds on a UPS?” They’re like, “Oh yeah, I can turn that off, can’t I?” Yeah, you can. And not just you, but if somebody else is in the environment, they can turn it off. And so do you want that call at 3:00 AM to say, “Hey, my whole server rack went down because somebody got into my UPS,” whether it was intentional or by mistake and just playing around, you don’t want that and the business impact that that can have.

It’s everything. Everything is connected. If you look at how even down to a machine on the floor, it’s talking to something in the local server room or at the data center. You have to protect that whole chain, right top to bottom that whole chain that’s allowing that machine to do its job has to be protected. You can’t just look at the one thing. It’s everything it’s attached to everything it’s communicated with, everything that powers it that has a network plug on it that’s plugged into something has to be protected.

John Vecchi:

Yeah, and I love you mentioning the UPS. I talk a lot about that in the sense that these things all sometimes show Shodan search just for UPSs connected to just on the open internet, let alone. There’ll be on any given day, 15, 18,000, 20 plus thousand. You’re thinking, why on earth would you have a UPS connected just to the open internet?

Again, I’m not saying anything that isn’t common knowledge, but most of these UPSs have the same default passwords and ABCABC. It just happens to be what they are. We’ve yet to find any of these devices when we find them that do not have default credentials.

Joel Goins:

I’ll be honest with you, I’d love to know what that use case is to have that UPS on the internet, right?

John Vecchi:

Isn’t it amazing?

Joel Goins:

I mean, just tell me, just please. I’m just curious, right? It could be a Calvin cartoon at the end of the day.

John Vecchi:

It’s so true.

But I mean, this is the state we’re dealing with. And so when you step back and look at that, the state of these and just how you can find these on the open internet, let alone go attack you just specifically in directly, there’s a lot threat actors can do.

We talk a lot about, I’ve mentioned there are nation states. Of course there are a lot of threat actors that are sponsored by nation states, but you have a lot of ransomware gangs, too. We’re seeing ransomware gangs target these devices because it could be the easiest ransomware attack you ever had. If you have thousands of PLCs and they’re all running with default credentials, I don’t know, what if I just go and log into all your PLCs and change your passwords? And then give you a call and say, “Hello, I just changed the passwords on all your PLCs and so give me a load of money.” It’s the easiest ransomware attack.

I mean, what attacks keep you up at night? Is that one of them? I mentioned that sometimes, and I see people nodding their head. Well, yeah, even in healthcare, you could do that to a life critical device and just lock everybody out there all deployed with default credentials. I mean, you could just start there and you could…

Joel Goins:

It is bigger these days because if you look at most organizations now, the IT, even security is normally not housed at a manufacturing plant or a corporate office these days. We’re all remote. If somebody gets in and through ransomware locks us out, there’s a chance we’ve got to go fly somewhere just to go get on the network to try to fix it, right? It’s not just, okay, let me get in the car and I’m 30 minutes away. No, it’s a flight, right? It could be across the world to go fix something like that.

Ransomware and locking somebody out, that’s a tough pill to swallow. It’s the time to recovery that is much longer now because of the complexity and sophistication of the attackers and the distance to that end device that most of the asset owners are from there.

The other thing you had mentioned about Shodan, so I’ll give you a quick story. When I was at oil and gas, we had an issue we weren’t even aware of, and this was probably eight years ago I want to say. We actually got a call from the FBI. The FBI and unbeknownst to us at the time, had started monitoring traffic through ISPs. And so they had identified this company X as an oil and gas midstream company and were monitoring our traffic.

They told us, they called our head of security and said, “Hey, we believe somebody is in your environment.” And so long story short, going back to Shodan is, at least in the US, the government is watching Shodan. I wouldn’t be surprised that if we didn’t start seeing penalties for companies whose devices show up on Shodan.

John Vecchi:

On Shodan, yeah.

Joel Goins:

You get a subpoena or a bill from the US government or a ticket or however they put that together on devices that are just out there on the web that shouldn’t be. That’ll a big driver to get things fixed, and quick because at that point, you’re…

Especially critical infrastructure. Who pays the bill at the end of the day if things go south? We do as taxpayers and we do through the government and comes and bails somebody out to go fix something. But if they can be more prescriptive on looking for things that are shouldn’t be on the internet, that I think we’ll all be better off.

Maybe we’ll see that happen someday. I think it would be a good carrot, if you will, for some companies to go make sure that their estate is in order.

John Vecchi:

Yeah, it’s a really great point. We use Shodan. I use Shodan just to make the point, right? I mean, God forbid you put in something like a printer. You start to see hundreds of thousands of these things show up. I mean, the fact that you can even have 15, 13,000 UPSs is startling. But start looking at things like printers and cameras and oh my God, cameras, you could have millions of them literally.

Joel Goins:

They’re Linux devices is unused compute.

John Vecchi:

Yep, they’re just Linux servers. Like I say, you see a camera, I see a Linux server. I mean, this is the state. It’s like you said, Shodan could be a very useful tool to open some eyes in certain organizations. I think that’s an interesting idea you have. You’re going to have to talk about that one.

Well, look at, Joel, we could continue talking. It’s so fascinating. I mean, again, this is an area that’s so small, the number of experts like yourself, that we need more. You guys are protecting our critical infrastructure. You have all these mission-critical devices, and in many times, these don’t just affect, say, the operations of the business. It could affect livelihoods and safety and public safety, right? I mean, it’s a really important thing, and there just aren’t enough of you guys. We appreciate having your thoughts and expertise in here today.

Any thoughts? Again, there are certain practitioners that will listen to this, those that are wanting to get into the OC side, OT side, and ICS side of security. Any advice that you would leave our listeners with today that might help them as they think about this?

Joel Goins:

Yeah, I think the biggest thing walk away with is go back and reiterate, know what’s on your network. With traditional tools you think you do, and I will tell you you don’t. You don’t know what’s there. You’ve got to get a tool that specifically can go find OT and IoT devices because I guarantee you, you don’t even have any idea what’s actually in your environment.

And then once you do that, then you can start fixing it and making your security risk much lower. Start there, right? Go discover, go see what you have, and then go fix.

John Vecchi:

Yeah, and as I think we learned today, Joel, you can actually do that in a way that’s safe.

Joel Goins:

Yes, absolutely.

John Vecchi:

Right? I think this is the test. This is why I love having guests like yourselves on. As people can see, wow, you really can. You’ve got you doing this. You can actually do it, and we can do it safely in a way that’s efficient, but safe and with full control so you can actually have that visibility, right?

Joel Goins:

Yeah. We have a big estate and we have yet to, I’m not knocking on wood, but we’ve scanned the environment many, many times and have zero complaints that we’ve impacted the network, that we have knocked over a device. They don’t even know we’re running it.

That’s the other beauty of Phosphorus is it’s so quiet. Scary quiet, but it’s quiet in a good way.

John Vecchi:

Yeah. That’s great. Well, what a great testament, amazing discussion. Joel, thanks so much for joining us today. Really, really appreciate it. We’d love to have you back again and continue the discussion.

Thanks everybody, and thanks again to our guest Joel today. We really appreciate that. Remember, the IoT Security Podcast is brought to you by Phosphorus, the leading provider of unified and proactive cyber-physical system security and remediation for the Extended Internet of Things.

Thanks again, Joel, and until we meet again, I’m John Vecchi. We’ll see you next time on Phosphorus Radio.