IoT Lessons We Learned in 2024

Transcript

Alex Nehmy: Hello everybody. This is the IoT Security Podcast. I’m Alex Niemi, Asia Pacific CTO at Phosphorus, your host for today and I’m really excited for what we’ve got for you. In 2024 we had some amazing guests on the podcast. These people work with IoT and OT every day and they bring all kinds of perspectives, stories and expert guidance to the conversation. What better time than the first month of a new to look back at some of those conversations and what we learned about the nature of IoT security, the trends that drive the industry and the community itself. 

Every story has to start somewhere and we always like to hear about how people got started in IoT security. Jason Tall, an established V SISO with an incredibly diverse background, started his path with a CPU and no keyboard. Let’s take a listen back to when he was on the show. This clip touches on the sensitivities of remediating devices in life-critical situations.

Jason Taule: My first computer had no hard drive. It had no keyboard and no monitor. Right. It was a, it was a CPU and a processor and I learned uh, how things worked, much like the old term hackers, people that take things apart and to figure out how they work. Uh, so I was really there seeing how everything was built. I was one of the first people uh, on the ARPANET, uh before we, before Al Gore’s legislation, uh, made it available for commercial purposes. Uh and I can remember thinking wow, I’m going to get, I’m going to get left behind. This Internet thing is crazy and I better get smart about it. Um, it’s the only thing I’ve ever done. My first job out of school was uh, with Booz Allen Hamilton. 1986 the computer security act got passed and a bunch of federal agencies didn’t have ah, programs for their edp. Or we didn’t call it EDP then we called I.T. uh, AIs Automated Information Systems. Right. So this is the late 80s. Um, my folks ah, at the team at Booze gave me a copy of the Rainbow series and it said stay a chapter ahead of the customer. Right. And now I’m a security expert because I’ve done a couple of these projects. After a while I did get pretty good at it and uh, I had the very good fortune of working with and for some of the smartest, best people, um, I helped build the VA’s program, I helped build NASA’s program and I learned really important lessons at each one of these. So for example, if you’re NASA and you’ve set Something in orbit that’s going out to the edges of the solar system and it has a vulnerability. Guess what you’re not doing? You’re not pushing a patch to that thing and restarting it because you might take all of the investment and spoil it if the thing doesn’t restart.

Alex Nehmy: Patrick Gillespie is now the OT practice lead at Guidepoint Security. Early in his career Patrick knew visibility was important but he used the wrong tools. IT tools not purpose made to safely discover xiot and he used this to inventory his IoT environment and it didn’t go well.

Patrick Gillespe: They’re all familiar with Kali Linux, right? Comes with a set of tools. Um, but before that when I got into security, Kali Linux didn’t exist. It was Backtrack Linux. So when I started going uh, back to school for my master’s in security I started um, testing my local networks, you know, at the railcar manufacturing facility I was at. So I started testing it. Well the OT was a separate physical network but of course you know, I had a desktop with dual NICs. So I plugged the OT network into my dual nicked it uh, computer and set up um, a Backtrack Linux server and put Open BOSS on it. I was like, well you know it worked well in it. Nobody cried. So then I decided to scan OT with Open Boss just to see what was out there. And the devices did not respond well. And actually there was a printer, I don’t remember the brand name uh, but it, when I scanned it we got a frantic call to the help desk. A lady, uh, I think it was in purchasing out in the plant, uh, at one of the paint uh, facilities for the real car called and says her printer wouldn’t stop printing papers. And they all said are you dead? She thought somebody was attacking her personally. And so I had to go, you know, apologize profusely. But it was because of the Open BOSS scan on, on that network that not only caused the weird printing but you know, caused some uh, outage denial of service essentially for those devices. So that was a lesson learned for me. And then once convergence started happening, especially during COVID a lot of, a lot of people started doing full blown enterprise vulnerability scanners in map port scanners. And yeah like, like you mentioned earlier, those, those devices do not behave well uh, against that kind of traffic.

Alex Nehmy: After you’ve achieved visibility, you’re going to find vulnerabilities to fix. Right? In this clip, Sean Tufts, managing partner at Optiv, highlights how remediation is so much more powerful than continual assessments. Remediation actually moves the Needle on reducing your cyber risk. And when it comes to legacy systems, remediation is a lot quicker and a lot easier than buying new gear. Just fix it.

Sean Tufts: John. I can’t tell you that in the last three months, starting at uh, Thanksgiving, October time frame, we stopped or we started hearing clients say more about, hey, I don’t need another assessment. Assess to death. Everyone’s always been assessed to death. Uh, they never weren’t assessed to death. Right. But they’re starting to say, fix the problem, please. Like let’s, let’s go actually jointly lower risk. Quit identifying, start lowering. Let’s get these things patched. Let’s go mobilize teams. Let’s get in front of this before we’re way behind it. So we, we’ve seen that very acutely in the last couple months where people want, are just tired of hearing about these dumb printers. They’re just tired of it, and they’re like, I want them wiped out. Go patch them, get them done. And that’s the expectation these days. There’s no money to replace the printer, but there’s a lot of focus, um, on taking care of those legacy systems if now that we can do it. Yeah. And is it the same approach, kind of low-hanging fruit as far as what, what they’re going to try to go fix initially. Do you take that similar approach? Let’s go get that low-hanging stuff with brain dead. Let’s go do that. Is that kind of the approach? Yep, that’s exactly it. We were um, we worked on a hydroelectric facility. Uh, the only thing you could see from the outside was the print server. They were properly segmented. They had all the Purdue model stuff set up exactly how you would want it, except for the one printer. Well, let’s go take care of that. Number one, let’s tuck it back where it should go. Number two, let’s make sure it’s patched. Let’s make sure it’s, it’s fixed and it’s not, you know, so addressable, so easily found. So if you can do those things, then all of a sudden your, your attack surface, the attack thing is a lot smaller again.

Alex Nehmy: Now we started to see that a big point to be made over the year was that segmentation isn’t the be all and end all. Rockwell Automation senior product security engineer Michael Lester came on the show and shared his experiences of how network segmentation in OT isn’t the only control necessary. It’s also critical to fix the cyber hygiene of these devices by changing default passwords and applying patches where necessary.
John Vecchi: So do you see that kind of awareness, uh, and how are they even thinking about those kind of hygiene issues on the OT side? Right,

Michael Lester: yeah, very much so. And that you know, in terms of that, that uh, separating uh, you know, devices and making sure that only uh, the ones that are supposed to communicate are communicating. And that’s part of that process I would add in making understanding what ports, protocols and services are being used, right when you, to understand an asset. And so you have those documented and you can be like, hey, we don’t need those. But also you make sure that your most critical assets are segmented properly. And you look at that, that you know, uh, that segment and you start, and I’ve seen many of them start looking at, know those default passwords, uh, making sure that if we, if it’s patched that it’s done, you know, correctly, that a test bed set up and been patched and tested previously made, uh, sure to patch during downtime, um, and applying just the basic cyber hygiene to the most critical part. Because when you segment correctly it that is supposed to be your most trusted zone, right? Your IT is less trusted than your OT network and you set, you segment it into your most trusted zones and you work on those. Ah, so there’s very much that want to uh, work on those critical areas first and make sure that you don’t have less secure devices, um, on the same network as a way to get in easily.

John Vecchi: No question about it.

Alex Nehmy: Even touching some devices the most sensitive of operational technology can be something you just never want to do. Let’s take a listen to this clip with our fantastic guest, Joel Goines. He discusses the challenges with legacy hardware and weighing the business risk of maintaining equipment that even the OEM vendor is struggling to support. Approaching remediation in OT is in some cases a daily conversation.

Joel Goins: You buy a machine, and it’s 10 years old, and the PLC is 10 years old. It’s never been, the firmware’s never been patched, uh, the passwords wrong or it’s got default creds on there. The keys in remote, right. There’s some things we can fix. You mentioned it, right? Flip the key to run mode, go fix the default creds on it. But the vendors, sometimes they don’t even want to go and help you because you bought the machine 10 years ago. The guy that wrote the code for that is retired. They have no idea if going from firmware version 20 to 30, if the code base is still going to work and they don’t have the talent, they really don’t care to get into it and go reprogram it. So we started looking at, you know, doing something like micro-segmentation around that machine, right? Understanding what does it need to talk to. And phosphorus kind of helps us there because when phosphorus does, it’s pulling. We could see the ports that are open. We say, hey, do you need port 80? Okay. No, you don’t need. Well, let’s kill that, right? Uh, port 22. Well, we ssh into it. We have to ask the question, right? Are you SSH into that or you’re, uh, remoting into that PLC because, you know, it’s a country away or is it just downstairs and it’s easier for you to do it at your desk and we have to have those conversations around, you know what, maybe you need to go down and plug in and do this and probably safer that way to go down there and hit plug in. The other thing you kind of touched on was even though we can’t patch in some cases, right. Whether it’s an eternal decision or the equipment’s so old that, you know, the company that made it’s out of business and, you know, you’re afraid to touch it or even reboot it. It’s knowing what assets you have and then doing a risk analysis on an asset. And at least you now know where your risk stands from a, uh, not just from a cybersecurity standpoint, but it could be also at a business risk that, hey, if this machine dies or does not want to start the next day, there is nobody out there to support it. And a lot of people haven’t done that type of analysis. So when you think about a, to me, that’s more of a top line revenue risk, right? I can’t produce product versus a bottom line security risk. I had to take this thing down and it’s costing me money or, you know, I’ve got to go, you know, they completely break the PLC and I’ve got to start from scratch. So it’s, you know, identifying those risks and then, you know, letting the powers that be, you know, that decision makers, you, uh, know, on that particular location decide, okay, we understand the risk and how we’re going to mitigate this risk. Right. And get it on the risk register, right. So somebody’s aware of it.

Alex Nehmy: IoT and OT security, like Joel said, are an ongoing conversation. That conversation, community and building up of expertise in the industry is something Sygenta senior security architect Khris Woodring is really passionate about. While we saw many initiatives, best practices and guidelines get deployed throughout the year. Chris came on the show to talk about how in OT security people are one of the most valuable resources. And as an OT security community, we need to collectively focus our energy on, on resolution of the cyber risks.

Khris Woodring: So for your executive, if there’s a lot of noise, a lot of risk, they’re like, my business is at risk, you know, production, um, may be hurt. Right. If we get shut down for some reason from a cyber attack, um, and you know that’s going to affect our reputation, our revenue stream and so on. From a ciso, it’s like this is the function of my job. I’m supposed to be protecting this company. So I’ve got to absorb all this information, understand every single different thing that’s out there that could possibly go wrong, that type of thing. And then from your operations standpoint, just being very blunt and uh, I, uh, don’t want to paint this with too broad a brush, but again, from an operations standpoint, these guys are just like, you know, tell us what we need to do to be safe. Right? We’re not cyber experts. We run a plant, you know, we design, we engineer, we, we implement and we run a plant. We’re trying to produce some, some product. If cyber is an element of this, tell us what we need to do. Now if we look at the, the broader information landscape, you know, all the news articles and things like that, that, that you, you mentioned here, I think we, we’re, we’re in a unique situation because we, it’s good to have awareness, but at a certain point the human mind begins to shut down with too much information. Right. Information overload. So a lot of what’s happening is it’s, I don’t want to say cry wolf because there’s a lot of stuff happening, but at a certain point we as an industry need to get beyond just saying, hey, this is happening, that’s happening, this is happening, you know, all these different things and move towards more of a, of an organized, ah, effort to make true progress. And I don’t want to discount the efforts that are happening towards that end. But the biggest risk by far that I see to um, critical infrastructure, industrial technologies and things like that that are potentially impacted by cyber is that there is a lot of information out there. There’s a lot of voices, a lot of news articles, but there’s not as much leadership driving resolution. Ah, right. Um, there’s not none. There is some and there are those who are like thought leaders and they’re trying to do everything they can. I think what we’re missing is, um, we’re missing people like a workforce to support what needs to be done. And we are missing to some extent, leadership at a, uh, governmental level. And this isn’t aimed at any given country. This is like worldwide, like organized frameworks on how we’re going to move forward as a world. You know, not just this country’s going to do this and that country’s going to do that. So they’re really big challenges.

Alex Nehmy: How do we shine a brighter light on these challenges? Unfortunately, the light often comes from attacks that impact many people. Thinking back on a few Here, Mike Holcomb, ICS, ot Cybersecurity Lead@ Flora, shares the moments that created real world impact and brought it ot security. More to top of mind, what could happen if this or that went down? Well, now we know. Now a lot of people know.

Mike Holcomb: Prior to Colonial Pipeline, all we used to talk about were state adversaries, right, Coming into your environment. It’s Russia, it’s China, it’s the United States, right, Coming into the environment to steal your data, steal information about how that plan operates, and then potentially position themselves to where they could make something happen. And when Colonial Pipeline hit, it was, you know, the United States largest gasoline pipeline going down, not from a state adversary, but a ransomware group. And they were just sending out emails, basically every email address they had of everybody in the world trying to get somebody to click on a link or open up an attachment and infect their systems. And because it burned down to the ground, then it had that impact into OT where they had to shut down the pipeline. And there was that impact, that real world impact. And that’s where it did open up a lot of. Yeah, you know, you were saying, which is fascinating, that literally in the past couple of years the needle has moved quite dramatically to, as you said, if you had the discussion, even as currently as a couple of years ago, it’d be very, very different. Was it was something like Colonial Pipeline, you know, was that a turning point? I mean, did it. In other words, was that one of the impact, uh, you know, events that had a big impact on moving that needle? And if, if not, what were some others over the past couple years in your eyes? Yeah, no. Colonial Pipeline to me was, was the one it was, you know, being a longtime IT cyber professional that, uh, to me, I liken it to target the target reach, because for me there were still a lot of companies out there that weren’t taking cybersecurity seriously from an IT perspective. And then all of a sudden the Target breach happened. Everybody had to replace their debit cards or their credit cards and they’re like oh, we should probably do something about cyber back at the office.

Alex Nehmy: Many of these attacks hinge on low bar vulnerabilities. Threat that could be mitigated with an increased level of cyber hygiene. We talked to Mark Matai, Global Director of Industrial Managed security services at 1898 & Co. About cyber hygiene and he gave us a real world example of risk mitigation in an environment that needs 247 uptime.

Mark Mattei: I think like getting a handle on all of devices and the hygiene is one thing and, and it net like that the patching problem and updating problem is never going to go away. Right? So, so like focusing on just trying to continually patch, sure you have to do some patching but you really got to focus on what your architecture looks like and how do you, you mitigate the, the threat versus patch the vulnerability. So having a plan to come up with and say when there’s vulnerabilities released, I’m going to have a, a ah, way to um, understand if that vulnerability is actually a risk to me or if because I have SMB blocked at my firewalls, I don’t have to worry about that. Or um, if I have you know, ah, you know, these different ports blocked, I don’t have to worry about that, that PLC um vulnerability because no one can ever get to it. Let me check that uh, mitigation to make sure it’s in place. Um, I provided um, uh, you know, incident response to a client that basically had Conficker uh, across one of their manufacturing plants. Um, and here’s a little bit of the difference in the, the incident response from a, for an OT perspective uh, versus an it uh perspective. Then, the um, when we made the incident response, of course, we found Conficker all over the manufacturing plant. Um, we put the initial mitigations in place to stop the bleeding. Um, and then we came up with a plan to remove the Conficker. Of course, the plant can’t shut down, so you can’t. Right, like you can’t, right? Conficker. The way it spreads. You would spend a lifetime going from machine to machine removing conflict or just for it to spread again in the local area. So we put the mitigations in place, and because Conficker is just a process running on a box, we left it there, right? Like it’s probably still running on uh, boxes within that manufacturing plant, but the mitigations are in place to stop any actual threat actors from Doing anything with that malware. As long as it’s not doing anything, then it’s just a software program running on those, those systems right now. And it has no effect or impact because the, the assessment was done to mitigate that risk a different way, not worry about patching the vulnerability or removing that malware from the, uh, those systems.

Alex Nehmy: We’re about to wind down. But there’s one last message that’s important to hear. There’s always something else, something to stay ahead of. And John Threat, also known as John Lee, a veteran of the great hacker war and known as Corrupt from the early 90s hacking group Mod M Masters of Deception, came to the show to remind us that even if the next big thing falls away, it will help birth the thing that actually takes off. Be aware of what’s next.

John Threat: My recommendation to everybody is just drop everything and learn AI. It doesn’t matter how you come to it. Like, I think for me, like, I instantly incorporated it in some of my practice. I mean that’s just in nature. I’m not like a, you know, uh, uh, I’m an optimistic person, but I also, you know, like to stay on, on the cutting edge of things, I think. But in the sense, in the sense that like, like, and this comes from the hacker spirit and that’s a part where I don’t like when I, I’ve gotten into this even on other podcasts when I’m on there with like older hackers or people, they’re like, ah, crypto. Oh, this, oh, AI. There’s like, bro, like this is the same shit as when, you know, when you were little. This is the precipice of something incredible. Do. Now what comes out of, let’s say like crypto prices, you know, crypto could fall for away. But something is going to come out of um, it. That will be the next thing. And those people, that people breaking the blockchain, breaking, um, the financial tools, you know, they’re cutting your teeth. They’re going to have a certain skill set. You only recognize the signature because you’re too busy. Like, well, I’m above that. That’s just tulips. All right, bro. All right, bro. As is chill. You are, uh, chill. But you know what I mean? But maybe you should think about not infecting other people with that if you are active in shaping people’s opinions about technology because you’re giving them a blank spot. And the same with like robotics. Like, Robotics for me is a whole nother platform that can be hacked, can be exploited, you know, um, um, for good or for bad, I think that, you know, whether you’re on the activism side or whether you’re on, you know what I’m saying? Or just like, yo, it’s just part of an evolving security practice I think is important.

Alex Nehmy: Thank you for joining me for that walk down memory lane, everyone. The conversations around visibility, remediation, and community have been truly inspiring, and we look forward to plenty more in this year and beyond. Thank you for listening. The IoT Security Podcast is brought to you by Phosphorus, the leading provider of proactive, unified security management for the extended Internet of Things.

Thanks so much to all of our guests from 2024. I’m Alex Niemi, and we’ll be back with more of the IoT Security Podcast.