From Vulnerabilities to Visibility: Enhancing OT Network Security with Michael Lester

Transcript

John Vecchi:

Well, hello everybody. You’re listening to the IoT Security Podcast live on Phosphorus Radio. I’m your host, John Vecchi, and today I’d love to introduce our very special guest, Michael Lester. Let me tell you a little bit about Michael. He’s been with Rockwell Automation in various security positions, now serving as the senior product security engineer. He owns LifecycleIQ Services and leads the DevSecOps Modernization for OT Cyber Offerings. He’s worked throughout his career focusing on automation and industrial control systems. Previously, he’s worked to design and implement cybersecurity solutions and products for industrial environments at scale.

Recently at Rockwell, Michael received the innovation award from Rockwell for automating large portions of manual processes, we love that, saving his team hours of time every time a new version is tested and released when the product is deployed in the field. Of course, I know recently Michael is a PhD candidate as he moves on his journey to be a full PhD. Pretty much an underachiever there. Michael, welcome to the podcast.

Michael Lester:

Hey, thank you for having me, John.

John Vecchi:

All right. So we’ve been trying to get you on here for a bit. It’s really a pleasure to have you here. So we’re going to dive into a host of things. As we get started, Michael, I gave a little bit of an introduction here, but can you just tell us a little bit, dive in a little bit deeper about your journey, how you got to where you are. I mean, it’s so fascinating. We talk about industrial OT kind of security and you know it’s just such a special kind of specialized area. So our listeners always love to hear kind of what was your journey, how did you get to where you are and specifically make it to kind of focus on the OT, ICS side?

Michael Lester:

Yeah. Really, I’ll start probably a little bit further back than most people would. In my teenage years, I really enjoyed video games, but one thing I loved about them was automating processes. Used to play RuneScape, so I’d make micros and automate some of the things to make my life easier. Along my path really was three distinct people that kind of got me onto the path for OT cybersecurity. One was my high school professor, physics professor. We used to dabble in Excel. He made an algorithm for solving Rubik’s cubes. So I really enjoyed that. That got me into computers and got me into Excel and writing scripts and whatnot.

Then graduating high school, I was like, “Man, really, what am I going to do?” My older brother, Daniel, he’s an electrical engineer, and I was like, “I want to do something difficult.” I was like, “I’m going to be an engineer.” So that started me down the path of engineering, and I knew I loved computers. I wasn’t sure electrical computer engineering. Just throughout the process, I had an internship with ESCO Automation as a junior controls engineer. So I got that controls aspect. From there, I was like, “That’s not quite what I want to do.” But I really liked this and I knew for a fact I wanted to help people.

In college, one of my senior projects was around facial recognition, I wanted to go. Drones were big, 2013, ’14, ’15, drones were just blowing up. I really wanted to do something for working with the police or FBI and with facial recognition and drones, help protect people. So I knew I wanted to protect people. My next internship was with Burns & McDonnell in their IT department. The third person that I met, his name was Fred Terry, and he was in a completely different department. He saw what I was capable of. He saw I loved scripting. I knew a little bit about OT environments, had that control system experience.

We sat down at lunch one day and he is like, “Hey, cyber is exploding. We are trying to build a team and I really want you to be a part of it.” I was like, “All right, you got me.” So I signed on with that team. I was the third member of the team and started there. We started with vulnerability assessments and we had a bunch of federal jobs doing RMF work, vulnerability assessments. With my automation background, of course, I couldn’t do those tasks manually. So I started automating all these processes, and as the team grew, I kind of became that person that built the tools for others to go and complete the task.

So I built our own sensor, basically had a sniffing interface and a management interface. If customers would let us put it on their network, we’d sniff traffic from their core switch, and then it would automatically scan, depending on the type of device, it would scan from the management interface and we could connect to it through wifi, do all these things. Eventually, we built the team up more and more, started getting into penetration testing, expanded into energy, wastewater, airports, other federal programs. I actually got big into nuclear for a little while. I was the lead consultant for the very first intrusion detection system installed on a nuclear power plant’s control network.

John Vecchi:

Wow.

Michael Lester:

Then that event, Burns & McDonnell kind of broke off into what it’s known to now in their consulting realm is 1898 & Co. So became a part of that and eventually decided it was time for a move and joined Rockwell Automation. At first I joined their network and security services group, kind of in that consulting role, doing network designs and whatnot. Then I met another group that actually built the tools for all the others to use and standardize processes. That’s where I’m at now. I do a lot of product evaluations. I built pipelines to test and integrate different products into our threat detection managed services, basically SOC-as-a-Service, OT SOC-as-a-Service specifically. That’s kind of my day to day now.

John Vecchi:

Wow, that’s phenomenal. I mean, you touched on so many things I kind of want to dive into. I mean, the first and foremost, look, I think all of us in cyber and in our professional careers, you see the importance of mentors guiding. I mean, absolutely you had mentors that just kind of pushed you and guided you on that journey, which is super cool. Of course you talked about the automation side. I mean, historically and just in general, kind of automation and OT, ICS don’t seem to kind of go together in just generally, right? It’s like don’t automate anything. So I’d really love to kind of hear a little bit about that. But before we do that, can you tell us a little bit, I mean, about your journey to the PhD? Is that based on kind of your knowledge and research in this area, or could I ask or is it a little bit different?

Michael Lester:

It’s a little bit different. I always knew I wanted to, if I was going to continue my education, I wanted to go the furthest I could. I was always about challenging myself. That’s kind of where that route started. Once I joined Rockwell Automation, I had a decent amount of time in the afternoons back. So I was like, “You know what? I’ve always wanted to continue my education. I’m going to start towards my PhD program.” Along with Rockwell is nice enough to pay for a good chunk of that as well.

John Vecchi:

Wow, that’s great.

Michael Lester:

You mentioned the mentorship, and that’s something I’ve always tried to stay a part of. I love presenting at colleges, high schools every chance I got at Burns & McDonnell, they’re very community involved, and every chance they brought in high school students or whoever it was, and they asked me if I could present, I was more than happy to present. So I’ve stayed in touch there and I wanted to keep in touch with that education side. So expand my network on the education platform was one of my goals with my PhD, be a part of that education system, give back. I’ve met several students that went to some of the classes with me. I started eight years in my career, cybersecurity career back going from my PhD when many of these students are just starting off.

John Vecchi:

Yeah, wow.

Michael Lester:

Or haven’t even been in the field yet. So I was able to mentor quite a few of them, even place them and get them jobs at different companies.

John Vecchi:

Wow.

Michael Lester:

I’ve maintained that kind of mentorship as well. In terms of education, I’ve written some projects for students to complete in terms of OT cybersecurity. They basically program a PLC, hack it.

John Vecchi:

Phenomenal.

Michael Lester:

One of the professors had his students do their senior design around it and they expanded on it. So it’s kind of evolving. I really love teaching and providing educational material as well.

John Vecchi:

Yeah. Well, look, I don’t need to tell you, we need that, right? I mean, cyber field, the field in general, there’s gaps. We need more women. We need more security professionals. We definitely need more kind of industrial OT, ICS experts, right? So that’s phenomenal. I have friends, I know friends who’d recently finally achieved their PhD. I know how incredibly difficult that is and how much time it takes. So that’s just phenomenal, man. Well done on that.

Michael Lester:

Thank you.

John Vecchi:

So let’s start by talking a little bit about just your thoughts on the evolution of kind of OT and OT security, right? I mean, obviously we talk in general terms about the kind of, I call it the head-on collision of IT and OT in all of that. But can you talk a little bit of your observation of this, just maybe even since your time at Rockwell. Is there anything specific that you’ve noticed that’s really kind of an evolution or maybe a drastic change in the right direction over the past several years, let’s say?

Michael Lester:

Yep. In terms of the last 10 years, we’re still saying the same thing. Get on a framework, take care of the low hanging fruit, etc. But the things that have really changed are the frameworks that come out, the tools and processes that have come out, the material that’s become available. Getting into this industry is not easy. Getting access to control systems, even in a test lab, that’s not easy.

Virtual PLCs, some vendors have come out with cheaper PLCs to get your hands on. Even microcontrollers, you can get some of that. Open source projects coming out for Modbus and those communications, communication libraries, and then even companies and vendors allowing others to use their material. So that’s one big change in the positive direction. MITRE just released their embedded framework, or not framework, but their threat model for critical infrastructure-

John Vecchi:

For embedded devices, right? For the embedded side. Yeah.

Michael Lester:

Yeah. So MITRE just came out with that. Nmap in their latest update last month, they included even more fingerprints for PLCs and ICS devices. So these are the things that we’ve moved forward with that have really helped, and along with the education. The education I think is the people are one of the most important parts and allowing them access to that material, allowing more people access to the material, it attracts more people to the industry. These podcasts are a prime example. It allows people’s voice to get heard that are working in the industry, possibly attract some other talent like, “Hey, that guy worked at nuclear power plants. That guy worked at a wastewater plant. That’s really cool.” Being able to protect people I think is a major attraction. That’s what attracted me was that aspect.

John Vecchi:

That’s awesome. I mean, look, I mean, if you sit back and frame the situation and you look at kind of the industrial sector of the OT, you’ve got this exploding amount of devices. I call it extended internet of things, simply because they all have similar things in common, right? They’re purpose built. They’ve got firmware, they’re IP, they speak TCP/IP. They’re network connected. You can’t put an agent on them and all of that, right? That includes IoT and OT and industrial IoT and industrial, medical devices and all these things, right? So you look at the IC side and you just have this incredible amount of devices, many of them very OT focused, PLCs, HMIs gateways, RTUs, sensors, I mean, robotics, gate systems. My God, you can go on and on.

But then the thing I like to remind people as well is within all of that, you have kind of what we would call more traditional IoT devices, right? You’ve got cameras everywhere. You’ve got perhaps smart controls. You’ve got door controllers and card readers. You’ve got printers, maybe ruggedized, kind of printer, all these things, right? So they all kind of matter. So you’ve got all these devices. You’ve got these disparate teams, and I’m sure you probably have some stories about that. I mean, you’ve got the IT guys, you’ve got the security guys, you’ve got the operators, you’ve got third party people all over the place in there and they’ve never even met each other, alone know they exist, right?

Then of course you have the threat landscape where we know that nation states and threat actors, even ransomware games are looking at this stuff, right? It’s vulnerable, terribly vulnerable, and why wouldn’t you look at it, right? So you kind of set that whole picture up. Is that kind of how you see it? Or are there things in there that are missing? Is that kind of the situation as you see it?

Michael Lester:

Oh, very much so. As you mentioned, the IT versus OT landscape, that was half my job as a consultant was getting the IT to sit down with OT. IT was like, “Hey, let’s go, go, go.” OT is like, “No, we’re not going anywhere. You aren’t putting anything on our networks.” It was getting both of them to work together. I think a lot of people starting their journey off nowadays have that same difficulty. I’ve talked to hiring managers and they’re along the lines of, “Well, it’s easier to take OT control people and train them in cybersecurity rather than bring IT people over.”

John Vecchi:

Got it, yeah.

Michael Lester:

Just because they’re used to that kind of environment. Safety is the main priority. Or availability and just having that aspect already ingrained into the person.

John Vecchi:

Yeah, yeah. I mean, I can see that. That’s probably a good way to go. I mean, take an OT person, let’s train them up on security. But how have you, throughout your career, kind of bridged these teams together? I mean, it has to be part of what you’re doing, especially when you look at deployments and you mentioned kind of the OT-SOC kind of, as you will, which is very fascinating. I mean, have you had to bridge these teams together? How are organizations succeeding in kind of bringing these teams together, right?

Michael Lester:

Yeah, and that’s a good question. Yes, that’s something we’ve had to deal with. It’s getting the right people in the room to sit down and have that kind of conversation. IT, you all have this maturity level. If you work with OT, you can bring OT onto that maturity level, look at the savings if you all work together and use some of these tools. OT doesn’t have to buy all these tools independently and you can save resources. At the end of the day, that’s the objective is to secure your OT side and still be able to make that margin.

John Vecchi:

Yeah. I mean, as you said, I mean, they’re called cyber physical systems, these devices for a reason. It’s very much a perfect kind of description of them on the OT, ICS side, because they will literally do potential physical harm, I mean, which could bring operations down. It could have a lot of kind of bad things to them. So you can understand kind of the position of that OT side saying, “Don’t do anything, don’t touch it.” Or even on the manufacturing, industrial side, to stop production in any way or bring operations down, the cost of that on a daily basis is now in I think it’s hitting sometimes in the tens of millions to millions of dollars, right? I mean, and I can see how that mindset is what kind of… They’re holding their hand out saying, “Stay away.” Is the IT side, the IT security side, doing better at understanding that mindset now or?

Michael Lester:

They’re moving in that direction. You mentioned kind of what makes a company successful or where do you see success the most, and it’s in terms of how OT and IT communicate. If I go in and they’re talking and they have clear delineation of how processes and procedures, IT might do switch upgrades or whatnot, but they’re approved by OT, right? If they have those kinds of clean processes and that clean communication, then they are much more likely to be successful in terms of reducing their threat landscape and the attack vectors.

IT, when they’re willing to work with OT, those teams I’ve seen improve immensely on their capabilities. Looking at OT as something different than just your IT devices, patch it, move on. They’re willing to put up test beds, do patching in test beds before moving it onto the network, making sure it’s approved or production, I should say, make sure it’s approved before it goes to production, all those types of things.

John Vecchi:

Yeah. Yeah. Well, I mean, let’s talk a little bit about specifically on the security side. Like I say, I speak a lot out there as well, and a host of industries, this is a big one for me, but healthcare and others as well. Again, if you just look at the past six months and focus on say, the IT, ICS side, whether it’s CISA, the NSA, the FBI, the CIA, Homeland Security, Department of Energy, just about every three-letter agency you could possibly name has issued statements, warnings, testified before Congress talking about the fact that this critical infrastructure, as they call it, if it is defined critical infrastructure, is under attack, its potentially already been compromised.

So you kind of have that, and then you have the state of these devices, right? Like I always say, and I’m sure people that listen to this podcast are probably tired of me saying it, but it’s like living in 1994 when you think in terms of the security of these OT, IoT, ICS and other devices, right? 75% of them running with default credentials, certain configurations. They’ve got open Telnet, SSH ports all over the place. They’re a configuration mess, firmware’s ancient, on average seven to nine years old with 68% of them have 8, 9 and 10 CVSs. I mean, it’s not a pretty picture, right? So when you look at that from a security perspective, do you see the OT side of the house kind of understanding that threat? If so, how do they digest that? How do they think about that?

Michael Lester:

That’s a good question. What’s funny is I’ve seen multiple perspectives, kind of that IT perspective coming into that conversation and OT perspective coming into that conversation. It really starts with your architecture. The architecture is one of the easiest things to reevaluate and design it so that devices that are supposed to be on the same networks and talking to each other are on a separate network than your cameras or your door readers, for example. Once you have that architecture defined and your firewall rules aren’t any things like that, then you move on to making sure that you understand your assets, not only document and know your assets, but understand what they’re doing and make sure you have them documented in some form or fashion. Those two things are the first approach to getting that under control, right?

John Vecchi:

Yep.

Michael Lester:

Segment your network, understand what’s on it and what’s talking.

John Vecchi:

Yeah, and I mean, we’ve heard a long time the segmentation side. I mean, I’ve got my own opinion on that. I think segmentation isn’t really security, it’s something different. But when it comes to things, and again, if you look at CISA and all of the advisories and even the CPG, kind of the CISA, CPG checklist, that cybersecurity performance goals, you see these checklists, you see these fact sheets and they literally talk about these three common things which I talk a lot about, which is default passwords, like don’t have 1111. How about that? How about let’s start there?

Change the default passwords, shut down configurations that are network connected that you don’t need. So extraneous insecure ports and protocols that are wide open, for example, and patch maybe, update your firmware. Now, I know on the OT side, that’s like heavy stuff, right? I mean, the whole idea of patching is like, “What?” But I do see that changing. I mean, to what extent do you think that the whole OT side, do they see these three things? I mean, I know they segment and shut things down. Again, I don’t think there’s any more air gap. I just don’t think it exists anymore, right?

So I mean, forget that. So the segmentation side, but when it comes to this fundamental security hygiene that, again, I say and I have hacking demonstration, I say, “Look, if I’m a hacker and your passwords are default at a start, I’m just going to start there.” I can just go to Google and find out what the password is and just pwn that thing right there, right? So do you see that kind of awareness and how are they even thinking about those kind of hygiene issues on the OT side, right?

Michael Lester:

Yeah, very much so. In terms of that separating devices and making sure that only the ones that are supposed to communicate are communicating. Part of that process I would add in making understanding what ports, protocols and services are being used when you need to understand an asset. So you have those documented and you can be like, “Hey, we don’t need those.” But also you make sure that your most critical assets are segmented properly, and you look at that segment and you start, and I’ve seen many of them start looking at those default passwords, making sure that if it’s patched, that it’s done correctly, that a testbed is set up and been patched and tested previously.

Make sure to patch during downtime and applying just the basic cyber hygiene to the most critical part. Because when you segment correctly, that is supposed to be your most trusted zone. IT is less trusted than your OT network, and you segment it into your most trusted zones and you work on those. So there’s very much that want to work on those critical areas first and make sure that you don’t have less secure devices on the same network as a way to get in easily.

John Vecchi:

Yeah, no question about it. So visibility obviously is a big thing, and most of the technologies that are on the OT side are kind of passive detection-based, network monitoring, deep packet inspection to provide that visibility. A lot of organizations I talked to, even on the OT, ICS will tell you, “I don’t think I know half of my stuff.” On the industrial side, maybe it’s critical infrastructure with multiple sites they have and now they’ve got the batteries. I mean it’s just there’s so much, right? How would you describe the state of visibility on the OT? I think on the IT side, we’ve got that covered. I mean, in droves, right? But on the IT side, even on just XIoT all the stuff, how would you kind of describe the state of visibility today?

Michael Lester:

Yeah, so it’s not quite there in my opinion. Many sites aren’t even at that capability to have monitoring, right? They might not have even switches that have spanning capabilities. You have deep packet inspection versus active querying, right? Tools, as I mentioned earlier, we have advanced these tools to where, you know what, it’s not active scanning, but we’re able to query in the native protocols, right? We have found that these are safer methods and unlike scanning, especially older devices where it used to shut them down or brick the CPU, memory, et cetera, now we have safe methods to query these devices. We’re getting there. I don’t think we’re all the way there yet, especially island and networks or networks below, rings below the PLCs. That visibility, especially when you’re looking at deep packet inspection, you have to have multiple sensors in order to collect from those areas.

John Vecchi:

To even see it, right?

Michael Lester:

Yeah, yeah. That’s a big issue. You can use RSPAN, and then you have overhead especially on your core switches coming up from the access switches, but each level has its pros and cons. Now that we are seeing these tools advance and doing these active queries, we’re able to get more information around the asset. Deep packet inspection, you can only get so much.

John Vecchi:

Exactly.

Michael Lester:

I’ve seen vendors adapt because they have realized this, right?

John Vecchi:

Yeah, yeah.

Michael Lester:

They’re like, “Oh, we started off deep packet inspection. You know what? We aren’t getting as much as we want.” They’ve added active querying tools into their capabilities, right?

John Vecchi:

Yeah. Yeah, yeah. So it’s getting better.

Michael Lester:

Yeah, it’s definitely getting better. As we mature, include more devices into those active querying, I don’t think we’ll ever be able to include them all, but maybe standards can come out that IoT and XIoT in general adapts and will make that easier, right?

John Vecchi:

Yeah. I mean, it sure would be nice if we had some help on the fed side with this. It seems to be just plenty of warnings and advisories and things, but not much beyond that to speak of there. But you talked about something very important. I mean, how could we be talking about OT, ICS if we don’t talk about active scanning discovery, right? This has been one of the things, I’m sure you see it all the time. I come up against the mindset every day, and if I’m ever anywhere on the OT, ICS side or speaking about it, this is just absolutely, it’s like a cemented pole in the ground and it’s not quite budging, although I do see movement there and it seems that you’re seeing that as well.

Again, it’s like I say, it’s kind of the current state of more kind of, I guess, you could call them legacy or whatever. Just all these technologies that are kind of based on similar, they’re kind of NMAP, they’re passive, they’re spanning and tapping in and taking packets and sniffing and all this stuff. Then you also have the traditional vulnerability management guys, which are the ones that they’re made. They do wonderful and beautiful on IT assets and traditional IT endpoints, but I call it waterboarding when they go talk to an OT device, right? They just kind of turn that thing over and just go through 10,000 ports. I mean, these devices just die. They do not like that. So you can understand where that mindset comes from, right?

Michael Lester:

Oh, yeah.

John Vecchi:

But I think the big thing that I see is that, and you mentioned them, there are technologies now that are evolving and you see some of the passive guys starting to doing some queries. You still need to deploy all that stuff. So it’s still kind of passive, but they’re doing some queries like that. Then you have now a whole new generation, which is actually what we call it, intelligent active discovery, it’s more probing at tiers, right? You start at tier zero, you query, you probe, and you’re literally speaking to that device, but maybe three packets, just tiny. Once you see what it is that got you, you’re a sensitive PLC or some mission critical device, maybe you’re an infusion pump, life critical. I got you. Stop, move on, open the aperture a bit wider, talk to them, stop, move on, right?

I think the big thing is kind of the idea that that technology exists, and then going from that to organizations being comfortable with actually trying that. Because what we found is the only way to actually kind of address some of the fundamental hygiene issues like the passwords and the configurations, the management of these device, firmwares, certificates, monitoring them for drift or what have you, is to be able to be communicating directly to that device in its own protocol. I mean, do you see any movement with the realization of these new technologies? Is that happening? I’m sure we still have a long way to go, but is there kind of crack in that wall a little bit?

Michael Lester:

Yep. I’ve always said OT trails IT by minimum 10 years. We had some of these capabilities 10 years ago to put devices in their native protocol. But yes, we are definitely seeing people more willing to adapt that. I think from an architectural and from an economical standpoint, it makes a lot more sense. What if we can add an agent or a small VM to this network segment or this entire site instead of multiple one or multiple network traffic sniffers, right?

John Vecchi:

Yeah.

Michael Lester:

I think from an economic standpoint, it makes a lot more sense. Along with, as we mentioned earlier, being able to get more details about an asset than you would in that passive deep packet inspection.

John Vecchi:

Right. More high fidelity, and actually, we want to work on that. I think our statistics showed nearly 60 plus percent of organizations said they had a massive visibility gap in general, right? So just improving on that. How about let’s try to get it to 80, 85, 90% visibility? I think those kinds of technologies, those type of kind of intelligent active, those querying technologies can also expand the level of visibility, right?

Michael Lester:

Yeah. I think everyone knows that’s an issue, not having visibility into their network. When more people are adapting, or as this software becomes more mature and more people are adapting, and it’s proven its maturity, it’s proven it’s safe, people see it as a viable option. I do see a lot more talk about deep packet inspection and the passive network monitoring, but in terms of the active querying, and I see that being adopted as well.

John Vecchi:

I mean, that’s a great indication, right? I mean, I think it’s moving in the right direction, right? Which is great. Again, back to you, you mentioned kind of the SOC side, that OT, SOC, and I know Rockwell, I mean, you guys are amazing. You have all your products, but you’re managing a lot of other manufactured products, not just your stuff, tons of other stuff, which is quite amazing. So you get insight into all of that. So can you talk a little bit kind of about that practice? I think I was in speaking at the ICS conference in Atlanta last fall when the news broke that you guys had acquired Verve Industrial. So increasing your tools, I’m assuming that is an important piece, that strategy, those kinds of acquisitions are probably critical when it comes to things like that, managed SOC kind of service. Is that safe to say?

Michael Lester:

Very much so. I mean, I’m part of that team. I did an evaluation on Phosphorus, working with Verve, working with all these companies and bringing them into our global portfolio so that when we standardize them, when something’s deployed in South America, Latin America, it’s the same as it’s deployed in Europe. That’s kind of my job, along with the challenge of not managing just a hundred sites, but managing a hundred sites for multiple customers, right? So those are some of the challenges that we face. So building that kind of enterprise and global architecture to be able to manage those, and then evaluating and bringing in other products and capabilities into our platform is what I’ve been working on and why we’ve acquired multiple cybersecurity companies in the past four years, just bringing in those capabilities to manage these environments for our customers.

John Vecchi:

Yeah. I mean, I’m sure Michael, it’s not a small undertaking to integrate these technologies in, right? Then it sounds like, I mean, you mentioned earlier the automation side, right? I mean, tell us that, you’re probably working on the automation side with them. So not only integrating these, but also trying to automate these tools, like you said, a hundred sites, but then for a hundred customers a hundred sites, automation has to come into play big time, right?

Michael Lester:

Oh yeah. That’s part of our evaluation. As we look at bringing in a capability, one of the aspects that we look at is can we automate it? How easy is it to deploy? What’s the man hours needed? If we’re able to automate and save all these man hours, then that’s massive for us. That’s a massive win. They look and they have the product has easy APIs, good API documentation, it integrates with other capabilities, has web hooks, et cetera. Those are all pluses. It’s like, okay, now we can automate this, we can scale it, and we can bring it in as a global capability because even one hour saved at one site that-

John Vecchi:

Well, again, if you multiply that by a hundred and then a hundred after that, it really makes a big, big difference. I mean, there’s no question you guys and Rockwell, you’re on the front lines, at tip of the spear of this stuff. We could probably dive into this thing all day. But as we work to get to a close here, any kind of, I don’t know, stories from the trenches, something that sticks out in your experience kind of out there that you might want to share that might maybe even be a lesson learned to advice, you know what I mean, for our listeners?

Michael Lester:

Oh yeah, man, I could probably go on for hours just with lessons learned.

John Vecchi:

I bet. I bet.

Michael Lester:

I always try to accept some challenges because I know I’ll mess up and I’ll learn something from it and I’ll be able to learn from it. I guess I’ll start within the trenches. Doing vulnerability assessments was always my favorite. This one time, it was in the industrial sector, but we had, and I did the analysis for this. Unfortunately I wasn’t on site to see this, but they’ve [inaudible 00:40:07] data, brought it back. I noticed that there were iPhones and Android devices on this network.

John Vecchi:

Oh man.

Michael Lester:

I was like, “What is going on here?” So I started looking at it more in depth, and I looked at the engineer’s report and they had open wifi as part of what they just gave guests.

John Vecchi:

Wow.

Michael Lester:

I was like, “Okay.” Then looking more in depth, that wifi Network was the same network that was on the controls network.

John Vecchi:

Wow.

Michael Lester:

It was all one flat network. So that was a fun finding. I’ll always remember that. I’m like, “Man, there are people that just don’t know.” We’re talking about where to start. These are the low hanging fruit we’re talking about. Someone could literally sit out in the parking lot and get access to the [inaudible 00:40:59].

John Vecchi:

It’s just wifi, right? Yeah. It’s incredible. I mean, it’s beyond shadow IT. It’s like full on dark IT going on on the outside.

Michael Lester:

Shadow IT was always another one of my favorite, seeing what people would do when they didn’t have wifi. Another time, there are these gentlemen that worked at a remote site and it had ethernet ran to it, but they didn’t have internet there for them to access. So they plugged in their own wifi router and gave themselves their own little wifi network.

John Vecchi:

Of course they did. That’s easy.

Michael Lester:

Yeah. So that was another fun find.

John Vecchi:

Wow.

Michael Lester:

Then I think the last one I have is more, I was talking with my wife and she reminded me of this the other day, or I reminded myself when we were talking about different stories. One time to save a project quite a bit of money, about 10,000 plus, there are all these devices that need shipped to us, and it would’ve taken, because there was red tape and documentation, all this stuff, it would’ve taken 30 to 45 days to get shipped to us, which meant that was a delay in our project. So I was like, “Well, I’ll just go pick it up.” So I hop on a flight, wake up at 4 AM, hop on a 6:30 AM flight, fly down there, get a minivan, load all the equipment and drive it eight and a half hours back, unload the equipment and get the minivan back to the airport rental before midnight. So we were only charged for one day.

John Vecchi:

Wow.

Michael Lester:

So that was my longest day I think that I worked, woke up at 4 AM and I ended up getting home around 1 AM the next day, so.

John Vecchi:

That sounds like the guys that run our XIoT security lab trying to go get devices and bring them back. I mean, that’s literally what you have to do, right? You have to have those devices right there, and people don’t think about it. You have to have them. I mean, that’s insane. Well, that’s amazing, Michael. What a great conversation. I mean, like we said, we could talk about this for hours, but it was really fantastic having you today. I know you said you’re out there talking a little bit. Anywhere of special, social, anywhere that our listeners can find you or should look for you, Michael.

Michael Lester:

Yeah, I’m on LinkedIn. I stay most active on LinkedIn. Post occasionally, but I’m always scrolling, so I don’t have as much time to study and get certs and do all these things as I used to. So I really enjoy when people are posting their certs, what they’re doing to study, what courses they’re taking, what they’re learning. That’s kind of how I learned new technology that’s coming around that’s not directly in front of me. So yeah, please feel free to hit me up on LinkedIn. That’s where I’m most active.

John Vecchi:

Awesome. So everybody, you heard it, follow Michael Lester on LinkedIn and try to catch him when he’s speaking. A fantastic discussion. Michael, thanks so much for joining us today. A reminder, everybody, the IoT Security Podcast is brought to you by Phosphorus, the leading provider of unified proactive security and remediation for the extended internet of things. Thanks so much again to our guest, Michael Lester, and until we meet again. I’m John Vecchi. We’ll see you next time on Phosphorus Radio.