Securing Critical Infrastructure: Challenges and Strategies with Sean Tufts

Transcript

John Vecchi:

Well, hello everybody. You’re listening to the IoT Security Podcast live on Phosphorus Radio. I’m John Vecchi.

Brian Contos:

And I’m Brian Contos, and we have a really special guest today, the managing partner for critical infrastructure at Optiv. The one, the only, Sean Tufts. Welcome to the show, Sean.

Sean Tufts:

Yes.

John Vecchi:

Welcome, Sean.

Brian Contos:

In the house.

Sean Tufts:

In the house. Thanks for having me, guys.

John Vecchi:

In the house finally. Finally in the house.

Brian Contos:

So Sean, before we get going, maybe you could give some of our listeners a bit of background about you and how you came up and what you do at Optiv.

Sean Tufts:

Oh yeah, great question. How far back do you want me to go?

Brian Contos:

All the way.

Sean Tufts:

The origin story?

Brian Contos:

All the way, Sean.

John Vecchi:

Yeah, man. Let’s hear it, Sean.

Brian Contos:

We want the Sean origin story.

Sean Tufts:

There’s a really small corner of the world that has I think there’s six of us that are former NFL players that got into cyber somehow. So there’s me, there’s a guy [inaudible 00:01:19], Pete Surrette, I don’t know where he is now. But there’s like five or six of us that all got our start, I was a linebacker for the Carolina Panthers. It was funny, I got out of football. Someone asked me that day, “Why’d you leave the NFL?” And I was like, “The collective will of 32 NFL teams. They all decided I was done. I disagreed.” But when I got out, I was like, “You know what? I don’t want to be in tech, I don’t want to be in oil and gas.” Those were the two and then I totally screwed that up. I’m in both.

John Vecchi:

Yeah. Look where you ended up.

Brian Contos:

That’s awesome. That’s awesome. So now your primary focus is on critical infrastructure, so what exactly does that mean? What are the areas that you’re really diving into down at Optiv?

Sean Tufts:

It’s changed a lot, actually. And people are going to write books about COVID, and I guess I’ll throw more fuel onto that fire. I worked at General Electric for a couple years, that’s where I learned cybersecurity. Built a couple wind farms in my day, worked at Baker Hughes, worked at a couple other spots, and we’ll go back to the origin story first. I got a cold email from GE that said, “Hey, we just bought this company called WorldTech. Who knows our clients that wants to learn cyber?” I was like, “That sounds like way more fun,” so I hopped in. We were doing nothing but oil and gas, heavy utilities, that was it. That was what we thought OT, IoT security was at the time. And then I tell you what, man, COVID hit and after Trump shut the world down, then that next week I had four conversations with toilet paper companies, all of them. They were all like, “We don’t know what to do. Apparently we’re critical infrastructure and we have nothing and we need something.”

Brian Contos:

“We ran out of trees.”

Sean Tufts:

Well, it was symptomatic, and then it’s been in these weird waves that are actually still continuing. For awhile obviously it went to pharma, and then it was logistics and shipping and distribution centers, and then the raw poultry crowd came, and then the raw meat crowd came. It’s been really interesting waves, watching new entrants come into the market and say, “Hey, we need to get our stuff together. We have all these physical assets, we don’t know what to do.” And it’s been an interesting ride since whatever that was in May, April 2020.

Brian Contos:

Yeah, I think we really redefined critical infrastructure during COVID.

Sean Tufts:

Well, of course.

John Vecchi:

I think so.

Brian Contos:

Who would’ve thought?

John Vecchi:

So Sean, tell us how did that then come to joining Optiv. Talk a little bit about your time there, and obviously right now you’re certainly considered an industry expert on the OT/ITS side and critical infrastructure side. What did that look like? How did you get to Optiv? And talk a little bit about your journey there to where you are today.

Sean Tufts:

Oh yeah, I took a great exit from GE. I went out in style. Actually, there’s a guy at one of our vendors we work with who watched me get laid off and I didn’t know it. I went up and met who was my new boss, and she was like five foot one and she turned around. I met her and shook her hand, it was nice. Turned around and she made this face I guess, that was like, “Oh,” and then I was gone the next week.

Brian Contos:

Oh geez.

Sean Tufts:

So I wrote a pretty good layoff out of GE. And you know what? We’ve had a lot of our peers that are kind of in similar situations right now in tight economies, tough tech market, and it was the best thing that ever happened to me. I was so concerned and uptight and just really anxious about getting laid off and thinking, “You’re a failure,” and all those kinds of things, best thing that ever happened to me. A guy over at Fortinet, Michael Hooper, shout out, he called a sales leader over here and said, “Hey, this guy knows a ton about a market. You guys stink in. Bring him in, see if he can make a fit.” And I started actually at Optiv in a sales role because I knew about all oil and gas and power and all those kinds of things.

So I got all our greenfield accounts and they were like, “Good luck,” and we started setting up relationships with new vendors coming to the market and new vendors and new vendors, and it kept getting bigger and bigger, and we kept finding more opportunities for bigger OT plays. And then my peers kept calling me and then their peers kept calling, and then it was like, “Okay, fine. Can I just do this for a living please? Can I just run the program?” So in that time, we grew to about a $50 million program between tech sales and services where we’re managing 3,000 sites for clients today, we’re doing risk assessments left and right. It’s been really fun.

Brian Contos:

It’s a rare breed of individual that can speak to the people that are wearing hard hats as well as the people that have pocket protectors, and not everybody can make that transition. And I’m just talking about the personas in this place, are you seeing a greater blend of the OT security folks and the traditional IT security folks more so than the early days, if you will, of the space?

Sean Tufts:

Yeah. The early days, it was a fight club episode, right? Remember that scene when a guy’s knocking on the door and he’s like, “If he’s tall, tell me he’s too tall. If he’s short, tell me he’s too short,” that kind of thing? All the OT heads when we first started this, when we were doing active scanning 15 years ago, they were like, “No, get out of here. Come back later,” and we kept coming back, kept coming back. I think right now it’s very dependent on industry, but I think it’s probably 50/50 where we have I think bridged those gaps and started to empower people that don’t look like your traditional PLC engineer that came from a vulnerability management side, that came from a SOC, that are starting to be allowed in and made friends with the OT heads that were really stodgy and kicking us out of rooms. So depending on the environment, right/ Distribution centers, a little more forward-leaning. Hospitals, definitely more forward-leaning. Refining and midstream, still a little behind, so it’s probably a 2:1 maybe, but I think it’s starting to change. We’re starting to get more entrants, which is good.

Brian Contos:

Where’s toilet paper land now? Because now you’ve really got me fixated on that.

Sean Tufts:

Just right on the TP. They’re coming around, I’ll say that.

John Vecchi:

They’re coming around. And Sean, one of the things that on the O/ICS side, even whether it’s industrial as well, is I think one of the challenges that we see a lot is just the various owners, the various teams that all have something to do with the whole idea of securing the OT/ICS, the kind of industrial IoT estates, and that’s a serious challenge. We see that time and time again. Do you see that, and what are some of the primary various teams that you tend to work with when you’re in some of these accounts with some of these different types of organizations?

Sean Tufts:

It’s bimodal. The first kind of teams we see is a cross-functional band of identity, network, engineering folks that come together and say, “Hey, we want to take this on.” They’ll usually deputize someone from the network team, maybe someone from engineering to go be that leader and take the first arrows in the back. And then the other type we see, and we’ve been seeing this more and more lately where they hire a really smart person, they say, “Hey, go fix all of the IoT OT stuff,” and the first thing that person does is say, “I’m one person. I need to go reach out and make friends there.” And sometimes they’re successful with setting up a program, sometimes they just wear their one hat and try to make incremental change. So we’ve seen both work, we’ve seen both fail. A lot of it depends on the ointment of the larger business.

Brian Contos:

So Sean, it was interesting. I was in Dubai last year and I know John’s going to be out there in the not too distant future, and of course critical infrastructure when it comes to power and energy, oil and gas, huge, huge opportunities there. And what I noticed was I was seeing a much higher percentage of teams in the Middle East that had OT security teams specifically just focused on OT security, and they were working very closely with the cybersecurity teams and the IT side and things like that, but it was a dedicated group. So it wasn’t like everyone kind of sort of had a responsibility, but it’s really the cybersecurity team running it, it was like look, there is a dedicated OT security group. Is that just a one-off that we’re seeing more out there or are you seeing that across the states as well?

Sean Tufts:

Geopolitically, we saw a lot more movement from cyber in critical infrastructure outside of the United States first. So going back to when I was at GE, we saw a lot more commitment from people in Middle East, we saw a lot more commitment from people in South America, Asia. There are some reasons behind that. We’ve always tried to identify what that leading piece was. Sure, some of it’s distance to a threat actor, some of it’s IT’s got a lot of really mature shops in North America. We’re pretty dang good, so we haven’t seen some of the legacy, legacy, legacy units that we would see in more developing countries. Don’t know where Dubai sits in that. So we saw them actually come together faster and they come together with more organization outside of the US, which I thought was funny and we could never really figure out. So we do see that in the Middle East. We haven’t seen it in Europe, actually. Europe is probably the last one to come. They’re starting to organize a little better over there now, I’d say they’re maybe 0.1 step behind us.

Brian Contos:

I always wondered when it came to that because you think of some of these regions, I won’t call out any specific countries, but they didn’t have the most advanced infrastructure. So the infrastructure that they are deploying now or they’ve recently deployed in some cases a lot newer than the stuff we’ve had in the US because we’ve just had it longer, and I think the analogy that I tie it to is credit card readers. They have had more advanced chip readers and things like that in Europe and parts of South America and Southeast Asia long before we had it in the US we already had this massive infrastructure and investment that was made in these older solutions, so we said, “Well, we’re not going to swap them out until they’ve expired.”

I always wondered if that was really what was happening for some of these big manufacturers or oil and gas and power energy as well. They’re like, “Man, we’re still depreciating this and these turbines aren’t like $5, they’re really expensive. So before we throw something new in here, we’ve got to get the life out of it.” So there was this combination of really new stuff as well as pretty old, some stuff that was really decades old, and you didn’t really see that much of that in other parts of the world.

Sean Tufts:

I’ll cosign that. I don’t have any data behind that, that’s my experience as well. When you see a global company and you start looking at their firewalls, for some reason there’s more modernization in the more developed countries, Europe, the US. So I think that is true. I don’t know if that’s a cause and effect thing, I have no idea, but that’s what I’ve seen as well, Brian.

John Vecchi:

And you mentioned a little bit ago, Sean, the geopolitical side. And recently I wrote a blog just with all of just look at the activity, if you go back even a few months for something like CISA and the kind of advisories they have, just the pace of those. I think at the time I wrote my blog, I think I looked back just a couple months and I think I said there was close to 60 different advisories on critical infrastructure, industrial PLCs, things like that. The thought was geez, at the same time, you kind of saw a little bit targeting that and I think from a geopolitical perspective, is it fair to say that some of the geopolitical things around the world could potentially mean these things are a bit more targeted? How do you feel about that and just some of the activities and focus on either it’s nation states or some of these cyber criminal gangs focusing on industrial and critical infrastructure? Is that real?

Sean Tufts:

Oh yeah, it’s very real.

John Vecchi:

What do you think?

Sean Tufts:

Actually, interesting, the numbers have gone down on the groups that are attacking oil and gas and utilities, which is a surprise. I am conjecturing, I don’t have any idea what’s going on in places I don’t want to talk about. When they saw that colonial pipeline was almost an act of war, at some point they were like, “Ooh, this seems like a bad fit for ransomware. We’re treading on some waters that could get very accelerated quickly.” So that was the inflection point where we saw them targeting other soft targets that wouldn’t necessarily carry that same weight, so that’s where we saw manufacturing, distribution centers, and there’s a lot of great research done by some people really seeing how much more there is. I also think too that manufacturing and those other markets and hospitals, softer targets. They haven’t moved as fast, so there was an easier target and less notable. So I think those two things are very true. We’ve seen that effort pivot a little bit in the last two years.

John Vecchi:

And real quick, finally, has that resulted and manifested itself in more urgency with your clients or is the desire to secure this stuff pretty much a steady state? What are you seeing?

Sean Tufts:

The whole market’s in this weird staring match. We know how big the problem is, we know the potential impact, but it’s still a Six Sigma event. So for a lot of companies coming forth and really committing from a personnel perspective, because FTEs are still our most expensive thing, from a personnel perspective, from a new technology perspective, from a new process perspective, from pulling off network people, pulling off identity, people pulling off cloud resources into other projects, it’s a big ticket item for a lot of operational people who don’t know how to manage that yet. So we still have this Six Sigma event we’re trying to protect against with a really big paycheck, so there’s still a lot of hand-wringing between those two.

Brian Contos:

I found that super interesting, the way you’re explaining that they targeted somebody and they’re maybe utilities or oil and gas that was right on the edge of maybe the response being a kinetic attack, and if prudence dictates that if I attack somebody that has the capability of calling up the Navy Seals to attack me back, maybe I change my targets that I’m going after. But now, like you said, they’re kind of moving to these soft targets. And my question on that is oil and gas, power and energy, pharma, maybe transportation, a few others, I think they over the last few years have really gotten it more, certainly so than they have historically. But now that they’re moving to these other targets maybe on the healthcare provider side and other arenas that have these IoT and OT large investments in this area, are they having to re-go through that education process again or did they learn from all the other guys?

Sean Tufts:

Nope. Didn’t learn from the other guys at all. Still walking in. It’s actually invaluable. No airport wants to hear how the ship channel is doing something right, there’s just like this hubris there. But when you can get someone to take a step back and listen and learn and say, “Look, this was what we saw. These were the activities we felt. This was the culture you were trying to break,” there’s a ton of commonality there. The hospital med tech is no different than the PLC engineer in a power plant from a Pavlovian perspective. I need uptime, I need my thing to be available and I need to ensure it’s not going to hurt anybody, and if I can do those things, then I don’t care what you do. But if you don’t tell me a great narrative on how to maintain my job while you’re doing your cute cybersecurity stuff, then I’m not going to let you in my building. We’re still facing that.

John Vecchi:

Yeah. Obviously, Sean, we focus a lot on the actual devices. These are smart devices and when you think in terms of OT/ICS and industrial environments, there’s a whole load of these and a lot of different types of devices. I’d love to get your thoughts just about some of the devices themselves, whether it’s PLCs, HMIs. You’ve got environmental sensors, you’ve got gateways, you’ve got ruggedized devices, robotics, all these. And a blend of that of course is things like cameras and ruggedized printers, which some might not even consider in the scope, but they actually can be pretty critical. When you see some of these devices, Sean, how do you think in terms of how vulnerable they are and what are some of those that come to top of mind for you, some of the basic critical vulnerabilities you see relative to the state of these devices?

Sean Tufts:

I separate them into things we can handle today and things we can’t. We just got done with the engagement in the Ohio River Basin somewhere, 10 million square feet of manufacturing. They had 20, what was it, almost 30,000 CVEs of nine or higher.

John Vecchi:

Wow.

Sean Tufts:

What are you going to do with that? Bring in a big firewall, set it on the edge and make sure it’s good, force everything through it. They’re not going to, and we as consultants or technologists are not going to advise them, “Hey, you should really go refresh every Windows box here.” That’s just not going to happen today, tomorrow, whenever. So from there we pivot to, “Let’s identify the crown jewels. What are the most important things?” For one of them, it was a recent client, it was the garage door. If that garage door didn’t open, product wasn’t going out the door. And they could move the product manually if they had to, if they could get the door open. So that was big, and then also there’s a funny contractual moment when the money change stands. When the product got out the door, that’s when the chick cashed, so that was another thing.

Then the other side of the fence is within that same environment, let’s say, there are things that we can fix today. We can go after the camera systems, we can go after the phone systems, the print servers. The little IoT guys that aren’t hard to solve, aren’t mission critical. Camera goes out for 25 minutes, that’s not going to be in theory a huge deal. Let’s go find those things and wipe them out because all too often, the access point to those hard devices are the not hard devices. So from a risk perspective, let’s take down the entry points. Let’s take down the known vulnerable devices that we can easily go fix. The camera’s running tele, right? That’s an easy one. We should be able to walk in and wipe out that entry point from a threat, what do we call it? Threat thingy, how big your threat is. What’s that called?

John Vecchi:

Yep. Like the attack surface.

Sean Tufts:

The attack surface. Thank you, John. CMO. Is that what I said, the attack thingy. Oh.

John Vecchi:

There it is.

Brian Contos:

Attack thingy, AKA attack surface. Everybody knows that, right? So Sean, I got to ask because as you’re going into these, it just keeps ringing off in my head that you must’ve seen some stuff. Are there any interesting stories from the trenches you can anonymize and share with us, just crazy stuff you’ve seen?

Sean Tufts:

We were sent to go find a bunch of highway cameras in a quasi-federal environment, three letter acronym. There was retail locations that they had set up that were part of the base and whatever they had, they were really worried about all the hick vision and all the stuff that we can’t have anymore. Sure. Okay, let’s go find those. Found them. They were doing dumb things, cool. In the process though, we found a bunch of smart TVs, a bunch. Smart TVs that were from good known locations, smart TVs from bad dumb locations, all the bad problems. But we found four of them that were actively streaming video out to Southeast Asia, so they just turned the camera on and we’re just trying to see what they could see. And we lost a little bit of traction after we identified it and saw where it was going. Incident response team came in, NDAs, all that kind of stuff. But it was eyeopening to think that they were just ready to wait and watch, and maybe someone with credentials walks through, maybe someone has a conversation they shouldn’t. That was interesting.

John Vecchi:

And since the FCC literally banned the sale, the importation, the distribution of a lot of these devices, we are kind of seeing that suddenly get some attention from clients just because they want to know, “Oh my god, do I have any of these?” Are you seeing just in general? Which you wouldn’t even, again, think I’m in an OT ICE industrial environment and I’m talking about band cameras, but is that something people care about?

Sean Tufts:

It is, yeah. A lot of places, obviously the straight commercial enterprises don’t really care that much. Anybody with a relationship with a government cares a lot. We had one of the Department of Corrections in the state, it was pretty funny, where we were working the IT staff and they were like, “We don’t have any highway cameras or Hikvision,” or I forgot what it was, the other one. And then we started looking at the Department of Corrections and they had a lot. And so the guy gets on the phone from the SOC and he is like, “You guys don’t have any of those foreign entity cameras, right? We talked about that, right? There’s none in your environment?” And he was like, “No, none.” He’s like, “Well, I see 50.” “Oh, where? What are you looking at?” And it was funny too, they were all named funny things. We couldn’t see the cameras and see the video feeds, but there was workout gym, there was weird things. We were like, “What are these cameras?” Straight off, they were right in the wrong places.

Brian Contos:

And then on the other side of the Pacific, they have a reality show called American Prisons and that’s where it’s all streaming from. You know what’s interesting about the Hikvision cameras is I was at a security event, it was physical security, it was logical security. They even had these hovercrafts, it was really interesting. But one of the number one sponsors of the conference, and actually to get into the facility, you had to walk under their arch branded arch was Hikvision. So even though it’s not welcome here in the US anymore, man, worldwide, there’s still a lot of that stuff and they’re not necessarily pulling it out, which is just crazy.

Sean Tufts:

Money wins, right? It meets the feature functionality, they work. You get video feed from it and it’s the right price, so a lot of people default to that. Security still isn’t the top of mind for every single entity that’s buying product and until we change that, we are going to live in that world still.

Brian Contos:

Well, sure enough. Now, do you think that there’s been so many government regulations and different frameworks and guidelines and best practices that are being pushed, whether it’s NIST or CIS or ISO or NERC or all these things that are out there, is it just word salad for advanced persistent auditors to come in and give you a headache or do you see that actually moving the needle and having an impact on increasing security within these organizations?

Sean Tufts:

No, I see it moving the needle and I see people that are invested into NIST branching out into the IoT version of NIST, into the critical manufacturing version of NIST. So from 53, you jumped to 82. And I think when we worked with clients who have that a strong CIS or whatever they want to do, when we turn that lens on it and say, “You’re meeting that and it’s really important to you, right?” “Yeah, we got to do that from our critical controls perspective.” “Okay. Well, you’re missing all these ones. Here’s a whole nother batch that you’re not aware of.”

That to me has been a big needle mover because we can show and highlight, “Here’s a standard, here’s your deviation from it. You believe in the standard and then on the right hand, you’re great. On the left hand, you’re awful. Let’s balance that.” Especially on clients where some of these companies, they don’t have venture funds, they don’t have a SaaS platform. They make things and if the making of the things gets disrupted, that’s a problem for the company. And so when we can protect that path and show them that they’re not protecting it today, that’s been a force multiplier.

John Vecchi:

And you mentioned a word right there, disruption. Right? And then earlier you mentioned the days of active discovery, active scanning. And the drill whenever we’re talking to anyone on the OT industrial side and it comes up to the whole idea of discovery and scanning, active’s a terrible, terrible, bad word and we kind know why, there’s been some bad things that have happened. And you can’t just throw a legacy scanner at this stuff for multitude of reasons. But do you see that changing, Sean? It seems there’s a little bit of a crack in that ceiling now where some are kind of coming back to the idea that as long as you can do this maybe safely and not disrupt the business, we might be open to it? Is that a fallacy or is there anything moving on that side?

Sean Tufts:

I see that very live right now. A lot of our passive scanners, great partners, conceptually you’re working on the margin. You’re working not in the kernel of a device, you’re taking some assumptive guesses and we’ve gotten great data there and really impactful data, but we’re still missing a chunk of it. We’ve got the always on component, but we need to be able to go interrogate these devices faster and to turn things over quicker and easier and get a correlation point because there’s still some things that come through like, “Hey, what is this? Are we going to spend two weeks trying to go research this or do we want to flash it and go?” I think that the passive group has really broken down the need for better data and in my view, having a passive and having active in the same breath where you can combine those two and do a one-two punch is really, really valuable.

And the new ways we’re doing that are so much smarter. The ways we used to interrogate a device was just hammer it until you get it and then you’re either going to kill the device or you’re going to get what you need out of it, right? Well, that doesn’t work in any vehicle, so let’s be smarter about what we look at in that device. Let’s address the device how it wants to be addressed. Once we do resolve what it is, let’s move off it, not come back. We don’t need to punish these things that aren’t able to take that beating. Let’s be smarter about it and release that really great, valuable tool in a way that can be absorbed. And we’ve seen a real big growth of people wanting that and thirsting for that most important bit of data, that last mile.

John Vecchi:

Yep. That’s encouraging.

Brian Contos:

Yes. Certainly, finesse is needed. We’ve seen the waterboard approach in IT forever, but when you start doing it to IoT and OT devices, they just roll over and say, “I’m out. That’s it.”

Sean Tufts:

Operators love it too. Well, let me go back. I’d say love is a new term, but they really want that data. They very much want it and they’ve never been able to get it, and they’ve always been mad about it. So when you can show them that you’re empathetic to their device legacies and you’ve got a way to play, then they really want to react to it. So they’re going to put you through your paces to make sure the active scanning piece can turn on in their environment but once that happens, they’re going to run wild and really start to do the information and the research they’ve been wanting to do for decades and they just haven’t been able to.

What was the big vulnerability a couple years ago? Oh yeah, the safety instrumentation systems from Triconic, right? No one knew how big those were. No one knew where they were. If you were able to in real time say, “I’m going to go find all these in my environment in 15 seconds,” well, now you can report to your board as opposed to we had one client in the oil and gas space, we asked that same question, “How many of these devices do you have?” “Somewhere between three and 3,000, we don’t know.” “Where are they?” “Earth, mostly.” They had no idea, and I can get that data in a push of a button or two.

Brian Contos:

Yeah. Having visibility, we’ve been talking about that in security since I think people were talking about security, but it’s just so foundational on this side of things because these devices are spread out in these strange locations. And to John’s point earlier, you’ve got these ruggedized devices sitting in a shack connected outside of a dam in the middle of nowhere, covered with black widows and stuff just sitting there running and operating, who knows what’s on it? The people that installed it are long gone and it seems to be one of those big problems out there. Are you seeing that organizations now, certainly working with Optiv, and are they getting their arms around that or they at least getting to the point where they’re able to take steps to find out where these devices are or is it still just, “It’s a blank space and I just don’t know what I’ve got”?

Sean Tufts:

The groups that are actually really pushing that you wouldn’t expect of the incident response groups because they’re like, “Guys, if you come to me and tell me that we have this problem and we have no other telemetry, I’m going to tell you to go find telemetry before I get involved.” And they’re putting up their internal SLAs to say, “We’re not playing with you unless you play with us, so we’re going to just not do that work.” And people are like, “Well, no, you have to do that work,” and they’re like, “Give me the data. If you can give me the data, then I can put my best of practice teams in play.”

And a lot of the new DFAR component, it’s funny when you read the OT DFAR part, first two chapters are all about asset inventory. That’s it. That’s all they need. And they’re not worried about advanced forensics, they’re not worried about quarantining, they’re not worried about all the things we see on the IT DFAR. They’re worried about solely, “Do we know what we have? How are you going to go research it?” That’s it. And I think when I read that, it was a really big eyeopener for me. How big is this problem? We really need to solve it.

John Vecchi:

And from there then, Sean, a big question. Obviously a lot of, like you said, the passive kind of tools, great tools, right? Amazing stuff, very detection-based, right? What are you seeing relative to an appetite to actually begin to think about in addition to seeing what you have in the visibility and the discovery, to actually go do something about it? So some of the underlying vulnerabilities, are you seeing an appetite now to people to actually say, “I get the detection stuff, I have it all over, but at the same time I do need to begin to think about going to fix some of this stuff,” whether it’s default passwords and credentials or the firmware? Like you said, town net’s wide open, whatever it might be. What are you seeing there?

Sean Tufts:

John, I can’t tell you that in the last three months, starting at Thanksgiving, October timeframe, we started hearing clients say more about, “Hey, I don’t need another assessment. I’m assessed to death.” Everyone’s always been assessed to death. They never weren’t assessed to death, but they’re starting to say, “Fix the problem, please. Let’s go actually jointly lower risk. Quit identifying, start lowering. Let’s get these things patched. Let’s go mobilize teams. Let’s get in front of this before we’re way behind it.” So we’ve seen that very acutely in the last couple months where people are just tired of hearing about these dumb printers. They’re just tired of it and they’re like, “I want them wiped out. Go patch them, get them done,” and that’s the expectation these days. There’s no money to replace the printer, but there’s a lot of focus on taking care of those legacy systems now that we can do it.

John Vecchi:

Yeah. And is it the same approach, kind of low-hanging fruit as far as what they’re going to try to go fix initially? Do you take that similar approach, let’s go get that low hanging stuff, brain dead, let’s go do that? Is that kind of the approach?

Sean Tufts:

Yep, that’s exactly it. We worked on a hydroelectric facility, the only thing you could see from the outside was the print server. They were properly segmented, they had all the Purdue model stuff set up exactly how you would want it except for the one printer. Well, let’s go take care of that. Number one, let’s tuck it back where it should go. Number two, let’s make sure it’s patched. Let’s make sure it’s fixed and it’s not so addressable, so easily found. So if you can do those things, then all of a sudden your attack surface, the attacky thing, it’s a lot smaller.

John Vecchi:

Again, the attack thingy.

Sean Tufts:

Yeah, the attack thingy. It’s smaller again. If you tuck that little guy in, that little printer and you patch it, now you’ve got a much more resilient and you can go from a risk of 10 down to risk of two really quickly.

Brian Contos:

So as you take everything that you’ve talked about during this conversation and looking at where we were and where we are, where do you see us in the next couple of years? Do you see more improvement from the manufacturer side in terms of the Siemens and Invensis and those folks actually creating devices that are more secure or do you see it really still being put on the shoulders of the operators or cybersecurity getting involved? What’s that future going to look like, do you think?

Sean Tufts:

OEM vendors or automation vendors are already doing that. They’re making better products today than they made before. Where was I? I was somewhere and heard Megan Sanford from Schneider talk, and she was like, “We want to know where these vulnerabilities are in our products so we can go fix them. Please highlight that, force it on us. The more you ask clients, the better we’ll be.” They want to be doing that earnestly. The problem is even if they fix every product they’re selling today, we still have 45 years of automation excellence that’s going to live in that world until it’s time to replace it. So even if the automation vendors totally revamp their programs, we have a whole earth full of stuff that just won’t get touched. So until we can really make viable progress, it’s going to fall to the operators. It’s going to fall to the network teams hiding this stuff until we can get in the right patch program.

So for me, I’m very optimistic that security can take the OT conversation for everywhere but one vehicle, and that’s vulnerability management patching. That needs to be a standalone team inside of an operator. They need to be able to be cross-functional, they don’t need to be siloed inside of cyber or siloed inside of engineering, and they need to be global in nature. So they need to be able to pivot and go find these things. What we found that manufacturing site in the Hyatt River Valley, 30,000 CVDs. You can’t just dump that into your environment that does CVS today. You can’t do it. It’s going to overrun it, you’re going to piss 100 people off, and then you’re going to mute everything that you don’t want to touch and then you’re going to forget to ever unmute it, right? Snore for three years, five years. We can’t do that. So those teams need to live on their own and be able to go impact change, do a crown jewel analysis, understand what’s important, and be able to go fix it.

When we build vulnerability management programs for clients, we create three buckets. The first one is the things you’ve patched so you can show progress. Most corporations when they see those big numbers are like, “Oh, you’re not moving the needle at all.” Let’s show where it did move the needle. The second bucket is the things that are coming up. We know we’ve got an outage window in Seattle, we’re going to get to Seattle in March. We know we have an outage window in Texas in April, we’re going to go there in April, and showing and publicizing what you’re going to be doing. That way when something flips, when something shifts, when something comes forward, something pushes back, you can migrate and work your teams accordingly.

And then the last bucket’s a big one, is the oh no, never bucket. We’re not going to do this. And then you’re developing your risk transfer methodology, whatever it is to terminate it, to transfer it, whatever. You’re showing that, most importantly, to your legal counsel. “We have decided to accept these risks. This will cost to remediate these activities, $200 million,” whatever it is. “We’re not going to do that today and if we give hat, we’re choosing to accept that, please increase our firewall budget.” That’s what you have to do, then we can show progress. We can start to build that mojo of, “Hey, we’ve done all the low-hanging fruit. We’ve got all the cameras, the printers, all that kind of stuff. Now we’re taking on some more important pieces,” and then you’re really lowering risk, and that’s what we’ve seen positive.

John Vecchi:

And even with the never bucket, they still need to understand where that stuff is, right? So there’s still a need and desire to see it and know where the heck it is so I can put it in that bucket. Is that safe to say?

Sean Tufts:

Oh yeah. If you don’t know what’s in the bucket, it’s not a bucket. If you don’t know what’s there, you can’t call it a risk mitigation because you just can’t. You have to be able to identify it and that way, if the IR teams or the operations teams come in, you can point them to where to go, but you have to have a seed in that game and if you don’t know what the assets are, you don’t have a seat,

Brian Contos:

That’s a bad actor’s dream, right? You’ve got all these devices that are really vulnerable and you don’t know where any of them are. I guarantee they do, and maybe Shodan even does, and they’re going to go ahead and go after those. You made a really interesting point that these manufacturers of these systems, these OEM systems, they want to do the right thing, but even if it was tomorrow and they released these just absolutely, completely secure systems, it’s not like all the existing systems are going to be washed out. We still have all these legacy devices to deal with it. It made me think of remember that movie Space Cowboy with Clint Eastwood? It was like from the ’90s.

It’s okay, it’s a good movie, but the idea was they were these astronauts during the Apollo era and they had built this satellite system and they had written the program, I don’t know what it was, Assembly or Cobalt maybe. Who knows? And it was up in space and there was nobody left that knew how to fix it, so they had to pull all these guys that had been retired for the last 25 years, stick them in a space shuttle and put them out to space so they could fix the satellite. That was the premise of the story, but there’s a lot of that.

And I remember back in probably 2000, 2002 timeframe going to some of these companies that had Windows NT4.0 as the operating system for some of these critical assets and at the time, it had been end of life for probably 15, 20 years and I said, “No, we probably won’t see that too much more in the future.” Just a couple of weeks ago, I was with a customer who still has not just Windows N4.0, but Windows NT3.51, which I think is almost 30 years end of life, which is just nuts. So these things are out there, they’re not going anywhere. So to your point, man, you better have a list and that thing better be on that list, so at least you know about it if you’re still not going to fix it.

Sean Tufts:

We found horror stories. We found active credentials or AD credentials that were older than active directory.

John Vecchi:

Oh my God.

Brian Contos:

Oh geez.

Sean Tufts:

So they’d been hard coded in whatever system they were using before, I don’t know how that happened, and for some reason then migrated over to AD sometime and because of that, it had a creation date older than them. And that was what, 93? When did that come out?

John Vecchi:

Yeah.

Brian Contos:

About then. Yeah.

Sean Tufts:

’89 was the credentials and they were still there, the password was not as complex as you would’ve wanted it.

John Vecchi:

Yeah. Well, pretty common on the OT industrial side too. Well, Sean, I tell you, we could talk a whole lot more, but as we wrap things up, obviously you are a trusted advisor for a lot of companies. You are an expert in this industry. I don’t know if our audience has ever been able to see Sean talk, you just have so much knowledge. And as we kind of say goodbye on this episode, what would be just a few points of advice you would give to all those operators out there, especially our listeners who might be on the OT industrial ICS side? Any just words of wisdom you might convey before we wrap up here today?

Sean Tufts:

It’s a people process and technology movement, it’s not one or the other. You can’t just build a policy and expect people to blindly follow it. You can’t just hope that you create a vulnerability management program and you don’t tell anyone, that’s not going to work. You can’t buy tech and just hope it’s going to do its thing, right? Getting sticky here and showing progress there takes a movement from all three of those parts, and there are definitely people out there that are willing to help advance all of those things at the same time. When we see too much real whack-a-Mole approach, I would say of the 200 clients we’ve worked with, there’s been only a handful that have really nailed that and been very fluid and bespoke about how they’re handling their own company and how they’re vocalizing programs, vocalizing new tech resources. Taking a moment to be thoughtful there will lead a lot of dividends in the next three to five years.

John Vecchi:

That’s awesome. Great advice.

Brian Contos:

So Sean, if people want to learn more about what Optiv’s doing in this space or maybe get in touch with you directly, what’s the best way to go around that?

Sean Tufts:

I’m like Coach Prime, I’m easy to find. LinkedIn, I’m out there. There’s one other Sean Tufts. He’s in Seattle. I have to call him the Sean Tufts after the Broncos lost. Shout out to the other Sean Tufts. So I’m on LinkedIn, and then the Optiv site has a really good OT platform. We built it there, and it’s great. Optiv.com/OT and all of our resources, case studies are right there. You can get in touch with Optiv persona right from that window as well.

John Vecchi:

What a great discussion. Sean Tufts, thanks so much for joining us today. Really, really appreciate that. And remember, everybody, the IoT Security Podcast is brought to you by Phosphorus, the leading provider of proactive full scope and unified security management for the extended internet of things. Thanks so much for joining us again, Sean Tufts. We greatly appreciate that. And until we meet again, everybody, I’m John Vecchi.

Brian Contos:

And I’m Brian Contos. We’ll see you all next time on Phosphorus Radio.