Getting Our Hands Around IoT Security: Dave Bang’s Journey

John Vecchi:

Hello everybody. You’re listening to the IoT Security Podcast live on Phosphorus Radio. I’m John Vecchi.

Brian Contos:

And I’m Brian Contos. We have an absolutely amazing guest today, Dave, the one and only, Bang. Welcome to the show, Dave Bang.

Dave Bang:

Hi. Thanks for having me.

Brian Contos:

Dave, I was just thinking this morning, we’ve known each other about a decade.

Dave Bang:

About a decade, yeah.

Brian Contos:

Maybe a little bit less, but yeah, right about there and have a lot in common. Every time we get together out in Texas, we always grab some dinner together and we talk about Marvel and DC for way too long. But if you could give some of our listeners a bit of background about you. You have a really cool, eclectic background, how you got into cyber and you’ve done some different things, so maybe you can kind of tell us your story.

Dave Bang:

Okay. I’ve been in the industry, in the IT industry, for almost 30 years now. I started at the company that I work for on a two-week contract, and that was in 1993. I did desktop support and I did OS life cycles. I like to tell people I started with Windows 3.1, And the last OS lifecycle I did was Windows XP. So I did a lot in between, and that’s about the time I moved into security on what was supposed to have been an 18-month rotation, and that was 2006. So I seem to be okay here.

I bounced around in security, did some firewall stuff, peripheral stuff, then did some application security and worked my way into architecture. Then I got fairly passionate about OT security. So I started partnering with the manufacturing folks and the guys on the shop floors and trying to bring their security game up to snuff. I’ve been doing that for the last probably a decade, but exclusively OT security for the last three years. Then I just recently took a new role doing security awareness, but my hope is to focus on the group of folks that are the operators and the engineers that don’t get a lot of security awareness time. At least that’s been my experience.

Brian Contos:

I was going to actually ask you that. What are maybe some of the big differences between working on the OT side juxtaposed to the IT side? You just mentioned one of them was probably general awareness and training and things like that about cyber, but what’s that like?

Dave Bang:

I would say what the topic always boils down to is which leg of the CIA triangle is most important. On the IT side, almost every security conversation talks about confidentiality, maybe a little bit of integrity, but it’s so ubiquitous that availability never really is talked about except in maybe like DDoS attacks or things like that.

On the OT side, it’s completely backwards or different, opposite. It’s all about availability. It’s all about keeping the manufacturing systems online and functioning and producing whatever it is that they produce. Integrity, probably is second. With Stuxnet and things like that, integrity’s become more important because the data that comes out is used to make business decisions as well as safety decisions, but most of them couldn’t really care less about confidentiality. That’s something for somebody else. That’s a security thing, not something they really care too much about on the shop floor. So I’d say that’s probably the biggest difference.

The things that people think about and they’re concerned about and they’re passionate about are different. One of my goals is to understand that passion and priorities and tweak the language and the messages such that it makes sense to those folks.

John Vecchi:

Yeah, David, it’s interesting-

Dave Bang:

Those guys don’t care. They barely check their email. They certainly don’t really understand why invoice fraud or anything like that would be important to them. But if I can talk about ransomware on the OT networks or things like that, that’s something that, it hits their availability. That’s something that’s a concern to them.

Brian Contos:

Absolutely, absolutely. As OT and IT continue to mature, but I guess cross pollinate a little bit more, and I think of energy production, and energy production they have to do. They’re producing more than they need, so they’re selling it off, and that’s all happening in real time. For that to occur, it has to connect to IT systems. So a lot of the theoretical unicorn air gaps that never really existed, but maybe they sort of did sometimes, they’ve disappeared. With that, it means that you’ve got to talk about unidirectional diodes and how you’re going to connect these environments together and who’s monitoring what. And over here it’s being monitored by SCADA and over here it’s SIM and these different groups. Are you finding that the folks on the sort of, I don’t want to call it legacy, but from the IT side of the house, are they starting to up their OT chops at the same time, just like the OT folks are starting to update their IT security chops?

Dave Bang:

My experience has been they’re not as quick to that as the OT folks are, right? The OT folks are starting to see, there’s big benefits to learning some of the IT stuff. There’s efficiencies to be gained, there’s automation to be used, things like that. The IT side of the house, they don’t have a lot of visibility into the OT side, so therefore gaining knowledge and experience around that is not something that they really think about. That’s been something I’ve been working on for the last few years is trying to get that visibility into the OT side and trying to get people to rotate through those kinds of roles or at least into positions where they interact with those roles, because I feel like that’s a big opportunity for that cross-pollination to happen.

Brian Contos:

We’ve both been doing this for too long and you’re one of the very few individuals I’ve met worldwide that’s spent a good amount of time on the OT security side and the IT security side, and I think you really hit the nail on the head is rotating these people, getting some of that cross-pollination between the teams. Because you said it so well, you think about it differently, whether it’s confidentiality, integrity, availability, or it’s just what are my day-to-day concerns?

It’s just a different mindset. So you might not be thinking about the threat actors and the potential of impact to your network in the same way. And the IT people have just as much to learn, I think, from the OT people. And it’s understanding the business, and that could be manufacturing, power generation, transportation, you name it. OT now is pretty much everywhere. It’s not just limited to a few critical infrastructure organizations. So I’m wondering, with that in mind, and the fact that you have lived in both of these worlds, what are you seeing in terms of the threats and the trends and the bright PRI business priorities really as it overlaps these two generally different roles?

Dave Bang:

So a lot of the same threats are there as far as threat actors, but their motivations are different. So what you’re seeing on the OT side is they are going… A ransomware attack is not there necessarily to get money. They’re looking to maybe disrupt services or gain money not for the release of the systems directly, but more for as an industry. So if they can attack one company and one OT network, that’s information that they could possibly use for the next one and the next one and the next one. And so the motivation’s different there. It’s not as much for monetary gain… I mean, example, but they can affect each other. So if an OT system goes down, that definitely affects the business because they may have to shut a plant down or shut a manufacturing line down or something like that, and vice versa.

We saw it with the colonial pipeline. Their business system was what was attacked, but because of the disruption in their business, they had to shut down the pipelines. It couldn’t deliver product to the East Coast. And so it got a lot more attention, I think, than what was originally intended by the bad actors. So, I think it’s not so much that you’re seeing different threats, they’re just different motivations. And because OT, for the most part, has not been a target large scale, it’s led to kind of a complacency in a lot of people that work in that environment, which is a different complacency than what IT people have. But they’re starting to, and I don’t want to say starting to, the last five years or so, that’s become a much bigger concern and you’re seeing a lot more people include OT security in their design of systems and their priorities, questions from the board level, things like that. And so you’re starting to see a lot more attention paid and a lot more funds released to do activities for securing the OT networks.

Brian Contos:

So Dave, we’ve talked a little bit about OT and we’ve talked a little bit about IT and how these two worlds are colliding, hopefully in a very positive way, and they have been for at least five years. But what I’m seeing now is this big embrace of xIoT. And xIoT, the way we think about it is everything from enterprise, IoT, so your printer is voice over IP, phones, security cameras, OT, all your industrial control systems, PLCs, SCADA, so on and so forth. And then network devices, these embedded devices like network-attached storage or switches, load balancers, wireless access points. So when we think about that, and we know the numbers are huge, usually there’s about three to five xIoT devices per employee in a company. So 10,000 people, somewhere between 30 to 50,000 devices. It’s going to be a little bit higher for those companies that have industrial control systems.

It’s also going to be higher for verticals like retail, a little bit lower for law firms, for example. But three to five is about the average. So I guess my question to you, as somebody, again, that’s lived on the IT and OT side, how do you think the industry is going to react now that they’re finding out I’ve got 50,000 Linux Android and BSD devices that are unsecure, unmanaged, default passwords, sitting all over my environment, making my attack surface footprint massive. Are we going to be able to get our hands around that quicker than we did when we said, hey, let’s start looking at IT and OT security together because now we had this huge attack surface? I guess my question is, are we ready for this?

Dave Bang:

I think there’s two questions, right? I’d say no, we are not ready for this, but I do think with the lessons that we’ve learned, I mean, it took us 20 years to think about security on the IT side. It took us 10 years to really think about security on the OT side, I figure two to five years to really get a handle around security on what you’re calling the xIoT. Because people are more aware now, this is not everybody’s first rodeo. The people that have an interest in bringing all these types of devices into their enterprise, I think are a little more aware of what the potential impact could be.

A lot of it is going to rely on the vendors, right? Because I mean, what we’ve seen with IoT devices being hacked in the past, it’s generally been a poor design problem, not thinking about, hey, who could possibly use this light bulb as a botnet to attack a bunch of other things? Those kinds of things. So it’s going to take vendors partnering with enterprises and enterprises requiring their vendors to help them be secure. And I think that’s something that’s been difficult to leverage in the past has been enterprises putting their foot down and go, hey, you are not the 800 pound gorilla in this situation, I am, or my industry is. And we are concerned about this stuff, and if you want to play at our table or in our playground, you need to think about these things.

And they need to do two things. Vendors, I think, have two responsibilities. One is making sure that their products are capable of being implemented in a safe and secure method, but I think they also have the responsibility to educate the people who buy their products and how to set them up in a safe and secure way. Now that doesn’t do much for the 30 to 40,000 devices sitting at the company you were talking about. Or in a case like a company like mine, probably closer to 80 or 90,000 devices, when you start adding in all the sensors and record keepers and things like that out in the plant. A lot of them are not hardwired anymore. They’re not a direct analog connection back to the PLC or whatever. They’re using some kind of wireless protocol to get the data back, because it’s more cost-effective and more reliable and things like that.

So companies have to stop and get a grasp on what do we have. I mean, we talk about, oh, we’ve got all these xIoT devices around, okay, what is it you have, right? That’s probably the first thing. And it’s just… We went through this with IT, we went through this with OT, we go through this every time is you wait until after the cow’s out of the barn before trying to count them, but getting a good inventory of what it is you have and where they are and how you’re using them lets you prioritize where your risk is and lets you focus on that, the higher risk stuff first.

John Vecchi:

Yeah. And let’s talk a little bit about… That’s a perfect time to talk a little bit about that visibility and what you can actually do to these devices. So we talked about xIoT. So again, a PLC is an ICS is an OT is an xIoT, but traditional IT tools oftentimes don’t work too well with these OT devices like they do on traditional IT assets, computers, endpoints, traditional things, right? So right there you have this difficulty in, A, getting the visibility you kind of need to understand how many devices you have, what’s there… Doing a posture assessment on them to understand what it is, and then actually being able to talk to them in a way where you can go do something about that. So can you talk a little bit about that, especially given that you come from traditional IT and IT security, how do you handle that? How do you approach that, given the difficulty of these devices and that traditional IT tools don’t really work too well, right?

Dave Bang:

Well, yeah. No, they don’t work well at all. And I think you’ve seen, again, in the last five to six years, you’ve seen a lot of the IT-based tools claiming to be capable of doing OT stuff. But the reality is they probably work okay with an OT device that was built in the last two or three years. But there’s… Most of those things that are out there have a 20-year, 15 to 20-year life cycle or longer. And so those devices don’t play well with the normal IT kinds of vulnerability scans, inventory tools, things like that. So to me, there’s two things. If you want to play in that space, you’ve got to understand how these things talk to each other and what they’re capable of accepting as an input or a query and be able to determine that and talk to them in their language. They’re much more likely to survive that conversation if you’re talking to them the way that they are programmed to talk.

And then the other piece to that is, in some cases, you’re not going to be able to directly query them or you’re not going to be able to put an agent or something like that on, so do you understand how they talk across the wire, right? Can you look at the bits as they go across the wire and say, oh, that’s an Emerson PLC talking to an Aspen Historian or whatever, and based on that traffic kind of know what firmware version it’s running or whatever. That to me is the only safe way to really understand your environment out there.

John Vecchi:

And as a follow-up to that, do you think that because of the challenge of tool sets trying to actually talk to these devices and understand them and therefore give you visibility of what they are, where they are, their status and state. Is that what is leading to a lot of hesitancy by practitioners and leaders in security like you to actually try to do anything with these devices? Or what is it that’s kind of leading the apprehension to actually go and try to do something about the state of these devices?

Dave Bang:

To answer your question, yes, I do believe that that helps fuel the apprehension. Some of the apprehension is just, hey, the last time we tried this a decade ago, half of my manufacturing floor puked on itself and it cost us millions of dollars to come back up again. That’s a big part of the apprehension. So a lot of it is prove to me you’re not going to do that again. But then the other half is we have tools that work on the IT side.

Now I’m being told I need to buy a different tool for the IoT or the OT or the xIoT, and now I’m back in the late ’90s, early 2000s of having to have a suite of tools that all do the same thing but only talk to specific types of equipment. Where is my single pane of glass for all of my environments? And I think we’ll get there. But again, I think that because these things are so independent and so different from each other, you kind of have to start with best of breed kinds of tools because proving the value of having that visibility will then allow you to get the vendors to join together, like we went through 20 years ago, and start to get tools that cross the platforms. But if you don’t show the value of it, it’s difficult to get the interest, right?

Brian Contos:

Yeah, so many great points there, Dave, to unpack. The first one is that organizations were burned, right? Because every cybersecurity company that learned how to spell the word SCADA said, oh, we have a SCADA solution too. I remember when this was happening and they started throwing out IDSs and then later IPSs and scans and all these tools that were built for SCADA, and they weren’t.

Maybe they even were able to spell Modbus or DNP3, but they weren’t. And they throw them out here and they didn’t really add a ton of value and sometimes it was negative. Sometimes it impacted that big capital A, availability. So they were gun shy. And I remember the early days of integrating like checkpoint firewall with the ISS/IDS and ISS could be set up to say, hey, if I see something that looks like an attack, I can automatically tell the firewall to block this. Well, then the CIO tries to get in and he’s got a weird VPN client or something else, and then fast-forward a week and you’re never doing that again because that realtime remediation burned you.

And I think you’re absolutely right. I think that’s what the industry saw a lot of. And it makes me think through the early days of Phosphorus, five years in stealth mode to essentially figure out a way to build an xIoT communication solution that just happens to do security. And the idea was it was like C3PO, right? It was a diplomatic droid that could speak the binary language of water evaporators. So you have to be able to speak all these disparate languages in a very safe way, and that’s a unique challenge. And you can’t just go out there and scan away and send malformed packets. And as John likes to say, you can’t waterboard these devices into, tell me what you are, because you’ll knock them over. So the point I’m trying to get to in a very long and drawn out way here is because of that and because of that apprehension around this and that tools like Phosphorus, we’ve been shipping for about a year.

So it’s not like a firewall. It’s not something that’s been around for 30 years. It’s still relatively new. A solution that can discover your xIoT devices, patch them, manage the credentials, the certs hardened, all these great things that we did back in the early ’90s that now we can actually apply to all of xIoT. This is a relatively new space, but that with the fact that people were burned right by some of these early entries into this market.

Do you think that the cyber criminals and the nation-states see this as a great opportunity because now they can count on organizations to be passive, they can count on organizations not to take the steps to secure these xIoT devices, just close their eyes, cross their fingers and hope that nothing bad’s going to happen, even though they do have 50, or to your point, 100,000-plus xIoT devices that are Linux and BST and Android with no passwords and high level of vulnerabilities and old firmware? Well, this is a great attack footprint. So do you see them going after this now as one of the new news for their targeting capabilities?

Dave Bang:

So I don’t think I’ve seen a lot of actual attacks, but to me, it seems like a greenfield attack vector, right? Because there’s a lot of it… There’s a people talking about it now, but that’s what they’re doing is talking, right? They’re recognizing where the gotchas could potentially be and they’re very hesitantly bringing that in. And I don’t think, again, because some of it, we’ve been burned, some of it is how big is the threat really, right? Is that where I should be spending my money? If I put my ubiquitous black hat on and I want to steal data or gain access or whatever? Absolutely, the IoT devices, I mean, we’ve seen it happen. We haven’t seen it on the industrial side very much, mainly because I think it’s not as easy to get to them. But once you do get to them, a lot of times they’re pretty wide open.

And that would be, I think, those things will start to become more of an attack vector, mainly because the industry’s starting to close down the attack vectors that have been available up until now. So you’re seeing guys like me that are bringing a lot of focus to securing the OT devices and doing the kind of basic block and tackling of patching and antivirus and reviewing firewall rules and doing those kinds of basic things. Well, that makes all those nice easy holes that were there. So the bad guys are going to go look for the next nice easy targets that people either aren’t paying attention to or haven’t realized they’re a threat yet.

John Vecchi:

Yeah, and it’s interesting, and again, it sounds like we hear this quite often, right? Until they’re kind of attacked in a very visible, painful way, organizations may tend to just lower that in priority, although again, we’ve seen attacks. You’ve got our RSACs where they’re targeting these kinds of devices for botnets, you’ve got the quiet exit where they’re going to breach you with a traditional, say, phishing attack, and then they’re going to pivot to one of these devices and then actually use it for xFill, for god’s sakes. And then you have things like the Siemens S7, the S7 crash, and Siemens kind of giving people like yourself a list and, hey, it should be easy. Just go update the firmware, rotate the passwords, shut off any extraneous services and ports and protocol. Yeah, it’s simple stuff, easy to do. So-

Dave Bang:

If you know where it is.

John Vecchi:

If you even know where it is and how to go do that, so we can see that there’s that kind of waiting period of, I have to see these things give me some pain and get attacked, even though we know there there’s attacks out there. But do you also think that we’ve, maybe from an OT perspective, maybe put our heads in the sand a little bit around the idea that, hey, they’re all isolated networks. I mean, again, and I just say Stuxnet was a pretty isolated network, that didn’t turn out too well for them. Is that also in play here, the idea that it’s isolated, you can’t get to it?

Dave Bang:

Well, I think the attitude is certainly there, that if you look at the Purdue model, those industrial Internet of Things sit down on the Layer 1, Layer 0 kind of areas. And the folks that recognize that security on the OT networks is an important thing, again, they focus on the Layer 3, where you’ve got your normal Windows type devices and then the Layer 2 where the control systems are, but they still see the Layer 1 and Layer 0 as kind of being not necessarily isolated, but unassailable, right? Because you have to get through these other things to get there. And the reality is the bad guys know that and you can go on… A quick Google search will tell you who a company gets their control systems from, you can go on eBay and buy some of those systems, especially the older ones for a few thousand dollars, set up your own lab, come up with a very targeted attack vector using those sensors and stuff, and it’s there.

So from a priority standpoint, I think the people have been focusing on those Layer 2, Layer 3 devices because they are the easier ones to get to. But I do think that the next level down is going to be the next set of attack vectors. Because again, you spend the time, you lock the front door, they’re going to figure out to go in the back door. You lock the back door, they’re going to start jimmying Windows, and wherever you’re not looking or wherever you haven’t spent your time and effort, they’ll figure it out.

Brian Contos:

You’re so right. I can’t count how many times we’ve gone into an organization where they’ve spent tens of millions of dollars, if not more, application security, network security, endpoint security, on and on and on. But a lot of these critical servers have KVM switches connected to them, lights-out management, UPS, they’re in racks. None of those devices are secured. They’re usually in Ubuntu Linux or something like that that’s plugged in. If you can get access to that KVM switch or that light-out management port, it’s smooth sailing, I can spawn a shell, virtual terminal, shut the system down, change the settings, upload software. Game over, and I don’t have to, to your point, bang my head against this very secure front door when, hey, somebody left the porch door open right here. I’ll just walk right through it. Look, I don’t have access to this one system now. Now I’ve got access to thousands of systems, thank you devices that manage other devices.

Dave Bang:

Right.

Brian Contos:

So with that said, Dave, some of that’s been in this space for so long, but do you have any interesting war stories you can share, whether it’s from the OT side or IT side or that kind of merge the two or anything like that?

Dave Bang:

So I just got back from Chicago from a ManuSec conference, which is one of the ones that I like to attend that does the IT/OT security crossover discussions, right?

Brian Contos:

Yeah.

Dave Bang:

And one of the stories that I told there was… So the organization that I just moved out of, we’re bringing security professionals to the OT side and matching them up with OT professionals that have a passion for security. We’d just hired a new person, young, right out of school, passion for security, passion for manufacturing, kind of that purple unicorn, and they were taking him around the plant, showing him the manufacturing unit, and here’s how things work. And when you talk about this, this is what that looks like, and just kind of giving him the grand tour. And he noticed there was a gentleman off the side that had a laptop that was plugged into one of the PLC devices.

And so they walked over to him and turns out he was the vendor representative that had manufactured that particular PLC name, shall we change to protect the guilty. And they were talking to him and he was saying that he was there to install some patches and do some firmware updates and things like that. And they watched him download a file from the internet and then upload it to the PLC. And the gentleman from my team goes, “Wait, you’re not supposed to be able to do that. There’s no access to the internet. We’re not supposed to be gaining access to the internet from this process control network.” And the vendor’s like, “Oh, no, I’m not doing it that way. I’ve got my phone connected to the internet and I’m tethered to my phone, so I’m just using my phone to connect to the internet and download these files.”

And he kind of scratched his head and was like, that just doesn’t sound right. But he didn’t say anything at the time because he is like, I’m new, maybe this is authorized or whatever. And so he brought it up to his manager and his manager called me and we’ve got to talking. They’re like, no, that should not happen. You need to go make sure they’re not doing that anymore. But it opened the question of is this happening other places? And as it turned out, lots of places were doing that, and it wasn’t malicious. They weren’t trying to do anything bad. They were trying to just do their job that we pay them to do. And it was easy, it was simple. They were able to do it and didn’t think anything about it. But what that opened up for us was, well, okay, this is a legitimate need.

These vendors need to be able to get patches, firmware updates, coding updates, whatever, from their systems on their network to our systems, how do they do that? And so it kicked off a big architecture discussion about how do we safely do that? But the initial thing was, holy crap, we spent, like you said, millions of dollars putting in the standard remote access solutions and file transfer solutions and stuff like that. And people are just… We’ve all seen the picture of the gate with the dirt roads driven around the gate, and that’s what happened. Always, it’s stuff like that reminds me that if you make it so secure that it’s inconvenient, people will bypass the security.

They’re smart, which is one of the reasons why I like to focus on, you talk about people, processes and technology. A lot of the technology… A lot of the security discussions focus on technology, and they don’t think about people and processes. And to me, if you don’t give all of them equal time, the people will go outside of the process, go outside of the technology to get their work done, because it’s easier or faster or whatever. So you have to think about all of them. That’s probably my favorite recent story is just because it was something none of us had ever thought of.

Again, it goes back to we assume things are air gapped, right? For years you couldn’t get a cell signal inside of a manufacturing plant, so it was never a concern. Well, guess what? There’s 10 times the number of cell towers and they’re a lot stronger and 5G penetrates a lot deeper, and now you’ve got this access that you never had to worry about that’s just out there in the air.

John Vecchi:

And I guess that leads me to kind of a follow-up question, is your advice on where should practitioners, teams start? We talked about many of the situation of these devices, majority of them have default passwords, if any at all, right? The firmware’s six, seven years old. The devices are very old. They’re loaded and riddled with vulnerabilities. If they do have certificates, they’re probably out of date or self-signed, or something’s wrong there. The firmware is very old, lots of potential extraneous ports and protocols that are turned on that shouldn’t. So a lot of things can be done. Where would you start? Where would you advise people to start? If you can go fix something first, from low-hanging fruit, what would that be?

Dave Bang:

So I would probably approach it from two directions. One is I would do a large scale pilot on the IT side, looking at the things people understand when you talk about them. So printers, network switches, refrigerators, whatever else you’ve got on the network that has low risk if something bad were to happen and kind of prove your bonafides about the fact that you can work with these things and not tip them over, but at the same time, find a place that’s got all the industrial internet of things that you’re worried about, but is potentially lower risk. So a place that’s got all the equipment that you’re looking for, but if something does tip over, it’s not going to go kaboom, right? Someplace that has a safe environment that, yeah, it may stop production, but it’s not super impactful. It’s not going to be risky to people’s lives, things like that.

And again, prove that you can deal with those things without tipping them over. And then you’ve got facts to go to the folks in the more risky areas to say, hey, look, here’s all the devices that we’ve detected here at your company that were in these situations. We were able to bring the visibility, we were able to prioritize, we were able to make the changes, and nothing lost connectivity. Nothing had to be restarted, or if it had to be restarted, it was restarted in a controlled way. That gets you past the apprehension, right?

Because a lot of that is once bitten, twice shy kind of things. But if you can approach those folks with the facts, this is what we did in your industry, this is what we did in your company, here’s the value of having it and here’s the lack of risk of implementing it. I think you get a lot stronger buy-in. I would say you’ve probably got the executive commitment, but you’ve got to get the trust from the people in the trenches or the ones that are going to have to clean up the mess if this thing barfs on itself.

Brian Contos:

Yeah, it’s funny you laid it out like that. That’s exactly how I’m seeing it rolled out in production. So a lot of folks will be, I want to say a little bit more cavalier when it comes to IoT devices, and they’re saying… And they’ll say, look, we’re just going to roll it out on and across all 50,000 printers. We have one hotel chain that’s close to a 100,000 printers. And they say, we’re going to try this state or these five facilities first, these five hotels. And then they start off with things like, well, let’s first add it to a PAM solution. We’ll rotate the credentials, so we’ll use Phosphorus for that. And then they’ll say, okay, that worked great. Now let’s turn off Telnet and Bluetooth and wireless on all those devices. Okay, that worked. Now maybe we’ll manage the certs and then we’ll go ahead and get into the firmware side of it, and then that works.

And then I see folks that also have OT. They’re almost always, well, let’s kick the tires on the IoT. And okay, you’re right, everything worked. Okay, now we’ll go ahead and do the password stuff. And they seem really keen on that, that’s a great first step. Some of these devices take a four digit numeric pin. Some of these devices can have 20 characters, but no special characters. Some can have special characters, but no backslash. So you have all these little -isms for all these different devices and they want to test that out, hardening. When it comes to firmware, I would say that’s probably the one where people need to be the most sensitive, as they should be, right? To your point, they’re the ones that have to clean up the mess and a password’s one thing, but if you screw up the ladder logic on something that’s completely something else, right?

Now all those set points are messed up. And what I’ve seen they like to do is they like to pick a couple devices, and you said it great, let’s pick a couple devices that don’t go boom, and let’s go ahead and manually push the firmware update from Phosphorus out to that device. If that works, now let’s try automating a dozen devices. Okay, that worked good. Now let’s try three dozen. And I actually think that’s the best way to do it. I would hate to be the person that comes in and says, do it everywhere all the time. Let it go, turn on the fire hose.

No one’s going to do that, especially people at OT that have been around for a while, they’re like, no, we’re not going to do it that way. So definitely taking those baby steps, I think that was great advice from your side. Absolutely. And it can go relatively quick. I mean, there doesn’t have to be a year between these baby steps. Sometimes there’s only a few days or a few weeks, but it does make sense to roll out in that process. Dave, as we wrap up here, I have a really important question to ask because we’re both Comicon and comic book guys. So Iron Man suit, Tony Stark suit.

Dave Bang:

Okay?

Brian Contos:

Jarvis, was Jarvis the operating system or was he just the Siri for the operating system on the front? I think he was integrated more. He was like the whole operating system, right?

Dave Bang:

He was the whole operating system, but he was the whole operating system for the house-

Brian Contos:

Everything.

Dave Bang:

For the cars, for everything. Yeah.

Brian Contos:

So within the suit, you’ve got Jarvis in this operating system, but I’ve got to imagine you’ve got a whole bunch of xIoT devices in that one suit. My question-

Dave Bang:

Oh, yeah. The whole thing is nothing but a collection of IoT devices. Yeah.

Brian Contos:

My question is, do you think there’s more xIoT devices in Iron Man’s suit or one Walmart store?

Dave Bang:

Well, okay, so I would say the Infinity War nanoparticle suits probably win for individual IoT devices. Yeah. But the Iron Man 1, 2, and 3 suits, definitely a lot less there than your average Walmart.

Brian Contos:

Well, very good. Well, awesome. Thanks so much.

Dave Bang:

Is that nerdy enough for you?

Brian Contos:

That that’s pretty perfect. That’s pretty perfect.

John Vecchi:

Well, I’m glad we settled that question on the podcast, as well as a host of other critical things, and it was really just such a fantastic discussion. Thanks so much, Dave, and amazing advice, tremendous advice. I hope our listeners caught all of that, and it was just wonderful to have you on the podcast today. Thanks to my co-host, Brian, and to our guest today, Dave Bang. Thanks so much, Dave, for joining us today.

Dave Bang:

My pleasure. I enjoyed it. Thanks for having me.

John Vecchi:

And remember, everybody, the IoT Security Podcast is brought to you by Phosphorus, the leading provider of proactive, full-scope security for the extended Internet of Things. And until we meet again, I’m John Vecchi.

Brian Contos:

And I’m Brian Contos.

Dave Bang:

And I’m Dave Bang.

John Vecchi:

And we’ll see you all next time on Phosphorus Radio.