The IoT Security Challenge: How Steven Edwards Tackles the Extended IoT Attack Surface

John Vecchi:

Well, hello everybody. You’re listening to the IoT Security podcast live on Phosphorus Radio. And I’m John Vecchi.

Brian Contos:

And I’m Brian Contos. So we’ve got a really special guest today, the original, the one and only international man of mystery, Steven Edwards. Welcome to the show, Steven.

Steven Edwards:

You guys are very excited about the introduction. That is awesome. Thank you.

Brian Contos:

So, Steven, before we get going, you and I have known each other for a while. You’ve got a great background. Can you tell our listeners how you kind of came up in the space, and what do you do in security, and what’s your day to day like?

Steven Edwards:

All right. Well, I try to be honest when I speak about this, I didn’t start off in security. I was a regular IT guy. I did desktop support, I did server admin work. I was living what I thought was the good IT life, and I decided to go back to school, and I was looking for where I wanted to take my career from there. And I got some good advice from a coworker when I said I was looking at doing my Cisco track or looking at maybe doing security, and he absolutely said, “Nope, you absolutely got to go do security. That is where the people are needed. That’s where the work is, that’s where you’re going to find the fun and excitement.”

So I did that. I went back to school, got a master’s degree, and I have not looked back since. Haven’t regretted in the slightest. Every day is a new adventure. Every day you can not only laugh, and smile, and enjoy what you’re doing, but you can also slap your forehead at what other people are doing and just get a little disgusted sometimes. But there is never a shortage of interesting things to work on and talk about.

Brian Contos:

Yeah, it’s always interesting hearing how people get into security. Some people take a direct path, but more times than not it’s a little bit of an indirect path. It’s a wiggly road. And I’m wondering, do you think having your base in that IT operation side, like the network and system administration really prepared you to possibly be more effective and secure, you look at things a bit differently?

Steven Edwards:

The different roads that you could go down to get into this field, it requires different sets of mindset. We need the straight up IT guy that just can help us run the tools and get them in place. But we also need that investigative mindset, the analyst role, that group of people that has that nebulous title of the analyst. It’s so far-reaching, so far stretching on what it is that is needed from you, the capabilities you need to have, the mindsets you need to walk in there with. But the number one thing is you have to be curious. And then number two, you have to be tenacious. So if you can be those things, you can find a long-term and very prosperous career in security.

John Vecchi:

Yeah, I can imagine. And I’m guessing, Steven, that probably was helpful and you started in IT as you said, and then you decided, look, I’m going to go back to school, I’m going to focus on cyber. I’m guessing that was probably helpful, the IT experience you had as you went into cyber. Is that a safe thing to assume? And second question is, tell us a little bit about, I mean, what is it in all these years you’ve been focused on cyber, what is it that just absolutely fascinates you the most that you love the most so far and all the time you’ve been switching and focusing on cyber?

Steven Edwards:

All right. Well, absolutely, when it comes to your IT experience. In my IT experience in particular, that is part of what led me to this space. You do have to know your technology to be able to work well in security. Like I said, it’s a shift in the mindset of how you look at things, how you go about it, how you have that tenacity and that just curiosity about what’s going on. What does that event log mean, what does that file do? You have to be a dog with a bone sometimes and just not want to give it up.

But the other part of it is that when I look back at my IT experience, I realize that I was doing a lot of those security things. I just didn’t understand that’s the label that it applied to. You read even your basic documentation from Microsoft back in 2003, and as they’re trying to walk you through how to set up a new domain, they’re explaining to you what you need to do for security. Maybe they just didn’t emphasize it as much as maybe you’d hope to, but it was there, the information was there, what you needed to do to try to keep that domain not only running correctly, but also keep it to where people you didn’t want getting in there couldn’t get in there. If we’d followed some of those best practices, some of the problems that coming on a little bit later may not have been as big or as painful.

But to your second question, what is it that interests me most about it? And like I said, what I always want for my personal life, for my professional life is I want to feel engaged. Like, “Hey, there’s something for me to do, there’s something for me to learn. There’s something for me to be interested in. I want to feel like every day is different from the next. If it starts getting too repetitive, I’m just sitting there doing the exact same thing every day. I don’t literally want to be the guy that just simply presses that button and then moves on to the next task and security does that.” I think other portions of IT have the same thing, but for my mindset, for my goals and the way I go about looking at things and the way I go about solving problems, security was absolutely the best field. Somebody could get that same satisfaction from being a program or somebody else can get that same satisfaction from being a network engineer, but for me it was security. So-

Brian Contos:

Yeah, I think a lot of people like that idea, that insecurity. If you’re doing the same thing twice, you’re probably doing it wrong the second time because it changes so quickly. Steven, what I’d like to know is, you’ve been with this very large organization before COVID, during COVID, and what I guess I’ll call now post COVID for the most part.

Steven Edwards:

Don’t jinx it. Don’t jinx us.

Brian Contos:

That’s right. That’s right. I’ll knock on wood. But from a security landscape perspective, did you notice any fundamental shifts, either from an attacker perspective, I know you work a lot with the executive leadership team, how they view security, any changes across those three distinct periods, I guess?

Steven Edwards:

No, man. One of the biggest things I’ve noticed is just simply that work remote that’s going on right now, people didn’t recognize the ability for employees to work remotely. Most of the things I’ve been doing on a daily basis, going all the way back to 2003, it didn’t really matter where I was doing it from. But because the technology was set up, the business model was just set up for you to be in an office, be standing there to be walking the data center floor. For some reason back in those days, they’d make me dry from Dallas to Fort Worth multiple times a week just so I could be on the data center floor. And even then you’re like, “I could have done this from anywhere.” That remote keyboard, laptop setup that they have in the data center, I can reach that from my home if I have to.

So when you got to COVID and everybody was forced to work from home as much as they were, it really changed the business mindset because they realized everyone could, we didn’t have to always be in the office. We didn’t have to have the big giant physical data center with the giant wall of glass, so everybody walking by could see all the blinking lights and everything. It wasn’t necessary. So you move everybody out of the office. But that changes a lot of things for security too, because it’s a completely different world when everybody is sitting in their own office, in their own house with their own laptop, and you’re trying to, “Secure the boundary of your organization, but now you don’t have a boundary.” Or if you have a boundary, 80% of your workforce isn’t within that boundary. So, what do you do to do that?

Now, you’re shifting a whole bunch of the controls that you had put in place out to the end user, and you’re putting it on their endpoint to now provide little mini boundaries around their system to give them the same level of protection as when they were in the office working on the full desktop. So I think, not only did the business change its mindset, and I think it was for the better, but I think that the nature of IT and the nature of IT security had to change their mindset to go along with it. And I think it could be for the better because… And I think we talked about this last time, we had a podcast, security professionals are very limited. There’s not enough of us. There’s enough people who would like a job and would like some money, but not enough people who have been trained and understand, didn’t have the necessary mindset to go into it.

So a lot of organizations have trouble finding that resource, especially when they were geographically confined on where they could find that resource. I have to find that resource within X number of miles of the Dallas area, otherwise I can’t have them. But now that boundary is gone and suddenly it’s okay for me to hire a guy who might need a job right now but lives in Florida, “Hey, bring them on. No problem.” So there’s been a lot of good things. Look, COVID was rough on everybody. I had people who were affected. It was not a joyful time period. But no matter what the tragedy is, it’s beholden upon us to learn from it to grow and improve, and I think we have done that in a lot of ways.

John Vecchi:

So, Steven, let’s talk a little bit about the attack surface and what you’re seeing there. Obviously this podcast, we’re all about what we call xIoT. That’s a broad category. It includes kind of classic enterprise IoT printers, VOIP phones, cameras, network attached stores, things like that. As well as network devices, wireless access points, load balancers, layer two switches, things like that. As well as the OT side, right? Skated devices and industrial control systems, PLCs, all of that. And what we’re seeing is up to 30%, oftentimes that can be part of the attack surface. What are you seeing overall as your attack surface, and how are you seeing xIoT kind of come into that?

Steven Edwards:

So yeah, the attack surface in our company is fairly broad. We attempt to limit that as much as we can. I mean, if you can walk out to the store and find a refrigerator that’s on the internet, things are getting bad. So yeah, my mindset has always been… And it’s probably because I’m old that, “Hey, not everything needs to be on the internet.” Just because it’s slightly easier doesn’t mean it needs to be on the internet. So I don’t need a refrigerator that can connect to the internet. But in our office, we do have quite a few things that don’t fall into a standard slot of devices, but it has a network connection. And if you’re not keeping an eye on things, you’re going to discover all kinds of stuff that you have on your network that you didn’t realize. Now, you can do the best you can go out there and isolate these network segments, put these firewall rules in place, don’t let X talk to X. The zero trust mindset comes into play.

But ultimately what it comes down to is most companies, they don’t have the time, the technology or the money to invest in what they really need to do to protect all of those different types of assets. So we have vending machines, network-based vending machines that they’re communicating with each other. Now, you can pay from your phone. So that thing has to have internet connectivity, it has logs, it has credit card information, they’re tokenizing it, but it’s going to have to get that payment somehow. It connects in with an app. There’s all kinds of things that it can do.

Now, every printer you have is on the network. Every scanner you have is hitting the network. We do have cameras all over the place and every one of those, it’s not the olden days of coax cables running all over the building to get to those cameras. No, it’s just simply plug that a thing into a switch and let’s put it on the network. So the problem has expanded greatly in finding an answer to that problem is a challenge. But in truth, the bigger challenge in my mind is to recognize that it is a problem, and to be able to convince somebody else, especially those that actually control the checkbook, that it is a problem and they need to go do something about it.

Brian Contos:

Yeah. What we’re seeing a lot of is we’ll go into an organization and we’ll say, “Look, you have 30,000, 40,000 50,000 xIoT devices. And that’s probably a lot of door locks, a lot of KVM switches UPS systems, voiceover IP phones, cameras, a lot of the things that you mentioned. And whenever they guess and say, “Well, we think we have X, we always know it’s times two.” They’re always off by 40%, 60%. I don’t know why it’s that, but that’s the number they always seem to be off of.

And then we explain these devices, they’re Linux, some are BSD or Android, Android’s just a flavor of Linux really. Some are running busy box, but they’re Linux servers, in a lot of cases, like these printers, these higher-end printers and even cameras, they’re more powerful than your laptop. So they’ve got more processing and memory and capability. Their Linux servers, the only difference is no one’s managing them. There’s no endpoint security, there’s usually no password or a default password or a very bad password. And they’re just opened up.

And when we expose that to executives like a CSO for example, and they look at it from that perspective, well, these are just unsecure Linux servers expanding my attack service disproportionately because I probably have a lot more of these than I even do traditional IT assets, in some cases, at least IT servers. It becomes a bit of an aha moment. But I try to look for IT analogies and I always think back to, is there going to have to be some major XIoT level catastrophe where people are able to get in, and pivot, and steal sensitive IT data and exfiltrated out from the 10,000 printers they compromised or these types of things. Is that going to have to happen or have we learned enough from those battle scars in IT to say, “Let’s not have to get to that point in the xIoT, let’s actually try to get in front of this problem before it becomes insurmountable”?

Steven Edwards:

Yeah, so the first interesting thing I keyed off on what you said there is that people are off by 40% to 60% when guessing that. That seems very polite on your part. I would’ve guessed that it was like four times the amount. If I’m saying I got 5,000 of them, then I got 20 because you don’t know the full extent. And the other big problem with it is, I guess there’s several that we could go into, but one of them that I can think of right off the bat is the fact that, yes, those are Linux boxes and Linux has a reputation of being secure, but Linux is also not a very highly publicized and attacked operating system either.

And when you go open source, and there’s some great things that happen there, and I’m glad for those things that have happened. But if everybody’s using the same open source library and Log4j comes to mind, then when that problem is discovered, it’s compounded so dramatically because now it’s within all of your other tools that are being used. And you can’t just simply go in there and make a little tweak and fix the problem. I mean, eventually with Log4j, they figured out that there was a setting, you’d go in there and make a change and you can solve it for most of the problem, but you’re waiting for all these other vendors to come about and provide you a solution to this problem. And they’re not doing it as fast because they probably didn’t write the library that was causing the problem begin with. So they can’t go change what it is. So they’re scrambling trying to figure out where do they go to get the answer.

And then like you said, on top of that, each one of those is basically just a really powerful Linux server sitting in your environment going unnoticed. And you find that we fall into the trap of wanting to protect the crown jewels. You always want to fall back to, “Hey, this is our data. This is the main thing, this is what we want to protect. But we don’t seem to take enough into account that the attacker isn’t going to start there.” I mean, that’s their ultimate goal, but they may not even know where that thing is, but they’re not going to start there. They’re going to start with the first thing they can to get a foothold and then start trying to figure out what you have. And one of the easiest spots they can go get a foothold is, like I said, that refrigerator sitting over there disconnect to the internet for some reason, maybe you don’t even know about it because just some guy in IT was following instructions and plugged it in.

That’s what happens with printers. And now we’re seeing monitors, big giant TVs that are connected to sound devices that are supposed to be WebEx hosted platforms, it’s real easy for them to get into a conference bridge and whatever. But they connected it to the network. It has that Linux system in the background, it has no patches for it. Nobody’s really producing massive updates. And even when they come out with an update for that system, who’s even taking notice that that update is out there and needs to be applied, and has a process to go do it doesn’t involve someone just walking around with a USB stick and plugging it in.

So there’s a big problem there that’s just simply not recognized for how big of a problem that it actually is. And so it’s not getting addressed in a manner that it needs to be. And the attackers are taking advantage of that. They use that as the door to get in and eventually given enough time if they go low and slow, they’re going to go ahead and get to your crown jewels. And it all started because of the printer in that VP’s office. That’s it.

John Vecchi:

Well, it’s interesting. I mean, we talked about the state of these devices, and you already addressed this problem with all the common libraries and white labeling and all of these, which means, you could have a VOIP phone and a camera, and some of the same vulnerabilities are common be between the two. So you’ve got lots of vulnerabilities, oftentimes shipping with these devices. You’ve got firmware that’s six years old, you’ve got all kinds of ports and protocols, HTTP, HTTPS, wireless, Bluetooth, town net, SSH, I mean there’s a lot of issues with those. What does security with these devices look like for you? We talked about the discovery issue, we don’t even know how many we have and where they are and what they are and what’s the state of those? Is it a discovery? Does it start with discovery for you, Steven? Or what does security look like for you given all of the devices that are out there and the state of these devices relative to your tax surface?

Steven Edwards:

It’s nothing but a challenge. And your standard vulnerability management scenarios just don’t seem to have a practical solution to the problem. So yes, you got to have a way of figuring out that it’s there, which means you’re taking your average vulnerability scanning tool. And what we’ve had to do is we just simply have to sweep across every private address space, even if it’s not supposed to be within our environment. It’s like, “No, we got to sweep across every single one of them because we don’t know where they are hiding stuff.” And we got to find. So once we found it, and then we got to figure out we have to categorize it. We got to figure out, “Hey, what is this thing?” And again, like you were talking about, here’s a bunch of phones, here’s a bunch of TVs that have been set up for WebEx sessions and some kind of office space or whatever.

Here’s all kinds of other weird devices. So once we’ve categorized it, now you have to think back, what do we do for a typical Windows box when we need to patch it? “Oh, okay, we’re going to turn to our standard solutions. Here’s SSCM, here’s IBM big fix, here’s Land Desk.” Is that even a product anymore. I don’t know. That was something I used way back a long time ago. And they’re going to have all these patches for us and we’re just going to be blasting those patches out to those machines, rebooting them overnight, all this kind of stuff, making sure they’re updated as best they can. Where is that solution or where’s that equivalent to handle all these phones? If you have these big vendors, Cisco, they’re putting out all of these phones, what’s their big management tool?

And then how do we integrate a process in to what we’re doing with the rest of our environment that makes sure that we are able to scan those devices and apply the patches necessary. On top of that, “Hey, if let’s say they have that solution to go patch it, what’s the notification process? Because I don’t know about you, but most of these vulnerability scanners, they aren’t really all that great for scanning that telephone and trying to figure out what’s wrong with it, where there might be a vulnerability in the thing.”

Brian Contos:

So yeah, Steven, that was great. I think you hit the nail on the head for so many of the big problems organizations are trying to address, just finding, managing, monitoring, building xIoT into their existing processes. So that as we close out your final comments here for our listeners, what kind of advice can you give to somebody listening in terms of addressing their cybersecurity strategy considering xIoT and other types of issues, maybe not just from a tech perspective, but again, because you work so much with business leadership and things like that process and people, what are some of the battle scars and lessons learned that you can share with anybody?

Steven Edwards:

So from the perspective of somebody at my level, some kind of manager, director, IT security guys that are directly responsible, they’re putting hands on it on a day-to-day basis. First you got to figure out how big the problem is. You have to some way of knowing as best you can, what is out there and what’s going on with it. Now, you’re going to have to walk into the rest of this, the rest of part of the engagement, ready to play the long game because you are going to have to convince your leadership, or your governance group, or whoever it is that you have to go to get the money when you’ve identified a problem to go find the right tool to solve that problem. You’re going to have to talk to them about the nature of the problem, how big the problem is, and you’re going to have to give them time and opportunity to think through it because they aren’t in the weeds of this stuff.

We are on a daily basis. They may not understand the aspect of it. You’re going to have to look for scenarios where possibly you can use, most everybody has to go through an annual pin test or something like that. We got to do it for PCI or whatever. “Hey, they got to come through, they got to do a pin test.” We’ll see if you can build something into your PIN test where they’re going to try to attack those IoT devices because if they attack it, they’re going to get through that thing. They’re going to get on one of your systems, they’re not going to stay there. They’re going to pivot to the next thing. But you’ve got to start showing them, and making sure that the things that were previously unknown are now known front and center, and being understood, and being a concern for those that have control of the purse string. And in control of your project roadmap for the next few years that understand this needs to be something that we need to address within our environment.

Because walking up to your financial management group or your VP or whatever, and just simply saying, “Give me x hundred thousand dollars to go solve this problem every year.” They’re just going to go, “What? I don’t know what that is. Why would I spend that money?” They’re going to say, “No.” It’s the same thing if it was just you and your checkbook at home. If somebody walks up and says, “Oh, hey, you got this particular problem in your soil and if you don’t go fix this…” And you’re like, “I don’t care. I don’t garden. I don’t have to go fix that problem, why do I worry about it?” It’s like, “No.” You got to show them over time what the problem is, why it’s going to be a problem if they don’t resolve it, how that problem can hurt the things that they do care about, which is also a trick of it is if you can’t get them to understand that if they leave this problem alone, it’s going to eventually affect the thing that they actually do care about most. And then things are going to get bad.

So like I said, for me, especially the way I’ve learned from my mentors, I have to be prepared to play a little bit of a long game to be able to show them and convince them the nature of the problem, and how going about solving that problem is ultimately in their best interest. I’m not saying I’m trying to trick them and make them think that the idea was theirs to begin with, but they have to understand it before they’re going to be willing to write the check to take care of it. But once they do understand it, once they do get a chance to see it, once they understand what it can mean, how it can affect their business operations, how it can affect their core data, how it can affect the crown jewels and the things they really care about, they’ll get on board and they’ll help you solve the problem, but you’re going to need time to resolve that. And you’re kind of going to need a little bit of a salesmanship to get there.

So hopefully, whatever VAR you work with, whatever reseller you work with, whatever product you’re looking at, hopefully you have good salesmen that can help you with that process because you are going to have to sell somebody, the fact that problem needs solving and this is the way to go solve it.

John Vecchi:

Well, listeners, you heard it right from the source. Precious advice and a fantastic discussion today. Steven, thanks so much for being our guest today. And again, thanks Brian, my co-host. But again, Steven Edwards, thank you so much for joining us on our podcast today.

Steven Edwards:

Thank you very much for having me. I appreciate it.

John Vecchi:

All right, everybody, and remember, the IoT Security podcast is brought to you by Phosphorus, the leading provider of proactive full-scope security for the extended internet of things. And until we all meet again, I’m John Vecchi.

Brian Contos:

And I’m Brian Contos.

John Vecchi:

And we’ll see you all next time on Phosphorus Radio.