Take the Path to Secure OT with Patrick Gillespie

Phosphorus LIVE: Bi-Weekly Webcast

Date: November 14, 2024
Host: Mike Huckaby
Co-Host: James McCarthy
Special Guest: Patrick Gillespie

Transcript

Mike Huckaby: All right. Okay, well, let’s go ahead and get started. Thank you, everybody, for joining us today. Welcome to Phosphorus Live! This is our bi-weekly look at the very interesting world of Phosphorus, the extended Internet of Things.

I’m your host, Mike Huckaby. I’m the Head of Go-to-Market Enablement here at Phosphorus, and I’m joined by our device-obsessed Director of Sales Engineering, James McCarthy, who, by the way, is celebrating his two-year anniversary with Phosphorus today. Congrats, James.

James McCarthy: Thank you, thank you. Two years went by quicker than I was expecting. We’ll say that.

Mike Huckaby: And here’s to another two years to come.

Mike Huckaby: We’re happy to have you join us. We get together like this every two weeks to provide some education on the connected devices we all use to run our businesses these days and to give some insight into what Phosphorus does to help keep those devices secure. We have a simple format: a quick discussion on a specific topic and then a Phosphorus technical take related to that topic.

And it’s a quick hit—it’s 30 minutes. Today, we’re going to do things just a little differently because we have a special guest who will join us. But before we get to that…

James, hey, what’s been happening at the Phosphorus news desk?

James McCarthy: Well, like everything in the IoT world, a lot of things are happening on a regular basis. Lots of news to talk through, but there are a couple here that I wanted to highlight. So, you know, we have some new research highlighting critical vulnerabilities in a widely used Z-Wave chipset up to the 700 series. This exposes millions of smart home and IoT devices to cyber threats.

Attackers within radio range can exploit the flaws to disrupt services, drain batteries, intercept data, and even potentially seize control of devices. While the newer chipsets—the 500 and 700 series—can fix this vulnerability with a firmware upgrade, older models covering the 1, 2, and 300 series cannot be secured through updates. So, unfortunately, they will remain perpetually vulnerable to these threats.

For me, that highlights two things. One is that, in the world of IoT, sometimes you don’t get the option to ever fix the problem. At the very least, understanding where these things are, the risks they pose, and being able to highlight them can be really helpful. The second is that vulnerabilities sometimes need to be patched in IoT devices, and this can go all the way down to the chip level. It’s critical to have that visibility.

At the same time, Check Point Research recently reported a 70% increase in cyberattacks on U.S. utility companies this year compared to the same time period last year. With the rapid digitalization of the power grid exposing these critical systems to more vulnerabilities, it’s more important than ever to focus on securing IoT and OT environments. Experts are warning of the urgent need for robust security in infrastructure, visibility into what’s running at different locations, and well-defined processes to remediate risks.

Mike Huckaby: Wow! We’ve talked a bit about critical infrastructure on this show, and it’s not surprising we’re getting this data from organizations like Check Point. It reminds us all that we have to continue paying attention in this realm, that’s for sure.

James McCarthy: Yeah, absolutely.

Mike Huckaby: So, as organizations around the globe deal with the amazing growth of IoT devices in their environments, starting on a path to becoming more secure can be absolutely daunting.

To help us with this discussion—spanning people, process, and technology—we’re very lucky to have Patrick Gillespie, the OT Practice Lead at GuidePoint Security, joining us today. Hi, Patrick, thanks so much for being here.

Patrick Gillespie: Yeah, thanks, Mike. Good to see you, Mike, and good to see you, James.

Mike Huckaby: Now, we were just together a couple of weeks ago in Dallas. I got to watch you present, Patrick, and I thought it was very interesting. You have such a diverse background that really adds value to what you do today. Can you share a bit about your journey to becoming the OT Practice Lead at GuidePoint?

Patrick Gillespie: Yeah, really, just lucky and in the right place at the right time, I guess. My career actually started on the manufacturing side as a CNC programmer before I transitioned into IT and architecture. I ended up doing a lot of automation and industrial controls for a railcar manufacturing facility that made trains and parts for trains. I did that for a long time before I got into pen testing.

That shift is what led me to GuidePoint about two and a half years ago. I joined to help lead the Red Team, which focuses on threat and attack simulations. As we started taking on more OT work, my industrial controls background became relevant, and GuidePoint asked me to lead the OT and IoT practice. I eventually transitioned from the Red Team and have been focused on building out the OT practice and team at GuidePoint ever since.

Mike Huckaby: That’s fascinating. For those who might not be familiar, could you explain what GuidePoint Security does?

Patrick Gillespie: Sure. GuidePoint Security is a cybersecurity value-added reseller (VAR). For those unfamiliar with the term, a VAR essentially partners with multiple vendors to offer tailored solutions. Since our inception in 2011, we’ve focused exclusively on cybersecurity. Over the years, we’ve worked with hundreds of vendors across IT security—firewalls, identity access management (IAM) systems, vulnerability management, patch management, and so on.

What makes us stand out is that we’re vendor-neutral. This means we focus on helping clients figure out the best solutions for their environments without being tied to any single vendor. A bit of a shameless plug, but I believe GuidePoint is the best cybersecurity VAR out there.

Mike Huckaby: Thanks for explaining that. So, you started with a pen-testing background and moved into OT. I imagine that was a bit of an adjustment. Security and pen testing are quite different from thinking about how to apply security in an OT environment. How did you adapt?

Patrick Gillespie: It was definitely a big learning curve. I left the railroad manufacturing industry in 2016 to dive into consulting and pen testing full-time. For several years, I focused entirely on that. But when I transitioned back into OT work, I had to revisit many of the concepts I dealt with early in my career. It felt like going back to square one in some ways.

I’ve always enjoyed architecture and the defensive side of security. That’s where I began my career, as a big Cisco guy back in the day. My time in the Army, with an intel background, also helped—it taught me to think like the enemy, which is a useful mindset for pen testing and red teaming.

When you’re on the Red Team, you’re essentially emulating the tactics and techniques of adversaries to improve defenses. GuidePoint supported my growth, allowing me to shift into leadership and build the OT practice. Switching back to OT was a challenge because you deal with outdated systems, devices that can’t be updated, and environments that require a totally different approach to security in teams that have not typically worked together, you know, between OT and IT over the years.

Mike Huckaby: So, what’s your approach then, working with the people to get them all on board? James and I have talked to people that are very OT-centric, and maybe they’ve been working with specific industrial controls for years and years. They’re very, very sensitive to anybody touching them or doing anything that won’t allow business to move forward the way it’s supposed to.

Mike Huckaby: So, what’s your approach to being able to get people all on board?

Patrick Gillespie: Yeah, so we do workshop-style meetings, whether we’re delivering a service or not. When we start meeting with the client, we’ll have our PM team—we have a project manager on every service we do, essentially on every project—coordinate times to meet individually with each team. Even in IT, you have your security teams, of course, your firewalls, and then OT and IT architecture. You still have switches and routers, and wireless devices to connect all these systems. You have identity teams, server teams for virtualization—you have multiple IT teams.

Then, you have to figure out who owns the assets and who’s going to be affected by any new process or technology. Who’s going to answer the phone when there’s an alert in your SIEM or when your SOC gets an alert about an OT device or an OT network? Does the asset owner even know that they’re going to be called at 2 a.m. when you see an alert or anomaly?

So, really, getting everybody on the same page, we meet with each team individually to hear their concerns. While IT is focused on confidentiality and integrity, and then availability, OT flips that—availability is the highest priority. Like you mentioned, Mike, getting that product out the door with high quality and efficiency. But availability also means safety.

If you’re going to push a system or deploy a technology into OT, and it’s going to risk the safety of their people or their equipment, you’ll be hard-pressed to get that technology implemented.

James McCarthy: Have you seen that kind of resistance cause technologies to fail in the past? For instance, something seems like a great fit and could solve a problem the OT team has, but because it introduces too much change, they ultimately decide not to do it, even if the outcome might be worth the effort. Has that happened often in your experience?

Patrick Gillespie: Yeah, not really the change itself—it’s the way it’s introduced to those teams. So, if the IT director, CISO, or whoever says, “Oh, we need to do intrusion detection in our OT environment,” IT is inheriting the security of that traffic. However, they don’t own the assets. If they come in and tell OT, “Hey, we’re going to do this,” and don’t get their input, that’s where we see a tool not get implemented or fully used.

When the company doesn’t get value out of the tool, they stop paying for it. That’s more the issue. When we get teams on the same page and gather everybody’s input on the roadmap, we find compromises. For example, if you get ransomware in IT, let’s say on an accountant’s computer, you might shut it down to protect confidentiality and data integrity. But in OT, if you’re dealing with a nuclear facility and someone gets ransomware, you can’t just shut off the systems keeping the core at a stable temperature. That could cause a meltdown, harming many people in a large radius.

Patrick Gillespie: So, it’s really more about the approach than the technology itself. If OT understands what they’re doing and sees that these tools are safe, they’ll work with you. Back when I was in manufacturing—and I think I mentioned this to John Becky on one of these calls a few months ago—when I started doing security, Kali Linux didn’t exist yet. We used BackTrack Linux back then.

I would accidentally cause issues on printers or industrial control systems because I didn’t know better. I’d scan the industrial controls network, and suddenly, PLCs would reset, or printers would spit out a bunch of junk and scare people. You learn from those experiences. So, yeah, hope that answers your question, James.

James McCarthy: Yeah, absolutely. That’s a great point of view. I like that you talk about the concept of confidentiality, integrity, and availability being flipped in the OT space, where availability is king, and everything else follows. It’s such a different way to think about solving problems in that space.

Patrick Gillespie: Yeah, just like with any relationship—especially between OT and IT—it comes down to compromise. In my presentations at GuidePoint conferences or industry conferences, I usually show two pictures. One is a major highway with several lanes, symbolizing your IT bandwidth.

You want 10 gig connections to all your closets and users so they can stream March Madness or whatever else during business hours. You’re driving at 100 miles an hour, trying to push as much traffic through as quickly as possible. Then you’ve got your OT road, which is a backcountry road with potholes. The locals know where all the potholes are and how fast to take the corners.

If you come into OT trying to go 100 miles an hour like you do in IT, you’re going to cause yourself problems, right? You know you’re going to potentially crash and burn in OT if you don’t listen to their concerns. It’s like, “Hey, you might want to watch out for that pothole in Turn 6,” or whatever. Once you realize that, you think, “Oh, I need to listen to these people, not just force technology on top of their job.”

Mike Huckaby: That’s a great analogy—I like that analogy. So, once you get the people moving in the right direction, what are the key processes you focus on? What needs to change in that kind of move from people-first to processes? What are the key ones you look at?

Patrick Gillespie: Yeah. The first one we ask is: Do you have an incident response plan specific to your OT environments? That’s really the number one control from the SANS Top 5 Critical Controls for Industrial Control Systems.

You’re never going to be 100% secure. You might have zero vulnerabilities in your environment—and then something like CrowdStrike happens. Sorry to pick on CrowdStrike, but you know what I mean. You’re never truly secure from systems being taken down.

Most successful attacks—by “successful,” I mean those that cause outages or damage—are insider attacks, and many of those are not malicious. They’re accidents. When I was at the railcar company, we had two instances where our ERP system, which ran on UNIX, was essentially wiped. It was just a script that a contractor ran when building the system that accidentally made it recurring annually. So, a year of data was lost because of an accident.

It’s not always a hacker from some other country writing code at 2 a.m. and creating backdoors. Yes, those stories make the news and get publicity. But from a process perspective, securing and maturing the security of your OT and IoT environments is a long-term effort. It takes a good three to five years to build and maintain momentum.

In the meantime, what will you do if an accountant gets ransomware, and you don’t have proper segmentation between IT and OT? Do you know which cables to unplug or ports to disable to ensure it doesn’t spread to OT? Maybe you still have flat networks, or you don’t have an industrial DMZ firewall. Who are you going to call if there’s a virus spreading through a Windows-based HMI running Windows XP?

Your SOC probably doesn’t have the authority to shut off OT ports, and doing so could cause outages or even physical damage, depending on the industry. So, having an OT-specific incident response plan and detailed playbooks for different scenarios is critical.

Patrick Gillespie: We involve the people in this process by conducting OT tabletop exercises. These simulate different scenarios, helping teams refine their response. You cannot build an incident response plan in the middle of an incident. If you don’t have a plan, you’re planning to fail.

Of course, plans don’t always go as expected. As the saying goes, everyone has a plan until they get punched in the face. But if you’re doing exercises together, IT and OT teams learn who to call and how to coordinate. For instance, if OT gets a call from the help desk but wasn’t expecting it, they might ignore it or hang up. Exercises ensure that won’t happen.

That’s how we start. You’ll inevitably face incidents—whether accidental or malicious—and need to know how to respond as your environment matures. Playbooks will also evolve as you implement new processes and technologies. Your incident response plan matures alongside your environment.

Mike Huckaby: Cool. Now, let’s talk a little about ownership around Phosphorus Security. Over time—from your railcar days to today—what changes have you seen? I remember you mentioning some shifts in the CISO role. How has ownership of Phosphorus Security evolved over time?

Patrick Gillespie: Over time, CISOs have had to battle for respect at the board level. Twenty or thirty years ago, before there was a true CISO role, these responsibilities often fell under the CIO or IT director. Once companies began hiring true CISOs, it took time for them to get out of that mindset and gain a seat at the table.

Now, OT and IoT security are going through a similar transformation. However, CISOs are now essentially inheriting the security of IoT, and oftentimes the assets as well. It’s very easy—you think of printers, for instance.

Mike Huckaby: Alrighty!

Patrick Gillespie: Printers typically fall under your CISO’s role. We still see dot matrix printers in production, but cameras, for example, are usually easier for a CISO or security team to manage. If you have thousands of cameras at manufacturing facilities or oil and gas field locations, those don’t typically affect production, so they’re easier to handle.

However, once you start deploying XIoT or industrial IoT devices—like sensors for measuring the temperature of a heat-treating furnace or detecting a flood due to a valve leak—you’re getting data that helps the front office make better business decisions. The CISO effectively inherits the security of these devices but not necessarily the assets themselves.

For instance, PLCs (Programmable Logic Controllers), RTUs (Remote Terminal Units), and other sensors are often part of the production or operations budget. The CISO, however, inherits the traffic and behavior of these devices, especially if something malicious is going on. That’s where the real challenge comes into play—when a CISO inherits the security responsibility but doesn’t truly own the asset. Does that make sense?

Mike Huckaby: Yeah, sure. So, what recommendations do you have for CISOs? They come in with no clue about what they’re supposed to do. They’ve inherited all of this. What do you tell them?

Patrick Gillespie: The first thing is, do you have a good asset inventory of your hardware and software? Depending on the environment, this varies. For example, NERC CIP compliance, which applies to electric utility companies, has been around for 12 years and is well-matured. These companies are usually on par with their asset inventory for both hardware and software. That’s also why we do a lot of pen testing in this sector—they have the people, processes, and technology in place to make pen testing valuable.

Manufacturing, however—especially private manufacturing companies with no regulations—might not even have an asset inventory. Maybe a vendor installed something in the 1990s, and there’s a handwritten notebook with IPs and PLC numbers that only a 40-year veteran knows about. That doesn’t scale well, especially if you have 10, 20, or 100 sites globally.

That’s where tools like Phosphorus come in. They help with asset discovery in a safe manner for OT and IoT devices. With these tools, you can build an asset inventory for each location, down to individual manufacturing lines. You can’t protect what you don’t know exists on your network.

This ties into incident response plans. From a technology standpoint, asset inventory is the starting point. The question is, do you have an asset inventory?

Mike Huckaby: Very cool. Of course, that’s an area where Phosphorus excels in helping CISOs get that information. We like to say that.

Alright, Patrick, this has been fantastic. Our time went by really quickly, but I’ve really enjoyed your insights. James, any final questions?

James McCarthy: Nothing other than to thank you for taking the time. I know we’ve been working together in the field lately, and our partnership is highly valued. Your input is always fantastic. It’s been great hearing your point of view on all this, and I look forward to more in the future.

Patrick Gillespie: Yeah, yeah, I really enjoy working with you too, James, especially out in the field helping our clients.

Mike Huckaby: So, we have one silly question for you—something we ask everyone who comes on the show. Since our headquarters are in Nashville, we all love Nashville hot chicken sandwiches. What’s your favorite regional specialty where you’re from?

Patrick Gillespie: For Memphis, and it’s still true today, it’s got to be barbecue. I’m not a huge Texas brisket fan, but I love Memphis barbecue. I also enjoy Kansas City barbecue, which has a sweeter taste. I’m kind of halfway between Memphis and Kansas City now, so I get to enjoy both regions’ barbecue.

Mike Huckaby: Nice. Do you barbecue yourself?

Patrick Gillespie: I would never cook myself. No.

Mike Huckaby: Yeah.

Patrick Gillespie: Sorry, Mike.

Mike Huckaby: That’s good. I gotta figure out how… Yeah.

Patrick Gillespie: No, I’m not much of a cook. I’ll cook a duck or goose from hunting or something, but no, I’ve never actually attempted to barbecue ribs or anything like that. I just really like to eat it.

Mike Huckaby: Me too! We’ll have to make our way to Memphis and join you for some barbecue. Thank you again, Patrick. I really enjoyed having you on the show.

Mike Huckaby: Everybody, thank you so much for joining us. Next week, we have a special Phosphorus Live hosted by our CEO Chris Rouland and our CISO John Terrell. The show is called Factory Takeover: Exploiting and Securing OT and ICS in Industrial and Manufacturing Environments. You’ll get to see how some of these devices can actually be exploited—it should be a lot of fun.

As for us, the following week is Thanksgiving, so we won’t be back here on this show until December 12. Until then, everybody stay secure. Thank you, and we’ll see you soon.

James McCarthy: See you, guys.