Mastering IoT Security with Julie Tsai: Lessons from a Cyber Pro

John Vecchi:
Hello everyone. You are listening to the IoT Security podcast live on Phosphorus Radio and I’m John Vecchi.

Brian Contos:
And I’m Brian Contos, and we have a really fantastic guest today. Welcome to the show, Julie Tsai.

Julie Tsai:
Hi Brian. I’m so glad to be here. Thanks for inviting me. I’m excited to get into security and IoT with you.

Brian Contos:
Absolutely. And we’ve been wanting to interview you for a while, so we’re going to just jump right into it. But could you give our listeners a little bit of background about your journey and how you got into cyber and what it is that you do now?

Julie Tsai:
Sure, absolutely. So I’ve been working in Silicon Valley for more than 25 years now, and I would say my cybersecurity journey was completely accidental and was unintended up until about maybe about 10 years ago. And in fact, when I had first gotten out of college in the mid-90s, I was like, oh, maybe I’ll just go try out a different city, a different part of the country. And there was never a reason to leave. There was always these really exciting things going on here in the Bay Area. I found myself moving into… Actually through the generosity of friends who I used to know from the school newspaper, started working as a network controller, which was my first hands-on job in tech debugging network interfaces for a small ISP that was run out of a basement and eventually bought by one of the major providers. And that was like the first early dot com years and a lot of fun, chaotic. And we’d go and you would learn from the people around you. You would go to the bookstore and grab a O’Reily book and just try to figure it out. And that was the beginning.
And as I got deeper and deeper into the technology at the time, just thinking I wanted to get more grounded and understand it better because I’d actually graduated from school with a liberal arts degree and around that time realized, okay, I’m interested in media. The most interesting things in media are happening on the internet. I just have to learn more. I just found myself drawn into the more and more empirical and hard parts of the technology.
So moved into system administration, did a little bit of everything full stack for my first few years, including development, front end. And then at some point realized, well actually I think I’m actually strongest at system administration, surprisingly, because of being able to look at all parts of the system and being interested in debugging complex interactions and things like that. So just went all the way into that.
Eventually I found some of the self-healing and automation technologies, which were at the early parents of today, what people use Chef and [inaudible 00:03:04] CFEngine and Puppet were the start. And in my eyes, some of the best theoretical as well as practical underpinnings were put down by Mark Burgess about how to orchestrate finely detailed systems across big groups. And it was a great way to begin [inaudible 00:03:24] further and deeper into system administration and coincidentally also security because a lot of the things that you would try to architect and enforce throughout a large distributed system, if you could use configuration management and self-healing methods, all of a sudden you were getting these big returns to scale and windfall wins in terms of consistency and transparency.
And that became my specialty as an individual contributor. Doing that sort of DevOps and DevSecOps stuff in the early days. And at the time, all I was trying to do, was trying to help solve my boss’s compliance problem without forcing my entire team to run around the office with clipboards because that to me just sounded like the worst way to work. So how are we going to make sure that we are secure and compliant without having the means to hire a ton of people? And the self automation healing led in that direction.
A few years after that, after every job I was working for a lot of little startups, I was doing security stuff on the side and saying like, okay, well we just got to get this in. We got to solve this compliance issue along with other work. A VP over at Walmart said, when I was applying for a DevOps role he said, “Have you considered going into information security?” I was like, “No, it’s siloed, it’s boring. I’m not going to do that.” But then I went and interviewed with the CISO at the time and found, oh, this actually looks really interesting. There’s a really big scale of interesting complex problems. You get to use all of your skills and just went deeper and deeper into it over the last 12 years.

Brian Contos:
Wow, that’s such an amazing journey. And it’s so funny when you mentioned the O’Reilly books. I had a similar experience. I used to collect them. Right? So I do remember one of my first gigs was having to administer sendmail, and I went and picked up the O’Reilly, which has a big bat on it. And if anybody ever had to do send mail, that book was like a phone book was a bit of a nightmare. And then like DNS BIND of course by Cricket Liu, which is-

Julie Tsai:
That’s a classic.

Brian Contos:
… I think I still have that one sendmail I think I burned. But I think I still have DNS and BIND. So out of curiosity, so what a great way to come up and I love the liberal arts background and then everything you did with the early startups on the telecom side with the ISPs, what were some of the early security challenges when you were first started getting into it? What were the threats you were concerned about and what were some of the things you were trying to mitigate at that particular point in your career?

Julie Tsai:
Yeah, it’s interesting. You kind of come full circle. In the very, very beginning, before you’re steeped in all this stuff. You’re just trying to keep the lights on it and understand what’s going on when something goes wrong and pray that it’s not a security issue. And I remember a few times we’d be working on a project and something unexpected would be happening. You’re like, okay, please don’t let this be that. And then you started to dig around and you’d know that sinking feeling of like, okay, this does not look good. This is unexpected here. This has been messed with. Okay, there’s no good reason. And then you start to put together those pieces.
And I feel like in the early years too, you still had to use a lot of the quaint, ground pounding research methods. Going and calling up the best experts that you knew, digging around a lot on your own in the source code and then the logs and figuring out all of the areas where you wish that had been instrumented better. So those are things that come up. Sometimes it would come up and it’d be benign and small and you just take care of it. There wasn’t a big compromise. And then sometimes, you know, would realize, okay, this is a company threatening issue and scrambling for resources because a lot of small startups in the early days, everyone’s just focused on growth. So you don’t have massive teams or instruments devoted to security. You might have one person who’s considered a security expert on the team and you go to them where they have certain responsibility, but there wasn’t in the early years as much of an awareness around chief security responsibilities and what it really means to do it right.

John Vecchi:
Yeah. And Julie, it’s interesting you talk specifically coming from a startup environment where you said you’re small, but yet you still have an attack surface. You’re still getting targeted. You still have all the same threats as the big guys have, but small and you have no team. So I mean, how has all that come together for you today in just dealing with the kind of attack surface we have today, which just seems to get broader and bigger and deeper and wider? And has that helped get to a strategy today to deal with that attack surface even despite challenges, resources, gaps, all that kind of thing?

Julie Tsai:
That’s a great question. I think some of those things you learn, even when you think that, “Oh, this isn’t the way that it’s supposed to be done right.” Are actually super, super useful. I think the two things that I’d call out specifically and encourage people to develop at any stage of their career are being able to deal with ambiguity. You’re going to come in with a lot, you’re going to have come in with theories as you get more knowledge, you’re going to have more and more deep frameworks in your head about how to address stuff and including all the literature that’s out there. But you have to be able to stay adept and light on your feet, and you have to be able to turn at moment’s notice to say actually the information, the data’s not showing this. You got to go down this other route. Time is of the essence. You cannot exhaust everything.
So being able to deal with the ambiguity, synthesize all the information that’s coming in and realize sometimes you’re going to be an uncharted territory and it’s going to figure it out. Your team’s going to figure it out. You’re just going to carve your way through that jungle.
I think the second thing that was super helpful throughout this period was learning how to deal scrappy. You know, got a mission-critical issue in front of you. No one necessarily knows anything. You’re going to have to figure out how bad it is, how to disclose it, how to run it up the chains of command and get the right resources on it. And sometimes in the beginning it will feel very lonely. It will feel like there wasn’t intention, there wasn’t interest in it. It’s potentially really rough. But I think that being able to look at that situation and say, “Hey, this what’s threatening us right now, and we have got to deal with what we have with who we have and just make it work.” I think that’s something every security team, every technology team on, anything that’s kind of cutting edge… It’s useful for you because when you’re truly in areas that are a little uncharted, the operational playbook hasn’t been set, and so you’re going to have to be able to just do the very best with the information that you have. Sometimes you’re going to infer things from lack of information, and that’s what you have also, but you’ll always try to put together that timeline.
I think that nowadays with as many threats and new situations that we’re coming into now with both this shifting dynamics in world order, the massive level of automation and technology going into every level of society, including old industry, day-to-day tools and people, we’re in this… We’re today encountering a number of threats and complexity of threats that I can only compare it to the world of ecology and biology. There’s a lot of things that are proliferating in areas that you aren’t necessarily going to see. They’re going to interact in unexpected ways, and you’re going to have to figure out how to mitigate even without all the tools at your disposal and I think that kind of thinking is really, really slow in today’s world where we are faced with a lot of very significant and complex challenges that are new.

Brian Contos:
Julie, I’d like to double click on the different thinking there just for a moment, because as you mentioned, you came up with a bit of a different background, liberal arts. And I actually say it a little bit of a different background, but a lot of people don’t come into security with a degree in security or even computer science and with your liberal arts background, and I know you have a minor in econ from Stanford, so clearly you’re a great underachiever. But has that background in economics because I’ve met a lot of people that have actually had accounting backgrounds and are doing this as well. Has that helped you in your role, whether it’s Walmart or Box or maybe the companies that you’re a board advisor for? Has that played into how you think about the problems and how you address it from a cyber perspective?

Julie Tsai:
I think it has. I think that the sort of root or the DNA of my thinking on that has, or my education at least actually has helped a lot because in political science, you’re trained to think of whole systems and complex interactions, some of which can be quantified and some can’t. And I think this is true in technology and security too, because especially in security, you have a lot of psychology going on. You’ve got financial motivations, emotional motivations, and unexpected data streams coming in from places. Economics helps you to quantify certain things and try to be really, really disciplined about what and what you don’t know. And I think that also really helps. The thing, the sort of hidden piece to this is that both of my parents were immigrants on… My father especially in technical scholarships, he was a chemical engineer. And years later, I only found that a lot of people in the DevOps world were saying, “Hey, actually a lot of the folks who are excelling in the system administration and more nascent fields actually come out of accounting or chemical engineering backgrounds,” which is very, very precision oriented. It’s very, very precise.
And my mom did her second degree in computer science, and so that sort of sense of precision and rigor is part of what I carried into that picture. And I was going through a phase where I wanted to try to learn and achieve things outside of the classroom. And in some sense, I set my degree to the side, which I think was necessary at the time to focus and to really just say, “Okay, I’m going to commit to this new thing. I’m going to commit to really understand the technology.”
I was in system administration for 13 years, and then later in my career, as I was starting to get problems with bigger scope and teams and organizational dynamics and speaking, I started to see how all those strands would come together, and other cybersecurity leaders have told me that, and in the generation above or partway… no one goes into this planning to go into this. And I think a lot of this of us in this particular generation of professionals were making it up on the fly and then just trying to really commit to it. And someone told me that actually, in the early days of computing, before it got really professionalized in the 60s and 70s, it was like that too. You’d be drawing in people from all sorts of very diverse backgrounds and they’d be bringing all of that stuff into what was happening.

John Vecchi:
Yeah, it’s fascinating. And so let’s talk about today your attack surface. So how does all that… To confluence, all those things together to dealing with your attack surface and then mapping that to what then are your priorities to address that attack surface? What are the tools that meet those priorities? And then how do you deal with all of that with the team you have and the gaps? You know what I mean? Is that-

Julie Tsai:
Yeah, totally.

John Vecchi:
What does that all look like today for you? Because you mentioned the point earlier, which was a great one, just society today, there’s so many data points coming in. I mean, the world we live in is so different, and we’ll talk about the IoT and xIoT side of it, but just that attack surface is so big. How do you even grasp that?

Julie Tsai:
Yeah, I think looking at it from that whole system point of view, if you’re coming into a new environment, either bootstrapping or repairing something that was there. There’s a few things that I would definitely highlight as being important. One of them, of course, is visibility. How much can you really understand about your infrastructure, your systems, your applications, your people? And if you don’t have enough data at all, then you start asking those generative questions about what does an attack look like? What is my revenue driver or crown jewels of data? And start focusing around some of your visibility areas there and your pipelines to that. It’s important to look at it from that three-dimensional view. So visibility’s always really key. If you have too much information, again, start going to those core questions and say, okay, what out of all of this do I really, really care about?
And am I getting good data on the important question that I need? So I think visibility, understanding, monitoring, understanding your stack, and the teams that are working with you on that are completely, completely critical. I think the other piece, and this is more of the qualitative side of things, is really understand your business. Who are your customers? Who are your competitors? Who are the people that have a inordinate interest in what your company does for whatever reason? Have there been areas that have been exploited in certain ways or that they’re able to use in unexpected ways?
I think one of the interesting and important things about security is that both your users as well as attackers, will use your application and your stack in unexpected ways. When your customers and users are using it. What you have as a product insight. When your attackers are using it that way, what they’re actually do is exploiting a vulnerability in some overlooked, some overlooked backwater of the technology environment.
And so being aware, usually what ends up happening is that a smart attacker is going to attack the thing that you’re paying the least attention to. In fact, with one of my team members for my last role when we were interviewing, did a lot of red teamwork. And so that’s basically what I’m going to do. Skip a lot of the more esoteric stuff, but if you as a business are not paying attention to X, that’s where I’m going.
I think that that’s where you have to really think as a business, okay, it is a weakest link in the chain problem. So when I look at the whole thing in terms of our user profiles, our technology stack, where do we keep data? How do our employees operate? Do we have a lot of non-employee people? Are they over international borders? How well-trained are they? How well have we provisioned them with safe tools? You know, you start looking at all those things, look at where those weak links are, and those are going to be some of your vulnerability points and as well as where you might be the most liable, right?
So really understanding your stack in that area, I think is important along with the visibility. So I think those are the two major things that I’ve looked at coming into a new environment.

Brian Contos:
Sure. Well, when you think about IoT today, and we can even talk about OT on the [inaudible 00:18:51] side, industrial control systems and things like that. Any of these specialized hardware, software defined devices that you’re unable to install a traditional agent on a crowd strip or a McAfee or anything like that. And there’s a lot of them. And I honestly didn’t even know how many there were until I really started working in this space. And I would go to some retailers, and in my mind, I’m thinking, I don’t know, maybe they have a couple hundred security cameras and door locks and things like that. No, they had thousands and some cases, tens of thousands per store and things like deep fryers and microwaves and ovens and cleaning equipment. They’re all running Linux. They’re running Linux or Android or a flavor of BSD, they’re all running a flavor of Linux, essentially. Look at all these Linux servers. And even in fast food restaurants. So now that we’re working with CISOs and other leaders in these companies, and a lot of it’s pretty shocking.
My question for you is how did this sneak up on us? It’s not like we didn’t know there was things, it’s just, it’s like, oh my God, we have so many and they’re so vulnerable and they’re all Linux servers.

Julie Tsai:
I think it is got to be that thing of gravitating towards the bright, shiny, “Hey, we can make everything smart. Let’s make everything talk to each other. Isn’t that neat?” Oh so the lights go off. See? Oh, they’re communicating. That’s great.
And of course, as the security practitioners, you have to thank, as soon as that’s happening, okay, we have another attack surface. And this is another method of information and compromise being communicated. And I think that we are in that stage of things where there’s still a little bit of that euphoria and enthusiasm over how cool and advanced we can make all these tools. And you layer that on top of a lot of legacy technologies that are either bootstrapping or retrofitting to some sort of simplified one XOS or something. Yeah. Now you have all these basically computer devices and things that didn’t used to be computer devices.
Does my lamp here have Bluetooth? Does my pen need to have it? And so now you have basically a house. If you look 20 years ago, how many computing nodes did it have aside from people’s PCs, right? But if you look today right at all the pieces, the refrigerator, all the appliances, the smart home security systems, the things in the yard, whatever, how many computing devices are there now? And that’s all a footprint that needs to be protected. And part of the issue in the compromise is some of these areas that are less well known and less protected from a product standpoint are also completely ripe for attack.
When you look at different breach kill chains, how many of them leverage a vendor who’s working on, let’s say, some weird HVAC system over here and they had a password in and someone was able to hang out and lie in wait and then start pivoting into systems. So you know, look at it and it’s now become a very complex terrain of generations of technology that are layered together, a lot of which have not been maintained or are really hard to maintain. Who’s going to update the OS on your refrigerator if it’s not doing it by itself?
And so over time, all of that starts to accumulate more security vulnerabilities that the regular consumer and even the technical consumer’s not going to have the patience or the bandwidth to manage. So we have a really, really interesting issue here. Tool can be used for good or for bad. Every flexibility or capability also leads to a potential area of compromise or break. And our tool set for hardening and insulating things is still to mature. And I think this is where companies like yours become very relevant. And I think that eventually we also have to come to a sophisticated understanding of the right balance of how much to network certain things versus not. I think when you look at any kind of evolution of a technology or product, there seems to be this arc of there’s this sweet spot where functionality and design and simplicity are all in the right place. And then at some point things start to get overloaded and start to bloat.
And I think right now we’re sort of at the beginning stage, and it will take both the marketplace as well as consumers to become smarter and smarter about what to do and what not to do. And I see signs of hope. I look at the younger generation and they seem like they’re more mindful than we are certainly, or the older generation is about when to turn the technology off. And I think it’s because they’re so immersed in it all the time that it’s just to get a break at all is such a big deal. Whereas a lot of us didn’t necessarily grow up that way.

John Vecchi:
I mean, you made a point, which is great, Julie that you have to pay attention to what is connecting to your network. And a lot of these devices, many of them maybe perhaps don’t need to be connected to your network, but many are. So which ties also to your point that attackers are going to focus on that thing that you’re paying the least amount of attention about or that you’re just not paying attention to at all. And so here you’re faced with the devices that are connected, and as Brian said, they’re network connected. They’re basic Linux servers running a bunch of flavors you can’t put an agent on them.
Traditional security tools interestingly, don’t particularly do a great job with them either. And then you look at the state of these devices, so half of them, 50% of them are deployed and operating with default passwords. There’s Loads and loads of vulnerabilities, many of them critical… Many, many of these devices ship with vulnerabilities just for you. It’s a nice gift, right? They just shipped with them.
They could have expired certificates, they’ve got every port and protocol open. I mean, they come shipped, you plug them in. It’s like Bluetooth, Wifi, telnet, SSH, HETPS, everything. So when you think about that, how do you think through that and then say, how do I prioritize this? What do you do about that? The state of these devices, because practically almost anyone you pick has these issues. And how does that then tie into how do you prioritize that and how would someone prioritize that with the tools we have today, right?

Julie Tsai:
Yeah. I would start… Any good security administrator when they get something to secure or harden, takes away the unnecessary things. And I think this is where this is the tension or the balance with people who are creating the features or creating the product. It’s like, okay, do we need this? It creates performance and security issues if we’re adding something we don’t need.
And I think that consumers also, I don’t want to recommend… Or a world that I think which is unsustainable, whereas your average household person has to learn how to regularly update and secure 50 different devices in their home of different types and flavors and is just impossible. Anybody who does this stuff professionally knows that even if you had to do that for your job, that’s not, it’s going to get old pretty fast. And so I think do think that the key for the home or personal consumer is to simplify what they connected network and what they don’t. And then also expect certain things, look at things like where what’s under warranty are the operating systems and the computing platforms, they’re under warranty as well. In an ideal world, we’re all able to maintain all of our stuff from beginning to end, but I don’t think that that’s necessarily realistic. So I think that’s that piece of it.
I think on the professional and industry and commercial side, what it requires is a better coordination maturing of the tools that are there to help synchronize consistency and secured configurations. I think that a lot of the tools are there, but they need to be working together in the right way. And I think having that focus around what’s actually going to help the builds come out secure and come out in a maintainable way. All this stuff takes time.
It used to be, if we use Linux as an example, in the earlier years, there’s a lot of custom carve outs of how you’re going to install something or mess around with the manifests and making sure that things are exactly the way you want. And then eventually the package managers got better and better and easier to do things at scale. And I think this is where the security component needs to come in too and say like, okay, how are we making this this truly industrialized and smooth?

Brian Contos:
Yeah. So Julie, I just love the measured approach you take to these because I can see it how from your system administration and network background and coming up through security, now applying to today’s modern threats, it’s really a similar methodology and thought process that’s applied. It’s a very mature thought process and how you attack risk mitigation. I’m wondering through the years of doing this, are there any cool war stories or lessons learned that stories from the trenches, if you will, that you can share with us?

Julie Tsai:
Yeah, I can. Let me think of something that I can share. I think that definitely in the early years when I’d be working for a few companies, more than one company I was working for would some application core or a platform system that was written by a developer or DevOps person they had had on contract and the person had just disappeared off the… Just went off into the ether. And I think there’s a number of things that contribute to that. But one of the things that is very, very challenging in those circumstances is to maintain and secure those things where there hasn’t been documentation. It’s been very handcrafted and you got to just learn it by working with it and watching it. And one of the things that I think I’ve learned the hard way on that is if you don’t have a good understanding of the technology stack you’re working with, either because it’s just the documentation wasn’t there, or you’re new to it or whatever, start mitigating your risks in the beginning in terms of understanding where your levers are if you had to turn certain things off.
And what are your indicators also of normal? And I would say the sworn, any organization have a really, really good understanding of your baselines of normal activity, normal usage. Security and performance issues often show up in a very similar looking kind of package. And so you want to have a clear idea when things are going right and the things are going wrong, even when you don’t have all of the 100% tooling that you want. And I think that doing that hard work in the beginning, even before things have happened, as a just a proactive measure is what I’ve learned. You want to use those peaceful times to really understand what you’re working with and to prepare. And I think that’s definitely something that I learned the hard way.

John Vecchi:
And does that also, Julie, apply to the actual tools? I mean, in some cases, you’re going to have to learn a new tool to address something on the fly. You’re going to have to bootstrap, you’re going to learn it. So that approach applies to the actual tools and technologies that you’re using and deploying. Is that safe to say?

Julie Tsai:
Oh, absolutely. Absolutely. You know, want to get in there. You want to be doing a deployment from end to end. You want to troubleshoot an incident from end to end in terms of when something goes wrong. How did we know about it? How did we contain it or stamp it out right away? And if those things aren’t happening organically, you want to, as I say, war room them out a bit proactively just to get those muscles working in your head.
If you’re working with a new custom stack or even one that’s well known, that’s a recommended stack of stuff. You want to be in there and tracking all those sort of interdependencies and really knowing what’s fragile and not fragile, because those things are often not going to be advertised as well. Whether people don’t always start right away and say, “Hey, these are all the areas where you need to look out for.” Every once in a while you’ll have people on your team who are going to be oriented that way. You want to keep those people close. But a lot of times you’re going to have to discover that the hard way. So I think that really there is no substitute for understanding what you’re working with and what you’re getting into. And I think that also is part of the huge challenge of IoT, right? Because you have that great diversity of different layers of things in the stack, the old stuff and the new stuff, as well as sometimes probably a lot of custom integrations and that kind of thing can get very, very complex to manage.

Brian Contos:
Yeah, absolutely. You hit the nail in the head so correctly in the old and the new, because we go into organizations, especially on the OT side today, where there’s, there’s tech that was built in the 80s, not initially designed to be on a TCPIP network, which is retrofitted. So it’s like running serial over ethernet or something like that and it’s got some module that was tacked on. And it’s just a digital device that controls physics, it’s flow voltage, pressure, temperature, things of that nature. And they’re chugging along, but because they depreciate turbines, for example, over decades, they depreciate the computer that’s connecting to the turbine to run it over decades as well, which we don’t see that often. And I didn’t even think my iPhone would work if I didn’t upgrade it after probably a year.

Julie Tsai:
That’s right.

Brian Contos:
Let alone 20 years.
But Julie, this has been great, and I’m sure we could talk for hours and hours about, this is just fascinating. I love your perspectives. Perhaps you could give some closing thoughts or words of wisdom to our listeners out there that are working in the cybersecurity space.

Julie Tsai:
Yes, absolutely. First of all, stay with it. We need you. Also, I think that there’s a time for accumulating information and there’s a time to simplify. And to me, a lot of the art of security is knowing when to pull in more information and when to simplify, either the way you’re analyzing something, or how you’re maintaining something. A lot of the great security administrators I worked with early in my career, I saw that a big part of it was just asking those hard questions about what do we really need to go into production? If you don’t need it, get rid of it.

John Vecchi:
Yep. Well, that’s fantastic, and it’s such an insightful, interesting discussion today, Julie. It’s wonderful to have you. And so thanks so much for joining us. Julie Tsai and again, to my co-host Brian Contos.

Brian Contos:
Thank you so much, John Vecchi.

Julie Tsai:
Thanks for having me.

John Vecchi:
And remember everybody, the IoT Security podcast is brought to you by Phosphorus, the leading provider of proactive, full scope security for the extended internet of things. And until we meet again, I’m John Vecchi.

Brian Contos:
And I’m Brian Contos.

John Vecchi:
We’ll see you next time on Phosphorus Radio.