Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
What’s Really Plaguing SOC Teams, Lessons for Vendors & Leaders
163 Plays24 days ago
Erik Bloch, security operations expert and longtime infosec leader, joins the show to talk about the real problems plaguing SOCs and why the industry keeps missing the mark!
George K and George A talk to Erik about:
- The massive disconnect between what vendors think SOCs do vs. reality - spoiler alert: most founders have never worked in the trenches
- Why we're still fighting the same problems from 20 years ago - false positives aren't even 10% of the actual work, so why do we keep obsessing over them?
- The real burnout formula: high responsibility + low control + endless busywork that has nothing to do with catching bad guys
- Business alignment that actually works: Stop talking about theoretical $48 bazillion losses and start tying security outcomes to actual business growth
Whether you're a vendor or an executive leader, there's gold -- and data! -- in this episode. The "AI will solve everything" narrative is misleading, hear what actually works in security operations!
------------
🏳️🌈 It's PRIDE month! ALL PROFITS from all sales of the Pride collection during the month of June will be donated to scholarships for LGBTQ+ students in cybersecurity.
And this year we have generous vendor sponsors who will match our donation!
Shop swag, help students. You can shop the collection here: https://bkbtpodcast.shop/collections/pride-in-cyber
Recommended
Transcript
Challenges in SOC and Information Gaps
00:00:00
Speaker
A lot of the the vendors going into the space are founded by people who have not worked in the space. And they're getting their information from, you know, hey, I've talked to 100 CISOs or 500 CISOs or 1,000 CISOs, right, to gather the information.
00:00:13
Speaker
And as you guys are probably aware, CISOs, getting their information from turning around and talking to someone running their street operations team. And they don't most people, most teams don't have you know metrics around what they're doing or even track their programs or whatnot, because security is still coming to the Wild West and in most organizations.
00:00:30
Speaker
If you're a huge company, if you're a Google, Salesforce, Facebook, you probably have yourself together. But for the rest, the other 98 percent of the people, um We're not run very well. There's no standard you know playbooks or best practices of how to run a security operations team or a SOC.
00:00:47
Speaker
So everyone's just kind of going around that you're doing it the best they can. Hence why when the CISOs get asked, you know hey, what's your biggest problem with the SOC? A lot of it becomes like an echo chamber. you Hey, alert fatigue, you know burnout, all these things.
00:00:59
Speaker
But can they actually define why? Most of them cannot. they'll Again, they'll turn around, pivot to the people who are working for them to kind of troll the information. Again, a lot of times they don't have any data around this type of stuff.
00:01:11
Speaker
um So again, it's is anecdotal information kind of being bubbled up versus factual data. And that's what i've kind of tracked across my my last three roles at Atlassian, Sprinkler, and Salesforce is what are we actually doing in the SOC?
Introduction to 'Bare Knuckles and Brass Tacks'
00:01:30
Speaker
All right. It's your favorite cybersecurity podcast. It's Bare Knuckles and Brass Tacks, where we tackle the human side of the industry. I am George K. with the vendor side. And I'm George a A., Chief Information Security Officer.
00:01:43
Speaker
And today our guest is Eric Block, longtime cybersecurity veteran and student of the SOC. I just made that up. Why? Because the dude has been studying SecOps like in his spare time longer than anyone I know in terms of trying to attach metrics to what really
Eric Block on SOC Alignment and Burnout
00:02:02
Speaker
matters.
00:02:02
Speaker
And you'll hear it in the episode. We're really excited to talk to him. He and I have been going back and forth for years. This the first time I've spoken to him in person, but it was ah delight and dude dropped some serious knowledge.
00:02:15
Speaker
For myself, someone who runs SOX, who's running a SOX right now, and, you know, I build SOX. This was one of the most enjoyable experiences in our three years of of doing the show.
00:02:26
Speaker
Eric is a wealth of knowledge. He understands the perspective and he brings not only that background of those three decades, but he understands, I think, better than almost anyone else we ever had on here.
00:02:37
Speaker
what the SOC is going to look like, how the SOC needs to be thought about, and what the future of security operations is. And I think this was one of the first real conversations we had with someone in the know that could start telling us what the future of of cybersecurity looks like.
00:02:54
Speaker
Yeah, man, this one is for everyone. Dude, if you're on the vendor side and you are not taking copious notes here, you are not paying attention because this guy is telling you where you are wrong in your messaging and how you are wrong in approaching the problem.
00:03:08
Speaker
And if you're CISO, he's got some really great stuff at the end about how to start measuring processes and just aligning it to business
Vendors' Misunderstanding of SOC Needs
00:03:16
Speaker
budgets. I mean, it's just incredible.
00:03:19
Speaker
Eric b Block, welcome to the show. Hey, thanks for having me, George. Happy to be here. As we said before recording, you and have been circling each other for a few years, always found your posts insightful. I know everyone says that, but yours really are.
00:03:33
Speaker
um And you're here today. And since you're the practitioner, I get first crack. So we are very interested in the problem of the SOC and the problem of burnout.
00:03:43
Speaker
We've both talked about this. You have spent maybe more time studying SecOps and at a more granular level sort of talking about it in public than most anyone else I know on LinkedIn.
00:03:59
Speaker
We have vendors frothing at the mouth to talk about how AI is going to replace tier one analysts, ah agents going to do this, more visibility, blah, blah, blah. So given your understanding, my question is, what would you tell vendors that they're missing in the way that they're thinking about how a SOC operates?
00:04:21
Speaker
um Well, George, from my experience over the years dealing with security operations centers and talking with vendors, um probably the the largest disconnect, and this probably isn't something new for just security operations, is a lot of the the vendors going into the space are founded by people who have not worked in the space.
00:04:39
Speaker
And they're getting their information from, you know, hey, I've talked to 100 CISOs or 500 CISOs or 1,000 CISOs, right, to gather the information. And as you guys are probably aware, CISOs, they're getting their information from turning around and talking to someone running their street operations Right.
00:04:54
Speaker
And they don't most people, most teams don't have you know metrics around what they're doing or even track their programs or whatnot, because security is still coming to the Wild West and in most organizations. If you're a huge company, if you're a Google, Salesforce, Facebook, you probably have your stuff together.
00:05:08
Speaker
But for the rest, the other 98% the people, um We're not run very well. There's no standard you know playbooks or best practices of how to run a security operations team or a SOC.
00:05:20
Speaker
So everyone's just kind of going around that you're doing it the best they can. Hence why when the CISOs get asked, you know hey, what's your biggest problem with the SOC? A lot of it becomes like an echo chamber. so Hey, alert fatigue, you know burnout, all these things.
00:05:32
Speaker
But can they actually define why? Most of them cannot. they'll Again, they'll turn around, pivot to the people who are working for them to kind of troll the information. Again, a lot of times they don't have any data around this type of stuff.
00:05:44
Speaker
um So again, it's iss anecdotal information kind of being bubbled up versus factual data.
Data-Driven Approach in SOC Operations
00:05:50
Speaker
And that's what i've kind of tracked across my my last three roles at Atlassian, Sprinklr, and Salesforce, is what are we actually doing in the SOC? And it started way back in 2017 when i was at Salesforce and our SOC was missing the SLA 70% of the time. And they kept kind of spitballing things that they thought would would ah make it work better.
00:06:11
Speaker
And what what kind of triggered me is this, is is our SOC team said, hey, look, we could do this much better we had PCAPs captured across the entire company.
00:06:22
Speaker
And my head blew up. or I'm like, you make sense, right? And so I started doing a case study on the Security Operations Center. I mean, Salesforce had one. It was in Bangalore, India, about 200 people. as I started to compile the data, I'm like, PCAPs have nothing to do with actual problem going on here.
00:06:37
Speaker
it what I discovered kind of the full scope of what they were doing, like all the inputs and the outputs and some of these things I've posted on my LinkedIn. And as I started to compare the data to kind of what the vendors the problems the vendors are going after, they were completely misaligned.
00:06:53
Speaker
um Hence why, as I've published my data around some of this stuff, I'm trying to point out, like this is what an enterprise SOC does, like for real. These are the business outcomes they deliver.
00:07:04
Speaker
And at the same time, kind of pointing out the problems around you know how there's always going to be too much work versus the capacity to actually do the work. And it kind of leaves them in it in a tough space where they're having to kind of pick and choose the things you're going to have your stock look at.
00:07:18
Speaker
Often that falls upon the managers or the directors the people managing that function to decide, okay, what going look at? And kind of putting the rest of it off. At the same time, like that's the kind of perfect storm for burnout. mean, burnout from talking to, you know, therapists, psychiatrists, whatnot, is um a high level responsibility mixed with a low level of control.
00:07:40
Speaker
Right. The guys in the soccer are like, oh, my God, look at all the things. Otherwise, the company will be out of business. um And they're constantly floored by you know doing too much work, low fidelity alerts, just all the work they're doing, hence why they get burned out. But again, if people just focus on, hey, you know, they're burning out because they have too many false positives. Like that's not legit the story.
00:08:00
Speaker
As I've broken down again, kind of on my on some of my LinkedIn posts. you know false positives or maybe 10% of all the work they do versus even true positives or maybe 5% of all the work a SOC does.
00:08:13
Speaker
you know that Most SOCs, especially at larger companies, they're dealing more with people problems versus machine um kind of based alerts. like They're dealing with more people because if your as your scale as your company scales literally, your your you know your people grow.
00:08:27
Speaker
those complaints, those reports from people will grow literally. As you get bigger and more mature, usually you become more mature in your detection systems.
AI's Role and Human Insight in SOCs
00:08:35
Speaker
You're better at filtering, you're better at automating. um And these are the things that, again, once you get you know five to 10,000 people in your company upwards, you start dealing with more of these odd problems than just false positive, true positives.
00:08:49
Speaker
Man, like i I don't even know where to begin because like I'm like, well, do i but i want to ask you? Have you found a way to... I will securely implement open source LLMs in a production environment or do I want to go down in a route of like, hey, so, you know, talking about alert fatigue, which which I think, you know, is the reason why I don't trust all the AI hype from a security software perspective. It's like it's clearly going to replace us or suddenly going to, you know, outdate operators to the point that there's no point even hiring. I mean, everyone might as well become an architect.
00:09:21
Speaker
I think that's not the case. I think. every piece of security software out there now, you know from the class up at like the Wiz, Ground Strike level, all the way down to new OEMs and the emerging market, which I think is extremely promising. i do love our odds with the new companies that are coming out right now out of California, out of Texas, of New York City, even in Canada. We got some really exciting tech that's since infancy.
00:09:45
Speaker
But my my issue is if you have a false positive on these devices that are reportedly or supposedly AI enabled with all this accuracy, then you can't really trust anything.
00:09:57
Speaker
And then the other part of that thought is you can't really replace human beings because there still fundamentally need to be human beings that assess the data and make a determination. So where do you see the direction of trying to actually staff a SOC operation? And and you know I came into cyber where I think I was really lucky that I had an experience where it was an actual brick and mortar on-site SOC, had hundreds of people in it.
MSP vs. Enterprise SOCs
00:10:22
Speaker
We had over 100 clients worldwide. The company I worked for employed at the time over 30,000 people, a large consultancy with a big security practice. I felt that we were part of an organized unit. We were a machine. Everyone had their role.
00:10:35
Speaker
A former military commander of mine was actually the VP who helped build and design the thing. So it was very militarized in its operation. I feel like gone are the days of those kinds of SOCs. So what does the SOC of the future, in your opinion, look like?
00:10:50
Speaker
um Well, if you you can still use that analogy, actually, because I mean, I worked for NTT for a while, which is a MSP. They're one of the largest MSPs on the planet. And I worked out of one of their socks. And for for MSPs, their socks are kind of like those brick and mortar places still. Right.
00:11:06
Speaker
If you go to like CrowdStrike, mean, their FedRAMP sock, it's an actual physical building with big brick walls and, you know, all the alarms controls on it and dark screens, um because when you're serving customers, you need that.
00:11:18
Speaker
Enterprise SOCs are a completely different animal nowadays. Like if you go to, you know, what's what's the SOC at Google? It's not even a place, right? And so when I compare the two, I say, hey, for men's service providers, the SOC is still a noun, right? it's ah It's a person, place thing.
00:11:33
Speaker
For enterprise SOCs, that SOC is more of a verb. It's a function that you provide regardless where the people are at in the world. um So I kind of draw that distinction first.
Autonomous SOCs and AI Experimentation
00:11:44
Speaker
If you go towards the enterprise side, they are dealing with different issues than what managed service SOC is, right? A managed service, they are providing a uniform service with kind of ah a floor, right? They're providing the same service to all their customers regardless what it is.
00:12:00
Speaker
So they know the inputs and the outputs. They know what they have to deliver on what and what they're gonna expect having come their side. Versus an enterprise stock where, like, you know, when i was at Salesforce, we had over 800 different vendors in our security ecosystem due to hundreds of acquisitions, different environments, different countries we were in, you know, federal environments, you know, low, medium, high.
00:12:19
Speaker
They all have their different tech stacks. And we got stuck with having to kind of figure all that stuff out. So there's kind of the... the the tech challenges around all the machine-based detections coming in. But your SOC also becomes, you know for enterprise, kind of the the default dumping ground for all the things. right So when you hit the little phishing report email button in the email, it goes to your SOC to look at.
00:12:39
Speaker
you know If you, ah you know oh, I clicked on a phishing link, I need my VPN password reset, whatever it is, they'd send to the SOC. Anything where you send security at your company name going to be dumped on those guys.
00:12:50
Speaker
So they're dealing with different work streams than an enterprise SOC is or an MSP SOC is. Hence why I kind of refer that as ah as a verb. It's a capacity you're delivering.
00:13:01
Speaker
um At the same time, like if you think about you know where you can roll in you know AI to help these guys, um like That is still up for debate. when When I was at RSA, if you guys were there too, um I walked around and looked at some of these companies calling themselves AI SOC companies or autonomous SOC companies.
00:13:19
Speaker
And the first thing I noticed from talking just a couple of these guys is what they define as an autonomous SOC or an AI SOC varies wildly from vendor to vendor, right? That's uptime. Yeah.
00:13:30
Speaker
Yeah, and like ah one of my favorite companies is Tynes. I like those guys, Owen and TK. They were practitioners that came out of DocuSign, right? They have lived the pain. That's why I think their product does good job of this.
00:13:42
Speaker
But they are calling themselves an autonomous SOC platform now, right? By adding LLMs that help you do automation, right? And that's vastly different than another autonomous SOC company that's using LLMs to like triage all your events and fix all the things.
00:13:58
Speaker
um So, kind of how they're attacking this problem can come from many different angles, but they're calling themselves the same thing. And so, what I've found is, um again, it's kind of hard to classify all these companies right now but because they're taking so many different approaches.
00:14:14
Speaker
Some of them, I think, have valid approaches that can help, you know, limited scale. Some of them, I think, are just completely going off the wrong direction, you know, um which is a whole other issue we can double click on. Oh, yeah, that's not to knock our co-hosts, but marketing sometimes takes way too much of a lead.
00:14:31
Speaker
george Agreed. Agreed. Thank you for that segue. Thinking about, well, it's like, Eric, you're reading my notes. I was thinking about this time versus energy expenditure the role of
Technology vs. Process in Security Industry
00:14:45
Speaker
technology.
00:14:46
Speaker
I've said it many times, right? We have this holy trinity of people processing technology, but it would feel like the industry is like 90% indexed on the technology part. and One of my favorite posts, because for this interview, I went back through ah the feed was that flow diagram that you had managed to show that like, even if you determine something as a true positive, a lot of the tooling doesn't touch the workflow.
00:15:09
Speaker
Which is now most of the labor. Right. So I guess I've been thinking about UX. I've been thinking about times torque. A lot of these companies that are trying to attack the process part instead of just like more visibility and detections.
00:15:25
Speaker
So you're there in the Bay Area. um You've also sort of flirted with your own startup and I'm sure you do some advising. So I guess my question is, now that you have this understanding of the problem and this gap where the technology is being developed or even how it's being messaged, what are we missing in terms of innovation that is like truly transformative for these teams? Yeah.
00:15:50
Speaker
it's i don't think i I don't think it's necessarily detection because we have a lot of detection, right? But I feel like that's where like so much of the effort is being applied. like because those are The technology side, I mean, that's where the money is to be made, right? Yeah.
00:16:07
Speaker
You know, i my like I have some friends, Jeremiah Grossman and Robert Hansen, who have tried to start a fund. Right. And their premise was that the security industry is kind of turning the pharmaceutical industry.
00:16:20
Speaker
but There's more money in giving you Band-Aids and aspirin than there is actually solving the problem. And from trying to do my own startup idea, I kind of realized that those guys were on to something. If you think about the what the motivation is for the venture capitalists who are investing who are investing in this company, their primary motivation is not to solve the problem. Their primary motivation is to make money right for their for their LPs on the backside.
00:16:42
Speaker
Hence, they're going to invest in a company or a product, repeat founders, whoever has connections that are going to have a successful idea to make the money, regardless if that solves the problem or not. right And we see this all the time. We see companies you know so pop up. They get you know some business, you know whether it's above board or below board. you Because you can go back to cyber starts paying CISOs off, whatever it is.
00:17:03
Speaker
And they'll they'll rock it up to some evaluation. They get acquired. Are any the problems solved? No. but We're still where were 20 years ago. We're still fighting the same basic problems, especially operations space.
00:17:15
Speaker
But people have gotten rich, right? Right. So the the incentives and the motivations for a lot of the venture capitalists investing in this space are not aligned to the actual operators' ah problems, which is probably the biggest thing I've i've noticed over the years.
00:17:30
Speaker
um Hence why they keep focusing on technology, you know, rather are going to help the people get burned out all the Are we going help them get with the fire hose to the face? Nah. You know, are we going to look for process and metrics and actually look at what we're doing? Nah.
00:17:44
Speaker
Like, let's just keep throwing technology at the problem until it sticks one day. um So when when it comes down to some of these companies that are you know trying to attack the process problem, like I hope they
Standardization in SOC Processes
00:17:56
Speaker
get better at it, right? Because one of the things that you guys probably know is when you have a security operations team, there's an input, which is like ah you some event happening. There's an output and once you do the whole investigation process of your company's you know back in business the same way it was before this happened.
00:18:11
Speaker
How companies get from point A to point B is completely different from company to company. um The technologies they use, the processes they use, they're all different. um Hence why I think a lot of the problem is involved in the process side of it, not the technology.
00:18:25
Speaker
If we fix the process, I think the technology would have to follow to fall in line with that process. um Because what what I'm seeing, you know, again, from looking a lot of my data is the true positives that come out of all the machine-based detections are about 5%.
00:18:40
Speaker
You know, it's that's the same percentage as like indeterminate detections where it tells you something. and You're like, what does this mean? Right. the The true positive benign, the false positives, the duplicates, those are vast majority of what these detection engines are detecting.
00:18:57
Speaker
Right. What they're what they're spitting out and giving in front of the people. And we keep doing more and more of that. Yeah, yeah we throw AI at it. Now we have you know more detections or detections in other places. Or we have higher fidelity protection. or Or more easily written reports about all of the false detection.
00:19:13
Speaker
Yeah. And this hasn't changed for 20 years. Like if you read some of Anton Chewbacca's blogs, the ones that he wrote 20 years ago you know are just as applicable as they are today.
Critique of Security Industry's Profit Focus
00:19:22
Speaker
And sometimes he recycles them and just says, hey, look, I feel like I'm writing about 2005 all over again.
00:19:28
Speaker
Repost. Yeah, that's kind of where we're at. And again, the the problems aren't being solved. But at the same time, there's a lot of founders out here running around in their new Ferraris and whatnot, right? there There's all kinds of checks being written. There's people make getting money off of this.
00:19:42
Speaker
and it's But our problems aren't being solved. Like, you know, last role, had 41 burnout the day I got there. Like, we're not solving the problems.
00:19:54
Speaker
are Hey listeners, it's Pride Month and that means the annual fundraiser is back. In the month of June, all profits from all sales of Pride merch in the BKBT swag shop are donated to scholarships for students in cybersecurity, both undergrad and graduate studies.
00:20:15
Speaker
We have a whole bunch new swag. So check it out at bkbtpodcast.shop and check out the Pride and Cyber collection. Anything you buy there, we're donating profits from.
00:20:28
Speaker
And this year, we have corporate sponsors who will be matching our donation. So let's see how much we can raise. Thank you for your support and thank you for listening.
00:20:45
Speaker
your To your credit as well, I've harped on this problem my entire time in the industry when I realized what it was when I started getting brought along by the seniors to some of these big vendor trips, big vendor meetings, like the fancy things. and you see all the money they spend on this stuff. And you're like you know you go to restaurants, you look at the menu, there's like a thousand dollars an item on this thing. like why ah i i look at code.
00:21:09
Speaker
What am I doing here? So like, you know, um I think that is at the very core, the toxicity of our industry is that there are so many people who look at it as a means to, you know, without very much skill, without very much blood, sweat and tears put into it, make fast money, figure out a formula and repeat it endlessly.
00:21:32
Speaker
Right. And I don't know how we as practitioners push back against this because we need to pay our bills. We got to pay our mortgages, rent, et cetera. But at the same time, this constant drive and race to the bottom while still constantly worrying about EBITDA over performance, over personnel, over stakeholders every single day.
00:21:54
Speaker
And you see these layoff stories. we We're in a another round of it right now. Microsoft just had a big one. CrowdStrike just had a big one. you know It's all for the sake of profitability. and and And I can't for the life of me, you know, when you you have a record or a near record setting year, don't know if it's set the record or not, I'd have to go back and read, but you have over a billion dollars in profit and you're still doing a workforce reduction.
00:22:21
Speaker
How do practitioners have any sort of fate in the future of this industry, Eric? Like, how do you see this? Is it only gonna get worse before it gets better? Or how does this go, man?
Regulation as a Driver for Security Innovation
00:22:32
Speaker
um You know, someone was telling me the other day, you know, America innovates, China imitates and Europe regulates.
00:22:42
Speaker
And that kind of resonated with me because a lot of the things that like we are having to do internally at some of my companies are due to European regulations, right? We invent a lot of stuff and then the Europeans kind of you know lay down the law of, yeah hey, this is how I want you to protect our data or whatnot.
00:22:58
Speaker
And so while um you know while you know compliance with equal security, I think compliance does help drive a lot of these initiatives and get them off the ground. Because I mean, think about if you are a company, right? And you want to sign a contract with somebody, and you're like, hey, want to see your SOC 2, I want see your ISO 27000K, want to see your PCI, I want to see whoever search, right?
00:23:18
Speaker
And you didn't have them, you wouldn't be in business. You wouldn't be taking on any customers. So from that aspect, the fact that they're forcing you to have these regulations means you have some baseline of security, right?
00:23:31
Speaker
They're forcing you to do something, even if it's just minimally, um which helps get some these initiatives off the ground. At the same time, if we didn't have that, I think a lot of companies would just skip out on security completely unless there were some requirements. like Not all of them, but a lot would.
00:23:44
Speaker
And so like compliance is helping drive this. Regulation is helping drive some of this. um We do have companies that actually do take security seriously. Like the Googles of the world, that are currently they have like massive you know thousands of people on the security trying to do the right thing um because it it it instills trust. if People want to use their products. So there are some motivating factors around that.
00:24:04
Speaker
but you know how do we fix it at at a macro scale? Like, i i I wish I had the idea if if I knew how to do that, I'd probably be be rich too. So hence why I've kind of focused on the micro scale of, you know, what can I fix? Well, I'm a guru in security operations. I'm a guru in the SOC.
00:24:22
Speaker
um I can point the problems out. I can suggest solutions. I can work with the vendors who want it or claim they want to solve the problem and tell them this is how you solve the problem.
Burnout in SOCs and Overlooked Challenges
00:24:32
Speaker
Some listen, some don't.
00:24:34
Speaker
um So, you know, You do what you can do in your little world and hope the people around you are also doing what they're doing to kind of change you know their little parts of the world. um But that's that's all I've been able to, at least from my angle. you kind of view internalize the lead by example. That's your approach. Yeah.
00:24:52
Speaker
And I've talked to, I mean, probably, i mean, a hundred startups in the last two years, all kind of jumping into the space and the security operations and and screw are ah the SOC space as well.
00:25:03
Speaker
And again, some of them have listened to me. Some of them have not. um you know Some of them are chasing the shiny object. Hey, we're going to catch the APT. you know We're going to catch the Chinese or the Iranians when they hack in. I'm like, cool. How often does that happen?
00:25:17
Speaker
like Once in a career for some people? you know Once a decade? i mean I think I've dealt with APTs four times in the last 30 years. i mean It doesn't happen every day. The fact that you can you catch them in triage with your cool AI thing, cool.
00:25:31
Speaker
What about the other 99.999% of the work they're doing in the SOC? What about the you know the phishing emails, the the false DOP alerts, you know that that all the tickets people are opening, requesting know random this and that.
00:25:43
Speaker
um How about all that garbage work? that those That's the thing that actually burns people out, the repetitive grind. um you know i I tell this story of when I was at Atlassian and we had the Chinese APTs attack us over there.
00:25:58
Speaker
My team worked for almost three weeks straight, 24-7. They were passing the baton between the three regional teams. And ah yeah my CISO, was really concerned about burnout. and And I i kind of kind of chuckled at first.
00:26:12
Speaker
And he was serious. He's like, no you guys are going to give them some timeouts. And I'm like, I'm not going to touch them. Like... These guys, like, why are they doing this? Why are they part of the SOC on the IR team, right?
00:26:24
Speaker
They want to catch bad guys. They want to do the right thing. They want to, you know, make a difference. And here they are. Like, some these guys in their you know late 20s, early 30s, this is their first chance dealing with a nation state. The fact nation is even attacking us, right?
00:26:37
Speaker
These guys worked for almost three weeks straight, passed the twenty four seven until they remediated the whole thing, and they were stoked about it. They were happy. This is why they wanted to work in the industry, right?
00:26:47
Speaker
Right. And this is the first time it ever happened to them. And some of these guys have been working there for five, six, seven, eight, 10 years. First time they've ever got to do something like this. The next day it was back to, this is a false phishing email. I'm going to close this new PM ticket, right?
00:27:03
Speaker
went back to that. That is the crap that burns them out and just crushes them. to throw a Chinese APT. Like these guys freaked out. I'm like, dude, we should be popping the champagne. We've made the big leagues.
00:27:14
Speaker
Like Atlassian, we're in the same category as Microsoft and Google, you know governments. They're coming after us because we're we're that important in the world. That means we are up there in that level, right? The team that was fighting them off,
00:27:27
Speaker
Like we are at that level where where they're coming after us and we're going punch back just as hard. like This guy's got excited. like They were stoked to be doing this. Yeah. You really, you really highlight there, right? Like one of the definitions of burnout is the work without purpose.
Establishing Metrics and Processes in SOCs
00:27:42
Speaker
Right. And you point out it's a it's a mission and it's like going through all the training and you you want to see action and an APT is action, the likes of which you'll you might never see again.
00:27:57
Speaker
um But ah yeah, so I want to. go back to you said in your time at Salesforce is when you had this light bulb moment of like, Oh, I should just try to like measure something, yeah right? There's no baseline measurement.
00:28:11
Speaker
What would be your advice to other CISOs who are listening? And I take your point that some haven't been in the sock or they just have been away from it for so long that they're relying on the team leads for the reports.
00:28:24
Speaker
So what would be your advice to them on like, okay, well, if you want to start getting your hands around this, like, start with XYZ? Like where where would where would they start? um I would start by making sure that your SOC has a process, even if it's you know just super simple basic process that they operate within.
00:28:44
Speaker
And this can be something as simple as like, hey, you know every alert gets thrown into Jira Ticket ServiceNow ticket someplace, right? And your SOC operators will work out of a ticketing queue, they'll pop a ticket out, they'll look at it, they have some SLA you set, you know half hour, hour, hour, whatever it is.
00:29:01
Speaker
that they have to triage this alert within that amount time. They close the ticket, go to the next one. Having that simple process will allow the CISO or SOC manager or or the SecOps director, whoever it is, to start measuring what their team is actually doing.
00:29:17
Speaker
And this goes to kind of doing some basic capacity planning, which know engineering, everybody else in the world has this down pat. We're horrible about it in security. They're saying, OK, how much capacity is my SOC team actually have?
00:29:29
Speaker
Right. So if you have, you know, and I i have some some slides I give here. If you have one person. Right. And say they have an SLA of one hour to work a ticket. in the SOC, okay? They're working an eight hour day.
00:29:41
Speaker
They can work on average eight tickets a day, right? This isn't rocket science. If you have 10 people in your SOC, you can work 80 tickets per day, okay? Again, not rocket science. If you're tracking their work, you know, like in JIRA, it's super simple to form a report saying, hey, look, show me a 30 day trend of how many tickets these guys are working.
00:29:59
Speaker
It should hopefully average out to eight per day per analyst. They might be a little bit higher, maybe a little lower. Sometimes you triage things quicker. Sometimes things take more time. But on average, across the span of 30 days, you should be right about that neutral point where they're doing about eight tickets a day.
00:30:14
Speaker
If you're way above that, you're giving them too much work. If you're way below that, you can lower the SLA, right? You you start to fine tune their work. Once you go beyond that and realize what capacity they have, then you start looking at what work are they actually doing?
00:30:28
Speaker
Like, what what's the work you're giving them, right? And often a lot of the times you can figure that out by what are the outcomes when these tickets are closed on the other side? Like, are any of them true positives? Are they all false positives? Are they all duplicates?
00:30:43
Speaker
Are they all just, you know, indeterminates? They can't figure out what to do with them? um And going back and looking at the work you're giving them to make sure you're giving them the highest impact work. I love that. I love that. yeah You want to prevent the burnout and you want them working on things that actually make an impact.
00:31:00
Speaker
But until you know what they're doing today, you'll you'll never figure that out. And that was kind of the first thing at Salesforce when I said they were missing this place 70% of the time. i was always started asking myself why.
00:31:11
Speaker
And that's when I realized, you know, they spent, I mean, our SOC there, because we had over 800 vendors, they spent over 50% of their time pivoting between tools. but you know that was That's the classic swivel chair problem, right?
00:31:24
Speaker
Yeah, 800 vendors, all these acquisitions, all these countries. they like, what tool do have to get to? yeah Where's the console at? How do i log in? you know That's half of their time. Until you when you realize that, you can't address it. but you know It's funny, though, because i and i have to cause you've made a really good...
Aligning SOC Metrics with Business Outcomes
00:31:43
Speaker
I guess delineation between two. Working like an enterprise SOC versus an MSP SOC are two different things. And so when I think of like life in the MSP SOC, and I had to deal with ArcSight and that stupid fucking alert waterfall thing, like any given shift, you'd be getting 25 to 50 alerts on your PryOne channel alone, right? But then each individual client might have their own PryOne channel, there might be their own instance of an ArcSight scene with us.
00:32:09
Speaker
So you're like, you have to scroll through basically like at least a dozen windows. And those are just your Pry one, Pry two. Pry three, Pry four doesn't even get looked at. Maybe at night if you're bored for some miracle reason.
00:32:22
Speaker
But then you know you're going through these alerts and you never win. And then all these alerts just go unchecked. And that was the thing that boggled my mind because we we sold these you know massive contracts.
00:32:33
Speaker
And it's like, hey, you know like we're only getting visibility on like a fraction of these. does Does no one actually authenticate? It's amazing that you realize or or you're able to articulate the problem with metrics and accountability because I think we, I'd love to get your opinion on this. I think the way that business values security and the KPIs that board level personnel assigned to us are incorrect.
00:32:58
Speaker
I think they're not an accurate representation of what our core functionality should be if you are working in a mainline SOC position. How do you feel about that? um That's true. I'm actually, i was working on my my ah deck that i have to present to the audit committee next week. I was just doing that room before I got this call. I'm still working on it because have to do the dry run tomorrow with the CFO.
00:33:21
Speaker
um But one of the things I was noticing was um the the prior decks to this um that you know my producer had given were the risks and the metrics they were measuring that they were giving to the board,
00:33:36
Speaker
didn't have anything to do with the actual business we were in. Like, oh, the biggest exist threat to us is generative You know, the the largest risk is we're at risk of losing, you know, 48 bazillion dollars if our IP lost, right?
00:33:50
Speaker
And this is why the the people on the board kind of rolled their eyes. They're like, okay, you got your two minutes. Get out of here, right? And I've seen this happen at Atlassian and other places, Spring Cloud and Burnett, where the CISOs are making their best guess to make a story to ask what they need.
00:34:03
Speaker
Again, rather than what I'm doing now is I'm tying the security outcomes to the outcomes of the business, right? The security or the business wants to continue to grow and scale on board amount of customers this year.
00:34:14
Speaker
i need to make sure its security can handle that, right? i want to grow with the business. want to make sure we can do all the all the the questionnaires coming from the customer, if you have all the compliance sorts in place so we can sign the business to make sure that you know from a a base level that all of our customer environments are covered by a security operations team that we're shipping secure code.
00:34:32
Speaker
And I wanna align all that to the business outcomes, right? And so rather than saying, hey, look, you know, we have 100 environments and i only have visibility into you know six of them. i need a bazillion dollars to do that.
00:34:43
Speaker
You know I'm like, no, we have, you know, X, know maybe four or five customer environments where our data lives. I need X resources to cover those. And if we're going to expand this year to add on two more environments because of our growth, I need to align my team so we can cover that as well.
00:34:59
Speaker
And so I'm tying my ass and my data and my metrics to what the company is telling me they're going to do this year. So if they're going to you double our business, OK, I need to be able to to help enable that to happen.
00:35:11
Speaker
whatever that I'm not going to double my team, but obviously i need to be able to to deal with that doubling the capacity of customers, whether that means more in cloud environments, more accounts. I need to bring in more tooling, more automation, you have some more you know certifications if we're going to multi-cloud, making sure we we continue to do that. The whole idea is you don't want security to become a blocker for the business.
00:35:30
Speaker
You want to security become an enabler to the business, right? That's why it's more of why we're here is to un enable the business to continue onboarding customer making making money.
SOC Alignment with Business Objectives
00:35:37
Speaker
If you align yourself to that, it becomes a much easier story because rather than saying, I need, you know, a bunch of resources because this or that might happen. We might lose our, you know, the the world will come to an end if we get hacked or we lose our internal property.
00:35:49
Speaker
You can say, no, look, if you guys really plan to onboard a thousand customers this year, right? In order for me to support that, I will start on the sales cycle so some of these customers don't leave because you know we don't have the right certifications.
00:36:02
Speaker
I need X and Y and Z to support that mission. And all of a sudden then, that conversation, it's not about, you know Eric, why you asking for a million dollars? The question becomes, are we confident we're really going to bring on a thousand customers?
00:36:15
Speaker
Because if we are, then he needs to have this in order for a lot to happen. And this is the same way engineering does it. you It's the same way, ah you know, sales and marketing does it. If we want to achieve X, I need Y to get there.
00:36:27
Speaker
Security is, you know, not everybody, but, you know, a lot of places are like, they don't like I need X, Y and Z because here's some theoretical scenario where we might, you know, the company might go to business because we might lose our own property. The customer data will be compromised, whatever it is.
00:36:43
Speaker
And that's not tied to the business goals and objectives. Those are theoretical you know scenarios that might happen that could put the company at risk. But what's really at risk is the company's revenue for that year. That's what the board wants to hear about.
00:36:57
Speaker
Right? It's something going to stop us from making our numbers this year. i I really like that you have mentioned capacity planning on the the software engineering side and sales and marketing. I often think that cyber would benefit from bringing in thoughts from other disciplines, but we're kind of like...
00:37:15
Speaker
really like believe in our own specialness. And it's like, no, like this is our thing. And we're, you know, I don't know. And maybe in your budget, you can set some money aside for pizza for the next APT or something.
00:37:29
Speaker
Yeah. um It has been interesting. Like it, you to me, it just kind of seemed obvious. Like i'm like, Oh, how are you guys getting money? Oh, we're doing capacity planning and a spreadsheet over the next four quarters. And we're balancing resources to the, what the company wants to do.
00:37:44
Speaker
you know, and I go to other teams, like the SRA teams. How do you guys measure what you're doing? Oh, we have all these metrics that measure, you know, what we're doing and how we're we're putting things through, we're enabling the business to continue to grow.
00:37:55
Speaker
I'm looking at us like, what do we, what do we show? Like, Nothing. know, how many people are there? Theoretical risks. Screenshots of scary headlines. Yeah. Yeah. Or a heat map where they drop dots someplace. this is what we think the risk is for this thing, right? Spider graphs. All about the spider graphs. Spider graphs that aren't calibrated correctly.
00:38:15
Speaker
Yeah, none of it's tied to actual hard data around, you know what do I need to achieve the the company's mission or goals? um And none of it's even tied back to, um again, what the company's trying to achieve. They're tied, you know, a lot of times to kind theoretical scenarios that may or may not happen.
00:38:32
Speaker
Yeah, that's awesome. um Yeah, well, that is ah that's a perfect place to come to an end. Eric, thank you so much for your time and attention. this I mean, ah dig the posts. I dig the research. I dig the the book that I'm sure is going to come out one day.
00:38:48
Speaker
But I'm glad that we were able to get you on and like talk this out, because I think there's just this huge divide between the perception, especially on the marketing side of, you know, tier one, tier two, sock this visibility that. And then we just need kind of like the cold water reality. Yeah, I got tell you to you, too, for my own selfish reasons, man, I hope we get to meet someday whatever and hang out because, uh.
00:39:11
Speaker
you know i I run a SOC, I build SOCs, I do both on the consulting side for my own primary employer and and I could really, really use kind of the insight and the the visionary kind of a perspective that you provide, man. because i Every day I try to figure out can we do this better?
00:39:30
Speaker
And you, in our three years doing this show, are one of the most accurate guys at just like finger on the pulse. You understand the problem as best as anyone out there right now, man. So it's really appreciated that you shared your insights with us tonight.
00:39:45
Speaker
um Thanks. I appreciate the kind words and I'm happy to to share it. um I've actually been on, well, Not that I want to do that, I'm kind of introvert, right? But the the company I'm at now is a security vendor and they they ah you know they sponsor events.
00:40:01
Speaker
And as a part of that, they get speaking spots. So they've been putting me in these speaking spots. you know i'm I'm sure much to the origin of our our marketing department, I'm not talking about what our product does or what we do. I'm talking about security operations, right?
00:40:13
Speaker
um And I gave a talk at the ISC squared a couple weeks ago. I gave a talk at Cloud Security Alliance here in the Bay Area. um And what I've seen, kind of to your point, is people are hungry for this.
00:40:24
Speaker
Like, you know, security operations is one of kind of the kind the fundamental pillars of security that nobody seems to know how to do very well. And, you know, we say it in the military and maybe it's military as well, that in the lack of a plan, even a bit, even a bad plan is a plan. Right.
00:40:40
Speaker
And so I might not be perfect you know or right on. I might not fit every place's organization, but it's something it's something to start from. It's something that I found works for me. um And yeah, if you ever want to you know compare notes or I can show you my deck, whatever, and you know, feel free to to ping me or or if you're ever in the Bay Area.
00:40:58
Speaker
Thank you, sir. Perfect is the enemy of good. And we appreciate your kindness. All right. Well, we'll catch you soon. Awesome. Thanks, guys.
00:41:08
Speaker
If you like this conversation, share it with friends and subscribe wherever you get your podcasts for a weekly ballistic payload of snark, insights, and laughs. New episodes of Bare Knuckles and Brass Tacks drop every Monday.
00:41:21
Speaker
If you're already subscribed, thank you for your support and your swagger. Please consider leaving a rating or a review. It helps others find the show. We'll catch you next week, but until then, stay real.