Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Episode 9 - jack sanford
165 Plays1 month ago
riptide and jack discuss how his audit competition/bounty platform Sherlock stacks up against the competition, why bounty hunters should focus their time there, how the platform has evolved over the years, addressing complaints with competitions to include: self-judging, spam, and insider bad behavior. Also how to create incentive based systems to obtain desired outcomes, what makes a good audit, and much, much more ...
Recommended
Transcript
Introduction to Bounty Hunters and Podcast Focus
00:00:07
Speaker
Oh yeah, ready to go. Welcome to Bounty Hunters. I'm here with Jack from Sherlock. How you doing? so That is an awesome intro. but That is an awesome intro. pumps us up, man. We are pumped to talk about bugs.
00:00:23
Speaker
I'm ready. Let's do it. Hey, man. So I got to say, for whatever reason, and we've never met before, I had you blocked on X, and I couldn't remember why. And then I unblocked you. And then I looked at some tweets, and I was like, oh, shit.
00:00:37
Speaker
I think he just tweets a lot of shit. And maybe I just blocked you from back in the day. i had no idea. Or he or likes to to stir up shit. I don't know. you've You've had me blocked for years, and I don't know what I said, but there was something, i want to say maybe 2021 is when I first noticed it.
Jack's Background and Career Transition
00:00:57
Speaker
But yeah, I say a lot of crazy things on Twitter, so it's not surprising. I don't have that many people who blocked me, though, so I did remember that you had blocked me. but's that's Did you get a notification? How did you know? 2021.
00:01:10
Speaker
Fuck, DeFi, summertime. Yeah. Who the fuck knows? I wish I knew. i think we, I'm not sure if we interacted directly on Twitter or maybe ah you were just saying, like, you just disagreed with something that I was saying. But don't yeah, I remember looking you up and being like, oh, Riptide blocks me. Well, listen, water under the bridge. I'm going to reset our our relationship here.
00:01:35
Speaker
Yeah, well, you know, maybe I'll say some things on this podcast too, and maybe you'll block me again after. Maybe it turns out you're an asshole. I don't know. Maybe I am too. We'll see. Anyway, man, thanks for taking the time coming on ah This is kind of a little different podcast because don't know if you listen to any of my other ones, but obviously I get Bounty Hunters on here. We talk about finding bugs.
00:01:56
Speaker
ah you're i don't Maybe you run the Sherlock account. I don't know. But you guys reached out said, hey, this be a good idea. And I thought about it, I thought it actually would be a really good idea because you run Sherlock and that is one of the top kind of contests and bounty spots that hunters go to.
00:02:18
Speaker
And so to get your perspective on a lot of things, I think would be interesting to a lot of bug hunters that listen to this and especially some of the comments we got in the discord and thanks for joining. But a lot of people, I guess, are are very interested in some of the back room I don't want to dealings in contests, but there's there's some stuff that's going on that um if you could help clarify some some things and put it out in the open, I think a lot of people would appreciate it.
00:02:45
Speaker
Yeah, no, sounds great. Would love to. um Yeah, honored to be on the podcast. I don't spend much time as a bug bounty hunter myself. I wish I had more time to do it, actually, and compete in contests.
00:02:58
Speaker
But yeah, that's one of the the downsides of the job is you just don't have much time to to do that kind of thing outside of the the day job. um But yeah, I've listened to, so I was just listening to NNEZ.
00:03:14
Speaker
Guy sounds like a legend, you know, just grinding out of out of Bangkok. Do you tell people where you're based, by the way? I think everyone knows, but if they ask, usually i try to keep it about the guest, but yeah, I'm just chilling low key in Europe, man.
00:03:30
Speaker
In Europe, nice. What about you? I was actually in Europe until about two months ago. I was in near Amsterdam in the Netherlands, and now I'm in New York. Okay.
00:03:42
Speaker
Good stuff. And real quick, what's
Journey into Crypto and DeFi
00:03:44
Speaker
what's your background? man I know nothing about you. You started this. Why? Why'd you start Sherlock? Yeah. Yeah, of course. um My background is all over the place. I was like from childhood training to be a professional tennis player. So that was like on the docket for me for...
00:04:01
Speaker
10 or 15 years, um played out at a really high level ah in university and college at Berkeley, ah but I had a lot of injuries. And so I kind of realized like, oh, this isn't gonna work out for me.
00:04:14
Speaker
So I focused on um more of the business and finance side of things. I really wanted to be a hedge fund manager. I thought that was a cool career path. So I did that for a few years. and made it all the way to being in the top group at Citadel, um you know, and like a really great team, crushing it on the hedge fund side of things.
00:04:33
Speaker
Is this in Chicago? This was in San Francisco. Yeah. one of the One of the best financial institutions groups in the global equity section of Citadel sits in San Francisco. ah One of the guys, the partner at Citadel.
00:04:47
Speaker
And so I was working for him and it was it was cool, but it was so boring. I was just like analyzing these regional banks that don't do anything. And that was when I got red pilled on the financial system and like how broken it is, how un-innovative it is.
00:05:02
Speaker
um So I left that to get into programming and ah like the startup world. So i was a self taught programmer for like two, two and a half years doing all kinds of stuff, building iPhone apps and Swift, building websites and JavaScript, a bunch of Python projects and then found crypto and DeFi. And I was like, oh, my God, this is so clearly where I want to spend.
00:05:25
Speaker
Like my career basically is creating a better financial system because i know how broken and kind of corrupt and just all these problems with the old financial system. And so that's what got me into DeFi. At that point, I knew programming enough to pick up solidity um and just became obsessed with it.
00:05:44
Speaker
And so the problem that me and my co-founder got really excited about was, hey, if you want crypto and DeFi to be successful, you can't have like your mother putting her life savings into DeFi and then it yeah getting hacked, you know?
00:05:59
Speaker
And so that's where we come to the security
Sherlock's Mission in Crypto Security
00:06:01
Speaker
side of things. And we basically said, look, we don't like a lot of the security initiatives and teams going after this problem in like 2020. um So we felt we should step in and try to kind of, you know, hold up that pillar for the crypto ecosystem and and try to make things safer. So that was my initial kind of introduction into the crypto security side of things.
00:06:26
Speaker
Very cool. From fig to crypto. Everyone's got their transition. That's it's interesting, man. Yeah, once you see the back rooms of banking, you just realize how how retarded it is and how nothing works. Yeah, you get so many guys coming over.
00:06:42
Speaker
That's cool. So you said, I'm going to build. And so at this point, what did you know you were going to focus on contests, like build some sort of contest platform? I think Code Arena was probably probably the first one that I ever heard of doing contests?
00:06:58
Speaker
Was that for you too? Yeah. Yeah. That's a great question. Coderino was definitely the first, I guess. So this is kind of a longer answer and I would love to hear, you know, what was your, I don't know anything about you either.
00:07:11
Speaker
What was your kind of background before you got into the space? Oh, I used to be, I was always doing stuff, um, on the technical side and uh, I did some military stuff and then went over to wall street and was, uh,
00:07:27
Speaker
was a banker over in Miami as well. So covered a few different groups. Fig was one of them briefly, but mostly transportation. ah And i kind of, I saw the banking side inside out and I was dabbling in Bitcoin since 09. I used to build, ah build computers, built a miner, blah, blah, blah.
00:07:49
Speaker
And didn't jump into ETH until much later, but then I saw the ecosystem. ah But then No, I didn't really go full bore until until about COVID. I got laid off, whole team got laid off and I just went hardcore crypto and I saw the bounties on ImmuneFi and i said, you know, I want to do something where I can control what's going to happen to me instead of just relying on the market.
00:08:15
Speaker
And so I just, you know, parlayed my my JavaScript PHP knowledge into Solidity and just put head to the grindstone until I started finding bounties. That's awesome. That's awesome. Yeah. The level of self-sufficiency has a as a bounty hunter is quite high, it seems like.
00:08:34
Speaker
You have to. I mean, there's ups and downs. I think it's better now with GPT and stuff because you can be deep in a weird code base and not have a clue and it could help explain it to you. Whereas that would take, you know, maybe a week of research on your own to learn however some complicated system works. so But yeah, it's very... um you know it's just you out there because you can't share your secrets or your tips yeah yeah um well that's i'll answer your question but i also would love to hear because there's such like a risk tolerance spectrum between doing traditional audits doing contests and then doing bug bounties and if you're a bug bounty hunter you're kind of at the extreme end of that of that risk spectrum in some ways so would love to kind of hear how you think about that um
00:09:24
Speaker
But i'll I'll answer your question first about getting into audit contests and and Sherlock's early days.
Challenges with Traditional Audit Firms
00:09:32
Speaker
So we started out, um I don't know if Code Arena was around when we first started.
00:09:38
Speaker
um What year did you kick off? We kicked off early 2021. So they might've been just kicking off as well, but we didn't do audit contests until much later.
00:09:50
Speaker
So they were definitely first to it. We started out doing smart contract insurance and we kind of figured, okay, if we only kind of underwrite the very best audit firms, then this is gonna go really well. We're gonna have almost no payouts and whatnot.
00:10:09
Speaker
And then we quickly realized even the best audit firms back in 2021 were missing bugs left and right. I mean, we we built a protocol, ah two protocols actually, for the smart contract insurance platform.
00:10:22
Speaker
And so we tried a bunch of different audits. We tried ah you know early code arena, we tried some traditional audit firms like top blue chip ones. And especially with the traditional audit firms, we had terrible experiences. Like everything you could imagine going wrong kind of went wrong for us. Like we paid a bunch of money, we waited many, many months.
00:10:42
Speaker
We had like this update audit we wanted to do that was a significant update to our protocol. And the traditional audit firm, who I won't name, ah they took off our original auditors, ah like two guys from that audit last minute.
00:10:58
Speaker
And they gave us a guy with six months of blockchain experience and a guy with three months of blockchain experience. And we were like, hmm. That's kind of weird. ah i guess these guys know like I guess these guys know more than we do.
00:11:10
Speaker
ah So you know we'll continue trusting them. We'll have them do this audit. We already paid them a long time ago. I don't even know if we could get our money back. And so we had them do this this update audit for us.
00:11:22
Speaker
and they didn't find much. And then we put that code on chain. We put out a ah bug bounty on it, pretty large bug bounty. and the first day we put up the bug bounty, basically ah a bounty hunter, you know, submitted something, dinged us, and we paid out like three times what we had paid this audit firm in auditing on like day one of the contracts being on bug bounty platform.
Adapting to Audit Contests for Improved Security
00:11:49
Speaker
So that was a wake up call for us. um And there's more to that story, but basically we kind of figured out, okay, we can't trust these traditional audit firms. We think the Coderita model is cool. We had a lot of changes we wanted to make to the Coderita model.
00:12:07
Speaker
And so we did that. We kind of launched it as a hybrid between traditional audits and Coderita. We had the lead senior Watson designation to make sure that somebody really top was going to be dedicated to every single audit and be trying really hard.
00:12:20
Speaker
And so we launched with that as like, hey, let's just try to get really good at auditing so that our smart contract insurance can be cheap enough that protocols can afford it. So that's why we got into auditing and audit contests originally.
00:12:34
Speaker
Okay. Can you, know so now I remember the first time I heard of you was the incident with Euler. Can you tell us what happened there? Yep. Yep. So Euler incident, super painful um period in Sherlock's history, probably the most painful um to give the full picture there.
00:12:54
Speaker
This was before we had gotten into audit contests. We were doing auditing for Euler and we had some really top guys. I think Chris Michelle did one of the audits. um we were We basically were doing the independent auditor model that Sherlock, now our Blackthorn brand and kind of like Spearbit, we were running that model.
00:13:15
Speaker
in like late 2021, early 2022, actually before Spearbit was even doing it on their side. And so we had all these really top guys. I would have to remember exactly who audited Euler, but Chris Michel, some of these guys, I think Watchpug looked at it as well. And then the Euler had a small update yeah EIP 14 or something that they wanted to, like Euler improvement proposal 14.
00:13:43
Speaker
And so we kind of scoped it for them and we said, yeah, this is pretty small, but we want to spend extra amount of time on it. So we kind of like doubled the scope of that audit. um and WatchPug did the audit um and you know nothing nothing bad happened. That was in like May of 2022.
00:14:04
Speaker
And a lot of things happened between May of 2022 and Euler got hacked. Like two months afterwards, we switched entirely to audit contests because we felt that that was
Incidents and Insights on Vulnerabilities and Bounties
00:14:14
Speaker
a more secure model than putting a few independent auditors together. can talk about that.
00:14:20
Speaker
um And so the Euler exploit happened in, I believe it was March of 2023. So it was about nine months where that code was live on chain. We had a million dollar bug bounty and Sherlock paid for the entire bug bounty because through our insurance protocol, we paid for people's bug bounties. So we paid for Euler's $1 million dollar bug bounty for nine months on chain after Watchpuck had done that audit.
00:14:45
Speaker
not a single bounty hunter found this exploit, which was this donate to reserves function where you could kind of mess up the... You could change the way that somebody who had deposited, you could change their position essentially by donating extra to it.
00:15:04
Speaker
um And this is how you can do the exploit. um And so nobody found that and you know they got exploited in March of 2023. And yeah, it was one of the worst periods in in Sherlock's history for sure. Obviously, one of the worst periods in Oilers history as well.
00:15:21
Speaker
you Do you remember their TVL at that time? Wasn't it about 200 mil, I think? Yep. Yep. About 200 million. And I think their bounty was half a mil. What do you think about that? Like, why why didn't this dude just say, hey, I'm going to report this? Do you think it's, and this is just the the overarching theme of, we we still don't know what the fuck to do, to how to properly incentivize would-be black hats with this model.
00:15:47
Speaker
But I mean, do you think um you think he just thought, hey, I can get away with it? I could pocket 200 mil, 500K's not enough. Do you think if the bounty was, you know, five mil, he would have done the same?
00:15:59
Speaker
Yeah, it's a great question. So the bounty was a million dollars, which maybe not that big of a difference, but it was one of the biggest bounties of the time, especially when we put it up in 2022. It was like, I don't know top 10 or top 20.
00:16:12
Speaker
um And obviously 200 million, you're looking at that, you're like, okay, that's half a percent of the TVL. By some, we can talk about what you think about this, but by some bounty hunter metrics, you know, 10% of TVL is kind of standard. Although once you get to big numbers, that becomes...
00:16:29
Speaker
you know, no protocol is offering 10% of TVL, at least before a hack. um So, yeah, I really don't know. i mean,
00:16:41
Speaker
There's certain parts of that situation that I probably shouldn't get too deep into in terms of like the hacker and kind of what their situation was. um But it's it's unfortunate. I've seen it a couple times too, ah behind the scenes and in different areas where you kind of wonder like, why did you hack this?
00:17:00
Speaker
ah you know for Especially for smaller amounts where there's a bug bounty, you know you can make a good amount of money, but they decide to hack it anyways. and it's It's very strange to understand that decision.
00:17:14
Speaker
um And I think that's changing a little bit with people getting prosecuted more now. like I think the Mango Markets guy and things like that, like hackers are kind of not thinking that they can get away with it as much anymore. But back two years ago, you know people would just kind of not think through hacking very much, it seemed like, and would just kind of go for it.
00:17:37
Speaker
Yeah, I remember the Euler hacker, I think he had to look up how to deploy a smart contract. like He really didn't have the best kind of knowledge of the whole process.
00:17:48
Speaker
Guys like that will get caught. But I think the human mindset, you know it's it's men, and men want to flex as well. So if they don't get caught, they're going to talk about it to someone, probably get caught later.
00:17:59
Speaker
Unless you do everything right, you know, but I hope it's changing. It is a a different landscape now, but a problem we're still working to solve, I would say. Yeah, it's crazy. I mean, you've got guys like, what's his name? And Dean, who's still like just a Canadian dude who's been on the run for how old? He's like 19 year old Canadian guy who's been on the run for two and a half years, hiding from world authorities.
00:18:26
Speaker
Which hack was this? This was indexed finance? But indexed. Yeah, but he he came out. Didn't he say no code is law? And didn't they go through the courts? um I have to check on that. i know i thought it was this Andine guy from Canada. And then he is thought to have done a second hack um more recently.
00:18:49
Speaker
But... Yeah, I don't know. i or he was He was called to court, but he didn't show up. And now he's been on the run ever since. That was my understanding. but Okay. On the run of the blockchain.
00:19:01
Speaker
I love it, man. This place is crazy. Well, what you're doing what you're doing is great, man. Like, Sherlock, ah that's great. let me Let me ask you something. like Because I'd say the top names right for bounty hunting would be Immunefy, Cantina.
Competing with Other Bounty Platforms
00:19:17
Speaker
ah I honestly haven't looked at your what bounties are offered, but the next name that comes to mind is Hack and Proof, and probably you guys as well. um how do you How do you kind of compete in this space with the other guys? Like, what are you what are you offering that's so different?
00:19:35
Speaker
Because you can offer a project, hey, we'll take X amount for fees, but hey, on top of that, come with us because all the best bounty hunters are coming here, which, you know, i don't know how you'd come up with that metric if you do say that because from ah bounty hunter perspective,
00:19:51
Speaker
We'll just look at projects and then see where the bounty's hosted sometimes. Like say I find something interesting and then I just say, well, you know, do they have a bounty? I don't care where it's hosted. it it doesn't matter to me.
00:20:03
Speaker
And then I'll go, yeah hopefully there's a bounty hosted somewhere. That'd be great. And it's not HackerOne or one of these like Web2 type platforms because then I know that the dollar amount is going to be shit.
00:20:15
Speaker
Right. But that's how that's how I kind of look at it at the space. Some people love going on ImmuneFi, Cantina, probably Sherlock and just look at the bounties and then look look at the assets and go that way.
00:20:26
Speaker
i was just curious, like how you how you kind of how you target your clients. Like, how do you say, hey, we're better than the other than the competitors. Here's why you should hunt with us. yeah Yeah, it's a great question. um And you know I think if we couldn't find differentiation in bug bounties, we wouldn't have gotten into the business at all.
00:20:47
Speaker
um And now, we just announced yesterday, have the biggest bug bounty ever in Web 2 or Web 3, the $16 million one with usual money. with usual money um and So there are some unique things that, that we're doing that I think are setting us apart, even though frankly, we're newer on the bug bounty side, like on the contest side, we were very early to that. We were right after code arena, um, on the bug bounty side, it was always something we wanted to get into because.
00:21:15
Speaker
we had we had run bug bounty programs for both of the protocols we built. So you know my co-founder is Solidity guy since 2018, and we built two protocols and we had that bug bounty, which I was telling you about earlier, which we had some ah payouts or at least one payout from.
00:21:31
Speaker
um And so we kind of knew from the project's perspective, all of the pain points. And there were some pretty significant ones. like For us, getting the bounty set up was difficult for a lot of our clients.
00:21:44
Speaker
We've tried to help them set up bounties so that you know we can get the insurance on them and things like that. um And they really it takes it takes weeks sometimes with other bug bounty platforms just to get the bounty set up.
Tackling Spam and Ensuring Serious Submissions
00:21:57
Speaker
And then one thing we experienced really strongly was At first we had, you know, criticals, highs, mediums, I think maybe lows listed on our on our bug bounty platform.
00:22:09
Speaker
And we just got so much spam and so much irrelevant stuff that we got rid of everything except for criticals. And even when we kept a criticals only bug bounty on this other bug bounty platform, which I won't mention, um we got so much spam. I mean, we have- This is before LLMs?
00:22:26
Speaker
This is before LLMs. um I mean, we haven't, updated those smart contracts in probably two and a half three years and we still a week or so ago i got another you know critical notification critical vulnerability notification you know fee on transfer like some bs um but your heart drops every time because you're like oh my god someone found a critical right so it's like pretty uh annoying from a project's perspective and then you have to put whoever you're Solidity developers are, maybe you have a security guy, like they can spend hours looking into, especially these LLM ones. I mean, we've seen some where you spend, I think we spent like a day and a half with one of our clients trying to figure out if this bug was real or not.
00:23:13
Speaker
And then it turned out like, oh, they misunderstood. the like VM dot warp ah foundry function like cheat code. And you know, you couldn't use it in this realistic, it was unrealistic way of using that um in this really massive exploit that took a ah lot of hours to figure out.
00:23:34
Speaker
um So the spam thing is actually a big problem for protocol teams because that you know that team, they didn't develop anything for two days because they were looking into this one spam submission. um And so we took some of those pain points and we basically got rid of all of them. So our bug bounty platform is actually really different from those other ones that you mentioned. We're the only ones who require $250 deposit from bug bounty hunters. I love that.
00:24:01
Speaker
That's a great idea. i've i've been saying that. I didn't know you do it. Great idea. That's good. to i was going to ask your thoughts on that because i thought you might hate it. But no, because the spam thing, I've been on your side to helping a project go through submissions for for a contest one time. And the amount of spam, this is pre LLM.
00:24:21
Speaker
And it's fucking ridiculous. The amount of spam that came through now. I mean, you need AI to deal with AI. It's just too much shit. Yeah. People need to have skin in the game. Right, right.
00:24:33
Speaker
And even the, you know, you can't trust the triagers either on some of the other platforms, unfortunately. So even the ones where the triager says, yeah, this doesn't seem valid, like most teams still look through all ah all of those anyways. So like the spam filtering isn't really helping you that much.
00:24:50
Speaker
um So we have that $250 deposit and the way it plays out, this is going to be more for a protocol team than maybe relevant for bug bounty hunters. But You can only use our bug bounty platform if you've done an audit or a contest with us, because if you've done that, then we know everything about your code base. We know the scope. We know the rules. like We build um teams bug bounty programs before they even ask us for it, because we just have it already. like We have all the information, so we just put it together and then say, hey, do you want to have a bug bounty program?
00:25:25
Speaker
here you go, it's already built. So that saves them weeks and saves them a lot of time trying to migrate stuff to a different platform. um And then The reason we only allow teams that have done audits or contests with us on our bug bounty program is because the triagers then, which I don't like the term triage, but the people who review the submissions are the top auditors from your audit or from your contest.
00:25:51
Speaker
um And so this is maybe more relevant for the bug bounty hunting side. Sure, you have to pay $250 as a deposit. ah You get it back if you're correct about the submission, by the way. And what you get instead of like this triage process or whatever, kind of a black hole for a week when you submit it, you immediately get a security researcher who knows this protocol better than anyone else telling you if the bug is real or not.
00:26:14
Speaker
um And this is great for the protocol team as well, obviously. And we use that $250 to pay that person to review the bug. um Now wonder if, and that's, I like that idea. And I wonder if the idea where you, and mean, you're selectively taking on protocols where you've worked with them in the past.
Protocol Ghosting and Trust Mechanisms
00:26:32
Speaker
Do you think this helps protocol ghosting, which is, which is a shitty thing that happens to to bug hunters where ah protocol gets listed and this happened on immune five, a few times with a few different but bug hunters and they'll list it. And then people submit bugs.
00:26:49
Speaker
They close it. They change the code, whatever they do, whatever they can to fix all the bugs. They get kicked off and they got a free audit or a free bounty hunt. Yeah. Yeah. Yeah. I think that's a terrible problem. a Really hard one to fix. Like I'm unifies come out with the vaults thing where it's like, okay, some protocol teams have some money.
00:27:08
Speaker
They show you that they have the money at least. And you know, there are even programmatic ways we actually have a flow where you can programmatically have a payout if you're on, if you're using our coverage so that the bounty hunter only trusts Sherlock and trusts our,
00:27:23
Speaker
on-chain claims mechanism, which we can go into later. um But so we solve it that way. If the protocol team says, hey, we want to have a 200K bounty, but we don't want to pay out the 200K. We want Sherlock to pay it out. We're going to pay Sherlock, you know, 10K a month for that privilege.
00:27:42
Speaker
then we actually decide on the payouts, which is a lot better than a random protocol team deciding on it from the bug bounty hunter's perspective, I would assume, because Sherlock has to play this, has to continue playing this game and our reputation has to continue, you know, being good into the future on this.
00:27:58
Speaker
So we do try to do what's right. And we have those on-chain mechanisms as well. Um, and then I think what you're also getting at is just the, uh, you know, it's like the opposite of an adverse selection problem where you're getting protocol teams that are willing to pay for security because they're already paying for Sherlock audits. Sherlock is not the cheapest auditor out there by any means. And so you're already getting this like selection of protocol teams that have demonstrated that they care about security and are willing to pay for it.
00:28:27
Speaker
So I think it does create a much better group of protocols. Mm-hmm. No, I think it's pretty good. I think everyone really needs just needs to think about human incentives with anything that they do. like what's Usually it's follow the money, but just think about what people want and how they can get it and what incentivizes people. And that usually should lead you to the right answer and how you implement your systems or or what human behavior will will be doing in a certain situation.
00:28:58
Speaker
um We had some some questions from Discord. I saw White Hat Badge was asking something about your bounty data. Are you going to make it public? Yeah, bounty data. me Maybe at some point, I think, does any bounty platform make their data public? I guess they they do some monthly recaps. I've seen those.
00:29:19
Speaker
Yeah. Well, say you're on a mutify, right? And you're you're looking on their bounties. And what I don't like is, so you'll see a list of projects and then some projects will say private for amount paid or amount submissions, whatever.
00:29:32
Speaker
i don't I don't like that. And I understand why they want to do it, but I don't like it at all because... it's like be transparent about your security posture and your response of everything should be out there this is open source this is a decentralized system and we're supposed to be looking at your open source code and we want to know if it's worth our time to look at it so how many bugs have been submitted how many have been paid out all kinds of these details would help us a lot to kind of screen projects and just find their targets to hunt because
00:30:03
Speaker
Last thing hundred wants to do is just fucking waste his time on something. That's, I mean, we waste our time all the time, but we at least want to be able to choose what we want to waste our time on, at least with some sort of background.
00:30:17
Speaker
Yeah. Yeah. I think that's, I think that's a really fair perspective for Sherlock. I think we'll release some data once we have, uh, we don't want it to be able to be reverse engineered. We don't have that many bounties still. I think we're getting close to like 20.
00:30:34
Speaker
um But for a long time, you know, there's been a few and the dollar amounts are specific. And so we don't want people to be able to reverse engineer things that the protocol teams don't want them to be able to reverse engineer.
00:30:48
Speaker
um So we're going to be a little bit careful about that. I will say in terms of data, we've only ever had one invalid submission in our history on the bug bounty platform side, ah which I think is interesting and kind of demonstrates that the $250 deposit makes people take it pretty seriously.
00:31:10
Speaker
um And then in terms of showing the the bounties that you've paid out, I agree with you on that. If I were a security researcher, I would 100% want that. It would make me feel much more comfortable hunting.
00:31:25
Speaker
for that protocol. um The problem is from the protocol's perspective, who they answer to are investors and users and investors and users have this completely unrealistic expectation of, okay, this team has really good developers.
00:31:41
Speaker
They're not going to have any findings in their audit. They're not going to pay out any bug bounties and they're definitely not going to get hacked.
Transparency and Security in Bounty Platforms
00:31:48
Speaker
And this is like the base case expectation from a lot of users and investors.
00:31:52
Speaker
So protocol teams feel if they show, hey, we paid out a critical bounty, they feel that it makes them look weaker and it reduces trust from their users and investors.
00:32:05
Speaker
Yeah, and no, ah I get it. I kind of miss the days when it was just devs, like little devs teams doing everything and there was no investors and you find a bug, you just talk to the devs and fuck, it just feels like everything's so professionalized now in this space, even though it's still very young.
00:32:25
Speaker
It's just like, you know, these, these are the realities of the current situation, unfortunately. Yeah, it's changed a lot. It's ah growing up for better or worse. Like KYC is probably a ah discussion we could have around bounty, you know, your thoughts on that.
00:32:43
Speaker
um But that's definitely becoming ah bigger deal as you're trying to onboard BlackRock into the space and things like that. You know, i think it's a waste of time. I think it's just a facade.
00:32:54
Speaker
We all know it could be gamed. Go open up a Telegram channel and you can get your fake docs physical. You can get your your fake virtual docs from AI. i mean, what are you are you mandating KYC on your bounties? um Let me think.
00:33:10
Speaker
i don't think we do, actually. We let the protocol team decide, and I know some have decided not to do it. um We don't mandate KYC in our audit contest either. We let the protocol team decide if they want it or not.
00:33:22
Speaker
um You know, early on, Sherlock and Code Arena, neither one had KYC and it was kind of like, look, this is game theory, you know, there's going to be hundreds of people looking at this. So you're going to get security through incentives and through the mass of people who are all trying to find the same bugs.
00:33:39
Speaker
um But we've definitely seen a trend towards, hey, we need to be you know buttoned up here. We need to be responsible in air quotes. At least say we're buttoned up. Have KYC in these things.
00:33:53
Speaker
Yeah. Exactly. Yeah. Have the appearance of being buttoned up. Yeah. What about, so I'm just thinking about like self-dealing, like all the shit that goes wrong with this kind of permissionless anonymous ah ecosystem that we're in.
00:34:06
Speaker
With your contests, right? I mean, man, I just saw a bunch of complaints in the Discord about all All the problems that go on. And I didn't really do contests, I wasn't aware of a lot of this. But then once people brought it up, I was like, well, yeah. I mean, just like people work at Coinbase and Binance front-run token listings. And everyone's going to try to...
00:34:28
Speaker
stay under the radar and do some shady shit every now and then because they're incentivized by money.
Gameability and Fairness in Audit Contests
00:34:34
Speaker
So how do you deal with your guys? Is it a gentleman's handshake where you're like, Hey man, don't, don't, don't submit a finding and then judge it from your a non account, like stealing submissions, you front running.
00:34:50
Speaker
How do you handle these issues on your platform?
00:34:56
Speaker
Yeah, yeah, it's a great question. i'll I'll take small issue with the complaints. We definitely have complaints in our Discord about how we can improve things and why someone didn't get a certain payout in a certain contest.
00:35:07
Speaker
But I think in terms of like... gameability real unfairness. We haven't had those situations that maybe have happened on other contest platforms where someone has done some self-dealing or there's been you know massive kind of duplication.
00:35:25
Speaker
I'll tell you something. like We try to solve everything through pure game theory and incentives because that's like the best way to do it if you can. Having to slap people on the wrist who do bad things after they do them is just not an effective way of of running a platform because there's an unlimited supply of people who are just going to try that stuff.
00:35:46
Speaker
um So I think Sherlock's been pretty clean. like We've seen very few instances where maybe somebody has submitted their finding twice. So I'll just give you some background on one way to game the system, which maybe shouldn't talk about public. Tell us all the ways.
00:36:05
Speaker
Yeah, I'll tell you, i can i can go through a few different ways. One of the most obvious ways is create multiple accounts. You know, okay, you have to have another Discord, another GitHub, another,
00:36:17
Speaker
whatever to to get around this, but you can create multiple accounts. Maybe you know your sister makes an account and you just use that account as your second one. And so you can submit findings through multiple accounts.
00:36:28
Speaker
And this is a way of earning extra money because of the Sybil formula that Sherlock and some other audit contest platforms use. But here's the thing, we can fix that.
00:36:39
Speaker
It would just make it a lot more punitive for any duplicated findings. So it would kind of hurt the system a little bit, but we can fix that to the point where there's no incentive to do it. The reason we don't is because even today, it's not a strong incentive because the way that you make money, if you duplicate your own finding,
00:37:00
Speaker
is if there's a lot of findings, if there's like 20, if there's, sorry, not a lot of findings, a lot of duplicates, if there's 20 duplicates, so you're in an audit contest, you find this vulnerability and 20 other people find it too and submit it in this contest.
00:37:14
Speaker
In this case, you can make like two extra dollars or maybe like five extra dollars by trying to civil it and creating multiple accounts and submitting it through multiple accounts. but the problem is if it happens that you are the only person to find that critical vulnerability and then you civil yourself uh you actually lose like in some cases three thousand dollars or one thousand dollars and so it's kind of like picking up pennies in front of a bulldozer, if you're familiar with that term, where yes, you can make $10 or $2 here and there by doing this.
00:37:49
Speaker
But as soon as you do it on a finding where there weren't a bunch of duplicates, you've just lost yourself like $1,000. ah thousand dollars So everything else you were making, like it gets blown away by that.
00:38:03
Speaker
um And we'd have to get a little bit into the technicals of like the algorithm and why it works that way. But that's the reason why we've seen ah maybe a sign here or there that somebody did a duplicate, ah you know, civil their finding.
00:38:16
Speaker
But it doesn't have a big impact on people's rewards. And the biggest impact it will have if it does have one is on that person. They're going to lose like $1,000 for doing it because of how the formula works.
00:38:29
Speaker
So but how do you how do you know as a hunter if if your finding was stolen? Like if you submit a critical and whatever, one other guy submits a similar one, how do you know that wasn't an insider?
00:38:44
Speaker
Okay, that's a great question. So that's a different topic. So the one before was you Sybil yourself, right? Like you create three accounts and you submit this finding through all three of your accounts.
00:38:55
Speaker
The one you're talking about is a different situation where you submit the finding and then it gets leaked somehow. You know, the team sees it or the maybe the Sherlock, like the protocol team sees it. Maybe the Sherlock team sees it and then someone else makes an account and submits it because they're like, wow, I'm going to earn $2,000 from this.
00:39:16
Speaker
um This one is is really interesting because when Sherlock I'll tell you ah I'll tell you in story form um how we've dealt with this.
00:39:27
Speaker
Sherlock started out saying, look, you can submit findings during the contest and we're going to share them with the team as soon as they're submitted. This creates a vulnerability where if the team is not honest, they can create an account and they can submit it in the contest and they can like get back some of what they paid in the contest.
00:39:46
Speaker
um Or if it's a rogue guy, they can just make some money on the side and no one knows about it. And so when it was just Sherlock and Code Arena, we realized this. We're like, oh, we're dumb. We didn't realize this was an issue. And so we changed it to where the team can only see all the findings at the very end. And even the Sherlock team, I think maybe there was maybe there were two people who could potentially even go and try to find those findings during the contest and submit them. So it was very like sectioned off.
00:40:11
Speaker
um And Sherlock and Code Arena both agreed. It was kind of a gentleman's agreement of like, this is good for security researchers. Let's not show the findings until after the contest is over.
00:40:24
Speaker
Where this changed was when other people got into the auto contest space, like Cantina and Immunify and some others.
Innovations and Educational Contests
00:40:32
Speaker
And we had built a really great reputation for auto contests over that year and a half. They were really growing. Reputation of auto contests was really going up.
00:40:41
Speaker
And so it was a little bit unfortunate in my view that some of these other players came in and they kind of said, oh, auto contests have a great reputation. let's do like a watered down version of the audit contest and let's compromise in a bunch of areas and let's show the team the findings during the contest. Let's even show them to a judge or our triaging team during the contest.
00:41:04
Speaker
Um, There's a bunch of other things too, like a conditional reward pots. um What else? But essentially- Why do you think they did that? why Why do you think they released it early? Is that just so, hey, two days later after the contest ends, look, we're already done because we've been working the whole time.
00:41:20
Speaker
Like what's justification? Yeah, they did that because customers love it. If customers can see a critical vulnerability, you know, at the beginning of their two week contest, they can start fixing it and they can be ready to launch by the end of the contest.
00:41:35
Speaker
So these bounty hunters don't like it. Bounty it's bad for security researchers, bounty hunters, great for the protocol team. Um, and this is like the story of audit contests and probably bug bounty platforms as well as balancing the needs and wants of those two parties.
00:41:56
Speaker
What do you think about, i think Immunify, they launched their attack-a-thons, ah which is which seems like a contest type thing. is that Have you looked into that at all as far as your competition?
00:42:08
Speaker
Yeah, yeah. So Immunify attack-a-thons, they're basically on a contest with some like educational material ahead of it so that people can get up to speed on the ah the code base before the audit contest starts.
00:42:24
Speaker
um But the the issue I take, yeah obviously I'm umm ah running one of these platforms. So these are our competitors. What we've seen is it's become less of audit contests and more of bug bounty contests.
00:42:40
Speaker
And what I mean by that is in an audit contest, we say, look, there's going to be $100,000 that's going to get paid out to people who find stuff. And so if you're ah bounty hunter or researcher, you're like, okay, cool. If I do a good job, I am going to get some of that $100,000.
00:42:56
Speaker
I feel comfortable spending the next week or two weeks of my time on this. Bug bounty contests are if there is a high or if there is a critical, then we will pay out $100,000 or whatever it is.
00:43:09
Speaker
And these are very different from a security researcher's perspective, as you can imagine, because now it's like, okay, even if I absolutely crush it for two weeks, there might not be a high or critical in this code base. And so I get paid nothing.
00:43:24
Speaker
And so what we've seen is, let's say for example, DeadRoses just put his yield door protocol on Sherlock for an audit contest. He got 600 participants or 601. It was like, I think it's a record. It's definitely a record for Sherlock.
00:43:40
Speaker
600 security researchers looking at this. We've seen blue chip protocols work with some of these other audit contest providers and put up larger dollar amounts, but it's a bug bounty contest where there's no guaranteed payout.
00:43:54
Speaker
And they got like 25 participants. So there's a huge difference between 600 and 25, as you can tell. So we're starting to see that play out a little bit more where the researchers are actually voting with their feet and saying, we don't like these types of contests where there's these huge variable pots, which those aren't, that's not a bad thing on its own.
00:44:18
Speaker
But if there's no fixed rewards, that's a bad thing for researchers. I agree. And so if you could share, it because I know Dead Rose is going to listen and the Bulgaria mafia is going to listen, how many criticals were found in his audit?
00:44:33
Speaker
Oh, that's a good question. I don't actually know that information. I could probably look it up here, but I know they had 750 submissions, which is a lot of submissions,
00:44:46
Speaker
with six hundred people participating you're It's actually on average only one submission per person. So if there was you know just one kind of obvious vulnerability, you know medium severity vulnerability in there, it could have been duplicated 600 times. which well And you say 600 blue chip security researchers. I don't think there are 600 in total blue chip security researchers. But like how do you how do you see that they're quality guys? Just they have a record at Sherlock for finding bugs?
00:45:17
Speaker
Oh, sorry. Yeah, if I said 600 blue chip, um so definitely some of, you know, we don't know who those 600 are. They're just people who want to be security researchers who spent time looking at this code base.
00:45:30
Speaker
Some of them definitely are blue chip security researchers. um Like it was either Simel, I think was Simel who was looking at that. I got first place in it. ah One of the best guys in in the world in auditing right now from a quantitative perspective in contests.
00:45:46
Speaker
And then there's you know probably people who are just learning solidity or just learning security and coming in out of that 600 too. Yeah. You know, that it's interesting from a project's standpoint, it depends how they operate, but for them, the optics are identical, whether they put up a contest and two people show up or 500 because they could put
Judging Process and Participation Strategies
00:46:07
Speaker
it on their site. They could show, hey, look, we had a contest.
00:46:10
Speaker
And most people, you know, investors, whatever, won't do the diligence is, hey, look, it's been audited. OK, great. it's at a contest. Great. But when you look, you know, as you know, the real deal about security, if you had 10 people show up versus 600, it's a different ballgame.
00:46:26
Speaker
Yeah. Yeah. And it's much easier to say, Hey, this is a million dollar contest. If it's only a million dollar contest, if a critical gets found, it's very difficult to say, yeah, this is a million dollar contest where we're paying out a million dollars, no matter what happens.
00:46:41
Speaker
So yeah I like that, man. I don't like the conditional pots. I like what you're saying. Like, Hey, listen, our budget's a million bucks, pick the right platform to put it on So if one guy finds the the fucking low than whatever. We, we said we're spending a mil and that's it.
00:46:59
Speaker
Yep. Yep. And, you know, teams hate it if they spend a million and then there's one low only they're like, wow, this was a huge waste of money, even though they probably got like a thousand person weeks of auditing done, you know, in that two weeks.
00:47:15
Speaker
But yeah. Yeah. All right Let me, let me put another one out there. So say you're, you're at Sherlock and you're, uh, you're a Watson, right? And you say, hey, you know, I'm going to be sneaky. I'm going to go submit a finding. And then I know I'm the judge, right? So I'm going to judge. Is the Watson the judge?
00:47:34
Speaker
sorry, the terminology. Yeah, it's a great question. So the way it works in Sherlock, we have an open... So we have people in the contest and some of them are dedicated, like the lead senior Watson is dedicated. And then there's a separate person who is the lead judge. So the lead senior Watson is never the lead judge. we don't Can the judge submit his own findings through another anon account or there are there two judges that have to review each finding?
00:48:02
Speaker
So the judge could submit findings through an A-non account. So that is an attack vector that is possible. And the way Sherlock's judging works, we have the most...
00:48:15
Speaker
like intricate judging. We have an entire community judging period where we actually want the people who participated in the contest to judge their own submissions. So I know, you know, someone might say, oh, it's bad if, ah you know, a security researcher in audit contest judges their own submission.
00:48:33
Speaker
But we have an entire application we built for judging with these crazy, the rules of it are are a black box. You can't see them, but we have ways of knowing like, okay, this researcher is judging their own submission here, or this researcher is very inaccurate at judging their own submission or whatever it is.
00:48:51
Speaker
So we have a lot of ways of dealing with that. We think it's a net positive if a if the person who submits the finding judges it during our community judging period, um because they just add extra context, you know, they give us more reasoning for why they think their submission is valid.
00:49:06
Speaker
And we can have a discussion or not about why they think it's valid and, and you know, get to the bottom of it. So it's actually a very open system.
00:49:19
Speaker
And every single judge has like a signal score. So at the end of the day, The final arbiter of things is the Sherlock judge. His name is Wang. ah He's absolutely incredible.
00:49:31
Speaker
um it's It's not easy to find somebody who can do that role and do it well. And I think he's gotten extremely good reviews. We've had a few different people be Sherlock judges and and Wang is is incredible at it. And so if a...
00:49:47
Speaker
lead judge were to submit their own finding in the contest, a lot of people would call it out in the community judging and say, you know, hey, this is valid or not. So the lead judge wouldn't get the final say, but the lead judge would be able to have input and they have more weight on their input.
00:50:01
Speaker
But if enough people say, hey, the lead judge is wrong about this finding, um, then Wang or the Sherlock judge comes in and works with the protocol team if necessary, talks to the people who are on both sides of the issue and gets to the bottom of it.
00:50:17
Speaker
So the idea of somebody in the contest or the lead judge being in the contest or some conflict there at the end of the process, it will likely get filtered out and caught. And then they're, uh,
00:50:32
Speaker
signal score and the judging app will go down and then they will earn less money for judging in the future. They will get less weight. their Their decisions will be weighted less by our black box algorithm in the future.
00:50:46
Speaker
So there's probably some ways here and there on the outsides or on the, uh, like edges to game the judging system in small ways. But in terms of big things, I, you'd have to get it past the Sherlock judge who's talking to all parties, talking to the protocol team, and that's going to be difficult.
00:51:05
Speaker
Shout out to big Wang. That's cool. So big Wang, my man, let me ask this. So, uh,
00:51:15
Speaker
Oh, fuck. Where was I going my point, man? I had a great one until I talked about Big Wang. um what What do you think? Like weaknesses in Sherlock's model. Where are you guys looking to improve to kind of attract more bounty hunters? Because I'm not trying to shill Sherlock. ah you know you've taught me a lot about your platform.
00:51:34
Speaker
Which sounds, it sounds cool, man. Like I have, I had some reservations, but what you've told me, ah you're trying to kind of use game theory and have people operate off incentives and you have a bond that people need to post and you're focused just on high severity bugs.
00:51:53
Speaker
To me, that's that's all good. As long as the projects are good actors, which you say you vet them, you kind of know the parties. All those are good signs, man. where Where do you guys think you could improve? Like what what challenges you face right now?
00:52:10
Speaker
Yeah, it's a great question. i think we have great participation on the audit contest side. um I think we need to do more, even more to incentivize top participation, like you know you and White Hat, Mage or Madge, still learning about Sherlock, maybe not um spending a ton of time there yet.
00:52:31
Speaker
and i think the The participation on the bug bounty side, we need to improve. it's It's difficult because Sherlock is like hardcore mode for bug bounties. um you know If you're trying to get into the space, the bug bounty space, submitting a bunch of mediums and lows and things like that, great way to get into it.
00:52:50
Speaker
You can't do that on Sherlock. You've got to pay $250. There's only criticals in a lot of the bug bounties. Um, these are all highly audited code bases. Like they've all been through Sherlock audits, if not, you know, two or three other audits from other firms.
00:53:04
Speaker
So it's really hardcore mode in terms of bug bounty platform. Um, and so I think that, you know, hurts participation a little bit. Um, but it does, you know, it's trade-offs like you just get way less spam and hopefully we get higher signal of top security researchers spending more time here.
00:53:23
Speaker
And you have your own in-house top guys, right? are the What are their titles? So, yeah, we have some guys through Blackthorn. Blackthorn, that's Who are exclusive to Sherlock. um A lot of the guys who have been top in the audit contest. I think in total, Blackthorn ah members have won 125 audit contests. So the the top of the top.
00:53:48
Speaker
And so if you get if you're a client and you say, hey I want this Black Dorn guy on my on my audit, And then is there any case where he just kind of, ah hes he gets paid to show up probably a ah minimum and then maybe incentivize on top of that.
00:54:02
Speaker
Have you had cases where maybe you had a couple of them on on one project and then one of the guys just kind of phones it in and just just, hey, I'm just going to take my 10 grand, whatever for showing up and not really do much? Or is he disincentivized to to do that because he's got a rep score or something like that?
Improving Traditional Audits with Contests
00:54:22
Speaker
this is This is a great question. this is This is a question that I've been thinking about for four years and why we got into audit contests in the first place, because you can you can say this about any traditional style audit, whether it's people through Blackthorn or Sherlock or you know through any traditional auditor where they're full-time employees.
00:54:44
Speaker
What if they you know just decide to play StarCraft for one of the weeks? How are you ever going to know that? What if they don't show up you know at all really and the other three guys on the audit carry them?
00:54:56
Speaker
um This, in my view, is one of the biggest ah like vulnerabilities in traditional audits. And this is why, Sherlock, for two years, we didn't offer traditional audits because we saw this problem. and we were like, man,
00:55:11
Speaker
I don't know. This this seems kind of you know kind of sketchy. um and And so I think it is it is a problem. we We think a lot internally about how can we make these things more competitive. it's It's always a question of does the collaboration of these three or four people does that outweigh the competitive aspect? If we said, look, it's PVP one versus one versus one of these four people, like what would get the better outcome?
00:55:42
Speaker
And we do see, it's tough because we do see collaboration being super valuable in a lot of audits. Like we see the discussions now, we see people riffing off each other and kind of finding things and going down rabbit holes and making each other more efficient, looking in different areas for each other.
00:55:59
Speaker
um So there is like collaborative done well is very, very powerful. Um, the problem is you can't always know that everyone is trying hard in a collaborative audit. Uh, and so that's why I think audit contests are super powerful.
00:56:17
Speaker
And that's why, like, I know guardian, I think has like a two V two model. We've thought a lot about kind of going in that direction. um So this is you're talking about a very open vulnerability in audits that, in my opinion, has been open for the last four years and every audit firm is kind of susceptible to it.
00:56:38
Speaker
Yeah, and unfortunately, this was the end of the podcast because we had connection issues. Or rather, I had them down in my podcast basement. Anyway, this was the first time I heard about Sherlock.
00:56:49
Speaker
I thought he had some really good points. I'm going to check it out. I think the coolest part is having a bond and putting that up because you're putting your money where your mouth is. It helps with the spam, and bounty hunter is going to think twice if they put up $250 in hard USDC cold cash.
00:57:07
Speaker
So anyway, if there's more demand for this kind of stuff, maybe I'll get the big bosses of the other platforms on. I think it's kind of cool just hearing from their end and seeing how we're kind of all working together, doing the same things and trying to secure the blockchain while making millions of dollars and staying jacked and ripped.
00:57:26
Speaker
Anyway, next time, see you on the blockchain.