Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Episode 24 - josselin feist
144 Plays8 days ago
riptide & montyly (josselin feist, humble slither creator) discuss his tenure at trailofbits, web2 vs web3 security posture, security tooling, internalizing security and solving the human problem, concolic execution/hybrid fuzzing, using LLMs as a force multiplier, an ALPHA drop, why putting underscores in your file names is 32337, and much, much, more ...
Recommended![Episode 19 - 0xe4669da [SPECIAL n00b EDITION] image](https://media.zencastr.com/cdn-cgi/image/width=112,quality=85/image-files/677fcded3f04b4e1c3f96cfc/1b4f61b7-5c8c-48b2-9d61-66ec9549c632.png)
Transcript
Introduction & Sponsorships
00:00:07
riptide
Welcome to Bounty Hunters. We're back with another episode. We are sponsored by ImmuneFi, the biggest bug bounty platform in Web3. If you're a white hat, this is where you earn real money and make crypto safer.
00:00:21
riptide
Don't know where you want to start? Hit the feeling lucky button and let the hunt choose you. We're also sponsored by Rare Skills, rareskills.io forward slash Riptide.
00:00:33
riptide
get 10% off a bootcamp. Go learn Uniswap V3, go learn ZK Math, learn it all, go find bugs and make millions.
00:00:43
riptide
And we are also sponsored by GitRecon.xyz forward slash Riptide. If you so launch a fuzzing engagement with ah the chads at GitRecon, you will get five grand off with code Riptide. So check it out.
Guest Introduction: Jocelyn Feist
00:01:00
riptide
Introduce our guest today is Jocelyn Feist, the creator of Slither and X-Trail of Bits. He was there eight years. Good day, sir. How are you?
00:01:12
Josselin
Hey, good morning. Thanks for having me. Looking forward for the discussion.
00:01:16
riptide
ah Very, very excited to have you on. You, I would say kind of laid low in the Twitter sphere until recently. And I'm i'm just saying this because i yeah I took more notice of you recently. I was like, oh, yeah, who's this guy? Because you were behind, you know, the the shield of trail of bits for so long. And now you you're kind of you're out there on your own.
00:01:40
Josselin
Yeah, I think that's correct. I was was never you know too active on Twitter in the past year. I'm trying to be a bit more active, but yeah, I'm still, I think, on on the low side.
00:01:51
riptide
And um
Career Transition & Current Roles
00:01:52
riptide
so what happened? why Why did you decide to to leave?
00:01:56
Josselin
Yeah, I think that's a good question. And, you know, a few people have asked me that. I've been at Trail of Beats for eight years and, you know, you know in the streets, like two lifetimes.
00:02:06
Josselin
So I just thought at some point I needed to do something different. I'm still in a really good relationship with you know everyone at Trail of Beats. I still talk to a lot of engineers there, but I just think after um eight years in the same place, I just needed a change.
00:02:06
riptide
yeah
00:02:21
riptide
Yeah, absolutely. And now, and you kind of, so now you're doing this private security reviews I saw on your, on your page. And then you also joined Perimeter, as you said, the the lead fuzzer over there, which is kind of cool.
00:02:31
Josselin
Yeah.
00:02:34
riptide
ah By the way, get recon.xyz fuzzing forward slash riptide. I just want to throw that in there. i don't know if you, if you've heard of Alex and, and recon and okay.
00:02:42
Josselin
ah Oh, yeah, of course. Of course.
00:02:44
riptide
Okay, cool.
00:02:44
Josselin
I've been discussing you know with Alex for a few years now, probably.
00:02:48
riptide
ah Okay, cool, cool. No, I think more the better in in the field of fuzzing and hybrid and symbolic execution. I think and think anything we can do to kind of find the bugs is is a good thing.
Development & Evolution of Security Tools
00:03:01
Josselin
Exactly.
00:03:01
riptide
um So on your end, I mean, you have, you know, I look at you and I'm like, man, I started with one of my first tools was with Slither. And you're, you know, I'm like in the crib and you're nursing me with the body. Like, here you go. Here's your here's your findings, young researcher.
00:03:19
riptide
And, you know, you know I thought that was so cool. I said, wow, this is... And think back then, I mean, when you first developed it and when that was kind of one of the few static analysis tools to look at, how cool it was to just be able to run through the code and find potential ah flaws. Like from from creation of it to where it is now to to kind of you know what we're doing with fuzzing, ah how do you think...
00:03:50
riptide
What's your view on like how this has evolved? Do you think what we have now is adequate? ah You know, I'm curious to get your viewpoint given so many tools out here now from when you first started.
00:04:02
Josselin
Yeah, I think that's a really, really good point. So to put things in perspective, you know when I started to to develop Slyther, was definitely really cool to see the outcome, but at the same time, you know I'm coming from like traditional cybersecurity and in the traditional like web to work, we already have like a lot of really, really good tools. We have a really good search analyzer. We have really good fuzzle. We have a lot of kind of like things you can use.
00:04:28
Josselin
i And when you know i started within the blockchain space, it was basically nothing, right? So on the one side, it was cool to see kind of like the impact of the tool, but on the other side, was like,
00:04:40
Josselin
Yeah, there's still a lot to be done just to get to a place where we are even close to what we can have in in traditional security. um So it does definitely evolve you know over all the past you know five, six, seven years.
00:04:53
Josselin
um We are in a better place now in terms of tooling, but it's still it's still far from what we have in traditional software security.
The Role of LLMs in Security
00:05:01
Josselin
um like Like the further are good, like Ekinah, Medusa, Foundry, they are doing really well.
00:05:05
riptide
Mm-hmm.
00:05:07
Josselin
But if you compare to what you can have in In traditional cybersecurity, the maturity of the tooling is just not there. um Even Slitter, I mean, I'm really happy where Slitter is, but in comparison to what we could have with more, you know, a couple of years, more time and more effort, I think there's still like a lot of opportunities there.
00:05:29
riptide
Yeah, I geeked out on your on your web page and like before you were doing the Web 2 bugs and and I looked at some of your submissions and the feedback is just like, yeah, fixed.
00:05:41
riptide
Thanks. You know, from. And, and ah you know, the Web3 space is like bounty, you know, multimillion dollar bounties hang is just fixed. Thanks. And that was that was really the expectation with a lot of I mean, many projects you just contribute. Hey, I found a bug. Yeah, OK.
00:05:58
riptide
But you didn't have millions and billions on the line.
00:06:03
Josselin
I mean, if you think about that, because I think what you you're referring is some of the bug I've reported in non-blockchain kind of project.
00:06:11
riptide
hmm.
00:06:12
Josselin
If you think about that like 20 years ago, when you were finding something somewhere you basically could report it, but if you were reporting it, you would have issue with law and you could get into trouble just because you were like genuinely reporting a bug to a protocol.
00:06:30
Josselin
um There were many cases where people were like, oh, okay, you know I found a way to you know get access to your bank account and do A, B, C.
00:06:38
riptide
Right.
00:06:38
Josselin
And the reaction 20 years ago would have been, oh, okay, this guy is a bad actor. We are going to send law enforcement against that person. ah So over, I think, like the past 15 years, it did improve the bug bounty space within Web2 has evolved and now we're in a better place.
00:06:59
Josselin
ah But yeah, for for many years, like but reporting,
Changing Perceptions & Impact of Bounty Platforms
00:07:02
Josselin
you would get and t-shirt maybe you know like a thanks you will be know happy and you know 15 years later when you do that in web3 you get like ah but a really good bounty so it's definitely it does improve
00:07:13
riptide
Yeah. Yeah. Cause guys, I think naturally, you know, with a good moral compass, they want to report, they want to help.
00:07:21
Josselin
yeah
00:07:22
riptide
I think the, my viewpoint on, on human society is that we like to contribute and we like to help each other. And we, we have a positive outlook and then you have the small subset, which is nefarious and they're going to do bad.
00:07:37
riptide
But by default, i yeah I think what you're saying is this wasn't rewarded. This wasn't recognized. There was just hackers. It wasn't black hat, white. It was, they're all bad. You're doing anything.
00:07:46
Josselin
Yeah.
00:07:47
riptide
What are you probing? This is the crime of curiosity, man. No, no, no Put you jail. Yeah, it's it's flipped significantly, I think, for the better. And I think these bounty platforms for Web3 have been such a ah great driver of that as well.
00:08:01
Josselin
Oh, yeah, definitely.
00:08:04
riptide
and um you know you have you have a ah crazy body of work here i i particularly liked your your post i didn't know it was from you until i browsed your page because i'd read it before the breaking of upgrade ability where you have the delegate call self-destruct pattern that kind of no one saw except you and you have a quote in the beginning you're just like even though, you know, we weren't engaged on the job. We just, I just, just opened up the contracts and just browsed and just yeah quickly took a look and found this devastating bug and, uh, for audit firms, I think didn't catch it is what you said. And, um,
00:08:42
riptide
Isn't that the case? Like we, and as this, you know, this moves on to kind of what I'm trying to get at here, but we have this human review problem where it still exists today from, this was in 2020, wrote this, it's 2025 now.
00:08:58
riptide
And it's the same thing. Now we have LLMs. We have, I think, better fuzzing tools. We have, um, you better, well, now we're getting concolic execution and symbolic execution, better tools there.
00:09:13
riptide
And I think LLMs are improving dramatically and we're working on something with that too. But you still have this human problem where you miss things. You know, what is your...
00:09:25
riptide
opinion on this. and And I also want to, not to give you too many things, but you posted something by Gustavo on the new Echidna capabilities around Cacolic execution. Could you talk a bit on that kind of where we're at now compared to five years ago?
00:09:40
Josselin
So I might take the first question and then you know we can move to like concordance execution.
00:09:44
riptide
Sure, sure.
00:09:46
Josselin
ah So I think now related to like the human problem you're mentioning, I think a lot of people and a lot of teams see security as something external to them.
00:09:57
Josselin
Like they are going to hire like a security provider, they are going to do like a contest, bug bounty, and then kind of externalize security. And I think that's kind of like the root of the problem. the will In my opinion, like the real long-term solution is for security to be embedded in the development lifecycle, to have maybe like external review along the path and maybe like know through through multiple cycles of the development.
00:10:11
riptide
Thank you.
00:10:22
Josselin
But it's really about enabling the developer to build safer code. because that's really what all of this is about there will always be you know an expert there will always be kind of like you know a new risk and the best the best weapon against that is really to enable developer to avoid making the mistake in the first place so so it's really about you know having a good design having a good testing strategy having kind of like a good process on how you develop the software rather than always kind of like thinking and and seeing security as a last step before deployment because that
00:10:56
Josselin
doesn't work as as we know.
00:10:59
riptide
Oh, absolutely. And I wonder if we're going backwards with the vibe coding LLMs. Like how can you internalize security? You're internalizing it to Claude, you know?
00:11:11
riptide
And I don't think the devs, I think what you're saying is right.
00:11:11
Josselin
Yeah.
00:11:14
riptide
Like the devs that prioritize that, you see that. And these are the top tier guys that build ground up with security in mind. And the other guys just say, well, let's kick it over to the trail bits and they'll they'll figure out everything.
00:11:26
Josselin
Yeah.
00:11:27
Josselin
Yeah. For, for, for LLM it's definitely going to be tricky because everyone is going to start using you know llm to write code but the quality of the code which is kind of delivered is not there as of today and i i actually don't recommend anyone to write production solidity code with llm it's really good if you want to write you know test if you want to do like you know this type of things but to write like the business logic through llm
00:11:28
riptide
Yeah, well,
00:11:41
riptide
Mm-hmm.
00:11:56
Josselin
most likely you're going to have a bug and it's going to be subtle and you're not going to notice it until it's too late. ah So anyone who is deploying a smart contract which is going to handle funds should not have a single line of code written by the LAMM in my opinion.
00:12:13
riptide
I would agree. And I would say principle, like I use it to write POCs all the time and you still have to double check those because they'll just make a crazy assertion and, and, or a crazy prank call.
00:12:25
riptide
But how do you, how can you write code, the production code with LLMs and put it out there? Even if you, you get the test running, whatever, and your contracts are out there, you don't understand it You don't know what it wrote.
00:12:38
Josselin
Yeah.
00:12:39
riptide
like Like just what you said, you don't know the design choices and trade-offs that it made.
00:12:44
Josselin
Exactly. And there's a lot of sub substantial, you know, like a lot of small and minor things within, within your cut base might have an impact, you know, down the path. So you need to control and understand every line of code that you're pushing out there. And if you didn't get that to LLM,
00:12:59
Josselin
it's it's not going to happen. And you know people are lazy. If you receive some code which is written by LLM, you're going to check it quickly, but you are not going to check it like you know in depth and really try to understand what is written and what's our so assumption.
00:13:13
Josselin
So I think as a you if i were if I was like a team lead in a DeFi protocol or this type of things, I would have as a strict guideline, no one can deploy code which was written by LLM.
00:13:28
Josselin
use LLM to write documentation, use LLM to write test, but do not use LLM to write production code for smart contract.
00:13:38
riptide
Absolutely agree. do you think How many projects you think are doing this, though?
00:13:43
Josselin
It's hard to tell.
00:13:45
riptide
That's the problem. It is hard to tell.
00:13:46
Josselin
I think, yeah, I think the ah team that are mature, they know about that, I think, and they probably don't use LLM. The team that are a bit more, you know, younger or they have like a a short time to market, you know, type of strategy, they're probably not going to do that and they're probably going to rely on LLM.
00:14:07
Josselin
So, yeah, bad things are going to happen for this team.
00:14:12
riptide
Yeah, like I was curious, what do you see is like, you know, when I was reading that, what you posted about congolic execution, which is they call it hybrid fuzzing or when you do symbolic execution and fuzzing.
00:14:25
riptide
do you Do you see this like maybe plus LLM reviews as maybe ah ah kind of a holy grail type thing to stopping most bugs?
00:14:35
Josselin
it is going to help it is not going to be bulletproof proof so one of the things i've also tried to to you know to push to to my client and to the industry is this notion of invariant driven development where really when you write smart contract what you want to know you know from even before writing a single line of code is the invariant. So invariant are things about the protocol that should remain true.
00:15:00
Josselin
A good example is in Uniswap where you have the K invariant and you know when you're doing the swap at the end, K should not change unless you have like some some fear counting there.
00:15:12
Josselin
ah But from the moment you know kind of like the invariant of the protocol, you can design the code base around them, you can design your testing strategy around them, and then you can use additional tools like fuzzing, symbolic execution, formal method to verify invariants.
00:15:28
Josselin
So there, fuzzing, symbolic execution, concordial execution, they are going to be a piece in the puzzle to improve the security But I will even go further and say it's really about the invariant, you know, from the first place. so And it's really about knowing what your code base is supposed to do.
00:15:46
riptide
and And so what if certain invariants are missed by the protocol team? Like as an auditor, do you kind of look for situations like that? when Like in your role as a fuzzer and in your prior role, are you going in there and maybe looking at their test suite and seeing, hey, what did you miss?
00:16:05
riptide
And then compare that with what you know about the code to kind of come up with with different pathways?
00:16:08
Josselin
Yeah.
00:16:10
Josselin
Yeah, exactly. like When I'm doing like an invariant engagement, I'm going to do exactly what you say. um So part of invariant engagement is going to be to write like the fuzzing harness and like all the setup for for the code base, which can be complex for for some code base.
00:16:25
Josselin
And part of it is also to to discuss with the engineer to understand the invariant, to understand how we are to cover, to understand what they have done so far and how to to to help them in the long term.
00:16:36
Josselin
um So there's definitely like a lot of kind of like ah exploration of invariant. um So yeah.
00:16:43
riptide
And when you were a trail biz, what could you explain their audit model? Because the only one um that I've seen clear transparency about is Hexans because they put it on their audit reports and and where they have basically have two teams.
00:16:55
riptide
It looks like they have one guy, you know, one set of teams review and then they send it to the other team. So they do a double review of each report. And I think that that sounds in theory like a great model.
00:17:01
Josselin
Mm-hmm.
00:17:04
riptide
How did you guys do it over there?
00:17:06
Josselin
Yeah, it's much more collaborative. um So we, like, know, when I was at Trail of Beach, we were not setting street guidelines into, okay, this is how you do a review. We, you know trusted the engineer to have, like, the right process by themselves.
00:17:20
Josselin
But basically, we foster collaboration. um We foster, like, internal discussion, discussion with the client also, because at the end of the day, the client have way more understanding than you would ever have.
00:17:31
Josselin
um you know, like if you review a code base for four weeks and they have been working on it for two years, they definitely have a lot of insight that you should try to extract. So it was less about competition and internal competition to know who is going to find the most bug, but really more about fostering a collaboration among the engineer so that they can build up on each other's understanding.
00:17:51
riptide
but said Did you always have a review no matter what is a final review by another team?
00:17:57
Josselin
No, no, it was not like multiple team in kind of, uh in different like segregation it was really like everyone together
00:18:06
riptide
Okay, and and and i'm not I'm not at all trying to flex on the great Joslin at all, but this is a follow question I have because i think your technical expertise is far beyond mine. And I just, you know, I threw some luck into here, but can we talk about the Arbitrum Nitro review from 2022, which you led?
00:18:25
Josselin
um Yeah, it might. It was a long time ago.
00:18:29
riptide
nothing Nothing crazy about it, right?
00:18:29
Josselin
We can try.
00:18:32
riptide
I mean, it's 150 page review. you know You guys went all in. And you did that in March, and I found that bug in September in Arbitrum. And the the root cause of this bug was due to this post-upgrade init function where they wiped three slots clean, and then they reinitialized two, and they left the third blank.
00:18:56
Josselin
okay
00:18:56
riptide
And they told me, and they had multiple auditors review.
00:19:00
riptide
I don't know if it was just you. They told me, oh, this we put that in there because of the feedback from one of our security partners as an optimization. And I didn't see anything in the review about it I've never got an answer for it.
00:19:14
riptide
who Who suggested it? And I was just curious if that was if you knew who was behind it.
00:19:19
Josselin
I mean, to be honest, from a review three years ago, don't remember very much. Usually we don't tend to recommend optimization because optimization are risky.
00:19:29
riptide
Yeah. Mm-hmm.
00:19:33
Josselin
um Because exactly what what you you you you you will say.
00:19:36
Josselin
I would like for optimization recommendation, I basically have two mode. Either the optimization is clearly having no security impact. Like for example, if you read twice a state variable and there is no external code, there's nothing and you can, you know, be sure like you can just cache it in a local variable.
00:19:36
riptide
yeah
00:19:55
Josselin
Sure, you can do it. But for optimization that are more like low-level, like you know you can use assembly or you can you know skip some other things, I would be surprised if it was you know something we recommended.
00:20:08
Josselin
Let's say from a report from three years ago, it's difficult to remember.
00:20:14
riptide
put you on the spot not at all flex just just curious yeah
00:20:15
Josselin
Yeah, especially especially if it's not in the report, I would be surprised like if it was coming from us.
00:20:23
riptide
yeah and uh you know also so like you put it you put on your page that you're doing security reviews and you specifically didn't use the word audit why is that
00:20:34
Josselin
So I've been using security review foe many years. And I know there is some kind of like, you know, Twitter fight about audit versus security review.
00:20:46
Josselin
um Security review does make more sense, but I might choose also audit, you know, from time to time because that's what people are expecting.
00:20:51
riptide
Mm-hmm.
00:20:53
Josselin
i To be honest, I don't think audit versus security review debate is that important to me reflects more like a kind of like ah internal bubble within the Web3 security space where you know people might debate about things that don't matter that much.
00:21:09
Josselin
um Like if you go to a client and if you go like to the developer and and you know, like that community, they probably don't care if we call it an audit or security review, even if security review is kind of like more Truth in in like in a straight sense.
00:21:25
riptide
Okay. Yeah, I was just curious how you were categorizing it because I use that terminology as as something I do, is which is like ah people come to me and I say, hey, wait until all your
Security Reviews vs. Audits
00:21:35
riptide
auditors review. And then you could also call it like a targeted bounty hunt.
00:21:39
riptide
But I say security review because I'm not going to give you gas optimizations.
00:21:39
Josselin
Hmm.
00:21:43
riptide
I'm not giving you low mediums. Like I'm only giving you things that will break the protocol, just critical size, just stuff like So that's how I defined it.
00:21:48
Josselin
OK.
00:21:51
riptide
That's why was curious how you were defining it.
00:21:52
Josselin
Okay. so So you are doing yeah really more like time bound, back bounty type of things.
00:21:59
riptide
Type of thing because beef, like say you, you get all your reviews and then your bug bounty is a million bucks. Well, Hey, just let me work on it first. And then if I find a critical, look, you're saving a million bucks and you're, I'm charging way less than that to do a a targeted review.
00:22:14
Josselin
That makes sense.
00:22:15
riptide
So, yeah.
00:22:16
Josselin
So the way i see it is that a security review can take many forms and it depends on the scope. So we can end up having like, you know, 10 different names for the services and for the products, but I think just, you know, having one name makes it a bit easier.
00:22:32
Josselin
And what I mean by that is that when you do a security review, depending on the context and the scope, you might adapt. For example, if you review something which is still in development and you know some components are not fully fleshed, some components are still like moving and the developer will know that, okay, this is going to be you know changing,
00:22:49
Josselin
During the review, you're not going to focus on that and you're going to maybe focus on on something different. um if you If the developer is more interested in like design because it's halfway through the development, you might focus more on long-term recommendation.
00:23:02
Josselin
While if it's like two weeks to deploy like the protocol, you might shift the focus into exactly what you say and more like know high and and and critical. ah um But you are still doing, and my you know in my mind, security of you, it's really more like you adapt based on needs of the client in that sense.
00:23:18
riptide
Right, right. No, good point. And what was I going to ask you? Oh, yeah. ah Something I have to know, like, what is in what is in your tool suite, your toolkit?
00:23:31
riptide
Like how are and you post something recently, like how are you using LLMs as a military term as a force multiplier in your kind of day to day work?
00:23:40
Josselin
that's That's a okay really good question. i use it for everything now. um
00:23:47
riptide
ah You admitted it.
00:23:48
Josselin
but Yeah, I mean, for for for example, you know, like I use cursor when I explore a code base to kind of chat with cursor to get some insight and, you know, draw some graph and um things like that.
00:23:58
Josselin
ah It's a really good assistant for for some of the things. ah When I need to write a proof of concept, I'm also going to delegate part of it, you know, through an LLM. ah It's not perfect, but it's going to, you know, scaffold like the test and this type of thing. So just saving time.
00:24:14
Josselin
um When I'm building tooling, now I'm also using LLM and I'm, you know, delegating part of the implementation to the LLM. um And this has saved me also like a lot of time. Like, yeah you know, I used to i know spend a day to write like a custom static analyzer for a specific target.
00:24:31
Josselin
Now I can do it in like two hours with an LLM. So it's pretty good.
00:24:35
riptide
That's what I know. It's, it's, there's so many ways to save time. It's just, it's, it's so hard not to rely on it, but then you have to you have to don't get lazy, right?
00:24:44
Josselin
Yeah, I think that's a real important piece, like don't get lazy, in the sense that you should not delegate your reasoning to the LLM.
00:24:45
riptide
That's it.
00:24:54
Josselin
You should still drive like you know what's important. You should still drive like the reasoning. ah You cannot just rely on LLM, for example, to find bugs, but you can rely on the LLM to help you to get an understanding quicker and so on.
00:25:08
riptide
Yeah, if you can if you could put it over the target, it could be pretty good. But if, hey, just go find all the bugs, it doesn't work that great. But if you can put it around, you know, your 25 meter target there, it could be, and and especially with, what was I using, Grok 4, where it has full web access, full research.
00:25:19
Josselin
Yeah.
00:25:26
riptide
My God, search whole repos. My God, what a time saver. If you're not using that for your bug hunting, you're crazy. Even though it messes up sometimes, it's like, my God, what a time saver.
00:25:36
riptide
Highly recommend
00:25:37
Josselin
Yeah. and have you Have you tried DeepWiki?
00:25:41
riptide
What is it, Deep Wiki?
00:25:43
Josselin
Yeah, it's it's from, I think, Cognito. It's basically like a code exploration for public GitHub, where you can just build any public GitHub and they have indexes, everything they created, like a summary and graph, and you can just query any code base remotely.
00:26:01
Josselin
um So if you do bug bounty and you explore like, oh, okay, like this new project on GitHub, DeepRicky is really, really good to give you a first insight.
00:26:10
riptide
I'm gonna check that out, DeepWiki. You know, GitHub Search, I've used that a lot and I cannot believe you still cannot do a regex on GitHub Search. Like, I feel like it's artificially neutered.
00:26:24
Josselin
might
00:26:24
riptide
you're you're You're probably laughing because you you built some secret backend API tool where you're regexing left and right.
00:26:30
riptide
And you're laughing at my novice, my fledgling behavior.
00:26:34
Josselin
No, I mean, i mean i'm I'm nothing because I'm thinking about like the backend of GitHub. And if you allow rejects, that's a lot of computing power which is needed. Like running regular expression can take some some some processing power if you if you allow any type of regular expression.
00:26:51
Josselin
um So it might not be that easy to do that.
00:26:55
riptide
Yeah, it's there's gotta be some reason.
00:26:57
Josselin
Yeah.
00:26:58
riptide
And then, ah I don't want to bring it up because it's it's such an old school tool like the Ethereum ah verified contract search. Like when you really want to get lazy, by God, it's the worst search ever, but it's it's it'll do it a pitch.
00:27:13
riptide
um ah what ah oh yeah Oh yeah, are you familiar with Lightchaser?
00:27:19
Josselin
Um, yeah, I never use it, but I'm familiar, like of the name basically.
00:27:24
riptide
So i I did an episode with Chase the Light and he has built his tool where he has, what does he call them? Not agents, but detectors.
00:27:35
Josselin
Mm hmm.
00:27:35
riptide
And he said he's amassed this list of, I think, over a thousand detectors.
00:27:41
riptide
And um he he says all hand-coded, you know, coded by hand. And he he does it all himself. Maybe works with one other person. But he he expressed very high confidence in his tool and the improvements in his tool in version four on where he's been able to consistently find high severity ah vulnerabilities just using ah static analysis.
00:28:08
riptide
I was curious on your thoughts on that.
00:28:10
Josselin
Oh yeah, I mean, I totally believe it. I mean, I found a lot of bugs in Slitter, so i'm not surprised if, you know, people building LightShare, know, have similar outcome. There are a lot of like simple, not necessarily simple, but some form of pattern which are really good for static analysis.
00:28:27
Josselin
um And you can find like bug at scale. I think the real challenge is not to find bug. And it's the same with LLM, like it's not to find bug, but it's to find,
00:28:38
Josselin
bug with a good ratio over the false alarm and the false positive. Because finding bug is the easy part. Removing false positive is difficulty here.
00:28:49
riptide
Mmm, yes, yes.
00:28:50
Josselin
Like even with Slitter, like the summer of the detector, Marc-Andre Vintour- there are some design shows where we could say okay we are going to cover more ground and you know we bought more but. Marc-Andre Vintour- we are going to be okay to miss some of the bug because we are going to reduce the ratio between false positive and and and true positive um so there is like a a lot of design direction and and and decision that you can make.
00:29:16
riptide
I feel like someone is is out there building the ultimate tool where they're combining like a light chaser with you know the right AI agents with fuzzing where it's a kind of one click, hey, look at this code base.
00:29:30
riptide
And it's just cranking on AI credits and it combines everything at once, fuzzing, everything.
00:29:30
Josselin
Yeah.
00:29:35
riptide
and And I mean, it's probably already in development, but I feel like that's that's kind of the ultimate auto tool that we need in this space.
00:29:44
Josselin
Yeah, from from what I can tell you not talking to a bunch of people, everyone is going in that direction. I think today, most of the last team are trying to build that capacity. And a lot of individual or like, you know, small group or like two or three people are also trying to build something like that.
00:30:02
Josselin
I don't know if you if you follow. um But there was like the AI XCC result a couple of months ago, which was a kind of a competition to find bug automatically using traditional program analysis and LLM in in like C and Java, funded by the DARPA.
00:30:18
Josselin
And it's exactly what you described before, like traditional analysis and traditional software.
00:30:19
riptide
Mm-hmm.
00:30:24
Josselin
So definitely everyone is going in that direction.
00:30:28
riptide
and And my thoughts on that are are, that's great, but I also know the blockchain and it's it's a place, when you when you were in DeFi summer, where protocols developed by, protocols, right?
00:30:42
riptide
Developed by 16 year old in India can amass, you know, 50 million.
00:30:44
Josselin
Thank you.
00:30:46
riptide
And this guy just picked up his Solidity course online and anyone could deploy things and attract capital. And so we can have all the tools necessary, but in in my eyes, we have the human element that is always vulnerable, not just in this case, but as far as just pushing out code and and permissionlessly.
00:31:07
riptide
So, you know what do we what do we do, Joseph? What do we do here?
00:31:10
Josselin
Yeah, that's what I was saying at the beginning, like the solution is not on the security researcher side. The solution is really on helping the developer to get better because that's how, like, you know like if everyone that start to learn Solidity or start to, you know, do all of that, get some flavor of, okay, you have to think about security, you have to, you know, have like all this mindset.
00:31:21
riptide
Hmm.
00:31:33
Josselin
um And if we educate kind of like, you know, more developers toward that direction, That's how we mature as an industry. That's how we improve in the long term.
00:31:43
riptide
Mm-hmm. Yeah. In some ways, I think our space is getting better and then a lot of lot of ways is getting worse as far as... like
00:31:52
Josselin
It's much better than like five years ago, let's say.
00:31:55
riptide
Yeah, I mean, there's there's definitely more awareness. Like a lot of people at least know how to read a smart contract and they know what's going on.
00:31:58
Josselin
yeah
00:32:02
riptide
But I mean, look look now, the amount of drainers and scammers and front end, like any any vector, when you have this much money involved, any vector is being exploited.
00:32:14
riptide
Now compared, way more now, I would say than compared to back then.
00:32:17
Josselin
Yeah, like I agree, like I think ah a lot of attacker moved away from smart contract and, you know, started to find any entry points to attack and steal money which are not smart contract related.
00:32:30
Josselin
So that's definitely like something we have seen. And for like end user, you know, like obviously like all the scammer or like the phishing and everything, they are making a killing for themselves.
00:32:41
Josselin
um
00:32:43
Josselin
So.
00:32:43
riptide
And salinity extensions or VS Code extensions.
00:32:46
riptide
like you know it's like It's like we're walking through a minefield every day.
00:32:46
Josselin
Yeah.
00:32:50
Josselin
Yeah.
00:32:52
riptide
And I remember you know past audit reports that you always look through and you guys would do it, everyone would do it, but it was always like, hey, um admin centralization, risk.
00:33:03
riptide
And everyone would write it off. Oh, it's not really a finding. And then we're seeing like, hey, really the wrench attack, the digital wrench attack is working. Offer them a job and then compromise their system.
00:33:15
riptide
Yeah, it's...
00:33:15
Josselin
Yeah. That's why I know like it's also important in the long term that like large team do like thread modeling and this type of activity for them to understand what are the risk, what are like the boundary in the protocol, um because there is so many ways to, you know, steal money from a protocol that are not related to issue with a smart contract.
Physical Security & Generational Approaches
00:33:38
riptide
Did you ever get into that at all? Like, you know, the physical side of it or exploring those kinds of avenues? Because in in our space, it's laughable. You'll see whole teams, the the whole multi-sig shows up at a conference. You know, it's so laughable. it's just, it's crazy.
00:33:56
Josselin
Yeah, so I've provided advice, but I'd never performed like a physical pen test or this type of things.
00:34:03
Josselin
It's less it's less of ah of a thing in Web3 just because everyone is remote. So like, you know, if you go if you go like in more traditional company, they have an office, they have like, you know, server, they have like all all of that, like like database somewhere.
00:34:04
riptide
Yeah.
00:34:13
riptide
Mm-hmm.
00:34:20
Josselin
um But it's a different set of that. Most of the blockchain team are remote and don't have even like or real office.
00:34:28
riptide
Yeah, yeah, I think the the only, I don't know, there's just an exploit all the time, but we have the hardware wallets, which was such an improvement off just writing down your keys.
00:34:39
riptide
i I think that that's really our are like iron gate, holding a lot of these attacks at bay, like these trainers, like thankfully you can do something that I wouldn't say 100% guarantees your funds not to be lost, but it's ah it's a very high wall to climb.
00:34:55
Josselin
Yeah. Yeah. You need like, you know, air gap machine or so, and you need a lot of different things because even if you have like a hardware wallet, like even when you need to sign a transaction, it's really difficult to understand what's going on with the transaction.
00:35:04
riptide
Mm-hmm.
00:35:07
Josselin
um So that's why you probably want to to have like an air gap machine machine that you just use to sign transaction if you manage any significant amount of money, right? um So there are a couple of like strategies that you can you can apply.
00:35:20
Josselin
um It depends on how much funds we are talking about, how many you know people are within the process, what's kind of like the context of your of your company.
00:35:31
Josselin
um But there are definitely like things to improve.
00:35:34
riptide
Yeah, and i don't I don't know your age, your maybe you're my age, something around there, but it I don't know if it's a ah thing with the younger groups because most of my listeners are 18 to 24.
00:35:46
riptide
And I've worked with some of them and I've seen how they work. And they're very, the the opposites of of what I would do, right? Like I'm very, um I don't trust the cloud.
00:35:57
riptide
I don't trust my file names have underscores in them, you know, like things like that.
00:36:01
Josselin
for
00:36:02
riptide
But they're they're very happy to put things in Google Docs. google Like everything's in the cloud. They're cool with it. They're running 10 different browser plugins that they're somehow cool with that.
00:36:15
riptide
And it's just a lot of things where I grew up kind of paranoid on the computer, where if the file's not on my computer, then i don't trust And like so we have all these things and all these kind of guideposts that are already out there. But do you feel like the younger generation, they just, you know, I guess a different mindset, they don't care as much or or don't know about it as much?
00:36:38
Josselin
I'm not sure it's a generational thing. It might be, but I'm not sure. And the reason for that is that a lot of people that are not security oriented, you know, above 30, 40, 50, they do the same.
00:36:55
Josselin
So I think it's it's more about like having kind of like a security mindset and having like a kind of an education around security um that change perspective on things like being a bit more paranoia, being a bit more, okay,
00:37:06
Josselin
how to trust something, what you can trust. And I think it cost comes more with kind of like your exposure to security.
00:37:15
riptide
Yeah. Yeah, I guess I would agree. True. yeah Yeah. Because i could I could talk to people that that are my age that would do some really dumb shit. That's true.
00:37:23
Josselin
Yeah, I think for a lot, one thing I've noticed for a lot of like, you know, younger people in the new web3 space, several or large majority came to security through Solidity.
00:37:24
riptide
I think you...
00:37:36
Josselin
They came through security, you know, through smart contract security. They never did any need like web security or traditional application security or this type of things. but They might not have like the traditional security background that you might have from, from people from other community.
00:37:51
Josselin
And because of that, they might not have this education about security because all they know, all they have experiment is around smart contract security, which is kind of like a a subset of what you can do.
00:38:03
Josselin
So I think like, and I've seen that, you know, with a lot of people where you have to kind of like help them to understand, okay, there's like all these things that can happen, there's like all these possible things, but it comes from exposure to like security risk.
00:38:18
riptide
that's That's a good point. Yeah, they they view security just through the Web3, yeah, the focus.
00:38:21
Josselin
Yeah.
00:38:24
riptide
They're not thinking Tempest attack on my ah network cables there. yeah And the Web2 thing is just like, you know, the the crowd now has grown up with the browser as the brains of the computer too, which is,
00:38:38
Josselin
Yeah.
00:38:38
riptide
which is totally different than what I'm used to. And I think that's affected a lot too, because the browser is now to me, I mean, it's critical infrastructure on my computer. So those, those extensions I've scrutinized highly and they're, they're only used absolutely when necessary.
00:38:56
riptide
And it just opens up so many things that I don't think people consider.
00:38:59
Josselin
Yeah. Or even, you know, having like a, like if you work, if you're an individual having ah laptop for work and a laptop for other things um is a practice that probably most people don't even follow if they don't work at a company.
00:39:00
riptide
Yeah.
00:39:12
Josselin
um So they end up, you know, mixing personal action with, you know, whatever using for work, which definitely increase the risk.
00:39:21
riptide
Yeah, definitely not good.
00:39:22
Josselin
I think it comes also from, in a way, I've noticed a lot of people, you know, started a journey through bug contest and you know going through like like individual and competition.
00:39:34
Josselin
When you do that, you might be missing a bit of the mentoring that you get at a company in terms of like traditional security, where you need to have something on your laptop that, you know, add like additional guarantee. You need like to have like some specific access control.
00:39:51
Josselin
You have a lot of kind of like hygiene that you get when you work in a security company that if you don't have again exposure to it, it's difficult to get by yourself.
00:40:01
riptide
ah Very good point. Yeah, you do kind of absorb things that are just around you at working for a company.
00:40:05
Josselin
Exactly.
00:40:06
riptide
That's true. No, good point. um I want to get to some questions, if that's all right with you from the discord.
Rebuilding Slither & Fuzzing Techniques
00:40:16
Josselin
Go ahead.
00:40:16
riptide
And you can, you you could choose the answer or decline, whatever.
00:40:19
riptide
I don't think they're too, they're too tough. This one's kind of interesting. So K42 says, if you were making slither from scratch today with all the wisdom and knowledge of today, but you only had two weeks to do it, what would you focus on for optimal results?
00:40:34
Josselin
Oh, two weeks.
00:40:37
riptide
Vibe code.
00:40:39
Josselin
Yeah, if it's if it's a two-week thing, I would focus on a few types of vulnerabilities that I want to discover and focus all the design, all the analysis, toward finding one or two specific vulnerabilities.
00:40:53
Josselin
Yeah, for two weeks.
00:40:54
riptide
What would those be?
00:40:56
Josselin
I mean, don't know, let's say you want to have the best Marc-Andr Goulettenau- only we enter on see detector in the world right and you have two weeks to do that, then you can focus on okay how to detect the pattern, and how to.
00:41:08
Josselin
Marc-Andr Goulettenau- Make sure you remove false positive what type of analysis, do you need to go through that um so it will be really about like what's the end result and how to work backward or that.
00:41:22
riptide
I love that class. Read only reentrancy. Do you remember who actually found that? I can't remember who did the write up.
00:41:27
Josselin
So.
00:41:30
Josselin
So who coined the term? I think it was chain security or it was open Zeppelin.
00:41:36
riptide
I think you're right. I i think it was chain security.
00:41:38
Josselin
Yeah, but people were finding this type of bug before, right?
00:41:39
riptide
um I'm going to say that.
00:41:42
Josselin
But they did coined the term and created like more awareness toward it.
00:41:47
riptide
You're saying curve. That wasn't the first one that it was founded.
00:41:49
Josselin
No, no, I'm saying people were finding like, you know, read only reentrancy before without calling that read only reentrancy.
00:41:49
riptide
Yeah.
00:41:57
riptide
Okay. Okay. No, that, yeah, that's a great one. I love that. That's, I think that's the the dream of a bug hunter is to find his own class of bug.
00:42:06
Josselin
yeah Okay.
00:42:07
riptide
Ideally he should get it named after himself, like the riptide recently. That's like the, the wall of nerd fame put me up on there. Okay. The next one is from a,
00:42:19
riptide
your hardcore competitor and and colleague, Alex. So he says he says, what is your setup for fuzzing? when When do you fuzz and when do you just do manual review?
00:42:32
Josselin
That's a good question. It depends on the context because if I am on a fuzzing engagement, then I'm going to do end-to-end fuzzing. I'm going to create like a complex harness. I'm going to discuss with the client to find the invariant and, you know, all of that. So that's like kind of like an AV process.
00:42:47
Josselin
Now when I do traditional in a security review and my goal is not to focus on fuzzing, the way I use fuzzing is as a targeted approach. ah For example, if I know there is some weird arithmetic going on and I notice, okay, like, you know I have a feeling that this rounding is not right. I have the feeling like, know, like this multiplication, this division, if something is wrong, i am going to use fuzzing in that direction and I'm going to you know, mark the contract, I'm going to create kind of a model of arithmetic in a separate out contract, and I will do targeted fuzzing ah based on the model of the contract.
00:43:22
Josselin
And I do that so that I can abstract all the harness, I can abstract all the kind of like configuration, and really focus on the part of the code that I want to fuzz. um So doing a manual review, that's really how I do it usually.
00:43:35
riptide
What would you say would be like some of those red flags where you're like, yes, im this is what I'm going to, going to fuzz.
00:43:41
Josselin
Like, arithmetics is really where I, for the most, like, you know, when I see no explicit winding direction, you know, over the code base, or there is like always like, you know, some, some um it's always winding up or something like that.
00:43:50
riptide
Hmm.
00:43:57
Josselin
Like, you know, when I see complex arithmetic, it's usually a place where I'm going to use a further. and And the reason for that is that even if I find the bug manually, I need like a confirmation because like arithmetic can be tricky.
00:44:09
riptide
Mm-hmm. And what about, do you do the do you do hybrid fuzzing as well?
00:44:14
Josselin
um By a hybrid, fuing you mean things like symbolic execution and combination?
00:44:16
riptide
Like the... Correct, yes.
00:44:19
Josselin
um Not directly. I do use from time to time almost when I need like some some level of confidence in in in what I'm doing. So I'm going to use Echina, Medusa.
00:44:31
Josselin
as as a first step to first. And then for some things where the further doesn't give me like information I want, I might switch to symbolic execution. However, that doesn't work all the time. And ah it depends on the complexity of the operation because if it's too complex, um like the solver is not going to be able to handle it.
00:44:49
riptide
Okay. And that that was his other question too. Do you use formal verification tools? Which ones and when? Mm-hmm.
00:44:54
Josselin
Yeah, so I use almost from time to time, not on a kind of like a you know continuous basis, but when there is a specific need, I'm going to use it.
00:45:02
riptide
Okay. And like when you do some of your fuzzing, is there, do you have like a specific, how do you come up with the amount of runs that you're going to do?
00:45:04
Josselin
Thank you.
00:45:11
riptide
and And do you have dedicated hardware for that? Like how, because I've i've seen, I've seen, I think it was Enigma Dark or what of these people, someone posted something where they had, they had things running for a couple of days on some fuzz tests.
00:45:24
Josselin
Yeah, so if it's for an environment development kind of engagement and I need to first for like multiple weeks, I might you know deploy something in the cloud, for example. But if I do it during manual review, that's more something I'm going to run for like you know over the night on my on my my ah personal machine.
00:45:42
Josselin
um So it depends on what I'm trying to do. Usually for targeted fuzzing, it's going to run for like a couple of hours at max because um um'm I'm not targeting like a large piece of code.
00:45:45
riptide
Okay.
00:45:52
Josselin
So it's going to find, or it's not going to find. For more complex engagement, it's going to depend.
00:45:58
riptide
Okay. and And what is your personal machine? Just out of curiosity, what are you running?
00:46:03
Josselin
um It's a desktop machine. So it's like a, I don't know, Ubuntu.
00:46:07
riptide
what What OS? What OS are you running? Ubuntu.
00:46:11
Josselin
Yeah, I have also Mac, so it depends what I'm doing.
00:46:12
riptide
You know, I got rugged on the Mac. I'll tell you what, like I've got just, I was running Ubuntu for many years and I was, you you know, you have your normal frustrations when it upgrades your, your network driver and you, you got to roll it back, you know, this and that, right.
00:46:16
Josselin
Oh, what happened?
00:46:29
riptide
And I, I eventually saw this stupid marketing, which they got me on, which was for this, the Mac M3 and I got an M3 max. And it was like, oh, it's so fast, this and that. And I saw, you know I've used Macs in the past. It is sleek.
00:46:43
riptide
I buy one. And it's just so disappointing. It's a Mac is great. Like the interface is great. And then it just does things so badly. Like the security updates all the time. There's always these holes in there.
00:46:56
riptide
And then it's just it just doesn't perform that the way you'd expect such high-end hardware to perform. So I think I was rugged big time. I think if I had this hardware and I was running Ubuntu on it, it'd be way better.
00:47:09
Josselin
I mean, I do have a Mac too, so it depends on the need I might switch, but yeah.
00:47:09
riptide
and then
00:47:10
riptide
the
00:47:13
riptide
You know what I mean. Yeah.
00:47:15
Josselin
and
00:47:15
riptide
Yeah. It's just disappointing. I think Steve Jobs is rolling in his grave. Anyway. Okay. So Alex has some other ones. He says, which improvements are still missing in security tooling from your perspective?
00:47:28
Josselin
um
00:47:30
Josselin
Okay, that's a recurring joke, but we are still missing a debugger. I mean, I remember on back in like 2017, 18, like people who were like, oh, we are finally going to have a good debugger.
00:47:42
Josselin
And you know, it's many years later and we don't have a good debugger.
00:47:46
riptide
I have one for you. Have you heard a Rumblefish?
00:47:47
Josselin
Oh, no.
00:47:49
riptide
I'm going to send this over. I met this guy at an ETH event and he wrote this on-chain debugger and he wants feedback.
00:47:55
Josselin
Okay.
00:47:56
riptide
I'll send it to you offline, man. But the thing is crazy. It's really cool.
00:48:01
riptide
So maybe we're part way there.
00:48:04
Josselin
Oh yeah, that would be perfect. Like if you have a good one.
00:48:06
riptide
yeah Yeah, sure. ah Okay, last question. He says also, in which security changes would you want to see to smart contracts and processes that projects go through?
00:48:19
Josselin
And processes, I think it's really,
00:48:20
riptide
In the context of of improvements, missing and security tooling.
00:48:23
Josselin
Oh, for security tooling. um I think it goes back to like the invariant-driven developments, because once you know you're invariant, you can use different tools and different techniques to verify them.
Invariant-Driven Development
00:48:35
Josselin
um But it's it's less about the tool, it's really more about the processes.
00:48:36
riptide
Mm-hmm.
00:48:39
Josselin
ah So the things I wish you know to see more across developer, is thinking about invariants. And in fairness, have seen an increase of that over the past two years. I see more developers coming with, so you know when you start review of like, okay, this is like 10 invariants we know about and this type of thing.
00:48:58
Josselin
So it is improving and it is going in that direction, but there is still a lot of room for improvement.
00:49:05
riptide
All right, cool. um Joseph, the final ah final boss question. And i didn't prepare yeah i don't prepare anyone for it. yeah i So I do an alpha drop where I drop some sort of alpha for the bug hunters that listen out there.
00:49:19
riptide
And it could be code specific thing. could be a bug pattern. It could be general life advice. It could be anything you like. um So I'd ask you for an alpha drop. I'll give you a minute to think about it while I drop some alpha. So this is on, um and it'll be kind of vague. i'm going to keep it vague because I like to.
00:49:39
riptide
So this is on 4246 vaults, ERC 4246, and it's about vault wrappers. So just check check before the vault wrappers kind of introduce a vulnerability depending on how they're coded up.
00:49:55
riptide
um that could just check the shares and check the share checks on those wrappers ah before things make their way and the logic flows to the vault because you may run into some interesting findings there because some certain things aren't considered because it's kind of out of the spec when someone writes a wrapper that talks to one of these vaults.
00:50:19
riptide
um Mr. Joslin, do you have any alpha for the listeners?
00:50:22
Josselin
um Okay, one thing I would recommend like people doing bug booted to look for are the hook architecture from like Uniswap, Balancer and all the protocol that to follow it because I'm kind of expecting that people building hooks on top of like major protocol might not necessarily have like you know a mature security process and probably like the integration some of the access control some of the data a validation they do through the hooks is probably not that great so if you want to you know hands for that ah there's probably some chance to find something
00:50:58
riptide
Very good. I like it. Well, cool. ah Hey, we're we're at about an hour. Thank you for coming on. It's been a ah real pleasure and an honor ah to get your expertise on the podcast.
00:51:11
riptide
And everyone, I think, will enjoy this episode. So thank you, Joseph, for coming on. And everyone, we will see you next time on the blockchain.