Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Episode 20 - chasethelight
170 Plays8 days ago
riptide & chasethelight discuss how getting rugged on BSC led him to create his automated bug finding tool Lightchaser, why programming in C and ASM can make learning new languages easier, why static and dynamic analysis trumps LLMs, why you should dig deeper to outperform automated bug detection, why we need bounty hunters and the importance of manual review, how Lightchaser V4 is leveling up bug detection, and much, much, more ...
Recommended![Episode 19 - 0xe4669da [SPECIAL n00b EDITION] image](https://media.zencastr.com/cdn-cgi/image/width=112,quality=85/image-files/677fcded3f04b4e1c3f96cfc/1b4f61b7-5c8c-48b2-9d61-66ec9549c632.png)
Transcript
Introduction and Special Guest
00:00:07
Speaker
Welcome back to Bounty Hunters Life on the Blockchain. I'm your humble host, Riptide. And we have a very interesting guest today, a little bit different.
Promotions and Offers
00:00:16
Speaker
but Before we kick off, we we all know we got a shout out about the fuzzing kings over at getrecon.xyz.
00:00:24
Speaker
Use forward slash Riptide on that to get... the hugest deal of your life five grand off for first-time customers for an invariant testing engagement these guys i mean guys if you want to fuzz you want to find the bugs you got to get recon if you're a project you want to deploy You want to put it out live on the blockchain. You're not fuzzing.
00:00:48
Speaker
You're crazy. You got to get recon.xyz forward slash Riptide. Get in there. Talk to Alex. Have a consult. He'll sell you on it. You'll be secure or nothing's perfect, but it's going to be much more secure than if you did not do a fuzz.
00:01:03
Speaker
If you deploy without fuzzing, you're crazy. You heard it here first. Also, if you want to get some skills for the bounty hunters out there, go to RearSkills.io forward slash Riptide, get 10% off a boot camp and let's kick it off.
Guest Introduction - Chase the Light
00:01:18
Speaker
So today we have Mr. Chase, the light, the creator of Light Chaser.
00:01:23
Speaker
How are you? I'm doing good, thank you. Thank you for having me on. Also, I love you, Injur. I'm just making it up as I go, man. No, thanks for thanks for joining, man. i have i'm you know I'm not a contest guy, but I have read read plenty of the after-action reports and contests, the bug write-ups.
00:01:43
Speaker
And I saw, I don't know when I saw you, probably more than a year ago. And I'm like, what is this light chaser dude? like Like, what is this? ah what What's this guy running here? How does he find all these? Because I remember before you were on the scene, people would get on there and they would submit things. They'd get like, oh, know, zero address missing.
00:02:01
Speaker
And they would earn 20 bucks or whatever it
Origin of Light Chaser
00:02:04
Speaker
was. And then obviously, you know, kind of tell us how you how you said, hey, I'm going to just enter into this arena, taking, know, like, i don't know if you base it off a Slither or Mithril or kind of kind of talk about the background of this tool and why you got started.
00:02:24
Speaker
Absolutely. um So Lightchaser has a bit of a weird start so i actually began development of light chaser in late 2021 so it's actually quite quite old um but its original purpose wasn't necessarily for vulnerability detection at least not initially um this is i'll do a quick rundown of the history of light chaser real quick So around 2021, 2022, obviously that was like DeFi summer and lot of these like little projects were launching.
00:03:05
Speaker
And I think on Reddit or something like that, I kind of saw these little projects. And when I kind of saw that, some people were essentially 100Xing like overnight. was like, that's interesting.
Smart Contracts and Learning Journey
00:03:17
Speaker
Um, so I kind of like read up on it a little bit and, uh, on the Binance smart chain, I kind of got on pancake swap, which is the Binance smart chain equivalent of the Uniswap.
00:03:30
Speaker
And, uh, yeah, I just kind of found a random token on, uh, Reddit and, uh, Yeah, i I think I put in like 15 or 20 dollars, like a small amount.
00:03:41
Speaker
And yeah, it went up like 50% or something like that. Oh, that's like pretty cool. So I just kind of just did that randomly. But then one time when I did that, my money essentially just magically vanished. i was like...
00:03:53
Speaker
What on earth happened? um ums so And obviously that happened to me quite futile and also i wanted to investigate what was going on. So I kind of looked into it and I knew about smart contracts, but I didn't really know what they were. So I kind of i looked into that and that, oh okay, okay. So they're essentially code that,
00:04:17
Speaker
that's what these UIs are connected to. So when yeah I'm doing something through PancakeSwap, it's actually calling these functions on the EVM. And once I started digging through, I started reading contracts. And because I had a C background, and if...
00:04:34
Speaker
C is one of your first few languages. if you In my personal opinion, if you learn C, learning any other language becomes far, far easier. So I actually learned Solidity very quickly. I didn't really need to do like a course or anything because I could understand the context and I could could i essentially just figure it out as I read more and more contracts. And that's when I began to learn, oh okay, okay, so this particular contract...
00:04:58
Speaker
there's a function that allows the admin to sell set the selfies, but there's no limits on it. So they could just set the selfies to 100 and basically use that as a essentially to make the contract a honeypot.
00:05:10
Speaker
And there were loads of other little things that these malicious um devs would do. Like another one which they would do is they would have like a secret mint function that...
00:05:21
Speaker
would essentially allow them to mint themselves like a trillion tokens and then they would suddenly just dump the entire thing and just crash the entire token. And there there would be other cool... Not not cool, but like interesting stuff that they would do.
00:05:33
Speaker
um Obviously, people started looking out for... um oh is the owner ownership renounced um because obviously if the ownership is renounced then there's less of risk because those they can't do stuff like adjust the buy and selfies and stuff like that but what they were actually doing is all these things in you could sneak them all in because no one was really you know deep diving it like you and me we were just degening in like ah we didn't know what the fuck we were doing ah Yeah, 100%. Like, that yeah, i've i've I've lost more money than want. I'm too embarrassed to admit. But anyways, so and so like they would do like other weird stuff. Like they would renounce the ownership and then you will see it on like BSC scan.
00:06:15
Speaker
Now, if you go onto the read tab and then you can see that, oh yeah, own the owner address is and address zero. So it's being renounced. But then what they would do is they would have like a random like state variable with that, would that had their address.
00:06:29
Speaker
And then they would have like another random hidden mint function that would look if the message center is that state variable, that constant. And basically they they would they still be able to do the same thing, but people would get fooled because they think that the ownership got renounced.
00:06:43
Speaker
So I kind of picked up on these patterns and there were quite a few of them. And, every time before I would ever like put in $15, $20 in a token, I would always check to see if it had any of these um like sketchy things in the code.
00:06:59
Speaker
But I started doing that for every single time. And I'm not sure if you know, but for micro-cap tokens, which I think is more of a formal term for these really small new tokens that used to come out, timing is very, very important. Let's say a new token comes out the difference between investing within the, within a few minutes of that token launching and investing like 20, 20 or 30 minutes later can be huge. It could be 10 X, 20 X, 30 X. So there was a massive difference.
Development of Light Chaser
00:07:30
Speaker
So speed was very important. And obviously me reading this stuff manually um took time. So um I thought, Hey, this can be automated.
00:07:40
Speaker
And that's, how the idea for Light Chaser came about. i mean, it wasn't even called Light Chaser back then. um But yeah, so i I created this program and at the time I didn't actually know that these other tools like Slither existed and stuff like that. So I was just kind of building it from scratch. And I personally always was in favor of building stuff from scratch anyway. So I just kind of went down that route. And was only when I was decently... but like down that road when I realized, oh, okay, these other tools exist as well.
00:08:13
Speaker
But obviously those were more like vulnerability detection. I was like, i want i wanted to build my own thing that was completely unique. So I kind of had like a self-imposed rule that I won't look at any online references or any online resources because I was a worried that if I did that, I would just kind of end up with like a Slither clone.
00:08:30
Speaker
wow So I just kind of went into it completely blind, but i I did have somewhat of an automation background. So that's how Lightchaser V1 came about. Oh, this is this is also something.
00:08:43
Speaker
um Yeah, Lightchaser was actually just one half of the overall project. I actually created two frameworks. um I created a contract analysis framework, which is obviously now we know as Lightchaser, but i also created a blockchain analysis framework, which would use ETH RPC calls um to like check stuff like for buying selfies. um I had a really cool module, which would essentially an ETH RPC call and check to see if there were any Uniswap swaps, and they would like grab that data,
00:09:10
Speaker
and it would look at the the amount that the user wanted and compared to how much they got and I would use that to determine the buy and sell fees. So it could also tell me very very accurate fees as well.
00:09:22
Speaker
And obviously that was a framework so you can actually code detectors for that framework as well. But that's data v1, I never developed that further unfortunately. um I only further developed Lightchaser so Yeah, that's kind of like the beginning of a fly chase. so that's That's really cool.
00:09:40
Speaker
and I'm curious. you so You started with C. You had a C background. And you said if you know C, you're you're clearly a C max. You said all languages after that kind of come easy. Why do you think that is?
00:09:54
Speaker
um I think because C kind of forces you to learn the programming fundamentals really, really well. Whereas a lot of more modern programming languages kind of abstract all of the very technical, difficult to learn stuff.
00:10:09
Speaker
um For example, with Python, you never even need to worry about stuff like pointers, really. So because it kind of forces you to learn those... deep fundamentals, it kind of makes other programming languages far easier. And I also have a very mild background in assembly as well. I've done some assembly as well, so and obviously c So I think that kind of gave me a very good foundation. But no, C wasn't actually my first language.
00:10:34
Speaker
Python was. But i fully got comfortable with C as well. And then that's when I learned Solidity. And I don't think it was my Python experience that made Solidity easy. It was rather my C experience that made it easier.
00:10:47
Speaker
Interesting. Yeah, I, it was, i I mean, I tried C way back when I was 16. I just, I knew the basics, but I could definitely see what you're talking about and, you know, probably C++ too, just because of the object oriented stuff and solidity, but,
00:11:02
Speaker
I would say for me, it was like a JavaScript. It was like such a clear carryover to solidity, but I didn't have that background that I think what you're saying is vital, like to have to have ah the knowledge behind just dealing with pointers, dealing with memory allocation, malloc and everything in C, and then having an assembly background.
00:11:22
Speaker
you know Now I see how beneficial that could be to really have that when you, do the deeper you dive in the EVM,
00:11:32
Speaker
Yeah, no, I completely agree. And I don't just think it's programming exclusive as well. I think in general, if you learn the harder variant or something, the stuff that you actually need to use on the day-to-day basis um becomes far more easier. So like um in the UK, you have A-levels. I'm not really sure what the US equivalent of that is, but I did further maths for A-levels. And because further maths was so much harder than the the normal A-level maths,
00:12:00
Speaker
It made the A level maths far, far easier. So if I, I personally believe if I just did the regular math, I would have found a harder than I actually, than did because i did further maths.
00:12:11
Speaker
And so when, when you made V1, what was like, did you have, like, what was the first kind of detector that you put in there?
Light Chaser's Capabilities and Challenges
00:12:20
Speaker
Oh, and the first detector was very simple because like I mentioned, um those like hidden mint functions, those were very, very dangerous. So I would just, the first detector I ever made was specifically for that. But unfortunately, with the timing of Lightchaser, when I actually completed it, I never actually got to use it for its original intended purpose. It kind of just met on the back burner and then coded when the announced bot race and said, hey, I have Lightchaser. mean, it wasn't called Lightchaser back then, but I already have something. Let me just use that and build a upon that. And then I quickly developed V2 and then V3 and now I'm at V4.
00:12:56
Speaker
And I mean, you you dominated with the bot races, right? I mean, i don't hear of any other bot out. there I mean, we all and we all heard the AI hype about all these AI auditors, but I mean, the bot race is for Code Arena. It's exclusively you that wins that, right?
00:13:12
Speaker
Oh, no, no, and not exclusively. I'd say like towards the but the tail end of the Coder winner bot races, that's when Lightchaser became very definitively the strongest bot.
00:13:25
Speaker
um So I'd say from late 2023 to that region, that was like the final...
00:13:36
Speaker
I'd say six or so months of the Cotarino boat races. ah think that's when Light Chesa became the strongest boat. But then unfortunately, Cotarino paused the boat races.
00:13:46
Speaker
So, yeah. Why? what What happened there? I didn't stay up on that. Why'd they pause it?
00:13:53
Speaker
um I don't 100% know for sure because I don't think they made a full official announcement on why they were paused. But i I have my own theories, ah which I'm not 100% sure if that was the reason. But I think at the time, they also got rid of analysis. So I think one of the main reasons was that they kind of wanted the primary focus to just be the high and medium pool. And obviously the bot races and the analysis...
00:14:21
Speaker
were taking away from the high medium pools. I think that was one of the core reasons as well. They wanted to put more focus on that. And I think another thing was that the way the boat races were going, I think the detectors became very, very, I'd say non-critical and gas heavy. So theque I think they kind of lost their direction in a weird way. I think that was primarily due to what the meta became to make the main focus.
00:14:47
Speaker
high and medium finding because during when bot races got paused so did the analysis reports and i believe the qa report part also got drastically reduced i mean they restructured that as well so i i think that was one of the main business reasons but perhaps another reason was that because the meta became very accuracy focused a lot of the meta essentially rewarded like having loads of non-critical and gas findings rather than the development of really cool high-medium findings. So I think that might have also played a role because as essentially what happened was that the penalty for essentially having a false positive in the high-medium detector became so great that you were almost better off just adding and a ton of
00:15:40
Speaker
some very very simple very very boring non-critical and gas finding so the the reports just became these really really big reports full of nc and gas finding so i think they began to lose their direction as well um i i did i did try to like uh advocate that uh perhaps we should do some like changes to the the system to alleviate that but uh Yeah, I think, typically speaking, that meta kind of just, once it it came about, it kind of just stayed that way.
00:16:17
Speaker
and i think like i a great example of this is uh let's say a low detector low detector was typically worth five points um but if your low detector had even one false positive the penalty could be up to 15 minus 15 points so instead of taking that risk of adding a new low detector you are better off adding another five natspec detectors which are borderline useless um but you know they're like that's what the meta became and i think that kind of accelerated end because they they began to lose their value over time and i really did kind of like try to like advocate for us like move away from that like one of the things i proposed was that we raise the bar off uh to get a grade because um i'm not sure if you know but um there was three grades that you could no four grades you can get there was the winner
00:17:12
Speaker
Grade A, Grade B, and Grade C. What would you say are like the limitations of of your bot? like why can't it find some like Why is it so hard to find a high critical for the bot?
00:17:26
Speaker
um Honestly, I would say now that is no longer a limitation for Lightchaser. I think now it's at a point where it can actually quite reliably find high-end criticals and high high critical and mediums as well.
00:17:41
Speaker
ah But that's that's kind of like a recent development. I think the advent of Lightchaser v4 really helped to get Lightchaser to that point. And the results that we've been getting privately in private engagements and contests as well, like we got a solo medium recently very recently, um is is looking pretty bright for ITSM now. And I think I'm confident that that limitation thanks to V4, has been mostly alleviated now.
00:18:13
Speaker
It's very rare for me to come across a vulnerability where I'm like, okay, um Light Chaser doesn't have the complexity ceiling to capture this.
00:18:23
Speaker
and And so you are strictly doing static and dynamic analysis. You're not using any LLMs with V4. That's right. um I personally believe that in the long run, static and dynamic analysis will...
00:18:39
Speaker
be superior to LLM-based detection. Why is it? I think primarily because of reliability and of the actual
Tool Reliability and Improvements
00:18:49
Speaker
findings themselves. The thing with AI detection is, that's I think there was a blog post about this. ah There was about a LLM finding a zero day. But when you actually read it, it said that something like ah out of the hundred times it ah it got asked to find a vulnerability in that piece of code, it only found it like seven times.
00:19:09
Speaker
And I think that's a major, major problem with LLMs. It just, it tends to hallucinate so much and its results are incredibly unreliable. Like even if you prime it really well, that like half the, or perhaps like 30 or 40% of the time, it will find the bug, but the other the times it won't find it it's it's kind of a guessing game and it tends to fall apart the bigger and more complex a code base gets.
00:19:34
Speaker
ah Whereas with static and dynamic analysis, you don't have that limitation. The limitation is the framework, the detector, and the detector developer, all things which I have full control over, which is why i am going all in on Lighthazer.
00:19:50
Speaker
and And obviously your code is not open source, but so I could just talk about Slither since I i used to use that quite a bit. And it would produce report in seconds.
00:20:02
Speaker
And a lot of the findings, you know, you'd look for anything that could be insightful, but the vast majority, it was it was to be ignored. If I see reentrancy, this, you know, a lot of that was all you look at it manually and you say, well, that this isn't really legit how does How does Lightchaser kind of do things differently?
00:20:25
Speaker
So I think what you're primarily talking about is false positives and false positive is something that in general, any detection tool struggles with quite a lot.
00:20:36
Speaker
And with Lightchaser, the way I'm trying to alleviate that is, let's say Lightchaser generates a generateive report. I will myself go through that report and tag any detectors that produce false positives and then essentially repair those detectors.
00:20:51
Speaker
And if you do that enough times, a detector gets fully matured and more or less stops producing false positives. So that is something that can be alleviated given enough time. But, uh,
00:21:03
Speaker
I definitely feel that that is a limitation of and detection tools in general. And how can how does your tool work? So say say if you have a cross-chain protocol and you you run it through, that and I don't know how you load the code base in your tool or anything, but you run it through the tool and say that that requires context of Solidity talking to Rust and then talking to another Solidity contract.
00:21:34
Speaker
Can it can it do that or no? Well, it can definitely abstract, like if you provide the, essentially the solidity code from both ends, the source chain and the destination chain, then it can detect some stuff.
00:21:48
Speaker
But actual, like the Rust stuff, it can't really do that. Is that like ah a V5? Is that a future version where you could have the whole flow kind of plugged in if it's in, you know, for the backend layer in different languages?
00:22:04
Speaker
Absolutely. i mean, I think a lot of the work that would required for that is creating a Rust parser as well. it's definitely something that's feasible.
00:22:16
Speaker
No, that would be very cool, man. And ah I mean, everyone's talking about LLM, this yeah ai all AI auditing and and all the all the the bug fighters and everyone's, they're you're kind of halfway spooked. They're like, okay, does somebody else have an edge that I don't have? And if they do, I want that edge. And then people try to use it and they get false positives, they get hallucinations.
00:22:41
Speaker
And you really, you I think this is a ah time where you can, you could develop a ah secret edge, like, like a lot of things in crypto, whether it's trading or MBV or whatever, you can have your own edge and just not say shit and no one's going to know about it.
00:22:56
Speaker
And, you know, maybe, maybe Lightchaser is one of those edges. And I kind of like that you're keeping it closed source because why not? It's your tool. You're able to, to make revenue off it and,
00:23:08
Speaker
and you're building it by yourself, kind of doing it your own way without saying, I'm going copy Slither or or anything like that. But like, like, what are you going to use this tool for outside of doing, the audit competitions? Are you going to also do bug bounties as well, or have you already done those? 100%. Um, so, uh, I think like the general pipeline for light is the worst that, um,
00:23:35
Speaker
first become the best tool in general, like amongst other tools. And I think Light Chaser has more or less accomplished that. And then the second mission statement after that was get to a point where Light Chaser can very reliably perform very well in contests. So myself and my awesome teammate, Nate, so we use Light Chaser to handle the detection and the write-ups and everything is handled by Nate.
00:23:59
Speaker
And we've begun to get some feedback very very promising results by doing that and the third state mission statement after that is to use our bug bounty so that's part of the pipeline and ah i i can't share it too much because some of the stuff is like and nda stuff but uh we 100% have found valid highs um on live stuff.
00:24:24
Speaker
um But that's pretty much all I can share in that regard. I verified my idea. Okay. Very, very scary stuff for us, man. what is our Is our career, are they in danger now because Light Chaser's on the prowl?
00:24:42
Speaker
So Light Chaser is definitely going to get better. And but I think one of the, with light shards in general is that I'm not necessarily trying to maximize it for monetary reasons, like monetary stuff, like in the sense of right now with light ah with as powerful as Lightchaser is and it has become, that I could definitely focus a lot more on outreach, get more clients.
00:25:07
Speaker
But because I'm so focused on these mission statements, I actually try to limit how much time I spend on client work so I could just focus on building itself.
00:25:18
Speaker
But like even when once Lightchaser gets to like the third mission statement, I don't believe that it will completely get rid of bug bounties at all.
Future of Light Chaser and Automation
00:25:27
Speaker
um It would definitely be able to compete, in my opinion, obviously I'm very biased and favourable, so I definitely think it would be able to compete alongside um bounty hunters, but I don't think it would replace them.
00:25:38
Speaker
like yeah This makes me never want to disclose any bug, any bug fix anywhere so you can't train your detectors on it. ah Fuck. Okay. No one's going to come public with any more write-ups after this.
00:25:54
Speaker
i had I had a question from a guy, an ex-Rami. He said, you know this whole light chaser thing says, did you have manual review experience? Oh, yeah. um So I'd say in 2022, did manual. Because obviously, like I mentioned, I got decently good at reading contracts because of the whole PancakeSwap stuff.
00:26:16
Speaker
So i was like, okay. um So I had that skill. That skill was generally quite rare at the time. um There was very few people who knew how to read contracts and find stuff in contracts. And um obviously, I started reading a lot more audit reports. But at the time, there was...
00:26:32
Speaker
very few audit reports to learn from. So I actually like learned a lot from Celtic reports at the time because like there were so many of them at the time. And back then, there wasn't any like stuff like solo date or anything like that.
00:26:47
Speaker
So I trained myself right up on that. And yeah, I did do manual audits for a while. um Then I stopped to focus on my studies. And then pretty much right after university, I started again.
00:27:01
Speaker
But then my full nine to five job began and then also I just, I had to leave it to focus on that. And then obviously now i'm all in. So like I've left that and I'm back in Web3 security, but as an automator.
00:27:18
Speaker
So, all right. So there was this tweet by Dacian and you're, I think you replied to it You're, you're aware of Dacian. that and He was talking about AI auditors. Yeah. and he, he said these three things. so he said, number one, once, and he was saying, you know, when, when we should be considering them seriously, number one, regularly finding 89% of what a junior auditor can find.
00:27:41
Speaker
You think light chaser is already at that, right? Yeah. I'm fairly confident at that. that i'm but I'm kind of basing it off like contest results and how it's performed as a known issues list.
00:27:51
Speaker
I'd say that for the contest platforms that Lightchaser does produce known issues list, I think it does actually find a lot of the stuff that junior auditors are submitting.
00:28:02
Speaker
um Yeah, like if you actually just go on to either Cantina or ah the Cypher Discord server and just tap down Lightchairs, you'll see a lot of people, but oh no, we found all my bugs and stuff like that. So I think it's either close or is ready at that point.
00:28:18
Speaker
um Yeah, that's the point one. Shit. Okay. All right new guys. Listen. Okay, you're fucked. Number two. I mean, no, to be honest, I mean, this it's this is progress, okay? Whether you like it or not, the train's moving the station, so you need to just level up your game.
00:28:37
Speaker
And the more complex bugs you could find that the computer can't find, is all the better use your human brain use your intuition and don't release your bug reports all right um number two is find the same or ideally more as static analyzers but with drastically lower false positive rates where would you say it is on that So obviously with my mission statements being primarily to be able to compete very, very well on the contest and then in the future bug bounties, my primary focus has been on building as many detectors as I can that are that are high quality. Like I don't just spam detectors and kind of coping for the best, but obviously with that being my mission statement, of course, Lighthizer is going to get false positives. And because my...
00:29:26
Speaker
primary focus is detector innovation um that can obviously result in more false positives but i do believe like i mentioned before the process to actually mature the detectors isn't super complex you just keep on obviously using light chase and over time you'll know okay so these detectors performed uh ah produced false positives so and just repair them. And you do that enough times and they they fully mature and they don't produce false positives anymore.
00:29:56
Speaker
So that is something that will 100% get alleviated with time, but it's not my primary focus right now. Right now, my primary focus is pure detection power. Okay.
00:30:07
Speaker
All right. and And the last one he said, which was the Holy Grail. Regularly find high impact findings that no or few other auditors can find in public contest or bug bounties. And you stated that...
00:30:20
Speaker
I think you're there, right? um I wouldn't say fully because although we do consistently get... We're there. getting there 100%, but we did actually get a solo finding a couple of weeks ago.
00:30:33
Speaker
so And obviously we found some some stuff which can't talk too much about. but So there is some like evidence that we are approaching that as well. But once...
00:30:46
Speaker
The point where I can say that yes, we are consistently getting solar high mediums, um that I can't say for sure, but I don't think it's too far away. At least I hope it's not too far away.
00:30:58
Speaker
Could you walk me through like, all right, so say, so you're, you're, you say your focus is adding detectors, honing detectors. When you want to add a new one, like say you see a new, I guess you stay up to date on audit reports and bug bounty findings and you look at something like, what do you, what do you do? You see something novel and you think, okay, I i don't have a detector for this.
00:31:19
Speaker
Can you walk us through it without sharing any trade secrets? Yeah, absolutely. um So Lightchaser has gone to a point where the complexity ceiling of the framework is capable of finding most bugs. like It's very rare for me to come across a bug report or a contest finding that Lightchaser doesn't have the capabilities of detecting this.
00:31:45
Speaker
Rather, what the limitation tends to be is, am I able to abstract that issue so it's applicable to other code bases? Because some findings are so protocol specific that I find it very difficult to abstract those to other protocols.
00:32:01
Speaker
So I think that comes that ability is very much experience based. I've been building detectors for years now, so I'm very good at spotting what detectors are capable of being applied to bases.
00:32:15
Speaker
other repos. But yeah, I think you you would definitely build that intuition. So that's like the first thing I look at. Can I abstract this? And let's say I found a finding that, yes, I can abstract that. And what you will find is that the bug bounty report or the order finding report on Coderino, wherever it may be, would be very, very long and very wordy.
00:32:40
Speaker
like it'd be so much detail. So as it automated, it's kind of hard to kind of figure out, okay, so how am i supposed to automate this? But a trade secret, which will hopefully be helpful to automators is, the trick is, is to just course go all the way down to the bottom and look at the mitigation.
00:32:58
Speaker
Because if you actually look at the mitigation, you can see exactly parts of the code caused that vulnerability in the first place, and then you can work your yourself you work your way backwards from there.
00:33:11
Speaker
um So that essentially what I'll do is um I'll look at that. Oh, okay. So these are the stuff to look out for. And yes, this this is something I can abstract. Then I would essentially, i have like an in tablet and I just kind of write down all the things that need to go wrong for this vulnerability to occur.
00:33:30
Speaker
And it's usually about anything between one to five things that need to go wrong. And sometimes more, like I have some really complex detects. Like I had a detector for, my one thousands detector I'll explain after this.
00:33:44
Speaker
But yeah, so there's usually one to five things that cause that vulnerability to happen in the first place. And then I create a separate detector for each one of those steps. And I just essentially daisy chain them all together um to look for, oh, okay, so that went wrong. Okay, then has this also gone wrong? Has that also gone wrong?
00:34:00
Speaker
And when everything goes wrong together, then you have a finding. um But yeah. So you have 1,000 detectors so far.
00:34:16
Speaker
And everything has been hand-coded, organically grass-fed coded by yourself. Oh, yes. I never use LLMs for coding. No vibe coding. Old school. I love it, man. You're now old school. as Yeah, 100%. I think it's better, man, because if you turn it over to to AI, you'll just look at it again. You won't understand it, and something will be fucked you'll have no idea what it was.
00:34:41
Speaker
Yeah, 100%. I personally, i as so useful as LLM coding may be, i don't think it will be able to replicate two avenues of coding, and that is very, very complex complex coding or very, very creative program but coding.
00:34:57
Speaker
So if you are building a project that's one of one or both of these things, then manual coding will be the way to go, in my opinion. who What do you think? Do you know, are you aware of Glider?
00:35:11
Speaker
Yes, I know Glider. So there they have, I guess you you frame it as like a query framework. I mean, it's it's a bit different, right?
00:35:22
Speaker
than what you're doing. Yeah, yeah, yeah. It's it's quite different. um But i think it is, the end of the day, a detection framework. So I would still consider it a bot, kind of.
00:35:37
Speaker
it like Is there a way, I like the idea behind it is, I mean, take all the contracts that are out there and and have this very specific kind of detection query framework where you can really hone in on on popular patterns or whatever you're looking for.
00:35:52
Speaker
do you Are you looking at like having LightChaser have that functionality as well to just scan all the all the deployed contracts out there with your all your detectors and find a way to kind of you know, process that that massive data into something useful?
00:36:09
Speaker
um That's not my pipeline right now. um I think my primary focus is to get good at contests, get good at bug bounties, but to be to actually scan all deployed code on EVM, that's not really something on my radar. Like instead of doing that, I'd more likely just run against all bug bounty programs instead.
00:36:32
Speaker
Is this just you? So in terms of the Light Chaser's development and programming, then yes, I'm one-man band for that. Although for Code Arena or Sherlock, I do compete with my team awesome teammate Nate, and he does like the write-ups, the POC, the remediations, but the detection is handled by Light Chaser.
00:36:53
Speaker
Okay. I highly recommend it. And I think that's a that's going to be a cool my team setup going forward. Yeah. You know, I, I'm looking at the future where you have a great bug bounty hunter or auditor teaming up with a tool like yours and yeah you work maybe hand in hand and maybe that, maybe that's the path going forward.
00:37:17
Speaker
Yeah, 100%. I mean, this is actually something that we've been offering privately. I mean, we just started offering it and we've been getting some very good results and great feedback as well.
00:37:29
Speaker
And just like I do have bit of a corny code name for it. I call it like like internally the LightShare and Mecha program because it sounds cool. But... Yeah, i like, ah um like, like, like the mech and like the hybrid auditors like the mega pilot.
00:37:46
Speaker
And that's like the setup that we have. And we used to compete on these contest platforms. And we've been getting promising results. So I i do feel that there's a lot of scope to grow that idea. So that is also something I'm very much looking forward to you ah Right now, it's just myself and Nate doing that approach.
00:38:06
Speaker
um But going like in the future, I might consider bringing you some more people in to do the hybrid auditing stuff. But although when that happens, Nate will still have like first pick on the the jobs and content, because obviously he was here from the start.
00:38:21
Speaker
But yeah. It's interesting. I just got an offer ah yesterday from a guy developing, you know, like ah an AI auditing thing. And he says it's posting some strong results and there's not that many false positives and he needs basically the same kind of setup to have some guy kind of review and and POC and make sure whatever legit. And there's like a revenue share, but you know, I have to look at these things cause I know it's progressing and and I don't know the code behind it. So I have to check it out.
00:38:54
Speaker
I mean, I do think these hybrid teams, they start coping up more and more as time goes on. Yeah, yeah. And what about, I mean, it's not a huge ecosystem, but what about Viper? are Is it worth putting the time into doing detectors for that?
00:39:09
Speaker
Yeah, so that that's something that I really had to consider is, do I support other languages with Viachaser? But the ah the reason why I ultimately decided against it was because of the time investment.
00:39:22
Speaker
And because the amount of time it would take for me, because I typically speaking, I pretty much always code my stuff from scratch. Like Lightchaser is 100%. Every single line of code is built from scratch.
00:39:33
Speaker
Like I don't even use any um like pip install libraries or anything like that. I think I import JSON and I used to use NumPy, but not anymore. But yeah, so like obviously with that kind of set up, um I would have to create a parser for these new languages and parsers take a tremendously tremendous amount of time to make and an even longer time to fully mature.
00:39:56
Speaker
Like my livechaser parser was producing parsing errors for a few years until I fully ironed out all the bugs and now it's very reliable and very stable.
00:40:08
Speaker
For me to do that for another language will take a considerable amount of time, but who knows in the future, um i I do have some plans for perhaps having move support or Rust support.
00:40:21
Speaker
But yeah, that's far in the future. Very cool, man. but and And so for the bug hunters out there there that are shaking in their boots thinking you're going to come take their paydays, what would you say are some limitations that you don't see?
00:40:38
Speaker
Maybe some insurmountable limitations for yeah maybe the next year or two. give Give the guys some hope. Honestly, in my honest opinion, I don't think manual auditing is going anywhere.
00:40:54
Speaker
And I don't just think that would be for the next year or two. I think that would be at least for the next five years. So if you are a manual auditor listening to this, my like i am like, obviously not not to sound too arrogant, but like but on the cutting edge and the automations out of things. And I don't believe manual auditing is going anywhere anytime soon. And that applies for bug bounty hunting and contests and also solo and team orders as well.
00:41:20
Speaker
I don't think that's going anywhere. Rather, the way I feel automation is going to be, it's going to be just another stage in the overall process. And even on my the Lightchaser website, i I mentioned this quite clearly that first you you get like a Lightchaser report, perhaps a hybrid report, and then you get teamop team manual audit, then a contest, then a bug bounty.
00:41:42
Speaker
So I just don't believe that manual ordering is going anywhere time soon. And one of the main reasons why is that a lot of these very important bugs that need to be found are very protocol specific.
00:41:57
Speaker
And these are things that, in my opinion, manual auditors will always be the best at. Like, even if lightchase again becomes really, really good, or these LLMs actually become worth the hype, which I don't think they will personally, but um um i so I still believe that those very protocol-specific or complex bugs will always be there.
00:42:23
Speaker
So if I'm reading this correct, your recommendation is is what I'm thinking, is that if you're trying to do just grab a protocol and and look for some bugs, those those kind of bugs might be already tackled. But if you're ah security researcher looking at a protocol and you're willing to get deep into the weeds on that protocol,
00:42:45
Speaker
you're going to be able to outperform these automated tools because you'll have that context tailored for that specific protocol because you put the time into it. Absolutely. i mean, I think like the main impact automated tools will have is it might like, do you know when an auditor just kind of goes through a report and they just submit like where do you see 20 findings and stuff like that or,
00:43:10
Speaker
Like these kinds of like findings that applicable to those the different protocols. I think those might fade away, but the very protocol specific stuff, um, will be here to stay for a very long time, I believe.
Impact of LLMs on Auditing Jobs
00:43:25
Speaker
ah Okay. And, and when your comment on the LLMs, why do you think they're not going to kind of reach, reach the hype expectation that everyone's putting out there? So I don't necessarily think that LLMs are aren like aren't useful. i just feel like they're overhyped maybe two or three times what they actually can do.
00:43:48
Speaker
And the reason why I don't think that they'll be able to like get better over time, I do think they'll get better over time, but I think the hype will always be like a couple of notches above where it's actually at.
00:44:01
Speaker
And but I think only time will tell. i think this is one of those things that my age well, my age poorly, but my personal gut feeling is that LLMs won't take away manual auditors jobs.
00:44:16
Speaker
Hmm. Yeah, I hope not. every Every time I plug something into it, I'm like, yeah, it's it still doesn't know what the fuck it's doing. But, you know, it's it's like i'm I'm not smart enough to understand. I looked at I told Nellon to explain to me how it works. And it's so fucking complicated. Like, I know the general idea behind it.
00:44:37
Speaker
But as far as like deep in the weeds, I don't get it. And I looked at um some write up from the Claude team, the Anthropoc team, and they said, you know, they had Claude run like a ah shop and it was doing things and and then it was hallucinating.
00:44:54
Speaker
And the creators of it said in their blog post, they said, we don't really know why it decided it was a real person and was coming to visit us and shit like that. Where I'm like, if the guys making it don't really understand why it's doing what it's doing.
00:45:09
Speaker
um Well, certainly don't. But I just think it's like, define the limitations of it. um i'm I'm just not the smartest guy in the room when it comes to that. But when I hear guys like yourself, you know, other guys on Twitter, like,
00:45:25
Speaker
Bernhard Mueller, Joseph Feist, when I hear them talk about like something fundamental is missing with, you know in quotes, the LLM-based reasoning for it to be a real ah real kind of player in this in this field, I respect those opinions and and and and I listen to them. Even though like everyone tries to hype it up, in the back of my head, I'm like, no, no, no. it's We still have a purpose here.
00:45:50
Speaker
No, 100%. I mean, I think the there are two main reasons. This is my personal opinion that people hype up AIs. One is, um, they don't want to be left behind. They don't want to be that one person who like didn't take it seriously. And then it turns out, Hey, I was supposed to hype and then they get replaced or something along those lines. And the second is, um, they kind of want to hype up wherever they're building.
00:46:12
Speaker
Oh, we incorporate AI and then they could go to venture capitalists and raise 10 or 20 million. Um, so, but, uh, Yeah, like, i I do get quite upset when I see manual auditors, like, get worried about it because I personally, i just don't see it happening anytime soon. and And I think that disheartening um isn't isn't great for the space because i feel like it's going to put a lot of people off from getting into manual auditing, which I don't, personally don't want.
00:46:42
Speaker
But, yeah, only time I tell, I'd say. um say Yeah, some people don't have a good view of of the bots and and all these tools. Like the guys, I had a new guy on the podcast I just put out. And you know guys like that, he was in Pakistan. And his his monthly all-in expenses, he says, for a good life over there, 700 bucks. Food, everything, rent, everything. ye and so trying to get into auditing or bug bounty hunting,
00:47:11
Speaker
you know, guys would submit these low findings to get a little, little cash in their pocket. And now that opportunity is basically gone. And I think the bar will continue to get raised, which is good for overall security. But I mean, you're not going to satisfy everyone. I think just that's just the nature of the beast.
00:47:28
Speaker
Yeah, no, 100%. But like on the flip side of that, um, I would personally consider Web3 security right now a blue ocean market in the sense that feel there's a lot of opportunities and the money is good and the work is interesting and fun.
00:47:43
Speaker
But when there isn't a barrier of entry, blue ocean markets very quickly become red ocean markets. So, in my personal opinion, there being some sort of barrier entry in the long run will be better not only for current auditors, manual auditors, but also future manual auditors who want to join because the gates to get in that will be open for a longer period of time because if the bar was so low that anyone can like start competing and get like 20 or 30 high mediums on their profile and then start offering solo audits. The market would essentially get so saturated that it would drive the prices down and that won't be great for anyone. And also the quality of the audits themselves, in my opinion, might be subpar because like if you do have a barrier of entry, like with Lightchairs, for example, it kind of forces auditors to become above that bar.
00:48:41
Speaker
And that will overall increase the quality and in my opinion, it will increase the payout for the auditors who are able to stick with it and actually get to the level where their service is superior to what automation can provide.
00:48:55
Speaker
Agreed, agreed. And for guys out there, if you're seeing a field becoming more and more competitive, which I think solidity, I think the auditing field solidities most popular smart contract language.
00:49:09
Speaker
When you see it becoming more and more competitive, which it is by every single day, go to a place where it's not as competitive, that less people are interested in. That could be Viper, that could be Rust, that could be Move, that could be some Moon Math, that could be whatever, just pick something. It may be more difficult. You may have ah a hard time, a big vertical ascent to try to get up to speed on it, but it doesn't mean that time's not worth it.
00:49:35
Speaker
And then suddenly you're in the same position that others were in with solidity, you know, a couple of years ago. And eventually, you know, the crowd moves on to that topic. But if you stay ahead of the game, I think there's opportunities for you out there.
Future Directions in Automated Tools
00:49:50
Speaker
100%. And yeah I completely agree with that. And that also somewhat even applies to people who want to get to automation. um You don't necessarily have to build a solidity automation tool.
00:50:02
Speaker
you can there's a huge, huge need for equivalence in other programming languages like Rust, Move, Cairo. So there's also a lot of opportunity there. And Web3 in general is um full of opportunities like ah Two of my closest friends, they're twins. um They're space architects. and They like design moon bases and cool stuff like that.
00:50:26
Speaker
And they are looking to get into Web3 as well. And ah i think with the avenue that they're going to try and get into is Web3 architecture for, do you know, like, I forgot what it's called, like, meta?
00:50:42
Speaker
As far as... You have those, like, virtual ah worlds and stuff. um ah so I forgot the metaverse, that's the word. And like both like design buildings for that and stuff. And there's actually a growing field there. So like Web2 in general has so many opportunities. So I think- I agree, man. it's It's massive. I wouldn't have started this podcast if I thought it was a growing market or or a shrinking market. I think it's yeah i think it's just beginning.
00:51:07
Speaker
I think, look at Web2 security and then I look at the future of crypto and it's like, dude, we haven't even started yet. So I'm a, I'm a optimist, man. There's always opportunity to be had. you just need to seize the day.
00:51:22
Speaker
Um, I wanted to ask you something else. You, you had ah tweet out there and I'm into my fitness, right? You had a tweet regretting not switching to free weights for weight training earlier.
00:51:33
Speaker
I've been getting PRS every week for months now. What happened? Oh, no, like this, this is like a really like tragic story. Like essentially i I've been obviously going to the gym for years now, ah but I kind of had to go to like a smaller gym that was close to my home because i didn't really have the time to commute to like the well-equipped gyms.
00:51:55
Speaker
But i was like, okay, it's fine. um They have plenty of machines, so that should be good enough. But I was at that gym for two years, going like at least three or four times a week and train at least I thought I trained hard, but none of my lifts improved.
00:52:12
Speaker
I pretty much stayed at the exact same strength for two years straight. And I was like, okay, ah let me leave this. Let me just try a free weight. so But obviously I didn't want to do the whole commute. So I just bought these adjustable dumbbells, power blocks. And as soon as I started those, every single week my lifts were going up and it was crazy. And it's been it's been like that for months now.
00:52:35
Speaker
And I really like wish I started that earlier. I feel like ah I lost a lot of time. I love to hear it, man. Free weights are the secret.
00:52:46
Speaker
it's yeah It's just, ah you're you're engaging every single muscle with that range of motion, whereas a machine very much limit you. So kettlebells, free weights, all that stuff is essential to becoming a hugely based giga-checked JAD with high testosterone, getting all the babes and finding all the bugs.
00:53:08
Speaker
Mr. Chase the Light, thank you very much for coming on. This is really cool, man. We will see everybody next time on the blockchain.