Transcript
Speaker: Hey everybody, welcome to Grounded Truth, a podcast where we gather some of the world's most influential data scientists, machine learning practitioners, and innovation leaders for conversations on the most relevant topics in AI today.
Speaker: I'm your host, John Singleton, co-founder and head of success here at Watchful, the machine teaching platform for data-centric AI.
Speaker: You can try Watchful for free today at www.watchful.io.
Speaker: And please, if you like the content and podcast, like, subscribe, and follow on Apple, Spotify, YouTube, or anywhere that you can find podcasts today.
Speaker: Joining me today is Emmanuel Turlay, CEO and founder at Symatic, the open source continuous machine learning platform that has recently launched Airtrain.ai, the no code batch compute platform for your LLM evaluation and tuning workloads.
Speaker: Prior to Symatic, Emmanuel is at Cruise, Instacart and world famous research organization CERN.
Speaker: We'll provide links to both Symatic and Airtrain.ai in the description below.
Speaker: Also joining me is my CEO and co-founder here at Watchful, Cheyenne Mahonte.
Speaker: How's it going, guys?
Speaker: Great.
Speaker: Great.
Speaker: Thanks for having me.
Speaker: Yeah, great to be here again.
Speaker: Really enjoyed that you were able to join us, Emmanuel.
Speaker: Before we dive into kind of our topic today, which is going to be RAG or Retrieval Augmented Generation versus fine tuning in LLM workloads, I'd love to give a chance for you to plug Airtrain.
Speaker: Tell us a little bit about it.
Speaker: Who listening should reach out and sign up for Airtrain or download today?
Speaker: Yeah, for sure.
Speaker: So the problem we're trying to solve with Airtrain, the initial problem we're trying to solve is really evaluation, like comparing different options.
Speaker: You know, people are bombarded every day with new models, new fine tunes, open source, closed source ones, and so on.
Speaker: And so figuring out which model is the best for a specific use case and a specific data set is actually quite challenging because, you know, you have metrics and benchmarks, but they're just high level indicators of quality that don't necessarily tell you how particular models work for your use case.
Speaker: So we built Airtrain to make it super easy for people to compare the different open source models and also GPT on their own application and their own data set.
Speaker: So it's actually free to use the evaluation product.
Speaker: So people can sign up at Airtrain.ai and we'll let them in as soon as we have our room.
Speaker: Awesome.
Speaker: And again, we'll provide a link to that.
Speaker: It sounds really interesting.
Speaker: And I think that's a great way.
Speaker: It's a great segue into framing this conversation of rag verse fine tuning.
Speaker: And before we kind of dive into that, I'd like to just like, let's define some terms here.
Speaker: What is rag and what is fine tuning?
Speaker: RAG is a Retrieval Augmented Generation.
Speaker: The idea is that when you use models, they're only trained until a certain time.
Speaker: I think the cutoff for GPT-4, I think it was maybe June last year or something like that, the latest release of it.
Speaker: And so the models don't know everything.
Speaker: They don't know everything about like recent events.
Speaker: They don't know any private information.
Speaker: They don't know anything about your company's private documents and so on.
Speaker: So the way to get those models to generate outputs that are aware of those things is to put this content inside the prompt, inside the context window.
Speaker: So you put the context in that context window and you ask the model to respond to the user's question.
Speaker: And the user can source the information from that
Speaker: that retrieves information and respond to the user.
Speaker: And so the way to pass this information, the relevant information to the model, is to do this retrieval.
Speaker: So finding the most relevant pieces of information, so whether it's using Bing, Google, or something like public source, or some private source like Wiki or our internal knowledge base, and feed this into the model's context window and ask the model to generate a response from that.
Speaker: So that obviously includes a component of finding those documents, so like some sort of
Speaker: similarity search, semantic search, regular search, where you can highlight the top five, top ten documents that are relevant for the question and then extract the contents of the document and passing it to the context window and then responding to the user.
Speaker: So that's RAG.
Speaker: It's a way to make models online, essentially.
Speaker: by knowing everything that's going on at any time.
Speaker: Fine-tuning is a different technique that essentially keeps training the model for a few more epochs on more data.
Speaker: So those big models are being trained at a very large scale by those providers,
Speaker: you know, tons of GPUs, you know, millions and billions of tokens.
Speaker: But they don't, so they're very generic.
Speaker: They know everything about a lot of different topics.
Speaker: But if you want them to act a certain way, for example, to speak like a lawyer or speak like Harry Potter or have a certain tone of voice or a certain format in which they're responding, you need to fine tune the models.
Speaker: For example, this is how models are able to chat, to receive questions from the user and respond because they have been fine tuned
Speaker: for the chat format.
Speaker: So fine tuning is essentially just training the model a little more.
Speaker: There's a few techniques out there to make fine tuning a lot cheaper and more practical, like LoRa, for example, all those PEF techniques.
Speaker: But so it's a different goal than RAG.
Speaker: It's not so much knowledge augmentation.
Speaker: It's more the form of the outputs and the tone of the outputs.
Speaker: And I think that's a very important distinction because one of the biggest
Speaker: misconceptions about fine tuning just in the general public, not even just general public, amongst practitioners is that fine tuning is how you inject new knowledge.
Speaker: When I think you put it best, it's really how do I get the output in the shape that I demand or desire for my particular use case?
Speaker: And I think that's a very subtle but important distinction that this is not putting new data into the model, but simply massaging the output that you expect or need for your particular use case.
Speaker: Yeah, that's right.
Speaker: I mean, you may teach the model new things, but the model is still very likely to hallucinate or confuse the fine-tuning data with the original pre-trained data.
Speaker: So if you want to really have high accuracy on the content, RAG is really the way to go because the content is right there in a context window in the ePoint prompt.
Speaker: and the model just has to extract data from it.
Speaker: And you can be very specific in your prompt, only use information that is specified here in this context window.
Speaker: So that's much more reliable way to produce a factual answer on some specific piece of information.
Speaker: Fine tuning is really more about form.
Speaker: Yep.
Speaker: Makes perfect sense.
Speaker: And so is this just a one size fits all decision?
Speaker: Just simple binary rag is obviously better than fine tuning or is it one or the other?
Speaker: Is it used both in most cases?
Speaker: You know, kind of give me an understanding of like, how do I evaluate the decision to invest more time into rag or more time?
Speaker: Or should I fine tune?
Speaker: Yeah, so I think in the long run, everybody will end up doing both.
Speaker: You cannot afford to have a model that is ignorant of recent information.
Speaker: For example, GPT-4 just included retrieval in their latest update since Dev Day a couple of weeks ago.
Speaker: And so now they fetch data with Bing, because obviously, if you were looking up anything more recent than the training cutoff date, then you wouldn't have any relevant information.
Speaker: So I think RAG is going to be necessary in almost any system, unless your system does not need to be aware of current information.
Speaker: So for example, if you just want to have a product that, for example, produces haiku in the voice of Harry Potter or something like that, you don't really necessarily need to fetch information.
Speaker: So maybe you don't need RAG in that use case.
Speaker: But for most enterprise applications, I see a case for both fine-tuning and RAG.
Speaker: Obviously, those are different projects with different types of compute profile.
Speaker: For example, RAG does not need any offline compute.
Speaker: You don't need to train the model offline.
Speaker: You need to do some ingestion, definitely, of your knowledge base into a vector database, typically.
Speaker: So that is some offline compute, but it's nowhere near as intensive as actual fine-tuning, where you need GPUs and very high-quality data and so on.
Speaker: So I think there's a progress for people to go from just the base model to, OK, what do I do next?
Speaker: The next step to improve the results is to do some prompt engineering, as you know.
Speaker: So you just try to get the best possible prompt, and that gets you some of the way there.
Speaker: After that, OK, now we need to make sure that the accuracy is higher, there's less hallucination,
Speaker: that the results are more grounded in factual documents.
Speaker: And so that's where I'm going to implement RAG.
Speaker: And so that's great.
Speaker: Suddenly your responses are a lot more relevant to your domain expertise.
Speaker: And then the last step is really to fine tune to get really the exact shape that you want.
Speaker: I think fine tune requires a little bit more expertise because you need to know how a training job works as opposed to simply just, you know, ingesting data into a database and then retrieving it, passing it to the context window.
Speaker: So I think that's probably the last thing that people do.
Speaker: The types of engineers that can do fine tuning is a different set of skills than regular RAG, which is more like a regular data infrastructure.
Speaker: So I think it's a matter of where are you in your journey towards a fully complete enterprise grade product.
Speaker: But eventually everybody will end up doing both.
Speaker: No, that makes perfect sense.
Speaker: And you mentioned going like as you're going through the process of prompt engineering and rag running into those walls.
Speaker: Could you tell me a little bit about how you're finding those walls?
Speaker: Like what do those mean?
Speaker: Are there metrics that are driving that kind of decision?
Speaker: When do I know that I now need to take the time, expense and effort of fine tuning?
Speaker: Or how do I get to that point?
Speaker: For sure, yeah.
Speaker: So LLM evaluation is actually still an open problem.
Speaker: If you look at other areas of machine learning, like old school regressions or decision trees, there were some easy metrics to figure out how good your model is.
Speaker: You can use root mean square error.
Speaker: You can use some precision recall, all those metrics.
Speaker: And in computer vision, you can use things like intersection over union, all those metrics.
Speaker: In LM space, because the output is so unstructured, it's just free text, it is really hard to evaluate the quality of an output.
Speaker: So there's a bunch of metrics out there, like blue and rouge, that are pretty popular.
Speaker: But they only apply to very specific use cases, like summarization, translation,
Speaker: But that's about it.
Speaker: And then you have benchmarks that will tell you how these models perform on things like high school math or international politics or very specific domains.
Speaker: But again, those are just indicators of how a model performs in one dimension.
Speaker: So to figure out how your model performs on your use case, there's not really any one size fits all for that.
Speaker: So usually what people do, the first thing that people do is what's called a vibe check and is exactly what it is.
Speaker: You basically prompt the model with like a half dozen prompts like, OK, well, it seems like it's fine.
Speaker: It seems like it's better than the other one.
Speaker: Obviously not a scientific decision, not rigorous.
Speaker: And so the way that people are doing it now is that they're building an eval data set.
Speaker: So it starts with like a few dozen prompts.
Speaker: And over time, every time you observe a failure in a model, then you use this prompt as a test case, just like you do in regular test-driven development.
Speaker: As a software engineer, every time you find a bug, you write a test case for that bug and then you fix the bug and then you keep that test case forever in your test suite.
Speaker: So it's the same thing for an eval data set.
Speaker: It's very much what we used to do at Cruise also, obviously not the language space, but every time there was a failure on the road, we would extract this as a test case in our test suite.
Speaker: So you build an eval data set.
Speaker: Ideally, you have also labels, so like ground truth, like what is a good answer for this prompt.
Speaker: And then once you have that, you're equipped for a real, like solid evaluation.
Speaker: So the most modern technique right now to do LLM evaluation is what is called LLM-assisted evaluation.
Speaker: So the kind of the big secret here is to use another LLM, typically a more powerful LLM, to evaluate the responses of the models you're trying to evaluate.
Speaker: So let's say you're trying to compare LAMA 7B with Mistral 7B, for example, like two fairly small-ish models in the grand scheme of things.
Speaker: And you want to see which one performs best at a particular task.
Speaker: So you're going to prompt them both on your entire eval data set.
Speaker: And then you're going to ask another LLM to evaluate those answers on specific criteria.
Speaker: So the criteria, the good thing about using this technique is that you can...
Speaker: you can target criteria that are abstract, not necessarily like measurable, quantifiable scientific criteria.
Speaker: So it could be things like groundedness or creativity or politeness or sentiment of the outputs.
Speaker: Because the LLM is able to score those things on
Speaker: on a gauge, for example, you can say on a scale of one to five, how creative, how playful is this output?
Speaker: And then the scoring model will be able to tell you, well, you know, LAMMA 7B or Mistral 7B is better at that.
Speaker: So obviously it takes more time because you need to build your evaluator set, you need to learn how to build an evaluator model, but this is really the way to go now.
Speaker: And so going back to your question,
Speaker: The vibe check is the moment where you're like, OK, my vibe check is very limited.
Speaker: I need to go further than that.
Speaker: So I need to build some more evaluation harness to really evaluate that.
Speaker: And so when do you decide to go from prompt engineering to Rack, for example?
Speaker: Well, as soon as you need your data grounded in more recent and factual documents.
Speaker: So if you're trying to get those models to generate data, to generate responses based on your internal knowledge base,
Speaker: or some private customer information that you don't want to leak out into a training job, then this is probably when you want to work towards rag.
Speaker: That's the point.
Speaker: And then going to fine tuning is really when the shape of the output is not satisfactory to you.
Speaker: When you want to make sure that your model always responds with the same format, the same tone, this is when you want to go towards fine tuning.
Speaker: That's probably your last step.
Speaker: Yeah, it's so interesting for those listening with maybe more traditional software development backgrounds or experiences, this idea of a vibe check, this completely very humanistic kind of like, well, we're all smart people that know approximately about this use case and what an output should look like.
Speaker: Does it feel right?
Speaker: I think we're very clearly lacking in kind of more measured and engineering based evaluation approaches.
Speaker: Yeah.
Speaker: It really shows that I think that we're still very, very early days in all of this, just in terms of tooling, how people are thinking about this problem.
Speaker: And then even, like you said, having a little bit more constructive or guided or scientific ways of other than, well, the vibe check failed.
Speaker: Like Gary said that it didn't really look right on first pass, so we need to go back.
Speaker: So the VibeCheck is kind of like a manual testing when you build software.
Speaker: You just click, click, click.
Speaker: Seems like it's fine.
Speaker: It doesn't show you the corner cases or the bugs and so on.
Speaker: And that's where you build an actual test suite.
Speaker: But so the interesting thing that you said about models that we are definitely very early
Speaker: And I see those big models as wild beasts in the sense of like they're very hard to tame.
Speaker: You don't really know how they're going to react to every external stimuli.
Speaker: And so you need to learn how to tame them.
Speaker: So you need to build an intuition and then build a framework around it to know how to contain it, like a cage, essentially.
Speaker: How do I cage Lama 7B, for example?
Speaker: How do I make sure that it always behave the way that I want it to behave?
Speaker: So you have to tame it and sort of train it to your own needs.
Speaker: And so it is definitely a new type of beast in the tech sphere where we're very used to very deterministic and predictable outcomes.
Speaker: And so in this case, it requires a very different sense of evaluation, essentially.
Speaker: I'd love to double click on that.
Speaker: You mentioned just sort of like LLMs evaluating the outputs of other LLMs.
Speaker: I'm curious about where you see limitations with that type of approach.
Speaker: Do you feel that that is a strong foundation to be built upon?
Speaker: Do you want to see other types of metrics sort of become more of the norm and move away from this?
Speaker: Tell me about sort of like where we are now and where you'd like to see the industry go in the future.
Speaker: Yeah.
Speaker: So we're definitely in the early phase of that practice.
Speaker: There's a few academic papers out there.
Speaker: There's the Judge LM paper, there's Prometheus.
Speaker: Another one was called Eval LM, I think.
Speaker: So it's definitely early days and there are some benefits, but also some drawbacks.
Speaker: So the benefits, as I mentioned, is that you can evaluate on arbitrary dimensions, arbitrary properties.
Speaker: The drawbacks is that obviously because an LLM by nature is non-deterministic, it means that you could prompt the same model to evaluate the same output and get different scores.
Speaker: If you ask your scoring model,
Speaker: on a scale of one to five, how creative or how playful is this response?
Speaker: If you prompt this like five times, you may get five different results.
Speaker: So maybe you actually want to do that to get an average score.
Speaker: So there is a problem of variance due to the stochastic nature of language models.
Speaker: Also, another aspect is how do those things correlate to human scores?
Speaker: So that's something we've been studying at Airtrain.ai.
Speaker: We wanted to make sure that our scoring model is able to score just as well as humans.
Speaker: So we created an eval data set.
Speaker: We asked people on the team and others outside the company, people, friends that were willing to spend a few hours scoring a model to evaluate with their own criteria.
Speaker: And even then,
Speaker: Humans themselves have a certain amount of variance.
Speaker: If you ask three people their opinion on something, you will have three different opinions.
Speaker: What is the truth?
Speaker: TBD, but at least you can try to align your scoring model to have results that are as close as possible.
Speaker: So we came up with a good technique to really have high correlation with human scores.
Speaker: But as humans are variable, so is the scoring model going to be.
Speaker: Another problem is that it's pretty costly because you need to prompt a model.
Speaker: Let's say you have a data set of like 10,000 evaluation use cases.
Speaker: you need to prompt not only the model you're trying to evaluate to generate the output, but also to prompt your scoring model.
Speaker: And so if you're paying per inference, for example, or if you have to host this model on a GPU, it costs you money.
Speaker: It's much more expensive than running some off-the-shelf Python metrics that you can just really quickly compute over this string of text.
Speaker: So cost is definitely still a problem, although it's probably an industry-wide problem, not just for evaluation.
Speaker: So it's definitely not a sort of a silver bullet to solve all your evaluation problems.
Speaker: It's just a very promising method.
Speaker: And so where I want to see the industry go, I think some models are going to be fine-tuned and specialized to be scorers, to be evaluators.
Speaker: maybe for specific properties.
Speaker: For example, everybody cares about safety and toxicity.
Speaker: If you're using a model to do, if you want to moderate or to generate human-like responses to forums or things like that or chats, you want to make sure that there's no toxic language or harmful ideas.
Speaker: And so you want to test for toxicity.
Speaker: So you could have
Speaker: a fine-tuned model that is much smaller and much cheaper to run that is just fine-tuned to the rate toxicity.
Speaker: So you create a data set of human ratings of toxicity, and then you can use that to fine-tune a model.
Speaker: And so you suddenly you have a toxicity evaluating model.
Speaker: You can do the same with other properties.
Speaker: Certainly, obviously, your model becomes specialized for one property.
Speaker: It's not arbitrary properties.
Speaker: but it's already a bit more reliable and also a bit cheaper to run.
Speaker: So I could see in the future a whole swarm or fleet of models that are specific to testing for specific things, whether it's, for example, kit-appropriate content.
Speaker: This model will tell you on a scale of one to five how kit-appropriate this content is.
Speaker: And so this is, I think, where things are going to go in the future.
Speaker: That's super interesting.
Speaker: And I think like one thing that you said earlier about sort of like the stochastic nature of these models really resonates in the sense that like so far we've been talking about bringing LLM evaluation closer to what you'd expect in like the software engineering world.
Speaker: But all of a sudden we have a massive departure where in software engineering, you're moving away from stochastic natures in software testing, right?
Speaker: Like if you have a test that is flaky,
Speaker: that is by definition a bad test.
Speaker: And you do everything you possibly can to make it not flaky.
Speaker: So that's why you have mocking frameworks.
Speaker: You have all these different things that have entered the testing zeitgeist to basically like get away from the stochastic nature of distributed systems.
Speaker: But now we're in this world where we can't fight the stochastic nature of models, right?
Speaker: It's just like part and parcel of how they are.
Speaker: So even if you're not using a model to evaluate, maybe you're just taking the output of a model and you're running some unsupervised mechanism on top of that, the outputs will vary.
Speaker: depending on when you decide to run that test, even if the input doesn't change.
Speaker: So now you have to have tests that are robust to that, which become very interesting.
Speaker: I'm really curious about your take on that topic on unsupervised versus supervised means of evaluation.
Speaker: And just to tee it up, one of the things that you said earlier about how there's a bunch of benchmarking and kind of like
Speaker: just sort of finger in the air type tests where people are evaluating LLMs versus other LLMs.
Speaker: So like how good is this particular model at high school math or how good is it at like summarization and all these other kind of like fairly common tasks.
Speaker: But the moment I want to use that LLM for something very specific for my business, for my organization, something that requires my data, something that requires my domain expertise, I'm now out in the deep end of evaluation.
Speaker: I don't know really where to start.
Speaker: Yeah.
Speaker: We can go down the path of building that fairly robust testing sort of harness where I have a bunch of evaluations, I have a bunch of ground truth, and I'm very pleased with that.
Speaker: But there's obviously the step before that, which is just like, I want to get some base level of metrics.
Speaker: Are there baseline metrics that I could look at?
Speaker: Or am I forced, the moment I decide to do something domain-specific, to start building that very robust test set?
Speaker: Yeah, so I think, as you mentioned, the first step is just to look at benchmarks, right?
Speaker: This is to give you a very high level sense of which models perform better.
Speaker: So that's definitely the first thing to do.
Speaker: It doesn't require any work.
Speaker: Just look at the leaderboards and pick out the best, right?
Speaker: Now, if you have access to ground truth, so the actual labels for your eval data sets,
Speaker: then there are some very measurable things you can do.
Speaker: You can do some similarity, like cosine similarity between the outputs, terminated by the model and the ground truth and see how close it is.
Speaker: So that's very quantifiable and you can actually have dashboards and really measure that very well.
Speaker: But even then, you will still have this kind of stochastic variation.
Speaker: So it's advised to maybe for the same prompt, prompt the model maybe five times and then apply your metrics five times and then get the average of that or some aggregated metric over it.
Speaker: Obviously, that costs more money.
Speaker: If you're doing this on GPT-4, for example, then you're going to pay for every inference.
Speaker: So that's a trade-off.
Speaker: As you're going through your journey, everybody starts with... I heard somebody say, if you have an idea for an AI application, start with building it on top of GPT-4.
Speaker: If it doesn't work with GPT-4, just give up.
Speaker: There's no hope, right?
Speaker: I think that's very fair.
Speaker: It is the best model in the market.
Speaker: It really performs extremely well, all benchmarks.
Speaker: Every new model is struggling to beat it.
Speaker: So it's pretty impressive.
Speaker: Obviously, using OpenAI's APIs has its own set of drawbacks.
Speaker: And so at some point, especially if you're in the enterprise and you care a lot about safety and security, you may want to migrate towards something on-prem.
Speaker: And this is when evaluation becomes really important because
Speaker: you can no longer rely on using the best-in-class model.
Speaker: You have to figure out yourself what pre-trained model to use, what rack technique, what fine-tuning, and so on.
Speaker: And so that's when you start looking at those evaluation metrics.
Speaker: So if you have access to ground truth, as I said, things are looking pretty good for you.
Speaker: But obviously, ground truth
Speaker: is cost money.
Speaker: You need to pay people to label or to generate those answers.
Speaker: So the next thing to do is, I think, would be the vibe check.
Speaker: It's probably the best thing to do after that.
Speaker: All those metrics like blue and rouge, for example, I honestly don't know why people still use them.
Speaker: I mean, unless you're trying to evaluate for translation or summarization.
Speaker: Yeah, I agree.
Speaker: I mean, what's the point?
Speaker: Benchmarks are better, I would say, because benchmarks are more like a harness that are both trying to figure out, is the model able to reason?
Speaker: Is the model able to understand English, to choose the right answer in a set of responses and so on?
Speaker: So they're better.
Speaker: And then after that, I would advise people to even to really think about
Speaker: their long-term strategy and building teams around those concepts.
Speaker: You cannot GPT-4 your way out of an AI strategy.
Speaker: At some point, you need to take things seriously and train your engineers to learn about evaluation and about the scientific process of evaluation.
Speaker: People were doing evaluation before LLMs.
Speaker: Back at Cruise, we had entire teams that were building evaluation tools for the vision space, obviously.
Speaker: But it is a whole area of research in machine learning.
Speaker: And so this seems to be taken very seriously.
Speaker: And so once the sort of initial prototype has been built around GPT-4 and people have some initial added value, I encourage people really to start thinking about their strategy longer term, what kind of teams they want to build around that, what kind of strategies and taking it seriously, because it really is here to stay, this new practice of using those non-deterministic outcomes.
Speaker: So yeah, it needs to be taken seriously.
Speaker: That makes sense.
Speaker: So if I can just like summarize what I heard,
Speaker: at sort of like the top level, you've got benchmarks to look at, which is really like an LLM to LLM comparison, just evaluating the overall capabilities of the model.
Speaker: And then when you get into something very domain specific, you have the vibe check, obviously.
Speaker: But after that, the moment you start getting more serious about your implementation, the moment you're starting to really think about putting this in production, it really becomes like an organizational challenge where you need to design processes around building that evaluation test set.
Speaker: You have to get the right labels.
Speaker: You have to feel that there is a process behind finding issues with your deployment and fixing them.
Speaker: and ideally building test cases around them.
Speaker: Am I thinking about that right?
Speaker: Yeah, that's right.
Speaker: I think, you know, as we saw this space explode, the space of LLMs and generative AI, we saw this practice go up the stack, as they say.
Speaker: It's no longer your PhD and your master's degrees that are doing the ML work.
Speaker: It's like your everyday developer, even sometimes task group developer, even sometimes non-technical person, like a PM, for example.
Speaker: that is doing this work.
Speaker: And it's great to get started quickly and get the first product off the ground, get some good ideas of what to build.
Speaker: But at some point, you need to bring back some amount of science into it.
Speaker: So whether it's training your developers to learn about evaluation or whether it's actually hiring people that have degrees in that space.
Speaker: So I think it's important for people to think about that as no longer just like it's grabs a quick opportunity and there's some quick value out of GPT-4.
Speaker: But really, how do we build this muscle internally at our company to know how to wrangle those wild beasts that are LLMs?
Speaker: I completely agree.
Speaker: And I'll add on top of that, my personal hope is that there will be more research into kind of like more unsupervised evaluation metrics, things that are like kind of can form baselines.
Speaker: So for instance, like I'd want to know if I'm in the period between sort of like generic prompt engineering and RAG, right?
Speaker: Like
Speaker: what metrics should I be looking at to figure out if I have more or less the optimal prompt for my task?
Speaker: And then at that point, I move on to the RAG part of this workflow.
Speaker: And then within RAG, it's like, should I be using cosine similarity?
Speaker: Should I be using Euclidean distance?
Speaker: Is this even right for my problem?
Speaker: Am I thinking about it correctly?
Speaker: Do I have the guarantee that if I embed the prompt and my documents that I will see similarity there?
Speaker: Or should I be thinking about this in a different way?
Speaker: Like, can I decompose these things so it doesn't mess up the metrics I already know are baseline for my prompt?
Speaker: But when I add the context, it suddenly like goes off the rails.
Speaker: Like I want, my hope is that there are more kind of like concrete,
Speaker: and discrete sort of like metrics that can be looked at where you can look at it and be like, oh yeah, this part of your prompt is like messed up in this quantifiable way.
Speaker: Go fix that.
Speaker: You know, that's kind of my hope.
Speaker: Yeah, for sure.
Speaker: I think especially evaluating RAG is even more complex because you have different parts of it.
Speaker: You have the ingestion into your vector database, you have the retrieval, and then once you pass the content to the context, does it extract the right thing?
Speaker: So it's important to evaluate each thing, like do unit testing almost.
Speaker: Exactly.
Speaker: Test your ingestion pipeline to make sure that
Speaker: documents are correctly indexed in your vector database.
Speaker: That's the first thing you need to test.
Speaker: Then test your retrieval mechanism for a particular question.
Speaker: Does it actually get the top 10, the most relevant response out of it?
Speaker: Luckily, there's a lot of work already on evaluating search algorithms because search has existed for 20 years now.
Speaker: And so you can already use those techniques to evaluate that.
Speaker: And then the final part, evaluating once you pass the content to the model, that is more sort of generic LLM evaluation because you've already evaluated all the steps.
Speaker: So what people recommend usually is to evaluate each step individually and then have a holistic evaluation as well.
Speaker: The same way that when you do software testing, you have unit testing that are just testing each function individually.
Speaker: And then you have integration testing that tests the end-to-end behavior to make sure that actually makes sense at a high level.
Speaker: Yeah.
Speaker: And I guess like last big question here, how much of this do you want to tackle with an air train?
Speaker: What do you see is like the future scope of air train and how much of what we've discussed do you think fits within that scope?
Speaker: How much is outside?
Speaker: That sort of thing.
Speaker: Yeah, so we started with Eval with Airtrain because we heard from the community that this is an unsolved problem and that people are struggling to decide what model to use.
Speaker: So we started with Eval and it's actually a free product because we actually don't believe people are willing to pay for Eval, even though it's a very important part of the lifecycle, we think is just one feature in a bigger platform.
Speaker: So we started with this because this was the fastest way to get started.
Speaker: But really what we want to go towards is no code fine tuning so that people can upload a data set, select some pre-trained models, set a few parameters for the fine tuning job, and then set up their evaluation harness.
Speaker: And then after a few hours of training, we
Speaker: we can tell them quickly, this model will perform better after the fine-tune for your particular use case.
Speaker: So we want to become a no-code fine-tuning and evaluation platform for LLMs.
Speaker: And then once the model has been fine-tuned, we would like to enable one-click deploy.
Speaker: So you have your models that fine-tune, you click a button, and then we serve it behind an API, and then you can directly integrate it within your app.
Speaker: We're not targeting the RAC side of things.
Speaker: We believe that there's already tons of companies doing that and that's amazing and we don't want to really compete with that.
Speaker: Our expertise at Sematic, the team, we all come from crews, so our expertise is really offline compute.
Speaker: We know how to run
Speaker: large workloads on distributed GPU clusters.
Speaker: We used to run all the training jobs for crews.
Speaker: So this is really our expertise.
Speaker: And so we want to really stay in our lane and focus on that and not so much the online orchestration that is RAG.
Speaker: Awesome.
Speaker: I'm really excited to see you guys grow.
Speaker: I'm excited to see the Airtrain product kind of take off.
Speaker: And I think my big takeaways here are, you know, there is a lot of opportunity in the space and the space, you know, being a LLM based solutions, however you want to define that.
Speaker: And a lot of the stuff happening, what I'll call the middle, the actual model inference in my mind is it's
Speaker: Us mere mortals, quite frankly, don't have a lot of input at this point.
Speaker: There's a few nation states, nation state capitalized research organizations and companies with nation state resources that are going to actually move the needle.
Speaker: And now because of this shift to this non deterministic programming paradigm, I guess is the phrase of the day.
Speaker: There's a lot of the quote unquote boring stuff from software engineering that now becomes outsizedly important and valuable to take advantage of the value that's being created by these, you know, I guess really LLM model companies or foundation model companies, however you want to describe it.
Speaker: And so you're definitely within good company for thinking that, you know, on the evaluation and the monitoring side is where there's significant opportunity and a lot of interest at a minimum as the entire industry is really just, I think you're right.
Speaker: They don't know if they are paying for evaluation because they don't even know what the bottom line they're trying to evaluate against.
Speaker: And so I'm really excited to be a part of it.
Speaker: And Emmanuel, I really enjoyed the conversation.
Speaker: Cheyenne, thank you, as always.
Speaker: Well, Emmanuel, before we head off, is there anything you'd like to plug regarding Airtrain, Somatic, anything personal, Twitter, anything like that?
Speaker: Yeah, for sure.
Speaker: So the website is airtrain.ai.
Speaker: One notable thing is that we have a YouTube channel where we try to do a mix of commentary on AI news, but also talk more deeply about academic papers and techniques that exist out there.
Speaker: So if you're interested in those two aspects, the hard stuff, but also the more gossipy stuff about the AI industry, subscribe to our channel and we post videos every week.
Speaker: And the channel is airtrain.ai on YouTube.
Speaker: What are you talking about, Emmanuel?
Speaker: There's nothing gossipy going on in AI lately.
Speaker: For those listening and maybe listening later, this is about a week and a half after the open AI incident.
Speaker: So no gossip whatsoever.
Speaker: Well, Emmanuel, I really enjoyed the conversation.
Speaker: Thank you so much.
Speaker: We'll include links to everything we talked about in the description here and everybody listening.
Speaker: Really appreciate it.
Speaker: Again, this is John Singleton, co-founder and head of success here at Watchful.
Speaker: Check us out at www.watchful.io.
Speaker: And again, please like, subscribe, follow anywhere you can find podcasts today.
Speaker: Appreciate you all and talk to you soon.
