Transcript
Speaker: Hey everybody, welcome to Grounded Truth, the podcast where we gather some of the world's most influential data scientists, machine learning practitioners, and innovation leaders for conversations on the most relevant topics in AI today.
Speaker: I'm your host, John Singleton, co-founder and head of success here at Watchful, the machine teaching platform for data-centric AI.
Speaker: You can try Watchful for free today at www.watchful.io.
Speaker: And if you like this content and want to get more, please like, subscribe, follow on Apple, Spotify, YouTube, or anywhere that you can get podcasts today.
Speaker: Joining me today is Manasi Vartic, founder and CEO at Virta, the platform that enables end-to-end automation for Gen AI application development.
Speaker: Prior to founding Virta, Manasi created ModelDB while a PhD student at MIT and cut her teeth in industry at a few small companies, Microsoft, Twitter, Facebook, and Google.
Speaker: Really glad to have you here, Manasi.
Speaker: Thank you for having me.
Speaker: Excited for this.
Speaker: Awesome.
Speaker: Also in the room here is, well, I guess not technically in the room, in the virtual room is my co-founder and Watchful CEO, Cheyenne Mahanti.
Speaker: How's it going, man?
Speaker: Hey, I'm good.
Speaker: Glad to be back.
Speaker: Absolutely.
Speaker: So today, well, and before we dive in, I want to give you the chance to plug Virta.
Speaker: Specifically, I have three questions for you.
Speaker: Who is Virta for?
Speaker: What problem do you solve?
Speaker: And why should they, why are you solving that problem now?
Speaker: Or why is now the time to solve that problem?
Speaker: Awesome.
Speaker: Thank you for that opportunity.
Speaker: So Virta is a ML infrastructure company.
Speaker: We started off doing model management and model serving.
Speaker: That's where my PhD work was and that's where my CTO and my expertise was as well.
Speaker: So this means keeping track of the models as you're building them, keeping lineage, where did the model come from, what dataset was used, how well did it perform?
Speaker: And then once you have a model actually running it in a production setting,
Speaker: So taking things that are usually batch oriented and then making them real time because you have, say, a customer at the other end that's looking for a response to your chat.
Speaker: Or you might have a scenario where you're trying to publish an insurance application and you want to do underwriting in real time.
Speaker: So those are the kinds of applications that we started with.
Speaker: And then this year, as everyone knows, has been the year of Gen AI.
Speaker: Happily for us, Gen AI models are just larger models.
Speaker: Much, much larger, but they're still models.
Speaker: And so our expertise on how do you manage them, how do you run them in production, how do you scale them, were super relevant.
Speaker: And so recently we released what we call the Gen AI Workbench,
Speaker: which takes someone with an idea for a Gen AI application from, let's call it zero to hero in a very short amount of time.
Speaker: So we help with a day zero where you're just building an app, then you're going to your day one to 10 call it, where you're optimizing your prompts, your data set, your fine tuning, and then day N where you want to scale it, you want to make it more efficient.
Speaker: So we aim to be the one platform that can take you through that process.
Speaker: And we're seeing some really great use cases and adoption on the platform.
Speaker: So if you're trying to do any set of those things, then go check out Virta at www.virta.ai.
Speaker: Awesome.
Speaker: And I think that's a very apt realm for our discussion today because what I wanted to dig in with you today is a little bit about
Speaker: the challenges and really the shift from traditional, like quote unquote classical ML model development and application development in this new world and new hot phrase or concatenation, whatever you want to call it, of Gen AI.
Speaker: And specifically around putting those applications in production.
Speaker: So not just, you know, I've done something on ChatGPT and it looks and feels and smells really cool.
Speaker: or I played with API for the first time, it's I've understood and discovered business value, I've already gone through the POC process, I might even have a couple of these applications that I want to put up, what's changed, what's painful about it, and how can we think about the workflow as it relates to large language model integration development.
Speaker: And so I was just going to start off, I think you already hit a little bit of it here.
Speaker: A week ago, approximately, today's December 7th, a week ago today, a one-year anniversary of the release of ChatGPT.
Speaker: I'm really curious, what's changed in your world and what's changed in your customer's world since that, you know, in that year?
Speaker: That question alone could take an hour.
Speaker: I'll give a condensed version.
Speaker: Anyone who's been in this space for more than a couple of years will have seen the technotic shift that happened this year.
Speaker: Last year, everyone was about MLOps.
Speaker: How do we run these models?
Speaker: Maybe there was model monitoring, management.
Speaker: And then GPT dropped on the scene end of last year.
Speaker: And then every conversation as the year progressed was really around Gen AI.
Speaker: So how do I use Gen AI?
Speaker: How is it different?
Speaker: How do I retool my teams so that we can leverage Gen AI?
Speaker: This was actually the first time I think in been in this field more than a decade at this point.
Speaker: This is the first time when business was so hungry for AI that every data science leader we talked to was like, we have way too many requests for data science coming all across the organization and we just don't have the resources to handle them.
Speaker: And so I feel GPT did something that the technologists who have been working in the space for many decades couldn't do.
Speaker: It really democratized it in a way that previously never had been possible.
Speaker: So there's various nuances about people trying to apply Gen AI where they can use traditional AI models and all of that that we might get into, but at a top level,
Speaker: it put AI on the map in every boardroom, in every team discussion in a really unprecedented manner.
Speaker: And so that was my big takeaway is like, this has done more to advance the field in terms of opportunities than anything else.
Speaker: So I'm just very excited for it.
Speaker: 100%.
Speaker: I firmly believe the brilliance, modeling aside, obviously is pretty incredible, but was in the interface by providing a chat interface to where one of my customers even said, now business unit leaders can POC themselves.
Speaker: They don't need to engage any technical resources, just they fire up chat GPT or insert model subscription service of choice and see a result and say, hey,
Speaker: why can't I do this now?
Speaker: Or we should be doing this.
Speaker: And it's caused both a lot of excitement and I think a lot of internal team headache.
Speaker: Yeah, alarm and scare.
Speaker: A lot of requests for Turing prizes and things that simply may never exist, but it's exciting nonetheless.
Speaker: 100%.
Speaker: So maybe walk me through it at like a super high level, the traditional kind of ML model development workflow and concerns and what has changed now in Gen AI.
Speaker: For sure.
Speaker: So when we say traditional, I'm going to maybe call out the various types of modeling might go in there.
Speaker: So you might start with regression where you're computing, say, how much inventory might I sell depending on the month of the year and geographically where you're located, all the way to deep learning models where you might have a fairly large model that's doing
Speaker: computer vision sort of detection or segmentation.
Speaker: I'm going to bucket all of them into the traditional, although they were not traditional a few years ago, as I like to remind people, like, this is still new stuff, deep learning.
Speaker: And what has the typical process there was you collect a lot of data, and by a lot, you know, it's not...
Speaker: a thousand it's going to be a millions of data points in the large settings that most people want to use this in they're going to collect a lot of data they're going to go off and train a model they're going to try different kinds of models different parameters to use they're going to evaluate that model on some sort of test data they're going to test it out some more and then they're going to deploy it
Speaker: And the process of building, or as we like to call it, experimentation, is very much how can we optimize it for the best performance.
Speaker: And on the other hand, you have the deployment world where you're taking this model and then you're running it in the efficient and robust manner so that
Speaker: you can use it in a real application setting.
Speaker: So that's where the experimentation and production worlds have also historically been very different because you don't care about performance, like latency throughput during the experimentation phase, but you care a lot about that on the production phase.
Speaker: So at a high level, I would say that's how traditional ML has been done.
Speaker: I assume you guys see it very similarly, right?
Speaker: I would be remiss to add one more step of data curation and cleaning as we started as a labeling automation company.
Speaker: So of course, but yeah, no, a hundred percent.
Speaker: We've definitely seen that same flow many a time.
Speaker: Perfect, yeah, 100%.
Speaker: So if you compare that flow to what is going on in GenEI, the biggest difference you're gonna notice is you don't need to train the models anymore.
Speaker: And so the whole experimentation phase is compressed significantly where you're using a GPT model or you're using LAMA or you're using Mistral, whatever model you pick, and you can get started immediately.
Speaker: So there isn't a training process, you don't even need to go and collect data.
Speaker: There's a caveat to that, that I'm going to get to in a second.
Speaker: But in principle, you do not need to collect the data, you do not need to train the model, the model exists, it's an API, and you're just building the application.
Speaker: And so that's one of the main things that
Speaker: we've seen is it's much less about the models is much more about the applications that you're enabling.
Speaker: And I think that's great as a field.
Speaker: One of the challenges with data science or AI is how do you show value and you show value when it's actually integrated into a business process.
Speaker: And previously that wasn't happening as much.
Speaker: So the shift to application orientedness, I think is a net plus.
Speaker: Um,
Speaker: So I'd say that's one.
Speaker: The second one is this, the production and the experimentation side are closer than before because you are using the final models, so to speak, in a lot of these settings.
Speaker: So where research could build a really inefficient model and not care how it performed in real world, that's no longer an option.
Speaker: So that's where the production requirements have now percolated all the way into experimentation, where you might not pick such a large model because it's going to be slow or expensive, or it's not as effective at the particular task.
Speaker: So those things we're seeing, there's a shift left almost.
Speaker: If you think about software, there's a shift left that's happening there.
Speaker: So that's second.
Speaker: The third one I would say is evaluation is really, really hard.
Speaker: And I think you folks have talked about that a little bit on previous podcast episodes, but I'll recap it to say in traditional model, there are good metrics that are accuracy, AUROC, precision recall, whichever one makes sense in your case.
Speaker: There are numeric objective metrics that you can use because with GenEI, you're creating open-ended,
Speaker: text or images a lot of times, or if you're doing reg, you're still, there's a human looking at whether the answer is correct.
Speaker: That leads to much more subjective metrics that end up being used.
Speaker: And it's much harder to quantify how HNI model or a system is behaving.
Speaker: So that's majorly different.
Speaker: It's a little surprising.
Speaker: Yeah.
Speaker: I would say it's a little surprising to tell an engineer that the first step in evaluation is going to be a, and I quote, vibe check.
Speaker: Does it feel right to lick your finger, put it up in the air?
Speaker: And does this look directionally correct is a far cry from an F1 score.
Speaker: And it's the funny thing and you folks are on the annotation space is like two people are not going to read the same piece of text similarly.
Speaker: So inter label or annotation, all of those things are becoming really mainstream and really critical for these types of workflows.
Speaker: Yes, it was very unsettling.
Speaker: We built our own Gen.AI features before launching the workbench.
Speaker: And one of our biggest challenges was, all right, how do we evaluate in such a way where it's not my product team going and actually labeling things in a spreadsheet, which is state of the art for a lot of people.
Speaker: Absolutely.
Speaker: So that's one of the things I hope will change in the upcoming year.
Speaker: Same here.
Speaker: And so...
Speaker: On the evaluation side, or really diving into the workflow.
Speaker: So I think that the phrase that you effectively communicated was the shift away from the requirement of large amount of data to simply being able to start is prompt engineering is the phrase of the day.
Speaker: Now I can simply apply, you know, context or instructions through my prompt and outcomes result generated text that is hopefully fingers crossed in the direction that I look, that I want.
Speaker: How do you,
Speaker: How do you measure or reason about how the effort into putting into your prompt versus let's say like fine tuning on a specialized data set?
Speaker: That's a great question.
Speaker: Maybe I'll point out a fun thing about prompts before even answering that.
Speaker: Prompting is an art and science, then a science, and people are developing new techniques.
Speaker: I did a talk at Scale by the Bay recently that I'll send you guys a link to.
Speaker: my go-to-market lead can write better prompts than I can and I have a PhD in CS, let's be clear.
Speaker: So that's where traditionally the people who have programmed these models are not necessarily the people who are gonna be best at prompting them because it depends on your ability to effectively communicate, which might not be what data scientists are trained for or engineers.
Speaker: And we just found that really interesting.
Speaker: Like the most effective prompts if we pulled our team
Speaker: came from folks who had more of a literary event than a programming event.
Speaker: And that's something I'm intrigued to see where that goes.
Speaker: But to your question, prompt versus fine tuning, practically, we believe that you should try prompting until perhaps you hit a wall and then you need to go for fine tuning.
Speaker: That's just because it's a lot cheaper to do prompting.
Speaker: You can spend a few days, you can try out the half dozen techniques that have been known to work well for prompting, see how far you can push it before you go to fine tuning.
Speaker: And fine tuning is odd because you need to, the good thing is you don't need such a large data set as compared to a training data set.
Speaker: but you still need the right kind of data set.
Speaker: So enough coverage on the axes that you care about, high quality, all of that, which in our experience can be a little bit hard for Gen AI applications.
Speaker: If they're open-ended, like you're turning some notes into a document or you are creating something from scratch, you might not have those many examples.
Speaker: So that's where we found that
Speaker: prompting is just a lower barrier to entry.
Speaker: But once you hit that wall, you want to do the next best thing, which is going to be fine tuning with highly curated data sets.
Speaker: So I'm personally very interested in quantifying those walls.
Speaker: And part of this is because, to your point, prompting is very much both an art and a science.
Speaker: I think we've said on a past podcast, it's like...
Speaker: It's really easy to just look at prompt engineering and be like, oh, it's a lot of like hocus pocus and like hand waving and like kind of dismiss it that way.
Speaker: But the reality is that some of the techniques that people have illustrated, you know, like few shot prompting as an example, do work, right?
Speaker: Therefore, there is some relationship there that can be quantified in some way.
Speaker: And so I guess my broad question to you is like, you mentioned earlier, just evaluating, like prompting outcomes is a fundamentally hard problem right now because of kind of like the infinite space that generative models play in.
Speaker: I'm curious if you think that there is a future where we do have discrete metrics to describe these things, perhaps not as directly as precision is.
Speaker: recall, you know, F scores, accuracy, you know, and so on and so forth.
Speaker: But maybe something analogous to the Gen AI world.
Speaker: Like, do you see that as like something that is inevitable, something that should be strived for?
Speaker: Or do you think it's more of like a workflow process thing that every business should adopt and kind of like have almost like vibe checks integrated into their workflow?
Speaker: I think it's the latter.
Speaker: Like the reason why accuracy or AUROC works is because the output that you're getting is from a finite set or like a set that you can define fairly tightly.
Speaker: And there are numeric quantities that let you compare two points in that space.
Speaker: versus for, and I'll separate out rag maybe from generative use cases, for the writing of text type use cases, I think it's going to be fairly hard to get a metric that is that tight and it's going to be
Speaker: it's going to be objective.
Speaker: I think what we're seeing and where I expect this to evolve is there might be different dimensions on which you evaluate the outputs.
Speaker: And the HELM benchmarks, if folks are familiar, they do some of this.
Speaker: It's like, what is fairness, bias, accuracy, and so on?
Speaker: There might be conciseness.
Speaker: There might be, is it factual or not?
Speaker: So more and more what we're seeing is there are going to be different axes on which the outputs are going to be evaluated.
Speaker: They could be prompting outputs or they could be just outputs from different models, fine-tuned or not.
Speaker: And you're going to be finding your best alternative in that multidimensional space.
Speaker: And I think it's going to be hard to
Speaker: without the use case knowledge to predict which one of those metrics is going to make the most sense.
Speaker: So my view of the world, and we actually bake this into our product, is you're going to have a leaderboard of the different models and the prompts, and there are going to be multiple scores on which you're going to score your results.
Speaker: And these could be
Speaker: Use case driven, there might be some more generic ones, like from the Helm benchmark, but ultimately you're going to define which of these metrics matter the most for your use case.
Speaker: So that's kind of my role view.
Speaker: That makes sense.
Speaker: Maybe develop metrics internally.
Speaker: Oh, that's what we're seeing from folks is because these are so use case specific.
Speaker: And this is where we did work on model monitoring.
Speaker: We see a lot of parallels there, but it's not quite the same technique.
Speaker: So you don't really care about data drift.
Speaker: It's more, does this match my expectations type metrics?
Speaker: Got it.
Speaker: I'm curious what you guys think too.
Speaker: Yeah, you know, like it's sort of like an open-ended question in my personal opinion.
Speaker: And we've started research to this end.
Speaker: I think like I largely agree with you that like this is a multidimensional problem.
Speaker: And it's also even hard to quantify like, for instance, things like conciseness, whether those even matter in most use cases, because it's possible that
Speaker: A, the use case might not even be NLP, right?
Speaker: It could be predicting over logs or something like that.
Speaker: It could be fraud detection within credit card transactions.
Speaker: And you're like, you know, few shot prompting a set of transactions and kind of like augmenting it with metadata.
Speaker: So there are a lot of cases where like you still want the broad generative foundation that come with these models, but it's possible that you're not
Speaker: actually outputting into an infinite space.
Speaker: It's possible that even if you are predicting like generic text, that you are actually predicting into a space that is more bounded than you might expect.
Speaker: So concrete examples is that
Speaker: You might prompt several times, same exact prompt, and you'll get different outcomes.
Speaker: But then if you embed those outcomes and you take cosine similarity or something of the outcomes, you'll see that they're actually very, very similar amongst one another.
Speaker: Even if the structural nature of them are different, they are conceptually aligned, which then implies that it is predicting into a bounded space just by definition.
Speaker: And then it's up to us to figure out what exactly is that space and quantify the various dimensions of it.
Speaker: So it's a long winded way of saying, I think there are metrics, but I can't quantify what those metrics are right now.
Speaker: You know?
Speaker: Yes.
Speaker: And your comment actually reminded me of there, there was a really interesting paper from Microsoft research on prompting techniques and ensembling and so on that really talks about what you were just describing.
Speaker: And it's really, really exciting.
Speaker: Yeah, I think there's a lot more work to be done here.
Speaker: And I think it's super interesting because for a while, all ML research and implementation was almost in pursuit of better and better outcomes for specific use cases.
Speaker: And now it's almost been inverted.
Speaker: We are getting pretty good results on a wide variety of use cases.
Speaker: We want to figure out why.
Speaker: So I'm really excited about that brand new tidal wave of research that is inevitably going to happen.
Speaker: And again, I'm hoping that in the future, there is a world where we don't have to just bake in the technical concept of a vibe check into everyone's workflow.
Speaker: I think that's a good thing to have overall, but I don't want it to be the first and only line of defense.
Speaker: I want there to be more, and I almost have to feel like there is more as an engineering-minded machine learning person.
Speaker: I just want there to be some quantification of why these things work the way they do.
Speaker: Yes, yes.
Speaker: And I'm optimistic that GPT-4 is supposed to be as good as a human grader.
Speaker: Maybe some of these things get automated and then there are metrics.
Speaker: So it's going to be exciting to see.
Speaker: Totally.
Speaker: So out there listening to this podcast, God bless them, is a data, God bless them.
Speaker: It's a data science manager, small team, maybe have their first model or two in production, have gotten some kudos from the business.
Speaker: It's actually returning on value.
Speaker: They're looking, you know, Gen AI, Gen AI, Gen AI.
Speaker: Their CEO has now told everybody that they're a Gen AI company.
Speaker: News to them.
Speaker: What are,
Speaker: What's some, maybe some sagely advice would you give to this individual looking to put their first LLM, you know, Gen AI applications in production?
Speaker: That is a great question.
Speaker: I might pick.
Speaker: maybe I'll bound it at three.
Speaker: There's a lot that I might want to say to this individual.
Speaker: The first one is make sure, and this is not even related to ML, make sure your data governance is in place.
Speaker: Like that's the number one thing, you know, that you can actually use this data.
Speaker: You're approved to use this data.
Speaker: There might be PII, PHI, and we,
Speaker: talk to a lot of customers who are in those settings.
Speaker: Like, can you actually use this without running into legal issues?
Speaker: The next one is, are you sending this data somewhere where you're not supposed to be sending it?
Speaker: And there's a lot of LLM providers.
Speaker: And part of what happened with OpenAI was
Speaker: folks are thinking about where are they going to run their models, where can they run them securely?
Speaker: And so protecting the data, I think is still a part of the data governance.
Speaker: Can you send this data to this provider for processing?
Speaker: So I feel like that's make sure your IT approves of it and you know, you won't get into trouble downstream.
Speaker: We have actually, a little bit related to that, we have done a bunch of work in model management where model governance is pretty critical and all of these applications have risk levels associated with it.
Speaker: And hopefully that thought has gone into selecting this particular use case.
Speaker: There's a pretty clear risk tiering that a lot of companies will employ for their applications to just make sure.
Speaker: you're in sync with that.
Speaker: So if we take that of the table, you know, what you're doing is, is approved by risk and it you're in good shape.
Speaker: The next thing that I would think about is cost, it can be surprisingly expensive to run GPT at scale, depending on what kind of business you're in, how many calls you're making all of that.
Speaker: And that's where
Speaker: I firmly believe that models in the future are going to be smaller and bespoke.
Speaker: Like GPT is great.
Speaker: This is a generalized model.
Speaker: It can do a few shot on so many things.
Speaker: That's one extreme, but it's also a very big model.
Speaker: On the other hand, if you look at the smaller models, like a Mistral, order of magnitude, Mistral has 7 billion parameters.
Speaker: GPT 3.5 had like 175 or something like that, and they're pretty on par.
Speaker: And so where I see it going is these models for use case specific scenarios are going to become smaller, and that will give an opportunity to fine tune the cost.
Speaker: So make sure you're not running out of your open AI or whichever Azure budgets as you're doing this.
Speaker: And then the third one is...
Speaker: human in the loop evaluation, you're going to have tested this on this application on a fairly small set of data.
Speaker: Maybe it's 1000 2000, if that, and we don't see people doing that.
Speaker: So like, if you're at the 1000s level, great, you're actually ahead of the pack.
Speaker: But what your customers are going to throw at you are going to be different data points.
Speaker: And so
Speaker: efficient tracking of those data points and then labeling again to make sure that your model is continuing to produce good outputs.
Speaker: I would say those three things are the main things that I would ask anyone who's putting a app in production to think hard about.
Speaker: I'd love to just like zoom in a little bit as well.
Speaker: So let's say that this hypothetical manager is tasking a team with building this like LLM application.
Speaker: And I see sort of like phases of development, right?
Speaker: Similarly to like an engineering workflow in any capacity.
Speaker: It's like you've got the initial development sort of work where you get to sort of like your MVP team.
Speaker: you have it tested to your point with some human in the loop evaluation.
Speaker: And you're iterating to get better and better results.
Speaker: Earlier in the episode, you mentioned once you feel like you hit a wall with prompt engineering, move on to fine tuning.
Speaker: Within prompt engineering, there's a lot of different techniques that you might look at.
Speaker: You've got
Speaker: Fushot, you've got Fushot with Rag, you've got like a whole bunch of like little things that people have put forth.
Speaker: I guess my question to you is like, how much of that
Speaker: Should this team be aware of?
Speaker: Where should they start?
Speaker: What walls might they hit?
Speaker: How do they get through them?
Speaker: How do I know I need rag?
Speaker: How do I know I need to do Fuse Shot?
Speaker: How do I know that my prompt is the problem and not something else?
Speaker: Talk me through what I would be experiencing as a developer of one of these applications and the walls that I might hit along the way.
Speaker: I compare the process of Gen.ai app development to the Gartner hype cycle.
Speaker: If you're in tech, you know what the Gartner hype cycle is, like peak of inflated expectations, valley of despair, and then slope of enlightenment or something, ladder of productivity.
Speaker: I map all of Gen AI development to that.
Speaker: You tinker with GPT.
Speaker: I'm going to change the world.
Speaker: This thing is so awesome.
Speaker: It's great.
Speaker: Soon you try it on real data and you're like, shoot, this sucks.
Speaker: This thing does not work.
Speaker: And then you keep trying to prompt engineering like AI is really dumb.
Speaker: This is never going to work.
Speaker: This is just like a dead end.
Speaker: And then slowly but surely you discover the better ways of prompting.
Speaker: Maybe it's few shot or you're doing chain of thought or you're doing chain of density and then you do fine tuning and you start creeping, you know, creeping up that path until you get to that plateau of productivity.
Speaker: So those are in our experience, the walls you hit are you can't effectively prompt and for productivity.
Speaker: A lot of people are building with Gen AI aren't necessarily from this space.
Speaker: And so knowing the different kinds of promoting strategies is really challenging for them.
Speaker: And for that plug here, so that's why what we did in the workbench was we apply the prompting strategies that exist.
Speaker: There's like six strategies that work.
Speaker: Depending on the use case, we apply those prompting strategies so people don't need to go figure out what they are themselves.
Speaker: In our experience, that's not very differentiated work.
Speaker: You should still go in and tune the prompts after that.
Speaker: But we'll do a basic lay of the land.
Speaker: Here's what you can get with different strategies assessment for you to keep going.
Speaker: So we found that to be really productive, getting to a baseline and then get to fine tuning.
Speaker: Like if you've done your basic strategies and you're not seeing significant difference or whoever is evaluating the outputs are still saying, OK, something's missing.
Speaker: It's not getting the tone right.
Speaker: It's not getting it's hallucinating.
Speaker: Great.
Speaker: OK, then move to more complicated cases.
Speaker: And so that's sort of my heuristic.
Speaker: And then the final wall they're going to hit is cost.
Speaker: And there's a whole different set of things that folks should do for that.
Speaker: That makes sense.
Speaker: So I think it's a good place to final question here for the both of you.
Speaker: Crystal ball time.
Speaker: We're at
Speaker: The end of 2023, it's December 7th, 2023.
Speaker: We are just over a year of ChatGPT's release and what a year it's been.
Speaker: It's been quite a whirlwind.
Speaker: I think we're all in very different places than we were 12 months ago.
Speaker: Crystal ball prediction of 2024.
Speaker: What do you see coming out in the world of AI in general?
Speaker: You don't have to box it to Gen AI.
Speaker: However, I think that's definitely the flavor of the day.
Speaker: Okay.
Speaker: All predictions are wrong, but they're helpful.
Speaker: Or was that models?
Speaker: I don't know.
Speaker: Whichever one of them is.
Speaker: So that, yeah, that phrase has been used a lot.
Speaker: I'll throw out three.
Speaker: That sounds like a good number.
Speaker: Multimodal is really catching up.
Speaker: The Gemini release from yesterday and just previous work.
Speaker: multimodal is really becoming real.
Speaker: So I'm excited to see where that goes.
Speaker: And I think we're going to see a lot of advances there.
Speaker: Text has been great.
Speaker: Some of the other areas are catching up now.
Speaker: So lots of new applications we'll get.
Speaker: enabled.
Speaker: Second one, I would say is smaller and more specialized models will, I don't think they will become ubiquitous next year, but we're going to see real progress and how small can we make these things and still have them have high performance.
Speaker: So that's something I think will happen.
Speaker: Um,
Speaker: Then I would say the third one is it's going to be interesting from a regulatory perspective on what happens in AI governance.
Speaker: That's where that might be more on the political or awareness level.
Speaker: I think we're going to see more with the EU AI Act hopefully going into effect and the Biden administration order
Speaker: I mean, we're going to see some changes there that will hopefully push this field towards more open, transparent, and accountable AI.
Speaker: And at a minimum, generate a lot of consulting spend.
Speaker: Yes, exactly.
Speaker: And yet McKinsey reduced their partner class by like 100 or something.
Speaker: So there is interesting market dynamics right now.
Speaker: Let's just say that.
Speaker: Yeah.
Speaker: Since you also directed the question to me, I'm going to yes and everything.
Speaker: I think all three of those predictions were spot on.
Speaker: Again, with the giant caveat that every prediction is wrong, etc., etc.
Speaker: I will add on to that.
Speaker: I think there, so one kind of like business level prediction is that I think like a lot of these companies and tools that exist that are kind of like a single Lego block in a larger set of things are naturally going to expand.
Speaker: I think moats right now are really, really small and everyone is currently afraid of open AI just coming in and eating their lunch, quite frankly.
Speaker: So I think there will be a defensive maneuver to eat more and more of the stack and gain more and more mindshare.
Speaker: So I would expect to see a lot of people shouting about their more and more integrated workflow and stack than the next person's.
Speaker: I think that's point one.
Speaker: Point two is I think out of that, hopefully, as I've been kind of like thumping my chest about it for a while now, is just like the emergence of stronger metrics, kind of like more grounded understanding in like an actual prompt engineering workflow,
Speaker: Like when do I actually start prompt engineering?
Speaker: When do I stop?
Speaker: When do I start fine tuning?
Speaker: What should I be looking for?
Speaker: At a minimum, start there.
Speaker: And then hopefully that comes out with some set of metrics or some framework, at least for people to think about it.
Speaker: And that's the second hope, less of a prediction, more of a hope there.
Speaker: I hope that happens next year, but we'll see.
Speaker: The third, and maybe this is controversial, maybe it's not.
Speaker: I think right now we're still very much in AI experimentation mode as an industry.
Speaker: I think there are actually very few real generative AI workflows that are in production that are not just chatbots.
Speaker: And I don't mean to belittle chatbots.
Speaker: They serve a very important purpose.
Speaker: But my hope
Speaker: is that next year there is a stronger emphasis on bringing more and more of this to production.
Speaker: So embedding more intelligence into the fabric of applications rather than it just being a widget of a chat bot that you can sometimes talk to and sometimes has integrations of things that you might do day to day.
Speaker: So I'd want more magic to be built into most of the applications that I use day to day.
Speaker: My hope, again, is that that happens next year.
Speaker: I'm with you on that last one.
Speaker: No more chatbots, please.
Speaker: Yeah, please.
Speaker: For the love of God.
Speaker: I don't need to know that it's there.
Speaker: Yeah.
Speaker: I'm most excited about not feeling that I am using a Gen AI powered tool.
Speaker: It should just solve the problem that...
Speaker: It was designed to solve and is better because there is a Gen AI component in the background doing the value part.
Speaker: I don't need to see another talk to your docs.
Speaker: It's fun.
Speaker: It's really cool.
Speaker: It is useful.
Speaker: But I'm really excited to see all of that quote unquote boring ML kind of maybe come back in a little bit of fashion using these newfangled techniques.
Speaker: That was awesome conversation.
Speaker: Manasi, Cheyenne, really enjoyed it.
Speaker: Just to kind of wrap up here, any last plugs for you, Manasi, with Virta, anything, Twitter, anything?
Speaker: Yeah.
Speaker: Check out the Gen AI Workbench.
Speaker: It's www.app.virta.ai.
Speaker: You can go and build the Gen AI application, experiment with prompts and models right in that same playground and
Speaker: You should have an app in less than 10 minutes if we have done ourselves right.
Speaker: If not, send me a note.
Speaker: Go check out VertaDataAI and get your app in 10 minutes.
Speaker: I think it's a pretty strong sell right there.
Speaker: This was great.
Speaker: Everybody, I really enjoyed it.
Speaker: Just to wrap up here, I'm your host, John Singleton, co-founder and head of success here at Watchful, the machine teaching platform for data centric AI.
Speaker: You can try Watchful for free at www.watchful.io.
Speaker: Guys, really had the great conversation.
Speaker: Look forward to chatting again and best of luck in 2024.
Speaker: Thanks so much.
Speaker: Thanks for having me.
Speaker: Absolutely.
Speaker: Thanks, Monasi.
Speaker: Bye.
