Transcript
Speaker: Hey everybody, welcome to Grounded Truth, a podcast where we gather some of the world's most influential data scientists, machine learning practitioners, and innovation leaders for conversations on the most relevant topics and AI today.
Speaker: I'm your host, John Singleton, co-founder and head of success here at Watchful, the machine teaching platform for data-centric AI.
Speaker: You can try Watchful for free at www.watchful.io.
Speaker: And please, if you like this content and want more of it, like, subscribe, follow us on Apple, YouTube, Spotify, or anywhere you get your podcasts today.
Speaker: Joining me today is my good friend and co-founder, Cheyenne Mahonte.
Speaker: Hello.
Speaker: Welcome Cheyenne.
Speaker: Thank you.
Speaker: Thanks for having me again.
Speaker: Yeah.
Speaker: I guess you could say you're somewhat of a regular.
Speaker: Yeah, somewhat now.
Speaker: I'm really excited to talk to you today.
Speaker: You just published a blog post about a week ago now, in addition to a little demo of the concepts that were researched in the blog post and also posted the repo on GitHub called A Surprisingly Effective Way to Estimate Token Importance in Large Language Model Prompts, which feels like a mouthful, but maybe I'll give it to you to translate that for me.
Speaker: Yeah, so I guess just the punchline is if you've ever written a prompt to get an LLM to do something and it doesn't quite do what you want it to, the question is what do I have to change in my prompt to achieve the outcome that I want?
Speaker: So the first step in that whole thing is what parts of my prompt was the model paying the most attention to?
Speaker: And it turns out that's actually like
Speaker: a fairly interesting and fairly hard question to answer, especially in today's world when we no longer have direct access to the models themselves.
Speaker: So our goal really was just to figure out, is there a way that we can estimate this?
Speaker: And if there is, how good of a job does it do at estimating token level importances relative to the real ground truth?
Speaker: How is this different than asking, I know chat GPT offers, like you can ask it for the attention directly and get the log prob odds directly from the model.
Speaker: How is this different?
Speaker: So just to be clear, ChatGPT and no existing sort of vendored LLM API will give you actual attention.
Speaker: You can prompt ChatGPT and be like, for this prompt, tell me how important each token is.
Speaker: But oftentimes when we tried that, we would get...
Speaker: like kind of arbitrary results.
Speaker: Oftentimes the model would just like put the same importance for every single token.
Speaker: It would be like 0.5.
Speaker: It didn't actually do what you wanted it to.
Speaker: That's sort of point one.
Speaker: Point two is that you can like estimate log probs and so on, but like
Speaker: That's on the output.
Speaker: So when the output is provided from the model, you give it a prompt, it gives you back a result.
Speaker: For certain models, so for instance, for like DaVinci 3 and now like, you know, GPT 3.5 Turbo Instruct, you can get log probs, which basically means that for the generated output,
Speaker: it'll give you, here are the top five other possible tokens that could have gone into this slot, and here were the probabilities that were associated with each one.
Speaker: But that's not telling you how the model got to that result.
Speaker: All it's telling you is what the result was and what the possible results could have been.
Speaker: But more importantly, or actually just differently importantly, I want to know,
Speaker: I gave you an input, what part of my input were you paying the most attention to and why?
Speaker: So right now there isn't really a direct way to do that.
Speaker: And actually even the ground truth way, or at least as close as we can get to ground truth, which is an approach called integrated gradients,
Speaker: Like those approaches are really expensive to run from a computational perspective.
Speaker: So it's not practical to be done on models of the size of like GPT-3, GPT-4, you know, let alone any other kind of modern language model right now.
Speaker: So maybe if I could summarize it, one, a key component of this is, you know, I keep on using the phrase, turn prompt engineering into less of an art and more of a science.
Speaker: Develop quantitative metrics driven ways to improve the quality of prompts.
Speaker: As someone who's written hundreds, if not thousands of prompts at this point, I can tell you it'd be a lot more useful to have tools that are better than, well, maybe if I'm more polite or if I'm more direct or just provide more examples, do I even know if they're providing any value to my task?
Speaker: So it's definitely welcome and appreciated.
Speaker: And two, there seems to be a limited number of models that you can even get some facsimile of attention, and in the best case, some log odds.
Speaker: But in cases where you do not own the model inference itself and the infrastructure it's running on,
Speaker: there's basically no way to get that any sort of quantitative insight into what the model's paying attention to in the prompt.
Speaker: And so what this is, is an attempt to provide a way that is hopefully model agnostic and inexpensive and efficient to run, to be able to provide that quantitative direction on improving prompt quality.
Speaker: Is that like a fair summation?
Speaker: Yeah.
Speaker: Generally speaking, yes.
Speaker: To put a fine point on it, it's like,
Speaker: Right now, there's a lot of just sort of noise around prompt engineering as a concept.
Speaker: Lots of folks are talking about prompt engineering being snake oil.
Speaker: In fact, there's like a Hacker News post about it, like literally yesterday, where it talked about how prompt engineering is just like a bunch of people who are scamming other people by, you know, creating...
Speaker: English or language-based giant prepended prompts and post-pended and so on, or appended.
Speaker: I've seen prompt marketplaces for prompt engineering.
Speaker: I mean, it's wild.
Speaker: So kind of what we're trying to do is get this back to something that we can reliably call a foundation.
Speaker: We're in this realm where everyone is trying to sprint forward as quickly as possible, and I think that's right.
Speaker: And I'm certainly not going to call prompt engineering snake oil.
Speaker: there are elements of it that are right, but it's not built on a real foundation.
Speaker: So for instance, like there are techniques that people have stumbled upon, like few shot learning or really like few shot prompting, uh, that seemed to work, you know, and like that is transferable across models.
Speaker: It does appear to be a technique that seems to work, but it's not a panacea.
Speaker: It's not a silver bullet.
Speaker: It's not like it will work all the time.
Speaker: And, you know,
Speaker: no matter what your few shot cases are, it will always work.
Speaker: It's just a tool in the toolbox.
Speaker: And then you have to rely really heavily on human intuition to figure out like, when do you change it?
Speaker: When do you not?
Speaker: That just means that there's something there, that there is a repeatable pattern.
Speaker: And that pattern does trend in the direction that you'd expect it to, which means that there's something that is learnable there.
Speaker: And there's something that has to be measurable.
Speaker: So what we're trying to do is start by measuring it.
Speaker: Can we measure it?
Speaker: That's really the first question.
Speaker: The second question is, to your point, can we measure something that is generically useful?
Speaker: Meaning, can we measure something that will work irrespective of which vendor you decide to use or if you're trying to use like an open source model like a Llama 2 or something, will these techniques apply to that type of architecture as well?
Speaker: So to save you from, you know, kind of like the positing here, the answer is it appears to be yes.
Speaker: We have found that there is a transferable approach that will work.
Speaker: And interestingly, we found it to be kind of architecturally and like vendor agnostic in pretty like surprising ways.
Speaker: So tell me a little bit about your approach here.
Speaker: Yeah, so it's actually like, again, surprisingly simple.
Speaker: So the approach is basically given a prompt, I wanted to measure what changes I would have to make to the prompt such that it radically changes the model's interpretation of it.
Speaker: But I wanted to do this in a way that was as cheap as possible, both from a compute perspective as well as from like a capital perspective and from a time perspective.
Speaker: So embeddings are kind of like a nice thing that a lot of vendors offer is like an API.
Speaker: Obviously there's a ton of embedding models out there and most
Speaker: pre-existing open source models will also allow you to sort of interact with their embedding spaces.
Speaker: So the approach was given a prompt, perturb it in various ways, and the most obvious and simplest perturbation is ablation, what's classically called an ablation study, which is literally just like given a prompt, drop each token, and see how that changes interpretation of the prompt.
Speaker: Now, interpretation of the prompt is an important thing here because you could do this and just prompt the model directly and be like, give me an output and then measure the difference between that.
Speaker: But if you have a giant prompt, that would be very, very expensive and it would take a really long time to do.
Speaker: So what we wanted to figure out is like, can we do this purely against the embedding space?
Speaker: and get something that is reliable.
Speaker: And so what we did was for each perturbation, we re-embedded that whole prompt minus that token.
Speaker: And we looked at the embedding that came back and we compared.
Speaker: And it turns out that the greatest changes in the resulting embedding
Speaker: came from dropping what are presumably very important tokens.
Speaker: And the way we validated this approach was we compared it to the closest thing we could get to ground truth, which was let's run this through like an integrated gradients approach.
Speaker: So we would take the prompt, run it against the model.
Speaker: What is integrated gradients for?
Speaker: Good question.
Speaker: Sorry.
Speaker: So integrated gradients is just an approach to figure out what features are most important for any neural networks decision, whether we're talking about transformers or not.
Speaker: Essentially, I'm going to talk like high level about it, but
Speaker: you basically like initialize a baseline.
Speaker: So think like pure neutrality.
Speaker: So there's nothing there.
Speaker: And then you have these various features.
Speaker: And the features that I wanted to sort of estimate importance for at this point were tokens.
Speaker: So you pick a neutral starting point.
Speaker: So just a pure baseline.
Speaker: And then you gradually transition from that baseline to the actual input.
Speaker: So you almost have like a bunch of dials across this baseline that you're slowly turning up or turning down depending on what your actual prompt is.
Speaker: And then for each sort of step along this path, we calculate how sensitive the model's output is to small changes in each of those features.
Speaker: And then we take an average of those sensitivities along that path.
Speaker: And then you can then multiply this average by the difference between the actual input and the baseline for each feature.
Speaker: So that gives you a measure of roughly how important each feature is to the model's decision.
Speaker: How much did it impact the model's interpretation of whatever your input was?
Speaker: So basically you have to like back prop on these models.
Speaker: So we do this on GPT-2 because that was like reasonable to be done and like reasonably fast.
Speaker: And we found that when using GPT-2 embeddings against GPT-2 integrated gradients, we got a fairly high degree of correlation.
Speaker: But what was most surprising to us was that when we did this same approach, but instead of GPT-2 embeddings, we did GPT-3 embeddings.
Speaker: So we used, I think, ADA002 embeddings from OpenAI.
Speaker: we got better results.
Speaker: And that was surprising to me because I assumed that the correlation was at the level of model architecture.
Speaker: So if you use the same embedding space for the same model and you compare the integrated gradients approach to whatever we got from the embedding space, I assume that those would be as closely correlated as they possibly can get because you're taking it essentially from the same space.
Speaker: You're learning from the same architecture.
Speaker: Right.
Speaker: But as it turns out, the quality of the embedding space is actually a greater predictor of model interpretation than anything else.
Speaker: So what that means is that in theory, you could take any super high quality embedding.
Speaker: Like tomorrow, OpenAI might come out with a better embedding or Cohere might or Anthropic might.
Speaker: And you pick whatever the best embedding space is for the language or the tasks that you're using.
Speaker: And you compare that using this approach, this ablation study approach, to what any other transformer-based language model is doing.
Speaker: And it should be an even better predictor of what that model is attempting.
Speaker: So that was a surprising indicator to us.
Speaker: So that essentially means that I could take Ada2 and use that to predict what Coheres model is doing for my prompt.
Speaker: So if we have an ability to compare, basically all we need is better embeddings, right?
Speaker: Better embeddings will enable us to give finer and finer detail into what our prompts like are doing or can do, and then potentially compare those against the performance of not only one model's architecture that may be similar to another, but another like even vendor's model completely outside, you know, had nothing to do with the original model being developed.
Speaker: Yeah, that's at least half true.
Speaker: One of the things that we also discovered in this is that sometimes this approach doesn't predict very well.
Speaker: And it was really interesting to dig into why that is.
Speaker: Our hypothesis was that as the prompts become more ambiguous, we
Speaker: the importance, like the integrated gradients approach would show us something different to what we would get from a pure embedding approach.
Speaker: And we actually saw that in the experiment.
Speaker: And you say ambiguous.
Speaker: So you mean like you're a coding assistant.
Speaker: I'm going to throw questions to you.
Speaker: You're going to help me develop a solution.
Speaker: Yeah.
Speaker: Would that be like maybe on the extreme end of ambiguity?
Speaker: What would something that would be extremely non-ambiguous be?
Speaker: Yeah, it's like in this article, tag every mention of Starbucks.
Speaker: It's the difference between something th at's a very direct
Speaker: a very direct command almost that has all the context necessary to approach it within the body of the prompt versus something that's like very open-ended, like a brainstorming prompt.
Speaker: Like I want you to help me brainstorm X, Y, or Z, or you are now a machine learning expert.
Speaker: Tell me what type of machine learning algorithm I should use if I have no labeled data.
Speaker: Basically the intuition is that if a prompt
Speaker: can have many possible answers, and all of those might potentially be within the realm of correctness, but all very different for one another, versus there being a single answer, then that is kind of what determines ambiguity.
Speaker: And that actually appears to be... It's not that the embedding estimation becomes bad or useless.
Speaker: It just becomes...
Speaker: it doesn't correlate quite as well as it would if there were very little ambiguity.
Speaker: So we think that there is several other metrics that need to be defined in order for us to actually understand how the model will react to a particular input.
Speaker: Part of it is what parts of my prompt are most important.
Speaker: Yes.
Speaker: Another part of it is how ambiguous is my prompt?
Speaker: Like how sort of spread will my answers be?
Speaker: And then you as a user will have to figure out like, you know, do I want this level of ambiguity?
Speaker: Like perhaps it's good because I'm trying to create a prompt that, uh,
Speaker: gives me a creative answer, in which case, like you actually do want the ambiguity.
Speaker: But in other cases, when you're like creating a prompt that needs to be really narrow and you're passing data through it constantly, you actually want as little ambiguity as possible so that you have reliable results on the other end.
Speaker: So, yeah.
Speaker: Yeah.
Speaker: So is it safe to say that this approach will probably have more impact and I think, which is a positive thing on discriminative tasks versus those that are, you know, purely generative in nature, like the creative tasks you mentioned?
Speaker: No, I think that there is like, here's another example.
Speaker: It's like, write a blog post about the impact of human labor in the process of labeling for machine learning.
Speaker: Focus specifically on human rights violations or labor laws or something like that.
Speaker: Like,
Speaker: That is kind of open-ended in the sense that there are a lot of different shapes of blog posts that would fit that criteria.
Speaker: Yeah, but virtually an infinite amount.
Speaker: Yeah, but you know that the guts of that content will be rooted in something that is kind of roughly in the same space.
Speaker: So that type of prompt would actually measure or should actually measure fairly low on the ambiguity scale, despite the fact that it is fundamentally a creative problem.
Speaker: So it's those types of things that are kind of in the middle where even though it's not a discriminative task, it's a purely generative task, you're still looking for something that is like,
Speaker: of a shape that you'd recognize.
Speaker: Another example is like, write me an email based on these four bullet points.
Speaker: You know, like there's a lot of different shapes of that email that would be acceptable, but you know that the core of that email should be centered around those four bullet points.
Speaker: So there are lots of cases where you'd want to understand not just, you know, how important were these tokens in my prompt to produce a particular output, but you'd also want to know like, okay, is my prompt
Speaker: reasonably good or will it have outputs that are predictable that I think will be acceptable?
Speaker: And from at least an anecdotal standpoint, I would say that definitely checks out from my experience of writing prompts.
Speaker: Every time that you have this logical leap of, oh, it should know this, it was trained on this, it'll pick up some nuance or context that's required for adequate completion of my task.
Speaker: You always get better and better results by figuring out more ways to be more direct and to the point less ambiguous.
Speaker: So
Speaker: So anecdotal evidence is the best evidence for what that's worth.
Speaker: Yeah, totally.
Speaker: So we found that we have an inexpensive, efficient way to at least directionally, it seems to work pretty well versus ground truth to estimate token importance.
Speaker: What's next?
Speaker: Yeah.
Speaker: So the intuition here is that, again, we want to like remove a lot of the murkiness around prompt engineering specifically.
Speaker: Yeah.
Speaker: There's a lot of reasons why that's important, but kind of like the biggest thing is that more and more people are starting to rely on LLMs and prompts in general, and being able to interpret what goes into these models to produce an output, and also being able to measure the output itself is super important.
Speaker: So the way we're thinking about it is as a user of these large language models,
Speaker: What would I need to know in order to improve my prompt?
Speaker: Let's say I write a prompt.
Speaker: I get an output that's not right.
Speaker: What do I do at that point?
Speaker: Ideally, there is an action.
Speaker: And ideally, that action is not, let's just try something different.
Speaker: In a perfect world, there's a very clear answer to what I should be doing next.
Speaker: Now, there's not always going to be that, but that's kind of like the goal.
Speaker: That's what we're striving for.
Speaker: So in a world where we magically have all the metrics that you might need to evaluate this as a human and figure out what the next step is, once you have that and that process is mostly on Rails, then it follows that there should be essentially a program that you could write that would just do this for you.
Speaker: Given a prompt, a starting prompt, and let's say a target of what you're trying to get the model to do,
Speaker: There should just be like a magic button that you can click that will automatically like refit the prompts to the task that you're trying to do and minimize for ambiguity, maximize for token density, you know, get rid of all the excess that you don't have to spend money on tokens that, you know, aren't affecting your output and so on.
Speaker: All that should just happen magically.
Speaker: Now,
Speaker: This first thing that we released, the simple way of kind of estimating token importance is really- I guess I should say, I don't think we mentioned the demo outside of the top.
Speaker: So this is something that is released that you can play with.
Speaker: Yeah, yeah.
Speaker: Please, for anyone listening to this, please play with it.
Speaker: The URL is heatmap.demos.watchful.io.
Speaker: And that will, of course, be included in the description.
Speaker: Of course.
Speaker: Yeah.
Speaker: You don't have to jot that down right now.
Speaker: Yeah, it's live.
Speaker: You can play with it.
Speaker: It's all open source as well.
Speaker: So if you want to look at the approach again, like there's a lot more to be done here.
Speaker: I think, you know, right now we're using a really simple like distance function.
Speaker: The reality is that like there's a lot to be
Speaker: to be had by improving the distance function.
Speaker: And we have lots of thoughts about how to do that.
Speaker: There's a lot of depth that you can get out of this.
Speaker: What we found is that there is a correlation between the embedding space and what the model is actually doing.
Speaker: And so by being able to extract more information about those changes from the embedding space, we'll be able to predict better for the model.
Speaker: Right now, we just kind of like scratched up the iceberg, and there's a lot more to be done there.
Speaker: But
Speaker: Moving from that, what we'd like to do is continue finding these metrics that theoretically I as a human, as a human user of large language models, would use to figure out what I should be doing to my prompt to make the output better.
Speaker: And how do I measure that?
Speaker: How do I know that I got a better output?
Speaker: How do I know that I will always get a better output, or at least usually get a better output?
Speaker: Can I measure how often I get a better output versus other times?
Speaker: Um, these are like questions that we're aiming to answer.
Speaker: So the very next thing that we're likely to do is, um, release some of our research on ambiguity estimation on prompting and outputs.
Speaker: Uh, and then we'll just kind of keep chugging from there.
Speaker: Awesome.
Speaker: Again, I would encourage everybody to check out the demo.
Speaker: We'll include the links as well as to the repo and the blog post wherever you found this podcast.
Speaker: Really excited to see the continued research.
Speaker: I know the team's really excited here at Watchful.
Speaker: And yeah, I mean, it's amazing.
Speaker: We need better tools is the clear takeaway.
Speaker: We need more quantitative approaches to be able to guide users to be more effective outside of, I mean, really, I feel sometimes with our customers, I'm talking black magic.
Speaker: I'm firmly in the realm of black magic.
Speaker: Maybe have you tried to be more polite?
Speaker: Can we take out this word?
Speaker: Have you tried a more direct verb?
Speaker: You know,
Speaker: provide more examples, it just it feels there's no one right answer, and at least even getting some directionally correct feedback on where I can improve and how I can improve.
Speaker: And I think in the dream world of, you know, if the culmination of this research could be something that is truly programmatic in nature, would be something revolutionary for all of us working in and around large language models today, dealing with prompts.
Speaker: kind of the ever-growing surface area of the world that's dealing with this whole new type of API, this type of interaction.
Speaker: Excited to see where it goes.
Speaker: Cheyenne, really appreciate it, man.
Speaker: Of course.
Speaker: Thanks for having me again.
Speaker: Yeah, I think I have to.
Speaker: You're technically my boss.
Speaker: So I blink twice if everything's okay.
Speaker: Watch out.
Speaker: But I really appreciate it.
Speaker: It's been a great episode.
Speaker: I encourage everybody to check out again the demo, the repo, and the blog post.
Speaker: If you have any thoughts, find it interesting or just want to chat about it, please reach out.
Speaker: and stay tuned for more.
Speaker: We're going to be releasing some more stuff.
Speaker: Don't want to talk too much about it, but shine alluded to some of the court questions that we're exploring and really, really, really excited about kind of the direction that's going and what, what could bear fruit.
Speaker: So thanks everybody.
Speaker: Again, this has been grounded truth.
Speaker: You can find us wherever you find your podcast today, YouTube, Spotify, Apple, et cetera.
Speaker: No longer Google as of yesterday, I learned Google podcasts is no more.
Speaker: So if you were a Google podcast listener,
Speaker: You're not listening to this now.
Speaker: Thanks so much again, John Singleton, co-founder and head of success here at Watchful, joined by Cheyenne Mahonte.
Speaker: Everybody enjoy your day.
