How Do I Evaluate an LLM In Healthcare?

In this week's Health Data Ethics podcast episode, I talk about healthcare-specific methods of evaluating LLM output - a recent paper on HumanELY, a web tool for evaluating LLM output across five separate axes, is a great lens for thinking about your ai tools. I also discuss a recent evaluation of GPT-4 Vision in which our AI friend ends up with the right answer to a medical case but can't quite tell us why.