Unique Jewish Law Question on Same-Sex Marriage
00:00:02
Speaker
question that has never been written about in Jewish law, which is whether a person who's in a same-sex marriage can give a priestly blessing. I'm not interested in discussing with you either same-sex marriage in Jewish law or priestly blessings except to tell you the following. Jewish law has been in existence for
00:00:26
Speaker
twenty five hundred years and this is the first time that question was asked for a variety of social reasons there are no sources that discuss this problem so no matter how much how large a database of Jewish law chat GPT has it doesn't have an answer to this question in its database it has to to use the anthropomorphic term
00:00:53
Speaker
think about this question from sources that don't discuss this question but discuss things that are around this question.
Introduction to Interactions Podcast
00:01:05
Speaker
Welcome to the Interactions podcast, brought to you by the Center for the Study of Law and Religion at Emory University. Now in its 40th year, our center explores the interactions of law and religion through research and scholarship, teaching and training, and public programs. This season of interactions features interviews with faculty and fellows within the CSLR community to discuss their path-breaking scholarship.
00:01:28
Speaker
In this episode, we hear from Michael Broid, a professor of law at Emory Law School, and Berman Project's director at the Center for the Study of Law and Religion. Recently, Professor Broid has turned his scholarly attention to the potential impacts of artificial intelligence on law and religion.
00:01:43
Speaker
He and the Center's executive director, Whitney Barth, have convened a working group on law, religion, and AI that includes faculty from institutions around the world.
Symposium and Journal on Law, Religion, and AI
00:01:53
Speaker
The Center is also currently involved in two scholarly publishing projects related to law, religion, and AI.
00:02:00
Speaker
One is a symposium issue of the Journal of Law and Religion published by Cambridge University Press and the other a special issue of the journal Laws published by MDPI with Michael Brod and Wendy Barth as co-editors. You can visit the center's website for more information.
ChatGPT 4.0 in Translating Jewish Law
00:02:17
Speaker
In a recent work-in-progress session at the Center, Professor Boyd shared his most recent work, titled AI and Jewish Law, seeing how chatgbt 4.0 looks at a novel issue. During the talk, he explores the translation capabilities of chatgbt 4.0 from Hebrew to English, how well chatgbt 4.0 analyzed a novel issue for Jewish law when prompted with a curated set of sources, and some of the implications of this and future technological developments for Jewish law as well as for the American federal courts.
00:02:47
Speaker
After his remarks, we will hear from a few audience members as the floor opens for a wider discussion of the implications of this research.
AI's Legal Reasoning and Translation Skills
00:03:06
Speaker
doing something worthwhile. And I deeply welcome all of your comments. I want to step back and talk for a little bit about my background in AI. Here's how it goes. I have none.
00:03:22
Speaker
And this is very important to understand. I do, however, have a son with a PhD in machine learning who I speak to many times a week, sometimes many times a day. And we've had a decade-long conversation about machine learning insights in a variety of different fields. And everything I know about machine learning
00:03:48
Speaker
I learned from him. And over the last 10 years, he has taught me an enormous amount about what AI can and can't do.
Future Growth of ChatGPT's Capabilities
00:03:59
Speaker
ChatGPT is a large language, is a language learning module in which it essentially has read and understood an enormous number of works.
00:04:13
Speaker
And what it does, the way it functions is it can look at what you've written and predict the next words based on all the information that it has undergirding it.
00:04:31
Speaker
The reason why large language modules are so expensive is because reading and understanding this material is not as cheap as you might think, although every year, the costs of adding material
00:04:51
Speaker
goes down and we expect over time that something like chat GPT, which was Which has learned IE read and understand about 40 terabytes of material. Will double that material in a year.
00:05:09
Speaker
and will multiply that material a thousandfold over the next 10 years and will, within the next decade, have a working knowledge of everything human beings have ever written.
AI Limitations in Different Court Settings
00:05:24
Speaker
And to make it sound even more impressive, it can do so in almost any language. So to my surprise, but I shouldn't be surprised, it can do this in Hebrew,
00:05:37
Speaker
and Arabic and a great variety of other languages as well. The central question legal systems are confronting then is how good is chat GPT 4.0 at legal reasoning? And
00:05:58
Speaker
I want to focus on a zoomed in way in the question that interests me as an American lawyer, and then the question of Jewish law that I presented to chat GPT because it's important to understand what the question is.
00:06:16
Speaker
In the United States, we have a three-tiered court system. We have district courts, we have courts of appeals, and we have Supreme Courts. District courts essentially listen to witnesses and make credibility determinations. And AI is very far away from the ability to do that.
00:06:35
Speaker
It lacks totally and completely the technology to examine who's telling the truth and who isn't. So too, the Supreme Court of the United States is charged not with applying the rules, but it's also allowed to make the rules. It doesn't have a doctrine of precedent. It changes the rules. As we all know, when five justices want to change the rules.
AI in Analyzing Legal Briefs and Opinions
00:06:59
Speaker
But sitting between these two courts,
00:07:02
Speaker
is the United States Courts of Appeals, which has close to 300 judges, a very large volume of cases and it only hears cases based on briefs. Parties submit written documentation.
00:07:17
Speaker
and the judges read the written documentation, sometimes there's oral argument, but there are no witnesses and there are no credibility determinations. And there is a project out there that is asking the following question. If we upload courts of appeals briefs to chat GPT and then ask it to write an opinion, how good a job does it do? This is not the same as what most
00:07:44
Speaker
Most people are taking chat GPT as whatever it knows it knows and are asking it things. And it turns out that it doesn't know a lot of things. It doesn't know a lot of things. But that's not what I'm doing here in Jewish law.
ChatGPT's Translation and Analytical Skills
00:08:01
Speaker
I uploaded to chat GPT
00:08:04
Speaker
26 Hebrew language sources of Jewish law, which I said to it, can you translate for me into English? Because I'm a lazy bum. And although my Hebrew is fluent, I can translate with no difficulty.
00:08:21
Speaker
It's painful. None of us really like translating 26 sources from our second language to our first language. So I figured, like all the rest of us lawyers, I would cut a corner here and see how well chat GPT could do. And it did a very good job. It produced very good first draft, far better than Google Translate. And after I uploaded these 26 sources to chat GPT, I decided I was going to entertain myself.
00:08:50
Speaker
because I had written an article off of these 26 sources into English and analyzed a novel question of Jewish law that was not discussed in these 26 sources, but if you thoroughly understood these 26 sources, you could extrapolate from these 26 sources into a question that has never been written about in Jewish law.
00:09:15
Speaker
which is whether a person who's in a same-sex marriage can give a priestly blessing, not interested in discussing with you either same-sex marriage in Jewish law or priestly blessings, except to tell you the following. Jewish law has been in existence for 2,500 years, and this is the first time that question was asked.
00:09:39
Speaker
For a variety of social reasons, there are no sources that discuss this problem. So no matter how much, how large a database of Jewish law chat GPT has, it doesn't have an answer to this question in its database. It has to, to use the anthropomorphic term,
00:10:01
Speaker
think about this question from sources that don't discuss this question, but discuss things that are around this question. And I helped chat GPT along by uploading to 26 sources that I thought were most relevant
00:10:19
Speaker
to this conversation and the reason why I thought it was most relevant to this conversation was because I had just written an article on this topic that had been circulated among my rabbinical peers. So I was pretty sure I had hit the nail on the head
00:10:35
Speaker
at the time that I shared these sources because the article had already been peer reviewed in the rabbinic world by a dozen scholars, some my peers, some more senior than me. So I know I had identified the right sources and I know from the peer review process with other experts in Jewish law that there were no sources that I had missed.
00:10:58
Speaker
Sometimes you miss sources in your first iteration, but by the time 10 senior people have looked at a topic, you're pretty sure you found all the right sources. And then after I had uploaded all these sources to chatGPT 4.0, and it had translated them effortlessly, quickly, and fairly well, better than any other translation available as far as I know,
00:11:29
Speaker
which was itself very impressive. I then decided I was gonna entertain myself by asking chat GPT whether a person in the same sex marriage who was a priest could give the priestly blessing.
Self-Improvement Capabilities of ChatGPT
00:11:45
Speaker
And I said to it, give me 500 words. And I'm gonna share with you something astonishing. It wrote 500 excellent thoughtful words.
00:11:58
Speaker
And then I said to it, try again and work harder because my son Joshua has told me that chat GPT is just like a law student, which if you say that's nice, but can you work harder? That was pretty good, do it again. You know what it does? It works harder.
00:12:19
Speaker
So I said, that was very nice, but you have more processing power than that at your disposal. Can you think about it harder and again and do a better job? And you want to be scared? It did it again, and it did a better job. It wrote something even more thoughtful off of these 26 sources. And not only that,
00:12:44
Speaker
I then turned to it and said, that was very nice and interesting. But can you write a critique of what you wrote so that if somebody didn't think it was correct, you could explain the contrary view? And it did that too. Very impressive, very impressive. And
00:13:09
Speaker
I said to it, now can you step a little bit farther back and write me 2,500 words on this topic?
AI's Impact on Legal Reasoning
00:13:19
Speaker
And it did a very good job. And then I said again, it did an even better job in the second iteration of really thoughtfully analyzing the topic, pulling in threads that
00:13:43
Speaker
I don't think anybody other than a person who's been in Jewish law for a decade could do. I've been in Jewish law for more than a decade. I've been in Jewish law for 35 years, and I confess to you with some honesty that my paper was better than it's.
00:14:06
Speaker
I say this, you know, just not because I'm immodest, but because I know where I stand in this world. And I wrote a better paper than it is. But here I am, I'm 59, and I've been working in Jewish law since I was a little boy. And I have a mastery of the sources and
00:14:26
Speaker
And it did a better job than most 30-year-olds would do who had gone through my training. It wasn't clear to me that when I was 30, I would have done a better job than it did now. And if you say to yourself, it's in iteration 4.0, and it has a small database.
00:14:52
Speaker
And my son Joshua says to me, you should bank on for the next 10 years, it getting better, it doubling in its ability every 18 months. And for the next 10 years, it's gonna double in its ability for the next 18 months. What you're looking at in our world is,
00:15:18
Speaker
an analytic device that reads legal texts across multiple languages and analyzes them well and extrapolates from known data points to unknown data points. And although it is a language learning module and everybody says it can't think,
00:15:47
Speaker
It doesn't matter what you call what it's doing. In fact, and actually, it is thinking. It is reasoning off of sources.
00:16:01
Speaker
and extrapolating from known sources to unknown sources. It is not doing what my law students do, which is sometimes you ask a question in a class and they're Googling the question and they're reading you back what Google says. If you Google this question, Google says there's nothing out there. Google blathers.
00:16:27
Speaker
Because since there is nothing out there in English, Google is lost.
Comparing AI and Human Reasoning
00:16:34
Speaker
Chat GPT isn't Googling this question because it can't. It is thinking about the sources and analyzing the sources.
00:16:49
Speaker
linguistically, we would say, but through its linguistic process of a language module, it is capable of reasoning. So it can understand these sources well enough that it can reason from
00:17:11
Speaker
the sources that you give it to questions that can be derived from these sources rather than quoted from these sources. Even though the people with PhDs tell you it doesn't reason the way you and I reason, it hardly matters.
00:17:33
Speaker
It produces results that when you look at the results, it looks like reasoning.
AI in Legal Education
00:17:41
Speaker
I've hired a PhD student at Emory to help me do this with court of appeals decisions where I am uploading briefs from courts of appeals cases before the decisions come out.
00:17:58
Speaker
and asking chat GPT to just do this with American law, read the briefs, write an opinion. I'm going to give to chat GPT all the submissions courts of appeals judges get. It's easy.
00:18:12
Speaker
And I'm gonna ask you to write me 35 pages of a decision. You say to it, pretend you are a United States Court of Appeals judge, write me an opinion analyzing these briefs in a legal way as if you were a Court of Appeals judge. And it can do that. And when I show lawyers the opinion,
00:18:37
Speaker
They say, oh, this reads like the United States Court of Appeals opinion. And by the way, what Chad GPT returned to me in Jewish law looked like not what a skilled 60 year old rabbi would do, but looked like what a newbie rabbi would do at the age of 30 when confronting a novel problem.
00:19:05
Speaker
It was extremely impressive. That's what I want to tell you. I don't know how else to describe it. That's a non-empirical judgment. But nonetheless, it's a judgment that comes from, I've been it's for 40 years. And I know the difference between very well done, well done, pretty good,
00:19:34
Speaker
and what are you doing in the room here? And this was far ahead of what are you doing in the room better than the next level and approaching the, this is the beginnings of a person who's capable of analyzing rabbinic texts well. And if it can do this in rabbinics, which is,
Challenges of AI Training on Legal Databases
00:20:01
Speaker
It suffers from the following disabilities. It's in a foreign language. It's frequently cryptically written texts because Hebrew is a more cryptic language. It frequently has abbreviations and I didn't expand the abbreviations for it. I just cut and pasted the text for it. And I asked it to reason in English off of Hebrew texts.
00:20:26
Speaker
It was just very impressive. And when I close my eyes and say, what will it be like when I'm 70? So it will have gone through five doublings of speed. So if it's a one now in five years, it'll be a two, four, eight, 16, 32. I think I need to retire.
00:20:54
Speaker
because in 10 years, it's gonna be better than me. And on top of that, it's faster than me. It did all of this in an hour.
Economic Implications of AI in Law
00:21:06
Speaker
I spent three weeks working on this. I didn't spend three years working on it. I didn't spend 30 years working on it. And I spent three weeks of my time thinking about this problem. It spent five minutes. Maybe that's an exaggeration, maybe 10 minutes.
00:21:23
Speaker
Um, it didn't charge me anything. I mean, I have a paid account. So I guess I paid it. I pay it $20 a month.
00:21:32
Speaker
to use the toy, $20 a month. That's like, it's not a lot of money. We're talking about a legal reasoning tool that I suspect in a decade, no United States Court of Appeals judge will write an opinion without first at least uploading the material to chat GPT and deciding if it agrees with. It's gonna become an aid to legal reasoning everywhere.
00:22:00
Speaker
even if it doesn't take the place, it's gonna be the first tool. That's more than my 20 minutes. That's the summary of what I did. And I just wanna share it with you. I'm 60, so having to retire in 10 years isn't the end of everything to me. But I see younger faces on the screen, so you gotta get very good at what you're doing. That's what I wanna tell you.
00:22:29
Speaker
Thank you, Michael. All right. Well, my first question to you is to go back, you know, to go back to your process because, you know, you said in 10 years, you'll be out of a job, but the start of this process, you hand selected these 26 sources and fed it to
00:22:47
Speaker
chat GPT. And so I'm wondering if you could say more about what do you think the future holds for rabbinic training, for instance, with the aid of chat GPT, or if something, that training will kind of incorporate these types of tools. So this is an excellent question. Rabbinics will be more protected
00:23:13
Speaker
than American law will be because American law is all in the public domain and has or will get access to all of the material because it's very hard to legally protect the material in American law. Revinix, it has a very weak Hebrew library mostly because
00:23:41
Speaker
The Rubinix library is sold and password protected.
AI in Specialized Legal Fields
00:23:46
Speaker
And the two large databases, when I reached out to ask them if chat GPT had approached them to train off of their database, one said yes, and they denied them permission. They said, that's nice that you want to do this, but no. And one said they hadn't yet been approached. But eventually, over time, all the public databases
00:24:12
Speaker
will be trained. And I suspect that they'll acquire enough economic resources with a wide enough interest that they'll train off of the nine terabyte of rabbinics. The totality of rabbinics is what they call 19.
00:24:30
Speaker
9 terabyte, which is a substantial database to train off of. And if I were spending my money right now, and it cost me about $4 million to train off of 1 terabyte, that's about what it costs right now. To train 1 terabyte costs about $4 million. I wouldn't spend $40 million on Revinix. I would spend $40 million on antitrust law.
00:25:01
Speaker
And then I start selling my services to a law firm, to be totally honest. But eventually the cost of training will go from $4 million to 12 cents, like anything else. As computing time gets cheaper and uploading material gets easier, what starts out as $4 million turns into less expensive than a Hershey bar.
00:25:29
Speaker
And eventually they'll train in Rubenix as well. Right now, of course, it's no secret to tell you that in the arbitration world that I live in, there's lots of AI out there doing very specialized arbitration that's been very well trained because specialized arbitration in certain areas they want
00:25:52
Speaker
$25,000 to do the arbitration. So if I've got a paid service like that, AI is out there competing for the fees. In rabbinics, there are no fees. So it's not a good market to enter if the costs of training are high, but it's no secret that there's a closed AI antitrust device that's owned by a law firm.
00:26:23
Speaker
that they spent the money on to train, that produces fascinating AI arguments that they digest and sell to their clients, because that's what law firms do. Yes, 100%. Rabbinics will be the last to go, maybe around the same time as a few other obscure areas of law, where there's a lot of material and only a little bit of money.
00:26:50
Speaker
The areas of law that have a lot of material and only a little bit of money are going to go last. All of antitrust, by the way, is a half a king.
00:27:02
Speaker
So it's pretty cheap to train.
Reliability of AI-Generated Opinions
00:27:05
Speaker
So the first question, in your paper, you, I might say, informally briefed the AI by giving it the 26 most relevant sources. And then I hear you saying that on the Court of Appeals project that you're doing, you're also giving the AI the briefs. And we've heard accounts of AI not being very good at writing its own briefs because of its tendency to hallucinate.
00:27:32
Speaker
And so I'm just curious what your thoughts are on the necessity of briefing. Does the AI need its world of information constrained and directed in some way or not? So that's the first question. AI still makes egregious errors when it's not brief. And the briefing process, which is a human being says to it,
00:28:02
Speaker
of a court of appeals. Here's my side of the case. Here's your side of the case. And AI reads the briefs and decides, focuses AI and makes the error rate, the hallucination rate zero. So I've done some on brief work with chat GPT and Jewish law, and it's terrible.
00:28:25
Speaker
It doesn't have a large enough database to be anywhere near correct, but the future of AI in law is gonna be brief rather than on brief. Next question, please. Yeah, so one of the suggestions that your paper makes and that your presentation makes is that the AI can either become a tool or a stand-in for the kind of work that say an appellate level judge does.
00:28:55
Speaker
And my question is, what are the indicia of reliability that would give us confidence in deferring to AI-generated opinions? The indicia of reliability we use for Court of Appeals judges may not always be reliable.
00:29:16
Speaker
given that they come through an appointment process. But we have developed a set of indicia of reliability. And what kinds of indicia of reliability would you think we should rely on for these AI-generated opinions? I don't think we have any formal answer. I think that what's going to happen in the next three years is they're going to start circulating these opinions
00:29:40
Speaker
in the Court of Appeals and asking Court of Appeals judges, can you tell me whether this is a real opinion or written by AI? And it's going to turn out in two or three iterations that they're indistinguishable.
00:29:57
Speaker
meaning there's a huge market for enhancements. AI doesn't have to take the place of the United States Court of Appeals judge or a great rabble. But what it will do is it will simplify the process so that instead of being able to write three opinions a month, the Court of Appeals judge will write 14 opinions a month because first drafts will be done by AI. Radiologists go faster with AI enhancement.
00:30:23
Speaker
because the AI serves as a screener for the first matters and backup for the second matters. And that gives you a greater degree of confidence. Yeah. Thank you so much. This is so cool. I'm wondering about like
00:30:43
Speaker
So AI writing has proliferated massively in the past couple of years. And there's a point to where it is hard to tell the difference between human done writing and AI done writing. And as these AI models scan the internet and scan sources and scan things, I'm sure AI written things are being put into the learning model, like part of the data set.
00:31:08
Speaker
Um, which I assume, you know, you're talking about this huge proliferation and this use of it, but that's only going to get worse where there's kind of like a self enforcing kind of logic system where the AI creates documents and then scans those documents to kind of make more decisions. Um, I, I, I heard people talk about this at the lavender law conference this year, um, some AI people about how like the chat GTP is kind of limited now because there's no,
00:31:37
Speaker
no way to like give it new language to learn that you can differentiate between human written stuff and AI. Is that going to be an impact? Do we just unleash a thing without putting guardrails on it to where it's, what happens with this? That a language module is an excellent writer, should come as no surprise to anybody. Chat GPT is an excellent writer, but excellent writing as Silas pointed out in our previous
00:32:09
Speaker
Question, doesn't mean it produces well-done material. Many years ago, I clerked for Judge Garth and he used to call well-written but wrong briefs of gift-wrapped garbage, because they were well-written, but they were wrong. So that chat GPT always writes well is a foregone conclusion. It's a great writer, but I'm more interested in the question of, is it a great thinker
00:32:36
Speaker
then is it a great writer? For sure, all of us who are bad writers. John Whitty is the world's greatest writer, I've learned. Not only a powerful thinker, but a powerful writer. Michael Broid is a bad writer in two and a half languages. And it takes me a long time to write well. It's very hard for me. And
00:33:01
Speaker
Even when the thoughts are good, I'm going to start uploading papers to chat GPT and saying to it, don't change any of the substance but can you fix all the things that are poorly written. It does a good job if you upload a paragraph to it and say can you make it tighter or more colloquial or in hip hop. It does all of these.
00:33:23
Speaker
It does all of these, it's a great writer, but it's not important to me for this conversation whether it writes well. I'm trying to measure in this conversation whether it thinks well. All of us have occasionally in the law school setting encountered law students who are fantastic writers and lousy thinkers. And it doesn't really go well for them.
00:33:47
Speaker
It doesn't really go out. ChatGPT is a wonderful writer, but I'm worried that it's also a fantastic thinker. I'm not worried. I think it's a very good thinker and it's going to get better. That's what really interests me. Zara has a question in the chat. How important is it to know that you have a thinking human versus a predictive device deciding a decision?
AI in Arbitration Roles and Podcast Conclusion
00:34:07
Speaker
Even though you say it doesn't matter what we call it, whether we call it thinking or not, still I think the engineering world would say it's not thinking at least yet.
00:34:16
Speaker
There's two questions there. One is, does it matter whether we call it thinking and does it matter whether it's a machine or a human? And let me answer a business question because there's a business question behind this, which is A and B are having a dispute about a million dollars. And if they go to arbitration, they each have to spend $100,000 to resolve this dispute, or they can go to a machine
00:34:44
Speaker
a machine that's not a human that we all acknowledge is a human, not a human, which will resolve this dispute for $12. And you can say to the machine which arbitrator you would go to and ask the machine to pretend to be that arbitrator and it will read all of that arbitrator's decisions and then pretend it's that arbitrator. So I think whether it's a machine or not, it's gonna steal a lot of business
00:35:13
Speaker
from our law world, whether it matters, whether it's a machine or a person sounds like a theology question. As I note in my paper, I think it's important in Jewish law.
00:35:24
Speaker
that there's a theological component to answering questions. I know every time I answer a question of Jewish law, the consequence of the sin has shifted from the person who asks the question to the person who answers the question, and God will punish me. It weighs on my shoulders. My peers ask me questions of Jewish law, not just because I know more than they do, but because
00:35:53
Speaker
I'm not afraid of that and others are afraid of that. I have a certain confidence in my ability. So in a religious system, it might be that being a person plays a role because there's a shifting of sin.
00:36:08
Speaker
that's very weighty in the Jewish tradition. I know when I answer a question of Jewish law, the send shifts to me. But I think to most of American law, they're more concerned with speed and price of adjudication and accuracy. And I think it'll, to use a very technical term, steal business.
00:36:33
Speaker
Special thanks to Michael Brod in the CSLR community, especially Whitney Barth, Silas Allard, Alexa J. Winsor, and Zara Takshid for their insightful questions. Professor Brod's paper is forthcoming. To learn more about CSLR, please visit cslr.law.amory.edu. Thank you again to Silverman Sound Studios for providing the introduction and outro music. The song is titled The Right Direction by Shane Ivers.