Polk Wagner
Subject: How AI Will Change Law Schools, Law Firms, and Users of Legal Services
Bio: Deputy Dean of Penn Law School
Transcript:
Larry Bernstein:
Welcome to Happens Next. My name is Larry Bernstein. What Happens Next is a podcast that covers economics, politics, and history, and today’s episode is how AI will change law schools, law firms, and users of legal services.
Our speaker is Polk Wagner, who’s the deputy dean of Penn Law School, who has been charged with figuring out how to adapt the law school curriculum to the AI revolution. I want to hear from Polk about how law schools will adapt as their graduate’s jobs will no longer focus on document creation, but instead more complex legal analysis.
This podcast was taped at a conference that I hosted so many of the questions will be asked by my friends. And in the audience was David Stellings, who’s the managing partner of law firm Leif Cabreze. I want to learn from David about how AI would change the structure of law firms as the work process will need to change and I want to find out if the by the hour billing practice will end and what will replace it.
I also want to hear from my friends who use legal services about how they plan to use AI to negotiate contracts, to understand complex legal agreements, and to draft documents that previously would have been delegated to their attorneys.
Polk, can you please begin with six minutes of opening remarks?
Polk Wagner:
This young transactional lawyer named Zach Shapiro had a small practice and he created a viral post called the Claude Native Law Firm where he described how he was able to replicate functions of a huge array of legal associates legal work, using mainly off-the- shelf Claude tools.
It does document review, it drafts, and it revises based on comments. He stays in the loop on the judgment calls, but really the work is being done by Claude. Artificial intelligence technologies are extremely good at text. Lawyers do a lot of interacting with documents and so it seemed only natural that this was going to be an area where there were significant disruption and change, but until I think this winter that was mainly theoretical. Now it’s coming more quickly than any of us thought. Giant law firms are probably 12 months behind. There’s a lot of friction involved in getting these tools to work in the real world. Scholarship by its nature takes years to get through the publication process.
And an even slower step than scholarship is the teaching. Law is by its nature a very conservative and slow profession. That is a virtue as well as a detriment, but that makes it even slower in law school. We teach people by making the students do things.
It’s the repetition, the writing of the briefs, the writing of exams, the papers is how we teach people to work through structured arguments, reasoning, judgment, assessing under conditions of ambiguity, all important lawyerly skills. We are seeing from these AI tools that the work piece is being mostly replaced. That raises a whole bunch of existential questions for a law school like mine and for law firms How should we be teaching students? We know that there are going to be a bunch of things that lawyers do that the AIs can’t, they’re not good at judgment, reasoning, analysis, people skills, all of these things that lawyers do that are important and probably going to become even more valuable in the future.
Traditionally, that’s not something we teach specifically. . Instead of producing entry-level lawyers, we need to be able to produce associates at a fifth-year level. People who can supervise the creation of work rather than doing the work themselves. They need to be able to supervise teams of AI agents and make judgment calls and choices almost from day one when they come into the firm. I’m doing a sort of a small-scale test with some of my colleagues where I have my AI bots write exams and I slip them into the pile of the exams that my faculty colleagues get and they grade them.
There have been four so far and the AIs have finished first in every single one and not by a small amount either. I’m doing it primarily as a wakeup call to my colleagues that we need to rethink how we’re assessing.
This particular study is offering AI its best possible advantages because it’s under high stress, tight time conditions, strict word limits. An unnatural situation where the AIs are going to be exceptional. It writes so much better than a law student under extreme time pressure. I don’t think that necessarily says that there will be no more lawyers. That’s interesting to think about as a pedagogical tool. Our toolkit has always been these exams and they’re not what we should be testing anymore.
Jay Greene:
Is this conversation among your colleagues about reducing the number of students you admit or perhaps eliminating your school or other schools like it? So even Education schools, there’s very little demand for new teachers relatively. I would try to convince my colleagues that we should shrink and they thought this was the most monstrous thing.
Polk Wagner:
That would not be a popular thing for sure, but you’re exactly right. There’s going to be significant displacement in the legal market. The question is timing. The roles of particular lawyers are going to become much more valuable. Those that can apply reasoning, judgment, handle ambiguity, all the things that AI can’t really do. Penn may be fortunate enough to be on the correct side of that curve, given the type of student profile we typically have. If I was predicting, I would say there are going to be fewer law schools for sure.
There are predictions about whether there’s going to be any more hiring of software coders anymore. On the other hand, there’s going to be a lot of demand for software because it becomes inexpensive and there’s going to be people who need to manage those systems. There might be more coders. We might see a similar phenomenon in law, the demand for legal services and access to the legal system is such that when the costs come down, you need more people to sort of manage that process.
Rory MacFarquhar:
Watching the guy discuss the two-person law firm, one version of it is these two people become incredibly valuable, but the other version says, fast forward 24 months and you have a zero- person law firm. The companies can do all the legal services that they need.
Polk Wagner:
One of the things that we are planning for is what we think of as legal work largely going to be done not at law firms in the future. It’s going to migrate into various places, maybe inside firms themselves where they can do versions of DIY legal work relatively straightforward. You can really harness these materials to allow a small group of in-house lawyers to run what used to require significant outside counsel. I still think there’s going to be roles for high-end lawyers at firms that are very specialized, but I do think that what we think of as legal practice is going to migrate into a whole bunch of different arenas.
We need to think about is how to teach our students to succeed in all these contexts. Whether they’re in-house or in the government where they’re not traditional lawyers, but they’re still doing legal work.
David Stellings:
My partners on the executive committee, there was a raging debate going on how we should change our AI policy. At the heart of that debate is the question of whether associates should be allowed to use AI to create a document and then check the document and improve the document or whether they should be required to draft the document themselves as lawyers have for hundreds of years. I think that they’re going to use AI regardless of whether we tell them they’re allowed to use AI. If they use AI, how are they ever going to become good writers?
As litigators our success is based largely on our ability to argue persuasively in writing and verbally and if they never develop that skill, how are they ever even going to be in a position to supervise the AI agents?
Polk Wagner:
We restructured the 1L curriculum this year for legal writing and they had no access to AI for the first two months and were taught the very beginning, how you write a brief,and then starting in the second semester they were given more and more access to use AI tools and then taught how to use them effectively. I mean, that’s where we are now, which is you got to learn how to do it.
David Stellings:
It takes so much longer than two semesters to learn how to write. I’ve been doing this for 30 years and I’m still learning how to write better, more persuasively. My wife who’s also a lawyer, we were talking about this last night and she’s like, “Well, maybe lawyers won’t need to write anymore.” What do you think about that?
Polk Wagner:
I have a colleague who does contract law whose hypothesis is that we won’t have contracts anymore. You just have your AI agent talk to someone else’s AI agent and they’ll work it out amongst themselves given the parameters that they understand about the business.
Larry Bernstein:
I had a golf outing the other day and David Brail created an app which kept score, handicap adjusted, skins, everything you can imagine. David has no skills whatsoever, period. It was amazing. How did David pull this thing off? He said it was pretty easy. I just told it what to do. Someone who has no skills to be able to get an A+ result.
Polk Wagner:
You understood immediately whether what the AI was doing made any sense or not and you understood the structure of the thing enough to know, “Oh yeah, these are the things I need to look at. These are the things I don’t.” How do we create these experts in understanding where the minefields are?
Coding is different because coding is very clear whether it works or it doesn’t. You run it and the thing crashes or it doesn’t. That’s not quite the same as what we talk about in law, particularly with long running agreements between corporations where the answer may be 15 years down the road some edge case arises that nobody ever considered.
Larry Bernstein:
Example, I have a will and a trust and these things are complicated. You sit down with a lawyer, you lay out all the conditions, they hand you this very extensive legal agreement. I’m sophisticated, but I don’t understand what’s going on in this document. I tell my parents that they should sign the same thing. And my mother says, “Larry, do you have any idea what this is? “ And I said, “Well, just sign it.”
Lawyers in general are very poor at articulating complex documents. We must take this leap of faith that this document reflects the needs and desires of this family. What AI can do is unbelievable. It’ll explain in plain English exactly what this thing does.
Polk Wagner:
AI tools make it so much easier to become familiar with subject matter areas that you’re less familiar with and it can draft simple documents. One of the roles that lawyers have always played is trust. The reason that you just sign whatever they put in front of you is because you trust that the lawyer that you’ve hired. Lawyers can provide that advice and trust to guide you and that is something I think that AI can’t necessarily replace, at least not in the foreseeable term.
We need to train the lawyers who can take the summary that you bring to them and engage with you in a way that gets to the point quickly, and gives you trust and reassurance that what you’re doing is going to meet your needs. That’s where the legal education and the practice of law need to change because traditionally, we’ve favored lawyers who can do the thing not talk about the thing. You do get the occasions where the lawyers don’t communicate well, but I think that’s going to be part of what we need to be teaching lawyers.
Mitch Feinman:
My wife is a trust and estates partner at a big law firm and a third of her job is being a therapist. If you look at the most common use of AI right now, therapist is number one on the list
and growing. I observe how my wife and her firm do things in terms of hardware and equipment and processes and I’m astounded. Talk about laggards on the technology adoption cycle and big law firms. I know you’re taking the perspective of graduating law students, but you’ve got people running these law firms who will be running them for a long time who are so behind on technology adoption. Do you believe they’re just going to have to adapt despite not being well
equipped to do so?
Polk Wagner:
That would be my expectation. I do think there’s a long transition period here. There’s lots of scholarship on the way that technology diffuses through the economy. It takes a long time. There’s an enormous amount of friction. The bots can create simple legal documents, but it’s not legal for them to do so right now because of various bar association rules. The firms are set up to maximize the amount of work because that’s what they can bill and that’s going to change. The question is how quickly and Walmart sent out a list to all their outside counsel f with things that were no longer billable because we want you to do it with AI.
If you’re a big, slow law firm and that letter comes from your major client, that is a serious wakeup call that is going to force change much more than the firms arguing amongst themselves. Once the clients start saying, “I will not pay anymore for the stuff that AI can do well,” then the firms are going to have to change.
David Stellings:
Most of these lawyers are getting paid by the hour and it seems to me like this is going to impede their ability to bill as many hours as they’re used to billing.
Polk Wagner:
100%. And this is going to dramatically force many law firms to change the way that they are doing business. How do we make it so that the students that we graduate are going to be successful in these new models? We’re adding to the curriculum a lot of courses on how to think about new models of legal practice, whether it’s contingency fee firms or modified fees or flat fees or all the different ways that don’t rely on the hours of work because that’s going to be gone.
Penn Law primarily sends lawyers to large New York City law firms. At least those firms appear to have made no changes at all in their hiring approach.
David Stellings:
Anecdotally, some of the top law firms are considering applying in the litigation context, which is where billable hours have reigned, a similar paradigm that they apply in the deal context with PE where they get a percentage of the deal amount either in a flat fee arrangement or even some hybrid with a contingent element.
Polk Wagner:
If law firms are going to survive, they’re going to have to have different business models andI do think we’re going to be seeing law being done in a lot of different places.Our students are going to end up in jobs that are not at law firms, working inside companies or for regulatory organizations, rather than at a traditional law firm.
This dramatic reduction in cost is going to ripple throughout the way that law is done.
Polk Wagner:
I think there’s enormous opportunities for truly dramatic public benefits. You could imagine landlord-tenant law. The landlords take advantage of people who otherwise don’t have access to the legal system or lawyers. You could imagine a city like Philadelphia setting up an AI system that resolves these disputes at pennies on the dollar and dramatically shifts the balance of power between landlords and tenants to the benefit of people who probably need that access to the legal system who don’t get it right now. We can’t even see yet all the ways that this is going to change the way that we think about the legal system.
I have a project right now with about 25 federal judges and my team of RAs and I work with them to explore how AI can improve litigation, and they are seeing a dramatic increase in filings because it’s very cheap and easy to file. They’re seeing this among prisoners at this point because they have access to ChatGPT in the prison library and they make very compelling briefs that sometimes make good points at a pace and scale that the system is not designed to deal with.
David Kostin:
Is there a thought process of how the sentencing process may use AI?
Polk Wagner:
There’s an enormous literature around biases in AI. AI models represent all of us and because of that they have all the good and bad parts of us. It has been documented in many cases that without significant guardrails, these models can behave in ways that are socially detrimental because there’s a lot of socially detrimental information that they’ve been trained on. The ethics of using these tools for sentencing is a big open question. There are courses at the law school right now exploring that and nobody knows the answer. The opportunities are tremendous in terms of the pace and scale at which you could resolve disputes. People sit for months waiting for their sentencing, why could we not do that in a day?
There’s the opportunity for those systems to be much more accurate than a judge guessing what the right answer is, but there’s also concern that these systems are not judgment machines. They are at best knowledge machines. What they do may look like judgment, but they’re not exercising any human judgment and where you keep the human in that loop is an interesting question.
Bryan Verona:
I take exception to your characterization of landlord tenant power structures. It’s not that simple. Time is the more valuable against landlords and the system is horribly clogged, at least in New York. Which clearly doesn’t need to be the case. his idea that criminal cases, certainly civil and/or litigation can be argued by bots conceptually overseen by a judge bot. Why do we need people? Who gets to calibrate the judgment and the sentencing? Why are theatrics in court relevant? Why are biases of judges relevant? Doesn’t this streamline these processes and certainly speed them up?
Polk Wagner:
I think that there’s a lot of opportunities for streamlining at every place in the legal spectrum. One of my colleagues is finishing up a study where he created two pools of humans, each with different political affiliations and asked them to interpret the contract terms. They gave different answers for those pools and then he asked an AI to also interpret them. These were all known terms that had been decided over many years by courts, and the AIs were always better. AIs don’t have those sorts of biases in them and obviously can be much faster. What is the role for humans? I think for a long time we’re going to need humans as trust and accountability markers.
Would you trust the outcome of a dispute you had to just pure AI over a judge ultimately making a call?
Bryan Verona:
Probably.
Polk Wagner:
Because you think that the AIs have less bias? or- Humans are
Zachary Schur:
Humans are imperfect things.
Larry Bernstein:
Our next question comes from Moira McDermott who a Math Professor at Syracuse is.
Moira McDermott:
I wanted to go back to the question of how much training do you need before you can evaluate and be a supervisor? We must have a statement on our syllabus about AI use and it can be of no use in these limited situations or go hog wild. I’ve always picked that middle one and tried to lay out when I thought you’d be able to use it. You can use it to help you study, making practice quizzes, that sort of thing. I tried to use Claude, ChatGPT to make extra practice problems for exams for my students and it was hard.
I give it my old homework assignments or old exams, and it just doesn’t do a very good job because like if you just change the numbers, either it could be very messy computationally or it misses the point. There’s a particular purpose for a question that I ask and there’s something in the problem that must work a certain way to be a good measure of the thing I’m trying to get at.
Polk Wagner:
The AIs do not have any sense of what problems they should be resolving. That is where the real skill of an advanced knowledgeable person matters a ton.
Moira McDermott:
The real thing is being able to ask the right questions, not answer them.
Polk Wagner:
Exactly. That is going to be a very durable skill for humans in the legal context. How do we get students from zero to a point where they can ask the correct questions? As someone in charge of figuring this out at a law school that is the big existential question that I worry about.
Rory MacFarquhar:
What are the legal and institutional protections on the role of human lawyers that would require actual change to eliminate?
Polk Wagner:
Various bar associations have taken various positions on the use of AI. Most end up along the lines of if you use it, you’re entirely responsible for anything that happens. Various judges have created their own local rules that apply to trials either in their venue or in their courtroom. Those range from, I don’t want to see anything that has any AI content at all to, if you use it, just don’t screw up.
The weird case by having a client ask legal questions of an AI that was deemed to be violating or piercing the lawyer client privilege, that seems like a one-off case. There’s a lot of fear around how these cases could turn out because as lawyers, unless we see something decided by a court, we don’t know what the answer is yet. There’s certainly a lot of regulations about who can be a lawyer and what you need to be a lawyer. I do think more lawyers are embracing the fact that these can be powerful assistance at this point more so than making the final decisions.
Paul Yablon:
People lie and part of the judge’s job is to determine who he believes and what is the truth. When we use AI, we are careful about the data sets we use, and we scrub the data because that is imperative to get the right outcome. If we knew that the data going into the decision making process was true, I would feel more confident that AI would make the right call. How do you deal with that?
Polk Wagner:
The more confident you are that the facts are going into the decision are the source of truth, the more confident you are that the output decision is going to be correct. The whole point of our litigation system is to surface what is as close to the ground truth as possible. Cross examination tries to prevent the lying or at least address it.
The judges that I’m working with have a lot of concern about the data getting worse. Traditionally we believed photos are compelling, or a recording of somebody’s voice. This is no longer ground truth. None of these things can be determined to be truth in the way that they were even six months ago. Lawyers and judges are seeing a lot more evidentiary challenges and concerns and incredible ambiguity about what is real, what is not real, because we have not gotten our arms around yet how to distinguish between something that’s AI created and not AI created.
It’s going to be an enormous challenge to work through that we’re not yet equipped to do. :
All of the major AI platforms have various proposals for how to tag or otherwise identify AI created artifacts as such. The judges I talk to are seeing this daily.
Larry Bernstein:
Our next speaker is Steve Kuhn who is the founder of Sportspredict.com.
Steve Kuhn:
Shani and I started sportspredict.com, where we have people predict sports events and we give rewards and status to people who are good predictors. I want to propose a potential idea for the world of law. Let’s say you have a case in front of the Supreme Court, and we’ve got tons of data on how every Justice Supreme Court has ruled on various cases before. We put all that in and we create a bot for each one that tries to mimic how they would decide a case and best predict how they would decide a case. Then as the legal team, your job is to come up with the brief and the set of arguments that maximizes the odds that you win. It becomes like law predict. How would you know whether they’re good? You would know by their ability to judge in historical cases or current cases what the odds were. We would assess them by saying, “Here’s a case for all nine justices, give us your odds.
Polk Wagner:
There’s a lawyer who has a blog that is doing something very similar\ He has a set of AIs that will write Supreme Court opinions and they are shockingly close to the ones that actually get written because it’s given enough data, it is relatively predictable how these cases turn out and the writing styles are easy because you’ve got a huge corpus of writing. So you can say, “Oh, here’s what Justice Thomas is going to write on this case.” And it’s almost always shockingly correct or shockingly close to correct.
Larry Bernstein:
One of the reasons we love arbitration is because it’s so much faster than going through the courts. I was involved in arbitration, and it ended up taking a long time, costing a lot of money. It surprised me. I have a feeling that if this arbitrator was required to use AI that this could speed up things.
Polk Wagner:
There’s a market for arbitrators, right? There’s going to be a lot of incentives and pressures for the arbitration firms to get efficient. The reasons that they are a popular option is that they can do things much more quickly and cheaply than traditional litigation. That is an area where you’re going to see rapid development. I would view the arbitration industry as a leading-edge indicator of when more automation of the law starts to occur.
One of our alums who started an entire AI law firm has identified employment law as an area of low hanging fruit for AI because the terms of those contracts are relatively detailed, straightforward and well worn. The AIs can do a good job of assessing the quality of agreements and how they’re going to play out.
Larry Bernstein:
I’m interested in how contract relationships between firms and individuals will change. With AI, we can negotiate bespoke user agreements.
Polk Wagner:
I have a browser extension in Chrome that I use that lights up different colors depending on how egregious the user agreement is of the website that you’re on. We are halfway there in having the identification of these things. Whether that makes the market much more efficient for me than negotiating with the websites to get my own version of the agreement, I don’t know. The next step would be that my browser negotiates with the server at the other end and says, “Here’s the set of agreements or set of terms we can agree on, “ and then we go from there.
Extended warranties are a version of that. People get to choose the product with the various menus, and you might have a longer or shorter warranty and pay more or less. There’s a lot of friction because there’s a lot of incentives on the part of the current incumbents to maintain this structure for as long as they possibly can. And it’s about when the market pushes.
David Stellings:
Consumer contracts almost always are contracts of adhesion, meaning there’s no negotiation. You accept their terms and conditions. I don’t see that changing anytime soon.
Larry Bernstein:
Contractual relationships are going to be in flux. The reason we have lawyers is because the businesspeople don’t really understand what’s going on, what happens is lawyers fight, fight, fight, and then they come to the businesspeople and say, “We’re fighting about this. What do you want me to do? “ And we can skip a step and just say, “Here are the issues, businessperson, what do you want to do? “ And then you can lay it out instead of having these teams of lawyers negotiate back and forth for days and just get to cut to the chase.
David Stellings:
There is one way those terms and conditions and consumer contracts change, which is through litigation. For many years had in its terms and conditions a forced arbitration provision. Forced arbitration provisions essentially prevent groups of people from gathering and suing companies and if they can’t gather, there’s no economically feasible way to stand up for themselves. But so Amazon had this arbitration provision and a smart firm said, “Okay, well, you want to go through arbitration, I’m going to market.” They got like 15 million bucks from a funder. They gathered something like 80,000 Amazon customers, all of them filed notices of arbitration.
Amazon wound up having to pay arbitration fees. Forget about whether they won or lost. Amazon was on the hook for like a thousand bucks per person who was filing for arbitration. Amazon removed the arbitration provision from its contract. Complaining’s not going to do it, but maybe that way.
David Stellings:
I was a defense lawyer for about five years and I’ve been a plaintiff’s contingency lawyer for almost 30 years. It’s so much better. I’ve always thought that the billable hour model has perverse incentives for the lawyers. Most lawyers are ethical people and are not intentionally spending additional hours doing work so that they could bill a client. But I do think that lurking in the back of their heads is the idea that if my group doesn’t bill $5 million this year, then our bonuses are going to be slashed relative to these PE guys who are making $30 million a year. I do think that it influences behavior. It makes them less likely to approach the other side even when they know that they’re eventually going to have to settle because it’s contrary to their business model.
I think this AI revolution is likely to finally push the billable hour away from the center of gravity of the legal system in the US.
Polk Wagner:
I think this is terrific because being a lawyer on a billable hour model sucks and it is a terrible lifestyle for young lawyers who feel like they’ve got to bill more hours, and the focus becomes on how much you work as to whether you’re doing a good job. I think that has a few perverse incentives, not only economic, but on people’s personal health. I think one of the interesting things about that viral Zach Shapiro essay was about how much better his job was now that Claude was doing most of the work. Because he could make the big decisions, the calls, work with the clients, and someone else was dealing with the writeups of the documents that’s so tedious and makes your job less fun as a lawyer.
Using Claude on a constant basis has made my job just so much more fun. I can do more, I can do better, I can put together a mini course in the morning instead of spending weeks on it.\ It’s a different model and frankly, I think it is better.
Law firms are going to have to change the practice of law and what we think of as law is going to have to change. There are enormous benefits both in the joy of doing what you love as well as monetary and economic benefits. I’m optimistic, but not without some clear-eyed views of the challenges ahead.
Larry Bernstein:
Thanks to Polk for joining us.
If you missed our previous podcast, it was fixing our healthcare system. Our speaker was Josh Gottlieb, who’s professor of economics at the University of Chicago’s Harris School of Public Health.
I want to learn from Josh about what are the barriers to educating more doctors and what we can do to solve the seemingly intractable problems to increase quality and productivity of our healthcare system.
You can find our previous episodes and transcripts on our website. Please follow us on a podcast or Spotify. I’m Larry Bernstein with the podcast, What Happens Next.
Check out our previous episode, Fixing Our Healthcare System, here.
