20 Top Lawyers Were Beaten By Legal AI (hackernoon.com) 123
An anonymous reader shares a report:In a landmark study, 20 top US corporate lawyers with decades of experience in corporate law and contract review were pitted against an AI. Their task was to spot issues in five Non-Disclosure Agreements (NDAs), which are a contractual basis for most business deals. The study, carried out with leading legal academics and experts, saw the LawGeex AI achieve an average 94% accuracy rate, higher than the lawyers who achieved an average rate of 85%. It took the lawyers an average of 92 minutes to complete the NDA issue spotting, compared to 26 seconds for the LawGeex AI. The longest time taken by a lawyer to complete the test was 156 minutes, and the shortest time was 51 minutes.
Shamelessly Stolen...from here I think (Score:5, Funny)
A computer once beat me at chess, but it was no match for me at kickboxing.
Re: (Score:2)
I am sure there is a Claude Van Damm robot in the works that will both play chess and make you bleed a lot.
Chess boxing robot (Score:2)
Success at MMA or any other contact sport largely depends on the robot built around the computer. Does the chess boxing [wikipedia.org] match in this joke take place before or after Boston Dynamics and its line of doglike robots?
Re: (Score:2)
Market Speak (Score:2)
Ya...AI has become Market Speak these days.
Anything that can do something faster than a person is, "AI".
Re: (Score:1)
Ya...AI has become Market Speak these days. Anything that can do something faster than a person is, "AI".
"Do" as in: do the thinking part better or faster than a human. Hence the intelligence in AI. And yes it's artificial - it doesn't grow in nature, humans put it together.
So AI is very much appropriate in this context, even if used as marketing speak.
Re: (Score:2)
But it isn't "intelligence".
It's probably just a fancier version of a spell checker essentially.
But we don't know because nothing has been explained, at least publicly.
Re: (Score:2)
But we don't know
That didn't stop you from speculating though.
Re: (Score:2)
The whole damned story is speculating.
We Need a Structured Legal Language (Score:5, Interesting)
Seems like most court cases involving contracts and interpretation of the law are mostly about unpacking the Bullshit written into the contract or law. It seems absurd that one of the functions of the court is to divine the intentions of the law's author.
A law can be complex in that there are many moving parts that can interact in different manners, but the parts themselves and the manner in which they can interact should be clearly defined and not subject to interpretation.
Re: (Score:2)
Re: (Score:2)
It seems absurd that one of the functions of the court is to divine the intentions of the law's author.
You have to have something like that otherwise how do you resolve the conflict between two competing interpretations of the same text?
Re: (Score:2)
What I'm saying is that there needs to be a methodology of creating this stuff that makes two competing interpretations of the same text a thing of the past.
You're one up on me (Score:2)
Shakespeare would be proud... (Score:2)
Lol (Score:1)
What is an "issue" with an NDA? What is the "right" NDA. You can't make objective measurements about this. The lawyers actually think and reason, the bot does what it is programmed to. Obviously the bot is better at comparing the NDA to a "correct" NDA. But the bot can't think about whether that NDA suits the context in which it is used.
As a programmer, I have this to say about AI: Bullshit, don't invest, it can not replace humans for non-repetitive tasks that require reasoned judement.
Re: (Score:2)
Re: (Score:2)
Re:Lol (Score:5, Insightful)
Boring, difficult tasks are where they bill most of their hours. Why would anyone who is paid $300/hr want to finish in 26 seconds? The lawyer who took 156 minutes just made $900.
Re:Lol (Score:4, Insightful)
Boring, difficult tasks are where they bill most of their hours. Why would anyone who is paid $300/hr want to finish in 26 seconds? The lawyer who took 156 minutes just made $900.
Proving once again that time spent is a lousy proxy for value.
Re: (Score:2)
Boring, difficult tasks are where they bill most of their hours.
Not for much longer if these AIs become available to the public.
Re: (Score:3)
You mean it will replace already poorly paid beginner lawyers in lawyer offices and make that occupation vanish into thin air.
Re: (Score:3)
Re: (Score:1)
Re:You think this is funny? (Score:5, Funny)
First they came for the lawyers, but I did nothing, for I am not a lawyer ...
Re: (Score:2)
Lawyers are the biggest roadblocks on the way of justice.
Re: (Score:1)
Lawyers are the biggest roadblocks on the way of justice.
Can you define "justice" for us?
Re: (Score:2)
Lawyers are the biggest roadblocks on the way of justice.
Can you define "justice" for us?
Probably not without spending way more time on it than I care to.
However, I can say that when the outcome of a legal process depends largely on the amount of money one is able to spend spend on their legal team, it is definitely NOT justice.
Re: (Score:1)
Lawyers are the biggest roadblocks on the way of justice.
Can you define "justice" for us?
Probably not without spending way more time on it than I care to.
However, I can say that when the outcome of a legal process depends largely on the amount of money one is able to spend spend on their legal team, it is definitely NOT justice.
Point taken. It could be one of those "bad system but better than anything else we can think of".
Who designed the test? (Score:2)
The document used for testing was not created by an AI, was it?
Re: (Score:2)
The document used for testing was not created by an AI, was it?
No clue but if it was created by a lawyer would you then claim that this would have given lawyers an advantage?
Re: (Score:2)
If it was designed by the team that developed the AI, yes, I'd be suspicious.
"spot issues"? (Score:2)
What kinds of "issues" are we talking about here. This seems highly subjective since one side's "issue" could be the other side's deal breaker.
Re: (Score:2)
What kinds of "issues" are we talking about here. This seems highly subjective since one side's "issue" could be the other side's deal breaker.
Exactly. I am reminded of when AI would replace doctors for diagnosing illnesses. Seems it is helpful in identifying symptoms and possible courses of suctions but humans are still needed to make sense of the information. The real value from lawyers is to explain the legal issues, their ramifications and advise on courses of action. Presenting an argument that the client understands is what is valuable; AI could help with tax by analyzing things and providing in sight. Law firms already use that, they're cal
Re: (Score:2)
The real value from lawyers is to explain the legal issues, their ramifications and advise on courses of action.
I came here to say essentially the same thing. I am not a lawyer but I can generally spot potential issues in legal documents presented to me. I would feel totally comfortable with an AI doing the same thing (as the summary points out, it does so much faster). However, whenever I have involved a lawyer in something it is because I also need to ask "what does this actually mean for em?" and then "so what do I do about it?" The AI is definitely not that far along yet.
Re: (Score:2)
Re: (Score:2)
Why use time? (Score:2)
The longest time taken by a lawyer to complete the test was 156 minutes, and the shortest time was 51 minutes.
Was time used as a measurement, i.e. were the lawyers told to work as fast as they could? Cause that would likely affect the diligence put into the review.
We already know that computers are faster at processing data. We also know that stressing the crap out of people tend to make them work faster but with less precision. Adding such an element to the human input would bias the result while providing no additional information of value.
Because Lawyers bill... (Score:2, Insightful)
by the hour?
Comment removed (Score:5, Funny)
Re:what the article doesnt say. (Score:4, Insightful)
20 top US corporate lawyers with decades of experience in corporate law and contract review were pitted against an AI.
What the article isnt saying is that these lawyers were pitted against a combat AI in a parking lot in the back of a derelict convenience store in the dark hours of the morning. The AI successfully dismembered virtually all of them, despite their decades of experience in corporate law.
I think I speak for everyone when I say, I'm ok with that, and gladly welcome our lawyer-dismembering overlords.
AI or Expert System? (Score:4, Insightful)
Re: (Score:2)
Yep. And my guess is that if these types of AIs start to become commonplace that the human lawyers would just figure out how to word things so that it flies under the radar of the AI. A non-reasoning AI is never going to be able to catch stuff written by a lawyer intentionally trying to obscure and deceive.
Re: (Score:2)
and what's it make us, if we're aware that we know that we're aware that we know that we're aware that we're distinct from our surroundings?
Re: (Score:2)
"non-reasoning" is not a term used in our field, so I am going to have to ask that you please define your own terminology.
Non-conscious, means without conscious. To be conscious, a being must be aware of itself as disinct from its surroundings and be aware that it knows that it aware of this. That second level of abstraction is critical and makes a human different from a fish, while the first makes a fish different from a sea sponge.
As far as we know, you don't have to be conscious to be able to reason. A fish or dog can make assumptions about their environment and make reasonable inferences. When we are talking about reading, it becomes even easier. AI might be able to process the paper but it can't comprehend the paper. Comprehending the paper would be having the ability to read the paper and then answer questions about the paper with or without being conscious. One step further would be to have the ability to answer subjective
First born child clause (Score:2)
It is better than just sneaking something small past an Asemi-I. Sneak something big past. Something that no human would ever let through. Like the surety being the first born child. Or possibly something more enforceable.
This happens to simplistic AIs that mark English essays. Well written gibberish can get high marks. The sad thing is that the AIs generally do a better job than human markers, which is probably more a statement about the humans doing the marking than the AI.
Re: (Score:2)
It is better than just sneaking something small past an Asemi-I. Sneak something big past. Something that no human would ever let through. Like the surety being the first born child. Or possibly something more enforceable.
This happens to simplistic AIs that mark English essays. Well written gibberish can get high marks. The sad thing is that the AIs generally do a better job than human markers, which is probably more a statement about the humans doing the marking than the AI.
AI and humans work better together. AI to catch common mistakes and humans to make sure it is sane. Almost all of the AI in recent years can be easily beat by AI and humans working together. We should stop thinking of it as an either/or and instead let the AI do what it is good at and humans do what they are good at.
Re: (Score:3)
Re: (Score:2)
intelligence: the ability to acquire and apply knowledge and skills.
Sounds like a perfectly appropriate word to use in this case.
Re: (Score:2)
Are these really AIs? Or are they just expert systems trained to do a job?
Expert systems are a form of AI, just not a form of machine learning. The AI field contained a lot of research on expert systems 30 years ago, as well as work on logic processing, and the logic processing element could be applicable here, combined with NLP.
I assume it wasn't Al (Score:2)
Now we need an AI to (Score:2)
Re: (Score:2)
And (A)I can't wait! (Score:5, Funny)
It took the lawyers an average of 92 minutes to complete the NDA issue spotting, compared to 26 seconds for the LawGeex AI.
I imagine sexbots will achieve similar results.
Little Bobby Tables (Score:3)
A clever lawyer might have added some 'special' text to the NDA https://xkcd.com/327/ [xkcd.com]
And then... (Score:1)
So ... (Score:2)
IOW we don't have to wait for long now before a couple of millions lawyers are sacked.
A good beginning.
I sincerely hope that the same AI will write the new laws in the future instead of the lobbyists.
Re: (Score:2)
Of course you take three hours to look at an NDA ... if you're billing by the hour.
No matter what, I bet the human attorneys were *much* better at double billing their time than the AI. Come on, AI researchers, gotta work on the hard stuff as well!
Can you work around the AI? (Score:4, Insightful)
I hypothesize that the AI is only good at spotting current and historic types of loopholes.
Here's my proposed test. A theoretical bad-actor NDA creator gets services of both a lawyer and the AI to review their document. They craft different ways to build in issues, with several cycles of submitting to both for feedback and modifications. (Since both of these would be available to someone trying t make a bad one.)
Final document is reviewed and scored.
My guess is that the human lawyers will be more adept at finding innovative issues in the NDA. But who knows until we test it.
It might be that the best path is a first pass by an AI to catch issues, then a lawyer-pass that can be significantly quicker since it doesn't have to look for the same issues the AI would.
We should all be worried (Score:2)
The mega corps figured that out years ago. They had Congress pass a law that made Arbitration agreements legally binding and had the Supreme Court stacked with pro corporatists who upheld it (even though it's a pretty obvious violation of due process
Re: (Score:2)
except their jobs are not at stake. read the other comments here about who the "top lawyers" were and what the best scores of the lawyers who did participate was.
the story and any conclusions are typical AI B.S. hype. Real world jobs not in any danger from AI since 1960s and that's still true.
Full study is contact-list-walled (Score:5, Insightful)
In the linked article, there is a button to get the full study, but downloading the study requires giving one's name, email, phone, company, and "contracts reviewed per month". Not exactly paywalled, but not exactly free.
The title of the Slashdot story says, "20 Top Lawyers Were Beaten By Legal AI", which is not true. The top AI and top human scores were identical at 94%. Furthermore, although the lowest human score was 67%, the average was 85%, indicating that the distribution is skewed toward higher human scores. There is a distribution of human scores, but just a single AI sample point, so it's not clear what the AI distribution would be.
There is also a comparison of AI vs. human time. Of course, that's a misleading comparison, similar to the misleading Jeopardy comparison from a few years back. Computers will always beat humans at text parsing and button pressing. That was true decades before AI.
The other aspect that I don't see in the article summary is whether the seeded risks were all in the AI training set or if any were deliberately left out of the training set. I'd expect the AI to do extremely well in detecting risks similar to the training set. However, I'd expect the humans to do better in risks that deviated from the training set.
Re: (Score:1)
Here's an idea for this tecnology (Score:5, Interesting)
They should put this AI on a website called something like "EULA-Buddy", where you can paste those 10-foot-long EULAs that come with every modern device or service. Then it could concisely tell you how much the EULA sucks. Maybe that would help reign in uncontrolled expansion of these ridiculous "contracts".
Re: (Score:1)
I'm pretty sure that would just prove that the all suck infinitely.
Re: (Score:1)
Legal Crime (Score:1)
Top 20 Lawyers? (Score:1)
charging by the hour? (Score:2)
Should make lawyers less expensive (Score:2)
Tools like this should make lawyers less expensive and available to more people as they won't have to spend their time examining documents when a computer can do that. (Actually the junior lawyers are for that and for researching now.) Let the lawyers spend time with clients, in talks with the other sides lawyers, or in court. Speeding up the discovery process should get cases to trial sooner too.
The lawyers that have to worry are the ones that are really good at researching. Those that are good with client
paperclip 2.0 (Score:1)