Legal tech companies have long preached that the machine learning tools they offer are accurate and efficient. But the lingering question has always been: Is the software as accurate as a human attorney?

Now, the company LawGeex is aiming to settle any qualms attorneys might have about embracing AI with a study that concludes its algorithms not only more correctly identified key language in a set of contracts, but were exponentially faster than humans in doing so.

“[L]awyers and the public generally believe that machines cannot match human intellect for accuracy in daily fundamental legal work,” LawGeex says in the resulting report. The paper issues a warning to the profession, arguing that “lawyers failing to capitalize on the competitive advantage of technology are unlikely to thrive into the next decade.”

The results of the study—which was administered by independent attorney Christopher Ray and involved a stable of high-profile legal scholars—are striking. Using five nondisclosure agreements from the Enron data set as the baseline, 20 lawyers were pitted against LawGeex's AI in parsing 30 provisions. On average, the LawGeex software achieved an accuracy rate of 94 percent. The humans? An average of 85 percent.

Here's the real kicker: The fastest human attorney completed the task in 51 minutes. The LawGeex software took 26 seconds.

“This experiment may actually understate the gain from AI in the legal profession,” USC Gould School of Law Professor Gillian Hadfield, who was involved in crafting the study, said in a statement. “The lawyers who reviewed these documents were fully focused on the task: It didn't sink to the bottom of a to-do list, it didn't get rushed through while waiting for a plane or with one eye on the clock to get out the door to pick up the kids.”

“The margin of efficiency is likely to be even greater than the results shown here,” she added.

The lawyers who went head-to-head with the software came from a range of backgrounds, some with prior experience at Big Law firms such as Alston & Bird and corporations such as Cisco. Hua Wang, one of the participants and a lawyer formerly at Proskauer Rose and K&L Gates, said in a statement that the review process she was tasked with was “logical and credible” and “similar to how I reviewed documents” at a major firm.

So what's the reaction from other practicing attorneys?

“I think this is the exact kind of pressure that all kinds of industries are going to be facing from AI in the future,” said Annette Hurst, a partner at Orrick, Herrington & Sutcliffe who leads a practice group at the firm focused on artificial intelligence and was not involved in the LawGeex study. Hurst added, though, that correctly identifying what a clause in a contract does is just one step in the process of giving legal advice.

“You still need an experienced lawyer to counsel the client, 'OK, we spotted this issue. What do we do now?'” There are other limitations, too, she pointed out. “No AI is going to be able to understand when you need an update to the standard contract because of a change in the law.”

Luis Villa, a lawyer and co-founder of software company Tidelift, sounded a note of caution about the findings on Twitter. “Worth stressing that NDAs are, essentially, toy contracts: extremely constrained in a variety of ways. So extrapolating from NDAs to other forms of contracts is not straightforward,” he tweeted. “Still, I could easily see this being a useful tool in a corporate legal team's toolkit, assuming the team can admit their own people are error prone (hard for lawyers!)”

Computer science professor Yonatan Aumann of Bar-Ilan University in Israel, one of the other experts consulted for the study, concluded that there was very little chance that the accuracy result was due to random chance. Specifically, he calculated a probability of 0.68 percent.

Erika Buell of Duke Law, who also advised on the study, noted in an email that the AI did not beat every lawyer every time. But, she added that she would expect the algorithm to gradually improve and stressed the efficiency gains that the technology could bring.

“Having the AI do a first review of an NDA, much like having a paralegal issue spot, would free up valuable time for lawyers to focus on client counseling and other higher-value work,” she said. “I strongly believe that law students and junior lawyers need to understand these AI tools, and other technologies, that will help make them better lawyers and shape future legal practice.”

Lest contract attorneys get too down on themselves, it's worth noting that the LawGeex algorithm took a fair bit of teaching. According to the report, the AI was trained on “tens of thousands of NDAs, using custom-built machine-learning and deep learning technology.”

“Training an AI is similar to training a new lawyer—exposure to different examples is crucial in developing a deep understanding of the legal practice,” The AI, however, had never been exposed to the five NDA contracts analyzed in this contest, the report says.

Other advisers involved in the study included former Morrison & Foerster deals lawyer Bruce Mann, PhD student Beverly Rich at the USC Marshall School of Business, and Stanford CodeX executive director Roland Vogl. LawGeex also released an infographic summarizing the study here.