Something jumped out at me early at the recent LegalTech Connect Law Firm Research & Innovation Conference with the legal research community. Greg Lambert asked how many attendees have encountered partners who don’t know their Westlaw or Lexis passwords. That question resonated throughout the room as most had experienced this.
It’s general knowledge that partners rarely conduct legal research, but if they don’t know how to log in to a legal research platform, what process are firms using for partners to access the underlying sources in a brief?
Over the past year, we’ve seen a steady stream of headlines about AI hallucinations where fake cases, fabricated quotes, and misstated facts made it through a review process and into court filings.
Technology Isn’t The Issue; It’s A Process Problem
While lawyers are responsible for their work, there is a tendency to blame the breakdown on the introduction of AI technology into the process. I think a better explanation is that AI exposed a flaw that already existed in most processes.
For decades, law firm drafting was built on partner guidance and associate writing, with a senior associate taking a pass and then the partner reviewing in a supervisory role. Citation tools such as Shepard’s or KeyCite are used by the associate along the way. But in practice, how, what, and when does the supervising attorney review the work?
If a case was reviewed earlier in the process, is it checked again at the end of the process, including the quotation pulled from the case?
Before AI, a mistake that squeaked through a gap in the process didn’t matter as much because potential errors were minor and could be corrected as errata if necessary. But that doesn’t work anymore. Human authorship is being augmented by AI-generated content and editing that can alter facts, fabricate quotes, and invent citations to nonexistent cases.
AI Exposes The Gaps That Were Always There
Anyone who uses AI grammar-checking software or Microsoft Copilot on a regular basis has had the experience of “uncorrecting” its revisions that are more grammatically correct but also change the meaning of a statement. Associates are still learning and may miss subtle errors introduced by these powerful new tools.
What was correct at an earlier stage of a document may very well be changed by AI in a later version.
Errata Versus Fabrication
The central issue here is that an AI hallucination is a fabrication. Fake cases are one issue, but more recent issues in filings include fake quotations. A complete review of everything, including foundational facts, needs to be conducted at the end of the process before filing.
The tried-and-true workflows that once assumed human authorship must change.
Drafting And Review Should Be Inseparable
Remember your high school science class? There are protons, neutrons, and electrons within an atom, but the atom itself cannot be separated. AI highlights this in document creation. The drafting and review processes are atomic. They can’t be separated without some kind of nuclear reaction.
AI creates efficiencies in drafting, but it imposes new burdens on the review process and highlights assumptions about who reviews what and when.
The process now needs to track who verifies citations, when verification occurs, and how it is communicated across the team drafting and reviewing the document. The same should happen for quotations and fact-checking.
Zero Trust principles
In Cybersecurity, there is a concept called Zero Trust. It’s basically “never trust, always verify”. Unfortunately, AI systems are probabilistic. They are literally guessing the next part of a word that fits a pattern. This means that hallucinations are a feature of the system’s design rather than a bug. There are techniques to minimize hallucinations, but they can’t be eliminated.
“Back to Basics” review processes
The consequences of AI incorporated into drafting are that errors can become fabrications that are well beyond the scope of errata.
Before the Internet, it was common to print out all authorities and keep them in binders for access and review. But the digital equivalent to that physical binder has been lost in some firms.
There are solutions available to help organize and improve processes. Creating hyperlinks within documents to a repository of referenced authoritative sources can help expedite review. If a link doesn’t exist to a case, then it’s a red flag that it may not exist. Clicking on a link can help review and verify that quotes or facts are correct, too.
Firms should take action now
AI Policy is not a process. Firms need to operationalize how attorneys can comply with the policy and provide the processes and tools needed to make it easier to meet the standards of Rule 11 Ethical responsibility.
The California Bar recently solicited comments on changes to the rules of professional conduct as they pertain to AI. And there is some discussion proposing an actual Hyperlink Rule to ensure sources are linked in filings. Firms shouldn’t wait for regulators to tell them what to do.
Conclusion
AI may have introduced gross fabrication into legal work, but it also removed the illusion that review processes were ever airtight.
There are more advanced solutions possible, including some I have suggested in prior writings on hallucinations. But before the industry considers more advanced solutions, it should ensure it addresses the basics first.
The firms that recognize and rebuild their review processes and tightly couple them to the drafting process will be in a much better position than those that treat this as a problem caused by technology or a problem to be solved with policy alone.
AI was used in the review of this article.
Ken Crutchfield has over forty years of experience in legal, tax, and other industries. Throughout his career, he has focused on growth, innovation, and business transformation. His consulting practice advises investors, legal tech startups and others. As a strategic thinker who understands markets and creating products to meet customer needs, he has worked in start-ups and large enterprises. He has served in General Management capacities in six businesses. Ken has a pulse on the trends affecting the market. Whether it was the Internet in the 1980s or Generative AI, he understands technology and how it can impact business. Crutchfield started his career as an intern with LexisNexis and has worked at Thomson Reuters, Bloomberg, Dun & Bradstreet, and Wolters Kluwer. Ken has an MBA and holds a B.S. in Electrical Engineering from The Ohio State University.
The post Back To Basics appeared first on Above the Law.