During the recent Legalweek, panelists debated what impact ChatGPT and generative AI could have on the legal industry, if any, but urged caution nonetheless
NEW YORK 鈥� It鈥檚 safe to say that ChatGPT and generative artificial intelligence (AI) as a whole have captured the imaginations of those in professional services like few technologies have before. The idea behind ChatGPT seems simple: You ask it a question, in plain language, and it provides a straightforward answer to your prompt.
However, the reality remains far from simple. There are layers of algorithms that create content (thus the generative part of the name) by continually predicting the next word, with extremely large data sets needed to make these predictions accurate. Further, the newly-released GPT-4 is even multi-modal, meaning it can accept both text and image inputs, which ratchets up the complexity even further.
As a result, even among the most optimistic technologists, there remains some generative AI risks that can鈥檛 be ignored. And as a key panel, Reshaping the Legal Profession: Thriving in the Age of Generative AI & ChatGPT, at the recent explored, the heavily hyped technology may be less of a do-it-all tool and more of a 鈥渕oderately bright, but very lazy first-year associate.鈥�
What generative AI is & what it isn鈥檛
That comparison came courtesy of panelist Aaron Crews, currently Chief Product & Innovation Officer at alternative legal service provider UnitedLex and formerly Chief Data Analytics Officer at law firm Littler Mendelson. Crews noted that while many legaltech types have high hopes for generative AI use in law, including himself, at its core the technology isn鈥檛 that revolutionary.
鈥淕enerative AI is fancy marketing-speak for a machine that anticipates where you want to go next,鈥� he said, adding that while there may be high expectations of a tool named artificial intelligence, in reality 鈥渋t鈥檚 not intelligent.鈥�
Indeed, generative AI is bounded by the data that is put into the system. That means that ChatGPT, developed by OpenAI and currently the most famous generative AI platform, has access to untold amounts of data to make its predictions 鈥� but that data is only current as of 2021, meaning it cannot adjust to newer events.
The tool also suffers from 鈥渉allucinations,鈥� meaning that sometimes the technology 鈥減redicts鈥� facts that have no actual basis in reality. In one notable case, as explained by panelist Foster Sayers, General Counsel & Chief Evangelist at software company Pramata, a Michigan judge tried asking ChatGPT about why a certain court decision was decided the way it was and found that ChatGPT completely made-up precedential cases 鈥� something the judge caught easily, since he had decided the case himself.
With the recent release of GPT-4, a factual accuracy rate between 70% and 80%, depending on the subject matter. But that 20% below perfect is 鈥渟ignificant鈥� in law, explained another panelist, Ilona Logvinova, Associate General Counsel at McKinsey & Company. Technologists in law often run into risk-averse attorneys and clients, where one bad experience can lead to a closed door for all future technological advancements. And although some companies and are hiring for a new role known as a prompt engineer to ask generative AI platforms more specific questions to get a desired outcome, it鈥檚 impossible to create a foolproof system.
鈥淧rompt engineers are getting more popular, but they鈥檙e also learning on the spot,鈥� Logvinova noted.
So where鈥檚 the use?
That鈥檚 not to say that generative AI will fall by the wayside, however. The panel identified a few potential use cases for generative AI in professional services as it now stands: document analysis, review and drafting; research and knowledge management; contract analysis and drafting; and chatbots and assistants. However, the technology is moving quickly, and so too are its potential applications, panelists added.
One panelist, Danielle Benecke, Founder of Baker McKenzie Machine Learning at law firm Baker McKenzie, noted that 鈥渇irms and other enterprises have been sitting on this unstructured data forever,鈥� which generative AI can help unlock.
However, while many regular generative AI use cases focus on the wide data sets the tool already has, it鈥檚 more interesting to start with the enterprise鈥檚 data and running AI against it, Benecke explained, adding that, for example, a firm鈥檚 M&A deal room could run generative AI against the data set and theoretically create a due diligence checklist based on the firm鈥檚 contracts that are already in place.
Pramata鈥檚 Sayers did question how much better generative AI is at producing new documents and contracts than simply using regular templates. While generative AI may produce bespoke work product, legal documents often have to be worded in a very specific way that鈥檚 tough to predict, he said. Contract experts such as TermScout鈥檚 Evan Harris have , finding that while generative AI can create a passable first draft contract, the outputs still require a good deal of editing and governance.
With these limitations in mind, Logvinova added 鈥渋t鈥檚 safer and less risky鈥� to use generative AI for internal purposes rather than for client-facing content or communications. Crews agreed, saying that he 鈥渁bsolutely would not鈥� use ChatGPT for client work as the technology currently stands, but that it may be helpful in fast-forwarding the data creation and ingestion process.
No matter the use case, however, all panelists agreed it鈥檚 paramount to avoid the temptation of adopting generative AI just out of curiosity. Due to its risks, and with the technology in its early stages, any use should be conducted with the firm鈥檚 overall data strategy underpinning the AI use and with a specific goal in mind.
Benecke said her exploration of generative AI primarily focuses on holistic applications across the firm rather than one-off use cases. Any time the firm adopts an AI tool, she said, it鈥檚 with the specific goal to 鈥渟upercharge the firm鈥檚 most valuable pre-existing service lines,鈥� directly tying the AI use with a firm strategic initiative.
Still, there remains a number of unknowns about generative AI鈥檚 use in professional services, and the balance between risk and innovation with generative AI weighing on the scale is one that firms are still working out.
鈥淏e forward-leaning, but be smart about your governance,鈥� Benecke warned. 鈥淵ou don鈥檛 want to be that cautionary tale.鈥�
