generative AI Archives - 成人VR视频 Institute https://blogs.thomsonreuters.com/en-us/innovation-topics/generative-ai/ 成人VR视频 Institute is a blog from 成人VR视频, the intelligence, technology and human expertise you need to find trusted answers. Mon, 06 Oct 2025 13:53:50 +0000 en-US hourly 1 https://wordpress.org/?v=6.8.3 CoCounsel Monthly Insider: Sharpening Your Competitive Edge /en-us/posts/innovation/cocounsel-monthly-insider-sharpening-your-competitive-edge/ Wed, 17 Sep 2025 20:22:59 +0000 https://blogs.thomsonreuters.com/en-us/?post_type=innovation_post&p=67579 Driven by our commitment to our customers, each month, 成人VR视频 is delivering enhancements to CoCounsel Legal and additional solutions, making them more intuitive, customizable, and effortless to use. In this September edition, we spotlight the latest updates, featuring major upgrades and subtle refinements, designed to boost efficiency and support the delivery of exceptional, high-quality work.

Redesigned drafting capabilities unify CoCounsel tools, content, expertise, and workflows

Informed by customer feedback, we’ve reimagined the legal drafting experience 鈥 making it more intuitive, intelligent, and seamlessly integrated. The drafting capabilities in CoCounsel fuse users鈥 institutional knowledge with trusted 成人VR视频 content and AI-powered technology to expedite the legal drafting process. The redesigned homepage puts everything users need right at their fingertips 鈥 CoCounsel Chat, skills, and powerful litigation and document analysis tools 鈥 all in one clean, intuitive space. Eliminating the need to jump between tabs or hunt for resources, it鈥檚 now an even smoother, faster experience that lets users stay focused, work smarter, and get more done with less friction.

Drafting homepage

 

Live Draft brings the ability to summarize and modify a document using natural language in Word. Live Draft also delivers contextual awareness of the document and understanding of the content and structure, so every suggestion and edit is tailored to the content. This helps to further reduce time spent producing a final draft, by delivering more accurate, relevant suggested changes.

Live Draft

 

Append Authorities enables users to combine all cited documents into a single file suitable for court use, reducing the risk of errors and increasing efficiency. Every cited document is linked for verification purposes, and a hyperlinked table of contents is included.

Append Authorities

 

Region settings customizes CoCounsel tools for global legal professionals

The new region settings capability enables users to select their geographic preference from U.S., UK, Australia or Canada. Based on the selected region, region settings will automatically adjust tools and prompts in the CoCounsel Library making the work product more relevant. Users can now automatically tailor their documents using specific regional requirements, including for the UK and Australia, British English spelling variations, legal terminology, grammar, and content formats. Similarly, this will be coming soon for Canadian English. Additionally, CoCounsel Library is now available in the UK, Canada and Australia.

HighQ integrates CoCounsel AI for intuitive, conversational client data access

With听CoCounsel鈥檚 Search a Database skill embedded within HighQ, this customer-driven development allows clients the ability to pose queries regarding their data and receive summarized, highly relevant answers. Sourced from pre-approved content within their site, clients can quickly review summaries, generate reports, and make informed decisions without waiting for manual responses.

Legal Tracker adds AI-powered capabilities

Legal Tracker鈥檚 new AI features help users manage legal spend more efficiently. The AI-powered PDF-to-LEDES converter and invoice review speed up invoice evaluation and ensure accurate billing. An AI assistant also streamlines reporting and reduces manual data handling.

Legal Tracker

 

These transformative features reinforce our commitment to empowering legal professionals with the tools and solutions they need to excel. or to see firsthand how they elevate work to new heights.

To stay abreast of newly added features, monthly releases, and more, please sign up for the .

]]>
The AI Implementation Gap Must Be Closed /en-us/posts/innovation/the-ai-implementation-gap-must-be-closed/ Mon, 15 Sep 2025 19:33:07 +0000 https://blogs.thomsonreuters.com/en-us/?post_type=innovation_post&p=67548 Law firms have shown they are very bullish on AI. Rightly so, when it comes to the core elements of the legal workflow 鈥 researching case law, pouring over documents to find the needles in the haystack, and drafting standardized documents like contracts, policies, and discovery requests 鈥 the agentic and generative AI (GenAI) solutions available today are helping firms cover more ground faster and more comprehensively than ever before possible.

Nearly half (47%) of law firm respondents from the Future of Professionals Report say their firms are already experiencing at least one type of benefit from AI adoption and 80% expect AI to fundamentally alter the course of their business over the next five years. Chief among those is time savings. On average, law firm professionals expect to free up 190 hours per year by using AI. At current average hourly rates, that works out to approximately $18,000 in savings per professional, per year 鈥 or a total of $20 billion for the U.S. legal industry.

Perception vs. Reality

For all the enthusiasm that exists for AI鈥檚 potential, however, there is a large gap emerging between law firms鈥 AI aspirations and their real-world AI strategies. Even though the majority of law firms expect AI to drive transformational change in the future and nearly half are experiencing some benefits now, far fewer (29%) expect to see high or transformational levels of change this year. When pressed further on what their firms are doing today to leverage AI, nearly one-third (32%) of law firm respondents say they feel their firms are moving too slowly on AI adoption, and just 22% say their firms have a visible AI strategy in place.

This gap between future ideals and current realities is a phenomenon 鈥渢he GenAI paradox,鈥 which occurs when businesses race to invest in AI pilot projects and buy new solutions, but struggle when it comes to implementing them and integrating them into everyday workflows. Versions of this struggle are playing out in virtually every industry right now as professionals come to grips with the fact that true transformation is not as simple as plugging in an off-the-shelf AI tool. It requires a clear strategy, a carefully planned roadmap, targeted integration of professional-grade AI solutions, and a commitment at all levels for the long haul. A firm cannot afford to sit on the sidelines any longer 鈥 it is imperative to have an AI strategy.

Key Steps to True AI Transformation

Over the course of our partnerships helping some of the world鈥檚 largest law firms not only access new AI capabilities, but , we鈥檝e found four levers that all firms need to engage to ensure the success of their AI initiatives.

  • AI Tools Without an AI Strategy will Never Reach Their Potential: Among the 22% of law firms that currently have a visible AI strategy in place, 71% are already experiencing a clear return on investment from AI. By contrast, for those firms that do not have a clear AI strategy in place, just 18% are experiencing a return on investment. That means law firms with a visible AI strategy are almost four times more likely to experience benefits compared to firms without any significant plans for AI adoption.
  • AI Leadership Comes from the Top: Law firms helmed by leaders who lead by example when introducing change, firms that have added new governance roles, and those that are actively investing in AI are consistently seeing more benefits than those that don鈥檛. For AI to truly add value, it needs to be implemented firm-wide, and that kind of sweeping change can only come with leadership support, clear goals and objectives, and widespread adoption.
  • Operations is Where the Hard Work Happens: Firm-wide AI integration is impossible without first understanding the need to change and reimagining workflows. To extract maximum value, AI-powered tools must be built directly into existing systems and processes. That requires making transformative changes to underlying business models, including how firms price, staff, and deliver legal work, and how they adapt related workflows and processes, while adding new roles and skills to support their operations.
  • End-User Adoption: The best AI technology and most well-thought-out strategy in the world will not mean anything if no one uses it. When users within law firms understand AI and feel empowerment, ownership, and accountability for its use, their law firms see results not only in terms of higher levels of AI adoption but in the additional benefits and ROI that they gain as well. Firms need to make sure they are educating staff, making tools readily available, and allowing time for a learning curve to take root.

While the detailed strategies and specific paths to AI implementation will vary from firm to firm, there are a handful of universal truths that apply to all. Foremost is the commitment to address the AI revolution for what it really is 鈥 a monumental transformation in the way legal work is conducted on par with the introduction of the personal computer, the internet, and the smartphone. It is not enough just to buy the latest greatest widget. Firms that want to extract real value from AI need to think hard about how it will affect everything they do and start addressing those changes now to unlock the full potential of the technology to transform their firm.

]]>
Forecasting the Future of the Law Firm /en-us/posts/innovation/forecasting-the-future-of-the-law-firm/ Wed, 05 Mar 2025 19:55:59 +0000 https://blogs.thomsonreuters.com/en-us/?post_type=innovation_post&p=65140 In the ever-evolving landscape of the legal industry, law firm leaders find themselves at a critical juncture, facing unprecedented challenges and opportunities. As we engage in conversations with leaders from global law firms, AmLaw 200 firms, and major independent law firms, a common thread emerges: the pressing need for decisive action in an environment of rapid change.

At the forefront of this transformation is the rise of generative AI (GenAI), which promises to reshape the very foundations of legal practice. This technological revolution is not just another trend; it’s poised to become the most influential force shaping law firms over the next five years, surpassing even economic factors and geopolitical instability in its potential impact. Looking at the implications of GenAI on the law firm business model, it becomes clear that the time for passive observation has passed. In today’s legal industry, there is simply no room for bystanders.

white paper looks at the current environment and forecasts the next three waves of how AI-driven technologies will reshape the legal industry.

Wave 1: Optimization of legal workflows

Law firms are increasingly pressured to adopt AI to reduce costs as their clients embrace these technologies, leading to shifts in cost structures and hiring practices. Despite these changes, the demand for legal services continues to grow as clients face business disruptions due to AI, prompting the need for new legal support. As AI adoption becomes more widespread, it is expected to significantly impact pricing strategies and workforce dynamics in the legal industry.

Wave 2: Legal market disruption and law firm re-engineering

Law firms are adopting more technology and project management strategies, which leads to fewer lawyers being hired and a re-engineering of business models to stay competitive. Legal departments are keeping more work in-house, but smaller firms can leverage AI to handle larger, complex tasks. The competitive landscape is evolving, with middle-market firms feeling pressure as routine work becomes productized and new AI-enabled services enter the market.

Wave 3: Disruption of legal services landscape and AI winners emerge

The use of AI in the legal industry will lead to a shift in how clients interact with law firms, with top-tier firms focusing on high-stakes and complex matters, while smaller firms move up the value chain with AI-enabled solutions. New AI-powered delivery models and self-serve legal products will transform the way legal services are bought and delivered, potentially leading to consolidation in the middle of the market. Ultimately, AI will have a profound impact on the law firm of the future, but it will work best when it complements, rather than substitutes for, legal professionals.

It’s crucial for law firm leaders to recognize that the emergence of AI and GenAI signifies a real and fundamental shift in the legal landscape, impacting how legal work is done. As these technologies promise to transform law firm operations, firms already grappling with pricing, talent, and competition must proactively manage AI adoption.

By addressing the interconnected challenges of client communication, talent acquisition, and AI-driven service pricing, firms can navigate the coming changes and avoid being left behind in this technological revolution.

Download your full copy of white paper.

]]>
Beauty Is in the AI of the Beholder /en-us/posts/innovation/beauty-is-in-the-ai-of-the-beholder/ Wed, 26 Feb 2025 17:04:29 +0000 https://blogs.thomsonreuters.com/en-us/?post_type=innovation_post&p=65079 鈥淲elcome to the era of the AI superlative. While the first two years of generative artificial intelligence (GenAI) development were an all-out sprint to create new models, establish proof-of-concept solutions, and define optimal use cases, the next phase to deliver increased efficiency and better work product to clients in the AI lifecycle will be dominated by marketing as well.鈥

Raghu Ramanathan, president of Legal Professionals at 成人VR视频, opened with these statements and shared his view on industry benchmarks in an article on Above the Law titled .

He noted as more companies develop AI solutions and start-ups seek capital investment, customers will look for benchmarks to evaluate these tools. 成人VR视频 does see value in benchmarking, however, Ramanathan added benchmarks must measure products the way they鈥檙e designed to be used and should focus on results customers care about.

鈥淭he challenge is that one-dimensional metrics do not offer a reliable representation of the real value of GenAI in the legal research process,鈥 stated Ramanathan. 鈥淣o LLM-based legal research products in the market today provide answers with 100% accuracy, so users must engage in a two-step process of 1) getting the answer and 2) checking the answer for accuracy.鈥

Chief Legal Operations Officer Meredith Williams-Range from Gibson, Dunn & Crutcher LLP discussed how they are using and seeing results from AI-enabled resources. 鈥淭here is a widespread misperception around how law firms are using AI and how we conduct legal research. We are not bringing in AI and saying: 鈥楪o do all the research and write a brief,鈥 and then replacing all of our junior associates with automated results. We鈥檙e using AI-enabled tools that are integrated directly into the research and drafting tools we were using already, and, as a result, we鈥檙e getting deeper, more nuanced, and more comprehensive insights faster. We have highly trained professionals doing sophisticated information analysis and reporting, augmented by technology.鈥

Read the full article on , and as Ramanathan concludes, 鈥渢he value of legal AI 鈥 of any technological innovation for that matter 鈥 is in how it gets used in the real world and how well all the different components come together to help lawyers do their jobs more effectively.鈥

]]>
Exploring AI’s Influence on the Legal Profession: Insights from Frost Brown Todd /en-us/posts/innovation/exploring-ais-influence-on-the-legal-profession-insights-from-frost-brown-todd/ Thu, 20 Feb 2025 17:01:09 +0000 https://blogs.thomsonreuters.com/en-us/?post_type=innovation_post&p=65017 The legal profession is no stranger to change 鈥 noting that the change and its impact on the industry may be viewed very differently. But the rapid evolution of technology, particularly artificial intelligence (AI), is presenting a unique set of opportunities and challenges.

, president of Legal Professionals at 成人VR视频, recently hosted his inaugural podcast episode with Cindy Thurston Bare, chief data and innovation officer, and Kayla Kotila, senior knowledge & research services manager, from Frost Brown Todd.

Highlights from the conversation include:

  • Adoption and Use Cases: Frost Brown Todd has approximately 250 legal professionals actively integrating generative AI into their daily operations with impressive results. This adoption is not limited to specific tasks; instead, AI is being woven into existing workflows, signaling a fundamental shift in how legal work is conducted.

  • Measuring Success and Adoption: Success is measured by client satisfaction and efficiency improvements, and the firm uses qualitative feedback and quantitative methods, like A/B testing, to assess AI’s impact. Adoption is widespread across different demographics, with varying use cases depending on experience levels. Junior lawyers are leveraging AI to streamline certain tasks, while partners are using it to enhance their recall and decision-making processes.
  • Client-Centric Innovation and Collaboration: One key measure of success for technological implementation lies in its ability to enhance client service. From expediting document review processes to uncovering critical insights for litigation, AI enables the firm to deliver faster, more accurate, and ultimately more valuable services. The importance of open communication with clients regarding their AI initiatives is key. Sharing success stories, addressing concerns, and exploring potential applications collaboratively fosters trust and ensures that AI implementation aligns with client needs and expectations that benefit both parties.

Technology continues to move forward and law firms must prioritize a culture of innovation, continuous learning, and client-centricity to thrive in an increasingly complex and competitive environment. The future of law is not about replacing lawyers with machines but rather empowering them with the tools and knowledge to deliver exceptional legal service.

As part of the Clarity podcast series from the 成人VR视频 Institute, Ramanathan will speak with customers, industry experts and colleagues, bringing perspectives from legal leaders and subject matter experts shaping the industry. The conversations aim to highlight the innovations driving the legal profession as well as the people and organizations implementing new technologies and approaches to maintain a competitive edge in the rapidly changing market.

You can listen to the full conversation on either or .

]]>
成人VR视频 Best Practices for Benchmarking AI for Legal Research /en-us/posts/innovation/thomson-reuters-best-practices-for-benchmarking-ai-for-legal-research/ Wed, 12 Feb 2025 15:38:20 +0000 https://blogs.thomsonreuters.com/en-us/?post_type=innovation_post&p=64911 At 成人VR视频, we do an enormous amount of AI testing in our efforts to improve our customers鈥 ability to move through legal work faster and more effectively. We鈥檝e noticed an increase in interest in AI testing generally, and in benchmarking AI applications for legal research specifically. We鈥檝e learned a lot in our thousands of hours of AI testing, as such we offer the following best practices for those interested in considering an updated or differentiated approach when testing or benchmarking AI for legal research.

1. Test for the results you care about most.

This would seem obvious, but we鈥檝e seen a lot of confusion about it, and if we could only make one recommendation, this would be it. It鈥檚 foundational for all other recommendations.

If you cared most about determining how long it takes to drive from one place to another, you wouldn鈥檛 just measure highway time, you鈥檇 measure total door-to-door time. If you cared most about car maintenance costs, you wouldn鈥檛 just measure the cost and frequency of brake repairs and maintenance.

With the use of AI for legal research, there are no LLMs nor any LLM-based solutions that offer 100% accuracy. Because of that, all answers generated by large language models or LLM-based solutions, even if they use Retrieval Augmented Generation (RAG), must be independently verified.

Some assume verification is a simple matter of checking the sources cited in an AI answer, but this is incorrect. We鈥檝e seen plenty of examples where an AI-generated answer is wrong, and the cited sources simply corroborate the wrong answer. Verification requires using additional tools (like a citator, statute annotations, etc.) to ensure the answer is correct.

This means every time an AI-generated answer is used for research, there is a three-step process the researcher must engage in: (1) review the answer, (2) review the cited material from the answer, (3) use traditional research tools to make sure the answer and cited material are correct.

When we talk with researchers about research generally and this process specifically, what they care about most is (a) getting to a correct answer or understanding of the relevant law, and (b) the time it takes to get to that correct answer or understanding.

Because of this, the two most important measures are:

  • Percentage of times using this three-step process the user can get to the right answer, and
  • Time it takes to complete all three steps

Surprisingly, the percentage of errors in answer in step 1 can have very little impact on the percentage of correct answers by the researcher using all three steps or the time to complete those steps (unless errors are excessive), as long as citations and links to primary law are good and those primary resources are current and easily verified. Focusing on step one is like trying to figure out door-to-door times by measuring highway speeds only. It鈥檚 not very useful.

For instance, which of the following systems would you rather use?

  • System where the initial AI answer is 92% accurate, but verification, on average, takes 18 minutes, and post-verification accuracy is 97%, or
  • System where the initial AI answer is 89% accurate, but verification, on average, takes 10 minutes, and post-verification accuracy is 99.9%

It鈥檚 a clear choice, but there is often a misplaced focus on measurement of the first step in the process to the exclusion of steps two and three. Measure what you care about most.

2. Use realistic, representative questions in your testing.

Presumably you want to evaluate AI for the typical legal research you or your organization does. For instance, if you look at the research your organization does and find the questions are roughly 20% simple questions, 60% medium complexity, and 20% very complex or difficult, and that roughly half are questions about IP law and half are about federal civil procedure, then a benchmark testing 90% simple questions about criminal law would not be very helpful to you.

At 成人VR视频, we model our testing based on the real-world questions we see from our customers every month. For your own testing, focus on the question types that best represent the researchers you鈥檙e focused on.

Testing mostly simple questions with clear-cut answers is easiest for testing, but if those types of questions don鈥檛 represent what your users do most (it doesn鈥檛 well represent most AI usage in Westlaw), then the results are not particularly helpful. Similarly, if you primarily test overly complex, extremely difficult and nuanced questions 鈥 or trick questions, those can be useful for testing the limits of a system, but they tend not to be very helpful for most real-world decision making.

3. Test a lot of questions.

In our own testing, we鈥檝e found that testing small sets of questions is rarely representative of actual performance with a larger set. Large language models can generate different responses each time, even with identical inputs. Additionally, if responses are long and complex, graders may disagree, even when judging identical responses. For just a quick general sense of direction, it鈥檚 fine to test with a sample of questions as small as 100 or so, but for comparing algorithms/LLMs against each other, we strongly recommend checking the results as you grade and testing until the measure of interest stabilizes. For example, if you are running a comparison between two systems to see which is preferred, you would test until the rate at which one system is preferred over the other stops changing dramatically with each new batch of questions. Another guide to the number of questions you should test is the confidence level and interval you want (see next section).

4. Calculate and report confidence levels and intervals.

Even with a relatively large set of questions, measurements of accuracy are only so precise. When using these measurements to make decisions, it鈥檚 important to understand the degree or range of accuracy of the measurement, often referred to as confidence level and confidence interval. You can think of confidence intervals and levels like margin of error in surveys. It lets you know how reliable or repeatable the measurement is expected to be.

For instance, testing AI accuracy based on 200 questions, if you ran the test again with the same questions/answers but different evaluators, or used the same evaluators but with a different 200 random, representative sample of questions, would you expect the exact same result? Typically, you wouldn鈥檛. You鈥檇 expect the result to fall within a certain range, so it鈥檚 important to report that range along with the results so decision makers understand the differences between algorithms/LLMs that are meaningful and those that are not meaningful. The proper way to report this is with confidence intervals and levels. You can read more about them . Using standard assumptions, when measuring an error rate of 10% from a sample of only 100 questions, you can be about 95% confident that the true error rate is between 4.1% and 15.9%. This is called a 95% confidence level, and the 鈥+/- 5.9%鈥 is the margin of error. If you measure an error rate of 10% from a sample of 500 questions, the 95% confidence interval would be between 7.4% and 12.6%, or 10% +/- 2.6%.

The basic power analysis to estimate a confidence interval assumes a perfect means of detecting the outcome you are trying to measure. If there is some uncertainty in that detection, e.g., if two independent evaluators disagree about the outcome some percentage of the time, then the margin of error increases. A grading process or measurement that鈥檚 unreliable ~5% of the time, might increase the margin of error from 5.9% to 7.3%, in our example above with 100 questions. It’s important to note that there are various methods for calculating standard error, and these examples make simplifying assumptions that likely underestimate the confidence intervals observed in practice.

5. Use a combination of automated and manual evaluation efforts.

Having human evaluators pore through lengthy answers to complex questions can be difficult and time-consuming. Ideally, we would just have AI evaluate the accuracy and quality of answers generated by AI. This is sometimes referred to as LLM as judge. But in the same way that AI makes mistakes when generating an answer, it can also make mistakes when evaluating the quality of an answer against a gold-standard answer written by a human. In our experience, modern LLMs are pretty good at evaluating AI-generated answers against gold-standard answers when answers are clear and relatively short. With length and complexity, we鈥檝e found the LLM as judge approach to be very unreliable.

For instance, has shown that LLMs tend to struggle when evaluating responses to complex and challenging questions like those requiring expert knowledge, reasoning, and math.

Since most test sets will contain a sample of simple/easy/clear questions and answers, it makes sense to use AI for automated evaluation of these, then use human evaluators for the rest, at least until AI improves to the point where more can be automated.

6. For human grading, use two separate human evaluators for each answer, and have a third (ideally more experienced) evaluator to resolve conflicts.

For assessments like these, can be a real issue. In our own testing, we鈥檝e found attorneys evaluating AI-generated answers for more complex legal research questions can disagree about the accuracy or quality of answers about 25% of the time, which makes single-grader evaluation unreliable. To improve reliability, we have two evaluators separately grade each answer, and where there are conflicts, we have a third, more experienced evaluator resolves the conflict.

7. When answers are wrong, investigate to see if the gold-standard answer might be wrong.

In the same way people make mistakes in evaluating answers, they can also make mistakes in coming up with the gold-standard answer for testing. In our experience, we鈥檝e found some instances where the AI-generated answer was evaluated as incorrect when compared to the gold-standard answer, but when we dug into it further, it turned out the AI was correct and the person who put together the gold-standard answer was wrong. Sometimes AI makes mistakes and sometimes humans make mistakes 鈥 you should check both.

8. If evaluating multiple algorithms/LLMs/solutions, make sure the evaluators are blind to which algorithm/LLM/solution the answer was generated by.

In our evaluations we try to avoid human bias in grading. Sometimes an evaluator has had bad experiences or great experiences with a certain product or LLM in the past, and we don鈥檛 want them to bring that bias to the current evaluation, so when evaluating different solutions, we first strip away anything that would identify the source of the solution, so results are not biased by past positive or negative experiences.

9. Grade the value of answers in addition to making a binary determination of whether the answer has an error.

What鈥檚 right or wrong in an answer can vary enormously in terms of positive value and negative impact. For instance, consider the following answers:

A. Answer is correct in every way but is short and high level. It just gives a basic description of the legal issue as it relates to the question but doesn鈥檛 provide any references to primary or secondary law for verification, nor any nuance regarding exceptions or other considerations.

B. Answer is lengthy and nuanced, addressing multiple aspects of the question and discussing important exceptions that might apply, and it provides references with citations and links for verification, and it’s correct in every way except in one of the citations, the date is incorrect, but that鈥檚 easily verified and corrected when clicking the link from the citation.

C. Answer is incorrect in every way and all its linked references point to primary law that simply corroborate the wrong answer.

If the evaluation is simply a binary view of the number of answers that contain an error, then answer A looks good and answers B and C look equally bad. In reality, answer C is far worse and more harmful than answer B, and Answer B is likely much more valuable to the researcher than answer A.

In our evaluations, we鈥檙e looking for answer attributes that are helpful to researchers, like depth of the answer and quality of the references, and we don鈥檛 just evaluate errors in a binary way. We consider answers that are totally wrong to be far worse than answers with erroneous statements in otherwise correct and helpful answers. Similarly, we consider erroneous statements in answers based on whether they address the core questions or are tangential to it, and whether they鈥檙e contradicted in the answer or easily verified with the linked references. We鈥檇 like to eradicate all errors, of course, but some are more harmful than others.

10. Look for errors beyond gold-standard answers.

Often LLMs generate answers with information beyond the scope of a gold-standard answer. For instance, the gold-standard answer might say the answer should state that the answer to the question is no, and it should explain that with X, Y, and Z, and it should specifically cite to cases A & B and statute C.

The LLM-generated answer might state the answer is no and explain X, Y, and Z with references to A, B, and C, but it might also add a few statements about exceptions or related issues or an additional case or statute. Sometimes these additional statements are incorrect, even when everything else is correct. So, if an LLM-as-judge or human evaluator only looks at the gold-standard answer to see if the AI-generated answer is correct, that evaluation can miss errors in the additional material. This means evaluators need to do independent research beyond simply looking at the gold-standard answers to determine if an answer has an error.

11. Consider testing reliability.

LLMs often have some randomness built into them. Many have a temperature setting that can be used to minimize or eliminate this, making answers more consistent when asking the same question multiple times.

But some LLMs are better at this than others, and some integrated solutions that use LLMs in conjunction with other techniques, like RAG, don鈥檛 set temperature low to allow for more creativity in answers.

For big decisions you might be making, consider testing reliability by running the same question 20 times and seeing if any of the answers are substantially worse than the other answers to the same question.

The above are our and learnings from our extensive expertise with AI, Gen AI and LLMs over the past 30 years. At 成人VR视频 we put the customer at the heart of each of these decisions we make and are transparent that at the point of use all our AI generated answers must be checked by a human.

As we work through testing our AI products, our teams do not follow each of these steps for every test we do, sometimes we prioritize speed over accuracy of testing or vice versa, but we ensure we clearly understand the trade-off in prioritizing some of these steps and communicate this with our teams. The bigger and more important the decision we鈥檙e trying to make, the more of these steps we follow.

This is a guest post from Mike Dahn, head of Westlaw Product, and Dasha Herrmannova, senior applied scientist, from 成人VR视频.

]]>
The $28 Billion Rise of Alternative Legal Services Providers and a Looming Bifurcation in the Legal Market /en-us/posts/innovation/the-28-billion-rise-of-alternative-legal-services-providers-and-a-looming-bifurcation-in-the-legal-market/ Mon, 03 Feb 2025 00:05:11 +0000 https://blogs.thomsonreuters.com/en-us/?post_type=innovation_post&p=64727 The legal landscape is changing, and a new report reveals just how significant the shift is becoming. According to the Alternative Legal Services Providers 2025 Report, released by 成人VR视频, the ALSP market has ballooned to an estimated $28.5 billion, fueled by an 18% compound annual growth rate from 2021 to 2023. This growth signifies a powerful trend: the increasing adoption and reliance on ALSPs by both corporate legal departments and traditional law firms.

One of the most intriguing aspects of this growth is the role of technology, particularly generative AI (GenAI). The report found that a significant percentage of both law firms (35%) and corporate legal departments (40%) find ALSPs leveraging GenAI to be more appealing.

However, the rise of ALSPs and the adoption of AI is also creating a division within the legal market. The report points to a bifurcation emerging between forward-looking legal entities embracing alternative delivery models and those clinging to traditional practices. This divide is especially notable as many corporate legal departments are signaling a decrease in spending with law firms hesitant to adapt to these evolving client expectations.

鈥淭he legal industry is going through significant transformation, driven by the adoption of GenAI technology,鈥 stated Laura Clayton McDonnell, president of Corporates at 成人VR视频. 鈥淎s legal departments become more sophisticated in their use of technology, they will increasingly expect their law firms and alternative legal service providers to deliver tech-enabled services that meet their evolving needs, driving a wave of innovation and efficiency across the entire legal industry.”

ALSPThis doesn’t mean traditional law firms are or will be obsolete. Many are successfully integrating ALSPs into their workflows, recognizing the value of their specialized expertise and cost-effectiveness. The key takeaway is clear: adaptability is critical. Those who embrace innovation, whether through incorporating ALSPs, leveraging AI, or adopting other technological advancements, are better positioned for success in this evolving landscape.

The legal market is at a crossroads. The future belongs to those who are willing to adapt and evolve, embracing new technologies and service models to deliver greater value and efficiency to their clients. The Alternative Legal Services Providers 2025 Report serves as a roadmap, highlighting the trends shaping the future of the legal profession.

]]>
2025 Report on the State of the Legal Market: Top Takeaways /en-us/posts/innovation/2025-report-on-the-state-of-the-legal-market-top-takeaways/ Tue, 07 Jan 2025 14:32:30 +0000 https://blogs.thomsonreuters.com/en-us/?post_type=innovation_post&p=64371 The legal industry is ripe for innovation and law firms focused on solving the fundamental challenges surrounding technology implementation are best positioned to drive sustainable growth in the legal market. These are among the findings of the 2025 Report on the State of the US Legal Market, released today by 成人VR视频 and the Center on Ethics and the Legal Profession at Georgetown Law.

The annual report relies on data from the 成人VR视频 Institute to review the performance of U.S. law firms and explore the trends and factors shaping the U.S. legal market. Below are five takeaways from the report.

  1. A transformative shift is under way in the legal profession. Amid the evolution from traditional practices to innovative business models, law firms need to continue innovating and adapting to remain competitive, including implementing the latest technology, employing new business models and prioritizing client-centric practices.
  2. Law firm leaders should 鈥渢ake advantage of the benefits of a lucrative 2024.鈥 The report noted that firms鈥 strong 2024 performance was defined by three metrics: demand, rates and expenses. Solid demand growth 鈥 across counter-cyclical and transactional practices 鈥 coupled with law firm billing rates accelerating at their fastest pace since the Great Financial Crisis contributed to law firms鈥 soaring profits, alongside expense growth levelling off.
  3. The impact of generative AI will continue to drive shifting market factors in 2025:
  • Strategic investment in technology: Firms need to prioritize technology investments to enhance productivity and adapt to the evolving legal tech landscape in order to drive long-term growth.
  • Shifting pricing paradigms: The traditional billable hour model will be challenged by alternative pricing structures that prioritize value and client-centric approaches.
  • Evolving talent models: The composition of law firms is in flux, with a shift toward more experienced lateral hires, growth in two-tier partner structures and less emphasis on junior associate hiring.
  1. Growth may be dampened in 2025 due to potential weaker demand and global economic uncertainty. Though firms may see demand weaken in 2025, the report notes that results of the U.S. presidential election could boost demand as greater levels of economic and geopolitical instability generally see clients turn to their lawyers to mitigate risk. In addition, the 2025 outlook includes expense growth remaining at elevated levels, putting more pressure on profits.
  2. 2025 will require firms to continue adapting to the impact of generative AI and emerging technologies. While firms took steps in 2024 to ensure sustainable growth in a changing market, innovative firms that invest in technologies and implement strategies to update their business model in 2025 鈥 including how they measure and reward lawyer performance 鈥 will be best positioned to achieve ongoing success.

Download the report鈥痜or strategies law firms can use to adapt their business models and implement new technologies to thrive amid changing market demands and clients鈥 needs.

]]>
How Generative AI Is Shaping the Future of Law: Challenges and Trends in the Legal Profession /en-us/posts/innovation/how-generative-ai-is-shaping-the-future-of-law-challenges-and-trends-in-the-legal-profession/ Thu, 02 Jan 2025 09:02:08 +0000 https://blogs.thomsonreuters.com/en-us/?post_type=innovation_post&p=64292 Helping our customers achieve their desired outcomes while using 成人VR视频 products, solutions and services is at the core of what we do. We continually talk with our customers to learn firsthand about their experiences and how 成人VR视频 can apply cutting-edge technology to solve their biggest pain points.听

These conversations are more important than ever as generative AI becomes more widely implemented across the legal industry. As our customers鈥 鈥 and their clients鈥 鈥 expectations change, it鈥檚 crucial that we鈥檙e closely aligned with their collective needs.听

With this mind, 成人VR视频 and Lexpert hosted a featuring law firm leaders and industry experts discussing the challenges and trends around the use of generative AI in the legal profession.鈥疊elow are insights from an engaging and informative discussion.

Lawyers are excited to implement generative AI solutions

Law firms may not have a reputation as early adopters of technology, yet many firms are already using generative AI solutions. David Cohen, senior director, client service delivery, McCarthy T茅trault, noted his firm鈥檚 response when it rolled out CoCounsel, the professional-grade GenAI assistant, a year ago. 鈥

鈥淲e put a call out for people to sign up and get a license, and the 175 seats were immediately filled,鈥 Cohen said. 鈥淭here was a wait list where people were pleading to get access to the tool 鈥 there鈥檚 certainly a lot of interest and excitement at our firm.鈥濃

Panelists noted how firms are increasingly implementing AI for drafting legal documents as well as for predictive analytics and other complex tasks.

Cohen said McCarthy T茅trault is using AI for higher-value work, including in litigation matters to scan data troves and identify where the firm has been successful in the past. 鈥

鈥淲e鈥檙e very interested in how we get a strategic advantage,鈥 Cohen said. 鈥淲hat arguments were used, what makes for a successful pleading 鈥 that鈥檚 something we can leverage.鈥 鈥

Rikin Morzaria, principal at Kinara Law, said he鈥檚 鈥渆xperimenting with and finding solutions that work long term,鈥 including using AI for advanced tasks such as drafting memos or reviewing his own work to identify potential areas for improvement.

鈥淚 treat it as something that would be submitted by a student or an intern, that needs to be reviewed again,鈥 Morzaria said. 鈥淚t ultimately produces a better product for me in the end.鈥

Unfounded concerns about robot lawyers

Panelists shared their perspectives on generative AI solutions augmenting, not replacing, humans.

鈥淎I doesn鈥檛 have human insight or instincts, it doesn鈥檛 know the client, it can鈥檛 read a room,鈥 said Valerie McConnell, senior director of CoCounsel Customer Success at 成人VR视频. 鈥淗ow lawyers allocate their time will shift: the hope and expectation is when we clear their plates, they鈥檒l have the space to think about strategic angles.鈥

This brought up how firms鈥 use of AI is changing talent management. Cohen noted that McCarthy T茅trault revamped its training program to incorporate the integration of AI into its workflows and processes.

鈥淗aving best-in-class prompts that are shared and available to our lawyers is something we certainly weren鈥檛 discussing a couple years ago,鈥 Cohen said. 鈥

Changing billing practices and elevating services

Panelists noted that generative AI is changing how lawyers bill their clients. They discussed an example of litigators relying more on flat-fee billing because they know they can handle certain aspects of a matter more quickly. 鈥

鈥淵our time in a discovery, trial, or mediation won鈥檛 change, but a lot of that time leading up to it will,鈥 Morzaria said. 鈥淭hat allows you to work more efficiently and take on work that may not have been feasible before.鈥 鈥

Panelists also shared how generative AI is helping firms elevate their services. They called out how firms in the litigation space are using it to surface key鈥痚vidence, while firms on the transactional side are using it for contract review.听

Managing and mitigating risks

Panelists discussed how firms are addressing concerns around emerging technologies including accuracy and privacy. Cohen noted firms are focusing on 鈥渁chieving the right balance where we鈥檙e generating benefits for our clients but also managing and mitigating the risks.鈥澨

McConnell emphasized that AI is not a replacement for a lawyer鈥檚 oversight. She recommended that firms create an AI policy, use a secure software platform that allows lawyers to easily verify output and doesn鈥檛 train on firm data, and provide employee training on responsible AI use. 鈥

Generative AI is shaping the future of law, and its impact on the legal profession will be even greater in 2025 and beyond. As the market matures, legal professionals will not just desire but require AI capabilities for their workflows.鈥疘nnovative firms at the forefront, including Kinara Law and McCarthy T茅trault, are best positioned to capitalize on the potential of this transformative technology to better serve their clients.

Check out highlights of the panelists鈥 conversation and listen to their full discussion .

This is a guest post from Raghu Ramanathan, president, Legal Professionals, 成人VR视频.

]]>
Raghu Ramanathan: Reflections on Legal Generative AI One Year In /en-us/posts/innovation/raghu-ramanathan-reflections-on-legal-generative-ai-one-year-in/ Mon, 30 Dec 2024 09:02:06 +0000 https://blogs.thomsonreuters.com/en-us/?post_type=innovation_post&p=64271 I recently talked with 鈥 Ben Joyner about generative AI in the legal space, touching on everything from our company鈥檚 M&A strategy to how CoCounsel is transitioning to a multi-model product. Talking with Ben about how generative AI has shaped our industry over the past year has me reflecting on my first year with 成人VR视频.

Raghu Ramanathan, president, Legal Professionals, 成人VR视频.

Continued climb in law firm productivity

I joined 成人VR视频 in February, and a notable way we鈥檝e seen the impact of generative AI solutions is the uptick in lawyer productivity. For the first time in years, Q2 saw a majority of law firms experience productivity growth. By Q3, an astounding 64% of law firms reported productivity growth, building on the gains made in Q2.听

This uptick underscores how technology is key to boosting law firm profitability. Law firms that invest in new technology as well as adopt AI and generative AI solutions to streamline workflows and improve the efficiency and quality of their work are best positioned to improve client satisfaction and drive sustainable productivity growth.

Build, buy, partner strategy

I鈥檓 pleased with the progress 成人VR视频 has made on our vision to provide all the legal professionals we serve with a professional-grade GenAI assistant to augment their work. We鈥檝e committed to investing $100 million annually in AI over the coming years, including investing more than $200 million to incorporate responsible AI into our solutionsin the past year alone.听

This year we continued investing in the latest technology through our build, buy, partner program. On the buy side in the legal space, our acquisition of Safe Sign Technologies 鈥 a UK legal large language model (LLM) startup 鈥 in August is proving a great fit. We鈥檙e incorporating Safe Sign鈥檚 tech and talent into our industry-leading content and expertise to bring customers even greater quality and performance from our AI solutions.鈥

On the build side, we introduced 19 legal generative AI solutions in 2024. Highlights include CoCounsel 2.0, the professional-grade GenAI assistant; Claims Explorer, a generative AI skill available in鈥; CoCounsel Drafting, an end-to-end drafting solution that streamlines and improves the drafting process for legal professionals within Microsoft Word; and Mischaracterization Identification in Quick Check and AI Jurisdictional Surveys 鈥 two generative AI research features that help customers save substantial time and deliver greater confidence that legal research is accurate, thorough, and complete. We also delivered deeper integration of CoCounsel into Westlaw and Practical Law.

On the partner side, we鈥檙e working with Microsoft, OpenAI, Google and others on plugins and integrations to enhance the generative AI-powered capabilities in our solutions.鈥疎very aspect of our build, buy, partner strategy is geared toward helping our customers automate their workflows, provide powerful insights to their clients and drive efficiencies.鈥

A maturing market听

2024 saw the implementation of legal generative AI solutions as well as efforts to benchmark these solutions. Our benchmarking support is reflected in our participation in studies including Vals.ai plus two consortium efforts 鈥 from Stanford and Litig 鈥 exploring how to best evaluate legal AI.听

I believe that benchmarking can improve both the development and the adoption of AI, but it鈥檚 just鈥痮ne component in how we consider and understand the benefits AI delivers for our customers.鈥疘 look forward to our ongoing collaboration with customers and industry partners as we continue working to minimize inaccuracies and increase the usefulness of the research outcomes for generative AI solutions.

To date, 15% of law firms have adopted and implemented legal-specific generative AI solutions. I anticipate we鈥檒l soon see a wave of fast followers 鈥 eager to be perceived as innovative 鈥 that will dramatically strengthen generative AI implementation.

I can鈥檛 think of a more exciting time to have joined a business. Where our industry is at now mirrors the early internet era: initial excitement, followed by strategic integration.听

We鈥檙e fast approaching a maturing market where legal professionals will not just desire but require AI capabilities for their workflows. We鈥檒l see more implementation of generative AI solutions among legal professionals as they increasingly realize the tangible benefits.

For more on how generative AI is shaping the future of the legal profession, please check out my interview.听

This is a guest post from Raghu Ramanathan, president, Legal Professionals, 成人VR视频.

]]>