In the legal profession, many daily tasks revolve around document analysis鈥攔eviewing contracts, transcripts, and other legal documents. As the leading AI legal assistant, CoCounsel鈥檚 ability to automate document-centric tasks is one of its core legal capabilities. Users can upload documents, and CoCounsel can automatically perform various tasks on these documents, saving lawyers valuable time.

Legal documents are often extensive. Deposition transcripts can easily exceed a hundred pages, and merger and acquisition agreements can be similarly lengthy. Additionally, some tasks require the simultaneous analysis of multiple documents, such as comparing contracts or testimonies. To perform these legal tasks effectively, solutions must handle long documents without losing track of critical information.

When GPT-4 was first released in 2023, it featured a context window of 8K tokens, equivalent to approximately 6,000 words or 20 pages of text. To process documents longer than this, it was necessary to split them into smaller chunks, process each chunk individually, and synthesize the final answer. Today, most major LLMs have context windows ranging from 128K to over 1M tokens. However, the ability to fit 1M tokens into an input window does not guarantee effective performance with that much text. Often, the more text included, the higher the risk of missing important details. To ensure CoCounsel鈥檚 effectiveness with long documents, we鈥檝e developed rigorous testing protocols to measure long context effectiveness.

Why a Multi-LLM Strategy鈥攁nd Trusted Testing Ground鈥擬atter

At 成人VR视频, we don鈥檛 believe in putting all our bets on a single model. The idea that one LLM can outperform across every task鈥攅specially in high-stakes professional domains鈥攊s a myth. That鈥檚 why we鈥檝e built a multi-LLM strategy into the core of our AI infrastructure. It鈥檚 not a fallback. It鈥檚 one of our competitive advantages.

Some models reason better. Others handle long documents more reliably. Some follow instructions more precisely. The only way to know what鈥檚 best for any given task is to test them鈥攔elentlessly. And that鈥檚 exactly what we do.

Because of our rigor and persistence in this space, 成人VR视频 is a trusted, early tester and collaborator for leading AI labs. When major providers want to understand how their newest models perform in high-stakes, real-world scenarios, they turn to us. Why? Because we鈥檙e uniquely positioned to pressure-test these models against the complexity, precision, and accountability that professionals demand 鈥� and few others can match.听

• Our legal, tax, and compliance workflows are complex, unforgiving, and grounded in real-world stakes
• Our proprietary content鈥攊ncluding Westlaw and Reuters News鈥攇ives us gold-standard input data for model evaluation
• Our SME-authored benchmarks and skill-specific test suites reflect how professionals actually work鈥攏ot how a model demo looks on paper听

When OpenAI was looking to train and validate a custom model built on o1-mini, 成人VR视频 was among the first. And when the next generation of long-context models hits the market, we are routinely early testers.

A multi-model strategy only works if you know which model to trust鈥攁nd when. Benchmarking is how we turn optionality into precision.

This disciplined, iterative approach isn鈥檛 just the best way to stay competitive鈥攊t鈥檚 the only way in a space that鈥檚 evolving daily. The technology is changing fast. New models are launching, improving, and shifting the landscape every week. Our ability to rapidly test and integrate the best model for the job鈥攁t any moment鈥攊sn鈥檛 just a technical strategy. It鈥檚 a business advantage.

And all of this isn鈥檛 just about immediate performance gains. It鈥檚 about building the foundation for truly agentic AI鈥攖he kind of intelligent assistant that can plan, reason, adapt, and act across professional workflows with precision and trust. That future won鈥檛 be built on rigid stacks or static decisions. It will be built by those who can move with the market, test with integrity, and deliver products that perform in the real world.

RAG vs. Long Context

In developing CoCounsel鈥檚 capabilities, a significant question was whether and how to utilize retrieval-augmented generation (RAG). A common pattern in RAG applications is to split documents into passages (e.g., sentences or paragraphs) and store them in a search index. When a user requests information from the application, the top N search results are retrieved and fed to the LLM in order to ground the response.

RAG is effective when searching through a vast collection of documents (such as all case law) or when looking for simple factoid questions easily found within a document (e.g., specific names or topics). However, some complex queries require a more sophisticated discovery process and more context from the underlying documents. For instance, the query 鈥淒id the defendant contradict himself in his testimony?鈥� requires comparing each statement in the testimony against all others; a semantic retrieval using that query would likely only return passages explicitly discussing contradictory testimony.

In our internal testing (more on this later), we found that inputting the full document text into the LLM鈥檚 input window (and chunking extremely long documents when necessary) generally outperformed RAG for most of our document-based skills. This finding is supported by studies in the literature^1,2. Consequently, in CoCounsel 2.0 we leverage long context LLMs to the greatest extent possible to ensure all relevant context is passed to the LLM. At the same time, RAG is reserved for skills that require searching through a repository of content.

Comparing the Current Long-Context Models + testing

As discussed in our previous post, before deploying an LLM into production, we conduct multiple stages of testing, each more rigorous than the last, to ensure peak performance.

Our initial benchmarks measure LLM performance across key capabilities critical to our skills. We use over 20,000 test samples from open and private benchmarks covering legal reasoning, contract understanding, hallucinations, instruction following, and long context capability. These tests have easily gradable answers (e.g., multiple-choice questions), allowing for full automation and easy evaluation of new LLM releases.

For our long context benchmarks, we use tests from LOFT³, which measures the ability to answer questions from Wikipedia passages, and NovelQA⁴, which assesses the ability to answer questions from English novels. Both tests accommodate up to 1M input tokens and measure key long context capabilities critical to our skills, such as multihop reasoning (synthesizing information from multiple locations in the input text) and multitarget reasoning (locating and returning multiple pieces of information). These capabilities are essential for applications like interpreting contracts or regulations, where the definition of a term in one part of the text determines how another part is interpreted or applied.

We track and evaluate all major LLM releases, both open and closed source, to ensure we are using the latest and most advanced models, such as the newly updated GPT-4.1 model with its much-improved long context capabilities.

Skill-Specific Benchmarks

The top-performing LLMs from our initial benchmarks are tested on our actual skills. This stage involves iteratively developing (sometimes very complex) prompt flows specific to each skill to ensure the LLM consistently generates accurate and comprehensive responses required for legal work.

Once a skill flow is fully developed, it undergoes evaluation using LLM-as-a-judge against attorney-authored criteria. For each skill, our team of attorney subject matter experts (SMEs) has generated hundreds of tests representing real use cases. Each test includes a user query (e.g., 鈥淲hat was the basis of Panda鈥檚 argument for why they believed they were entitled to an insurance payout?鈥�), one or more source documents (e.g., a complaint and demand for jury trial), and an ideal minimum viable answer capturing the key data elements necessary for the answer to be useful in a legal context. Our SMEs and engineers collaborate to create grading prompts so that an LLM judge can score skill outputs against the ideal answers written by our SMEs. This is an iterative process, where LLM-as-a-judge scores are manually reviewed, grading prompts are adjusted, and ideal answers are refined until the LLM-as-a-judge scores align with our SME scores. More details on our skill-specific benchmarks are discussed in our previous post.

Our test samples are carefully curated by our SMEs to be representative of the use cases of our users, including context length. For each skill, we have test samples utilizing one or more source documents with a total input length of up to 1M tokens. Additionally, we have constructed specialized long context test sets where all test samples use one or more source documents totaling 100K鈥�1M tokens in length. These long context tests are crucial because we have found that the effective context windows of LLMs, where they perform accurately and reliably, are often much smaller than their available context window.

In our testing, we have observed that the more complex and challenging a skill, the smaller an LLM鈥檚 effective context window for that skill. For more straightforward skills, where we search a document for one or a few data elements, most LLMs can accurately generate answers at input lengths up to several hundred thousand tokens. However, for more complex tasks, where many different data elements must be tracked and returned, LLMs may struggle with recall to a greater degree. Therefore, even with long context models, we still split documents into smaller chunks to ensure important information isn鈥檛 missed.听

When you look at the advertised context window for leading models today, don鈥檛 be fooled into thinking this is a solved problem. It is exactly the kind of complex, reasoning-heavy real-world problem where that effective context window shrinks. Our challenge to the model builders: keep stretching and stress-testing that boundary!

Final Manual Review

All new LLMs undergo rigorous manual review by our attorney SMEs before deployment. Our SMEs can capture nuanced details missed by automated graders and provide feedback to our engineers for improvement. These SMEs further provide the final check to verify that the new LLM flow performs better than the previously deployed solution and meets the exacting standards for reliability and accuracy in legal use.

Looking Ahead: From Benchmarks to Agents

Our long-context benchmarking work is more than just performance testing 鈥� it鈥檚 a blueprint for what comes next. We鈥檙e not just optimizing for prompt-and-response AI. We鈥檙e laying the technical foundation for truly agentic systems: AI that can not only read and reason, but plan, execute, and adapt across complex legal workflows.

Imagine an AI assistant that doesn鈥檛 just answer a question, but knows when to dig deeper, when to ask for clarification, and how to take the next step 鈥� whether that鈥檚 reviewing a deposition, cross-referencing contracts, or preparing a case summary. That鈥檚 where we鈥檙e headed.

This next chapter requires everything we鈥檝e built so far: long-context capabilities, multi-model orchestration, SME-driven evaluation, and deep integration into the professional鈥檚 real-world tasks. We鈥檙e closer than you think.

Stay tuned 鈥� more on that soon.

听——-

1. Li, Xinze, et al. “Long Context vs. RAG for LLMs: An Evaluation and Revisits.” arXiv preprint arXiv:2501.01880 (2024).听
2. Li, Zhuowan, et al. “Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach.” Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2024.听
3. Lee, Jinhyuk, et al. “Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?.” arXiv preprint arXiv:2406.13121 (2024).听
4. Wang, Cunxiang, et al. “Novelqa: Benchmarking question answering on documents exceeding 200k tokens.” arXiv preprint arXiv:2403.12766 (2024).听

听

听

The Emergence of LLMs in Automatic Evaluation

Traditional evaluation metrics in NLG primarily rely on comparing the generated text to reference texts using word overlap measures. These metrics, while useful, often fall short of capturing the nuances of language quality, coherence, and relevance. For example, suppose we have a reference summary 鈥淭he cat is on the mat.鈥� and a generated summary from a model 鈥淎 feline is resting on a rug.鈥�. If we use ROUGE as a metric for the evaluation, ROUGE only considers lexical overlap and cannot capture the semantic similarity between words or phrases. These summaries have the same meaning but would score poorly on ROUGE due to low word overlap.

LLMs have demonstrated an exceptional understanding of language, context, and semantics, making them attractive candidates for evaluating generated text. They can assess factors like fluency, coherence, and even factual accuracy, which are crucial for more sophisticated and context-aware evaluations. For instance, LLMs can be fine-tuned to understand the specific jargon and style of a particular domain, such as medical or legal texts, making them great potential evaluators.

The Promise of LLMs in Automated Evaluation

The recent technological advancements of LLMs have encouraged the development of LLM-based evaluation methods in various tasks and systems. LLMs can offer several advantages in the evaluation of NLG tasks:

• Context-Aware Evaluation: Unlike traditional metrics, LLMs can comprehend the context and generate evaluations that account for the subtleties and intricacies of human language.
• Scalability: LLMs can evaluate large volumes of text quickly and consistently, offering scalability that human evaluators cannot match.
• Reduced Subjectivity: Automated evaluation can minimize the subjective bias that human evaluators might introduce, leading to more consistent and objective assessments.

How to use LLM as an Evaluator

LLM-based evaluators are conceptually much simpler than traditional automatic evaluators for evaluation. While traditional evaluation methods rely on predefined metrics and comparisons to reference datasets, LLM-based evaluators work by directly assessing the generated text.

Figure 1 shows an overview of the LLM-based evaluation frameworks. To evaluate the quality of the text, you embed it into a prompt template that contains the evaluation criteria, then provide this prompt to an LLM. The LLM then analyzes the text based on the given criteria and provides feedback on its quality. This approach bypasses the need for extensive preprocessing and reference comparisons, making the evaluation process more straightforward and versatile.

Figure 2 illustrates the step-by-step workflow of evaluating summaries using clarity as a metric. A document and its corresponding summary are inputs. The summary is then embedded into a pre-formulated prompt template that includes detailed evaluation criteria, such as clarity, defined on a 1-to-5 scale. This prompt is then inputted to an LLM for automated analysis and evaluation. Based on the specified criteria, the LLM evaluator reviews the summary, assigns a score, and provides justification for the rating.

The Limitations of LLMs: Why Humans Are Still Indispensable

Despite the promising capabilities of LLMs, there are significant limitations that necessitate the continued involvement of human experts in the evaluation process, especially for domain-specific tasks:

• Lacking Specialized Domain Knowledge: Domain-specific tasks often involve complex knowledge. LLMs are typically trained as general-purpose assistants, and they still lack specialised domain knowledge. In contrast, subject matter experts bring in- depth domain knowledge obtained by years of dedicated training and education.
• Evolving Knowledge: Especially in fast-evolving fields like medicine and technology, staying up-to-date with the latest information is challenging for static models. Human experts, however, continuously learn and adapt to new knowledge and standards.
• Handling Ambiguities: In specialized domains, the language can be highly ambiguous and complex, and the ability to disambiguate based on deep contextual knowledge is something LLMs still struggle with.
• Ethical and Bias Concerns: LLMs can inadvertently reinforce biases present in their training data. Human oversight is crucial to identify and mitigate these biases, ensuring fair and ethical evaluations.

The Human-in-the-Loop Model: Best of Both Worlds

To harness the strengths of LLMs while addressing their limitations, a human-in-the-loop approach is essential. This combines the efficiency and scalability of LLMs with the expertise and judgment of human evaluators:

• Initial Screening: LLMs can perform initial screenings and provide preliminary evaluations, identifying clear cases of high or low quality.
• Expert Review: Human experts then review and refine these evaluations, focusing on cases that require nuanced understanding or where the LLM鈥檚 assessment is inadequate.
• Continuous Feedback Loop: Feedback from human evaluators can be used to fine-tune and improve LLMs, creating a continuous improvement cycle.

Conclusion

The integration of LLMs into the automatic evaluation of NLG tasks marks a significant step forward in the field of NLP. However, for domain-specific evaluations, the complexity and nuance of human language still necessitate human experts. By adopting a human-in- the-loop approach, we can leverage the best of both worlds: the speed and scalability of LLMs and the depth and discernment of human evaluators. This constructive interaction ensures that we maintain high standards of accuracy, fairness, and relevance in evaluating natural language generation tasks, ultimately driving the field towards more sophisticated and reliable applications.

This post was written by Grace Lee, lead applied scientist at 成人VR视频 Labs (TR Labs).

Note. This work has been done as part of the internship of Hossein A. (Seed) Rahmani at 成人VR视频 Labs (TR Labs).

Continued climb in law firm productivity听

I joined 成人VR视频 in February, and a notable way we鈥檝e seen the impact of generative AI solutions is the uptick in lawyer productivity. For the first time in years, Q2 saw a majority of law firms experience productivity growth. By Q3, an astounding 64% of law firms reported productivity growth, building on the gains made in Q2.听听

This uptick underscores how technology is key to boosting law firm profitability. Law firms that invest in new technology as well as adopt AI and generative AI solutions to streamline workflows and improve the efficiency and quality of their work are best positioned to improve client satisfaction and drive sustainable productivity growth.听

Build, buy, partner strategy听

I鈥檓 pleased with the progress 成人VR视频 has made on our vision to provide all the legal professionals we serve with a professional-grade GenAI assistant to augment their work. We鈥檝e committed to investing $100 million annually in AI over the coming years, including investing more than $200 million to incorporate responsible AI into our solutions鈥�in the past year alone.听听听

This year we continued investing in the latest technology through our build, buy, partner program. On the buy side in the legal space, our acquisition of Safe Sign Technologies 鈥� a UK legal large language model (LLM) startup 鈥� in August is proving a great fit. We鈥檙e incorporating Safe Sign鈥檚 tech and talent into our industry-leading content and expertise to bring customers even greater quality and performance from our AI solutions.鈥�听

On the build side, we introduced 19 legal generative AI solutions in 2024. Highlights include CoCounsel 2.0, the professional-grade GenAI assistant; Claims Explorer, a generative AI skill available in鈥�; CoCounsel Drafting, an end-to-end drafting solution that streamlines and improves the drafting process for legal professionals within Microsoft Word; and Mischaracterization Identification in Quick Check and AI Jurisdictional Surveys 鈥� two generative AI research features that help customers save substantial time and deliver greater confidence that legal research is accurate, thorough, and complete. We also delivered deeper integration of CoCounsel into Westlaw and Practical Law.听

On the partner side, we鈥檙e working with Microsoft, OpenAI, Google and others on plugins and integrations to enhance the generative AI-powered capabilities in our solutions.鈥疎very aspect of our build, buy, partner strategy is geared toward helping our customers automate their workflows, provide powerful insights to their clients and drive efficiencies.鈥��听

A maturing market听听

2024 saw the implementation of legal generative AI solutions as well as efforts to benchmark these solutions. Our benchmarking support is reflected in our participation in studies including Vals.ai plus two consortium efforts 鈥� from Stanford and Litig 鈥� exploring how to best evaluate legal AI.听听

I believe that benchmarking can improve both the development and the adoption of AI, but it鈥檚 just鈥痮ne component in how we consider and understand the benefits AI delivers for our customers.鈥疘 look forward to our ongoing collaboration with customers and industry partners as we continue working to minimize inaccuracies and increase the usefulness of the research outcomes for generative AI solutions.听

To date, 15% of law firms have adopted and implemented legal-specific generative AI solutions. I anticipate we鈥檒l soon see a wave of fast followers 鈥� eager to be perceived as innovative 鈥� that will dramatically strengthen generative AI implementation.听

I 肠补苍鈥檛 think of a more exciting time to have joined a business. Where our industry is at now mirrors the early internet era: initial excitement, followed by strategic integration.听听

We鈥檙e fast approaching a maturing market where legal professionals will not just desire but require AI capabilities for their workflows. We鈥檒l see more implementation of generative AI solutions among legal professionals as they increasingly realize the tangible benefits.鈥�鈥�听

For more on how generative AI is shaping the future of the legal profession, please check out my interview.听听

This is a guest post from Raghu Ramanathan, president, Legal Professionals, 成人VR视频.听

成人VR视频 President and CEO Steve Hasker shared a state of the industry outlook, noting generative AI is as disruptive and transformative as previous technology shifts yet is happening even faster. He emphasized what differentiates 成人VR视频, including investments the company is making in generative AI to enable professionals to accelerate and streamline entire workflows and deliver more value for clients. 听听

Hasker said 成人VR视频 has invested more than $200M in AI in the last year. He discussed the company鈥檚 vision to provide each professional it serves with an AI assistant; the launch of CoCounsel 2.0, which generates answers three times faster than the previous version; and new work with Microsoft on autonomous agents to increase revenue, reduce costs, and scale impact for customers.鈥�听

Tax, Audit & Accounting听听

鈥淎I is not just changing the landscape of accounting, it’s reshaping it.鈥� That was the message from Elizabeth Beastrom, president of Tax & Accounting at 成人VR视频. 听

While the profession sees AI as a game-changer to help them work differently, tax and accounting professionals also continue to wrestle with the perennial challenge of a talent shortage. This, combined with escalating complexity and more tax regulations, as well as changing client expectations, leaves tax professionals in need of a critical solution.听

成人VR视频 sees the potential of AI to help alleviate these challenges by augmenting human capabilities. Automating mundane, time-consuming tasks will enhance efficiency for tax professionals, helping them reclaim time to channel into higher value tasks. 成人VR视频 is working to bring the power of generative AI, machine learning and automation into its solutions in the following ways:

1. Saving time in tax preparation:

Coming in beta during the upcoming busy season, 成人VR视频 will launch an AI-assisted tax preparation experience to increase firm efficiency. The solution combines the power of CoCounsel, 成人VR视频 professional-grade generative AI assistant, with workflow automation and software integrations. It supports the delegation of data gathering to simplify mundane tasks and automate tax preparation. 成人VR视频 research shows that customers using this solution will save at least two hours per 1040 tax return on average.听

2. Supporting firms鈥� growth with advisory: 听

As client expectations continue to evolve, they鈥檙e increasingly looking to their accountants as trusted advisors. Firms of all sizes are focusing on growing their advisory practices to help bring their clients additional value, as well as supporting their growth. In 2025, the 成人VR视频 Advisory Solution will combine the power of CoCounsel and Checkpoint content to identify advisory opportunities. Advisory services are integrated directly into a firm鈥檚 practice, with technology empowering junior staff to take on higher-value advisory work and seasoned professionals to move beyond technical expertise to value-added synthesis. 听

鈥淚t helps firms build their advisory practice with confidence to deliver unprecedented value to meet clients鈥� evolving needs,鈥� said Nancy Hawkins, vice president of Product Management, Research.听

3. Transforming audit efficiency:听

Halving sample sizes, boosting efficiency and sharpening the focus on high-risk areas are all at the heart of 成人VR视频 Audit Intelligence Analyze solution, which launched in October. Further functionality will be coming in 2025 as it expands the Audit Intelligence suite capabilities. 鈥楾est鈥� will support with automating substantive testing with dynamic transaction tracing, while 鈥楶lan鈥� will harness full data populations with cutting-edge analytics for superior risk assessment. Both will launch with beta programs next year, along with the addition of CoCounsel to the Audit Intelligence suite.听

All three solutions 鈥� Review Ready, 成人VR视频 Advisory Solution and the Audit Intelligence suite 鈥� will be further enhanced with 惭补迟别谤颈补鈥檚 generative and agentic AI capabilities.听

Corporates听

Laura Clayton McDonnell, president of the Corporates segment, shared how enterprise technology, including AI and generative AI, is revolutionizing the profession with innovative and emerging solutions. She emphasized that companies are taking a streamlined and proactive approach to addressing risk and compliance across the enterprise, while driving towards their business goals, will maintain their competitive advantage. Clayton McDonnell also shared how organizations are using solutions including ONESOURCE Pagero, CoCounsel Core, Legal Tracker, Checkpoint Edge with CoCounsel and CLEAR to solve challenges and realize value for their business.听

In addition, Ray Grove, head of Corporate Tax and Trade, 成人VR视频, highlighted the company鈥檚 efforts to build a seamless, integrated compliance network, and Kevin Appold, vice president of US Public Records, 成人VR视频, shared how the company鈥檚 risk and fraud solutions play a critical role in the convergence of compliance and commerce. Also, Valerie McConnell, senior director of CoCounsel Customer Success, discussed how CoCounsel is transforming the general counsel鈥檚 office.听

Legal听

A highlight from the Legal Professionals segment included an in-depth look at the 成人VR视频 2025 AI product roadmap from David Wong, chief product officer; Mike Dahn, head of Westlaw Product; and Valerie McConnell, senior director of CoCounsel Customer Success. They outlined upcoming generative AI features and innovations to support legal professionals, including deeper integration of CoCounsel 2.0 in Westlaw and Practical Law plus generative AI research features including Claims Explorer, Mischaracterization Identification in Quick Check and AI Jurisdictional Surveys.听

Legal SYNERGY attendees also participated in interactive sessions and CLE courses on advanced prompting techniques, the science behind large language models, and optimizing generative AI for tasks like drafting and legal research. Sessions offered attendees a comprehensive view of the future of AI in law.鈥�听

SYNERGY 2024 also included several customer panels and executive briefing sessions. Watch the Innovation Blog for highlights from these sessions and for 2025 product and innovation highlights.听

鈥淎s fast as it started, it really feels like in the last year, there’s been an even more rapid acceleration, and many companies racing to become leaders in this field, including 成人VR视频,鈥� said Joel Hron, chief technology officer, 成人VR视频.

Hron and David Wong, chief product officer, 成人VR视频, shared their takes on the most significant advances in generative AI technology, including improvements in accessibility to the technology, with more developer tools alongside reduced costs and more out-of-the-box capabilities.

Wong said he鈥檚 most excited about large language models鈥� ability to have longer context windows, enabling them to keep more information in their short-term memory and answer ever-more complex questions.

鈥淭hat鈥檚 critical for the way 成人VR视频 uses a lot of these models,鈥� Wong said.

鈥淭he agentic behaviors of the models have become more robust in their ability to plan and ability to use reason over complex information,鈥� Hron added.

They also discussed balancing the need to innovate and go fast with the need for ethical, responsible and high-quality AI development.

Wong noted how 成人VR视频 is best positioned to develop professional-grade AI, grounded in fact and data. He emphasized customers鈥� need for measurable solutions, so they can discern tools鈥� accuracy rates, as well as the need for security and privacy.

Wong said 成人VR视频 has the scale and infrastructure to understand customers鈥� needs and develop solutions to solve their biggest challenges, guided by a philosophy and process that ensures the right balance between moving fast and ensuring quality.

Hron said the company鈥檚 human-centric approach to AI development is key.

鈥淥ur human expertise at 成人VR视频 and the level of rigor and quality we put behind both our content and our products for many years has really been a cornerstone of our brand,鈥� Hron said.

Hron said the iterations between technology and domain experts are crucial to how 成人VR视频 helps customers streamline their workflows with AI, such as with AI-Assisted Research on Westlaw Precision and CoCounsel Core.

They also highlighted the 成人VR视频 acquisition of Materia, an AI assistant and platform for accounting and auditing professionals.

鈥淚t鈥檚 a reinforcement of our belief in AI assistants being in the hands of every professional and a reinforcement of our commitment around AI across our entire product portfolio,鈥� Hron said.

He added that 惭补迟别谤颈补鈥檚 strengths have included leaning into the long context and multimodal capabilities of generative AI as well as enabling agentic behavior.

Hear more of Wong and Hron鈥檚 insights on Materia as well as the evolution of generative AI in the of the TechConnect series, which brings diverse and dynamic perspectives from all corners of the technology world with thought-provoking questions and conversation.

Why evaluating legal skills is complicated听

Traditional legal benchmarks usually rely on multiple-choice, true/false, or short-answer formats for easy evaluation. But these methods aren鈥檛 enough to assess the complex, open-ended tasks lawyers encounter daily and that large language model (LLM)-powered solutions like CoCounsel are built to perform.听

CoCounsel鈥檚 skills produce nuanced outputs that must meet multiple criteria, including factual accuracy, adherence to source documents, and logical consistency. These are difficult outputs to evaluate using true/false tests. On top of that, assessing the 鈥渃orrectness鈥� of legal outputs can be subjective. For instance, some users prefer detailed summaries, others prefer concise ones. Neither is 鈥渨rong,鈥� it just comes down to preference, which makes it difficult to consistently automate evaluations.听

To make it even more complicated, each CoCounsel skill often involves multiple components, with the LLM handling only the final stage of answer generation. For example, the Search a Database skill first uses various non-LLM-based search systems to retrieve relevant documents before the LLM synthesizes an answer. If the initial retrieval process is substandard, the LLM’s performance will be compromised. So, our evaluation must consider both LLM-based and non-LLM-based aspects, to make sure our assessment of the whole is accurate.听

How we benchmark听

Our benchmarking process begins long before putting CoCounsel through its paces. Whenever a significant new LLM is released, we test it across a wide suite of public and private legal tests, such as the dataset created by our Stanford collaborators, to assess their aptitude for legal review and analysis. We then integrate the LLMs that perform well in these initial tests with the CoCounsel platform, in a staging environment, to evaluate how they perform under real-world conditions.听

Then we use an automated platform to run a battery of test cases created by our Trust Team (more on this below), to evaluate the output that comes from this experimental integration. If the results are promising, we conduct additional manual reviews using a skilled team of attorneys. When we see an improvement in performance compared to previous benchmarks, then we start talking as a team about how it might improve the CoCounsel experience for our users.听

How we test听

Our Trust Team has been around as long as CoCounsel has.听 This group of experienced attorneys from diverse backgrounds 鈥� in-house counsel, large and small law firms, government, public policy 鈥� is dedicated to continually rigorously testing CoCounsel performance. 听

We continue to follow a process that鈥檚 been integral to all our performance evaluation since CoCounsel鈥檚 inception: Our Trust Team creates tests representative of the real work attorneys use CoCounsel for and runs these tests against CoCounsel skills. When creating a test, they first consider what the skill鈥檚 for and how it might be used, based on their own insights, customer feedback, and secondary sources. Once the test is created, the attorney tester manually completes the test task, just as a lawyer would, to create an answer key 鈥� what we refer to as an 鈥渋deal response.鈥� These tests and their corresponding ideal responses then undergo peer review. Being this meticulous is crucial, because the quality of our ideal responses determines the benchmark for a passing score.鈥��听

Once the ideal response has been created, a member of the Trust Team runs the test, using the applicable CoCounsel skill to complete the task just as a user would. An attorney tester reviews the output, referred to as our 鈥渕odel response.鈥� Then they point-by-point compare CoCounsel鈥檚 response to the ideal response, identifying differences and assessing whether these differences deviate from the ideal response in a way or to a degree that would make the skill’s output incomplete, incorrect, or misleading. It is important to note that a test can be failed for many reasons, not just because an answer is outright wrong. For instance, some skills (e.g., Summarize) are more inherently subjective than others and do not lend themselves to a right or wrong answer. Accordingly, even though attorneys鈥� opinions of the 鈥渃orrect鈥� answer can reasonably vary, we try to maintain the highest grading standards and 鈥渇ail鈥� the test if it is missing a single detail the attorney-tester considers to be Critical.听听

Based on this manual evaluation, we create a set of evaluation instructions tailored to the ideal output unique to each skill, capturing the needs of each skill so an LLM can鈥痬atch our human reviewers鈥� process. To illustrate this, let’s look at a real example using CoCounsel鈥檚 Extract Contract Data skill. After running the test in our automated evaluation platform, here鈥檚 what an attorney reviewer would see when reconciling the model response with the LLM鈥檚 evaluation of it:鈥�听

• Question: If the customer is acquired through a merger, can the agreement be assigned to the acquiring entity without obtaining the provider’s prior written consent?鈥�听

• • Trust Team ideal response:听

• • Short answer: Yes, with 30 days鈥� prior written notice. 鈥�听

• • Detailed answer: Yes, if the customer is acquired through a merger, the agreement can be assigned to the acquiring entity without obtaining the provider’s prior written consent; the customer is only required to provide the provider with 30 days鈥� prior written notice. The clause specifies that either party may assign its rights or delegate its obligations without such consent, provided that 30 days鈥� prior written notice is given to the other party, to an entity that acquires all or substantially all of the business or assets of such party, including by merger.鈥�听

• Model response:鈥�听

• • 鈥疭hort answer: Yes, with notice.鈥�听

• • Detailed answer: The agreement can be assigned to the acquiring entity without obtaining the provider’s prior written consent if the customer is acquired through a merger. The clause provides an exception to the general rule requiring consent for assignment. Specifically, it states that either party may assign its rights or delegate its obligations without such consent to an entity that acquires all or substantially all of the business or assets of the party, including through a merger, provided that 30 days鈥� prior written notice is given to the other party.鈥�听

• • Automated evaluation analysis: The model response captures the correct information that the agreement can be assigned to the acquiring entity without obtaining the provider’s prior written consent but requires 30 days鈥� prior written notice. However, it incorrectly states the short answer as 鈥淵es, with notice鈥� instead of 鈥淵es, with 30 days鈥� prior written notice,鈥� as per the ideal answer. This is a minor difference and does not change the meaning significantly.鈥�听

In this instance, the model response included a minor discrepancy from the attorney-authored ideal response. But the LLM鈥檚 evaluation of the response accurately determined that the answer remained sufficient, because it captured the complete notice requirement elsewhere in the response.听听

Our ideal-response approach provides two key advantages over assertion-based evaluations. It excels at identifying deviations from attorney expectations, including hallucinations. And it pinpoints extraneous or inconsistent information that, while not technically a hallucination, could make even a complete response incorrect if that information introduces logical inconsistencies, which would result in a failing score.鈥�
鈥�
We rely on our Trust Team to create well-defined ideal responses and auto-evaluation instructions and to determine if a test case passes or fails. A skill鈥檚 output definitively fails if it falls short of this ideal because of material omissions, factual incorrectness, or hallucinations. However, we recognize that many legal issues aren鈥檛 black-and-white, and the 鈥渃orrect鈥� answer could be open to reasonable disagreement. To address this, we peer review ideal responses in cases when the answer might require a second opinion. And we might eliminate tests when we find insufficient agreement among the attorney testers. This is how we both ensure that our passing criteria remain rigorous and account for the nuanced nature of legal analysis.鈥�

Maintenance and improvement听

Creating a skill test set is only the beginning. Once we begin using it, the Trust Team continually monitors and refines it by manually reviewing failure cases from the automated tests and spot-checking passing samples to make sure the automated evaluation is in line with human judgments. We also regularly add tests to cover more use cases and capture user-reported issues, which could lead to further iterations of the tests submitted for automated evaluation and their success criteria. 鈥�听

By following this process, every night we can execute, across all CoCounsel skills, more than 1,500 tests on our automated platform under attorney oversight, which combined with manual testing means we鈥檝e run more than 1,000,000 tests since CoCounsel鈥檚 launch. And it empowers us to quickly identify areas for improvement, which is vital to ensuring CoCounsel remains the most trustworthy AI legal assistant available.听听

Conclusion听

, we explored what it means for an AI tool to be “professional-grade” and why that standard is crucial for professionals in high-stakes fields like law. This post takes that concept further by diving into how we benchmark CoCounsel to ensure it meets those rigorous standards. By understanding the extensive testing that goes into evaluating its performance, you can see how CoCounsel consistently delivers the reliability and accuracy expected of a true professional-grade GenAI solution.听

To promote the transparency my team and I believe is necessary in the legal AI field, we鈥檝e decided to release some of our performance statistics for the first time and a sample of the tests that are used to arrive at the figures below applying the criteria referenced within this article. Check out our results .

This听is a guest听post from听Jake Heller,听head听of听CoCounsel,听Thomson听Reuters.

Below, CJ Lechtenberg, senior director, Westlaw Product Management, 成人VR视频, shares her insights on developing Mischaracterization Identification, a generative AI capability to help detect mischaracterizations and omissions in legal briefs.

In the five years since Quick Check was introduced, you鈥檝e added many enhancements including Quick Check Contrary Authority Identification, Quick Check Judicial and Quick Check Quotation Analysis. How did integrating generative AI make the Mischaracterization Identification enhancement different than previous ones?

Lechtenberg: This enhancement takes researchers beyond the step of knowing what might be a potential mischaracterization to an explanation of why something might be a potential mischaracterization 鈥� and that is radically different from any feature we鈥檝e deployed in Quick Check before.

I鈥檓 sure it鈥檒l come as no surprise when I say that generative AI is just a completely different beast. Lay people may think about the law as being black and white.听You can do this; you 肠补苍鈥檛 do that.听But legal professionals know that the law is really a sea of varying shades of gray. With machine learning, we wrestled with how we could ever give the machine enough data to figure out all the different ways an attorney may mischaracterize the law.

In Quick Check Quotation Analysis prior to the Mischaracterization Identification enhancement, we highlighted the actual textual differences 鈥� additions, omissions, and changes 鈥� in the quotations and showed the context around the quotes.听Doing so certainly saved researchers a significant amount of time and helped them spot issues they might not otherwise find, but the onus was still on researchers to review everything and determine what the precise differences were and how material they might be, if at all.听Even with the additional context provided, it could still be difficult to determine whether the quotations were taken out of context, especially if the quotes themselves didn鈥檛 appear to be different.

In developing Mischaracterization Identification, we recognized that the task of analyzing quotations and their context is so nuanced that attorneys will have different expectations for whether a mischaracterization occurred, so we needed to provide more than just categorizations. We found that large language models (LLMs) can generate nuanced descriptions of potential mischaracterizations, versus just explicit categorizations, and do it well, which is hugely beneficial for this type of task.

How will using Mischaracterization Identification give legal professionals and law firms a competitive advantage? How will judges using it benefit?听

Lechtenberg: The advantages of using the new Mischaracterization Identification are substantial for both legal professionals and the judiciary 鈥� both in terms of speed of review and quality of work product.听When we launched Quick Check Quotation Analysis in 2020, customers, both legal professionals and the judiciary, lamented about how time-consuming it is for them to review quotations and how challenging it is to spot differences. It is a mentally taxing task and often our brains fill in the blanks 鈥� interpreting what we think a brief maybe should say but actually doesn鈥檛.听 Attorneys never have a surplus of time, so the last thing they want to do is spend the little bit they have on the most tedious of tasks and still end up missing potential problems.

For attorneys, Mischaracterization Identification will help them efficiently and accurately make contextual misstatement and omission determinations for their opponents鈥� and their own quotations and the context surrounding those quotations. The fear of missing their own mistakes is very real for attorneys, but the possibility of missing the opportunity to capitalize on their opponents鈥� mistakes is an even larger concern. This new enhancement reduces both of those worries and will help attorneys be even better advocates for their clients.

Judges will also be able to effectively review the filings of parties in matters before them much faster. Attorneys owe a duty of candor to the judiciary and the Mischaracterization Identification feature will help flag any potential issues quickly. An added benefit, which members of the judiciary or their staff perhaps haven鈥檛 considered, is the ability to analyze their own orders and opinions to ensure that they haven鈥檛 made mistakes that could be appealed. This new enhancement will help alert judges and law clerks to potential issues before they finalize their opinions.

What early feedback are you hearing from customers?

Lechtenberg: In a recent survey, 93% of law firm professionals told us they鈥檝e seen opposing counsel misuse a quotation, 66% said they鈥檝e seen misrepresentations by an associate or colleague, and 65% of corporate respondents said they check the accuracy of outside counsel鈥檚 quotations.听The need to review opposing counsels鈥� and colleagues鈥� briefs for mischaracterizations of the law is still a very real issue for attorneys. Likewise, attorneys have said they鈥檙e always concerned about the accuracy of their work and that maintaining their reputation as a credible litigator with courts and opposing counsel is incredibly important.

Customers are extremely excited about this new Quick Check enhancement to help combat these concerns and we鈥檝e received positive feedback from them.听One law firm managing partner stated that they would use this tool a lot.听They cite-check their opponents鈥� briefs, so any shortcuts are beneficial to them. They recognize that most of the time, errors are harmless, but occasionally there are things they want to bring to the court鈥檚 attention and this feature will help them spot those issues more quickly and accurately.

Another law firm partner said this new feature is the 鈥渦ltimate security blanket鈥� because everything attorneys do is based on their credibility, and this feature alerting them to quotes being taken out of context before filing with the court would calm some of those fears.

Any surprising or unexpected moments as the team worked on developing or launching Mischaracterization Identification?

Lechtenberg: The fact that we鈥檝e accomplished this now with the use of LLMs is exciting, a little surprising and a long time coming. I鈥檓 an attorney who leads a team of attorneys; we鈥檙e literally trained to question everything and have a healthy dose of skepticism.听But I have been dreaming about a mischaracterization identification feature in Quick Check ever since we developed Quotation Analysis more than five years ago. At my core, I believed someday this could be achieved, but for years traditional machine learning approaches were just not powerful or nuanced enough to do it well.

Leveraging LLMs for a use case like this is a new frontier like we鈥檝e never seen before.听The LLM鈥檚 ability to analyze text from an uploaded document and compare that text to the text from the cited case used to support the argument and then go beyond highlighting textual differences and provide an actual explanation of what may be problematic 鈥� whether that鈥檚 a selective quote, omitted context or a misinterpreted holding 鈥� has been absolutely astounding.

What鈥檚 the one thing you want everyone to know about Mischaracterization Identification?

Lechtenberg: Mischaracterization Identification will not only help researchers spot contextual misstatements and omissions in their opponents鈥� or their own quotations and contextual statements faster and with more accuracy, but most importantly it will help them understand why those misstatements or omissions may be problematic. And, spoiler alert: Mischaracterization Identification is just the beginning of how 成人VR视频 will harness the power of generative AI in Quick Check to solve important customer problems.

For more on Mischaracterization Identification, read the press release or check out the by Mike Dahn, head of Westlaw Product Management, 成人VR视频.

In April of this year, we shared our vision to provide a GenAI assistant for each professional we serve. CoCounsel embodies our ongoing efforts to augment professionals鈥� work with GenAI skills, enabling professionals to accelerate and streamline entire workflows to increase efficiency, produce better work, and deliver more value for their clients.鈥疧ur continued investment in GenAI is driven to enable professionals across industries to accelerate and streamline entire workflows through a single GenAI assistant.

We believe our investment in GenAI 鈥� along with our integration to customer data as well as third-party integrations 鈥� extends the value customers derive from CoCounsel beyond our connected experience and our verified and trusted content. Our work with Microsoft, for example, includes CoCounsel integrations across Word, Outlook and Teams 鈥� meeting professionals where they鈥檙e already working.

AI and large language models are proving to be powerful tools that deliver efficiency gains and strengthen research practices for our customers. Yet our efforts to redefine work with GenAI are rooted in our strong foundation of editorial enhancements, authoritative content and technological expertise, alongside our long history of working closely with customers. That鈥檚 why we continue to build out AI- and non-AI-powered solutions to help with the entire workflow for legal, tax, and risk and compliance professionals. While AI may not be perfect, it can significantly help professionals reduce the amount of work and manage more complex and substantive work more efficiently. We collaborate with our customers to help them understand that AI is an accelerant rather than a replacement for their own research.

Benchmarking expectations

As a leader in innovation and AI research, we recognize the role that independent benchmarking brings in ensuring the accuracy, transparency, and accountability of evolving GenAI solutions. We believe that benchmarking can improve both the development and the adoption of AI. We also see it as one component in a broad range of ways we consider and understand the benefits AI delivers for our customers. We work with our customers as their trusted partners for change, helping them to confidently understand and adopt new technologies, looking at both their immediate value and role in long-term transformation, and leveraging our deep understanding of their businesses.

At 成人VR视频, our understanding of the holistic value of our products is based on customers鈥� usage and the benefits they derive. Our customers have run more than 2.5M searches through AI-Assisted Research鈥痮n Westlaw Precision since its launch late last year, and they tell us it’s saving time and improving productivity. Similarly, internal testing of CoCounsel’s skills has yielded impressive results, particularly with regards to CoCounsel鈥檚 document review capabilities.

Our benchmarking support is reflected in our participation in studies including Vals.ai as well as two consortium efforts 鈥� from Stanford and Litig 鈥� exploring how to best evaluate legal AI. We are submitting CoCounsel AI skills to the Vals.ai benchmarking study in five areas of evaluation 鈥� Doc Q&A, Data Extraction, Document Summarization, Chronology Generation, and E-Discovery.

is a first attempt at establishing a standard, and so we should view this work as the first iteration and an opportunity to learn versus treating it like a gold standard. For example, one limitation of the benchmarking methodology is that each vendor鈥檚 results are evaluated based on the text output alone, removed from the interface and experiences of the individual products. This discounts the work each vendor has done to design interfaces and safety features to minimize the harms of errors. This reinforces the need for a holistic evaluation of each product being tested, ideally as designed for the user.

Looking ahead, my expectation is that, while accuracy will continue to improve, no products will produce answers entirely free of errors. And as we鈥檝e shared with our customers, every AI product requires human expertise for verification and review 鈥� regardless of the accuracy rate. As the current approach to benchmarking rates an accuracy percentage 鈥� we need to be very clear on this point 鈥� whether the product produces a score in the low or high 90th percentile, all answers still must be checked 100% of the time.

I look forward to our ongoing collaboration with customers and industry partners as we continue our work towards minimizing inaccuracies and increasing the usefulness of the research outcomes for GenAI tools and all our solutions.

However, not using GenAI for work at all is a non-option. 77% of professionals believe AI will have a high or transformational impact on their work over the next five years, and 78% call AI a 鈥渇orce for good鈥� in their profession. In fact, 50% of law firms named AI among their top five strategic priorities for the next 18 months. If there were still doubt, there definitely isn鈥檛 anymore: GenAI is here to stay.听

So how can conscientious 鈥� and forward-thinking 鈥� professionals make the most of this generational technology while guarding against its drawbacks? How do you know if the GenAI solution you鈥檙e considering will live up to your professional obligation to work ethically and ensure your clients鈥� data is securely handled? Is any GenAI product trustworthy? Is it even possible for tools built on large language models (LLMs) such as GPT-4o from OpenAI and Google鈥檚 Gemini, all of which are known to hallucinate, to be safe enough to use professionally?听

Yes, it is possible. 鈥淏uilt on鈥� is the key. When we launched our GenAI assistant, CoCounsel, our product and engineering teams delivered on the challenge of creating a product that could take advantage of LLMs鈥� tremendous raw power while eliminating as many of their serious limitations as possible 鈥� like hallucinations 鈥� that curb the professional utility of models when used on their own. What makes the current generation of LLMs truly extraordinary, then, is not what they alone can do, but what they enable.听听听听听听

Using a model directly should be done with great caution and exposes users to risk if they use the output professionally. CoCounsel, on the other hand, harnesses that power and has engineered robust, well-tested accuracy, privacy, and security controls around it. In short: LLMs are the world鈥檚 most incredible engines. CoCounsel uses that engine to take you incredible places 鈥� places you couldn鈥檛 reach without these LLMs 鈥� safely.听

Why can professionals trust CoCounsel?听

We鈥檝e applied our technical and domain expertise to leading LLMs in creating and continuing to optimize CoCounsel, a first-of-its-kind product that both does more than LLMs can and corrects the problems that make them unsuitable on their own.听

In short, CoCounsel is a professional-grade GenAI assistant. And no professional should use a GenAI solution that isn鈥檛.听

What does it take for a GenAI assistant to be professional-grade? At a bare minimum 鈥� without which it should not be trusted for your work 鈥� it must be:听

1. Built for domain-specific use and grounded in reliable sources of data relevant to that use.鈥疉 professional-grade solution, such as CoCounsel, harnesses the power of LLMs but limits the source of knowledge to known, reliable data sources 鈥� such as profession-specific domains or professionals鈥� or their clients鈥� databases 鈥� which rigorously limits the possibility of inaccuracies.
2. Built to make verifying its output easy. CoCounsel was not designed to replace the role of the professional, but rather to help them accomplish more and higher-quality work in less time. So just as lawyers review all work delegated to a junior associate or paralegal, they must validate CoCounsel鈥檚 output. We鈥檝e made it easy to do so: all answers link to their origin in the source documents, so it鈥檚 simple to 鈥渢rust, but verify.鈥�听
3. Developed by technical teams with deep GenAI expertise.鈥� Though GenAI has only been broadly talked about since 2022, it鈥檚 been around since 1961. 成人VR视频 AI engineers and research teams have worked with LLMs since their invention, were among the first to build with GPT-4, and have invented patented approaches to applying LLMs to professional use cases.鈥�听
4. Continually and consistently tested and authenticated by a dedicated team of domain experts.鈥� 成人VR视频听AI engineers and Trust Team attorneys together filter, rank, and score CoCounsel鈥檚 responses to a daily battery of thousands of tests developed to simulate real-life legal use cases and ensure the assistant鈥檚 answers are consistent and accurate. To date we鈥檝e run more than 1,000,000 such tests against CoCounsel.鈥�听
5. Secure and private, because it interacts with third-party LLMs the right way. 成人VR视频 GenAI solutions access third-party LLMs through dedicated, private servers, and through an 鈥渆yes off鈥� API. No LLM partner employees can see customer queries or documents, and our LLM access is contractually 鈥渮ero retention.鈥� Our LLM partners cannot store customer data longer than it takes to process the request. Our product data is never used to train any third-party models. And all product data is encrypted in transit and at rest, and subject to 成人VR视频 rigorous security policies and practices. 听

Why 鈥渕akes minimal mistakes鈥� isn鈥檛 enough听

As important as the above five characteristics are, they鈥檝e become price-of-entry criteria for professional-grade GenAI. And given how rapidly the technology is evolving, what you expect from a professional-grade solution should, as well. Remember: GenAI has the power to do so much more than help you complete jobs. It can transform what it means to be a professional, freeing you for more strategic, creative, valuable work that a machine just cannot do 鈥� which can transform not only how you do business, but also how much business you do.听听

To take full advantage of this potential, you need a GenAI assistant that fulfills two more key requirements:听

1. Professional-grade means intelligently and seamlessly handling workflows, not just completing a series of tasks. A true GenAI assistant goes beyond responding to your requests, instead guiding you through the steps required to finish long, complex, even open-ended projects. Only this kind of product, such as CoCounsel, can truly unlock the full potential of GenAI.

Through an expanding set of capabilities and deep connections with both your documents and the tools you use every day 鈥� e.g., Microsoft 365 鈥� a professional-grade GenAI assistant can traffic an entire deliverable from task to task, program to program, in a continuous stream, prompting you forward through the next steps and simultaneously handling multiple pieces of the work itself.

CoCounsel is built for workflows. It鈥檚 accessible across multiple 成人VR视频 products, bringing together both fundamental capabilities such as summarization and document review with specialized functions such as legal research, for a smooth transition from one type of work to the next. And it鈥檚 integrated with both Microsoft 365 and document management systems, available for you literally wherever you鈥檙e working, from client communication to research to document drafting and beyond.

2. Professional-grade means providing partnership, not just product. Without in-depth, sustained support, you鈥檙e unlikely to get the most possible value from your investment. A professional-grade vendor offers a success and support team that will be there for you long after you鈥檝e signed a subscription agreement. They work with you to deeply understand your most prevalent use cases and how their solution will help you tackle them, and of course are available when something鈥檚 not working the way it should. They keep you informed about product changes and improvements and continue creating content you can use to increase your knowledge, such as videos, webinars, and written materials. A truly professional-grade team has a vision for their GenAI assistant, ensuring it will increase in power and capability as the technology advances. And most important, they are as invested in you as you are in them.

Upon adopting CoCounsel, legal professional users are trained by the 成人VR视频 Customer Success team, made up of licensed attorneys, many still practicing, with dozens of years of combined experience in both litigation and transactional law. And many of these trainers are prompting engineer specialists, who in addition to being licensed attorneys have a background in computer science. After onboarding, we ensure everyone using CoCounsel has the opportunity to attend live trainings and watch recorded webinars, get individual help through live chat and email, and access dozens of video tutorials鈥攁 resource pool that will only keep growing.

This is a guest post from Jake Heller, head of CoCounsel, Thomson Reuters, and Erin Nelson, CoCounsel content strategist, 成人VR视频.听听

In the latest , Blanchfield shared how the power of generative AI 鈥� and the dramatic leap it鈥檚 making in how professionals across industries can access large quantities of data 鈥� is transforming the legal industry and beyond. Blanchfield credits the more than 1,500 attorney editors on her team, who create and enhance content, with harnessing the power of generative AI for legal research.

Human expertise is just one component of how 成人VR视频 is capitalizing on the potential of generative AI. Three elements are critical, as David Wong, chief product officer, 成人VR视频, noted in his comments about the launch of CoCounsel 2.0 at ILTACON: 鈥淲e have the data, the expertise, and the tech. Few have all three in such quantity and depth.鈥�

In the new , Wong focused on the role of human domain experts, noting they鈥檙e key to the process of creating and validating data used by AI models for professional research.

鈥淭here’s a lot of both prompt engineering, fine tuning, and system refinement that’s necessary to get quality to a usable spot,鈥� Wong said. 鈥淓xperts, experienced researchers and experienced lawyers can help to gauge whether or not the systems are correct. We couldn’t have an objective, quantified measure of quality on these systems without the editors, without those experts.鈥�

Wong and Blanchfield discussed the importance of human experts in ensuring the accuracy and reliability of AI.

鈥淢aintaining accuracy is at the heart of what the editorial team does,鈥� Blanchfield said. 鈥淚t鈥檚 the number-one priority across every editorial team. We maintain our content to be accurate and trusted.鈥�

Wong acknowledged it鈥檚 challenging for the team to process and update unstructured, constantly changing data in real time. He said that 成人VR视频 ensures that its AI models are customized and meeting the varying needs of various jurisdictions through a combination of software and algorithms that take advantage of the LLMs.

鈥淪o when you ask a question of , for example, we are running an end-to-end algorithm which runs search, retrieves data, re-ranks, interprets and then ultimately passes that information to a large language model to synthesize and produce the answer,鈥� Wong said. 鈥淚t’s a very complicated system which involves multiple types of technology, multiple types of information retrieval.听Processing unstructured, dynamic data and customizing AI models requires integrating multiple technologies and algorithms to optimize performance.鈥�

Hear more of Wong and Blanchfield鈥檚 insights on integrating AI into professional tools and ensuring that information is trustworthy in the of the TechConnect series, which brings diverse and dynamic perspectives from all corners of the technology world with thought-provoking questions and conversation.