These conversations are more important than ever as generative AI becomes more widely implemented across the legal industry. As our customers鈥� 鈥� and their clients鈥� 鈥� expectations change, it鈥檚 crucial that we鈥檙e closely aligned with their collective needs.听听

With this mind, 成人VR视频 and Lexpert hosted a featuring law firm leaders and industry experts discussing the challenges and trends around the use of generative AI in the legal profession.鈥疊elow are insights from an engaging and informative discussion.听

Lawyers are excited to implement generative AI solutions听

Law firms may not have a reputation as early adopters of technology, yet many firms are already using generative AI solutions. David Cohen, senior director, client service delivery, McCarthy T茅trault, noted his firm鈥檚 response when it rolled out CoCounsel, the professional-grade GenAI assistant, a year ago. 鈥�听

鈥�鈥淲e put a call out for people to sign up and get a license, and the 175 seats were immediately filled,鈥� Cohen said. 鈥淭here was a wait list where people were pleading to get access to the tool 鈥� there鈥檚 certainly a lot of interest and excitement at our firm.鈥濃��听

Panelists noted how firms are increasingly implementing AI for drafting legal documents as well as for predictive analytics and other complex tasks.听

Cohen said McCarthy T茅trault is using AI for higher-value work, including in litigation matters to scan data troves and identify where the firm has been successful in the past. 鈥�听

鈥�鈥淲e鈥檙e very interested in how we get a strategic advantage,鈥� Cohen said. 鈥淲hat arguments were used, what makes for a successful pleading 鈥� that鈥檚 something we can leverage.鈥� 鈥�听

鈥�Rikin Morzaria, principal at Kinara Law, said he鈥檚 鈥渆xperimenting with and finding solutions that work long term,鈥� including using AI for advanced tasks such as drafting memos or reviewing his own work to identify potential areas for improvement.听

鈥�鈥淚 treat it as something that would be submitted by a student or an intern, that needs to be reviewed again,鈥� Morzaria said. 鈥淚t ultimately produces a better product for me in the end.鈥�听

鈥�Unfounded concerns about robot lawyers听

Panelists shared their perspectives on generative AI solutions augmenting, not replacing, humans.听

鈥淎I doesn鈥檛 have human insight or instincts, it doesn鈥檛 know the client, it 肠补苍鈥檛 read a room,鈥� said Valerie McConnell, senior director of CoCounsel Customer Success at 成人VR视频. 鈥淗ow lawyers allocate their time will shift: the hope and expectation is when we clear their plates, they鈥檒l have the space to think about strategic angles.鈥�听

鈥�This brought up how firms鈥� use of AI is changing talent management. Cohen noted that McCarthy T茅trault revamped its training program to incorporate the integration of AI into its workflows and processes.听

鈥�鈥淗aving best-in-class prompts that are shared and available to our lawyers is something we certainly weren鈥檛 discussing a couple years ago,鈥� Cohen said. 鈥�听

鈥�Changing billing practices and elevating services听

Panelists noted that generative AI is changing how lawyers bill their clients. They discussed an example of litigators relying more on flat-fee billing because they know they can handle certain aspects of a matter more quickly. 鈥�听

鈥�鈥淵our time in a discovery, trial, or mediation won鈥檛 change, but a lot of that time leading up to it will,鈥� Morzaria said. 鈥淭hat allows you to work more efficiently and take on work that may not have been feasible before.鈥� 鈥�听

Panelists also shared how generative AI is helping firms elevate their services. They called out how firms in the litigation space are using it to surface key鈥痚vidence, while firms on the transactional side are using it for contract review.听听

Managing and mitigating risks听

Panelists discussed how firms are addressing concerns around emerging technologies including accuracy and privacy. Cohen noted firms are focusing on 鈥渁chieving the right balance where we鈥檙e generating benefits for our clients but also managing and mitigating the risks.鈥澨�听

McConnell emphasized that AI is not a replacement for a lawyer鈥檚 oversight. She recommended that firms create an AI policy, use a secure software platform that allows lawyers to easily verify output and doesn鈥檛 train on firm data, and provide employee training on responsible AI use. 鈥�听

Generative AI is shaping the future of law, and its impact on the legal profession will be even greater in 2025 and beyond. As the market matures, legal professionals will not just desire but require AI capabilities for their workflows.鈥疘nnovative firms at the forefront, including Kinara Law and McCarthy T茅trault, are best positioned to capitalize on the potential of this transformative technology to better serve their clients.听

鈥�Check out highlights of the panelists鈥� conversation and listen to their full discussion .听

This is a guest post from Raghu Ramanathan, president, Legal Professionals, 成人VR视频.听

The generative AI-driven solution enables UK legal professionals to find the best starting point from their own databases or Practical Law templates, use content from Practical Law alongside their legal department鈥檚 or firm鈥檚 contract repository to draft or revise clauses, leverage Practical Law contract playbooks, and correct common drafting errors. It uses AI to refine and review documents, producing more accurate, higher-quality work 鈥� without leaving Microsoft Word.

I recently shared a CoCounsel Drafting demo with 鈥檚 Richard Tromans.

鈥淭he offering has been especially shaped for legal needs here and will directly engage with TR鈥檚 contract data in Practical Law,鈥� Tromans said in .

I highlighted CoCounsel Drafting capabilities and features for Tromans, and we discussed how 鈥淟LM-supported drafting could be truly transformative.鈥� We talked about how users will be able to use the large language model (LLM) to modify a clause, for example, and how CoCounsel Drafting uses generative AI grounded in retrieval augmented generation (RAG) to access听Practical Law content and notes.

We also discussed how CoCounsel Drafting delivers a level of detail similar to what a junior associate may produce when asked to draft a contract. I noted that CoCounsel Drafting allows customers to use their own playbook to draft a preferred set of terms for any contract type or use a Practical Law playbook for a range of popular contract types.

Finally, Tromans and I discussed integrations 鈥� with HighQ and with document management systems (DMS) including iManage 鈥� as well as security.

On the other side of the pond, U.S. customer feedback on CoCounsel Drafting, which we introduced earlier this year, has been overwhelmingly positive. I hope that UK legal professionals are equally pleased with the time savings and productivity increases CoCounsel Drafting offers.

As a former lawyer turned AI product leader, I’ve lived the pain of tedious legal tasks firsthand. That’s why I’m passionate about harnessing generative AI to transform legal drafting. I know how much time and energy gets wasted on repetitive tasks, taking away from the work that really matters. Now, I’m excited to help lawyers reclaim that time and focus on what they love 鈥� whether that’s helping clients, developing strategy, or simply enjoying a better work-life balance.

You can learn more about CoCounsel Drafting , and check out Tromans鈥� blog .

This is a guest post from Rawia Ashraf, vice president of Product Management, 成人VR视频.

1. Lawyer productivity gains are boosted by technology

In Q2 2024, for the first time in years, a majority of law firms experienced productivity growth. In Q3, a whopping 64% of law firms reported productivity growth, building on the gains seen in Q2.

鈥淭he continued climb in law firm average productivity 鈥� in stark contrast to previous years 鈥� is a key factor boosting law firm profitability,鈥� said Raghu Ramanathan, president of Legal Professionals, 成人VR视频. “Law firms that not only invest in new technology but also adopt AI and generative AI solutions to streamline workflows and improve the efficiency and quality of their work will be best positioned to improve client satisfaction and experience sustained productivity growth.鈥�

2. Demand continues to rise

Overall demand for legal services is up by a solid 3.6% compared to the same quarter last year. Leading the charge are counter-cyclical practices, with litigation demand up by 4.0% and labor & employment not far behind at 2.9%. Real estate led the growth in transactional practices, up 3.7%. The balanced growth across practice areas indicates a more sustainable expansion than what law firms experienced during the post-pandemic boom.

3. Both full-time equivalent headcount and productivity are up

Q3 2024 was the first time in three years that law firms increased both their full-time equivalent headcount and productivity. While there鈥檚 been a slight uptick in expenses 鈥� 5.7% for direct and 5.3% for overhead 鈥� the modest hiring rate compared to the boom years of 2021 and 2022 is helping firms keep these costs in check.

4. Stable economy provides ideal growth environment

The strong performance for law firms was supported by a stable and healthier-than-expected U.S. economy. Despite earlier concerns about sustained inflation and slow economic growth, the economic environment has provided a foundation for law firms to thrive. Law firms are positioned for a strong close to 2024.

Download the full report for additional insights on the factors shaping the future of law firm profitability and productivity.

Why evaluating legal skills is complicated听

Traditional legal benchmarks usually rely on multiple-choice, true/false, or short-answer formats for easy evaluation. But these methods aren鈥檛 enough to assess the complex, open-ended tasks lawyers encounter daily and that large language model (LLM)-powered solutions like CoCounsel are built to perform.听

CoCounsel鈥檚 skills produce nuanced outputs that must meet multiple criteria, including factual accuracy, adherence to source documents, and logical consistency. These are difficult outputs to evaluate using true/false tests. On top of that, assessing the 鈥渃orrectness鈥� of legal outputs can be subjective. For instance, some users prefer detailed summaries, others prefer concise ones. Neither is 鈥渨rong,鈥� it just comes down to preference, which makes it difficult to consistently automate evaluations.听

To make it even more complicated, each CoCounsel skill often involves multiple components, with the LLM handling only the final stage of answer generation. For example, the Search a Database skill first uses various non-LLM-based search systems to retrieve relevant documents before the LLM synthesizes an answer. If the initial retrieval process is substandard, the LLM’s performance will be compromised. So, our evaluation must consider both LLM-based and non-LLM-based aspects, to make sure our assessment of the whole is accurate.听

How we benchmark听

Our benchmarking process begins long before putting CoCounsel through its paces. Whenever a significant new LLM is released, we test it across a wide suite of public and private legal tests, such as the dataset created by our Stanford collaborators, to assess their aptitude for legal review and analysis. We then integrate the LLMs that perform well in these initial tests with the CoCounsel platform, in a staging environment, to evaluate how they perform under real-world conditions.听

Then we use an automated platform to run a battery of test cases created by our Trust Team (more on this below), to evaluate the output that comes from this experimental integration. If the results are promising, we conduct additional manual reviews using a skilled team of attorneys. When we see an improvement in performance compared to previous benchmarks, then we start talking as a team about how it might improve the CoCounsel experience for our users.听

How we test听

Our Trust Team has been around as long as CoCounsel has.听 This group of experienced attorneys from diverse backgrounds 鈥� in-house counsel, large and small law firms, government, public policy 鈥� is dedicated to continually rigorously testing CoCounsel performance. 听

We continue to follow a process that鈥檚 been integral to all our performance evaluation since CoCounsel鈥檚 inception: Our Trust Team creates tests representative of the real work attorneys use CoCounsel for and runs these tests against CoCounsel skills. When creating a test, they first consider what the skill鈥檚 for and how it might be used, based on their own insights, customer feedback, and secondary sources. Once the test is created, the attorney tester manually completes the test task, just as a lawyer would, to create an answer key 鈥� what we refer to as an 鈥渋deal response.鈥� These tests and their corresponding ideal responses then undergo peer review. Being this meticulous is crucial, because the quality of our ideal responses determines the benchmark for a passing score.鈥��听

Once the ideal response has been created, a member of the Trust Team runs the test, using the applicable CoCounsel skill to complete the task just as a user would. An attorney tester reviews the output, referred to as our 鈥渕odel response.鈥� Then they point-by-point compare CoCounsel鈥檚 response to the ideal response, identifying differences and assessing whether these differences deviate from the ideal response in a way or to a degree that would make the skill’s output incomplete, incorrect, or misleading. It is important to note that a test can be failed for many reasons, not just because an answer is outright wrong. For instance, some skills (e.g., Summarize) are more inherently subjective than others and do not lend themselves to a right or wrong answer. Accordingly, even though attorneys鈥� opinions of the 鈥渃orrect鈥� answer can reasonably vary, we try to maintain the highest grading standards and 鈥渇ail鈥� the test if it is missing a single detail the attorney-tester considers to be Critical.听听

Based on this manual evaluation, we create a set of evaluation instructions tailored to the ideal output unique to each skill, capturing the needs of each skill so an LLM can鈥痬atch our human reviewers鈥� process. To illustrate this, let’s look at a real example using CoCounsel鈥檚 Extract Contract Data skill. After running the test in our automated evaluation platform, here鈥檚 what an attorney reviewer would see when reconciling the model response with the LLM鈥檚 evaluation of it:鈥�听

• Question: If the customer is acquired through a merger, can the agreement be assigned to the acquiring entity without obtaining the provider’s prior written consent?鈥�听

• • Trust Team ideal response:听

• • Short answer: Yes, with 30 days鈥� prior written notice. 鈥�听

• • Detailed answer: Yes, if the customer is acquired through a merger, the agreement can be assigned to the acquiring entity without obtaining the provider’s prior written consent; the customer is only required to provide the provider with 30 days鈥� prior written notice. The clause specifies that either party may assign its rights or delegate its obligations without such consent, provided that 30 days鈥� prior written notice is given to the other party, to an entity that acquires all or substantially all of the business or assets of such party, including by merger.鈥�听

• Model response:鈥�听

• • 鈥疭hort answer: Yes, with notice.鈥�听

• • Detailed answer: The agreement can be assigned to the acquiring entity without obtaining the provider’s prior written consent if the customer is acquired through a merger. The clause provides an exception to the general rule requiring consent for assignment. Specifically, it states that either party may assign its rights or delegate its obligations without such consent to an entity that acquires all or substantially all of the business or assets of the party, including through a merger, provided that 30 days鈥� prior written notice is given to the other party.鈥�听

• • Automated evaluation analysis: The model response captures the correct information that the agreement can be assigned to the acquiring entity without obtaining the provider’s prior written consent but requires 30 days鈥� prior written notice. However, it incorrectly states the short answer as 鈥淵es, with notice鈥� instead of 鈥淵es, with 30 days鈥� prior written notice,鈥� as per the ideal answer. This is a minor difference and does not change the meaning significantly.鈥�听

In this instance, the model response included a minor discrepancy from the attorney-authored ideal response. But the LLM鈥檚 evaluation of the response accurately determined that the answer remained sufficient, because it captured the complete notice requirement elsewhere in the response.听听

Our ideal-response approach provides two key advantages over assertion-based evaluations. It excels at identifying deviations from attorney expectations, including hallucinations. And it pinpoints extraneous or inconsistent information that, while not technically a hallucination, could make even a complete response incorrect if that information introduces logical inconsistencies, which would result in a failing score.鈥�
鈥�
We rely on our Trust Team to create well-defined ideal responses and auto-evaluation instructions and to determine if a test case passes or fails. A skill鈥檚 output definitively fails if it falls short of this ideal because of material omissions, factual incorrectness, or hallucinations. However, we recognize that many legal issues aren鈥檛 black-and-white, and the 鈥渃orrect鈥� answer could be open to reasonable disagreement. To address this, we peer review ideal responses in cases when the answer might require a second opinion. And we might eliminate tests when we find insufficient agreement among the attorney testers. This is how we both ensure that our passing criteria remain rigorous and account for the nuanced nature of legal analysis.鈥�

Maintenance and improvement听

Creating a skill test set is only the beginning. Once we begin using it, the Trust Team continually monitors and refines it by manually reviewing failure cases from the automated tests and spot-checking passing samples to make sure the automated evaluation is in line with human judgments. We also regularly add tests to cover more use cases and capture user-reported issues, which could lead to further iterations of the tests submitted for automated evaluation and their success criteria. 鈥�听

By following this process, every night we can execute, across all CoCounsel skills, more than 1,500 tests on our automated platform under attorney oversight, which combined with manual testing means we鈥檝e run more than 1,000,000 tests since CoCounsel鈥檚 launch. And it empowers us to quickly identify areas for improvement, which is vital to ensuring CoCounsel remains the most trustworthy AI legal assistant available.听听

Conclusion听

, we explored what it means for an AI tool to be “professional-grade” and why that standard is crucial for professionals in high-stakes fields like law. This post takes that concept further by diving into how we benchmark CoCounsel to ensure it meets those rigorous standards. By understanding the extensive testing that goes into evaluating its performance, you can see how CoCounsel consistently delivers the reliability and accuracy expected of a true professional-grade GenAI solution.听

To promote the transparency my team and I believe is necessary in the legal AI field, we鈥檝e decided to release some of our performance statistics for the first time and a sample of the tests that are used to arrive at the figures below applying the criteria referenced within this article. Check out our results .

This听is a guest听post from听Jake Heller,听head听of听CoCounsel,听Thomson听Reuters.

Below, CJ Lechtenberg, senior director, Westlaw Product Management, 成人VR视频, shares her insights on developing Mischaracterization Identification, a generative AI capability to help detect mischaracterizations and omissions in legal briefs.

In the five years since Quick Check was introduced, you鈥檝e added many enhancements including Quick Check Contrary Authority Identification, Quick Check Judicial and Quick Check Quotation Analysis. How did integrating generative AI make the Mischaracterization Identification enhancement different than previous ones?

Lechtenberg: This enhancement takes researchers beyond the step of knowing what might be a potential mischaracterization to an explanation of why something might be a potential mischaracterization 鈥� and that is radically different from any feature we鈥檝e deployed in Quick Check before.

I鈥檓 sure it鈥檒l come as no surprise when I say that generative AI is just a completely different beast. Lay people may think about the law as being black and white.听You can do this; you 肠补苍鈥檛 do that.听But legal professionals know that the law is really a sea of varying shades of gray. With machine learning, we wrestled with how we could ever give the machine enough data to figure out all the different ways an attorney may mischaracterize the law.

In Quick Check Quotation Analysis prior to the Mischaracterization Identification enhancement, we highlighted the actual textual differences 鈥� additions, omissions, and changes 鈥� in the quotations and showed the context around the quotes.听Doing so certainly saved researchers a significant amount of time and helped them spot issues they might not otherwise find, but the onus was still on researchers to review everything and determine what the precise differences were and how material they might be, if at all.听Even with the additional context provided, it could still be difficult to determine whether the quotations were taken out of context, especially if the quotes themselves didn鈥檛 appear to be different.

In developing Mischaracterization Identification, we recognized that the task of analyzing quotations and their context is so nuanced that attorneys will have different expectations for whether a mischaracterization occurred, so we needed to provide more than just categorizations. We found that large language models (LLMs) can generate nuanced descriptions of potential mischaracterizations, versus just explicit categorizations, and do it well, which is hugely beneficial for this type of task.

How will using Mischaracterization Identification give legal professionals and law firms a competitive advantage? How will judges using it benefit?听

Lechtenberg: The advantages of using the new Mischaracterization Identification are substantial for both legal professionals and the judiciary 鈥� both in terms of speed of review and quality of work product.听When we launched Quick Check Quotation Analysis in 2020, customers, both legal professionals and the judiciary, lamented about how time-consuming it is for them to review quotations and how challenging it is to spot differences. It is a mentally taxing task and often our brains fill in the blanks 鈥� interpreting what we think a brief maybe should say but actually doesn鈥檛.听 Attorneys never have a surplus of time, so the last thing they want to do is spend the little bit they have on the most tedious of tasks and still end up missing potential problems.

For attorneys, Mischaracterization Identification will help them efficiently and accurately make contextual misstatement and omission determinations for their opponents鈥� and their own quotations and the context surrounding those quotations. The fear of missing their own mistakes is very real for attorneys, but the possibility of missing the opportunity to capitalize on their opponents鈥� mistakes is an even larger concern. This new enhancement reduces both of those worries and will help attorneys be even better advocates for their clients.

Judges will also be able to effectively review the filings of parties in matters before them much faster. Attorneys owe a duty of candor to the judiciary and the Mischaracterization Identification feature will help flag any potential issues quickly. An added benefit, which members of the judiciary or their staff perhaps haven鈥檛 considered, is the ability to analyze their own orders and opinions to ensure that they haven鈥檛 made mistakes that could be appealed. This new enhancement will help alert judges and law clerks to potential issues before they finalize their opinions.

What early feedback are you hearing from customers?

Lechtenberg: In a recent survey, 93% of law firm professionals told us they鈥檝e seen opposing counsel misuse a quotation, 66% said they鈥檝e seen misrepresentations by an associate or colleague, and 65% of corporate respondents said they check the accuracy of outside counsel鈥檚 quotations.听The need to review opposing counsels鈥� and colleagues鈥� briefs for mischaracterizations of the law is still a very real issue for attorneys. Likewise, attorneys have said they鈥檙e always concerned about the accuracy of their work and that maintaining their reputation as a credible litigator with courts and opposing counsel is incredibly important.

Customers are extremely excited about this new Quick Check enhancement to help combat these concerns and we鈥檝e received positive feedback from them.听One law firm managing partner stated that they would use this tool a lot.听They cite-check their opponents鈥� briefs, so any shortcuts are beneficial to them. They recognize that most of the time, errors are harmless, but occasionally there are things they want to bring to the court鈥檚 attention and this feature will help them spot those issues more quickly and accurately.

Another law firm partner said this new feature is the 鈥渦ltimate security blanket鈥� because everything attorneys do is based on their credibility, and this feature alerting them to quotes being taken out of context before filing with the court would calm some of those fears.

Any surprising or unexpected moments as the team worked on developing or launching Mischaracterization Identification?

Lechtenberg: The fact that we鈥檝e accomplished this now with the use of LLMs is exciting, a little surprising and a long time coming. I鈥檓 an attorney who leads a team of attorneys; we鈥檙e literally trained to question everything and have a healthy dose of skepticism.听But I have been dreaming about a mischaracterization identification feature in Quick Check ever since we developed Quotation Analysis more than five years ago. At my core, I believed someday this could be achieved, but for years traditional machine learning approaches were just not powerful or nuanced enough to do it well.

Leveraging LLMs for a use case like this is a new frontier like we鈥檝e never seen before.听The LLM鈥檚 ability to analyze text from an uploaded document and compare that text to the text from the cited case used to support the argument and then go beyond highlighting textual differences and provide an actual explanation of what may be problematic 鈥� whether that鈥檚 a selective quote, omitted context or a misinterpreted holding 鈥� has been absolutely astounding.

What鈥檚 the one thing you want everyone to know about Mischaracterization Identification?

Lechtenberg: Mischaracterization Identification will not only help researchers spot contextual misstatements and omissions in their opponents鈥� or their own quotations and contextual statements faster and with more accuracy, but most importantly it will help them understand why those misstatements or omissions may be problematic. And, spoiler alert: Mischaracterization Identification is just the beginning of how 成人VR视频 will harness the power of generative AI in Quick Check to solve important customer problems.

For more on Mischaracterization Identification, read the press release or check out the by Mike Dahn, head of Westlaw Product Management, 成人VR视频.

A shift in the legal industry’s approach to technology

What stood out to me at ILTACON was the momentum behind the adoption of generative AI across the legal industry. It was striking to see how many law firms, regardless of size, have already integrated generative AI into their practices.

My personal highlight at ILTACON was participating in a panel discussion on the application of generative AI in legal practice. We received thoughtful and insightful questions from the audience, reflecting a deep interest in maximizing the potential of generative AI.

During the panel, we conducted a quick audience survey, and nearly every attendee indicated that their firm was already using generative AI. This widespread adoption reflects a significant shift in the legal industry’s approach to technology, signaling that generative AI is already playing a pivotal role in shaping legal practice.

The primary focus of interest in my panel was on the practicalities of implementing generative AI solutions within law firms. Attendees seemed particularly interested to learn about best practices for conducting a successful proof-of-concept, including strategies for testing the accuracy and reliability of generative AI software.

Attendees also seemed interested in understanding how to effectively measure ROI from adopting generative AI. The focus was not on whether law firms should adopt generative AI, but rather how to adopt this technology and derive the most value.

From skepticism to enthusiasm

One of the most surprising and encouraging moments at ILTACON was seeing the shift in attitudes toward generative AI. Where there was once skepticism and anxiety, I now see a growing acceptance and even enthusiasm.

The concerns that previously dominated discussions 鈥� like the potential impact on the billable hour 鈥� seem to have softened. Instead, there’s a growing recognition that generative AI is not just a disruptive force but a transformative tool that can enhance legal practice.

Many attendees expressed optimism that generative AI will ultimately expand the reach of legal services, enabling more people to engage with the legal system, pursue legal claims, and explore innovative legal strategies.

This is a guest post from Valerie McConnell, senior director, CoCounsel, 成人VR视频. Check out other 成人VR视频 highlights from ILTACON, including the launch of CoCounsel 2.0.

They offered advice on evaluating a vendor鈥檚 technology expertise, support services, and transparency. Also, they shared how firms and organizations can mitigate the risks posed by acquiring a vendor鈥檚 AI services while maximizing their investment in AI. Below are highlights from their conversation.

Keys to choosing AI vendors

To start the process of assessing potential AI vendors, Yaghoubi emphasized the importance of reviewing their experience and expertise 鈥渢o allow your business to make informed decisions about whether to engage the vendor.鈥�

Wong said that in addition to performance and cost, firms should consider safety and trust factors.

Ramanathan noted it鈥檚 important to consider whether you want a consumer- grade model 鈥� that鈥檚 cheaper 鈥� or a more reliable professional-grade model. He emphasized three criteria to focus on when choosing an AI vendor:

1. 鈥淲hat鈥檚 your philosophy and principles around how AI should be used?鈥� He said asking a vendor this question allows you to see if your firm鈥檚 vision and long-term strategy and roadmap are aligned with the vendor鈥檚 approach.
2. Request a vendor鈥檚 references and testimonials. Ramanathan explained that firms and organizations should ask vendors how many customers are already using their solutions. 鈥淎I is still a game of scale,鈥� he said. 鈥淵ou don鈥檛 want to be the first customer training a model.鈥�
3. Clarify the level of support and training a vendor provides. Ramanathan said this is key to ensuring that all levels of staff are trained and can use the AI solutions constructively.

Wong added that the questions he receives from potential clients focus on data, technology, and talent. He warned that some companies simply repackaged existing large language models (LLMs) for legal use cases without adding much.

鈥淐lients that are working with companies that are building AI have a say,鈥� Wong said. 鈥淭hey can contribute, iterate, and build the products.鈥�

Also, he stressed the importance of working with a vendor that knows how to customize solutions and integrate customer feedback into product development.

How vendors use data

Kapoor asked what customers should consider regarding how vendors use their data. Wong said that understanding the data flow and how the data is processed are key, as well as understanding licensing and data rights, including intellectual property usage rights, cyber risk, and data leakage.

Ramanathan noted encryption standards as well as access control are critical as is demanding transparency from vendors: 鈥淵ou have the right to ask how the data your inputting is used.鈥�

Ramanathan added, 鈥淕ood vendors should have governance systems that answer鈥� details such as where data is stored and who has access to it.

鈥淟ook for transparency鈥� on data output

Wong advised firms to 鈥渓ook for transparency鈥� from vendors, making sure they provide qualitative and quantitative information about the quality of their outputs. He said vendors should be guided by a set of AI principles and should follow a data governance and AI model governance process to mitigate hallucinations and potential risks.

Ramanathan noted that good vendors conduct regular model validation on a periodic basis. He also flagged that professional-grade AI solutions 鈥� unlike consumer-grade AI solutions 鈥� give a sense for the reliability of the answer.

Data output considerations also include encryption standards as well as vendors鈥� privacy and security policies. Ramanathan said a baseline is compliance with standards such as GDPR and CCPA.

鈥淭he privacy and security measures a vendor takes are a result of their philosophy about AI and how to use AI,鈥� Ramanathan said. 鈥淚t gives you a clue as to what you can expect downstream in terms of execution.鈥�

Ramanathan added that vendors should share their risk management framework and enterprise risk framework as well as disclose how frequently they conduct audits and what mitigating actions they put in place.

Wong added that most firms and organizations have 鈥渢ried and tested approaches for technology procurement鈥� that they should apply to assessing AI vendors too.

Lack of AI-Specific SLAs

When exploring initial and ongoing training and documentation, Shokoh asked if AI service-level agreements (SLAs) are similar to those offered for SaaS-type platforms.

Ramanathan said there are elements of SLAs similar to cloud software 鈥渢hat you can and should expect,鈥� such as uptime and maintenance. He noted the hard part is the lack of industry standards for AI-specific SLAs to address issues like response time and accuracy.

In the absence of industry standards, Ramanathan recommended asking questions around issues like product reliability controls and internal testing programs.

Going above the legal requirements

Part of assessing an AI vendor involves anticipating it will adapt to new and changing AI regulations, given the lack of a comprehensive federal law in United States and various states implementing their own guidance.

鈥淭here鈥檚 wide range and little consistency across the market,鈥� Wong said. 鈥淲hat 成人VR视频 has done is look at AI standards in all the markets that we operate in and identify the most restrictive standards. We use a combination of the NIST standards and the EU AI directive as the basis for much of our governance framework.鈥�

Wong added that 成人VR视频 applies this viewpoint to its risk management framework and to its data and AI model governance framework.

鈥淲e projected what the regulation would be rather than look at where the regulation is today,鈥� Ramanathan explained. 鈥淲e proactively defined what we call our Data and AI Ethics principles, which are very hard-coded guidelines that go into engineering our products as well.鈥�

To watch a recording of the webinar, .