The 2022 ILTA Tech Survey found that nearly two-thirds of law firm IT departments reported using no metrics to measure their success. But there are many ways to start collecting that data
The data explosion within law firms has seen the transformation of both external functions like litigation and contracting, as well as internal functions like financial and document management. Technology-enabled services have let legal teams not only better serve their clients, but collect metrics to determine what鈥檚 working and what鈥檚 not as well.
However, there remains one area within law firms that has seemed more reluctant to embrace metrics to measure department success, and it鈥檚 perhaps a surprising one: IT departments themselves.
The 鈥檚 (ILTA) 2022 Tech Survey revealed that 63% of surveyed law firm IT departments reported using 鈥渘one/non-applicable鈥� metrics to measure IT department performance. That figure was lower, but only slightly, from 2021鈥檚 67%. The most popular available metrics used were downtime per month (37% of surveyed firms), Help Desk closed ticket feedback (36%), and Help Desk ticket resolution times (32%).
The ILTA report did note that 鈥淸t]he larger the firm, the more likely they are to actually poll users on satisfaction with regard to the performance of the IT department.鈥� Regardless of size, however, the fact that nearly two-thirds of firms weren鈥檛 using any metrics to measure IT department performance doesn鈥檛 surprise Kenneth Jones, chief technologist at law firm Tanenbaum Keale and ILTA committee member.
Jones notes that in the corporate world, technology projects tend to be tied directly to return-on-investment, with collecting metrics baked directly into employment. At his previous work at pharmaceutical giant Bristol Myers-Squibb, this attitude was prevalent, Jones says. 鈥淥ne of the measurements was, try not to put 40% of your time into 1% of your business,鈥� he adds. 鈥淐onversely, in the legal field, I think sometimes鈥� whoever screams the loudest tends to get the most attention, and it may not be the project that’s delivering the most revenue or profit to a firm.鈥�
Indeed, in the ILTA survey, less than 30% of surveyed firms reported measuring law firm success via IT capital costs per lawyer (29%), IT operating costs per capital lawyer (25%), or IT spending as a percentage of revenue (22%). Jones views this as a missed opportunity. The goal of any IT department, he says, should be to make sure as many dollars as possible are actively being utilized.
鈥淗ow many of us have Hulu accounts or Netflix accounts, or this subscription or that subscription, and we don’t use it anymore?鈥� he asks. 鈥淎nd so, that’s probably a useful metric also, just to be sure that as many dollars as possible are truly being allocated productively to business.鈥�
For those who haven鈥檛 built out metrics programs for the IT department, Jones says that 鈥渁 good first step is having the data collection and measurement in place.鈥� He separates the data needed for this type of program into two buckets: i) what technologies are actually being used and how effectively; and ii) where spend is going in an organization. The latter, he adds, can be tricky 鈥� spend shouldn鈥檛 just include official IT department projects, but also what individual attorneys are purchasing for themselves to augment their daily work (such as subscription licenses, discovery costs, technology consultants, and more).
Building out the metrics program
Consultant and legal survey expert Rees Morrison says that to actually begin collecting IT department data, a firm鈥檚 options can be broken down into four different buckets. The first is rather simple: focus groups. 鈥淚t’s a way of not only collecting some numbers, but also subjective views and opinions,鈥� Morrison says. 鈥淎nd so a focus group, it’s a flexible tool, but it’s also somewhat expensive and you have to have people who know how to run them.鈥�
Morrison鈥檚 second choice is perhaps the way most firms operate today: short email surveys, which he called a 鈥渃rude but simple way of collecting some data.鈥� The most effective email surveys are short 鈥� at about two or three questions 鈥� in order to encourage a higher response rate. However, he adds, these are tough to do en masse, as IT department leaders would also then be responsible for collecting, sorting, and displaying the data.
His third option is one that is perhaps more underutilized for law firm IT departments: formalized surveys. In fact, the ILTA survey found that just 16% of firms utilized user-satisfaction surveys to measure success, and 20% incorporated user feedback in general. While surveys may require more work on the front end, they also reveal greater insights on the back end, Morrison explains. 鈥淵ou can ask more questions. It’s easier for people to answer, because a lot of them, it’s 鈥榗heck a box鈥� or 鈥榝ill in a number鈥� as compared to an email, where it鈥檚 a little less formatted,鈥� he says. 鈥淎nd the huge advantage is, it then brings everything together, often in a spreadsheet so you can analyze it how you want.鈥�
Finally, IT departments can utilize what Morrison calls exhaust data: logs of how often a product is being used. This type of data can provide insights that the other types can鈥檛, he says, because it鈥檚 agnostic of attorney or legal professional biases. 鈥淭hey’re not thinking probably about how much time they’re online doing that or how many times they do it,鈥� he adds. 鈥淏ut somebody else can start drawing some patterns or making decisions whether a subscription is worthwhile or not, or get more of the sorts of ancillary data that flows from it.鈥�
Morrison notes that collecting these sorts of metrics can have multiple positive effects. There is, of course, the increased insight into firm technology usage and better ROI that comes from increased efficiency. but there is also positive firm engagement that can come from IT metrics projects, he says.
鈥淚 do believe that folks in general like their opinion being asked,鈥� he explains. 鈥淥therwise they feel they’re just serfs and nobody cares 鈥� I don’t happen to like this contract management system, but what can I say? But if they come and ask you on a scale of 1-to-10, how do you like it and a few questions related to that, I get a voice.鈥�
The work does not stop here, however. Law firm IT metrics are not a 鈥渟et it and forget it鈥� proposition. Law firms looking to collect this data also need to plan how they鈥檙e going to incorporate feedback around the data they鈥檝e collected 鈥� and make sure those they鈥檝e collected data from are heard, Morrison notes.
鈥淚f you gather metrics from people, I think you owe it to get back to them something,鈥� he says. 鈥淚t shouldn’t go into the black box of the firm and the associates never find out what happened. You give me some time and your thoughts and estimates and numbers, and I’m going to return with some findings and describe what we’re going to do about it.鈥�
