Data scientists they are not, but today's lawyers are getting less different from their tech counterparts by the day. The past few years have seen more corporate legal departments leveraging data to evaluate their outside counsel—powered, in no small part, by data management platforms and even artificial intelligence-powered analytics tools.

Of course, such technology is not exclusive to corporate law. Some financial institutions, for instance, use the same core technology to determine who qualifies for loans. Various U.S. courts also rely on it to assess offenders' criminogenic risks, which can factor into sentencing and pretrial release decisions. To be sure, these use cases come with their fair share of controversy, including concerns over algorithmic bias.

But such apprehensions seem to be a world away from the legal market. For all the available technology at its fingertips, the way in-house teams create and use metrics to evaluate outside counsel is still fairly uncomplicated. "I don't see much in the way of algorithms. It's more where there's 10 criteria on performance, and we're going to rate you on a one to five scale," says Kevin Clem, chief commercial officer at HBR Consulting, which helps legal departments evaluate their outside counsel and internal processes.

Indeed, many of the metrics legal departments use to measure outside counsel relies on subjective information gleaned from surveys or largely uncontested quantitative data extracted from spend and matter management tools. But that doesn't mean there aren't problems. Even basic timekeeping metrics, for instance, can be skewed if not created using the same methodology across the board. Despite efforts to the contrary, standardization for the way in-house evaluates their law firms isn't widespread.

Still, few outside counsel express concerns over in-house's metrics, or how they are created. But there's often a glaring reason: sharing these metrics, while possible, isn't common. Either outside counsel are not asking for, or being told about, these evaluations in the first place.

"That feedback loop doesn't exist to say, this is why you're a preferred provider versus why you're not," says Jamal Stockton, a board member at the Corporate Legal Operations Consortium and vice president and head of legal innovation and technology at Fidelity. "But it should—we need to get there, but we're just not there yet."

Setting the Standard

Many of the metrics legal departments use to evaluate its law firms are seemingly straightforward. David Cunningham, chief information officer of Winston & Strawn, says many are just "straight accounting," so there's "not much concern over accuracy." He explains, "The most common are going to be the financial metrics, just to understand potential savings and efficiencies … how they can get value or save money, basically. That's always the core."

In addition, legal departments also commonly look at timekeeping metrics, such as the time it took a firm to close a matter and the breakdown of how long each firm staff ­member spent on a matter. Most, if not all, of these financial and ­timekeeping metrics are calculated automatically by legal technology ­platforms, such as e-billing and matter management tools.

But while there are few accuracy concerns, not all metrics are created the same. Cunningham, for instance, has heard from several legal departments that the time to complete a matter is a pivotal data point. "Well, how do you measure that?" he asks. "Is it from the first day you call me about it, or is it the first time you tell me we're approved? It's not that we think someone is doing it wrong, it's just that we may look at that number differently."

Kevin Clem chief commercial officer with HBR Consulting.

The lack of standardization is even more of a problem when trying to measure something more elusive: quality of work. For some, this metric is vital in allowing meaningful comparisons that go beyond just basic accounting.

Stockton, for instance, notes that by using a standard set of qualitative metrics, two legal departments working on a similar matters but using different firms could compare the "quality of the engagement" of each firm. In practice, however, "We don't do that, but that is the power of scalability and community sharing of information."

Most of the subjective metrics legal departments create for their outside counsel comes from internal surveys and rating systems. For instance, Fidelity's legal department has a proprietary "Amazon-style review tool that utilizes a five star rating system" across certain criteria, including "subject matter expertise, the law firm's responsiveness or timeliness, what people feel the value-to-quality ratio is, communication skills, [and] whether or not they achieved the desired outcome," Stockton says.

Other legal department surveys can differ in their criteria and rating systems. But in recent years, there has been a concerted push to get everyone on the same page. Along with stakeholders across the industry, Stockton was part of a CLOC effort to design a universal framework for qualitatively evaluating outside counsel.

HBR's Clem also participated in the CLOC initiative and says his organization has been a backer of the CLOC standards, which were released in 2017. He adds that he's seen HBR's clients embrace the standards as well. "To my knowledge, ever since the CLOC standard was adopted, every time we help the client with an evaluation, we recommended the standards and they've been willing to use it," Clem says.

Still, CLOC's standards haven't been adopted by the industry at large. Clem chalks this up to the "classic challenge of, are you willing to compromise to benefit the industry, or do you want to tailor it to your own?"

And some of that, to an extent, is to be expected. "Generally speaking, what you're going to find is that every in-house counsel department is going to have different set of criteria based on discussion with their general counsel, staff and attorneys to determine what's important to them," Stockton says.

But the lack of progress on standardization also speaks to something equally as fundamental: qualitative metrics are still fairly new, and legal teams may be too focused on getting them off the ground to think of the industry-wide implications yet. "The legal ops person in the legal department, I believe, has a lot of trouble getting their lawyers to complete the after-matter survey at all," Cunningham says.

If You Build it, Will They Share?

The potentially arduous creation and management of metrics is a painpoint legal technology platforms have sought to address. HBR, for instance, has a survey platform to help legal departments easily collect and organize survey data. The platform connects to HBR's legal analytics solution Counsel Command, which can display these qualitative metrics, alongside more quantitative ones, in dashboards.

Similarly, the LegalVIEW Predictive Insights module from Wolters Kluwer aims to streamline how legal ­departments create and use metrics. When a matter is opened in Wolters Kluwer's legal management platform Passport, an artificial intelligence-powered module that analyzes the matter's characteristics and cross-references it with a list of the law firms the user has worked with in the past. It then provides a list of the top 10 firms for the matter, taking into account both objective metrics like billing and subjective ones like internal peer reviews, which each user weighs ­differently. "We are very transparent with the rating; the user is ­ultimately the one behind the decision," says Vincent ­Venturella, product director at Wolters Kluwer's ELM Solutions.

Jamal Stockton, board member at the Corporate Legal Operations Consortium (CLOC), and vice president and head of legal innovation and technology at Fidelity. .

The tool also offers predictions around firm budgets and cycle times for specific matters by tapping into LegalVIEW's database of customer spend and matter management data. While platforms like these offer their userbase some standardization—in LegalVIEW's case, around how financial and cycle time metrics are calculated—they also allow sharing of metrics between in-house and outside counsel. "All of the data we deliver is fully reportable, we deliver it in such a way that clients can extract and share" with their firms, Venturella explains.

However, there are few signs that much sharing is actually happening. "I think the reality is that law firms never see the information," Cunningham says. He calls the situation akin to restaurants not having access to their customers' reviews on Yelp, and therefore not knowing what they need to change. The metrics in-house gathers "are generally not shared with the law firms or law companies so there is nothing to contest—they just don't get hired anymore," he says.

There are a few likely reasons why sharing isn't widespread. For one, law firms aren't asking clients for metrics or feedback. "In most firms, there is not yet someone who has the role to reach out to the customer and say, what are your metrics and how do we look to you?" Cunningham notes.

But legal departments aren't being proactive either in discussing their metrics with their outside counsel. One director of legal operations for a California-based Fortune 200 company notes that much of reticence to discuss metrics comes down to in-house not wanting to offend its law firm colleagues. "I think there is an old fashioned stigma where the relationship kind of got in the way, because it was somehow awkward or inappropriate to bring up money" or even other performance notes such as "how many people showed up at a meeting or at a conference call." However, the legal ops director adds that communication is improving these days, partly because the use of metrics is growing. "I think the some of the former barriers are breaking down."

Still, there is a long way to go. But like CLOC, others aren't sitting idly by. Cunningham, for instance, says he "is working with legal departments and law firms to develop a legal ops dashboard that has shareable [metrics] information," Cunningham says. The effort, which aims to get off the ground in early 2020 and exist more as guidance than overall standards, looks to spark agreement on defining key metrics and how to calculate them.

The project, however, is a key example of the problem facing legal metrics: the technology is there, but the communication is not. "I know there are certainly law firms that have legal ops positions now. I hope those people are going proactively to clients and talking about these metrics," Cunningham says. "But the people I have talked to are not yet doing that. It's very early."