Data scientists they are not, but today's lawyers are growing more similar to their tech counterparts by the day. The past few years have seen more corporate legal departments leveraging data to evaluate their outside counsel—powered, in no small part, by data management platforms and even artificial intelligence-powered analytics tools.

Of course, such technology is not exclusive to corporate law. Some financial institutions, for instance, use the same core technology to determine who qualifies for loans. Various U.S. courts also rely on it to assess offenders' criminogenic risks, which can factor into sentencing and pretrial release decisions. To be sure, these use cases come with their fair share of controversy, including concerns over algorithmic bias.

But such apprehensions seem to be a world away from the legal market. For all the available technology at their fingertips, the way in-house teams create and use metrics to evaluate outside counsel is still fairly uncomplicated. "I don't see much in the way of algorithms. It's more where there's 10 criteria on performance, and we're going to rate you on a one to five scale," says Kevin Clem, chief commercial officer at HBR Consulting, which helps legal departments evaluate their outside counsel and internal processes.

Indeed, many of the metrics legal departments use to measure outside counsel rely on subjective information gleaned from surveys, or largely uncontested quantitative data ­extracted from spend and matter management tools. But that doesn't mean there aren't problems. Even basic timekeeping metrics, for instance, can be skewed if not created using the same methodology across the board. Despite efforts to the contrary, there is little standardization of the way in-house teams evaluate their law firms.

Few outside counsel express concerns over in-house metrics or how they are created. But there's often a glaring reason: sharing these metrics isn't common. Outside counsel either are not asking for or are not being told about these evaluations in the first place.

"That feedback loop doesn't exist to say, This is why you're a preferred provider versus why you're not," says Jamal Stockton, a board member at the Corporate Legal Operations Consortium and vice president and head of legal innovation and technology at Fidelity. "But it should—we need to get there, but we're just not there yet."

|

Setting the Standard

Many of the metrics legal departments use to evaluate law firms are seemingly straightforward. David Cunningham, chief information officer at Winston & Strawn, says many are just "straight accounting," so there's "not much concern over accuracy." He explains that the most common methods are based on financial metrics, to help the department understand potential savings and efficiencies.

Legal departments also often look at timekeeping metrics, such as the time it took a firm to close a matter and the breakdown of how long each staff member spent on it. Most, if not all, of these financial and timekeeping metrics are calculated automatically by legal technology platforms, such as e-billing and matter management tools.

But while there are few accuracy concerns, not all metrics are created the same. Cunningham, for instance, has heard from several legal departments that the time to complete a matter is a pivotal data point. "Well, how do you measure that?" he asks. "Is it from the first day you call me about it, or is it the first time you tell me we're approved? It's not that we think someone is doing it wrong, it's just that we may look at that number differently."

The lack of standardization is even more of a problem when trying to measure something more elusive: quality of work. For some, this metric is vital in allowing meaningful comparisons that go beyond basic accounting.

Stockton notes that by using a standard set of qualitative metrics, two legal departments working on similar matters but using different firms could compare the "quality of the engagement" of each firm. "That is the power of scalability and community sharing of information," he says.

Most of the subjective metrics legal departments create for their outside counsel come from internal surveys and rating systems. For instance, Fidelity's legal department has a proprietary "Amazon-style review tool" with a five-star rating system covering criteria including subject-matter expertise, a law firm's timeliness, the value-to-quality ratio, communication skills and desired outcomes, Stockton says.

Other legal department surveys can differ in their criteria and rating systems. But in recent years, there has been a concerted push to get everyone on the same page. Along with stakeholders across the industry, Stockton was part of a CLOC effort to design a universal framework for qualitatively evaluating outside counsel.

HBR's Clem also participated in the CLOC initiative and says his organization has been a backer of the CLOC standards, which were released in 2017. He has seen HBR's clients embrace the standards as well. "To my knowledge, ever since the CLOC standard was adopted, every time we help the client with an evaluation, we recommend the standards and they've been willing to use it," he says.

Still, CLOC's standards haven't been adopted by industry at large. Clem chalks this up to what he calls a "classic challenge": "Are you willing to compromise to benefit the industry, or do you want to tailor it to your own?"

Some of that is to be expected.

"Every in-house counsel department is going to have a different set of criteria based on discussions with their ­general counsel, staff and attorneys to determine what's important to them," Stockton says.

But the lack of progress on standardization also speaks to something equally as fundamental: Qualitative metrics are still fairly new, and legal teams may be too focused on getting them off the ground to think of the industrywide ­implications. "The legal ops person in the legal department, I believe, has a lot of trouble getting their lawyers to complete the after-matter survey at all," Cunningham says.

|

If You Build It, Will They Share?

Legal technology platforms have sought to address the potentially arduous creation and management of metrics. HBR has a survey platform to help legal departments easily collect and ­organize survey data. The platform connects to HBR's legal analytics solution, CounselCommand, which can display these qualitative metrics, alongside more quantitative ones, in dashboards.

Similarly, the LegalVIEW Predictive Insights Module from Wolters Kluwer aims to streamline how legal departments create and use metrics. When a matter is opened in Wolters Kluwer's legal management platform, Passport, an AI-powered module analyzes the matter's characteristics and cross-references it with a list of the law firms the user has worked with in the past. It then provides a list of the top 10 firms for the matter, taking into account both objective metrics, such as billing, and subjective ones, such as internal peer reviews, which each user weighs differently. "We are very transparent with the rating; the user is ultimately the one behind the decision," says Vincent Venturella, product director at Wolters Kluwer's ELM Solutions.

The tool also offers predictions around firm budgets and cycle times for specific matters by tapping into LegalVIEW's database of customer spend and matter management data. While platforms like these offer their users some ­standardization—in LegalVIEW's case, around how financial and cycle time metrics are calculated—they also allow sharing of metrics between in-house and outside counsel. "All of the data we deliver is fully reportable," Venturella explains. "We deliver it in such a way that clients can extract and share" with their firms.

There are, however, few signs that much sharing is actually happening. "The reality is that law firms never see the information," Cunningham says. It's like if restaurants didn't have access to their customers' reviews on Yelp, and didn't know what they needed to change. For law firms that miss the mark, "there is nothing to contest—they just don't get hired anymore," he says.

There are a few likely reasons why sharing isn't widespread. For one, law firms aren't asking clients for metrics or feedback. "In most firms, there is not yet someone who has the role to reach out to the customer and say, 'What are your metrics and how do we look to you?'" Cunningham notes.

For their part, legal departments aren't proactively discussing their metrics with their outside counsel. One director of legal operations for a California-based Fortune 200 company notes that much of the reticence to discuss metrics comes down to in-house not wanting to offend its law firm colleagues. "There is an old-fashioned stigma where the relationship kind of got in the way, because it was somehow awkward or inappropriate to bring up money" or even other performance notes, such as the attendance for a meeting or conference call, the director says.

But the director adds that communication is improving these days, partly because the use of metrics is growing, and "some of the former barriers are breaking down."

There is a long way to go. But like CLOC, others are taking action. Winston & Strawn is working with legal departments and law firms to develop a legal ops dashboard with readily shareable metrics, Cunningham says. The effort, which aims to get off the ground in early 2020 and will be more about guidance than overall standards, looks to spark agreement on defining key metrics and how to calculate them.

The project, however, is a key example of the problem facing legal metrics: the technology is there, but the communication is not.

"There are certainly law firms that have legal ops positions now. I hope those people are going proactively to clients and talking about these metrics," Cunningham says. "But the people I have talked to are not yet doing that. It's very early."