We write today with what may become the most controversial finding from the GC Thought Leaders Experiment. Based on in-house evaluations of over 1,400 legal matters, the largest and most established law firms lag the rest of the Am Law 200 in delivering high quality client service. There are exceptions, but this overall finding holds, and it is statistically significant.

More specifically, the Am Law 21-200 is outpacing the Am Law 20 (the largest 20 firms by revenue) on several key service metrics, including responsiveness, efficiency, quality of work, and solutions focus.  Likewise, when we look at firms with high revenue per lawyer (an even better indicator of a firm's prestige and ability to charge premium rates) we see a similar storyline.

Note that this isn't an opinion survey: no one ranked firms. These are 28 companies' evaluations of over 1,400 legal matters; and because we knew which firms and lawyers handled each matter, we were able to determine which firms performed better or worse, as well as identify trends. We also controlled for many variables (such as type of legal work, length of client/firm relationship, size of legal matter, amount of spend with the firm, and the like) to test and re-test our findings. But the result was the same. On the whole, when clients look to hire firms, bigger or more pedigreed is not necessarily better.

In the spider chart below, the brown lines show in-house lawyers' evaluation scores on actual legal matters that Am Law 20 firms worked on, and the blue lines show scores for the rest of the Am Law 200. In all, the 28 clients—including Panasonic, Nike, Petco, PayPal, DXC, Keurig, and Mastercard—work with 126 of the Am Law 200, including 18 of the Am Law 20, and this formed the basis of our analysis. On a scale of one to five, the chart plots higher scores farther out on the “web” for each of the performance metrics shown.

What's striking here is that the blue shape is larger on all sides.  In plain English, while Am Law 20 firms performed well, on average the firms in the rest of the Am Law 200 exceeded them on every metric. (And the differences were even larger when we compared the Am Law 20 to the Second Hundred, or the Am Law 101-200.)

Let's unpack the data in the above chart. First, Am Law 20 cost/efficiency scores averaged 3.80 (less than “good,” which was 4), versus 4.31 for the Am Law 21-200, a statistically significant difference. (And it's 4.45 for the Am Law 101-200.) This finding isn't too surprising, as larger, more pedigreed firms typically charge higher rates and their staffing leverage often requires the use of teams of associates. (Similarly, firms in the Am Law 200 with high revenue per lawyer—a good proxy for prestige—earned lower cost/efficiency scores than other firms; the top quartile, for example, underperformed the bottom quartile.)

Arguably the most important data point in this spider chart relates to solutions focus: is the outside lawyer demonstrating creativity, strategic thinking, and a “yes, and here's how” (rather than “no”) mindset?  Again, we have a statistically significant difference between the Am Law 20 (4.34) and Am Law 21-200 (4.53). And it's 4.56 when we look only at the Am Law 101-200. (Likewise, firms with higher revenue per lawyer earned lower scores on solutions focus than other firms.)

This, too, may be a by-product of staffing leverage (associate to partner ratio) at the largest and most pedigreed firms: as bigger teams liaise with the client, the lead partner may not learn quite as much about the business or matter. And this may be exacerbated by the reality that in-house lawyers are stretched too thin with less time to work with outside lawyers; more than before in-house teams are expected to focus their time and energy collaborating with internal clients to drive business results.

To be clear, there are exceptions to this finding—some of the Am Law 20 (and firms with high revenue per lawyer) outperform; and through our own discussions with several of their chairs and managing partners, we observe quite varied attitudes and approaches to client service.

As for responsiveness, the gap in performance—4.61 for the Am Law 21-200 vs. 4.52 for the Am Law 20—is not large, but it's interesting because responsiveness is such a core element of top client service, and most of us assume large, pedigreed firms offer attentive white-glove treatment. But as the data show, this isn't the actual experience of GCs and their teams.

Related, a few years back, we shared opinions of 88 GCs from Fortune 500 companies, and only 11 percent thought Am Law 20 and Magic Circle firms were more responsive than other firms. Today's data empirically validate some of those impressions. The root cause may be that there is so much high-stakes work coming through the door that it's hard for clients to win the “war for talent” and receive top partners' attention, or perhaps responsiveness suffers when teams of partners or associates are larger. This may also be why the Am Law 20 put up slightly lower scores in quality of work and in expertise—it becomes less likely clients will have a top partner actively working on (vs. overseeing) their matters.

Also on the spider chart is outcome vs. expectation, which is the in-house lawyer's assessment of whether the “business or legal outcome was better or worse than expected” (on closed matters). As shown, the Am Law 21-200 outperform the Am Law 20, though the difference is not statistically significant.

There is, however, another important and statistically significant difference in the spider chart: client likelihood to recommend. The average score is 4.06 for the Am Law 20 and 4.26 for the Am Law 21-200. (And it's 4.34 for the Am Law 101-200.) This is important because likelihood to recommend is clearly the most all-encompassing metric of law firm success. So to help us understand what drives this metric, we set out to see which performance measures were most important in generating higher likelihood to recommend.

As shown in the bar graph, the factor most correlated with likelihood to recommend is solutions focus, and the least important (though it still matters) is low hourly rate. (Interestingly, efficiency is more important to clients than the lead partner's hourly rate. This is something we've always believed at AdvanceLaw, but until now we lacked data-based evidence.)

The pre-eminence of solutions focus also corresponds to the experience of our clients. Across the past two weeks, we assembled 50 of our chief legal officers (most from global companies in the Fortune 500) for day-long discussions. Many of the GCs voiced concerns over law firms' general lack of solutions focus, including an unwillingness to offer different paths forward and an assessment of the risk of each path. (To their credit, the GCs understood their teams don't always have time to educate outside counsel about the business, making it more difficult for firms and lawyers to offer solutions.)

Of course, the message here is not that, as a rule, there is something wrong with the Am Law 20 or, by extension, the Magic Circle, or Canada's Seven Sisters. We've worked with many of these firms, and spent time getting to know many of their leadership teams—each is a success story in its own right. That said, some firms perform better than others, and neither size nor pedigree is controlling—there is significant variance across direct competitors.

For example, below is the distribution of solutions focus scores (vertical axis) by law firm revenue (horizontal axis) for Am Law 200 firms where we had a sizable number of legal matters. The regression line (or “line of best fit”) shows the trend—on average, the larger the firm, the lower its score on solutions focus, but there is of course marked variance.

In sum, there's a natural tension among size, pedigree, and profitability—each of which tends to require higher associate to partner ratios—and solutions focus, which is what clients desire the most. (It may be challenging for lead partners to get to know the business and offer solutions if they're disintermediated by teams of partners and associates.) How a law firm resolves this tension may well determine its future, as clients are increasingly demanding more practical, solutions-oriented advice. We believe getting this right (being on the north side of the above chart) will be the key challenge for firms—especially large ones—across the next decade.

As to how it's possible that the largest and most esteemed firms have been doing so well if they are not consistently delivering the best client service, we have a few thoughts.

First, the growing size and profitability of the largest and most pedigreed firms, and their corresponding growth in associate to partner ratios, may have reached a tipping point in the past few years—and until that point this wasn't materially impacting service quality. On the flip side, some of the firms in the Am Law 21 to 200 only recently reached a size that gave them enough depth of expertise in key practice areas to handle high-stakes work; so there were fewer options for clients looking to deviate from big, branded firms.

Second, switching costs in the legal industry are high—incumbents know the client and have institutional knowledge—so change happens at a much slower pace. Related, some legal matters have long tails and switching firms mid-course is especially difficult. And remember, it is not that the large, pedigreed firms are performing poorly (just, on the whole, not as well as other firms).

This brings us to the third and most important reason for the disparity between firm performance and firm success. It's that the legal market has an information problem: we don't know which firms are performing better, so we are forced to default to brand (and high cost) as proxies for quality. But that is changing, as discussed in this Above the Law article. Clients, in essence, are finding out that the blue chip firms of the past are not necessarily the blue chip firms of the present (and future). Likewise, the “insurance” rationale for choosing certain firms for high-stakes work based on historical reputation should dissipate over time (but won't disappear).

Frankly, we were pleasantly surprised by what we heard from the 50 GCs we recently assembled. We offered them a sneak peek at some of our findings, and many noted this data had changed their long-held beliefs, and would impact their firm selection decisions. Our own view is that many GCs have wanted to migrate more high-stakes work to promising but less branded firms for quite some time, but were understandably risk-averse.  Regardless, this data finding, as well as additional data on specific law firm performance, should be a game-changer in the market for legal services.

Of course, this change won't happen overnight, but it will be interesting to look at the Am Law 200 rankings in three or five years. One thing is certain: clients should prefer actual performance data to historical pedigree. Clients who think otherwise will pay for it—likely in more ways than one.

Firoz Dattu is founder and chair of AdvanceLaw and Aaron Kotok is managing director.