When Alcoa Inc.'s senior attorneys sat down in late 2004 for their annual review of outside counsel costs, they wanted to get a feel for how their peer group's expenses compared.

The team first scrutinized data from off-the-shelf law department benchmarking surveys, but found that wasn't helpful because the universe of companies with more than $15 billion in revenue was too small. So the team hired Rees Morrison, co-chair of legal department consulting at Hildebrandt International, to conduct a benchmarking survey.

Seven months and many hours of internal work later, Alcoa had the results of a 10-question survey of 20 peer group companies that answered some of their questions and raised others. It provided a broad baseline for validating law department staffing and spending levels, but with less specificity than they had hoped.

Alcoa's experience says a lot about the state of law department benchmarking today. Looking for ideas for dealing with rising costs–and searching for evidence to justify staffing and spending to the CFO and CEO–legal departments are turning to benchmarking in a big way. A 2005 survey of 200 Fortune 1000 law departments by Bottomline Technologies Inc. found that just over half of respondents were benchmarking outside counsel spending and performance–and more than half of those had started benchmarking within the past five years.

Proponents say benchmarking is an important tool in law department management, providing valuable insight into how other companies manage legal costs. Critics cite methodological flaws and “apples and oranges” comparisons and contend that benchmarking promotes mediocrity rather than aspirations to the exceptional. And both sides agree that beyond its “cover-your-ass” value, benchmarking is pointless unless it is the jumping off point for an internal action plan.

“You may find you meet the numerical benchmark, but is that good enough? Do you want to settle for that?” asks James Wilber, principal at Altman Weil, a legal management consultancy. “As a consultant, the question we get all the time is, 'how do others do this?' Maybe the better question is 'how should we be doing it?'”

Before You Benchmark

There are two basic ways to benchmark against other companies. The first is to compare your data to the results of one or more of the available survey products that legal department consultants and professional organizations sell, or to compare it against surveys conducted by trade magazines. The second approach is to hire an outside consultant to conduct a benchmarking survey of a peer group you define.

Regardless of which route you take, there are several things you should do before launching the project. First, recognize that benchmarking isn't a panacea, and in fact may have unintended consequences.

“Benchmarking can be extremely dangerous because oftentimes all it accomplishes is setting forth a common mediocrity,” says Susan Sneider, partner in New Vistas Consulting, a legal PR and management-consulting firm. “By definition, benchmarking tells you what everybody else is doing. It doesn't tell you best practices. In some cases you can infer it, and in other cases you can't.”

Second, acknowledge that like all statistical data, benchmarking metrics can be misinterpreted and misused.

“Benchmarking is often done to justify the status quo,” says Aileen Leventon, partner in Blaqwell Inc., a legal industry consultancy. “The numbers are analyzed and presented to support a point of view, while data indicating a contrary point of view is ignored.”

With those concerns on the table, the third step is to determine the objective of the project, the audience for the results and how you will use the information.

“Benchmarking should be done to accomplish an objective, rather than as an end in itself,” Leventon says. “The point is to have information on which you can take action, that you can use to inform your planning.”

For example, she says, benchmarking may be done to identify opportunities for improvement. It may be done because the CEO has told the GC to cut costs. Or it may be done because the company is entering a merger and the legal department wants to position itself as the trophy department.

Fourth, define precisely what information you want to get and then find out whether that information is obtainable. You may discover that the particular data point you are looking for can't be determined.

The Alcoa survey, for example, did not answer a key concern that propelled the department into benchmarking in the first place.

“We tried to put the group of competitors together to see if our spike in legacy [environmental] lawsuits related to old plants was consistent with others,” Alcoa's Deputy General Counsel John Wilson says. “We found that there are so many variables that the question is probably not worth pursuing. We did not get data that gave us an 'aha' moment–we had to stand back and say 'that is too hard to isolate without doing forensic work on the cases.'”

Fifth, take the time to do some thoughtful self-analysis first.

“You need to look internally first and figure out what is happening in your own company,” says Rob Thomas, vice president of strategic development for Serengeti Law and director of the ACC/Serengeti Managing Outside Counsel Survey. “I'm amazed at how many GCs don't do that. Get your arms around what is going on in your own shop, and understand the trends within your own company. Then you can look at others.”

Leventon agrees, citing as an example a company involved in a regulatory investigation, with legal spending far exceeding that of another company in the same industry that was not under investigation.

“Comparisons with others are not meaningful unless you ask who am I and what drives my spending, compared to who are they and what drives their spending,” she says.

Finally, decide which approach to take. Look at the benchmarks in the surveys, see if they match up to your objectives and figure out whether you can get similar data on yourself. If so, you may be in a good position to compare survey metrics with your own numbers. If not, you may want to engage a consultant to do a custom benchmarking study, Wilber says.

Scrutinizing Surveys

If you decide to use a publicly available survey, you'll want to take a close look at the range of products available. Prices top out at $3,500 (with discounts for survey participants) for the Hildebrandt International U.S. Law Department Survey, which covers the broadest range of metrics including staffing, spending, organization, outside counsel management and compensation data. Other surveys that focus on one or two of those areas are generally priced from $500 to $1,000. Magazine surveys are free to subscribers but have less rigorous methodology.

If you decide to use a consultancy-produced survey, first consider being a survey participant. Benefits can include substantial discounts on the results, receiving a separate report tailored to your company, or an invitation to a seminar where professionals analyze the benchmarking metrics.

When choosing a survey, look for a sample group that is as close to your company as possible. For example, one-third of the participants in the 2005 Hildebrandt U.S. Law Department Survey were from companies with more than $20 billion in revenue, and two-thirds were companies with more than $5 billion in revenue. On the other hand, the Altman Weil 2005 Law Department Management Benchmarks Survey focused more on small- and medium-sized companies–less than one-third of participants were from companies with $5 billion of revenue or more. And in the 2005 ACC/Serengeti Managing Outside Counsel Survey, about one-third of participants are from companies with revenues greater than $1 billion.

Recognize that different surveys cover different metrics and choose one that meets your objectives. Also look for what breakdowns are available by region, industry and company size.

And realize that while the surveys cover hundreds of categories, the one you want may not be there.

Jeffrey Carr, vice president, general counsel and secretary of FMC Technologies Inc., a Houston-based producer of systems for energy exploration, food processing and airport ground support, draws a wide range of data from several surveys when benchmarking his department. Still he is frustrated by the lack of metrics on outside counsel costs per matter broken down by type, complexity and region.

“Benchmarking [fees per matter] would be useful, but the data is not available,” he says. “We have to rely on our own data going back 20 years [to determine a baseline for comparing litigation costs] and we don't know if our experience is normal, above normal or below normal.”

Consider A Consultant

To drill into specific questions or to obtain a better-defined group of participants, you may decide to hire a consultant to do a tailored study of a peer group you define.

In addition to a written survey of empirical data, the study may also include interviews exploring processes and practices that cannot be reduced to numbers. Fee estimates from major law department consulting firms range from $20,000 to $50,000, depending on the scope of the study requested.

This process yields better apples to apples comparisons because data from commercially available surveys typically defines industry groups very broadly. But Morrison acknowledges that the process is not without its limitations.

“You can always attack benchmarking data methodically,” says Morrison, who literally wrote the book on law department benchmarking, “Law Department Benchmarks: Myths, Metrics and Management,” published by Glasser LegalWorks. “You can't get enough participants to make correlations. Typically, you will have six to eight companies in a sample [for a tailored study.] It's not statistically reliable, but it will give you an idea of where you stand.”

Because the number of respondents will be small, selection of the target group is key, and of that group, expect at best a 50 percent response, Morrison says.

Isolating a peer group and tailoring questions to ask them proved to be a more challenging task than the Alcoa attorneys had anticipated.

“We had to determine the universe,” Wilson says. “We sat down and said, 'how do we define people who look like us?' The hardest and most important part of the work was drafting characteristics of the model group. We thought we'd just whip the questions out, but both refining the questions and trying to figure out the comparative group got more detailed than we thought.”

Wilson advises taking control of the process from the beginning, rather than delegating too much authority to a consultant.

“The overwhelming thing is that you need to be well organized internally about what you want to find out–otherwise you are at the mercy of the consultant,” he says.

Morrison recommends limiting the number of survey questions to 10 to encourage participation. He also says it is important to promise anonymity to participants and to disclose the name of the sponsoring company to maximize the participation rate. Then add an interview component.

“Those intent on improving need to do interviews (in conjunction with the written survey) on topics where numbers don't help you,” he says. Interview topics can cover a wide range, from “How do you control outside spending costs” to “What have you done in terms of succession planning?”

Cautionary Words

At Alcoa, the legal team feels its investment of time and money was worthwhile, even though the results were less precise than they wanted, and they plan to dig deeper with another survey this year in conjunction with one of the companies that participated in the initial research. At the minimum, the survey validated the team's gut feeling that the company has a thin internal legal staff relative to its peers and provided an external validation for certain costs.

“It triangulated where we sat in that group of companies and gave us a reasonable benchmark for where all this sorts out,” Wilson says. “It has given us a little bit of a baseline in terms of measuring our performance that we didn't have before. It's useful, but you have to be careful about the conclusions you draw.”

Recognizing the limitations of benchmarking and having reasonable expectations are important. Even those consultants who make their living from benchmarking warn not to expect too much.

“What companies gain is an idea of how their practices and operations compare with other groups, providing context on where they stand,” Wilber says. “What benchmarking does is give you a roadmap of what you need to consider and whether you want to make changes. I always point out that benchmarking doesn't tell you if you are doing something right or wrong–it shows you where you diverge.”

Jonathan Bellis, co-chair of law department consulting at Hildebrandt International, who has directed what is now the Hildebrandt survey since it was first developed at PriceWaterhouse in the mid-1980s, says the key is viewing benchmarking as just one piece of the law department management puzzle.

“Benchmarking is a tool, not a panacea,” he says. “Many tools should be considered. It's certainly a key thing that should be done, but it's not the only thing. You start with the survey data, but pretty quickly you need to get to best practices, based on an awareness of innovation. Otherwise, it's an empty discussion.”