Last month, in As World Embraces Statistics, Lawyers Sit on Sideline, I lamented the lack of statistical mindset in the legal market. Why should we care? 

One reason: statistics can be very important at trial. Bob Ambrogi, in Statistics Surge as Evidence in Trials (IMS Expert Services August 2009 Newsletter, explains the growing importance of statistical evidence.

Another reason: to spot potentially unreliable statistics or analysis, whether for law practice or firm management. To illustrate, I deconstruct the Incisive Legal Intelligence law firm Economic Confidence Index (PDF) released in August (referenced at (an August 4th Incisive blog post).

Not Describing the Sample. To interpret a survey, you must understand who responded: how many and what demographic. Small, skewed, or unknown samples may yield unreliable results. I did not find a description of who – position/title, location, or firm size – responded to the Confidence Index. Who knows if the sample reflects large or small law firms or a mix.

Comparing Surveys from Different Points in Time. Comparing responses at two points in time requires like populations or samples. Incisive writes “[c]ompared to the beginning of 2009, the majority of firms report that business conditions have either improved or stayed the same rather than worsened.” To start, I was perplexed because the text says this is the first month of the Confidence Index. If so, it is not clear how any comparison whatsoever is possible. To boot, if Incisive conducted an earlier survey, there is no information on whether the sample was comparable.

Ask Unambiguous Questions and Help Readers Interpret Surprising Results. The Confidence Index survey asked respondents “In which of the following ways has your firm taken steps toward cost cutting/maintaining profitability as a response to the current economy?” This question does not specify a time period so answers presumably could refer to last month, 2009 year to date, since the crisis began (exactly when was that?), or some other time period. Does this matter? I think so. In this survey, 33% of firms report lay-offs in answer to the question. Compare that to data at at LawShucks, which show about 250 reported AmLaw 200 lay-offs. Surveys should not leave readers wondering about basic interpretation. If results are surprising in comparison to other widely reported data, then offer an explanation.

Report Data Accurately and Consistently. All the Incisive graphics and tables report on questions referring to 2009, the past six or twelve months, or in the next three or six months. Yet the text states “looking forward to 2010…[60% anticipate improving conditions]” What is the basis for Incisive comments about 2010? Three explanations come to mind: the survey asked questions about 2010 not reported in the write-up, the authors confuse 2009 and 2010, or the authors assume 2009 second-half data predict 2010. Write-ups should accurately reflect the data collected and the analysis.

Imagine presenting the Confidence Index to a judge who happens to understand stats and data. If you were the lead litigator, would you want to put this Incisive survey into evidence?

It’s a new era for BigLaw. For both the management of large law firms and the proper handling of litigation, law firm managers and litigators will increasingly need to collect and analyze and present data. Let’s hope, as Bob Ambrogi suggests, some lawyers and staff really can deal with data properly.

For those who do see a need to learn more about stats, law department management consultant and blogger Rees Morrison suggests 10 free courses to learn more about statistics in his September 13th post.