Last month I suggested in Wall Street Traders Trust Computers, Why Don’t Lawyers? that the time has come to rely on automated document review in discovery. A comment by Keith H. Mullen of Winstead Attorneys raises a great question, which I address here. 

[I drafted this post before the New York Times article yesterday (5 March 2011), Armies of Expensive Lawyers, Replaced by Cheaper Software. I will discuss that at the end of this post.]

Mr. Mullen commented:

“Good comment and question. My first thought: predictive coding should be one screening tool – but like artificial intelligence, it is ‘artificial’ – at some point, there are subtle issues that even humans view differently – don’t reasonable people disagree? So, just as the artificial intelligence movement failed (even Susskind moved on), at some point it takes a human being to be the ultimate judge or screener. What do you think?”

I am not entirely sure of the ‘right’ response to Mr. Mullen’s comment. Consider how the TREC study, an e-discovery full-text evaluation sponsored by NIST, deals with document designation challenge. It names a senior law as the ‘topic authority’ who is the arbiter of document designations. By naming one and only one person with final say, TREC avoids the problem of conflicting expert opinions.

Mr. Mullen is indeed correct that two or more human experts often reach different conclusions. In a case with multiple topic authorities who do not agree on a document designation, I can think of two possibilities:

  • with enough work, they could reach unanimous agreement or
  • no amount of work suffices to reach unanimity.

If the former is true, then I suspect we could extract the underlying ‘know-how’ or logic and embed that logic in a computer system. Whether economics and practical considerations actually so allows is a different issue. If the latter is true, then presumably a majority vote of the experts wins.

“Majority wins” is bred in our bones but is not an empirically sound basis to reject the analysis of sufficiently sophisticated software, especially where that software embeds the heuristics of multiple experts. Given the cost of actually causing multiple topic authorities to review more than a small sample of documents, I suggest society is better off relying on computers.

If the experts mainly agreed and if computer designations frequently varied from the expert consensus, then I would reject computerized review. The limited empirical studies to date, however, suggest that experts rarely reach consensus and that computers are more consistent at document review. That is, computers may not t always get the right answer but that does not mean we are better off with the divergent views of multiple topic authorities.

So I will stick with the conclusion of my prior post that “courts should reverse the presumption. They should presume that predictive coding is reliable. The burden of proof should shift to predictive coding opponents to show that it is not reliable.” Let the proponents of human review explain to the court why diverging expert views is better than consistent computers.

The Times article does not present information new to e-discovery professionals It is important though because of the impact it might have on not just the substantive discussion but who is in on the discussion of predictive coding. When CEOs and CFOs read it, they may well ask their GCs, “are you doing that that articles talks about?” GCs will find that they likely have to explain and quantify the risk of relying more heavily on computers. Today, I suspect that many a GC is not well prepared for that conversation.