Two recent e-discovery webinars suggest that the legal profession has yet to settle on the best approach to e-discovery document review.  

E-Discovery: Search Engines & The Lawyer’s Role in Review (law.com, 2/28/07) addressed using key words versus concept searching to speed document review. Implicit in the discussion was that it’s not a matter of if but when most litigants will rely on software assisted review. I share that assumption.

The panelists had a spirited discussion about simple Boolean search versus concept searching and what directions the courts would take. Some panelists took what I consider the only sustainable position: determining the best approach is an empirical question. They also pointed out that we will need a series of published opinions to settle the question.

Information Risk Management and E-Discovery: Alternative Approaches to Document Review (H5 Technologies, 3/1/07) included a presentation describing just such an empirical test: about 50,000 documents reviewed by both humans and an advanced concept search tool. The software performed better than the reviewers. Someone in the audience asked about the risk of the software missing a document. The panelist rightly answered that the question was backwards given that her data showed the software did better than the humans.

Here’s my take after listening to both. Many lawyers appears honestly to believe that human review is accurate, the “gold standard” for document review. “Honestly held” and “right” can diverge. I, for one, have never seen data to support the commonly accepted “gold standard.” I suspect that the proponents of it are the same ones who postponed dealing with e-data as long as they could.

Absent a well-controlled study with widely accepted findings, we seem doomed to years of costly litigation and a trickle of published decisions to establish a new standard. If I seem jaded, well, I am. Boolean search has been available as early as 1980. A seminal scholarly article (Blair and Moran, 1985) showed that Boolean search alone found less than 20% of the relevant documents. In 1989, I used a concept search engine (PLS) that generally found more documents than Boolean searching alone. In 1992, there was good anecdotal evidence that machines did better than humans in objective coding.

At least there is a debate today and at least some EDD thinkers are at least talking about an empirical approach.