Computer v Human Search Revisited
Can lawyers rely on search to find documents in e-discovery? Recent decisions suggest perhaps not.
Craig Ball’s blog post, Grimm Prognosis for ESI Search (6 June 2008) recaps decisions by Magistrate Judge John Facciola in U.S v. O’Keefe and Equity Analytics v. Lundin and by Magistrate Judge Paul Grimm in Victor Stanley, Inc. v. Creative Pipe, Inc.. Ball captures a central issue: “It’s assumed that lawyers are qualified to review documents and decide their relevance, responsiveness and privileged character. But are we qualified to craft proxies for our judgment in the form of keyword searches?”
Ball quotes Judge Grimm’s ruling that testing (“utmost care in selecting methodology”) is required to rely on software searches. I’m all for process and testing and the utmost care – if applied uniformly, including to lawyer performance on document review. Most judges and lawyers seem to assume that lawyers are (1) qualified to review documents and (2) better at doing so than software. The truth of either is purely an empirical question; the presumption of lawyer accuracy may well be false.
I’ve seen no evidence that passing the bar qualifies lawyers to review documents accurately. Even if it does, I’ve seen no evidence that lawyers can maintain accuracy reviewing documents hours on end (which is the norm). In fact, I’ve seen contrary evidence. Even if lawyers are qualified and, almost miraculously, do not falter as the hours roll by, do they always agree on how to classify a document? I have seen many instances where lawyers disagree. Of course, if qualified and alert lawyers can disagree about document designations, where does that leave software?
I’ve raised a similar point previously in response to a Law Tech News October 2006 article proposing to place a high burden on EDD vendors. LTN published my reader response arguing that the same burden should apply to lawyers. My logic there applies here.
The legal profession seems ready to set high standards for vendors and for software. Let’s be consistent and apply the same to lawyers reviewing documents. If we do, will any approach meet the standard of care implicit in Judge Grimm’s decision?
As the profession ponders how to conduct e-discovery, consider some practical ramifications. Blogger-lawyers Jim Beck and Mark Hermann of the Drug and Device law blog wrote a great post called Will Technology Increasingly Favor Plaintiffs?. The gist is that the long-standing gap between the discovery effort for plaintiffs and defendants has grown dramatically in the age of digital data. They note that
“We fear that search technology will increasingly permit plaintiffs to identify with relative ease the specific information that they need, but the technology will be far less helpful to defendants trying to separate the wheat from the chaffe.”
They rightfully fret that search technology “assume[s] that you know what you’re looking for” but observe the defense lawyer’s job is to put information in context and that context may not be knowable in advance.
As a profession, we need to
1. Develop consistent, evidence-based standards for how we manage discovery and review and
2. Consider whether the “e” in “e-discovery” has tilted the playing field unacceptably and, if so, whether there are policies we can create to fix the problem.
- Alternative Legal Provider (44)
- Artificial Intelligence (AI) (57)
- Bar Regulation (13)
- Best Practices (39)
- Big Data and Data Science (14)
- Blockchain (10)
- Bloomberg Biz of Law Summit – Live (6)
- Business Intelligence (21)
- Contract Management (21)
- Cool Legal Conferences (13)
- COVID-19 (11)
- Design (5)
- Do Less Law (40)
- eDiscovery and Litigation Support (165)
- Experience Management (12)
- Extranets (11)
- General (194)
- Innovation and Change Management (188)
- Interesting Technology (105)
- Knowledge Management (229)
- Law Department Management (20)
- Law Departments / Client Service (120)
- Law Factory v. Bet the Farm (30)
- Law Firm Service Delivery (128)
- Law Firm Staffing (27)
- Law Libraries (6)
- Legal market survey featured (6)
- Legal Process Improvement (27)
- Legal Project Management (26)
- Legal Secretaries – Their Future (17)
- Legal Tech Start-Ups (18)
- Litigation Finance (5)
- Low Cost Law Firm Centers (22)
- Management and Technology (179)
- Notices re this Blog (10)
- Online Legal Services (64)
- Outsourcing (141)
- Personal Productivity (40)
- Roundup (58)
- Structure of Legal Business (2)
- Supplier News (13)
- Visual Intelligence (14)