Measuring the Consistency of Legal Documents
Transactional lawyers spend huge amounts of time drafting and negotiating documents. Yet few have the tools to assess how their documents compare to similar ones drafted by other lawyers. That is changing.
An emerging product applies statistical metrics to compare like documents and help both lawyers and clients understand how to improve document drafting. More specifically, it facilitates building standard templates and knowing where to focus efforts on customizing. Kingsley Martin, known to many knowledge management professionals, has formed KIIAC LLC. His web site, which has the documents used for his initial analysis, is available at www.kiiac.com.
I have his permission to publish an e-mail message he sent to me. His note describes some interesting initial findings from his metrics-driven approach:
As part of our work to create document templates automatically, quantify differences among like documents, and develop very accurate searches for transactional documents, our research has discovered an interesting correlation: the more complex the transaction, the more likely the document consists of standard terms and conditions.
The table below shows a range of agreements and their consistency, measured by document structure commonality and clause language consistency. We base our analysis on 250-500 publicly available samples of each document type. We need to increase the sample set but the early trends of consistency from the document collection are emerging from our research:
Document Type Consistency Interest Rate Swap Agreement 97% Merger Agreement 90% Finance Agreement: (e.g. Term Loans, Credit Agreements etc.) 85% Corporate Formation: (e.g. Articles of Incorporation, Bylaws) 85% Employment, Consulting Agreements 65% Purchase or Lease of Real Property 60% Supply Agreements 55%
The statistical methods used to measure commonality are based on three main elements, simplified here for purposes of explanation.
- First, the presence of articles, clauses and sub-sections, namely the building blocks of a deal document. For example, the technology identifies whether each document has survival, amendment and waiver clauses, irrespective of where they may appear in a document. We also identify and count the number of deal-specific clauses that do not typically appear in a particular type of document. The ratio of standard to non-standard clauses gives us the clause commonality measure.
- Second, for clauses that have sub-sections, we measure the commonality of such sub-clauses. For example, in a merger agreement, what are the clauses in the representations and warranties article and how do they compare to the list clauses in this section from other documents? The ratio of common sub-clauses to non-standard clauses gives us sub-clause commonality measure.
- Third, the analysis measures the commonality of the words in each of the matching building blocks. The analysis identifies the common words for a particular clause, and then using this information computes the uncommon or deal specific terms. The ratio of common words to uncommon words in each matching clause gives us the measure of word commonality.
Using standard statistical techniques, we aggregate the commonality measures for each element to compute the overall document score.
We’ve performed the statistics. We are eager to hear from readers, especially practicing deal lawyers, why more sophisticated transactions tend to be more standard. Is it because sample documents for complex deals are more available online, thus causing a de facto trend to standards? On a related note, are the deal-specific terms the critical differentiator that marks the value of the document and the negotiating skill of the author?
If anyone has answers to Kingsley’s questions, you can e-mail him (kingsley dot martin at kiiac dot com) or leave a comment or contact me.
- Alternative Legal Provider (34)
- Artificial Intelligence (AI) (49)
- Bar Regulation (13)
- Best Practices (39)
- Big Data and Data Science (8)
- Blockchain (10)
- Bloomberg Biz of Law Summit – Live (6)
- Business Intelligence (19)
- Contract Management (19)
- Do Less Law (37)
- eDiscovery and Litigation Support (165)
- Experience Management (6)
- Extranets (11)
- General (189)
- Innovation and Change Management (157)
- Interesting Technology (95)
- Knowledge Management (219)
- Law Department Management (13)
- Law Departments / Client Service (110)
- Law Factory v. Bet the Farm (27)
- Law Firm Service Delivery (103)
- Law Firm Staffing (25)
- Legal market survey featured (5)
- Legal Process Improvement (21)
- Legal Project Management (26)
- Legal Secretaries – Their Future (17)
- Legal Tech Start-Ups (1)
- Litigation Finance (5)
- Low Cost Law Firm Centers (19)
- Management and Technology (177)
- Notices re this Blog (10)
- Online Legal Services (63)
- Outsourcing (134)
- Personal Productivity (38)
- ReInvent Law (10)
- Roundup (58)
- Structure of Legal Business (1)
- Supplier News (13)