The final Day One session of the 10th Annual Ark KM conference addresses enterprise search in “So You Think You Know Search…?”, which addresses “the disconnect between what law firms think they know about Search—and what’s actually happening.” (Live blog post – please forgive typos or errors. The end of this post has the text of a Michael Mills hand-out – a search best practices checklist.)


  • Michael Mills, President & Chief Strategy Officer, Neota Logic,
  • Oz Benamram, Chief Knowledge Officer, White & Case,
  • Kingsley Martin, President & CEO, KMStandards,
  • Evan Shenkman, Senior Knowledge Management Counsel, Ogletree, Deakins, Nash, Smoak & Stewart, P.C.
  • Joshua Fireman, Fireman & Company, moderator


Joshua kicks-off the session, noting that search was a big KM topic a few years ago but not today. He asks why? Some say the reason is that the search problem has been solved. The tools exist today to find documents, experts, and matters. But as panel did its prep, they realized that search has, not in fact, been conquered. So the panel will explain why we should still think about and focus on search.

Evan will talk about search at Ogletree; Michael will talk about why most do not use search appropriately.

Oz characterizes search as basic KM building block. If you don’t have it, give up on KM.

Evan on Recommind Decisiv Search at Ogletree Deakins…  The firm is integrating search into multiple applications. That means pre-specified search terms for specific topics that users regularly need access to, for example, issues around ADA Act. Within KM documents, the firm builds in pre-built searches to take users to more detailed information. These are highly crafted searches that are typically much better than typical user could construct.  Another example is a pre-built search to find 50-state surveys on particular topics. (RF: this presumably solves the problem that surveys mention many legal terms but are not about those terms. So searching them is tricky.)

Someone asks if pre-built searches are a transition to a time when lawyers learn more sophisticated searching or a substitute. I read the answer as both but more the former. In using pre-build searches. users will see the construction of searches, showing by example how to construct a complex search.

Michael – “Eat Your Search Vegetables” – a checklist of what you must do for effective search….  [see below for longer, handout version of the checklist]

  • Rigorous Metrics.   Few firms track how search is used. Contrast that to Google or Bing. Search logs are rich sources of information about what users look for and, with some effort, about how successful the searches are. Firms should have a metric of successful searches, for example, percent of searches that yield useful results. Suggests compiling stats quarterly and distributing to KM team. Bold KM leads will share with firm management. Oz suggests that logs also show who is not using search – and remediate (shares example of office that was not using search because PSL was on leave, so firm trained users).
  • Training.  At roll-out, firms train on search. But they overlook that people have an immense capacity to forget. So recurring training or awareness building is essential.
  • Weighting Search Results.  Put weights into system to improve relevancy ranking. Google constantly tunes algorithms (with an army of engineers).  Modern law firm search engines have methods to tune results. For example, put extra weight on vetted precedents or adjusted how system deals with time by using time multiplied by billing rate instead to give more weight to more experienced lawyers).
  • Capture the Multiple Authors of Documents.  On the back-end, you can capture the lawyers who worked on a document and add all those names to the author field for purposes of search.  This overcomes the problem that the DMS author is often a secretary or junior associate.


Joshua tees-up the next part of the discussion… how we can do more with search.  Cites as example using search to identify lawyers based on billing rate, realization, and jurisdiction to help staff matters.  This might not be an app rolled out to all in the firm but would be useful to anyone responsible for staffing matters.  This information can, with off the shelf tech, be presented in easy-to-read tabular format.

Oz says the real future of search is not search but is data analytics to anticipate problems.

Michael – at search result time, you can display contextual information from other systems to enrich results.

Kingsley…. Crystal Ball portion…. asks if something is beyond search that will disrupt market. Tech has moved from data profiling and search to classification and analytics to data extraction and big data predictive analysis. Suggests that we can eventually – within a decades – use these developments to replace some associate work. Eventually we will be able to predict outcomes. To get there, we will need developments in data extraction, which remains weak today. Suggests that IBM Watson JD will substitute for associates by the 2020’s.

[Q&A not captured.]


ADDENDUM – Michael Mills List – Eat Your Search Vegetables!  – Check List for Good Search

ü Review search logs regularly, at least monthly.
ü Track search patterns and trends—by user, practice group, location, content type, etc.
ü Attempt to measure search success—for a given query, were the results returned reasonably useful (precision) and reasonably complete (recall).
ü Develop a search metrics report and publish it quarterly to the search team and firm management.
ü Conduct annual user feedback sessions, in person or by survey.
ü Review and update training materials (particularly videos) to reflect current software capabilities and current content.
ü Embed training materials and links in the search UI.
ü Develop practice-specific and content-specific training programs, integrated with practice group substantive training so eligible for CLE.
ü Develop training programs for administrative and other staff.
ü Remember that many people don’t know even very basic search techniques, such as using quotes for phrases or starting a search with a minimum number of keywords.
ü Remember that the firm has new associates (and other people) every year; they don’t know about enterprise search.
ü Have expert searchers develop and publish stored queries for use by others.
ü Advertise search on the firm intranet, with “success stories.”
ü Analyze ways in which search could be used to improve productivity, responsiveness or quality of IT, HR and other firm internal operations.
ü Integrate enterprise search with other firm applications.
ü Survey firm systems annually to find new content sources that should be indexed in the enterprise search system.
ü Review and update the user interface.
ü Provide tablet and phone access to search, without a VPN connection.
ü Install the latest version from the vendor, and all patches.
ü Exploit the “recommended results” feature of the engine.
ü Exploit the “related results” feature of the engine.
ü Weight relevance ranking of results by the searcher’s role (lawyer, etc.), practice group, office location, matter type, or other extrinsic criteria.
ü Use the annotation features of the engine, enabling users to rate or comment on individual results.
ü Include best practices collections in the search index and tune ranking so they rise to the top of search results.
ü Use a single interface for all search processes, rather than a separate search for personnel lookups, library catalogue queries, expertise location, client/matter lookups, intranet search, and so on.
ü Apply tools like Lexis Search Advantage to enrich content.
ü As new data elements are added to HR, CRM, finance, docket, and other systems, use them as new metadata filters (facets).