" "

A Brief History of Technology Assisted Review

Technology Assisted Review (TAR) is now so widely used in e-discovery and so widely accepted by judges that one federal magistrate-judge recently declared it to be “black letter law.” But it was only three years earlier when that same judge, Andrew J. Peck, issued the first decision ever to approve the use of TAR. And it has been just five years since the terms “TAR” and “predictive coding” first began to filter into the legal profession’s vernacular.

So, how did TAR take root among lawyers? And how did it become so widespread so quickly?

Neither question has an easy answer. For TAR, there was no “Mr. Watson, come here!” moment of invention. Rather, TAR developed and took hold gradually at first, as understanding grew that it offered a viable solution for spiraling litigation costs driven by rapidly escalating quantities of data in litigation.

Tracing TAR’s Roots

One of the earliest articles to describe anything akin to TAR was by Anne Kershaw in 2005, “Automated Document Review Proves Its Reliability.” She described a study she conducted that compared a human review team against an automated document assessment system. While the humans identified only 51% of relevant documents, the automated system identified more than 95%.

The technology her article described was not what we think of today as TAR or predictive coding. But it similarly used statistical techniques to determine which documents were relevant and her analysis foreshadowed TAR’s eventual refinement.

“Automated document review and analysis provides significant new opportunities for attorneys in law firms and in corporate legal departments,” Kershaw wrote. “Legal review can be a more efficient, less costly, and a more proactive process that aids the legal team in managing the case.”

A key step forward came in 2006. That year, the Text Retrieval Conference, an organization started in 1992 by the National Institute of Standards and Technology (NIST) and the U.S. Department of Defense to study information retrieval techniques, launched the TREC Legal Track devoted to the use of search and information retrieval in law. Its annual research projects provided (and continue to provide) critical evidence of the efficacy of these techniques in e-discovery.

TAR Enters the Vernacular

It is impossible to pinpoint when any of the names we now use for this process—not just TAR, but also “computer-assisted review” and “predictive coding”—first came into use. They are mentioned nowhere in the TREC studies from 2006-2010. They are also nowhere to be found in another seminal early guide, The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery, published in 2007. A 2006 CLE handout referenced “machine-assisted review.”

What is clear, however, is that by 2010, these concepts had entered the legal profession’s vernacular. In fact, the years 2010-2013 were the critical juncture in the growth of TAR, with a series of developments that signaled that this evolving technology deserved serious attention.

The first of these developments was the 2010 publication of a study comparing the results of computer-assisted review to manual review. The study, Document Categorization in Legal Electronic Discovery: Computer Classification vs. Manual Review, was conducted by Kershaw along with Herbert L. Roitblat and Patrick Oot, who were all at the time affiliated with the Electronic Discovery Institute.

The study challenged the conventional wisdom that human review is the gold standard. It sought to determine “whether there was a benefit to engaging a traditional human review or whether computer systems could be relied on to produce comparable results.” Their conclusion: “On every measure, the performance of the two computer systems was at least as accurate (measured against the original review) as that of human re-review.”

Two Key Developments

A year later, another study even further upset the conventional wisdom. Two e-discovery researchers, Maura R. Grossman, counsel at Wachtell, Lipton, Rosen & Katz, and Gordon V. Cormack, co-director of the Information Retrieval Group at the University of Waterloo, analyzed data from the 2009 TREC Legal Track involving the use of TAR processes. They concluded that TAR was not only more effective than human review at finding relevant documents, but also much cheaper.

“Overall, the myth that exhaustive manual review is the most effective—and therefore the most defensible—approach to document review is strongly refuted,” they wrote in the Richmond Journal of Law and Technology. “Technology-assisted review can (and does) yield more accurate results than exhaustive manual review, with much lower effort.” Their study found that TAR produced a 50-fold savings in cost over manual review.

For lawyers and clients facing spiraling e-discovery costs, this was big news. It provided evidence that TAR could be both more effective than manual review and also significantly cheaper. But even with this evidence, many lawyers still saw one big obstacle to using TAR: the uncertainty of its acceptability to the courts.

That obstacle was significantly lowered on February 24, 2012, when Judge Peck issued his opinion endorsing the use of TAR in a case pending before him, Da Silva Moore v. Publicis Groupe. It was the first judicial opinion anywhere to endorse the use of TAR. “Computer-assisted review appears to be better than the available alternatives, and thus should be used in appropriate cases,” he wrote, opening the door to a sea of change in how lawyers conduct e-discovery.

The second major milestone in the courts came just two months later with the case Global Aerospace, Inc. v. Landow Aviation, L.P., No. CL 61040 (Vir. Cir. Ct. Apr. 23, 2012). In Da Silva Moore, the court approved TAR based on the parties’ agreement. Global Aerospace was the first case to approve the use of TAR over the opponent’s objection. It was also the first state court case to approve the use of TAR.

(The co-author of this article, Thomas Gricks, was lead e-discovery counsel in Global Aerospace. His work on that case was highlighted in a 2013 Wall Street Journal article, “How a Computer Did the Work of Many Lawyers.”)

 The Evolving Capabilities of TAR

There remained one factor limiting TAR’s widespread adoption: the limits of the technology itself. While first-generation TAR systems (TAR 1.0) represented a major—indeed, revolutionary—advance over manual review and keyword searching, they had shortcomings that limited their usefulness in many real-world contexts.

For one, TAR 1.0 systems required that a senior attorney be involved in training the system. The senior attorney would have to review and code hundreds or thousands of random documents until the system stabilized. Given that the goal of TAR was to reduce costs, requiring the involvement of a senior attorney was not a cost-effective way to go about it. In addition, requiring involvement of a senior attorney also frequently delayed the process from ever getting started in the first place.

Another shortcoming of TAR 1.0 systems was that they required legal teams to have all their documents at the start. If a subsequent batch of documents arrived later, as is typical in practice, the training would have to begin all over again.

Those shortcomings were overcome with the development of a new generation of TAR systems—so-called TAR 2.0 systems. Now gaining wide popularity, these newer systems no longer require senior attorneys for training. Instead, through a process known as “continuous active learning,” the review team can simply begin reviewing documents and the system will continuously learn from their coding calls and improve its results.

Also, because the system is continually learning and refreshing its rankings, new documents can be added at any time. This conforms to the way litigation occurs in the real world, where discovery documents typically arrive on a rolling basis.

In a 2014 peer-reviewed controlled study, Evaluation of Machine-Learning Protocols for Technology-Assisted Review in Electronic Discovery, Grossman and Cormack documented the superiority of CAL systems over earlier forms of TAR.  Their study found that CAL yielded “generally superior results” to other TAR systems and required “substantially and significantly less human review effort.”

TAR’s History Continues

Five years ago, TAR was virtually unheard of in litigation. Today, it has become essential litigation technology for its ability to prioritize documents and reduce the time and cost of review. Last year, more than half of U.S. corporations reported using TAR in litigation. There can be no doubt that percentage will continue to grow.

The history of TAR is still taking shape. But its place in the legal industry is now firmly embedded.

About Thomas C. Gricks III and Robert J. Ambrogi

Thomas C. Gricks III and Robert J. Ambrogi
Robert Ambrogi, Esq. is a lawyer and veteran legal-technology journalist and blogger. A practicing lawyer in Massachusetts, Bob also works with Catalyst as director of communications. Thomas Gricks, Esq. is a prominent e-discovery lawyer and one of the nation's leading authorities on the use of TAR in litigation. Tom is Managing Director of Professional Services for Catalyst and advises corporations and law firms on best practices for applying Catalyst's TAR technology, Insight Predict, to reduce the time and cost of discovery. He has more than 25 years’ experience as a trial lawyer and in-house counsel, most recently with the law firm Schnader Harrison Segal & Lewis, where he was a partner and chair of the e-Discovery Practice Group.

Check Also

legal services

Navigating Group Legal Services with Technology, Part III

The customer is not always right in legal services but the customer is always the customer. Today, more than ever, this is important to remember. 

  • Jake McCoy

    I switched from Cosmolex to practicepanther. This works better than some of the other software I tried to use. It is pretty easy to use. I’m on its trial since it is free ‘till the first month of the following year, I’ve been planning to use this permanently… I’ll keep you
    updated on what I finalize soon