Technology Assisted Review: An Acceptable Standard

Technology Assisted Review (also known as TAR or Predictive Coding) has become a common and important method of legal document review when dealing with large document populations upwards of 50,000 documents.  In the same way streaming music is replacing MP3’s, CD’s and dare I say it Vinyl, we are progressing through the life-cycle of new technology to a point where it is now an accepted standard.

As with any disruptive technology, progression starts with the innovators, these are people who take an educated guess (or often a leap-of-faith) and rely on new technology without knowing how they will fair. After the innovators have survived the fall the early adopters move in, they are keen to get in there early with the hope the innovators are onto something good.  Then it’s over to the early majority who bathe the comfort of proven success (and often believe they are innovators).  Followed behind are the late majority who basically get on board because they don’t what to be laggards, and finally onto the laggards who aren’t able to resist any longer and are forced to change (and who no doubt owns a vinyl record collection).

In TAR terms, I believe we are currently at a tipping point and in transition from the early adopters to early majority. So how did it evolve to this point?

In the beginning

I started using this technology about 8 years ago with a product called Equivio Relevance.  It was then I got to learn about the methods of training software and how to determine how well the software was performing by measuring the f-measure – which is used to determine statistical validation.  Back then it was definitely the innovators who were leading the way and the project I was working on was for a pharma company who was forward thinking and prepared to try something else if this didn’t work.  I was astonished at the results and the positive feedback I got from the legal team, but trying to convince others of its effectiveness was a very difficult thing to do. This technology was nothing more than a conjuring trick to many people, it lacked transparency and defensibility as there was little evidence to support the claims of its effectiveness.

Since then we have seen a number of leaps forward in both case law and scientific evidence.

Case Law

2012: Da Silva Moore v. Publicis Group, 11 Civ. 01279 (S.D.N.Y.)

In the US, Judge Peck’s judgement on the Da Silva Moore matter in 2012 held the first judicial endorsement of TAR.  This generated a large amount of debate and controversy around the courts involvement with the use of the technology, but it was a ground-breaking decision which has paved the way for its widespread use.

Following this decision there was a wave of other decisions around the US advocating the use of this technology, but internationally in other jurisdictions the courts were not providing their stamp of approval and litigators cautiously were not involving themselves in debate.

2015: IBRC & Ors. v. Sean Quinn & Ors [2015] IEHC 175

Finally, in 2015 the first landmark judgement outside of the US came from across the pond in Ireland. The Irish High Court gave its judgment in IBRC & Ors. v. Sean Quinn & Ors on the proposed use of TAR.  Citing US case law and academic studies, Justice Fullam held that “technology assisted review using predictive coding discharges a party’s disclosure obligations”.

2016: Pyrrho Investments Ltd v MWB Property Ltd [2016] EWHC 256 (Ch) &

David Brown v BCA Trading Limited & Ors [2016] EWHC 1464 (Ch)

The English courts jumped on the bandwagon in 2016, and like London buses, there had been a long wait and then two came at once.  First there was the Pyrrho Investments case where both parties seek approval in the use of TAR, which was agreed; and then David Brown v BCA Trading, where the use of TAR was contested but the judgement held “no factors of any weight” pointing against use of predictive coding.

2016: McConnell Dowell Constructors (Aust) Pty Ltd v Santam Ltd & Ors [2016] VSC 734

Australia is the latest to get on board.  Late in 2016 Justice Vickery appointed a Special Referee in the McConnell Dowell Constructors (Aust) Pty Ltd v Santam Ltd & Ors matter to deliver a report to the court addressing the appropriate management of discovery in the proceeding. The Special Referee said TAR was “consistent with the obligations of all parties”. This judgement also cited a number of previous international rulings.

Shortly after Justice Vickery from the Supreme Court of the State of Victoria passed this judgement, the court went on to amending their practise note in relation to Technology in Civil Litigation to include reference to the use of TAR and predictive coding.  Practice note SC Gen 5 states “technology assisted review will ordinarily be an accepted method of conducting a reasonable search”. This forward thinking leapfrogs the State of Victoria further ahead of any other jurisdiction in the world on its position relating to TAR, which provides me with certainly that others will soon follow.

Academic Studies

Lawyers and technologist often come from two very different mindsets.  Judge Perk talked about “bringing your Geek to court” to explain how the technology works but thankfully we now have some solid research to save us from those awkward moments.

The most notable study on the subject of TAR was published by Maura R. Grossman & Gordon V. Cormack in 2011.  The paper “Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review” used data from the TREC 2009 conference and concluded that two TAR processes (one using Machine Learning and one using a Rule Base) generally achieved better Recall, better Precision, and greater efficiency than the TREC Manual Review process.

After already recognising that TAR technology was effective, Grossman & Cormack published a further study in 2014, delving further into the different methods used to perform TAR and set out to understand the difference between the effectiveness of each method.

So where is this all heading from here?

There appears to be a perfect blend of case law and evidence to allow the late majority to step in and start applying this technology to their voluminous matters.  The question now is what is the most effective method and is this technology being used correctly. We have started to see requests for supporting evidence which proves the technology was applied in the right way, to ensure for example that the sample set was an accurate representation and the results were interpreted correctly.  These types of requests will continue as the technology matures and evolves.  It will become increasingly important for law firms to understand how the technology works, how it was applied to their matter and what steps they should take to become more familiar with sophisticated technology which they will be required to rely upon.

In my next post I will be describing how TAR works and discussing the different types of technology used. ->Published by Martin Flavell, Director of Sky Discovery UK Ltd



Leave a comment