Use of technology-assisted review, or “TAR,” increasingly has become part of e-discovery practice. Predictive coding, which is often used synonymously with TAR, is a process and technology used to prioritize the review or automate the coding of documents. While there are various methodologies and approaches to predictive coding, they typically rely on subject-matter expert reviewers to train software, which then identifies or codes conceptually similar documents.

Since the first major decision about TAR and predictive coding in 2012, Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182 (S.D.N.Y. 2012), there has been a debate over the appropriate level of transparency between parties when TAR is used. While judges generally have encouraged parties to share such information openly, they have been reluctant to order such disclosure, given that it may not be warranted under the Federal Rules of Civil Procedure. Judges typically have allowed such “discovery on discovery” only when a requesting party can show deficiencies in the producing party's processes.

In the context of TAR, the question that remains is how judges can assess whether such deficiencies exist. In a recent decision that addressed this issue, a judge determined that reasonableness and proportionality—not perfection—should be the guiding principles for assessing the quality of TAR.

'Winfield'

In Winfield v. City of New York, 2017 WL 5664852 (S.D.N.Y. Nov. 27, 2017), an action concerning affordable housing, Magistrate Judge Katharine Parker presided over a discovery dispute questioning the integrity and reliability of the defendant's predictive coding TAR system. During the first phase of discovery, in response to the plaintiffs' complaints about the pace of the City's manual document review, Judge Parker directed the City to use TAR, specifically predictive coding, “to hasten the identification, review, and production of documents responsive to Plaintiffs' document requests.” Id. at *4. The City complied with the court's directive. It put in place a TAR system, trained the system using documents from its manual review, and then conducted several additional rounds of training. After training had been completed, the TAR system assigned a score to each of the remaining unreviewed documents. The City then determined an appropriate cut-off score: Documents above this threshold were reviewed for responsiveness and privilege, while items that fell below were deemed non-responsive and were not subject to further review.

The plaintiffs complained about the quality of the City's TAR process. In advancing this challenge, the plaintiffs cited five documents that the City had classified as non-responsive, but nonetheless produced in fully redacted form given that they were part of a responsive document family. As a result of a production error, however, the plaintiffs were able to review the full text of the documents the City had attempted to redact. The plaintiffs claimed that these documents were actually responsive to their production requests. With the quality of the process in question, Judge Parker ordered an in camera submission by the City detailing its TAR process and its training for document reviewers.

Looking for more details, the plaintiffs requested that Judge Parker order the City to provide various random samples of documents that either fell above the predictive coding ranking cut-off and were nonetheless coded non-responsive, or that fell just below the cut-off and thus were not reviewed. The plaintiffs also requested, among other items, “information about the ranking system used (i.e., what cut-off was used, and how many documents were deemed responsive and unresponsive at each ranking),” as well as “materials submitted by the City for the Court's in camera review relating to predictive coding[.]” Id. at 6.

Judge Parker's Decision

In her decision, Judge Parker, citing both to recent precedent and to Principle 6 of The Sedona Principles, stressed the maxim that “the producing party is in the best position to 'evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own electronically stored information.'“ Id. at *9. She noted that “courts have not micro-managed parties' internal review processes for a number of reasons:”

First, attorneys, as officers of the court, are expected to comply with Rules 26 and 34 in connection with their search, collection, review and production of documents, including ESI. Second, internal attorney ESI work processes may reveal work product, litigation tactics, and trial strategy. Third, as noted above, the producing party is better equipped than the court to identify and utilize the best process for producing their own ESI consistent with their obligations under the Federal Rules of Civil Procedure. Fourth, perfection in ESI discovery is not required; rather, a producing party must take reasonable steps to identify and produce relevant documents.

Id. (citations omitted). Judge Parker wrote that she was “of the view that there is nothing so exceptional about ESI production that should cause courts to insert themselves as super-managers of the parties' internal review processes, including training of TAR software, or to permit discovery about such process, in the absence of evidence of good cause such as a showing of gross negligence in the review and production process, the failure to produce relevant specific documents known to exist or that are likely to exist, or other malfeasance.” Id.

With this in mind, Judge Parker determined that her in camera review of the City's TAR processes “reveal that the City appropriately trained and utilized its TAR system” and that the City's efforts to train its document review team were sufficient. Id. at 10. She ultimately concluded that “neither this Court nor Plaintiffs have identified anything in the TAR process itself that is inherently defective; rather, Plaintiffs' objections are premised upon human error in categorizing a small subset of documents as responsive or non-responsive.” Id. at 11.

Nonetheless, citing the human error that did occur, the low responsiveness rate of documents as categorized by TAR, and the increased transparency that would result, Judge Parker granted in part the plaintiffs' requests for samples of documents. She directed the City to provide the plaintiffs with a random sample of 400 non-privileged, non-responsive documents “on an attorneys' and experts' eyes-only basis.” Id. And, should the sample set raise additional questions regarding the completeness of the City's document productions, Judge Parker encouraged the parties to meet and confer “to determine whether additional training and review is necessary with the understanding that reasonableness and proportionality, not perfection and scorched-earth, must be their guiding principles.” Id.

Judge Parker denied the plaintiffs' requests for more information regarding the City's TAR ranking system and for the materials relating to her in camera review, stating that it was “unclear how this information is even potentially relevant to the claims and defenses in this litigation, as required under Federal Rule of Civil Procedure 26.” Id. at 12. Even so, she encouraged the City to provide this information “in the interests of transparency and cooperation in the discovery process.” Id.

Conclusion

The decision in Winfield relies on the notion that “discovery on discovery” is generally only appropriate with some showing of a deficiency in the discovery processes. Here, having found such a showing only in part, Judge Parker rightly applied a strict test that defers to the decisions of the producing party in the absence of compelling evidence to the contrary. Winfield adds to the growing body of law that as long as a producing party's use of technology-assisted review tools, including predictive coding, is reasonable and proportional in the context of a matter, the mechanics of such efforts should not be open to scrutiny by an opposing party. With this in mind, Winfield provides additional comfort to those who are looking to increase their use of advanced technology in an effort to streamline document review.

Christopher Boehning and Daniel J. Toal are litigation partners at Paul, Weiss, Rifkind, Wharton & Garrison. Ross M. Gotler, e-discovery counsel, and Lidia M. Kekis, e-discovery attorney, assisted in the preparation of this article.