Technology-assisted review (TAR)—an algorithm-based method of reviewing documents for relevance—is a powerful tool that can improve the accuracy of document review while reducing cost. Some litigators have been reluctant to utilize it for fear of expensive collateral litigation about its implementation. Numerous decisions have endorsed TAR; however, they also reveal that courts may insert themselves into the discovery process under the guise of transparency. In Winfield v. City of New York, No. 15-CV-05236, 2017 U.S. Dist. LEXIS 194413 (S.D.N.Y. Nov. 27, 2017), the court went so far as to order the production of samples of nonresponsive documents to satisfy the plaintiffs' request for transparency.

Below, we review Winfield against the backdrop of three significant TAR cases authored by S.D.N.Y. Magistrate Judge Peck, as well as other illustrative cases, and offer the following best practices so litigants can retain control of their TAR workflow and maximize its efficiencies:

1. Counsel should be prepared to educate the court and opposing counsel on applicable electronic discovery law and foundational principles.

2. The Federal Rules of Civil Procedure apply a reasonableness standard; they do not require perfection. Counsel should be prepared to leverage the metrics and statistics provided by most TAR systems to establish that their process was reasonable and defensible.

3. Counsel should evaluate what, if any, aspects of their TAR process they will share to demonstrate cooperation, a practice many courts have found meaningful, when addressing challenges.

The Winfield Case

The Winfield plaintiffs alleged that New York City's housing programs constituted intentional racial discrimination and sought wide-ranging discovery. The city employed search terms to identify a review population and commenced a manual responsiveness review. When the plaintiffs complained about the pace of discovery, the court directed the city to use TAR to hasten the process.

The plaintiffs thereafter complained that the city had over-designated documents as nonresponsive and, consequentially, disabled the TAR algorithm from properly recognizing responsive documents. The plaintiffs proffered five documents that were incorrectly designated as nonresponsive (but inadvertently produced) and sought an order requiring the city to produce a sample of nonresponsive documents, all materials submitted by the city for in camera review relating to predictive coding, and information about the relevancy ranking system and the relevance threshold cutoff.

Noting that courts have not traditionally micromanaged parties' document review, the court held that production of electronically stored information (ESI) was not so exceptional as to warrant inserting itself into:

internal review processes, including training of TAR software, or to permit discovery about such process, in the absence of evidence of good cause such as a showing of gross negligence in the review and production process, the failure to produce relevant specific documents known to exist or that are likely to exist, or other malfeasance.

Id. at *19-20. The court determined that the few documents that were incorrectly coded as nonresponsive did not constitute a sufficient basis to question the accuracy and reliability of the city's TAR process and found no basis to hold that the city engaged in gross negligence. Notwithstanding its holding, the court found “sufficient evidence to justify” the plaintiffs' request for samples of nonprivileged documents, noting that the “sample sets will increase transparency.”

While it denied the plaintiffs' request for documents submitted in camera, the court encouraged the city to share the information with the plaintiffs “in the interests of transparency and cooperation.” Id. at *35-38. Measured against the leading cases in the area, the Winfield court's interference with the parties' use of TAR is intrusive.

Foundational Authorities

Several decisions by Magistrate Judge Peck provide guidance to parties wishing to use TAR.

In Da Silva Moore v. Publicis Groupe, 287 F.R.D. 182 (S.D.N.Y. 2012), a gender-discrimination suit, Judge Peck found that “computer-assisted review [CAR] is an acceptable way to search for relevant ESI in appropriate cases.” Reasoning that linear review would be too expensive and that it was a “myth” that manual review was the most reliable approach, Judge Peck rejected the plaintiffs' challenge to the use of TAR. Judge Peck acknowledged that, although not all counsel would be willing to share their seed sets, parties should “at least” be willing to be transparent as to the procedures employed. Id. at 190-92.

In Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125 (S.D.N.Y. 2015), Judge Peck found that “it is now black letter law that where the producing party wants to utilize TAR for document review, courts will permit it.” Id. at 127. Judge Peck noted that courts customarily leave it to the parties to decide how best to respond to discovery. Reviewing relevant authorities, Judge Peck found that when the parties do not agree to transparency, decisions are split. He reasoned that requesting parties can validate the sufficiency of the producing party's TAR training and review by other means, such as a statistical estimation of recall, looking for production gaps, or a quality-control review of a sample of nonresponsive documents. Judge Peck stressed that TAR should not be held to a higher standard than keyword or manual review, recognizing that doing so would discourage parties from using it.

In Hyles v. New York City, No. 10 Civ. 3119 (S.D.N.Y. Aug. 1, 2016), the plaintiff sought to force the city to use TAR, a proposition rejected by Judge Peck, who noted that while he would have liked the city to use TAR, it is not up to the court or another party to tell the city how to search its own ESI.

Illustrative Case Law

Courts differ as to the level of cooperation and transparency required when parties use TAR. Several courts have left it to the discretion of the producing party to decide the best method of review and production. Other courts have inserted themselves into the process and imposed higher standards on TAR than on traditional linear review.

In In re Biomet M2a Magnum Hip Implant Products Liability Litigation, No. 3:12-MD-2391, 2013 U.S. Dist. LEXIS 84440 (N.D. Ind. Apr. 18, 2013), a multidistrict litigation involving millions of documents, the court rejected a steering committee's claims that utilizing keyword searches before predictive coding “tainted” the training process. The steering committee argued that Biomet should be ordered to apply predictive coding to the universe of documents initially collected, and that the parties should jointly train the predictive coding software. The court declined, finding that Biomet's process complied with the Seventh Circuit's ESI Discovery Principles, publications from the Sedona Conference, and Federal Rule of Civil Procedure 26(b). The court explained that cooperation does not require “counsel from both sides to sit in adjoining seats while rummaging through millions of files that haven't been reviewed for confidentiality or privilege.” Id. at *6. The court also denied the request for Biomet's “seed set,” holding that the steering committee had “no right to discovery of irrelevant or privileged documents.” Id. at *3-4. Although the court suggested that Biomet identify seed set documents that had been produced, it held that it did not have the “authority to compel discovery of information not made discoverable by the Federal Rules.” Id.

Similarly, the court in Aurora Cooperative Elevator Co. v. Aventine Renewable Energy - Aurora West, No. 4:12CV230, 2015 U.S. Dist. LEXIS 177331 (D. Neb. Jan. 6, 2015), rejected the plaintiff's argument that the defendant should be required to produce irrelevant seed set documents in the interest of transparency, reasoning that the discovery rules “do not authorize ordering the defendants to disclose irrelevant information.” Id. at *6.

Other decisions have employed more hands-on approaches. In In re Broiler Chicken Antitrust Litigation, No. 1:16-cv-08637, Document #586 (N.D. Ill. Jan. 3, 2018), the special master issued an ESI order that required a party electing to use TAR to disclose, among other things, a general description of how the TAR process will work, including how the producing party will train the algorithm; a general description of the categories or sources of the documents included or excluded from the TAR process; and the quality control measures to be taken. The special master recognized that a requesting party “does not get to 'micromanag[e]'” the discovery process nor require disclosure of work product or other privileged information. Nonetheless, it provided the requesting party with a right to input into the TAR protocol.

Likewise, the special master in United States v. Education Management, No. 2:07-cv-00461, 2013 U.S. Dist. LEXIS 195974 (W.D. Pa. Nov. 24, 2013), contemplated interfering with the document review process. After producing more than six million pages, the defendants sought to use CAR to prioritize documents for manual review. The court found no reason to limit the defendants' use of the technology, but conditioned that logic on the unreasonable requirement that “no documents [were] actually excluded from review because of the [CAR] system.” Id. at *114. Furthermore, the court stated that “[g]ranting one party the authority to unilaterally determine the method through which CAR may be employed with the effect of excluding certain documents would run counter to the purposes of and need for cooperative discovery.” Id. at *114-15.

Finally, the court in Progressive Casualty Insurance Co. v. Delaney, No. 2:11-cv-00678-LRH-PAL, 2014 U.S. Dist. LEXIS 69166 (D. Nev. May 20, 2014), rejected Progressive's unilateral attempt to use predictive coding based on lack of cooperation and transparency. The court rejected the notion that Progressive should have “exclusive responsibility for training the predictive coding software.” Id. at *28-30.

In sum, to successfully employ TAR, counsel should be prepared to establish that their process was reasonable and defensible. Additionally counsel will need to demonstrate their willingness to cooperate and be transparent.

Fisher is a partner in the Health Sciences Department of Pepper Hamilton, resident in the Harrisburg, Pennsylvania, office. Hamilton is a partner in the firm's Health Sciences Department in Philadelphia. Adams is a discovery attorney with Pepper Hamilton in Harrisburg.