As part of fulfilling their discovery obligations, parties may employ technology-assisted review, or “TAR,” in the document review process. TAR is a general term covering the use of analytics and other advanced technologies to add automation and efficiency to document review, such as prioritizing documents for review or reducing the total review population.

TAR is, by definition, a mixture of technology and process used as part of document review. It is document review assisted by technology, not necessarily an “easy button” for discovery. It often can make the discovery process faster, less expensive and possibly more accurate, but at times it can fall short at one or all of these objectives. This is especially true in situations where the process part of TAR may be reasonably called into question, as occurred in a recent decision where a party's motion to extend discovery deadlines was granted by a receptive court.

Domestic Airline Litigation

In In re Domestic Airline Travel Antitrust Litigation, 2018 WL 4441507 (D.D.C. Sept. 13, 2018), a multidistrict class action litigation, the court had set “a strict schedule for discovery” and “exhorted the parties to comply with the deadlines therein[.]” Id. at *1. On Aug. 24, 2018, however, the plaintiffs filed to extend the fact discovery deadline by six months due to issues they noted in the April 30, 2018 document production of defendant United Airlines, which contained approximately 3.5 million documents. Defendants Delta Air Lines and United both opposed the extension. Id. at *2.

The plaintiffs asserted that following United's use of TAR, only about 17 percent, or 600,000, of the 3.5 million documents in the production were responsive to their document requests. The plaintiffs requested more time because of the need to review all of the documents to determine which were actually responsive. Id.

The plaintiffs and United had entered into a TAR agreement, also referred to as a validation protocol, “to ensure accuracy and completeness.” Id. at *3. This protocol set forth expectations relating to the measures of precision and recall—precision being the percentage of documents produced that are actually responsive and recall being the percentage of responsive documents in the entire document collection that are present in the document production. For example, if a document collection is reviewed using TAR and 100 documents are produced as responsive, but a closer review indicates that only 85 of the 100 are truly responsive, then the production's precision would be 85/100 or 85 percent. If it is also determined (for example, through statistically valid sampling) that the balance of the document collection of not produced documents contains 40 additional responsive documents (meaning that there were, in actuality, 125 responsive documents in the collection), then the recall in the production set would be measured at 85/125, or 68 percent.

In the protocol, United agreed to “set a minimum estimated recall rate of 75% but will endeavor to achieve a higher estimated recall rate if that rate may be obtained with a reasonable level of precision through reasonable additional training effort[.]” Id. at 4. To allow measurement of recall and precision, United agreed to “engage in validation testing by reviewing a statistically representative sample of documents to test the accuracy of TAR as to the responsiveness of the documents[.]” Id.

Just prior to producing to the plaintiffs, United provided the results of their review of a “control set”—a random sample of the documents categorized by TAR—indicating an estimated recall of 85 percent and precision of 58 percent. Along with this information, United provided metrics of separate validation sampling they had conducted of the actual production set. Analyzing these metrics, the plaintiffs “found that the statistics from the validation sample indicated that the TAR process resulted in a recall of 97.4% and precision of 16.7%,” in contrast to the lower recall and much higher precision indicated by United's control set. Id. After exchanges between parties to understand the discrepancy, United stated in late July 2018 that it had incorrectly reported the control set metrics and that in reality they were aligned with the metrics from the validation sample. Id.

The parties attempted to confer to resolve the problem, but as the court notes, “the answer seems to be that unless United starts the process over, Plaintiffs must review all the documents.” Id. As such, the plaintiffs asked for an extension of discovery so that they might “segregate the large number of nonresponsive documents from the responsive documents … to prepare for depositions, motions practice, and trial.” Id.

In its analysis, among other factors, the court looked to determine the plaintiffs' level of diligence with respect to discovery. The court generally accepted the narratives and arguments offered by the plaintiffs, including that they employed 70 document review attorneys as soon as the production was available and that they “'could not have foreseen United's voluminous document production made up [of] predominantly non-responsive documents resulting from its deficient TAR process[.]'“ Id. at *3. The court set aside some counter-arguments raised by the defendants criticizing the plaintiffs' review processes, stating that such contentions failed to address the issue of whether United's production led to plaintiffs having to deal with “unforeseen or unanticipated matters, which justify Plaintiffs' request for additional time.” Id. The court additionally rejected United's argument that the plaintiffs got what they wanted—a high level of recall without regard for precision, observing that the language of the protocol itself “notes that 'a reasonable level of precision' was a concern.” Id. at *5. United also attempted to argue that the precision level of less than 17 percent was reasonable, but the court refrained from commenting on this, finding such an argument irrelevant since the issue at hand was whether the production “containing numerous nonresponsive documents” was unanticipated by the plaintiffs. Id.

After determining that the factor of the plaintiffs' diligence was met and analyzing other factors, the court granted the plaintiffs' motion to extend discovery.

Conclusion

In re Domestic Airline Travel Antitrust Litigation provides a number of lessons for practitioners who rely on TAR. First, while TAR can be a useful tool for litigants, it is not a panacea for document review. Second, while TAR protocols and other discovery agreements can be very helpful, parties should carefully review them and ensure that they are able to demonstrate compliance, as courts will rely on them in resolving future related issues. And third, as demonstrated in this decision, parties should endeavor to ensure that TAR efforts are reasonable, and appropriately mix process with technology, or they risk rework and motion practice that could outweigh any savings.

Christopher Boehning and Daniel J. Toal are litigation partners at Paul, Weiss, Rifkind, Wharton & Garrison. Ross M. Gotler, e-discovery counsel, and Adam Strayer, e-discovery analysis & review technology manager, assisted in the preparation of this article.