Editor's note: This is the second in a two-part series.

Last week, I discussed Winfield v. New York, 15-CV-05236 (S.D.N.Y. Nov. 27, 2017), where Magistrate Judge Katharine H. Parker ruled on the plaintiffs' claims that the defendant did not properly produce e-discovery. This week, I'll discuss the court's response and other current issues.

Current Discovery Issues

Plaintiffs presented the court with “three interrelated issues.” The plaintiffs objected to the city's TAR training and review process, claiming that the city searched the DCP/Banks review population with inappropriate search terms, that it searched that review population for “documents … responsive to only a subset of the plaintiffs' document requests,” and that it “over-designated documents as nonresponsive when in fact they were responsive” to the plaintiffs' document requests. To “increase transparency and confidence in the city's document review,” the plaintiffs proposed that the city provide for review random samples of each of the following, with size and methodology to be negotiated by counsel and their e-discovery experts or determined by the court in the absence of agreement: nonprivileged DCP/Banks documents and HPD/mayor's office documents that were subject to TAR that were above, or just below, the predictive-coding ranking cutoff used by the city, but where the city ultimately determined that the documents were nonresponsive. To explore how the city used TAR, the plaintiffs also sought information about the ranking system used (i.e., what cutoff was used, and how many documents were deemed responsive and unresponsive at each ranking); all materials submitted by the city for the court's in camera review relating to predictive coding, other than materials that are in fact privileged; an order requiring the city to search for documents responsive to all of the plaintiffs' document requests other than those specifically withdrawn by the plaintiffs or limited or stricken by the court; and, an order requiring the city to review of the additional search terms for the DCP/Banks review population document review process immediately (as opposed to waiting until after the settlement conference as the city proposes) and turning first to documents from Banks (so that some subset of his documents can be produced before his Nov. 29 deposition). The city, of course, moved the court to decline to consider the plaintiffs' requests.

The Court's Response

The court's response was lengthy and considered. What makes it so interesting (at least to me) is that, to use a hyperbolic adjective, the response is “boring.” I do not mean it will put you to sleep. What I do mean is that it treats all of the issues reviewed above as regular discovery issues, as opposed to “special” e-discovery issues whose explanation requires a vocabulary no one understands and carries with it an unstated assumption that the rules of discovery do not really apply to e-discovery and so will have to be twisted to get any answers to the legal questions at hand.

The court begins with a brief discussion of Federal Rules of Civil Procedure 34 and 26. Rules 34 and 26 provide that a party may serve on any other party a request to produce documents and ESI within the party's possession, custody or control so long as the documents and information are “relevant to any party's claim or defense and proportional to the needs of the case, considering the importance of the issues at stake in the action, the amount in controversy, the parties' relative access to relevant information, the parties' resources, the importance of the discovery in resolving the issues, and whether the burden or expense of the proposed discovery outweighs its likely benefit.” Rule 26 “also makes clear that discovery is not limited to information that will be admissible at trial.” Under Rule 34, if the producing party objects to the request, it must “state with specificity” the grounds of objection. Counsel representing the producing party must, under Rule 26(g), certify that productions in response to e-discovery requests are, to the best of counsel's “knowledge, information, and belief formed after a reasonable inquiry … complete and correct as of the time it is made,” and that any objection or response is “not interposed for any improper purpose,” including delay. The court treats e-discovery no differently than it does all other discovery.

Search Terms for the DCP/Banks Review

The court first addressed the 665 additional search terms proposed by the plaintiffs for the DCP and Banks reviews. The court cited William A. Gross Construction v. American Manufacturers Mutual Insurance, 256 F.R.D. 134, 136 (S.D.N.Y. 2009), for the principal that courts “have recognized that keyword searches used to search for and collect documents for review must be carefully crafted.” While the city agreed to perform the supplemental search, requiring “review of 90,000 additional documents at a cost of approximately $248,000,” the plaintiffs objected to the city's proposal because they believed that the city's TAR processes resulted “in the over-designation of documents as nonresponsive.” The court found “no evidence of gross negligence or unreasonableness in the city's TAR training or review processes” and so denied the plaintiffs' objections and ordered the city to conduct the supplemental review as described.

The plaintiffs objected to the city's continued use of its TAR system because they contended that the system improperly trained reviewers. Their evidence for such contention was that “the city's human document reviewers over-designated documents as nonresponsive during both the linear review and during the TAR training stages.” Discussing the objection, the court noted that other courts, as well as the Sedona Conference, have found that the producing party was in the best position to “evaluate the procedures, methodologies, and technologies appropriate for preserving and producing their own electronically stored information.” As well, the court observed that courts “have not micro-managed parties' internal review processes” for several reasons: counsel for the producing parties were expected to comply with Rules 26 and 34 in connection with their search, collection, review and production of documents; review of internal attorney ESI work processes could “reveal work product, litigation tactics and trial strategy;” the producing party was “better equipped than the court to identify and utilize the best process for producing their own ESI consistent with their obligations under the Federal Rules of Civil Procedure; and, “perfection in ESI discovery” was not required; rather, a producing party need only “take reasonable steps to identify and produce relevant documents.”

The court further observed that courts were “split as to the degree of transparency required by the producing party as to its predictive coding process.” In the instant matter, the court did not require the city to instruct the plaintiffs as to its predictive coding process and training, it did require the city “to provide in camera submissions addressing these subjects,” which included a seed set—made up of over 7,200 “randomly selected documents, as well as pre-coded example documents such as the pleadings, the plaintiffs' document requests, and other relevant, privileged and nonprivileged documents”—which was reviewed and marked (responsive or nonresponsive) by the city's document review in order to train the system.

The TAR training phase included five full training rounds, at the conclusion of which the city conducted a validation process. Therein, the city provided the review team with “detailed training … as to the issues in the case” as well as with “all of the plaintiffs' document requests to use in connection with their review and designation of documents as responsive and nonresponsive.” The court found that the city's training and review processes and protocols provided nothing close to a basis “for finding that the city engaged in gross negligence in connection with its ESI discovery.”

Turning to the plaintiffs' objections that a consequence of the city's use of TAR was that it “over-designated documents as nonresponsive,” the court noted that the objections relied upon a few documents “that were inadvertently produced by the city” that it contended were responsive to discovery requests and should have been produced, as well as to a few documents which the city had designated as nonresponsive but which, upon challenged, conceded that they were “arguably responsive.” The plaintiffs also challenged the city's refusal to produce documents withheld as privileged. The court noted that the city produced in camera a sample of 80 of these documents, after which the city withdrew its claim of “deliberative process privilege as to 59” of them and then changed its responsiveness designation to nonresponsive for 15 documents. The court reviewed those 15 documents and determined that nearly all of them were “responsive to the plaintiffs' document requests to some degree, though most” were “only of marginal relevance.”

While, then, the court agreed with the plaintiffs' assessment of the facts with regard to the above-discussed objection, it did not agree with plaintiffs' conclusion. The court found that labeling five documents out of 20 incorrectly as “no-responsive” in the context of “the 100,000 documents that have been reviewed thus far in this case” was not “sufficient to question the accuracy and reliability of the city's TAR process as a whole. Citing Da Silva Moore v. Publicis Groupe, the court noted that in any ESI review, “'the Federal Rules of Civil Procedure do not require perfection,'” Instead, the court reasoned, “the proper inquiry” was (citing Hyles v. New York City, No. 10-cv-3119 (AT) (AJP), 2016 WL 4077114, at *3 (S.D.N.Y. Aug. 1, 2016)), whether the “search results” were “reasonable and proportional.” In the instant matter, the plaintiffs failed to identify “anything in the TAR process itself” that was “inherently defective,” instead pointing to “human error in … a small subset of documents” categorized as “responsive or nonresponsive.” Given that the city “produced over 12,500 documents that were all designated as responsive,” such errors were negligible. Moreover, given that the seed set was comprised of over 7,000 documents, the court found that “the misdesignations identified by the plaintiffs would” not have affected the city's TAR processes “in any meaningful way.” Its finding was confirmed by the city's validation process, which the court reviewed in the city's in camera submission and found proper.

The court nevertheless granted the plaintiffs' request for sample sets of nonprivileged documents. The court reasoned that the sample sets would “increase transparency,” a request that was “not unreasonable in light of the volume of documents collected from the custodians, the low responsiveness rate of documents pulled for review by the TAR software, and the examples that plaintiffs” presented, which suggested that “some human error in categorization” may have “led to gaps in the city's production.” The court provided instructions for what sample sets should be produced, and then instructed that if the plaintiffs contend that the sampling suggests that responsive documents “within the HPD/mayor's office or DCP/Banks review populations” are responsive and relevant “but have not been produced,” the parties would meet and confer “to determine whether additional training and review” was necessary, but “with the understanding that reasonableness and proportionality, not perfection and scorched-earth,” had to be “their guiding principles.”

Common Sense in Applying Remedies

The court evinced common sense not just in its substantive rulings but in how it expected its commands to be followed. The court recognized that the handful of days between the issuance of its opinion and the scheduled deposition of Banks. It instructed the parties to conduct the deposition and reserve and object where the plaintiffs believed it needed the results of the city's second look at its discovery production, so that the court could fashion a remedy (e.g., additional deposition based upon the results of the second look) should one be needed.

Finally, the Court denied the plaintiffs' “request for information about the ranking system used by the plaintiffs (i.e., what cutoff was used, and how many documents deemed responsive and unresponsive are at each ranking) and for all materials submitted by defendant in camera relating to predictive coding.” The court reasoned that the plaintiffs failed to explain why information about the city's ranking system was needed, particularly since the plaintiffs' request to sample nonresponsive documents from the review populations was granted. The court further observed that it was “also unclear how this information” was “even potentially relevant to the claims and defenses in this litigation, as required under Federal Rule of Civil Procedure 26.” Nevertheless, to promote “transparency and cooperation in the discovery process,” the court encouraged the city “to share such information with the plaintiffs.”

Common Sense Is Not That Common

Winfield dealt with the same e-discovery issues so many other courts have addressed and puzzled over—how the judge the effectiveness of TAR or search terms, how much of such reviews should be open to judicial or public scrutiny, and so on. What makes the opinion so refreshing is that it does not treat these issues as if they were strange and unusual. The principal reason for that is that the opinion does not go on and on explaining Technology Assisted Review, nor is there any reason to in a world when you can talk to your television and other things in the Internet of Things—users are comfortable with such technology and have a basic understanding of how it works, so the opinion can concentrate on how to look at such technology by applying the rules and the basic concepts of fairness to come to a just and understandable result. Transparency is important, albeit not controlling. Evidence that TAR may not have uncovered all responsive files is a reason to tweak protocols but not to reject the process.

Finally, and fundamentally, the issue is the legal one of whether the parties have been treated justly, and not the philosophical one of whether technology is good or bad. If such factors make opinions regarding e-discovery “boring,” I for one am quite willing to say goodbye to “interesting” opinions and hello to ones in which e-discovery is no longer controversial and the law can be interpreted to come up with fair holdings based upon just, transparent reasoning.

Leonard Deutchman is vice president, Legal for KrolLDiscovery, which he helped build into the largest e-discovery provider in the United States. Before joining KrolLDiscovery, he was a chief assistant district attorney at the Philadelphia District Attorney's Office, where he founded the Cyber Crime Unit and conducted and oversaw hundreds of long-term investigations involving cybercrime, fraud, drug trafficking and other offenses.