Mustafa Awad
The 20% Problem: Why Manual Document Review Misses Critical Facts
Your last document review missed roughly four out of every five relevant facts in the case file. That's not a critique of your team. It's what the empirical literature on legal document review has been finding since 1985.
Before this gets dismissed as another BigLaw discovery piece, it isn't. The 20% problem hits 12-attorney boutiques and 40-attorney mid-market firms harder than it hits a 200-attorney litigation department. A BigLaw discovery group has a defensibility problem. A boutique handling a 50,000-page case file has a strategy problem, because missed facts translate directly into missed depositions and weaker settlement leverage. If you're the partner or associate actually running the matter, this is for you.
The 1985 recall study that still defines legal document review
One empirical study has shaped legal document review for 40 years. Blair and Maron studied a real litigation-support corpus: just under 40,000 documents across roughly 350,000 pages. Lawyers and trained paralegals used a state-of-the-art full-text retrieval system to answer 40 information requests.
Before evaluation, the lawyers estimated they had found about 75% of the relevant material.
The measured number was 20.26%.
The distinction is precision versus recall. Precision measured the documents retrieved: about 79% of what the system returned was relevant. Recall measured the documents missed: the search found only 20.26% of all relevant material in the corpus. So the visible results looked accurate, while roughly four out of five relevant documents remained hidden.
This became the founding data point of modern legal information retrieval. The exact percentage is not a universal law, but the lesson has survived four decades of research: confident search is not the same as measured recall.
Forty years of replication: from Blair-Maron to TAR
Blair and Maron's number could have been a 1985 artifact. It wasn't.
When the National Institute of Standards and Technology built the TREC Legal Track benchmark between 2006 and 2011 on a 6.9-million-document tobacco litigation corpus, an expert manual searcher found an additional 11% of known relevant documents beyond a carefully negotiated Boolean run. A meaningful residue was found by neither.
A 2010 study by Roitblat, Kershaw, and Oot re-reviewed a real Department of Justice "Second Request" matter originally handled by 225 attorneys. Two human re-review teams scored F1 values of 0.280 and 0.273. Two automated systems scored 0.341 and 0.378. A signal-detection analysis on the same data found that human reviewers were no better than the machines at distinguishing responsive from non-responsive documents.
In 2011, Grossman and Cormack used raw TREC 2009 data to compare technology-assisted review against the official exhaustive manual review. TAR outperformed manual review on recall, precision, and F1, while reviewing a fraction of the documents.
The defensible reading is not that humans are bad at reviewing documents. It is that any review process operating without measurement and validation is epistemically fragile, regardless of how many lawyers are staffed on it. Manual review feels thorough. The math says it isn't.
The cost dimension nobody at a 30-attorney firm can afford to ignore
The RAND Corporation’s 2012 monograph Where the Money Goes studied e-discovery spend across eight large companies and 57 cases. Two findings should sit on the wall of every partner running litigation matters: review consumed 73% of production costs, and outside counsel accounted for 70% of total e-discovery spend.
For mid-market litigators, the lesson is direct: the largest review cost is often tied to the least measured part of the matter. Blair and Maron showed why that matters. Lawyer-directed search can look precise while still missing most of the relevant material.
For a BigLaw discovery group, that 73% may be a procurement question. For a 30-attorney firm, it is a strategy question. You do not have a litigation support department to absorb inefficient review. Every hour spent grinding through documents is an hour not spent on case theory, deposition prep, expert work, or settlement leverage.
A larger firm can throw more people at the recall problem. A boutique cannot. If you are in the second category, you may have the strongest economic case for changing how review actually gets done.
Why your current document review workflow produces 20% recall
Consider the standard mid-size or boutique workflow:
One associate, plus a paralegal, plus a senior partner doing spot checks
Keyword searches generated from experience, prior matters and gut instinct
A document-by-document linear pass through whatever the keywords surface
Margin notes in a binder or a shared spreadsheet
The senior partner trusts the team's read; the team trusts the senior partner's instincts
This is the workflow that produces 20% recall. Not because anyone is doing anything wrong, but because the math of unstructured search across a heterogeneous case file does not bend to effort. Reading harder does not surface what keywords never returned.
From search confidence to measured defensibility: the legal document review shift
The defensibility question used to be procedural. Did you run reasonable keywords? Did you produce in good faith?
That bar is moving. Courts are increasingly receptive to TAR. The Sedona Conference standardized the vocabulary. The Grossman-Cormack Glossary gave judges stable terminology for predictive coding, gold standards, and precision-recall trade-offs. Rule 26 amendments emphasize proportionality and early ESI cooperation. The trend is clear. Defensibility now depends on what you can measure, not what you can attest to.
The important shift is not that AI replaces lawyer review. It is that human effort, standing alone, does not establish completeness or reliability. The empirical literature reviewed here supports documented, sampled, human-in-the-loop processes that can be tested for recall, precision, error, and proportionality. Measurement does not replace legal judgment; it makes the review process more capable of being explained and defended.
How Newcase.ai closes the 20% recall gap
Newcase.ai is an AI litigation intelligence platform purpose-built for litigation teams that need to find the material facts buried across depositions, exhibits, and case records without expanding review teams at the same rate as document volume. Its core capability, Never Miss a Fact, is built around the same problem Blair and Maron identified in 1985: confidence in search is not the same as measured recall.
Newcase processes the case file into a unified, searchable litigation intelligence layer. Extracted facts are linked to source citations, including page-and-line references where available, so lawyers can verify the output against the record rather than relying on unsupported AI summaries. The point is not that a keyword set happened to surface something. The point is that the review process becomes visible, testable, and source-grounded.
Newcase reports validation across 100,000+ manually reviewed pages and positions the platform around high-recall fact extraction, citation verification, and human-in-the-loop review. That is the defensibility lesson from the empirical literature: machine assistance is strongest when paired with lawyer judgment, sampling, adjudication, and documentation. The litigator remains responsible for consequential judgments; the system handles scale, ranking, and fact surfacing.
FAQ: AI for legal document review
How accurate is AI for legal document review?
Comparison is the honest answer. Blair and Maron (1985) showed that lawyer-directed full-text search recovered only about 20% of relevant material despite high precision. Roitblat, Kershaw, and Oot (2010) found automated systems outscored two human re-review teams on F1 in a real DOJ matter. Grossman and Cormack (2011) found carefully designed TAR workflows could outperform exhaustive manual review on recall, precision, and F1.
Newcase benchmarks Newcase against 100,000+ manually reviewed pages and reports 100% extraction of key facts with citation-verifiable, zero-hallucination standards. The lesson is not that AI is accurate in the abstract. It is that legal-review accuracy depends on measurable, source-linked, human-validated workflows.
How long does it take to summarize a 300-page deposition?
Twenty-five seconds with Newcase deposition summaries, with page-and-line citations on every extracted fact. A manual summary of the same deposition typically takes 6 to 8 hours of associate time, and the output is rarely citation-verifiable in the way trial work requires. The 15× faster figure published across Newcase deployments is the conservative aggregate, and depositions specifically run faster than that.
Does Newcase replace human legal judgment?
No. Newcase is built on the principle that human + AI outperforms either alone, which is the exact finding the empirical literature has been demonstrating for decades. The platform handles extraction, cross-referencing, and surfacing. You make every consequential decision. Newcase's workflow design matches what the research recommends.
Newcase is the AI Litigation Intelligence platform that connects depositions, attorney strategy, expert testimony, and case facts into a single searchable intelligence layer.
