Column
Human-on-the-Loop Orchestration for AI-Assisted Legal Discovery
Deploying LLM agents autonomously in legal document review triggers a phenomenon called trajectory collapse, where early misclassifications propagate through multi-step reasoning chains and can invalidate privilege review. A four-layer verification system with Human-on-the-Loop escalation reduces privilege waiver risk by up to 61% compared to full autonomy, while keeping attorney review to under one quarter of total documents.
Hello. I’m Inoshita from the Affectosphere Group.
When deploying AI in a legal department, what is the risk you should fear most?
Not “the answer is wrong.” Rather: “a mistake cascades, and by the time anyone notices, an irreversible judgment has already been made.”
A study published on arXiv (Sinha et al., arXiv:2606.19812) identifies a failure mode called trajectory collapse that occurs when LLM agents are autonomously deployed in legal document review — and presents a four-layer verification system to prevent it.
The context is legal discovery, eDiscovery, and due diligence: domains defined by document volumes that strain human capacity and by mistakes whose consequences cannot be undone.
Three Key Points
- Autonomous LLM agents experience trajectory collapse — early misclassifications propagate through multi-step reasoning chains and can invalidate privilege review entirely.
- A Human-on-the-Loop design with four-layer verification covering planning through uncertainty quantification reduces privilege waiver risk by up to 61% compared to full autonomy.
- Attorney review is required for fewer than one quarter of total documents — an architecture that achieves both safety and cost efficiency.
What Is Trajectory Collapse?
Legal discovery — or eDiscovery — is the process of identifying and disclosing relevant documents in litigation or investigation. The task requires sorting through massive document sets to separate privileged communications (attorney-client exchanges, for instance), relevant documents, and irrelevant ones.
What happens when LLM agents handle this autonomously?
The researchers identify what they call trajectory collapse. When a misclassification occurs early in a reasoning chain, that error propagates through every subsequent step. If a privileged document is misclassified as non-privileged, all downstream reasoning builds on that flawed premise — and the entire privilege review can be rendered invalid.
This is fundamentally different from a simple answer error. A one-off mistake can be corrected. Trajectory collapse means the entire reasoning direction has drifted off course, and by the time the error is detected, the downstream state may already be irreversible.
In legal contexts, the damage materializes as privilege waiver — the inadvertent disclosure of information that should have remained protected, or the reverse. These are not errors you can correct after the fact by saying “the AI made a mistake.”
A Taxonomy of Agent Failures by Functional Stage
A second contribution of this research is a systematic taxonomy of agent failures organized by functional stage.
Planning, document retrieval, classification, reasoning — each stage has its own characteristic failure patterns. Mapping “what breaks where” makes it possible to determine precisely where verification checkpoints should be inserted.
Particularly important is the treatment of uncertainty quantification as a distinct stage. LLMs tend to generate answers even when their internal confidence is low. Without explicitly measuring and managing how certain an agent is, you end up with a system that proceeds with apparent confidence even when the underlying judgment is shaky.
This connects directly to legal risk. The documents that most require expert attention are precisely the gray-zone cases. If an agent cannot surface its own uncertainty, those gray-zone cases get processed without escalation.
Four-Layer Verification × Human-on-the-Loop
The solution the researchers propose combines four-layer verification — covering planning, execution, evaluation, and uncertainty quantification — with a Human-on-the-Loop (HotL) escalation design.
The distinction from Human-in-the-Loop (HitL) matters.
HitL means humans monitor and approve every step. That is safe, but it does not scale. In eDiscovery processing tens or hundreds of thousands of documents, requiring attorney review of every item defeats the purpose.
Human-on-the-Loop means human intervention is reserved for cases where it is actually needed. Agents process autonomously, but escalate to humans when uncertainty exceeds a threshold or when specific risk signals emerge.
The simulation results show that compared to full autonomy, this HotL design reduces privilege waiver risk by up to 61%.
And attorney review is required for fewer than one quarter of all documents.
That number is significant. In eDiscovery, the dominant cost driver is expert time. Reducing the review population to under a quarter of the total translates directly to cost compression — while simultaneously improving risk management. The architecture achieves both goals in the same design.
How to Pilot This in Your Organization
How might the insights from this research translate into concrete use cases for legal teams, eDiscovery vendors, or due diligence workflows?
The most accessible entry point is using LLM agents as a first-pass screening layer in M&A due diligence.
Target contracts, internal policies, and key communications are submitted to an agent pipeline that applies initial classification and flags documents for human review. Attorneys and analysts focus exclusively on the flagged subset.
Two KPIs worth tracking from the start: attorney review volume reduction rate (measuring cost efficiency) and privileged document misclassification count (measuring risk management effectiveness).
Drawing on the four-layer verification framework this research presents, the most important design principle for that pilot is building uncertainty score thresholds directly into the escalation logic. Cases where the agent signals low confidence should be automatically routed to humans. That routing needs to be designed in from the beginning — retrofitting it after the fact is far harder.
Contract review is another natural application. For high-volume, structurally uniform contract sets — NDAs, service agreements, SLAs — an agent can perform anomaly detection on specific clause types (penalty clauses outside standard ranges, unusual termination provisions) and escalate only the high-anomaly instances to reviewers.
Human-on-the-Loop as a Design Philosophy
The most important insight this research offers is not a technical solution — it is a design philosophy.
Full autonomy versus full manual review: Human-on-the-Loop is the third path beyond that binary.
AI handles the majority of processing; humans intervene only at moments of high uncertainty or high-stakes judgment. This architectural pattern is not specific to legal work. Medical decision support, credit underwriting, compliance screening — all of these are domains with the same underlying structure: high-volume routine judgments combined with a smaller set of non-routine, high-risk decisions.
The starting point for HotL design is always the same question: where exactly is human judgment genuinely necessary? Answering that question requires agents that can report their own uncertainty honestly. An architecture that prevents agents from presenting “I don’t know” as “I know” — that is the core challenge of next-generation AI workflow design.
Reference
- Sinha, A., Ranganathan, S., Dharmaratnakar, A., & Das, D. (2026). Human-on-the-Loop Orchestration for AI-Assisted Legal Discovery. arXiv preprint arXiv:2606.19812.
* This article was written in part with AI assistance and may contain inaccuracies.