Is human-in-the-loop slower than fully autonomous agents?

Per-task, yes, by however long the checkpoint takes. System-wide, usually no. A loop that ships bad output autonomously spends more time on rollback, re-prompting, and cleanup than it saved by skipping review. The right comparison isn't "with a human" vs "without": it's "review now, cheaply" vs "review later, expensively, after the bug is in production."

Can I start on the loop and move in if things go wrong?

That's backwards from how it should work operationally, but it's how most teams actually adopt it, and it's fine as a starting point if you're watching closely at first. Better: start in the loop while the agent is new to a codebase or task type, then graduate to on the loop once you've seen enough successful iterations to trust the pattern.

What's the difference between human-in-the-loop and human-on-the-loop?

In the loop means the agent stops and waits for your input before proceeding: you're a required step in the sequence. On the loop means the agent runs to completion on its own and you review the output afterward, on your own schedule, with the ability to intervene if something's wrong. Same human, same oversight, different position relative to the agent's execution.

Does the orchestrator have to be code, or can it be a person deciding manually?

It can be either. Some teams hardcode the routing rules (file path patterns, diff size, test coverage) into their CI or agent harness. Others just have a rule of thumb a senior engineer applies when triaging a queue of agent output. The logic is the same either way: the difference is whether it's automated or applied by hand.

How do confidence thresholds actually get set in practice?

Most teams start conservative (review everything) and loosen the threshold as they build a track record. A common pattern: track how often a given task category needed a correction after review, and once that rate drops below some bar (say, under 5% over 20 iterations) for a category, move it from in-the-loop to on-the-loop. The threshold is empirical, not theoretical.

Where does Patchrooms fit into an in-the-loop vs on-the-loop setup?

Patchrooms is the checkpoint itself. It's the widget on your staging build where a reviewer clicks the broken thing and leaves a comment, and the export that turns that click into a structured prompt block the agent reads back. Whether you're in the loop on every commit or on the loop reviewing a batch, Patchrooms is what makes that review cheap enough to actually keep doing.

Blog

Human in the Loop vs On the Loop: Where to Put Your Attention

July 11, 2026 8 min read

Every agent loop eventually asks the same question: where does the human stand? In the loop, blocking each step until you say go, or on the loop, watching the output roll by and stepping in only when something's wrong. Get this placement wrong and you either bottleneck a system that should run itself, or you ship autonomous output nobody checked until a customer found the bug. The right answer isn't one or the other. It's a routing decision made per task, and it's usually made better by your orchestrator than by you scrolling a diff at 11pm.

Two positions, same human

"Human in the loop" means the agent stops and waits. It generates a plan, a diff, a draft, then pauses until you approve, reject, or correct it. Nothing downstream happens until you act. You're a required step in the sequence, and the loop's throughput is capped by how fast you respond.

"Human on the loop" means the agent runs to completion without waiting for you. It plans, executes, and ships, and you review the result afterward, on your own schedule, with the standing ability to intervene, roll back, or redirect the next iteration. You're supervising the system, not gating each step of it.

Both are legitimate oversight models. Neither is inherently safer or faster: it depends entirely on what the task can afford to get wrong, and for how long.

When you need to be in the loop

Put a human in the loop when a mistake is expensive to undo, expensive to leave live, or both. Three signals push a task toward in-the-loop review:

Irreversible actions: anything that can't be trivially rolled back. A database migration, a production deploy that touches billing, a message sent to a real customer, an email drafted and auto-sent. If "undo" means a support ticket instead of a git revert, stop and check first.

Customer-facing surfaces: copy, pricing, UI states a real user will see before you do. An agent hallucinating a wrong price or a broken checkout flow is a different order of problem than an agent breaking a staging build nobody's looking at yet.

High blast radius: changes to shared infrastructure, auth, permissions, anything where one bad output doesn't stay contained to the task it came from. If a wrong answer here cascades into other systems, the cost of checking upfront is small next to the cost of catching it downstream.

The common thread isn't "is the agent likely to be wrong." Good agents are usually right. It's "what does it cost me if this specific output is wrong and nobody caught it before it landed."

When on the loop is enough

Put a human on the loop when the task is reversible, contained, and cheap to inspect after the fact. A component refactor on a feature branch nobody's shipped yet. A copy pass across internal docs. A batch of UI tweaks on a preview deploy that hasn't touched main. In these cases, blocking the agent on your approval doesn't buy you much safety: it just adds latency, and latency compounds. A loop that stops for approval on every commit isn't a loop anymore, it's a queue with a robot typing into it.

The test is simple: if you can review the output after it's done and the cost of a wrong output surviving until your next review pass is low, don't gate the step. Let it run, batch the review, and spend your attention on judgment instead of babysitting.

The orchestrator decides, not the model

This is the part teams get wrong first: the decision of whether a given task needs a human in the loop shouldn't be made by the model doing the task. A model asked "should I check with a human before doing this" will either always say yes (useless: you're back to blocking on everything) or confidently say no on exactly the task where it was wrong (worse: you lose the checkpoint precisely when you needed it).

The routing decision belongs to the orchestrator: the layer above the model that knows what kind of task this is before the model starts working on it. That's a rule, not a judgment call made mid-generation: touches `/billing`? Require review. Touches a component with no tests? Require review. Isolated CSS tweak on a branch with a green test suite? Ship it, review on the loop. The orchestrator applies this rule before dispatch, so the model never has to decide how much scrutiny its own output deserves, which is a decision it's structurally bad at making about itself.

Confidence thresholds are the routing logic

In practice, "orchestrator decides" cashes out as a threshold: route to a human reviewer when confidence is below a bar, let it run unattended when confidence is above it. Confidence here isn't just the model's self-reported certainty: that's noisy and often miscalibrated. Better signals are structural: has this exact task type succeeded N times in a row without correction? Does the diff touch files with existing test coverage? Is the change size within the range that's historically needed no fixup?

This is why the threshold moves over time. Early on, when an agent is new to a codebase, everything routes to a reviewer: the threshold is strict because you have no track record yet. As a task category racks up clean iterations, the threshold relaxes and more of that category moves to on-the-loop. The threshold isn't a constant you set once. It's a dial you tune per task category as evidence accumulates.

The checkpoint that lets you move from in to on

There's a reason most teams default to in-the-loop longer than they need to: on-the-loop review only works if reviewing after the fact is actually cheap. If checking an agent's output means spinning up the branch, running the app, and hunting for what changed, you'll keep gating everything up front just to avoid that overhead later, even on tasks that don't need it.

The fix isn't a better threshold. It's a lower-friction checkpoint. If review takes one click on a live preview instead of a local setup, the cost of moving a task from in-the-loop to on-the-loop drops to near zero, and the threshold can relax faster because the fallback (catching it after the fact) is no longer expensive. Cheap review is what makes on-the-loop trustworthy instead of just optimistic.

Where Patchrooms fits

Patchrooms is that low-friction checkpoint. Embed one script tag on the preview, staging, or artifact your agent just produced, and reviewing it stops requiring a local setup: click the element, leave a comment or a voice note, done. No account for the reviewer, free and unlimited.

That matters for both positions in the loop. In-the-loop tasks get a fast approve/reject surface instead of a slow one, so gating a step doesn't cost you the latency it used to. On-the-loop tasks get a review queue that's actually cheap to clear, so batching review after the agent runs stays honest instead of turning into a pile you never get to.

Either way, the output isn't just a comment a person reads. "Copy agent feedback" turns the click into a Markdown prompt block (selector, goal, constraints, screenshot reference) that pastes straight into Claude Code, Cursor, or whatever's driving the next iteration. The MCP server lets the agent pull reports directly (`list_reports`, `get_report` with inline screenshots) and close them once fixed (`set_status`). Whichever side of the loop you're standing on, the checkpoint feeds back into the system as an instruction, not just a note.

FAQ

Is human-in-the-loop slower than fully autonomous agents?: Per-task, yes, by however long the checkpoint takes. System-wide, usually no. A loop that ships bad output autonomously spends more time on rollback, re-prompting, and cleanup than it saved by skipping review. The right comparison isn't "with a human" vs "without": it's "review now, cheaply" vs "review later, expensively, after the bug is in production."
Can I start on the loop and move in if things go wrong?: That's backwards from how it should work operationally, but it's how most teams actually adopt it, and it's fine as a starting point if you're watching closely at first. Better: start in the loop while the agent is new to a codebase or task type, then graduate to on the loop once you've seen enough successful iterations to trust the pattern.
What's the difference between human-in-the-loop and human-on-the-loop?: In the loop means the agent stops and waits for your input before proceeding: you're a required step in the sequence. On the loop means the agent runs to completion on its own and you review the output afterward, on your own schedule, with the ability to intervene if something's wrong. Same human, same oversight, different position relative to the agent's execution.
Does the orchestrator have to be code, or can it be a person deciding manually?: It can be either. Some teams hardcode the routing rules (file path patterns, diff size, test coverage) into their CI or agent harness. Others just have a rule of thumb a senior engineer applies when triaging a queue of agent output. The logic is the same either way: the difference is whether it's automated or applied by hand.
How do confidence thresholds actually get set in practice?: Most teams start conservative (review everything) and loosen the threshold as they build a track record. A common pattern: track how often a given task category needed a correction after review, and once that rate drops below some bar (say, under 5% over 20 iterations) for a category, move it from in-the-loop to on-the-loop. The threshold is empirical, not theoretical.
Where does Patchrooms fit into an in-the-loop vs on-the-loop setup?: Patchrooms is the checkpoint itself. It's the widget on your staging build where a reviewer clicks the broken thing and leaves a comment, and the export that turns that click into a structured prompt block the agent reads back. Whether you're in the loop on every commit or on the loop reviewing a batch, Patchrooms is what makes that review cheap enough to actually keep doing.