Building AI Agents That Know When to Ask for Help

Your AI agent just deleted the wrong database table. It was 95% confident it was right.

This scenario repeats across organizations because we've optimized AI systems for autonomy without building in humility. A truly reliable agent doesn't maximize task completion—it maximizes task completion correctly, which sometimes means stopping and asking for help.

The Confidence Problem

Large language models are notoriously overconfident. They generate plausible-sounding answers even when uncertain. In production systems, this isn't quirky—it's dangerous. An agent executing database migrations, managing infrastructure, or making financial decisions needs to recognize the boundaries of what it can safely handle alone.

The fix isn't perfect reasoning (impossible) or removing autonomy (defeats the purpose). It's building explicit uncertainty detection into your agent's decision loop.

Implementing Uncertainty Detection

Confidence Thresholds with Structured Output

Force your model to return confidence scores alongside decisions. Use structured output to make this non-negotiable:

pythonCopy
from pydantic import BaseModel
from typing import Literal

class AgentDecision(BaseModel):
    action: Literal["execute", "escalate", "gather_info"]
    confidence: float  # 0.0 to 1.0
    reasoning: str
    risk_level: Literal["low", "medium", "high"]
    escalation_reason: str | None = None

def evaluate_action(task: str, context: dict) -> AgentDecision:
    # Your LLM call with structured output
    response = client.beta.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1024,
        messages=[{"role": "user", "content": task}],
        response_model=AgentDecision
    )
    return response

# In your agent loop
decision = evaluate_action(task, context)
if decision.confidence < 0.75 or decision.risk_level == "high":
    escalate_to_human(task, decision)
else:
    execute_action(decision)

Multi-Signal Escalation Logic

Confidence alone isn't enough. Build escalation on multiple signals:

typescriptCopy
interface EscalationSignal {
  type: 'low_confidence' | 'anomaly' | 'irreversible_action' | 'missing_data';
  severity: number;
  context: string;
}

function shouldEscalate(signals: EscalationSignal[]): boolean {
  const severitySum = signals.reduce((sum, s) => sum + s.severity, 0);
  const hasIrreversible = signals.some(s => s.type === 'irreversible_action');
  const hasMissingData = signals.some(s => s.type === 'missing_data');
  
  return severitySum > 1.5 || 
         (hasIrreversible && severitySum > 0.5) ||
         hasMissingData;
}

Designing the Escalation Path

Clear Handoff Protocol

When your agent escalates, it shouldn't just throw the problem at a human. Provide context:

pythonCopy
class EscalationTicket:
    agent_id: str
    task_description: str
    attempted_analysis: dict
    confidence_score: float
    recommended_next_steps: list[str]
    required_information: list[str]
    time_sensitivity: str

def create_escalation(decision: AgentDecision, task: str) -> EscalationTicket:
    return EscalationTicket(
        agent_id="agent-procurement-001",
        task_description=task,
        attempted_analysis=decision.model_dump(),
        confidence_score=decision.confidence,
        recommended_next_steps=[
            "Review procurement policy section 4.2",
            "Check vendor approval list",
            "Confirm budget allocation"
        ],
        required_information=["VP approval", "Current vendor contract"],
        time_sensitivity="standard"
    )

Getting This Right in Practice

At LavaPi, we've learned that the teams shipping the most reliable AI systems share one trait: they treat uncertainty as a first-class feature, not an edge case. They instrument their agents heavily, track what triggers escalations, and feed that data back into the confidence thresholds.

Start by logging every escalation decision for a week. You'll see patterns in what your agent struggles with. Use those patterns to calibrate your thresholds and improve your context gathering.

The Bottom Line

A good AI agent isn't one that never fails. It's one that fails explicitly and early, before the damage compounds. Build escalation paths from day one. Your future self—and your ops team—will thank you.