SS
Back to Blog
2026-06-20 · 10 min read

Technical Note: Designing Reliable Multi-Agent Systems

Notes on orchestration, context management, and fault tolerance for specialized LLM agent teams. Based on building AgentForge and AI Research Orchestrator.

Multi-agent systems look simple on the surface: one orchestrator, a few specialist agents, and a shared task.

The hard part is the glue. Once multiple agents start reasoning in parallel, you inherit the same problems that show up in distributed systems: partial failure, inconsistent state, and coordination overhead.

This note distills lessons from building AgentForge (local multi-agent coding assistant) and AI Research Orchestrator (Aster — literature synthesis pipeline).

Failure Modes

1. Context Drift

Agents operating on free-text context lose nuance across stages. By stage 3, the original intent is diluted.

Solution: Structured intermediate representations (JSON schemas) at every boundary. Each agent consumes and produces validated structures.

2. Cascading Hallucinations

One agent's hallucinated API becomes the next agent's assumed fact.

Solution:

  • Critic/validator agent at each stage boundary
  • Retrieval-augmented context for factual grounding
  • Human-in-the-loop for critical decisions

3. State Inconsistency

Parallel agents mutate shared state without coordination.

Solution:

  • Immutable context passed between stages (functional style)
  • Explicit merge/consensus step for parallel outputs
  • Versioned context snapshots for rollback

4. Partial Failure

One agent times out or errors; others complete. What does the orchestrator do?

Solution:

  • Retries with exponential backoff as first-class design, not exception
  • Checkpointing: save intermediate state after each stage
  • Graceful degradation: return partial results with clear failure markers

Context Management

Keep State Structured

{
  "stage": "architecture",
  "input": { "requirements": [...], "constraints": {...} },
  "output": { "components": [...], "interfaces": {...} },
  "metadata": { "agent": "architect", "model": "qwen2.5:14b", "tokens": 2341 }
}

Structured state enables:

  • Programmatic validation at boundaries
  • Debugging: inspect any stage independently
  • Replay: re-run from any checkpoint
  • Testing: golden examples per stage

Limit What Each Agent Sees

Agent A doesn't need Agent C's internal reasoning. Pass only what's necessary:

  • Input contract: What this agent receives
  • Output contract: What this agent produces
  • No side channels: No shared mutable state

Coordination Patterns

Sequential Pipeline (Aster / AI Research Orchestrator)

Planner → Retriever → Analyzer → Synthesizer → Reporter

Simple, debuggable, but latency = sum of all stages.

Parallel + Consensus (AgentForge)

Analyst || Architect → Critic → Planner → Generator → Consensus

Lower latency, but needs explicit consensus logic.

Hybrid

Use sequential for dependent stages, parallel for independent ones.

Retries as Design

Don't treat retries as error handling. Design for them:

async def run_with_retry(agent, input, max_retries=3):
    for attempt in range(max_retries):
        try:
            return await agent.run(input)
        except TransientError as e:
            if attempt == max_retries - 1:
                raise
            await asyncio.sleep(2 ** attempt)  # exponential backoff

Key distinction: Transient (rate limit, timeout) vs. Permanent (schema violation, logic error). Only retry transient.

Observability

Every agent execution should emit:

  • Structured logs: stage, agent, model, input_hash, output_hash, latency, success/failure
  • Metrics: tokens, cost, retry count, validation pass/fail
  • Traces: end-to-end request flow with timing per stage

This turns "it's slow" into "Analyzer stage p99 latency is 12s due to model X."

Testing Strategy

  1. Unit: Each agent in isolation with golden input/output pairs
  2. Integration: Full pipeline with mocked LLM responses
  3. Regression: Golden pipeline runs saved as snapshots
  4. Chaos: Inject failures (timeouts, malformed outputs) at each stage

Lessons Learned

  1. Role specialization > prompt engineering: Explicit roles with structured contracts beat "you are a senior engineer"
  2. Structured intermediates enable debugging: Can inspect each stage's output independently
  3. Consensus/synthesis is the hard part: Cross-document reasoning, not just summarization
  4. Offline mode is a feature: Deterministic mock responses enable development, demos, CI/CD
  5. Retries are part of the design: Not an afterthought
  6. Observability from day one: You can't debug what you can't see

Related Projects

This technical note draws from two project case studies:

Resources