AI Research Orchestrator: Multi-Agent Literature Synthesis

Literature review workflows involve repeated search, filtering, reading, synthesis, and citation verification steps. This project automates that pipeline with specialized agents.

The Problem

Traditional LLM research assistants fail because:

• No access to current literature (training cutoff)
• Hallucinated citations (plausible but fake)
• No workflow structure (single prompt → single output)
• No traceability (can't verify claims against sources)

The Orchestrator solves this with modular agents + real academic APIs.

Two Operating Modes

Production / Integration Mode

• Semantic Scholar API (with/without key for higher rate limits)
• OpenAlex API (generous 100k req/day)
• Real paper retrieval, deduplication, and analysis
• Full pipeline with live data

Offline Demonstration Mode

• Deterministic mock responses per research domain
• Pre-cached paper sets for common topics
• Full pipeline runs without network connectivity
• Enables development, demos, CI/CD without API keys

Both modes execute the identical agent pipeline—only the retrieval layer differs.

Architecture

Research Question
      ↓
Planning Agent → Structured Plan (objectives, keywords, sub-questions)
      ↓
Literature Retrieval → Semantic Scholar + OpenAlex (deduped, ranked)
      ↓
Analysis Agent → Per-paper: methodology, datasets, metrics, limitations
      ↓
Synthesis Agent → Cross-paper: trends, gaps, contradictions, opportunities
      ↓
Report Generator → Structured technical report + references

Agent Specialization

Agent	Input	Output	Key Capability
Planner	Research question	JSON plan	Decomposes into searchable sub-questions
Retriever	Plan + keywords	Paper metadata	Multi-source, dedup, citation extraction
Analyzer	Paper abstracts	Structured analyses	Per-paper methodology/metrics extraction
Synthesizer	Analyses	Synthesis report	Cross-paper pattern detection
Reporter	All above	Final document	Sectioned report with grounded citations

Technical Stack

• Runtime: Node.js / TypeScript
• Academic APIs: Semantic Scholar, OpenAlex (free tiers)
• LLM Integration: Modular provider abstraction (OpenAI, Anthropic, local via Ollama)
• State Management: Structured JSON intermediates between stages
• Offline Mode: Deterministic mock responses for demos/development

Key Engineering Challenges

1. API Rate Limiting

• Semantic Scholar: 100 req/5min (no key), 1000 req/5min (with key)
• OpenAlex: 100k req/day (generous)
• Solution: Exponential backoff, request batching, caching layer, offline mode

2. Paper Deduplication

• Same paper in multiple sources with different IDs
• Solution: Title+author fuzzy matching + DOI normalization + canonical ID assignment

3. Paper Ranking & Relevance

• Raw API returns 1000s of papers
• Solution:
  1. Keyword overlap scoring
  2. Citation count + recency weighting
  3. Venue quality heuristic
  4. LLM-based relevance filtering (top 50 → top 20)

4. Structured Output Enforcement

• LLMs drift from schema
• Solution:
  • Zod schemas for each stage output
  • Retry with schema validation
  • TypeScript types = runtime validation

5. Offline Demonstration Mode

• APIs unavailable during hackathons/demos
• Solution:
  • Deterministic mock responses per research domain
  • Pre-cached paper sets for common topics
  • Full pipeline runs without network

Illustrative Workflow Example

Note: The following is an illustrative run from the offline demonstration mode with pre-cached data. It demonstrates the pipeline flow but does not represent live API results.

Input: "Compare transformer architectures for long-context language modeling"

Planner Output:

{
  "objectives": ["Survey long-context attention mechanisms", "Compare efficiency/quality tradeoffs"],
  "keywords": ["long-context", "transformer", "attention", "linear attention", "ring attention"],
  "sub_questions": ["How do sparse attention patterns compare?", "What are memory scaling laws?"],
  "evaluation_criteria": ["Perplexity", "Throughput", "Memory", "Max context"]
}

Retriever (Illustrative): Fetches 150 papers → dedupes to 89 → ranks → top 25

Analyzer (per paper): Extracts architecture variant, context length, perplexity, hardware, training cost

Synthesizer: Identifies clusters (RoPE, ALiBi, Ring Attention, Linear/Recurrent), compares scaling

Reporter: Generates 12-page report with tables, citations, gaps, future work

Capabilities Status

Capability	Status
Multi-source retrieval	✅ Semantic Scholar + OpenAlex
Structured planning	✅ JSON schema validated
Per-paper analysis	✅ Methodology, metrics, limitations
Cross-paper synthesis	✅ Trends, gaps, contradictions
Report generation	✅ Executive summary → references
Offline mode	✅ Deterministic demo
Provider abstraction	✅ OpenAI, Anthropic, Ollama

Lessons Learned

1. Specialized agents > monolithic prompts: Each stage has different reasoning needs
2. External retrieval grounds the system: Real papers beat parametric knowledge for citations
3. Structured intermediates enable debugging: Can inspect Analyzer output before Synthesis
4. Consensus/synthesis is the hard part: Requires cross-document reasoning, not just summarization
5. Offline mode is a feature, not a hack: Enables development, demos, CI/CD without API keys

Limitations & Future Work

Limitation	Future Direction
Sequential only (no parallel)	Parallel retrieval + analysis
No citation verification	Cross-ref with DOI/PDF extraction
Limited evaluation metrics	Automated report quality scoring
Single research domain per run	Multi-domain comparative studies
No persistent knowledge base	Vector DB of analyzed papers
No human-in-the-loop	Interactive refinement at each stage

Future Directions

• Distributed agent execution: Each agent on separate infrastructure
• RAG over analyzed papers: Query past syntheses
• Automated experiment planning: From gaps → experiment designs
• Integration with lab notebooks: Export to Notion, Obsidian, Overleaf
• Multi-modal papers: Figures, tables, supplementary materials
• Citation graph analysis: Influence mapping, cluster detection
• Real-time literature monitoring: Alerts on new relevant papers

Related Project

This blog post accompanies the AI Research Orchestrator project case study (also known as Aster).

View Project Case Study →

Resources

• Semantic Scholar API
• OpenAlex API
• TypeScript
• Zod Validation
• Agent Orchestration Patterns