> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orcbot.buzzchat.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Multi-Agent Orchestration

> Spawn worker agents for parallel task execution with real-time coordination

## Overview

OrcBot's multi-agent orchestration system enables parallel task execution by spawning isolated worker processes. Each worker is a full Agent instance that can execute tasks independently while coordinating with the main agent through IPC (Inter-Process Communication).

<Note>
  Multi-agent orchestration is ideal for parallelizing independent tasks, background research, or long-running operations that shouldn't block the main agent.
</Note>

## Architecture

```mermaid theme={null}
flowchart TB
    Main[Main Agent]
    Orch[AgentOrchestrator]
    W1[Worker 1]
    W2[Worker 2]
    W3[Worker 3]
    
    Main -->|spawn_agent| Orch
    Orch -->|fork| W1
    Orch -->|fork| W2
    Orch -->|fork| W3
    
    W1 -->|results via IPC| Main
    W2 -->|results via IPC| Main
    W3 -->|results via IPC| Main
```

### Key Components

* **AgentOrchestrator** (`src/core/AgentOrchestrator.ts`) - Manages worker lifecycle
* **AgentWorker** (`src/core/AgentWorker.ts`) - Worker process wrapper
* **WorkerProfileManager** (`src/core/WorkerProfile.ts`) - Worker metadata and persistence
* **MessageBus** - IPC communication channel

## Spawning Workers

### Using spawn\_agent Skill

<ParamField path="name" type="string" required>
  Unique identifier for the worker agent
</ParamField>

<ParamField path="type" type="string" default="worker">
  Agent type: `worker` (isolated process) or `peer` (network agent)
</ParamField>

<ParamField path="config" type="object">
  Optional configuration overrides for the worker
</ParamField>

**Example:**

```json theme={null}
{
  "skillName": "spawn_agent",
  "name": "researcher",
  "type": "worker",
  "config": {
    "modelName": "gpt-4o",
    "maxStepsPerAction": 20
  }
}
```

### Using delegate\_task Skill

<ParamField path="task" type="string" required>
  Task description to delegate
</ParamField>

<ParamField path="priority" type="number" default="5">
  Task priority (1-10)
</ParamField>

<ParamField path="targetAgent" type="string">
  Specific worker name, or omit for auto-assignment
</ParamField>

**Example:**

```json theme={null}
{
  "skillName": "delegate_task",
  "task": "Research the latest developments in quantum computing and summarize findings",
  "priority": 8,
  "targetAgent": "researcher"
}
```

## Worker Lifecycle

<Steps>
  <Step title="Spawn">
    Main agent calls `spawn_agent` with a unique name and optional config.

    ```bash theme={null}
    orcbot push "Spawn a researcher agent for background tasks"
    ```

    The orchestrator forks a new Node.js process running `AgentWorker.ts`.
  </Step>

  <Step title="Initialize">
    Worker loads its own Agent instance with shared or custom configuration.

    Worker has access to:

    * All core skills
    * Shared memory (read-only for safety)
    * Its own action queue
    * IPC channel to main agent
  </Step>

  <Step title="Execute">
    Worker processes tasks from its queue autonomously.

    ```typescript theme={null}
    // Worker executes independently
    worker.processAction({
      actionId: "research-task-1",
      task: "Find latest AI papers",
      priority: 8
    });
    ```
  </Step>

  <Step title="Report">
    Worker sends results back to main agent via IPC.

    ```typescript theme={null}
    worker.sendMessage({
      type: "task_complete",
      taskId: "research-task-1",
      result: "Found 5 relevant papers..."
    });
    ```
  </Step>

  <Step title="Cleanup">
    Worker terminates after task completion or timeout.

    The orchestrator cleans up resources and removes the worker profile.
  </Step>
</Steps>

## Monitoring Workers

### In the TUI

```bash theme={null}
orcbot ui
```

Navigate to **Worker Processes** to see:

* Worker name and PID
* Current status (idle, busy, error)
* Task queue depth
* Memory usage
* Uptime

### Using orchestrator\_status Skill

```json theme={null}
{
  "skillName": "orchestrator_status"
}
```

**Response:**

```json theme={null}
{
  "activeWorkers": 2,
  "workers": [
    {
      "name": "researcher",
      "pid": 12345,
      "status": "busy",
      "queueDepth": 3,
      "uptime": "00:05:23"
    },
    {
      "name": "scraper",
      "pid": 12346,
      "status": "idle",
      "queueDepth": 0,
      "uptime": "00:02:15"
    }
  ]
}
```

## Inter-Agent Communication

### Sending Messages

<ParamField path="targetAgent" type="string" required>
  Name of the worker to message
</ParamField>

<ParamField path="message" type="string" required>
  Message content
</ParamField>

<ParamField path="priority" type="number" default="5">
  Message priority
</ParamField>

**Example:**

```json theme={null}
{
  "skillName": "send_agent_message",
  "targetAgent": "researcher",
  "message": "Prioritize quantum computing papers from 2024",
  "priority": 9
}
```

### Broadcasting

```json theme={null}
{
  "skillName": "broadcast_to_agents",
  "message": "System maintenance in 5 minutes - wrap up current tasks",
  "excludeAgents": ["critical-worker"]
}
```

## Use Cases

<CardGroup cols={2}>
  <Card title="Parallel Research" icon="magnifying-glass">
    Spawn multiple researchers to investigate different topics simultaneously.

    ```bash theme={null}
    orcbot push "Spawn 3 researchers to compare React, Vue, and Angular"
    ```
  </Card>

  <Card title="Background Monitoring" icon="radar">
    Delegate long-running monitoring tasks to workers.

    ```bash theme={null}
    orcbot push "Spawn a monitor agent to watch server metrics every 5 minutes"
    ```
  </Card>

  <Card title="Data Processing Pipeline" icon="gears">
    Chain workers for ETL (Extract, Transform, Load) operations.

    * Worker 1: Scrape data
    * Worker 2: Transform and validate
    * Worker 3: Store in database
  </Card>

  <Card title="Multi-User Support" icon="users">
    Dedicate workers to handle specific users or channels.

    ```bash theme={null}
    orcbot push "Spawn a customer support agent for VIP users"
    ```
  </Card>
</CardGroup>

## Configuration

### Worker-Specific Settings

```yaml theme={null}
# orcbot.config.yaml
orchestrator:
  maxWorkers: 5
  workerIdleTimeout: 600000  # 10 minutes
  workerMemoryLimit: 512     # MB
  
workerDefaults:
  modelName: gpt-4o-mini      # Use cheaper model for workers
  maxStepsPerAction: 10
  autonomyEnabled: false      # Workers don't need heartbeat
```

### Per-Worker Overrides

```json theme={null}
{
  "skillName": "spawn_agent",
  "name": "premium-researcher",
  "config": {
    "modelName": "gpt-4o",
    "maxStepsPerAction": 30,
    "memoryContextLimit": 20000
  }
}
```

## Best Practices

<Warning>
  **Resource limits:** Each worker is a full Node.js process. Monitor system resources when spawning many workers.

  Recommended limits:

  * 5 workers max on 8GB RAM
  * 10 workers max on 16GB RAM
  * 20 workers max on 32GB RAM
</Warning>

<Tip>
  **Task granularity:** Delegate coarse-grained tasks ("research topic X") rather than fine-grained operations ("search for one keyword"). Workers have initialization overhead.
</Tip>

### Do's

* Use workers for independent, parallelizable tasks
* Set reasonable timeouts for worker tasks
* Monitor worker health in production
* Use cheaper models for worker agents
* Clean up idle workers to free resources

### Don'ts

* Don't spawn workers for trivial tasks (use the main agent)
* Don't share mutable state between workers and main agent
* Don't rely on worker execution order (they're async)
* Don't spawn workers recursively (workers can't spawn workers)

## Troubleshooting

### Worker Not Starting

**Symptoms:** `spawn_agent` completes but worker doesn't appear in `orchestrator_status`.

**Causes:**

* Insufficient system resources
* Worker name conflicts
* Invalid configuration

**Solution:**

```bash theme={null}
# Check system resources
orcbot push "Run system_check"

# Verify unique worker names
orcbot push "Show orchestrator status"

# Check worker logs
tail -f ~/.orcbot/logs/workers/researcher.log
```

### Worker Crashes

**Symptoms:** Worker disappears from orchestrator status mid-task.

**Causes:**

* Out of memory
* Uncaught exception in worker code
* Task timeout exceeded

**Solution:**

```bash theme={null}
# Check worker logs
cat ~/.orcbot/logs/workers/researcher.log

# Increase memory limit
# In orcbot.config.yaml:
workerMemoryLimit: 1024

# Increase task timeout
maxStepsPerAction: 20
```

### Tasks Not Distributing

**Symptoms:** All tasks go to one worker instead of distributing.

**Causes:**

* Only one worker marked as available
* Task routing rules favor specific worker

**Solution:**

```json theme={null}
{
  "skillName": "delegate_task",
  "task": "Your task here",
  "targetAgent": null  // Let orchestrator auto-assign
}
```

## Performance Metrics

### Worker Pool Efficiency

```bash theme={null}
orcbot push "Show orchestrator metrics"
```

**Key Metrics:**

* **Utilization:** Percentage of time workers are busy vs idle
* **Queue depth:** Average number of pending tasks per worker
* **Throughput:** Tasks completed per minute
* **Latency:** Average time from task delegation to completion

**Optimizations:**

* **High utilization (greater than 80%):** Spawn more workers
* **Low utilization (less than 20%):** Reduce worker count
* **High queue depth:** Increase worker capacity or add workers
* **High latency:** Check for slow tasks or resource bottlenecks

## Related Skills

<CardGroup cols={2}>
  <Card title="spawn_agent" href="/api/skills/orchestration#spawn_agent">
    Create a new worker agent
  </Card>

  <Card title="delegate_task" href="/api/skills/orchestration#delegate_task">
    Assign a task to a worker
  </Card>

  <Card title="orchestrator_status" href="/api/skills/orchestration#orchestrator_status">
    View active workers and their status
  </Card>

  <Card title="send_agent_message" href="/api/skills/orchestration#send_agent_message">
    Send a message to a specific worker
  </Card>
</CardGroup>