Provider Routing and Streaming

Direct calls from the desktop client would embed provider credentials and require different payloads per provider. The routing layer keeps keys on the server, exposes a single request format, and maintains consistent streaming behavior.

All requests go through a single endpoint: /api/llm/chat/completions. The router determines the appropriate provider based on the model ID in the request payload. Each provider has dedicated handlers in server/src/handlers/proxy/.

Job processors submit a normalized payload with task ID, job ID, prompt content, and model selection. Theprovider_transformers module maps that payload into provider-specific request shapes.

// Normalized request from desktop
{
  "model": "anthropic/claude-opus-4-5-20251101",
  "messages": [
    { "role": "system", "content": "..." },
    { "role": "user", "content": "..." }
  ],
  "max_tokens": 16384,
  "temperature": 0.7,
  "stream": true,
  "metadata": {
    "job_id": "uuid-...",
    "session_id": "uuid-...",
    "task_type": "implementation_plan"
  }
}

// Transformed for Anthropic
{
  "model": "claude-opus-4-5-20251101",
  "system": "...",
  "messages": [{ "role": "user", "content": "..." }],
  "max_tokens": 16384,
  "stream": true
}

Responses are streamed back to the desktop client through ModernStreamHandler, enabling real-time UI updates and progressive plan rendering.

// ModernStreamHandler processing loop
async fn handle_stream(
    response: Response,
    job_id: &str,
) -> Result<StreamResult> {
    let mut stream = response.bytes_stream();
    let mut accumulated = String::new();

    while let Some(chunk) = stream.next().await {
        let text = parse_sse_chunk(&chunk?)?;
        accumulated.push_str(&text);

        // Emit event to desktop client
        emit_stream_event(job_id, StreamEvent::Chunk {
            content: text,
            accumulated_tokens: count_tokens(&accumulated),
        });
    }

    // Final usage from provider response
    let usage = extract_final_usage(&accumulated)?;
    Ok(StreamResult { content: accumulated, usage })
}

When a primary provider fails (rate limit, outage, or error), the routing layer can automatically retry through OpenRouter as a fallback aggregator. This provides resilience without requiring user intervention.

Every request records usage metadata so teams can audit cost and performance. Token counts come from provider responses when available, with fallback to tiktoken-based estimation.

// Usage record stored per request
{
  "tokens_input": 4521,
  "tokens_output": 2847,
  "cache_read_tokens": 1200,   // Anthropic prompt caching
  "cache_write_tokens": 0,
  "cost": 0.0234,              // USD based on model pricing
  "service_name": "anthropic/claude-opus-4-5-20251101",
  "request_id": "550e8400-e29b-41d4-a716-446655440000"  // Server-generated UUID
}

Requests containing images are validated before routing to ensure the selected model supports vision capabilities. Invalid requests fail fast with clear error messages.

If a provider fails or no provider is configured, the job is marked failed and the error payload is stored. Users can retry or run the job with another model instead of relying on silent fallbacks.

API keys stay in the server configuration. The desktop client only receives allowed model lists and never embeds provider credentials.

If you are building a similar architecture, the key components to implement are: