Skip to main content

⚖️ A/B Testing Backends

Use per-model routing rules to send traffic to different providers based on the requested model. This lets you compare provider quality, cost, or latency for the same task.

📋 Scenario​

You want /v1/chat/completions requests routed differently depending on the model:

  • gpt-4o → OpenAI
  • claude-sonnet → Anthropic

⚙️ Setup​

downstreams:
  - id: openai
    name: OpenAI
    api_formats: [openai]
    base_url: https://api.openai.com/v1
    api_key: sk-proj-...
    output_model_ids:
      - gpt-4o

  - id: anthropic
    name: Anthropic
    api_formats: [anthropic]
    base_url: https://api.anthropic.com
    api_key: sk-ant-...
    output_model_ids:
      - claude-sonnet-4-20250514

rules:
  - id: gpt4o-rule
    name: GPT-4o Route
    pattern_path: /v1/chat/completions
    pattern_model: gpt-4o
    match_downstreams: [openai]
    pipeline_config: []
    is_enabled: true

  - id: sonnet-rule
    name: Claude Sonnet Route
    pattern_path: /v1/chat/completions
    pattern_model: claude-sonnet-4-20250514
    match_format: [openai]
    match_downstream_format: [anthropic]
    match_downstreams: [anthropic]
    pipeline_config:
      - plugin_id: anthropic2openai
    is_enabled: true

🔍 How It Works​

Tresor's rule matching priority collects all matching rules, sorted by specificity. Format filters are applied after the initial path+model match:

  1. Request for gpt-4o → matches gpt4o-rule (exact path + exact model, downstream filter passes) → forwarded to OpenAI with no transform
  2. Request for claude-sonnet-4-20250514 → matches sonnet-rule (exact path + exact model, format filters pass) → forwarded to Anthropic

The anthropic2openai plugin converts Anthropic's response format back to OpenAI format, so the client sees a consistent interface regardless of which provider handled the request.

Auto-Translation note: Since the Anthropic downstream declares api_formats: [anthropic] and the request path /v1/chat/completions is OpenAI format, auto-translation would insert openai2anthropic on the request side. The explicit anthropic2openai in the rule handles the response side. Deduplication ensures no double-conversion.

➕ Adding More Providers​

Simply add more downstreams and rules:

  - id: azure-openai
    name: Azure OpenAI
    api_formats: [openai]
    base_url: https://my-resource.openai.azure.com/openai
    api_key: azure-key-...
    output_model_ids:
      - gpt-4o

rules:
  - id: azure-rule
    name: Azure GPT-4o Route
    pattern_path: /v1/chat/completions
    pattern_model: azure-gpt4o
    match_downstreams: [azure-openai]
    pipeline_config: []
    is_enabled: true

Now requests for azure-gpt4o route to Azure OpenAI, giving you three providers to compare.