LiteLLM: OpenRouter Integration and Fallbacks Configuration
0 (0)

By | 07/31/2026
0 Comment
Click to rate this post!
[Total: 0 Average: 0]

OpenRouter recently announced a 50% discount on OpenAI, so we decided to give it a try.

We’ll switch things over on LiteLLM, which already handles requests from all our services – so the switch should be pretty simple.

Before OpenRouter, we’ll set up fallbacks straight to OpenAI in LiteLLM – for cases when requests to OpenRouter fail.

Connecting OpenRouter to LiteLLM

Basically, all we need is to describe a new LiteLLM Deployment, a model, and set its api_base to OpenRouter with our own API Key.

The only difference is the model name: for direct calls to OpenAI we use model = openai/gpt-4.1-nano, but for OpenRouter it’s model = openrouter/openai/gpt-4.1-nano, see OpenRouter Completion Models.

You can get the full list of models from the OpenRouter API, see the Models docs.

$ curl -s "https://openrouter.ai/api/v1/models" | jq ".data[].id" | head
"qwen/qwen3.7-flash"
"anthropic/claude-opus-5-fast"
"anthropic/claude-opus-5"
"inclusionai/ling-3.0-flash:free"
"poolside/laguna-s-2.1"
"poolside/laguna-s-2.1:free"
"google/gemini-3.6-flash"
"google/gemini-3.6-flash:batch"
"google/gemini-3.5-flash-lite"

Creating an OpenRouter API Key

Sign up for OpenRouter, go to API Keys for the OpenRouter Workspace you need, and create an OpenRouter API Key:

LiteLLM: OpenRouter and Fallbacks configuration

By the way, in OpenRouter you can set your own limits on a key:

LiteLLM: OpenRouter and Fallbacks configuration

LiteLLM Proxy Config for OpenRouter

Let’s add a new model – for testing we’ll set our own name in model_name, but in production we use a “generic” one that clients expect.

In api_base we override the URL with the OpenRouter API one, see Using the OpenRouter API, and in api_key – the API Key we created above:

...
    model_list:

      - model_name: or-gpt-4.1-mini
        litellm_params:
          model: openrouter/openai/gpt-4.1-mini
          api_base: https://openrouter.ai/api/v1
          api_key: os.environ/OPENROUTER_API_KEY
...

While testing, we can pass the key variable through Helm’s envVars; in production this is done via External Secrets Operator (see AWS: Kubernetes and External Secrets Operator for AWS Secrets Manager and LiteLLM: AI Gateway in Kubernetes and metrics to VictoriaMetrics):

...
  envVars:
    OPENROUTER_API_KEY: "sk-or-v1-***"
...

Let’s check with curl:

$ curl -sS -D /tmp/headers 'https://aigw.test.example.co/chat/completions' -H 'Content-Type: application/json' -H "Authorization: Bearer $LITELLM_TESTING_KEY" --data ' { "model": "or-gpt-4.1-mini", "messages": [ { "role": "user", "content": "what llm are you" } ] } '  | jq
{
  "id": "gen-1785490640-Tbfwc5Rlx7cyhqpxzSB6",
  "created": 1785490640,
  "model": "or-gpt-4.1-mini",
  ...

The headers give us all the interesting info (I use -D into a file so jq works correctly):

$ cat /tmp/headers 
...
x-litellm-model-name: openrouter/openai/gpt-4.1-mini
x-litellm-model-api-base: https://openrouter.ai/api/v1
...

But once we rolled OpenRouter out to “test production” – we started catching 429 errors, so we had to add fallbacks to OpenAI.

LiteLLM Fallbacks

The limits are described in Credit Limits and Rate Limits, and even though our models aren’t free – we still hit errors.

And in any case, you should have fallbacks anyway.

OpenRouter also supports Automatic failover between models, but that requires changes on the client side – and we want this as transparent as possible, with minimal changes on the clients.

So we’ll set it up on LiteLLM instead, see Fallbacks (Provider Failover).

The LiteLLM docs are a bit of a mess here (not the first time, btw) – one example describes it as litellm_settings.fallbacks, another as router_settings.fallbacks.

But router_settings is the correct one, since it takes priority – router.py:

...

      _fallbacks = fallbacks or litellm.fallbacks

...

And fallbacks “arrive” from router_params in proxy_server.py:

...
router = litellm.Router(
    **router_params,
...

The format is pretty simple – “model to set the fallback for : list of models to route to on failure”:

router_settings:
  fallbacks: [{"<QUERY_MODEL>": ["<FALLBACK_MODEL1>","<FALLBACK_MODEL2>"]}]

You can also set a general fallback instead of a model-specific one, see Default Fallbacks:

router_settings:
  default_fallbacks: ["claude-opus"]

(again – in the docs it’s set under litellm_settings instead of router_settings, though it works the same either way)

Alright, let’s try it and see how it works.

Adding a Model Fallback

Let’s add a fallback for our test model:

router_settings:
  fallbacks: [{"or-gpt-4.1-mini": ["gpt-4.1-mini"]}]

And in the model itself – we “break” the api_base with a wrong port:

model_list:
  - model_name: or-gpt-4.1-mini
    litellm_params:
      model: openrouter/openai/gpt-4.1-mini
      api_base: https://openrouter.ai:8000/api/v1
      api_key: os.environ/OPENROUTER_API_KEY

Let’s repeat the request:

$ curl -sS -D /tmp/headers 'https://aigw.test.example.co/chat/completions' -H 'Content-Type: application/json' -H "Authorization: Bearer $LITELLM_TESTING_KEY" --data ' { "model": "or-gpt-4.1-mini", "messages": [ { "role": "user", "content": "what llm are you" } ] } '  | jq
{
  "id": "chatcmpl-E7eVkBzpdbWp5n6JagYlYHTlEEI1f",
  "created": 1785492728,
  "model": "gpt-4.1-mini-2025-04-14",
  ...

And the headers – now the response is from OpenAI:

$ cat /tmp/headers | grep x-litellm-model
x-litellm-model-id: 8b2d27fc9982e18f87ed59ca8c8b0d02c52546f26ece314f17d14433ed6498c2
x-litellm-model-name: openai/gpt-4.1-mini
x-litellm-model-api-base: https://api.openai.com
x-litellm-model-group: gpt-4.1-mini

Fallback Options

See Fallbacks + Retries + Timeouts + Cooldowns.

For fallback routing you can set a few useful options:

  • num_retries: how many times to retry the request against the primary model group before switching to the fallback model
    • if the model group has several deployments, a retry can go to a different deployment without falling back
  • timeout: how long to wait for a response before redirecting requests to the fallback model
    • after each timeout, the next num_retries attempt starts (if set to more than 1), and only then does it move on to the fallback mode
  • allowed_fails: how many failed requests to a model are allowed before it goes into cooldown
    • with a value of 3, cooldown kicks in on the fourth failed attempt
  • cooldown_time: how many seconds the model will be “disabled” from general routing

Again, the docs are a bit… rough here, since, for example, it says “allowed_fails: 3 # cooldown model if it fails > 1 call in a minute“, and instead of “in a minute“, they probably meant 30 seconds, matching the cooldown_time example.

Fallbacks and Prometheus metrics

Of course, it’s a good idea to monitor when fallbacks kick in – and when they fail.

LiteLLM has built-in metrics for this, see Fallback (Failover) Metrics:

LiteLLM: OpenRouter and Fallbacks configuration

Here:

  • litellm_deployment_cooled_down: how many times a deployment (model) was put into cooldown
  • litellm_deployment_successful_fallbacks: number of successful fallback triggers
  • litellm_deployment_failed_fallbacks: number of failed fallback attempts

And an example alert:

- alert: LiteLLM Deployment Failed Fallback
  expr: |
    sum by (requested_model, fallback_model, api_key_alias, team_alias, exception_class, exception_status) (
      increase(litellm_deployment_failed_fallbacks_total[5m])
    ) > 0
  for: 30s
  labels:
    component: devops
    environment: ops
    severity: critical
    ilert_routingkey: devops-ops-critical
  annotations:
    summary: LiteLLM Deployment Failed Fallback
    description: |-
      LiteLLM failed to handle a request using a fallback model during the last 5 minutes.
      *Failed fallbacks*: `{{ "{{" }} $value }}`
      *Requested model*: `{{ "{{" }} $labels.requested_model }}`
      *Fallback model*: `{{ "{{" }} $labels.fallback_model }}`
      *API key alias*: `{{ "{{" }} $labels.api_key_alias }}`
      *Team alias*: `{{ "{{" }} $labels.team_alias }}`
      *Exception class*: `{{ "{{" }} $labels.exception_class }}`
      *Exception status*: `{{ "{{" }} $labels.exception_status }}`
      <https://{{ $.Values.monitoring.root_url }}/d/adtt9jj/adrmshg/litellm-system-overview |:grafana: LiteLLM System overview>

Advanced routing

We have services that want to keep going to OpenAI directly – but we don’t want to change anything in their code, meaning model_name has to stay as-is.

There are a few options here, see the Tag Based Routing docs:

  • use Tags on API Keys
  • use Tags from headers (with some nuances)

Routing by API Key Tags

We create a new key, though when creating a key you can’t set Tags on it right away, since “This feature is only available for LiteLLM Enterprise“.

But you can create the key – and then set the Tag you need in its Settings afterward:

LiteLLM: OpenRouter and Fallbacks configuration

And similarly – a key tagged “direct-openrouter“.

In router_settings we add enable_tag_filtering=true, and in model_list we describe a model group – two deployments with the same model_name, but different conditions in tags:

router_settings:
  enable_tag_filtering: True

model_list:

  - model_name: or-gpt-4.1-mini
    litellm_params:
      model: openrouter/openai/gpt-4.1-mini
      api_base: https://openrouter.ai/api/v1
      api_key: os.environ/OPENROUTER_API_KEY
      tags: ["direct-openrouter"]

  - model_name: or-gpt-4.1-mini
    litellm_params:
      model: openai/gpt-4.1-mini
      api_key: os.environ/OPENAI_API_KEY
      tags: ["direct-openai"]

Let’s make a request with the “direct-openai” key:

$ curl -sS -D /tmp/openai 'https://aigw.test.example.co/chat/completions' -H 'Content-Type: application/json' -H "Authorization: Bearer sk-***" --data ' { "model": "or-gpt-4.1-mini", "messages": [ { "role": "user", "content": "what llm are you" } ] } ' | jq
{
  "id": "chatcmpl-E7f89zMEtza5liQnm8uBVQaaBcyVv",
  "created": 1785495109,
  "model": "gpt-4.1-mini-2025-04-14",
  ...

We get a response from OpenAI:

$ cat /tmp/openai | grep x-litellm-model
x-litellm-model-id: b1cd50178d84ad641058301b47c9f171d86337bcc7f0c08dc425f9f44ec2dffd
x-litellm-model-name: openai/gpt-4.1-mini
x-litellm-model-api-base: https://api.openai.com

And with the “direct-openrouter” key:

$ curl -sS -D /tmp/openrouter 'https://aigw.test.example.co/chat/completions' -H 'Content-Type: application/json' -H "Authorization: Bearer sk-***" --data ' { "model": "or-gpt-4.1-mini", "messages": [ { "role": "user", "content": "what llm are you" } ] } ' | jq
{
  "id": "gen-1785495093-Toi3dLdjHJ692Ue8bFgl",
  "created": 1785495093,
  "model": "or-gpt-4.1-mini",
  ...

We get a response from OpenRouter:

$ cat /tmp/openrouter | grep x-litellm-model
x-litellm-model-name: openrouter/openai/gpt-4.1-mini
x-litellm-model-api-base: https://openrouter.ai/api/v1

Routing by the User Agent header

Another option – instead of adding tags manually, just use the User Agent from the headers.

The docs on Regex-based tag routing (tag_regex) describe tag_regex as “Use tag_regex on a deployment to match incoming requests by their headers” – so it sounds like it can match any header, but in reality only User-Agent is taken into account (if I’m reading the code right):

...
        # Build header strings for regex matching from what the proxy already stores.
        # Currently we match against User-Agent; format matches "^User-Agent: claude-code/..."
        user_agent = metadata.get("user_agent", "")
        header_strings: list[str] = [f"User-Agent: {user_agent}"] if user_agent else []
...

Still, in my case that’s enough – one of our services is TypeScript with user-agent = "OpenAI/JS 6.26.0", and the other is Python with user-agent = "OpenAI/Python 2.45.0".

Let’s change the conditions in the models to use a regex:

- model_name: or-gpt-4.1-mini
  litellm_params:
    model: openrouter/openai/gpt-4.1-mini
    api_base: https://openrouter.ai/api/v1
    api_key: os.environ/OPENROUTER_API_KEY
    tag_regex:
      - '^User-Agent: OpenAI/JS '

- model_name: or-gpt-4.1-mini
  litellm_params:
    model: openai/gpt-4.1-mini
    api_key: os.environ/OPENAI_API_KEY
    tag_regex:
      - '^User-Agent: OpenAI/Python '

Let’s make a request with the “main” test key (which has no tags), but explicitly pass -H 'User-Agent: OpenAI/JS 6.26.0':

$ curl -sS -D /tmp/js-headers 'https://aigw.test.example.co/chat/completions' -H 'Content-Type: application/json' -H "Authorization: Bearer $LITELLM_TESTING_KEY" -H 'User-Agent: OpenAI/JS 6.26.0'   --data '{    "model": "or-gpt-4.1-mini",
    "messages": [{
      "role": "user",
      "content": "JS routing test unique-001"
    }]                                           
  }' | jq
{
  "id": "gen-1785495531-69Gu3ziqrjBmlNyEx3Jo",
  "created": 1785495531,
  "model": "or-gpt-4.1-mini",
  ...

We got a response from OpenRouter:

$ cat /tmp/js-headers
...
x-litellm-model-name: openrouter/openai/gpt-4.1-mini
x-litellm-model-api-base: https://openrouter.ai/api/v1
...

And a similar request, but with -H 'User-Agent: OpenAI/Python 2.45.0':

$ curl -sS -D /tmp/python-headers   'https://aigw.test.example.co/chat/completions'   -H 'Content-Type: application/json'   -H "Authorization: Bearer $LITELLM_TESTING_KEY"   -H 'User-Agent: OpenAI/Python 2.45.0'   --data '{
    "model": "or-gpt-4.1-mini",
    "messages": [{
      "role": "user",
      "content": "Python routing test unique-001"
    }]
  }' | jq
{
  "id": "chatcmpl-E7fFeVdB0ZWynqc5JxO3d4lUA4Gqf",
  "created": 1785495574,
  "model": "gpt-4.1-mini-2025-04-14",
  ...

And this time we got a response from OpenAI:

$ cat /tmp/python-headers
x-litellm-model-name: openai/gpt-4.1-mini
x-litellm-model-api-base: https://api.openai.com

Done.

Loading