LLM Gateway

llmgatewayText calls one OpenAI-compatible endpoint that routes across providers, falls back on failure, and records usage and cost.

1. Install

shell

npm i @tanstack/ai-llmgateway

shell

pnpm add @tanstack/ai-llmgateway

shell

yarn add @tanstack/ai-llmgateway

shell

bun add @tanstack/ai-llmgateway

2. Set the API key

Get a key from the LLM Gateway dashboard. Put it in the environment of your server:

shell

LLM_GATEWAY_API_KEY=llmgtwy_your-api-key

llmGatewayText and llmGatewaySummarize read LLM_GATEWAY_API_KEY. To pass the key in code, use the create* factory of the same adapter:

import { createLLMGatewayText } from "@tanstack/ai-llmgateway";

const adapter = createLLMGatewayText("gpt-5.6-terra", process.env.LLM_GATEWAY_API_KEY!);

3. Stream a chat from the server

import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
} from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

export async function POST(request: Request) {
  const { messages, threadId, runId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
    threadId,
    runId,
  });

  return toServerSentEventsResponse(stream);
}

4. Render the chat

The client is the same for every provider. Point useChat at your route:

tsx

import { useState } from "react";
import { useChat, fetchServerSentEvents } from "@tanstack/ai-react";

export function Chat() {
  const [input, setInput] = useState("");
  const { messages, sendMessage } = useChat({
    connection: fetchServerSentEvents("/api/chat"),
    threadId: "support-chat",
  });

  return (
    <form
      onSubmit={(event) => {
        event.preventDefault();
        sendMessage(input);
        setInput("");
      }}
    >
      {messages.map((message) => (
        <p key={message.id}>
          {message.parts
            .map((part) => (part.type === "text" ? part.content : ""))
            .join("")}
        </p>
      ))}
      <input value={input} onChange={(event) => setInput(event.target.value)} />
    </form>
  );
}

For Vue, Solid, Svelte, Angular, and the other frameworks, see Quick Start.

You now have a streaming chat through LLM Gateway. To try another model, change the id, for example to claude-opus-5 or kimi-k3.

Pick a model and a provider

Any model on llmgateway.io/models works. Pass its id as the model name:

A bare id lets the gateway pick the best provider that is available.
A provider/ prefix always sends the request to that provider.

plaintext

model: "gpt-5.6-terra"          // the gateway picks the provider
model: "claude-sonnet-5"        // the gateway picks the provider
model: "moonshot/kimi-k3"       // always sent to Moonshot
model: "fireworks/kimi-k3"      // always sent to Fireworks

Tools

Tools work the same as with every adapter:

import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
  toolDefinition,
} from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";
import { z } from "zod";

const getWeatherDef = toolDefinition({
  name: "get_weather",
  description: "Get the current weather",
  inputSchema: z.object({
    location: z.string(),
  }),
});

const getWeather = getWeatherDef.server(async ({ location }) => {
  return { location, temperature: 72, conditions: "sunny" };
});

export async function POST(request: Request) {
  const { messages, threadId, runId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
    threadId,
    runId,
    tools: [getWeather],
  });

  return toServerSentEventsResponse(stream);
}

See Tools.

Summarization

import { summarize } from "@tanstack/ai";
import { llmGatewaySummarize } from "@tanstack/ai-llmgateway";

const result = await summarize({
  adapter: llmGatewaySummarize("gpt-5.4-mini"),
  text: "Long article text...",
  threadId: "thread-1",
  stream: false,
});

console.log(result.summary);

Models and options

Model options

The gateway takes the standard Chat Completions parameters and sends them to the provider that it picks. The gateway removes the parameters that a provider does not support. Sampling options go in modelOptions:

import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("kimi-k3"),
  messages: [{ role: "user", content: "Hello!" }],
  threadId: "thread-1",
  modelOptions: {
    temperature: 0.7,
    max_completion_tokens: 4096,
  },
});

Reasoning

Set reasoning on chat():

import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("kimi-k3"),
  messages: [{ role: "user", content: "Plan a database migration." }],
  threadId: "thread-1",
  reasoning: "high",
});

It goes out as reasoning_effort. The types take the levels of that model from the LLM Gateway catalog. No model takes budgetTokens.

Reasoning models stream their thinking as reasoning_content deltas. The adapter turns them into AG-UI REASONING_* events. See Reasoning.

Models

LLMGATEWAY_CHAT_MODELS is the chat model list from the LLM Gateway catalog. Each model has type data for its input kinds and its reasoning levels. Autocomplete in the editor uses this list.

An id that is not in the list still works. It gets text-only input and the generic options.

adapter.inputModalities lists the input kinds of the selected model. It is undefined for a model that this package does not know. See Check what a model accepts.

Advanced

Self-hosted gateway

To use your own deployment, set baseURL:

import { createLLMGatewayText } from "@tanstack/ai-llmgateway";

const adapter = createLLMGatewayText("gpt-5.6-terra", process.env.LLM_GATEWAY_API_KEY!, {
  baseURL: "https://api.llmgateway.io/v1", // the hosted default. Set your own URL here.
});

The adapter works the same against a self-hosted gateway.

Need a header on each request, or a log of each model call? This adapter supports wrapFetch.

API reference

Factories	Creates
llmGatewayText(model, config?) / createLLMGatewayText(model, apiKey, config?)	Chat adapter
llmGatewaySummarize(model, config?) / createLLMGatewaySummarize(model, apiKey, config?)	Summarization adapter

The short factory reads LLM_GATEWAY_API_KEY. The create* factory takes an explicit API key.