Compaction

Compaction shortens a transcript before it passes the model context limit. A long chat or agent loop otherwise fails on the next call.

Add withCompaction to your chat() call. It runs before each model call.

Quick setup#

1. Install the package#

npm i @tanstack/ai-compaction

pnpm add @tanstack/ai-compaction

yarn add @tanstack/ai-compaction

bun add @tanstack/ai-compaction

2. Add the middleware to your chat endpoint#

import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
} from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { withCompaction } from "@tanstack/ai-compaction";

export async function POST(request: Request) {
  const { messages, threadId, runId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter: openaiText("gpt-6.1-sol"),
    messages,
    threadId,
    runId,
    middleware: [withCompaction({ maxTokens: 100_000 })],
  });

  return toServerSentEventsResponse(stream);
}

The client does not change. Your useChat code stays the same.

3. Watch it run#

Use a small maxTokens and an onCompact log while you try it:

import { withCompaction } from "@tanstack/ai-compaction";

const compaction = withCompaction({
  maxTokens: 2_000,
  onCompact: (info) => {
    console.log(`Compacted from ${info.before} to ${info.after} tokens`);
  },
});

Put compaction in the middleware array and chat for a few turns. The server logs a line each time the history shrinks, and the chat keeps answering.

How it works#

• withCompaction is a chat middleware. Before each model call, it estimates the token count of the messages (characters / 4).
• When the count passes maxTokens, a strategy rewrites the messages that go to the provider.
• The full transcript does not change. Persistence and the UI still see every message.
• The system prompt is never dropped. chat() keeps it separate from messages.
• Tool calls stay paired with their results, so the request stays valid.

Pick a strategy#

Pass strategy to change how the history shrinks. Three are built in:

evictOldest#

This strategy is the cheapest. It keeps the recent messages, drops the older ones, and puts a short marker in their place. It is the default, so name it only to set keepRecentTokens:

import { withCompaction, evictOldest } from "@tanstack/ai-compaction";

withCompaction({
  maxTokens: 100_000,
  strategy: evictOldest({ keepRecentTokens: 40_000 }),
});

summarizeOldest#

This strategy keeps the meaning of old turns and costs one summary call. Give it a summarize function. The function gets the messages to drop and an input with the threadId of the conversation. It returns the summary. Give the summary call a thread id of its own, so it does not share state with the conversation:

import {
  chat,
  chatParamsFromRequest,
  summarize,
  toServerSentEventsResponse,
} from "@tanstack/ai";
import type { ModelMessage } from "@tanstack/ai";
import { openaiSummarize, openaiText } from "@tanstack/ai-openai";
import {
  summarizeOldest,
  withCompaction,
  type SummarizeInput,
} from "@tanstack/ai-compaction";

async function summarizeHistory(
  messages: Array<ModelMessage>,
  { threadId }: SummarizeInput,
) {
  const text = messages
    .map((m) => `${m.role}: ${typeof m.content === "string" ? m.content : ""}`)
    .join("\n");

  // Return the whole result, so compaction can report its usage.
  return summarize({
    adapter: openaiSummarize("gpt-6.1-sol"),
    text,
    threadId: `${threadId}:compaction`,
  });
}

export async function POST(request: Request) {
  const { messages, threadId, runId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter: openaiText("gpt-6.1-sol"),
    messages,
    threadId,
    runId,
    middleware: [
      withCompaction({
        maxTokens: 100_000,
        strategy: summarizeOldest({ summarize: summarizeHistory }),
      }),
    ],
  });

  return toServerSentEventsResponse(stream);
}

summarize can return the summary text, or an object with summary and usage. Return the summarize() result as it is, so that compaction can report the cost of the summary.

For a coding agent, use the ready summarizer in Better summaries.

clearToolResults#

This strategy is the best fit for agent loops. Tool output (file reads, command output) is usually most of the tokens. The strategy replaces the content of old tool results with a stub. Every message and every tool-call pair stays in place.

import { withCompaction, clearToolResults } from "@tanstack/ai-compaction";

withCompaction({
  maxTokens: 100_000,
  // Keep the 5 most recent tool results in full. Stub the older ones.
  strategy: clearToolResults({ keepRecentToolResults: 5 }),
});

Combine strategies#

composeStrategies runs several strategies in order. It stops as soon as the result is under maxTokens. Put the cheap, targeted strategy first and a broad fallback last.

This example clears old tool output first. It drops old messages only when that is not enough:

import {
  withCompaction,
  composeStrategies,
  clearToolResults,
  evictOldest,
} from "@tanstack/ai-compaction";

withCompaction({
  maxTokens: 100_000,
  strategy: composeStrategies(clearToolResults(), evictOldest()),
});

Let the provider compact#

Some providers can compact the conversation on their own servers. The model then keeps more of the old turns than a marker gives. It costs one compaction call.

Pass the adapter of the chat() call as native. When the count passes maxTokens, the compact method of the adapter runs in place of strategy:

import {
  chat,
  chatParamsFromRequest,
  toServerSentEventsResponse,
} from "@tanstack/ai";
import { openaiText } from "@tanstack/ai-openai";
import { withCompaction } from "@tanstack/ai-compaction";

// Create the adapter once, outside the request handler.
const adapter = openaiText("gpt-6.1-sol");

export async function POST(request: Request) {
  const { messages, threadId, runId } = await chatParamsFromRequest(request);

  const stream = chat({
    adapter,
    messages,
    threadId,
    runId,
    middleware: [withCompaction({ maxTokens: 100_000, native: adapter })],
  });

  return toServerSentEventsResponse(stream);
}

• An adapter with no compact method uses strategy.
• When compact fails, strategy runs in its place.

The OpenAI Responses adapters have compact. See Native compaction. For what happens on each kind of failure, see Native compaction failures.

When the chat is still too long#

The token count is an estimate. So sometimes a call is still too long for the model, and the provider rejects it.

You do not have to catch that error. When a call fails with CONTEXT_OVERFLOW_CODE, withCompaction compacts and calls the model again, in the same run. Your code only sees the answer. This also works with auto: false.

It retries once for each model call. If the compacted call is still too long, the run ends with a RUN_ERROR:

• code is CONTEXT_OVERFLOW_CODE ('context_overflow').
• message is the text from the provider.

These adapters set this code:

• OpenAI, Anthropic, Gemini, and Bedrock.
• OpenRouter, Grok, Cloudflare, and BytePlus.
• Other OpenAI-compatible adapters, when the provider sends context_length_exceeded.

To make sure that the retry always has something to drop, see Give the retry something to cut.

Compaction and persistence#

Compaction and server-side withPersistence use two views of the messages:

• messages is the full transcript. Persistence saves this view.
• providerMessages is the context for the model. Compaction rewrites this view.

The order of the middleware does not change this split. Dropped, summarized, and stubbed content stays in the message store.

If the persistence adapter has a metadata store, compaction also saves a small checkpoint. The next request then starts from the last compacted result. See Checkpoints for the details.

Advanced#

Compaction: Advanced has the deep details:

• Every option of withCompaction and of each strategy.
• Your own strategy, and compaction on your own schedule.
• Counts from the real usage of the provider, and summaries that run in the background.
• DevTools events, checkpoints, and session logs.