Skip to content
TanStackAI
Media

World Generation

generateWorld() builds a world on the server from a prompt. The result is one of two kinds:

  • A live world (Reactor). The server mints a session token. The browser connects, plays the world, and changes it with new prompts.
  • A finished 3D world (World Labs Marble). The server waits for the job, then returns a viewer URL and asset links.

Warning

World generation is experimental. The API can change. Live world models bill for each second that a session holds a GPU. World Labs bills in credits for each generation.

Quick setup: a live world

This streams a live Reactor Orbis world. It takes about 10 minutes.

  1. Install the packages:
shell
npm i @tanstack/ai @tanstack/ai-reactor @reactor-team/js-sdk
  1. Put your key in .env on the server:
shell
REACTOR_API_KEY=your-reactor-key
  1. Add a POST route at /api/world that mints a session:
ts
import { generateWorld } from '@tanstack/ai'
import { reactorWorld } from '@tanstack/ai-reactor'

export async function POST(request: Request) {
  const body: unknown = await request.json()
  const prompt =
    typeof body === 'object' &&
    body !== null &&
    'prompt' in body &&
    typeof body.prompt === 'string'
      ? body.prompt
      : ''
  if (prompt.length === 0) {
    return Response.json({ error: 'prompt is required' }, { status: 400 })
  }

  const world = await generateWorld({
    adapter: reactorWorld('visko-orbis-stable'),
    prompt,
    threadId: 'thread-1',
  })

  return Response.json({
    token: world.token,
    model: world.model,
    prompt: world.prompt,
    expiresAt: world.expiresAt,
  })
}
  1. In the browser, connect and play the world in a <video> element:
ts
import { Reactor } from '@reactor-team/js-sdk'

const video = document.querySelector('video')
if (!video) {
  throw new Error('Missing video element')
}

const world = await fetch('/api/world', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    prompt:
      'A dramatic coastline of black volcanic cliffs at golden hour, a single unbroken take.',
  }),
}).then(async (response) => {
  if (!response.ok) {
    throw new Error('World session failed')
  }
  return response.json()
})

const token = typeof world.token === 'string' ? world.token : ''
const model = typeof world.model === 'string' ? world.model : ''
const prompt = typeof world.prompt === 'string' ? world.prompt : ''
if (token.length === 0 || model.length === 0 || prompt.length === 0) {
  throw new Error('World payload is incomplete')
}

const reactor = new Reactor({ modelName: model })

reactor.on('trackReceived', (name, _track, stream) => {
  if (name !== 'main_video') return
  video.muted = true
  const attach = () => {
    video.srcObject = null
    video.srcObject = stream
    void video.play().catch((error: unknown) => {
      if (error instanceof DOMException && error.name === 'AbortError') return
    })
  }
  attach()
  for (const track of stream.getTracks()) {
    track.addEventListener('unmute', attach)
  }
})

await reactor.connect(token)
await reactor.sendCommand('set_resolution', { resolution: '1080p' })
await reactor.sendCommand('set_prompt', { prompt })
await reactor.sendCommand('start', {})

The video element now plays a live world. To steer it, send a new set_prompt. The scene changes at the next chunk.

How it works

The work splits in two:

  • The server calls generateWorld() with a world adapter. A live adapter returns a short-lived token, the model slug to connect to, and the prompt. The API key stays on the server.
  • The browser uses the Reactor SDK to connect with the token, play the stream, and send commands.

There is no client hook for worlds. For the shared model of other media types, see Generations.

Live world models

reactorWorld() takes these ids. The model field on the result is the Reactor connect slug.

IdConnect slugWhat it does
visko-orbis-stablereactor/visko-orbis-stableSteerable video with live audio
visko-orbis-dynamicreactor/visko-orbis-dynamicSame family, can change resolution while it runs
lingbot-world-2reactor/lingbot-world-2A world you can move through, anchored on an image
lingbotreactor/lingbotVideo you can move through, anchored on an image
heliosreactor/heliosInteractive live video

More notes:

  • Happy Oyster (happy-oyster-adventure, happy-oyster-director) uses createWorld and startTravel after connect.
  • Helios also works with generateLiveVideo() and reactorVideo(). See Live Generation.
  • Every id is in the Reactor adapter.

Finished 3D worlds (World Labs)

World Labs Marble runs a job, not a live stream. generateWorld() waits until the world is ready. Then it returns a viewer URL and asset links. Set WORLDLABS_API_KEY on the server.

ts
import { generateWorld } from '@tanstack/ai'
import { worldlabsWorld } from '@tanstack/ai-worldlabs'

const world = await generateWorld({
  adapter: worldlabsWorld('marble-1.1'),
  prompt: 'A mystical forest with glowing mushrooms',
  threadId: 'thread-1',
})

console.log(world.url) // The Marble viewer

What you get back:

  • world.url is the Marble viewer. Do not put it in an iframe.
  • world.assets holds splat, mesh, and panorama URLs when the provider returns them. These URLs are often signed.

Models: marble-1.1-plus, marble-1.1 (the World Labs default), marble-1.0, and marble-1.0-draft. See the World Labs adapter for image and video inputs.

A full app is in examples/ts-react-media. Open the World tab. Pick a Reactor model for a live stream, or a Marble model for a finished 3D world. What the example shows:

  • If splat files are present, it loads one SPZ in Spark. If not, it shows a thumbnail and an Open in Marble link.
  • It can download the Marble files when the response includes them.
  • It can save a Reactor session as MP4 when recording is on for that model or plan.

Advanced

Start LingBot from a seed image

LingBot and LingBot World 2 start from an image. Pass a File from <input type="file" accept="image/png,image/jpeg">. The SDK uploads the file. Do not send base64.

start also needs set_prompt. Write the prompt so it describes what the image shows. If the prompt and the image disagree, the image wins and the world drifts.

ts
const picker = document.querySelector('input[type="file"]')
if (!(picker instanceof HTMLInputElement)) {
  throw new Error('Pick a seed image')
}
const file = picker.files?.[0]
if (file === undefined) {
  throw new Error('Pick a seed image')
}
const image = await reactor.uploadFile(file)
await reactor.sendCommand('set_image', { image })
await reactor.sendCommand('set_prompt', { prompt })
await reactor.sendCommand('start', {})

Let users bring their own Reactor key

The browser can save a Reactor key and send it in an x-byok-reactor header. The server reads that header first, then REACTOR_API_KEY. Do not put the key in the JSON body.

Server:

ts
import { generateWorld } from '@tanstack/ai'
import { byokMissing, getByokKey } from '@tanstack/ai/byok/server'
import { reactorWorld } from '@tanstack/ai-reactor'
import { reactorByok } from '@tanstack/ai-reactor/byok'

export async function POST(request: Request) {
  const apiKey = getByokKey(request, reactorByok)
  if (!apiKey) return byokMissing(reactorByok)

  const world = await generateWorld({
    adapter: reactorWorld('visko-orbis-stable', { apiKey }),
    prompt: 'A dramatic coastline of black volcanic cliffs at golden hour',
    threadId: 'thread-1',
  })

  return Response.json({
    token: world.token,
    model: world.model,
    prompt: world.prompt,
  })
}

Browser:

ts
import { defineByok, defaultByokStorage } from '@tanstack/ai-client/byok'
import { reactorByok } from '@tanstack/ai-reactor/byok'

const byok = defineByok({
  storage: defaultByokStorage(),
  providers: [reactorByok],
})
byok.setServerCoverage(true)

const world = await fetch('/api/world', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    ...byok.headers(reactorByok.id),
  },
}).then((response) => response.json())

See Bring Your Own Key for the key store and a save UI.

World Labs jobs that do not wait

Pass modelOptions: { wait: false } to generateWorld(). Then it returns at once with an operationId. This SDK cannot resume that id. See the World Labs adapter.

Edit on GitHub
Live Generation
Generation Hooks