Skip to content
TanStackAI
Media

Text-to-Speech

generateSpeech() sends text to a text-to-speech model and returns audio the browser can play.

Quick setup

This reads a sentence out loud with OpenAI. It takes about 5 minutes.

  1. Install the packages:
shell
npm i @tanstack/ai @tanstack/ai-react @tanstack/ai-openai
  1. Put your key in .env:
shell
OPENAI_API_KEY=sk-...
  1. Add a POST route at /api/generate/speech:
ts
import {
  generateSpeech,
  generationParamsFromRequest,
  toServerSentEventsResponse,
} from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

export async function POST(request: Request) {
  const { input, threadId } = await generationParamsFromRequest('tts', request)

  const stream = generateSpeech({
    adapter: openaiSpeech('tts-1-hd'),
    text: input.text,
    threadId,
    voice: input.voice ?? 'alloy',
    stream: true,
  })
  return toServerSentEventsResponse(stream)
}
  1. Call the route from a component and play the result:
tsx
import { fetchServerSentEvents, useGenerateSpeech } from '@tanstack/ai-react'

export function SpeechPlayer() {
  const { generate, result, isLoading, error } = useGenerateSpeech({
    connection: fetchServerSentEvents('/api/generate/speech'),
    threadId: 'speech',
  })

  return (
    <div>
      <button
        onClick={() => generate({ text: 'Hello, welcome to TanStack AI!' })}
        disabled={isLoading}
      >
        {isLoading ? 'Generating...' : 'Speak'}
      </button>
      {error && <p role="alert">Error: {error.message}</p>}
      {result && (
        <audio
          src={`data:${result.contentType ?? 'audio/mpeg'};base64,${result.audio}`}
          controls
          autoPlay
        />
      )}
    </div>
  )
}

Click Speak. The audio player shows up and plays the sentence.

How it works

  • The server. generateSpeech() sends text and voice to the provider. The result holds the audio as a base64 string, plus its format and contentType.
  • The stream. With stream: true, the route sends the result to the browser as Server-Sent Events.
  • The hook. useGenerateSpeech() stores the result on result. A data: URL turns the base64 audio into something an <audio> element can play.

The hook takes the same input fields as generateSpeech(). They are listed in Generation Hooks. To use a server function instead of an API route, see Transports in full.

Each provider has a fixed set of voices. If none of them fit, create your own voice and pass its id as voice.

Pick a provider

Swap the adapter to change the provider. The rest of the route stays the same.

ProviderAdapterModelsAPI key
OpenAIopenaiSpeech from @tanstack/ai-openaitts-1-hd, tts-1, gpt-4o-mini-ttsOPENAI_API_KEY
Gemini (experimental)geminiSpeech from @tanstack/ai-geminigemini-3.1-flash-tts-preview, gemini-2.5-flash-preview-tts, gemini-2.5-pro-preview-ttsGOOGLE_API_KEY or GEMINI_API_KEY
ElevenLabselevenlabsSpeech from @tanstack/ai-elevenlabseleven_v4, eleven_v4_turbo, eleven_v3, and moreELEVENLABS_API_KEY
BytePlusbyteplusSpeech from @tanstack/ai-byteplusseed-audio-1.0BYTEPLUS_VOICE_API_KEY
fal.aifalSpeech from @tanstack/ai-falKokoro, ElevenLabs, MiniMax, Gemini TTS, Chatterbox, and moreFAL_KEY
GrokgrokSpeech from @tanstack/ai-grokgrok-ttsXAI_API_KEY
GroqgroqSpeech from @tanstack/ai-groqcanopylabs/orpheus-v1-english, canopylabs/orpheus-arabic-saudiGROQ_API_KEY

For example, to use ElevenLabs:

ts
import { generateSpeech } from '@tanstack/ai'
import { elevenlabsSpeech } from '@tanstack/ai-elevenlabs'

const result = await generateSpeech({
  adapter: elevenlabsSpeech('eleven_v4'),
  text: 'Welcome to TanStack AI.',
  threadId: 'thread-1',
})

More provider samples are in Provider examples.

Dialogue with two or more voices

You want a short scene with two speakers. One text with one voice cannot do that. Pass turns instead. Each turn has its own text and its own voice:

ts
import { generateSpeech } from '@tanstack/ai'
import { byteplusSpeech } from '@tanstack/ai-byteplus'

// Replace SECOND_VOICE with another voice id from the BytePlus voice list.
const SECOND_VOICE = 'your-second-voice-id'

const result = await generateSpeech({
  adapter: byteplusSpeech('seed-audio-1.0'),
  turns: [
    {
      text: 'Do you have a left-handed bass?',
      voice: 'en_female_stokie_uranus_bigtts',
    },
    { text: 'We do. Come and try it.', voice: SECOND_VOICE },
  ],
  format: 'mp3',
  threadId: 'thread-1',
})

Send turns or text, not both.

Each adapter declares how many voices it accepts in one request. A request with too many voices fails before it reaches the provider:

AdapterDifferent voices per request
byteplusSpeech3
elevenlabsSpeech10
geminiSpeech2
Other TTS adaptersNo dialogue

Read the limits at runtime from adapter.capabilities:

ts
import { byteplusSpeech } from '@tanstack/ai-byteplus'

const adapter = byteplusSpeech('seed-audio-1.0')

console.log(adapter.capabilities?.maxSpeakers) // 3
console.log(adapter.capabilities?.timestamps) // true

Timings for each word and turn

result.duration is the length of the file. It does not tell you where the speech stops, or where each turn starts. You need that to trim silence at the end or to show captions.

Set timestamps: true to get the timings:

ts
import { generateSpeech } from '@tanstack/ai'
import { byteplusSpeech } from '@tanstack/ai-byteplus'

const result = await generateSpeech({
  adapter: byteplusSpeech('seed-audio-1.0'),
  text: 'Welcome to the guitar store.',
  threadId: 'thread-1',
  timestamps: true,
})

// Where the speech ends. This is earlier than the end of the file.
const speechEnds = result.alignment?.endSeconds.at(-1)

for (const segment of result.segments ?? []) {
  console.log(segment.startSeconds, segment.endSeconds, segment.text)
}

The result has two fields with timings:

  • alignment: one entry per character or per word, with start and end times. alignment.unit tells you which one the provider sent.
  • segments: one entry per turn (dialogue) or per sentence (one voice). A dialogue segment also has turnIndex and voice.

All times are in seconds. If a provider sends milliseconds, the adapter converts them.

An adapter that cannot return timings rejects timestamps. It does not send back an empty result. Read adapter.capabilities?.timestamps before you ask.

Your app now speaks text with the voice and the provider you picked.

Advanced

Provider examples

Gemini (experimental)

ts
import { generateSpeech } from '@tanstack/ai'
import { geminiSpeech } from '@tanstack/ai-gemini'

const result = await generateSpeech({
  adapter: geminiSpeech('gemini-3.1-flash-tts-preview'),
  text: 'Hello from Gemini TTS!',
  threadId: 'thread-1',
})

fal.ai

fal.ai has many TTS models. Pass the model id as a string literal to get typed modelOptions.

ts
import { generateSpeech } from '@tanstack/ai'
import { falSpeech } from '@tanstack/ai-fal'

// Gemini 3.1 Flash TTS: 80+ languages and expressive audio tags
const result = await generateSpeech({
  adapter: falSpeech('fal-ai/gemini-3.1-flash-tts'),
  text: '[warm, enthusiastic] Welcome to TanStack AI!',
  threadId: 'thread-1',
  voice: 'Kore',
})
ts
import { generateSpeech } from '@tanstack/ai'
import { falSpeech } from '@tanstack/ai-fal'

// Kokoro, American English
const result = await generateSpeech({
  adapter: falSpeech('fal-ai/kokoro/american-english'),
  text: 'Hello from fal!',
  threadId: 'thread-1',
  voice: 'af_heart',
  speed: 1.0,
})

console.log(result.format) // for example "wav"

The fal adapter puts the top-level voice and speed into the model input. Use modelOptions only for keys of that one model:

ts
import { generateSpeech } from '@tanstack/ai'
import { falSpeech } from '@tanstack/ai-fal'

const result = await generateSpeech({
  adapter: falSpeech('fal-ai/elevenlabs/tts/eleven-v3'),
  text: 'Welcome to TanStack AI.',
  threadId: 'thread-1',
  voice: 'Rachel',
  modelOptions: {
    stability: 0.5,
  },
})

BytePlus Seed Speech

Seed Speech is a different BytePlus product from ModelArk. It uses its own key, BYTEPLUS_VOICE_API_KEY. The ARK_API_KEY of the chat, image, and video adapters does not work here.

ts
import { generateSpeech } from '@tanstack/ai'
import { byteplusSpeech } from '@tanstack/ai-byteplus'

const result = await generateSpeech({
  adapter: byteplusSpeech('seed-audio-1.0'),
  text: 'Welcome to TanStack AI!',
  threadId: 'thread-1',
  voice: 'en_female_stokie_uranus_bigtts',
  format: 'mp3',
})

console.log(result.contentType) // "audio/mpeg"

Seed Speech limits:

  • The formats are wav, mp3, pcm, and ogg_opus.
  • One call makes at most 120 seconds of audio.

Seed Speech has no top-level speaker field. The adapter sends voice as references: [{ speaker }]. If you set modelOptions.references for voice cloning, it replaces that array, and voice is lost. To keep a stock voice, add a speaker member yourself. The voice id names are in the BytePlus adapter page.

OpenAI voices, formats, and options

OpenAI voices:

VoiceSound
alloyNeutral and balanced. The default.
echoWarm and conversational
fableExpressive, good for stories
onyxDeep and authoritative
novaFriendly and upbeat
shimmerClear and gentle
ashCalm and measured
balladMelodic and flowing
coralBright and energetic
sageWise and thoughtful
versePoetic and rhythmic

OpenAI formats:

FormatUse
mp3General use. The default.
opusStreaming
aacAAC audio
flacLossless audio
wavUncompressed audio, for more processing
pcmRaw PCM audio

OpenAI models:

ModelQualitySpeedUse it for
tts-1StandardFastRealtime apps
tts-1-hdHighSlowerProduction audio
gpt-4o-mini-ttsHighestChangesVoice style control

voice, format, and speed are top-level options of generateSpeech(). They are not modelOptions keys. speed goes from 0.25 to 4.0.

modelOptions.instructions sets the speaking style. Only gpt-4o-mini-tts accepts it. On tts-1 and tts-1-hd, the call throws.

ts
import { generateSpeech } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

const result = await generateSpeech({
  adapter: openaiSpeech('gpt-4o-mini-tts'),
  text: 'High quality speech synthesis',
  threadId: 'thread-1',
  voice: 'nova',
  format: 'mp3',
  speed: 1.0,
  modelOptions: {
    instructions: 'Speak in a calm, measured tone',
  },
})

OpenAI accepts at most 4096 characters of text in one request. For longer text, split it into parts.

The fixed voice lists are also exported as types: OpenAITTSVoice from @tanstack/ai-openai, and GeminiTTSVoices from @tanstack/ai-gemini.

Result shape

ts
import type { PersistedArtifactRef, TokenUsage, TTSAlignment, TTSSegment } from '@tanstack/ai'

interface TTSResult {
  id: string // id of this generation
  model: string // the model that made the audio
  audio: string // base64-encoded audio
  format: string // audio format, for example "mp3"
  duration?: number // length in seconds, if the provider reports it
  alignment?: TTSAlignment // with timestamps: true
  segments?: Array<TTSSegment> // with timestamps: true
  contentType?: string // MIME type, for example "audio/mpeg"
  usage?: TokenUsage // token usage, if the adapter reports it
  artifacts?: Array<PersistedArtifactRef> // stored files, with persistence
}

Play audio without a data URL

A long clip makes a long data: URL. A Blob URL is lighter. Convert the base64 audio to bytes, then play it:

ts
import type { TTSResult } from '@tanstack/ai'

function playAudio(result: TTSResult) {
  const audioData = atob(result.audio)
  const bytes = new Uint8Array(audioData.length)
  for (let i = 0; i < audioData.length; i++) {
    bytes[i] = audioData.charCodeAt(i)
  }

  const blob = new Blob([bytes], { type: result.contentType })
  const url = URL.createObjectURL(blob)

  const audio = new Audio(url)
  void audio.play()
  audio.onended = () => URL.revokeObjectURL(url)
}

To do this once per result, put the conversion in onResult. See Result transforms.

Save audio to a file (Node.js)

ts
import { writeFile } from 'node:fs/promises'
import { generateSpeech } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

const result = await generateSpeech({
  adapter: openaiSpeech('tts-1-hd'),
  text: 'Hello world!',
  threadId: 'thread-1',
})

await writeFile('output.mp3', Buffer.from(result.audio, 'base64'))

Errors

The OpenAI adapter checks the input before it calls the API:

ts
import { generateSpeech } from '@tanstack/ai'
import { openaiSpeech } from '@tanstack/ai-openai'

try {
  await generateSpeech({
    adapter: openaiSpeech('tts-1-hd'),
    text: 'Hello!',
    threadId: 'thread-1',
  })
} catch (error) {
  if (error instanceof Error) {
    if (error.message.includes('exceeds maximum length')) {
      console.error('Text is too long (max 4096 characters)')
    } else if (error.message.includes('Speed must be between')) {
      console.error('Invalid speed value')
    } else {
      console.error('TTS error:', error.message)
    }
  }
}

If a request fails or the audio sounds wrong, pass debug: true to generateSpeech(). It logs the request, every raw chunk from the provider, and any error. See Debug Logging.

Explicit API keys

Each adapter reads its key from the environment. To pass the key yourself, use the create* factory:

ts
import { createOpenaiSpeech } from '@tanstack/ai-openai'
import { createGeminiSpeech } from '@tanstack/ai-gemini'

const openaiAdapter = createOpenaiSpeech('tts-1-hd', 'your-openai-api-key')

const geminiAdapter = createGeminiSpeech(
  'gemini-3.1-flash-tts-preview',
  'your-google-api-key',
)

Tips

  • Cache the audio. Store each result, so that you do not generate the same text again.
  • Pick a voice for the content. For example, onyx fits authoritative content, and nova fits friendly messages.
  • Pick a format for the use. mp3 is for general use, opus is for streaming, and wav is for more processing.
Edit on GitHub
Realtime Voice Chat
Transcription