|
| 1 | +--- |
| 2 | +title: Generations |
| 3 | +id: generations |
| 4 | +order: 13 |
| 5 | +--- |
| 6 | + |
| 7 | +# Generations |
| 8 | + |
| 9 | +TanStack AI provides a unified pattern for non-chat AI activities: **image generation**, **text-to-speech**, **transcription**, **summarization**, and **video generation**. These are collectively called "generations" — single request/response operations (as opposed to multi-turn chat). |
| 10 | + |
| 11 | +All generations follow the same architecture, making it easy to learn one and apply it to the rest. |
| 12 | + |
| 13 | +## Architecture |
| 14 | + |
| 15 | +```mermaid |
| 16 | +flowchart TB |
| 17 | + subgraph Server ["Server"] |
| 18 | + direction TB |
| 19 | + activities["generateImage({ ..., stream: true }) |
| 20 | +generateSpeech({ ..., stream: true }) |
| 21 | +generateTranscription({ ..., stream: true }) |
| 22 | +summarize({ ..., stream: true }) |
| 23 | +generateVideo({ ..., stream: true })"] |
| 24 | + transport["toServerSentEventsResponse()"] |
| 25 | + activities --> transport |
| 26 | + end |
| 27 | +
|
| 28 | + transport -- "StreamChunks via SSE |
| 29 | +RUN_STARTED → generation:result → RUN_FINISHED" --> adapter |
| 30 | +
|
| 31 | + subgraph Client ["Client"] |
| 32 | + direction TB |
| 33 | + adapter["fetchServerSentEvents('/api/...')"] |
| 34 | + gc["GenerationClient |
| 35 | +(state machine)"] |
| 36 | + hooks["Framework Hooks |
| 37 | +useGenerateImage() · useGenerateSpeech() |
| 38 | +useGenerateVideo() · useSummarize() |
| 39 | +useTranscription()"] |
| 40 | + adapter --> gc |
| 41 | + gc -- "result, isLoading, error, status" --> hooks |
| 42 | + end |
| 43 | +``` |
| 44 | + |
| 45 | +The key insight: **every generation activity on the server is just an async function that returns a result**. By passing `stream: true`, the function returns a `StreamChunk` iterable instead of a plain result, which the client already knows how to consume. |
| 46 | + |
| 47 | +## Three Transport Modes |
| 48 | + |
| 49 | +### Streaming Mode (Connection Adapter) |
| 50 | + |
| 51 | +The server passes `stream: true` to the generation function and sends the result as SSE. The client uses `fetchServerSentEvents()` to consume the stream. |
| 52 | + |
| 53 | +**Server:** |
| 54 | + |
| 55 | +```typescript |
| 56 | +import { generateImage, toServerSentEventsResponse } from '@tanstack/ai' |
| 57 | +import { openaiImage } from '@tanstack/ai-openai' |
| 58 | + |
| 59 | +// In your API route handler |
| 60 | +const stream = generateImage({ |
| 61 | + adapter: openaiImage('dall-e-3'), |
| 62 | + prompt: 'A sunset over mountains', |
| 63 | + stream: true, |
| 64 | +}) |
| 65 | + |
| 66 | +return toServerSentEventsResponse(stream) |
| 67 | +``` |
| 68 | + |
| 69 | +**Client:** |
| 70 | + |
| 71 | +```tsx |
| 72 | +import { useGenerateImage } from '@tanstack/ai-react' |
| 73 | +import { fetchServerSentEvents } from '@tanstack/ai-client' |
| 74 | + |
| 75 | +const { generate, result, isLoading } = useGenerateImage({ |
| 76 | + connection: fetchServerSentEvents('/api/generate/image'), |
| 77 | +}) |
| 78 | +``` |
| 79 | + |
| 80 | +### Direct Mode (Fetcher) |
| 81 | + |
| 82 | +The client calls a server function directly and receives the result as JSON. No streaming protocol needed. |
| 83 | + |
| 84 | +**Server:** |
| 85 | + |
| 86 | +```typescript |
| 87 | +import { createServerFn } from '@tanstack/react-start' |
| 88 | +import { generateImage } from '@tanstack/ai' |
| 89 | +import { openaiImage } from '@tanstack/ai-openai' |
| 90 | + |
| 91 | +export const generateImageFn = createServerFn({ method: 'POST' }) |
| 92 | + .inputValidator((data: { prompt: string }) => data) |
| 93 | + .handler(async ({ data }) => { |
| 94 | + return generateImage({ |
| 95 | + adapter: openaiImage('dall-e-3'), |
| 96 | + prompt: data.prompt, |
| 97 | + }) |
| 98 | + }) |
| 99 | +``` |
| 100 | + |
| 101 | +**Client:** |
| 102 | + |
| 103 | +```tsx |
| 104 | +import { useGenerateImage } from '@tanstack/ai-react' |
| 105 | +import { generateImageFn } from '../lib/server-functions' |
| 106 | + |
| 107 | +const { generate, result, isLoading } = useGenerateImage({ |
| 108 | + fetcher: (input) => generateImageFn({ data: input }), |
| 109 | +}) |
| 110 | +``` |
| 111 | + |
| 112 | +### Server Function Streaming (Fetcher + Response) |
| 113 | + |
| 114 | +Combines the best of both: **type-safe input** from the fetcher pattern with **streaming** from a server function that returns an SSE `Response`. When the fetcher returns a `Response` object (instead of a plain result), the client automatically parses it as an SSE stream. |
| 115 | + |
| 116 | +**Server:** |
| 117 | + |
| 118 | +```typescript |
| 119 | +import { createServerFn } from '@tanstack/react-start' |
| 120 | +import { generateImage, toServerSentEventsResponse } from '@tanstack/ai' |
| 121 | +import { openaiImage } from '@tanstack/ai-openai' |
| 122 | + |
| 123 | +export const generateImageStreamFn = createServerFn({ method: 'POST' }) |
| 124 | + .inputValidator((data: { prompt: string }) => data) |
| 125 | + .handler(({ data }) => { |
| 126 | + return toServerSentEventsResponse( |
| 127 | + generateImage({ |
| 128 | + adapter: openaiImage('dall-e-3'), |
| 129 | + prompt: data.prompt, |
| 130 | + stream: true, |
| 131 | + }), |
| 132 | + ) |
| 133 | + }) |
| 134 | +``` |
| 135 | + |
| 136 | +**Client:** |
| 137 | + |
| 138 | +```tsx |
| 139 | +import { useGenerateImage } from '@tanstack/ai-react' |
| 140 | +import { generateImageStreamFn } from '../lib/server-functions' |
| 141 | + |
| 142 | +const { generate, result, isLoading } = useGenerateImage({ |
| 143 | + fetcher: (input) => generateImageStreamFn({ data: input }), |
| 144 | +}) |
| 145 | +``` |
| 146 | + |
| 147 | +This is the recommended approach when using TanStack Start server functions — the input is fully typed (e.g., `ImageGenerateInput`), and the streaming protocol is handled transparently. |
| 148 | + |
| 149 | +## How Streaming Works |
| 150 | + |
| 151 | +When you pass `stream: true` to any generation function, it returns an async iterable of `StreamChunk` events instead of a plain result: |
| 152 | + |
| 153 | +``` |
| 154 | +1. RUN_STARTED → Client sets status to 'generating' |
| 155 | +2. CUSTOM → Client receives the result |
| 156 | + name: 'generation:result' |
| 157 | + value: <your result> |
| 158 | +3. RUN_FINISHED → Client sets status to 'success' |
| 159 | +``` |
| 160 | + |
| 161 | +If the function throws, a `RUN_ERROR` event is emitted instead: |
| 162 | + |
| 163 | +``` |
| 164 | +1. RUN_STARTED → Client sets status to 'generating' |
| 165 | +2. RUN_ERROR → Client sets error + status to 'error' |
| 166 | + error: { message: '...' } |
| 167 | +``` |
| 168 | + |
| 169 | +This is the same event protocol used by chat streaming, so the same transport layer (`toServerSentEventsResponse`, `fetchServerSentEvents`) works for both. |
| 170 | + |
| 171 | +## Common Hook API |
| 172 | + |
| 173 | +All generation hooks share the same interface: |
| 174 | + |
| 175 | +| Option | Type | Description | |
| 176 | +|--------|------|-------------| |
| 177 | +| `connection` | `ConnectionAdapter` | Streaming transport (SSE, HTTP stream, custom) | |
| 178 | +| `fetcher` | `(input) => Promise<Result \| Response>` | Direct async function, or server function returning an SSE `Response` | |
| 179 | +| `id` | `string` | Unique identifier for this instance | |
| 180 | +| `body` | `Record<string, any>` | Additional body parameters (connection mode) | |
| 181 | +| `onResult` | `(result) => T \| null \| void` | Transform or react to the result | |
| 182 | +| `onError` | `(error) => void` | Error callback | |
| 183 | +| `onProgress` | `(progress, message?) => void` | Progress updates (0-100) | |
| 184 | + |
| 185 | +| Return | Type | Description | |
| 186 | +|--------|------|-------------| |
| 187 | +| `generate` | `(input) => Promise<void>` | Trigger generation | |
| 188 | +| `result` | `T \| null` | The result (optionally transformed), or null | |
| 189 | +| `isLoading` | `boolean` | Whether generation is in progress | |
| 190 | +| `error` | `Error \| undefined` | Current error, if any | |
| 191 | +| `status` | `GenerationClientState` | `'idle'` \| `'generating'` \| `'success'` \| `'error'` | |
| 192 | +| `stop` | `() => void` | Abort the current generation | |
| 193 | +| `reset` | `() => void` | Clear all state, return to idle | |
| 194 | + |
| 195 | +### Result Transform |
| 196 | + |
| 197 | +The `onResult` callback can optionally transform the stored result: |
| 198 | + |
| 199 | +- Return a **non-null value** — replaces the stored result with the transformed value |
| 200 | +- Return **`null`** — keeps the previous result unchanged (useful for filtering) |
| 201 | +- Return **nothing** (`void`) — stores the raw result as-is |
| 202 | + |
| 203 | +TypeScript automatically infers the result type from your `onResult` return value — no explicit generic parameter needed. |
| 204 | + |
| 205 | +```tsx |
| 206 | +const { result } = useGenerateSpeech({ |
| 207 | + connection: fetchServerSentEvents('/api/generate/speech'), |
| 208 | + onResult: (raw) => ({ |
| 209 | + audioUrl: `data:${raw.contentType};base64,${raw.audio}`, |
| 210 | + duration: raw.duration, |
| 211 | + }), |
| 212 | +}) |
| 213 | +// result is typed as { audioUrl: string; duration?: number } | null |
| 214 | +``` |
| 215 | + |
| 216 | +## Available Generations |
| 217 | + |
| 218 | +| Activity | Server Function | Client Hook (React) | Guide | |
| 219 | +|----------|----------------|---------------------|-------| |
| 220 | +| Image generation | `generateImage()` | `useGenerateImage()` | [Image Generation](./image-generation) | |
| 221 | +| Text-to-speech | `generateSpeech()` | `useGenerateSpeech()` | [Text-to-Speech](./text-to-speech) | |
| 222 | +| Transcription | `generateTranscription()` | `useTranscription()` | [Transcription](./transcription) | |
| 223 | +| Summarization | `summarize()` | `useSummarize()` | - | |
| 224 | +| Video generation | `generateVideo()` | `useGenerateVideo()` | [Video Generation](./video-generation) | |
| 225 | + |
| 226 | +> **Note:** Video generation uses a jobs/polling architecture. The `useGenerateVideo` hook additionally exposes `jobId`, `videoStatus`, `onJobCreated`, and `onStatusUpdate` for tracking the polling lifecycle. See the [Video Generation](./video-generation) guide for details. |
![autofix-ci[bot]](https://avatars.githubusercontent.com/in/243519?v=4&size=40){kind=avatar}
0 commit comments