A model that takes eight seconds to answer feels broken if the screen sits blank, and instant if the words start appearing in one. Same latency, completely different experience. The fix is streaming the tokens as they generate and appending them to React state, so the answer types itself out.

Stream from the server

The route handler returns a ReadableStream instead of a finished string. As the model emits tokens, they are enqueued and flushed to the client immediately.

export async function POST(req: Request) {
  const stream = new ReadableStream({
    async start(controller) {
      for await (const chunk of model.stream(prompt)) {
        controller.enqueue(new TextEncoder().encode(chunk));
      }
      controller.close();
    },
  });
  return new Response(stream);
}

The first token can reach the browser in a few hundred milliseconds even when the full answer takes many seconds, and that first token is what kills the feeling of waiting.

Read and append on the client

The client reads the stream chunk by chunk and appends each piece to state. React re-renders on every append, so the text grows in place.

const res = await fetch("/api/chat", { method: "POST", body });
const reader = res.body!.getReader();
const decoder = new TextDecoder();
let text = "";
for (;;) {
  const { done, value } = await reader.read();
  if (done) break;
  text += decoder.decode(value, { stream: true });
  setMessage(text);
}

decoder.decode(value, { stream: true }) matters because a multibyte character can be split across two chunks, and decoding each chunk in isolation would garble it.

Let the user stop

A streaming request the user cannot cancel is a bad citizen. An AbortController wired to a stop button cancels the fetch, and the server should notice the disconnect and stop spending tokens on an answer nobody is reading.

const controller = new AbortController();
fetch("/api/chat", { signal: controller.signal, /* ... */ });
// stop button: controller.abort();

Handle the failure mid-stream

A stream can die halfway through, leaving a half-finished sentence. The client needs to catch a read error and show that the response was interrupted rather than pretending the partial text is complete. I keep the partial content and mark it as incomplete, so the user sees what arrived and knows it was cut off.

The whole pattern is a server that flushes tokens, a client that appends them, a way to cancel, and honest handling when it breaks. None of it is exotic, and together it turns a slow model into an interface that feels alive while it thinks.