The hardest part of an AI support desk is not the answering. It is knowing when to stop answering. A confident wrong answer costs more trust than saying nothing, so the whole system is built around one number: a confidence score attached to every AI reply.

The confidence gate

Each model response comes back with a score between 0 and 1. Below 0.6, or when the user types something like "talk to a person", the conversation is flagged for handoff. The gate runs server side, never in the browser, so a user cannot skip it by editing a request.

function shouldHandoff(reply: AiReply, message: string): boolean {
  if (reply.confidence < 0.6) return true;
  if (/human|agent|real person|speak to someone/i.test(message)) return true;
  return false;
}

The threshold is not magic. I started at 0.5, watched a week of transcripts, and moved it to 0.6 because the band between 0.5 and 0.6 was where most of the polite-but-wrong answers lived.

Inheriting context

When the gate trips, the agent does not start from zero. The conversation, the customer record, and a short AI-written summary of what has happened so far all land in the agent console at once. The customer never repeats themselves, which is the single thing people hate most about support.

Memory is scoped per user and persisted in Postgres, keyed by conversation id. The AI reads the last N turns plus a rolling summary, so a long thread does not blow the context window. When a human replies, their messages are written to the same table with an author of agent, so the history stays in one place.

Presence and routing

A conversation that needs a human goes into a queue, not a void. Online agents are tracked over a WebSocket connection, and the queue assigns the waiting conversation to whoever has the fewest active threads. If no one is online, the customer gets an honest message and the thread is held for the next agent who logs in.

What it feels like

The result is a desk that answers instantly when the model is sure and routes to a person the moment it is not. Customers get speed without the usual cost of a bot that bluffs. Agents get context instead of a cold open. The score is doing the quiet work in the middle, and tuning that one number moved satisfaction more than any prompt change I tried.