Systems | Development | Analytics | API | Testing

HTTP streaming – the default transport underneath every major agent framework – was never designed for sessions that survive a tab close or hand off cleanly between participants. Two failures surface consistently in production CX products because of this. Both generate support tickets about conversation state and prompt quality. Both trace to the transport layer. The scenario that illustrates them: a customer contacts support about an order that's partially shipped and partially stuck.

Every major layer of the AI stack now has a name. Model providers - OpenAI, Anthropic, Google - handle inference. Agent frameworks - Vercel AI SDK, LangGraph, CrewAI - handle orchestration. Durable execution platforms like Temporal make backend workflows crash-proof.

There’s no shortage of noise in AI right now. New frameworks, protocols, demos, and acronyms appear almost weekly. But when you speak directly to the teams actually shipping AI to users at scale, a different picture emerges. This is what we've learned over the last few months from speaking to CTOs, AI engineering leads, and product leaders from unicorns, public companies, and fast-growing platforms across industries where humans interact directly with AI.

Shared state is a hard problem. Not hard in the abstract, computer-science sense (the concepts are well understood). Hard in the someone has to actually build this sense, where every team that wants a live leaderboard, a shared config panel, or a poll that updates in real time ends up reinventing the same wheels: conflict resolution, reconnection handling, state recovery. Most teams do not want to spend their time building and maintaining that layer. They want to ship the feature that depends on it.

Most AI applications start the same way: wire up an LLM, stream tokens to the browser, ship. That works for simple request-response. It breaks when sessions outlast a connection, when users switch devices, or when an agent needs to hand off to a human. The cracks appear in the delivery layer, not the model. Every serious production team discovers this independently and builds their own workaround. Those workarounds don't hold once users start hitting them in production.

When an AI response reaches token 150 and the connection drops, most implementations have one answer: start over. The user re-prompts, you pay for the same tokens twice, and the experience breaks. Resume tokens and last-event IDs are the mechanism that prevents this. They make streams addressable – every message gets an identifier, clients track their position, and reconnections pick up from exactly where they left off. The concept is straightforward.

Building AI agents that work reliably in production requires solving problems that have nothing to do with AI. While teams focus on prompt engineering, model selection, and agent orchestration, a different class of challenges emerges at deployment. These have little to do with LLMs and everything to do with keeping agents and clients synchronized in realtime. Over the past few months, we've spoken with engineers at over 40 companies building AI assistants, copilots, and agentic workflows.

Your AI assistant is mid-sentence explaining a complex debugging strategy. The user refreshes the page. The response starts over from the beginning, or worse, vanishes entirely. This isn't a model problem. It's a delivery problem.