An assistant can produce an accurate response and still be unusable. With a support tool, a team does not judge a model's quality during a prepared demonstration; it judges it when a customer is waiting, an employee is looking for a procedure, or an agent needs information before proceeding with a case. At that point, “it's slow” becomes a business problem. But first, you need to know what that means.
This phrase often refers to five different delays: preparing the request, searching a document repository, calling the model, calling third-party software, and then displaying the response. Confusing them leads to buying a more powerful model to compensate for a slow connector, or shortening a prompt when the time is actually being spent in a queue. The first task is therefore to make the pipeline visible, not to start a benchmark race.
OpenAI's documentation on latency optimization offers a useful distinction: actual system time and the time perceived by the user do not always coincide. Streaming can make the first words appear sooner without speeding up the full process. It improves the experience, but fixes neither poorly configured document retrieval nor an unavailable dependency.
A slow response is often a series of small delays
Consider an internal assistant that answers questions using a procedure repository. Before the model starts writing, the application may identify the user, check their permissions, turn their question into a search query, retrieve passages, possibly rerank them, assemble the instruction, and call the model. If the assistant can then open a ticket or consult the CRM, each tool adds its own delay. The final response conceals this machinery.
The model matters, of course. Model size, the number of tokens it must generate, and infrastructure load all affect response time. But a long response generally has more impact than a few additional lines in the instruction. The same documentation recommends reducing unnecessary generation first, then limiting sequential requests. An assistant that reformulates a response three times before displaying it is slow by design, even if each of its steps is correct.
Context calls for a less instinctive approach. Putting the entire document repository or the full history of an exchange into every request may seem reassuring: the information will be in there somewhere. In practice, this increases the volume being processed and makes it harder to select what is relevant. Anthropic's guidance on long contexts emphasizes the organization and placement of documents in a long request. The context window is not permission to pour everything into the model.
This is particularly important for a custom AI development project. A useful document repository is not about giving the system more text. It requires selecting the sources that can be consulted, dividing content coherently, filtering according to permissions, and retaining the passages actually used. This discipline improves speed, but also the ability to explain where a response came from.
Measuring before fixing prevents false economies
To understand a slowdown, separate timings must be recorded: receipt of the request, retrieval, any reranking, context construction, first token, end of generation, tool calls, and final response. They must be observed for common cases as well as difficult ones. A flattering average says nothing about a spike that leaves users unable to work; percentiles, particularly p95, provide a more honest picture of the experience.
Measurements must be matched to the type of request. A simple question about a stable rule can be resolved with a prepared answer or direct search. A request that requires cross-referencing a procedure, a case file, and a status in the ERP follows a different path. Handling both with the same model, the same context, and the same number of calls is convenient for the architecture, but rarely for the user.
Reducing the number of steps can be more effective than optimizing each one. When two classifications are independent, they can be run in parallel. When a check is necessary only for a minority of cases, it should not block every request. Conversely, an action with consequences should not be accelerated by removing the control that makes it safe. Useful speed is speed that respects the workflow, not speed that produces the first text as quickly as possible.
The NIST AI Risk Management Framework reminds us that evaluating a system is not limited to its model. For an assistant, this also applies to source availability, access permissions, tool errors, and the ability to fall back to a degraded response. If the CRM does not respond, the interface should be able to say that it cannot verify a piece of information, rather than suggesting that the model is still thinking.
Designing an acceptable response time for each use case
The right target depends on the work. For an employee preparing a summary, a slower but properly sourced response may be acceptable. For a customer service agent looking for a procedure during a call, waiting does not have the same cost. For a system that triggers an action, speed comes after the ability to verify and stop the process.
This distinction changes how AI and automation agents are designed. An agent can prepare the information needed for a case in the background, then present a proposal for approval. It does not necessarily need to respond instantly or act on the first result. This saves time where it matters, without turning every delay into a risk.
Before replacing a model, a company can therefore ask three simple questions: at which stage does the delay arise, which cases actually suffer from it, and what fallback response remains honest when the system cannot respond quickly? AI scoping serves precisely to connect these measurements to a use case and a credible service level. The result is not an assistant that displays a promise of speed. It is a tool whose team knows why it waits, when it should wait, and when it should be told to stop.