How to Reduce LLM Latency: Caching and Edge Strategies

Reducing LLM latency is one of the most critical challenges for engineers building responsive AI applications. While Large Language Models (LLMs) keep growing in capability, their token-by-token generation can create frustrating bottlenecks for end users, and long wait times lead directly to lower engagement and application dropouts. Optimising your inference pipelines for speed is therefore a core developer requirement. This guide outlines how to configure prompt caching, implement response streaming, structure edge network routing, and use serverless configurations to cut processing delays.

[!TIP] Performance Metric Tip: When measuring API delays, isolate Time to First Token (TTFT) from overall generation speed. A low TTFT makes an application feel instant to the user, even if the total output generation takes several seconds, because text starts rendering immediately.

Key Takeaways:

• Prompt Caching: Reuse static prefix headers to bypass parsing states and reduce TTFT by 80%.
• Response Streaming: Push tokens via Server-Sent Events (SSE) so users see instant text generation.
• Edge Workers: Run authorisation and request routing at regional edge centres close to users.
• Model Routing: Divert straightforward user requests to lightweight models to optimise speeds.

The Components of LLM API Latency

To decrease response times, you must first understand what elements dictate overall API delay. Overall latency is the cumulative total of three distinct variables.

First, network transit time measures how long a request takes to travel from the client to your server, and then on to the model provider’s API. This makes transit distance a major bottleneck.

Second, Time to First Token (TTFT) represents the duration between the model receiving the request and generating its first output token, which is why prompt caching matters so much.

Finally, token generation speed measures the rate at which the hardware outputs subsequent tokens. Hardware constraints dictate generation speed, but developers retain full control over transit time and TTFT, so smart routing and caching can drastically reduce LLM latency.

Prerequisites

Before you start wiring up the optimisations below, make sure you have the following in place. None of them are exotic, but skipping one tends to cause confusing failures later.

• A middleware or edge runtime you control. The examples use Cloudflare Workers, but any serverless platform that can proxy a request works. You need somewhere to sit between the browser and the model provider.
• API credentials for a provider that supports streaming and caching. OpenAI and Anthropic both do. Store the key as a secret (a Wrangler secret or environment variable), never in client-side code.
• Node.js 18 or later if you want to test Workers locally with wrangler dev. The global fetch and ReadableStream APIs used throughout are available in that runtime and in modern browsers.
• A baseline measurement. Capture your current Time to First Token and total response time before changing anything, so you can prove each optimisation actually helped. The benchmarking section below shows how.
• Familiarity with Server-Sent Events (SSE). Streaming responses arrive as a sequence of data: lines, and you will parse them on the client.

Implementing Prompt Caching

Prompt caching is the most effective way to optimise TTFT for applications built around large system prompts. When a request contains a long static instruction block (such as an agent’s system prompt or a RAG reference document), the model provider must parse and encode those tokens on every execution. Both Anthropic and OpenAI support prompt caching, which saves the parsed token states in memory. Subsequent requests that share the same prefix then bypass the parsing stage, reducing TTFT by up to 80%.

Cache lifetimes vary between providers. Anthropic maintains the cache for roughly five minutes of inactivity, whereas OpenAI uses a dynamic decay model. Scheduling regular background fetch pings can therefore keep critical system instructions active in server memory.

The mechanism differs slightly between the two providers, and getting the request structure right is what determines whether the cache actually engages. With Anthropic, you mark a cache breakpoint explicitly using cache_control. Everything before the breakpoint is stored, so the stable, static content must come first and the volatile per-request content must come last:

 1import Anthropic from "@anthropic-ai/sdk";
 2
 3const anthropic = new Anthropic();
 4
 5const response = await anthropic.messages.create({
 6  model: "claude-opus-4-8",
 7  max_tokens: 1024,
 8  system: [
 9    {
10      type: "text",
11      text: SYSTEM_INSTRUCTIONS       // small, sent on every request
12    },
13    {
14      type: "text",
15      text: KNOWLEDGE_BASE,           // large, static reference block
16      cache_control: { type: "ephemeral" }
17    }
18  ],
19  messages: [
20    { role: "user", content: userQuestion }   // volatile — after the breakpoint
21  ]
22});
23
24// Confirm the cache is working
25console.log(response.usage.cache_read_input_tokens);

The single most common mistake is placing a timestamp, request ID, or any per-request string ahead of the cached block. Because caching is a prefix match, one changed byte anywhere before the breakpoint invalidates everything after it, and the cache silently never hits. Verify it is working by reading usage.cache_read_input_tokens from the response: if that stays at zero across identical requests, something dynamic has crept into the prefix. Note too that the cached prefix must clear a minimum length (in the region of 1,024 to 4,096 tokens depending on the model) before caching engages at all.

OpenAI takes a simpler approach: caching is automatic for prompts above roughly 1,024 tokens, with no cache_control flag to set. The same discipline still applies, though. Keep the static instruction block at the very start of your messages array and append the changing user input at the end, so the reusable prefix stays byte-for-byte identical between requests.

To see the pricing and parameter structures of prompt caching, consult the Anthropic Prompt Caching Guide .

Edge Compute and Serverless Routing

Processing LLM requests on a single centralised server introduces massive network hops for global users. Deploying your API middleware on serverless edge networks (like Cloudflare Workers) shortens those paths dramatically.

The edge worker receives the client request, authorises the session, and routes it to the closest model provider datacentre. This serverless structure delivers tokens to the user’s screen the moment they are computed, so the interface feels highly responsive. The JavaScript middleware below demonstrates how to configure streaming responses directly from an edge runtime:

 1export default {
 2  async fetch(request, env) {
 3    const payload = await request.json();
 4
 5    // Call the streaming LLM endpoint
 6    const response = await fetch("https://api.openai.com/v1/chat/completions", {
 7      method: "POST",
 8      headers: {
 9        "Authorization": `Bearer ${env.OPENAI_API_KEY}`,
10        "Content-Type": "application/json"
11      },
12      body: JSON.stringify({
13        model: "gpt-4o-mini",
14        messages: payload.messages,
15        stream: true
16      })
17    });
18
19    // Forward the stream directly to the client browser
20    return new Response(response.body, {
21      headers: { "Content-Type": "text/event-stream" }
22    });
23  }
24};

This serverless structure delivers tokens to the user’s screen instantly as they are computed. To learn how to build edge-optimised backends, read our guide on building a serverless API with Cloudflare Workers .

Parsing the Stream on the Client

Forwarding the stream from the edge is only half the job. The browser still has to read those chunks as they arrive and render each token, otherwise the response accumulates in a buffer and appears all at once, defeating the point. This is the step most tutorials skip, and it is where the perceived performance is actually won or lost.

The response body is a ReadableStream of raw bytes. SSE frames arrive as data: lines, but a single network chunk can contain several frames, or split one frame across two chunks, so you must buffer partial lines rather than assuming each chunk is a complete message:

 1async function streamCompletion(messages, onToken) {
 2  const response = await fetch("/api/chat", {
 3    method: "POST",
 4    headers: { "Content-Type": "application/json" },
 5    body: JSON.stringify({ messages })
 6  });
 7
 8  const reader = response.body.getReader();
 9  const decoder = new TextDecoder();
10  let buffer = "";
11
12  while (true) {
13    const { value, done } = await reader.read();
14    if (done) break;
15
16    buffer += decoder.decode(value, { stream: true });
17    const lines = buffer.split("\n");
18    buffer = lines.pop();              // keep the trailing partial line
19
20    for (const line of lines) {
21      if (!line.startsWith("data: ")) continue;
22      const payload = line.slice(6).trim();
23      if (payload === "[DONE]") return;
24
25      try {
26        const json = JSON.parse(payload);
27        const token = json.choices?.[0]?.delta?.content;
28        if (token) onToken(token);
29      } catch {
30        // ignore keep-alive comments and malformed partial frames
31      }
32    }
33  }
34}

The buffer.split("\n") followed by lines.pop() is the important detail: it retains any incomplete line until the next chunk completes it. Wrapping JSON.parse in a try/catch keeps the loop alive when a keep-alive comment or a half-received frame arrives. The onToken callback then appends each fragment to the DOM, so text appears the instant the model produces it.

Measuring and Benchmarking Latency

You cannot optimise what you have not measured. Before and after each change, capture Time to First Token and total generation time so you can attribute an improvement to the right cause. The quickest way to sample TTFT is with curl, using time_starttransfer as a close proxy for the first byte reaching the client:

1curl -w "TTFT: %{time_starttransfer}s\nTotal: %{time_total}s\n" \
2  -X POST https://your-worker.example.com/chat \
3  -H "Content-Type: application/json" \
4  -d '{"messages":[{"role":"user","content":"Hello"}]}' \
5  -o /dev/null -s

For application-level numbers, instrument the client parser directly. Stamp the clock when the request leaves and again when the first token lands:

1const start = performance.now();
2let firstTokenAt = null;
3
4await streamCompletion(messages, (token) => {
5  if (firstTokenAt === null) firstTokenAt = performance.now();
6  render(token);
7});
8
9console.log(`TTFT: ${Math.round(firstTokenAt - start)}ms`);

Run each measurement several times and take the median rather than a single sample, since network variance and cold starts can distort one-off readings. The table below gives illustrative ranges for where the time typically goes on a well-behaved global request; treat them as a shape to compare against, not fixed figures, because your own numbers will depend on region, model, and prompt size.

The two rows worth staring at are the cached versus cold TTFT figures. That gap is the single largest win available to most applications, which is why prompt caching sits at the top of the optimisation list. Generation speed, by contrast, is fixed by the provider, so routing simpler requests to a smaller model is the only lever there.

Step-by-Step Optimisation Workflow

To optimise your software application speed, start by separating static system instructions from dynamic user inputs. This division lets you target cache entry points cleanly.

Next, activate prompt caching flags inside your API payloads to ensure the model provider saves your text tokens in memory.

Always configure response streaming using standard SSE endpoints. By writing lightweight frontend parsers to process chunks as they arrive, you improve user-perceived performance. Then establish model fallback paths: route basic customer inputs to smaller models, reserving larger reasoning models for advanced tasks. Finally, profile network hops to confirm serverless workers reduce transit delays. For details on database optimisation, check our guide on WordPress vs custom web development .

Common Pitfalls and Troubleshooting

A handful of failures come up again and again when teams first roll out streaming and caching. Recognising the symptom saves hours of guesswork.

A subtler trap is buffering at the edge itself. If you await response.text() inside the worker before returning, you have quietly turned a streaming response back into a blocking one. Always pass the stream body straight through. Equally, watch worker CPU limits: heavy per-request work in the middleware adds directly to TTFT, so keep authorisation and routing logic minimal and defer anything expensive.

Production Considerations

Getting a demo streaming in a browser is straightforward; running it reliably under real traffic needs a few more guards.

Set a sensible request timeout on the upstream call so a stalled provider connection cannot hold a worker open indefinitely, and pair it with a retry that falls back to a second provider or a smaller model when the primary times out. Because prompt caching charges a small premium on cache writes and a large discount on reads, it only pays off when a prefix is reused; a background ping every few minutes keeps a hot system prompt resident without paying to rewrite it on every user request.

Instrument continuously rather than only at launch. Log TTFT and tokens-per-second per request and alert when the median drifts, since a provider-side regression or a change in prompt size will show up there first. Finally, respect provider rate limits: a burst of concurrent streams can trip them, so queue or shed load gracefully instead of letting requests fail silently. These measures turn a fast prototype into an application that stays fast when it matters.

Key Takeaways

• Target Time to First Token (TTFT) and transit time to reduce LLM latency.
• Leverage prompt caching on model APIs to bypass systemic instruction parsing overhead.
• Use response streaming to deliver tokens in real-time, improving perceived speed.
• Deploy API middleware on serverless edge runtimes to shorten global network paths.
• Route simpler user queries to lightweight models to optimise execution speeds.
• Set up performance monitoring tools to continuously analyse and reduce latency under real user conditions.

Frequently Asked Questions

How do I reduce LLM latency in production? To reduce LLM latency in production, you should implement prompt caching for static instructions, enable token streaming, and deploy edge workers to optimise request routing. By deploying serverless orchestration runtimes closer to global clients, developers bypass multiple network routing hops and deliver the first response token in real-time.

What is prompt caching? Prompt caching is an API feature that stores parsed text states in server memory, allowing subsequent requests using the same prefix to run much faster. By bypassing the systemic parsing cycle for large instruction datasets, this optimisation reduces Time to First Token (TTFT) by up to eighty percent.

Does model size affect latency? Yes, smaller models have much faster token generation speeds, making them ideal for simple tasks where latency is a primary concern. Routing straightforward classification or extraction requests to specialised edge models ensures quick turnaround times while reserving dense models for reasoning tasks.

How does Server-Sent Events (SSE) streaming help reduce perceived latency? SSE streaming pushes text output tokens from the model host to the client screen in real-time as they are compiled. Although this does not reduce total execution duration, it minimises Time to First Token (TTFT) and gives the user a responsive, active application interface.

How do I cache dynamic LLM responses at the edge? You can cache dynamic responses at the edge using KV databases or Redis instances with short TTL (Time to Live) limits. Caching dynamic responses is effective for repetitive user queries or common customer service intents, preventing model provider network calls entirely.