Most teams call a model provider straight from application code. The API key sits in an environment variable, the SDK call is three lines, and it works first time. Cloudflare AI Gateway exists because of what happens next. The bill arrives and nobody can say which feature caused it. The provider has a bad afternoon and takes your product with it. A system prompt gets edited and there is no record of what the old one returned.

A gateway is a proxy between your application and the provider. Every request passes through it, so every request can be counted, logged, cached, rate limited and, when the provider fails, retried elsewhere. It is the cheapest structural fix for three problems that otherwise get solved late and by hand.

What follows is what it does, what it enforces as opposed to merely observes, what it costs in money and in milliseconds, and where the honest limits sit.

Does putting a gateway in front of your model API actually save money? Not directly. The core of Cloudflare AI Gateway is free on all plans and adds no markup to inference, so the saving comes from what it shows you rather than what it blocks. Per-feature cost attribution tells you which part of the product is expensive, caching removes repeated identical calls where that is safe, and fallback routing stops a provider outage becoming your outage. Budget enforcement exists, but what happens when the budget runs out is a product decision, not a configuration one.

The Three Questions a Direct Model Call Cannot Answer

Every argument for a gateway reduces to one of three questions a direct SDK call leaves unanswerable. The abstract version, visibility and control, persuades nobody who has to justify the work.

Nobody can attribute the bill

Per-token billing is usage based and provider invoices aggregate. You get a monthly total per API key, not a total per feature. If a summariser, a chat assistant and a nightly classification job share one key, the invoice cannot tell you which of them tripled. The usual response is a key per feature, which works until you have eleven features and a rotation policy.

Nobody can reproduce the failure

When a call fails inside application code, what survives is whatever your logger captured, usually a status code and a truncated message. Rarely the exact prompt, the model version that answered, or the latency at which it gave up. Reproducing an incident a day later becomes guesswork about inputs. Meanwhile a provider outage propagates straight through to your users.

Nothing caps a runaway loop

An agent that retries itself, a queue consumer that redelivers on failure, or a loop whose termination condition a model keeps judging unmet: each can generate thousands of calls before anyone notices. Without something in the path counting, the first signal is the invoice. Per-user limits and hard stops do not exist unless a component in the path enforces them, and application code is a poor place for them because every call site has to remember.

What Cloudflare AI Gateway Is and Where It Sits in the Request Path

Cloudflare’s AI Gateway overview describes a service sitting between your application and AI model providers so you can monitor usage and manage how the application scales. The features are analytics, logging, caching, rate limiting, and request retry with fallback, on all Cloudflare plans. Cloudflare claims it takes one line of code to start, and the integration shape is why that is close to true.

The integration is a base URL change

Instead of pointing your SDK at the provider, you point it at a gateway address carrying your account identifier, your gateway identifier and the provider name. Cloudflare documents the OpenAI form as https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/openai, supplied as the baseURL when constructing the client. Model names, parameters, streaming and response parsing stay as they were. Cloudflare’s provider list covers two dozen services on the same pattern, including OpenAI, Anthropic, Google Vertex AI, Amazon Bedrock, Azure OpenAI, Mistral, Groq, DeepSeek, xAI and Cloudflare’s own Workers AI.

What that shape implies

Adoption is genuinely cheap: one configuration value per service, reversible by changing it back, which makes this one of the few infrastructure changes you can trial without a project plan.

Your provider key now travels through Cloudflare, because the proxy has to forward it, unless you move to stored keys or Cloudflare-managed credentials. That is a trust decision, not a detail.

And everything on offer is bounded by what a proxy can see. It sees requests and responses, not your application’s intent, which is why attribution requires you to label requests rather than expecting the gateway to infer what a call was for.

Analytics and Logging Are the Features That Save the Money

The gateway counts every request. Cloudflare’s logging documentation lists what an entry holds: the user prompt, the model response, provider, timestamp, request status, token usage, cost, duration and the client user agent. Per request rather than per month, and queryable.

Be blunt about why this, rather than the enforcement features, is where the money is. Blocking spend saves the cost of the calls you stopped, which is bounded. Knowing where spend goes changes what you build, which is not.

Attribution is not automatic. Custom metadata attaches your own labels, such as feature name or tenant, so analytics can group by them. Skip that and you get a total, which is what the invoice already gave you.

Caching, and the Cases Where a Cache Hit Ruins the Product

Caching is the feature most often switched on for the wrong reason and the one most capable of silently breaking a product.

How the cache key is built

Cloudflare’s caching documentation explains the key as a SHA-256 hash of the provider, endpoint, model, authentication header and the full request body. An exact match on all of it is a hit, anything else a miss. So a hit requires a byte-identical request, which for a chat endpoint carrying accumulated history is rare after the first turn. Minimum time to live is 60 seconds and the maximum is one month. Per-request control comes from cf-aig-cache-ttl, cf-aig-skip-cache and cf-aig-cache-key, while cf-aig-cache-status returns HIT or MISS so you can measure the real hit rate.

When a hit is safe and when it is not

A hit is safe where identical input should produce identical output and staleness is acceptable: classification, structured extraction, translation of fixed strings, embeddings of unchanged documents, evaluation traffic. It is unsafe wherever the product’s value depends on variation. If two users ask the same question and the second receives a response generated for the first, your temperature setting is decorative.

The worse failure is a privacy one. If the request body carries nothing that distinguishes users, a cached response can cross a user boundary, which is a disclosure rather than a quality issue. Caching also covers text and image responses only.

Rate Limiting, Retries and Fallback Routing

These three get described together as reliability features. Only one of them meaningfully constrains cost.

Rate limiting caps requests, not tokens

Cloudflare offers a fixed window and a sliding window, and requests over the limit receive a 429. Note what is counted: requests. A single call carrying a very large context costs far more than a short one and the limiter cannot tell them apart. Rate limiting protects you from a runaway loop and from an abused endpoint, not from an expensive prompt, and treating it as cost control is the common mistake.

Retries and timeouts

Cloudflare’s request handling headers allow up to 5 attempts with cf-aig-max-attempts, a delay up to 5000 milliseconds with cf-aig-retry-delay, and a constant, linear or exponential strategy with cf-aig-backoff. The header cf-aig-request-timeout is measured from when the first part of the response arrives, so on a streaming call it is a time to first byte limit rather than a total duration limit. On the final attempt the gateway waits until the request completes however long that takes.

Fallback routing has changed shape

The Universal Endpoint, which took an array of provider objects and walked down it on failure, is deprecated. Cloudflare now directs new integrations to the OpenAI-compatible endpoint, and to Dynamic Routing for fallbacks, retries and conditional routing. A dynamic route is a named, versioned flow built visually or as JSON, made of model nodes, conditional nodes branching on request body, headers or metadata, percentage nodes for A/B tests, and rate limit and budget limit nodes that divert to a fallback when exceeded. You invoke it by putting the route name where the model name goes.

Inspecting What Goes In and What Comes Back

Because the proxy holds both halves of the exchange, it can evaluate them. Cloudflare’s Guardrails intercept user prompts and model responses, flag content for review or block it from proceeding, and apply one policy regardless of which provider answered.

The argument for policy in the path rather than in code is that application-level moderation has to be written once per call site, and the call site added last week is the one that misses it. The cost is that inspection is itself inference: Guardrails are billed as Workers AI token-based usage, so the price scales with the length of what you inspect. Data loss prevention scanning is free on all plans.

What Changed During Agents Week in August 2026

Cloudflare ran its first Agents Week from 3 to 7 August 2026 and on the final day published the unification of Workers AI and AI Gateway into a single control plane. Separate what shipped from what was announced.

What shipped: one AI binding instead of two, so env.AI.run() covers both Workers AI models and external providers. Gateway routing became the default for Workers AI rather than opt-in, with gateway: { id: 'default' } auto-creating the gateway on first use and delivering request logging, token tracking and cost attribution with no other change. Credits became spendable across providers, so Workers AI can be paid from the same prepaid balance as OpenAI or Anthropic.

What did not ship: model-first routing, where you request a model and the platform picks the provider, is next rather than available, and a prompt-classifying smart router is an internal pilot. This is a consolidation of billing, bindings and dashboards, not a new capability. If you were already calling Workers AI at the edge without a gateway, you now get observability by default.

What Cloudflare AI Gateway Costs

Cloudflare’s AI Gateway pricing page states that the core features available today are offered for free, covering dashboard analytics, caching and rate limiting on all plans. All figures below are Cloudflare’s published US dollar prices, quoted in the currency Cloudflare publishes them in.

Where the charges actually appear

Persistent logs are free to use but capped by plan: 100,000 logs across all gateways on Workers Free, and 10 million per gateway on Workers Paid. Logpush, which exports those logs elsewhere, is a Paid feature at 10 million per month with $0.05 per additional million. Workers AI itself, per its pricing page, includes 10,000 neurons per day at no cost and charges $0.011 per 1,000 neurons beyond that on the Paid plan, a neuron being Cloudflare’s unit for GPU compute.

Unified billing costs 5 percent

If you use Cloudflare-managed credentials rather than your own provider keys, credit purchases carry a 5 percent fee. Cloudflare’s own example is a $100 credit purchase billed at $105. Inference is passed through with no markup. Bring your own key and unified billing does not apply, because a request carrying provider authentication or a stored key skips it.

Cost Control Done Properly

A gateway reliably enforces three things: a request rate, a budget limit inside a dynamic route, and whether a request is served from cache. Everything else it does is measurement. Confusing the two categories is why teams install one, tick every box, and are still surprised by the invoice.

A hard budget cap is a business decision before it is a configuration. Something has to happen when the cap is reached and every option is bad in its own way. Failing the request degrades the product for whoever asked last, which looks like a bug. Falling back to a cheaper model degrades quality silently. Queueing converts a cost problem into a latency problem. Choosing between those is the work.

The saving that reaches an invoice usually comes from observability rather than enforcement. Once spend groups by feature, the expensive thing is nearly always fixable without changing the model: an oversized system prompt sent every turn, full conversation history resent where a rolling summary would do, a retry loop firing against a failure that was never going to succeed. A log finds all of those. A rate limiter finds none.

The Latency You Are Adding

An extra hop is not free, and pretending otherwise is how a good decision gets made for a bad reason. Your request now terminates at a Cloudflare data centre, is processed there and forwarded to the provider, which adds a TLS handshake and a network leg you were not previously paying for.

In the normal case that is small relative to what it wraps. A chat completion runs from hundreds of milliseconds to several seconds and is dominated by generation time, and the gateway sits on Cloudflare’s edge network, so the first leg terminates near the caller. Against a two second completion the overhead is noise.

Where it stops being noise is short, cheap, high-volume calls. An embedding request returning in well under a tenth of a second is a case where fixed overhead is a visible percentage. Streaming is the other, because the number users feel is time to first token, and anything added before that token lands on the metric that matters. Measure it from the duration the gateway logs, against the same call made directly.

Multi-Provider Strategy and the Lock-In You Do Not Escape

The pitch is that a gateway makes providers interchangeable, and at the transport layer that is true. The OpenAI-compatible endpoint gives you one request shape, dynamic routing gives you failover without a deployment, and changing a model name becomes configuration rather than code.

The transport layer was never the expensive part. Switching models is expensive because models behave differently. A system prompt tuned over months against one model produces different output against another. Tool calling formats and reliability differ. Refusal behaviour differs, so content that passed before gets declined. Adherence to a requested JSON shape differs, and a parser written against one model’s habits breaks against another’s. Context windows differ.

What you do buy is that a switch becomes an afternoon of evaluation rather than a sprint of plumbing, and that outage-driven failover exists without your building it. That is worth having, but it is not portability. One second-order dependency deserves naming: the gateway now sits in the path of every model call, so its availability becomes your availability. That is the same trade we examined comparing Cloudflare Workers and AWS Lambda.

Logging Prompts Means Processing Personal Data

This is the section that gets skipped and the one with legal exposure attached. Logging is on by default for each gateway, and an entry contains the user prompt and the model response in full. If your users type anything about themselves, paste a document, or describe a medical or financial situation to your assistant, that content is personal data in a third-party log store, and under UK GDPR you remain the controller of it.

Redact before it leaves your application

The only reliable place to remove something is before it is sent. The gateway offers cf-aig-collect-log-payload: false to keep metadata while dropping bodies, and cf-aig-collect-log: false to log nothing. But data minimisation is about not collecting more than you need in the first place, so the durable fix is upstream: strip account numbers, identifiers and free-text fields the model does not need before the request leaves your process.

Retention is a decision you have to make

The ICO’s guidance on storage limitation sets no fixed periods. It requires that you do not keep personal data longer than you need it, that you can justify the period chosen, that you have a policy setting standard periods, and that you review and then erase or anonymise what you no longer need. A log store with a plan-level cap is not a retention policy, because a cap is a storage limit rather than a justified period.

Alternatives, Named Fairly

There are three genuine alternatives, and the choice is mostly about who operates the thing.

A self-hosted open source LLM proxy gives you the same request path with the logs on infrastructure you control, which is right when the sensitivity of the prompts is itself the problem. The cost is operating a component on the critical path of every model call.

A specialist LLM observability vendor goes deeper on evaluation, prompt versioning and trace inspection. If your problem is that outputs are wrong rather than that costs are opaque, that fits better, and the two are not mutually exclusive.

Building it yourself is defensible when the need is narrow. A wrapper recording request, response, token counts and a feature label into your existing observability stack is perhaps two days of work and answers the attribution question. What you will not get cheaply is caching with a correct key, and fallback across providers.

The rule: if nobody knows where the spend goes, a gateway is the fastest fix and the free tier proves the point. If prompts must not leave your infrastructure, self-host. If the outputs are wrong, no gateway helps and you need evaluation tooling.

What Implementation Takes, and What It Returns

For a single service already calling one provider, half a day. Create the gateway, change the base URL in configuration, deploy behind a flag so a revert is an environment variable rather than a release, watch the logs populate, confirm streaming still behaves.

For a real product, budget three to five engineer days, and most of that is not the gateway. It is deciding a metadata schema so attribution answers questions you will actually ask, auditing every call site including scheduled jobs and code nobody has opened in a year, working out which endpoints can safely be cached, and writing the redaction step. Add a day for fallback routing, because a fallback is only worth having once you have tested that the fallback model produces acceptable output.

The return is unglamorous. You stop being surprised by the invoice, which for most teams beats the absolute saving. A provider outage becomes a degraded response rather than an incident. Teams building agent workflows benefit most, because one user action fanning out into dozens of model calls is where estimates diverge from reality, a pattern we covered in what AI agents cost and where they fail.

Putting One In Without Breaking Production

The sequence that works is boring on purpose. Put the gateway in the path with logging on and nothing else enabled. Collect a week of data. Read the attribution. Then enable exactly the features the data justifies, and enable caching last, only on endpoints where you can articulate why a repeated answer is a correct answer.

Mecanik builds and operates this layer as part of our AI integration work, and our OpenAI API integration service covers the provider side. Engagements almost always start the same way, with a week of logging and no other changes, because the attribution data usually reorders the priority list.

Frequently Asked Questions

Is Cloudflare AI Gateway free? The core features are free on all plans, which Cloudflare’s pricing page describes as covering dashboard analytics, caching and rate limiting, and data loss prevention scanning is free too. Charges appear around the edges: Logpush is a Paid plan feature, Guardrails are billed as Workers AI token-based inference, and log storage is capped by plan rather than priced per log.

Does an AI gateway add latency to model calls? Yes, it adds a TLS handshake and a network leg. Against a chat completion taking hundreds of milliseconds to several seconds, that overhead is usually negligible. It becomes visible on short high-volume calls such as embeddings or small classifications, and on streaming responses where the metric users feel is time to first token rather than total duration.

When is it unsafe to cache LLM responses? Whenever the product’s value depends on variation between responses, or whenever the request body does not distinguish one user from another. Cloudflare builds the cache key from the provider, endpoint, model, authentication header and full request body, so a hit means a byte-identical request. Caching suits classification, extraction and embeddings, not conversational responses that should differ per user.

Does a gateway make it easy to switch model providers? At the transport layer, yes. One request shape, one configuration change, and failover without a deployment. The expensive part of switching is untouched: prompts tuned against one model behave differently on another, tool calling formats and refusal behaviour differ, JSON adherence differs, and context windows differ. A gateway removes the plumbing, not the evaluation work.

What did Cloudflare change in August 2026? On 7 August 2026, at the end of Agents Week, Cloudflare unified Workers AI and AI Gateway into a single control plane: one AI binding covering both Workers AI and external providers, gateway routing on by default for Workers AI, and credits spendable across providers from one prepaid balance. Model-first routing was announced as coming next rather than shipped.