Deploying Llama 3 on the Edge With Cloudflare Workers AI

This Cloudflare Workers AI tutorial shows you how to deploy and run machine learning models directly on Cloudflare’s global edge network. With Cloudflare Workers AI , you can execute Large Language Models (LLMs), text translation, image generation, and audio transcription close to your users without managing complex GPU servers. The steps below cover how to configure Wrangler, write a fetch handler, run a Llama model, and optimise API costs at the edge.

TL;DR

• Run AI models serverless; Cloudflare Workers AI manages the underlying GPU infrastructure, billing you only for active compute.
• Configure bindings in wrangler.toml; link your worker directly to the AI binding without managing API keys.
• Write a fetch handler to receive user requests, execute the model, and stream JSON responses.
• Select optimised models from Cloudflare’s catalogue (such as Llama 3, Whisper, or Stable Diffusion) to balance speed and accuracy.
• Silo billing and rate limits to protect your endpoint from high-token cost abuse in production.

Why Run AI Models on the Edge?

Traditionally, integrating AI features required calling external APIs (like OpenAI) or hosting open-source models on expensive cloud servers.

Calling external APIs presents latency and data privacy concerns. Running your own GPU servers introduces maintenance challenges and scales poorly. Cloudflare Workers AI resolves this dilemma. The platform hosts open-source models on GPUs deployed across Cloudflare’s global network. Your code runs as a lightweight Worker, executing models close to your users with near-zero latency. For an introduction to building basic edge structures, check our guide on building a serverless API with Cloudflare Workers .

Prerequisites

Before you begin, make sure you have the following in place:

• A Cloudflare account with Workers enabled. The free plan includes a daily inference allocation, which is more than enough to follow this guide.
• Node.js 18 or newer installed locally, so you can run the tooling and the local development server.
• Wrangler, the Workers CLI, installed and authenticated. Install it with npm install -g wrangler, then run wrangler login to link it to your account.
• Basic familiarity with JavaScript and the fetch request/response model that every Worker is built around.

If you are starting from a blank slate, scaffold a project with the create-cloudflare (C3) tool. It generates a ready-to-deploy Worker, a wrangler.toml, and a sensible compatibility_date:

1npm create cloudflare@latest my-edge-ai
2cd my-edge-ai

Choose the “Hello World” Worker template when prompted. That gives you a clean starting point to which you can add the AI binding in the next step.

Step 1: Configuring wrangler.toml

To access AI models, you first need to define the AI binding in your project’s configuration file.

Open your wrangler.toml (or wrangler.json) and add the following block:

1[ai]
2binding = "AI"

This binding makes the AI service available on the environment argument (env.AI) in your worker code. You do not need to manage API keys, configure endpoints, or deal with connection strings. Wrangler handles the authentication automatically when you deploy.

A complete wrangler.toml for this project looks like this:

1name = "my-edge-ai"
2main = "src/index.js"
3compatibility_date = "2025-01-01"
4
5[ai]
6binding = "AI"

The compatibility_date matters: it pins runtime behaviour so future platform changes cannot silently alter how your Worker runs. Omit it and deployment fails with a configuration error, which is one of the most common first-run stumbles.

Step 2: Writing the Worker Code

Once the binding is configured, you can call the AI service from your fetch handler. The example below shows how to receive a JSON payload containing a user prompt and generate a text completion using Llama 3.

 1export default {
 2  async fetch(request, env) {
 3    if (request.method !== "POST") {
 4      return new Response("Method not allowed", { status: 405 });
 5    }
 6
 7    try {
 8      const { prompt } = await request.json();
 9      if (!prompt) {
10        return Response.json({ error: "Missing prompt" }, { status: 400 });
11      }
12
13      // Call the model using the AI binding
14      const response = await env.AI.run("@cf/meta/llama-3-8b-instruct", {
15        prompt: prompt,
16        max_tokens: 256
17      });
18
19      return Response.json(response);
20    } catch (err) {
21      return Response.json({ error: err.message }, { status: 500 });
22    }
23  }
24};

This simple structure allows you to build text summarisers, sentiment analysis tools, or content generators. If you want to connect these features to your database, you can bind to a serverless SQL database; see Cloudflare D1 database setup for details.

Using the chat message format

The prompt field is convenient for one-shot completions, but conversational models behave best with a structured messages array. This lets you set a system instruction that shapes the model’s tone and keep it separate from the user’s input:

 1const messages = [
 2  { role: "system", content: "You are a concise technical assistant." },
 3  { role: "user", content: prompt }
 4];
 5
 6const response = await env.AI.run("@cf/meta/llama-3-8b-instruct", {
 7  messages,
 8  max_tokens: 512
 9});
10
11// Text models return the generated text on the `response` property
12return Response.json({ answer: response.response });

Note that text-generation models wrap their output in a response field, so the completed text lives at response.response rather than at the top level. This catches out many first-time users, who log the raw object and see an unexpected shape.

Choosing the Right Model

Cloudflare’s catalogue includes dozens of models, and choosing well is a trade-off between speed, quality, and cost. Smaller models reply faster and consume fewer resources; larger models reason better but cost more per request. The table below compares a few popular text-generation options.

As a rule of thumb, start with an 8B model such as Llama 3 8B Instruct. It handles the majority of summarisation, classification, and chat workloads at a fraction of the latency and cost of a 70B model. Only move up to a larger model once your own evaluations show the smaller one genuinely falling short.

Step 3: Streaming Responses for Better User Experience

For chat applications, waiting for the full response to generate can result in poor user experience.

You can configure the AI binding to stream the response token by token as it generates. To achieve this, pass stream: true in your request options. The worker will return a ReadableStream that you can forward directly to the browser client, creating a fast, interactive chat interface. To compare this edge runtime approach with traditional cloud hosting, read our comparison of Cloudflare Workers vs AWS Lambda .

Here is a complete streaming handler. Rather than returning parsed JSON, you forward the stream to the client with the correct content-type so the browser can consume it as Server-Sent Events:

 1export default {
 2  async fetch(request, env) {
 3    const { prompt } = await request.json();
 4
 5    const stream = await env.AI.run("@cf/meta/llama-3-8b-instruct", {
 6      prompt,
 7      max_tokens: 512,
 8      stream: true
 9    });
10
11    return new Response(stream, {
12      headers: { "content-type": "text/event-stream" }
13    });
14  }
15};

The key difference from the non-streaming version is the response itself: you pass the stream straight into new Response() and set content-type to text/event-stream. Wrap a stream in Response.json() instead and it will not serialise correctly, so the client receives an empty body.

Testing and Deploying Your Worker

With the binding and handler in place, run the Worker locally before you ship it:

1npx wrangler dev

Because inference runs on Cloudflare’s GPUs rather than on your machine, the AI binding reaches out to the network even during local development. Wrangler manages this for you, but it does mean local testing needs connectivity and a valid login to work.

Send a test request with curl while wrangler dev is running:

1curl -X POST http://localhost:8787 \
2  -H "Content-Type: application/json" \
3  -d '{"prompt": "Summarise the benefits of edge computing in one sentence."}'

Once you are happy with the output, deploy to the global network:

1npx wrangler deploy

Wrangler uploads your Worker and returns a *.workers.dev URL. Your endpoint is now live in every Cloudflare data centre, and each request is routed to the location closest to the user automatically.

Common Pitfalls and Troubleshooting

A handful of issues recur when teams first deploy inference at the edge. Knowing the fix in advance saves hours of debugging.

• Cannot read properties of undefined (reading 'run') — the env.AI binding is undefined. This almost always means the [ai] block is missing from wrangler.toml, or you edited the file but did not restart wrangler dev (or redeploy). Confirm the binding name matches exactly.
• No such model or a 400 error on run — the model identifier is wrong. Model names are case-sensitive and must include the full path, for example @cf/meta/llama-3-8b-instruct. Copy them from the model catalogue rather than typing from memory.
• Empty or malformed streamed responses — you returned the stream through Response.json() instead of new Response(stream, ...). Streams must be forwarded, not serialised.
• Capacity or 429 errors under load — the model is temporarily saturated or you have hit an account limit. Add a short retry with back-off, and consider a smaller model for bursts of traffic.
• Truncated answers — the reply stops mid-sentence because max_tokens is too low. Raise the limit, but remember that larger values increase both latency and cost.

If a request fails silently, watch the live logs with npx wrangler tail while you send a test request. It streams runtime errors and console.log output from the deployed Worker, which is the quickest way to see what the model actually returned.

Optimising Production Costs and Security

While Workers AI is highly cost-effective, running model inference in production requires careful controls.

• Implement Rate Limiting: Limit how often a client can call your endpoint. GPU compute costs scale with usage.
• Sanitise Prompts: Validate inputs to prevent injection attacks and ensure the model outputs stay on brand.
• Silo Context Size: Keep prompt histories short. Sending massive context buffers on each call increases token processing costs.

For a detailed look at context management, check our guide on building an OpenAI API chatbot .

Key Takeaways

• Cloudflare Workers AI provides serverless access to open-source models without GPU maintenance.
• Configure access by adding the [ai] binding block to your wrangler.toml.
• Write fetch handlers to execute text generation, translation, or image tasks directly at the edge.
• Enable token streaming with stream: true to optimise mobile and chat experiences.
• Implement rate limits and prompt sanitisation to protect your edge endpoints in production.

Build Your AI Applications on the Edge

Building reliable AI integrations requires expert knowledge of serverless architecture. Mecanik specialises in AI integration services and custom API setups through our OpenAI API integration service . We build edge-native tools that scale seamlessly, optimising costs and user experiences. Contact us today to discuss your project requirements.

Frequently Asked Questions (FAQ)

What is Cloudflare Workers AI? It is a serverless platform that allows you to run machine learning models (text generation, translation, speech-to-text, image generation) on Cloudflare’s global GPU network.

Do I need an API key to use Workers AI? No. Once you declare the AI binding in your wrangler.toml file, Cloudflare handles credentials internally, making the service accessible via env.AI as shown in this cloudflare workers ai tutorial.

What models are available on Cloudflare Workers AI? The platform hosts popular open-source models, including Meta Llama, Mistral, OpenAI Whisper, and Stable Diffusion, which are updated regularly in their model catalogue.

Can I stream text responses from Workers AI? Yes. By setting stream: true in the parameters of your env.AI.run call, the worker returns a standard ReadableStream to stream text to the browser.

How does billing work for Workers AI? Billing is based on the number of tokens processed (for text models) or the duration of compute time (for other models), offering a cost-effective utility model.