AI Gateway: One Endpoint, Many Models

An AI gateway is a single entry point between your applications and your AI models. Every request, whether it's a voice command, a photo, a chat message or a business decision, goes through the gateway. The gateway checks it, picks the model best suited to the job and returns the answer through one consistent API.

We're building our own custom AI gateway at Axis Technologies, because the projects we deliver no longer use just one model. A single customer workflow might need to listen to a phone call, read a scanned delivery note, write a reply and decide whether a refund needs approval. No single model is the best, cheapest and fastest at all four.

This article explains what an AI gateway does, how intelligent model routing works, and what we've learned so far building one for voice, vision, text and decisions.

When teams start with AI, they usually connect one app to one large model. It works, until it doesn't. Four problems tend to appear at the same time:

• Cost. Most requests are simple: "summarise this", "classify that", "is this an invoice?". Sending them all to the largest model is like hiring a senior engineer to sort the post.
• Speed. Large models take longer to answer. For a voice assistant or a live chat, a two-second delay feels broken.
• Different kinds of input. Text models don't hear and many don't see. Voice needs speech-to-text and text-to-speech. Images need a vision model. Document search needs embedding and reranking models.
• Control. Some data shouldn't leave your network. Some tasks need an audit trail. Some providers have outages. You need one place to manage all of that.

An AI gateway solves these problems in one layer instead of in every application.

Think of the gateway as a smart receptionist. Every request comes to the same desk. The receptionist checks your badge, makes sure the request is safe, understands what you need and sends you to the right specialist.

Because the gateway speaks the same API format as OpenAI, our apps don't need special code. They point at the gateway and ask for the model auto:

import OpenAI from 'openai';export const ai = new OpenAI({  baseURL: process.env.AI_GATEWAY_URL, // our gateway, not a provider  apiKey: process.env.AI_GATEWAY_KEY,});const reply = await ai.chat.completions.create({  model: 'auto', // let the gateway choose the right model  messages: [{ role: 'user', content: 'Summarise this delivery note and list any missing items.' }],});

Switching a model, adding a new one or moving a task from the cloud to our own hardware happens in the gateway. None of the applications change.

Routing is the heart of an AI gateway. For every request, the router answers one question: which model should handle this?

The router itself is a small, fast model. Its only job is to make that decision in a few milliseconds. It considers:

• Modality. Does the request contain audio, an image, or only text?
• Difficulty. A quick yes/no or a short summary is "easy". Multi-step reasoning, code or long documents are "hard". Easy requests go to the small model; everything else goes to the large one.
• Capabilities. Does the request need tool calling? Is it longer than a model's context window? Does the answer need to be long? A model that can't meet those needs is never chosen, however cheap it is.
• Policy. Some apps or data types must stay on our own servers. Some tasks always need the strongest model.

Every response tells the app which model was chosen, so we can audit the router and spot bad decisions. Open-source projects such as vLLM Semantic Router have made this kind of routing much more practical in the last year.

Voice is where a gateway really pays off, because one spoken request uses several models in a row.

Speed matters more than anything here. That's why speech models run on dedicated hardware close to the user, and why voice requests usually go to a small, quick language model. A brilliant answer that arrives three seconds late is worse than a good answer that arrives now.

Typical uses we're designing for: phone and helpdesk assistants, hands-free checklists for operators on the shop floor, and voice control for internal tools.

Many business processes still run on pictures: scanned invoices, photos of delivery notes, screenshots of error messages, quality-control photos from a production line.

When a request contains an image, the gateway sends it through a vision encoder first. The encoder turns the picture into something a language model can reason about. Then the router decides whether a small model can handle the question ("what's the invoice total?") or whether it needs the large one ("compare this delivery note with the order and list what's missing").

The lesson we keep relearning: getting a vision model to describe a picture is easy. Getting it to read the picture, use a tool and take the right action reliably is a separate step that has to be tested end to end.

Not every request is a question. Many are decisions: should this refund be approved? Is this email a complaint? Which team should handle this ticket?

For decisions, the gateway works together with our AI agents. The gateway provides the right model for each step. The agent, built with Next.js and Mastra, owns the logic: which tools to call, which rules apply and when to ask a person.

A few rules we follow for decisions:

• The model suggests, the rules decide. Anything with money, safety or legal weight goes through a fixed workflow with clear thresholds.
• Confidence matters. Low-confidence decisions go to a person automatically.
• Everything is traceable. For every decision we can see which model was used, what it saw and why it chose what it did.

Putting all AI traffic through one gateway is also a big security win. Instead of protecting every app separately, we protect one door.

• Prompt injection detection. A dedicated classifier checks requests for attempts to override instructions or extract hidden data. We start in monitoring mode, so it only records what it finds. That lets us measure false alarms on real traffic before we block anything.
• Personal data masking. Names, emails and other sensitive fields can be masked before a request reaches an external provider.
• Keys and limits per app. Every application has its own key, budget and rate limit. A bug in one app can't burn through everyone's budget.
• A full audit log. Who asked what, which model answered, and what it cost.

One of the main reasons to build a gateway is the freedom to choose where each model runs.

A point that surprises many people: you don't need the biggest, newest GPU to run useful AI. Giving each model one focused job lets you match it to modest hardware. A small model for quick answers, an embedding model for search and speech models on a low-power chip can share one server. The large model is only woken up when it's really needed.

People often ask whether their existing API gateway can do this. Partly, yes: authentication, rate limits and logging are the same idea. The difference is that an AI gateway understands the request. It knows whether there's an image inside, how hard the question is, how many tokens it will cost and which model can handle it. It also deals with risks that classic gateways were never built for, like prompt injection.

In practice we often run both: the API gateway at the edge of the network, and the AI gateway behind it for everything that touches a model.

Building our AI gateway has taught us a few lessons we'd share with anyone starting out:

1. Get routing right before chasing savings. A wrong routing decision looks exactly like a bad model. Check that requests go where they should before you optimise anything.
2. Capabilities beat clever prompts. Whether a model supports tools, how big its context is and how long its answers can be matter more than how you word the prompt.
3. A good search score isn't a correct answer. In document search, check that the retrieved passages are really the right evidence, not just similar text.
4. Test every path end to end. "The vision model runs" is not the same as "the vision workflow works". Each modality needs its own real-world tests.
5. Log the decision, not just the answer. Knowing which model answered, and why, is what makes the system debuggable.

As soon as your business uses AI for more than one thing, an AI gateway stops being a nice-to-have. It gives every app one simple API, sends each request to the model that fits it best, keeps sensitive data where it belongs and shows you exactly what your AI is doing and what it costs.

We're building ours for voice, vision, text and decisions, and using it inside the agent-powered software we deliver to clients. If you're running into the limits of a single model, or want to bring AI onto your own hardware, get in touch. We'd be happy to show you how routing could work for your workflows.

Further watching: Techno Tim's video I Built My Own Intelligent AI Gateway is a great hands-on look at running several self-hosted models behind one intelligent router.