MCP at Scale: Gateways

Part of the free Generative AI course on LogicWiz, module: Nova Joins the Network.

Episode 36: MCP at Scale — Gateways

Nova now had 600 tools. She could do anything — and chose wrong half the time. "It's not that she's dumb," Arjun realised. "It's that we handed her a phone book and asked her to pick a name."

Success Has a New Problem: Too Many Tools

MCP made adding tools cheap — so organisations added hundreds. Connect an agent to a few servers and you might expose 600 tools. That success creates a new problem, and it's a sneaky one.

Every tool the agent can use has its definition and signature injected into the model's context — because that's how the model knows the tool exists. Six hundred tools means six hundred schemas riding along in every prompt, before the user has said a word.

Watch what that does:

{{visual:context-bloat}}

Reasoning Drowns in Options

Two things go wrong as the toolset grows, and they compound:

  • Cost climbs. More tool schemas = more tokens in the context window = a bigger bill on every single call. You pay for those 600 definitions whether or not any get used.
  • Accuracy falls. This is the subtle one. Give a model an excessive number of tools and its selection gets worse — the right tool is harder to spot in a crowd of near-duplicates. The metadata can even "bloat" the context enough to dilute the user's actual instruction.

📌 The optimisation problem at the heart of MCP-at-scale: give the agent a small, highly-relevant subset of tools — broad capability, but high decision accuracy and low token cost. Everything below is a strategy for choosing that subset.

Four Ways to Slim the Toolset

There's no single "right" answer — the best approach depends on your setup — but four strategies recur:

  • Group-based access. Use auth to scope tools to a user's role. A finance agent sees only the ~150 finance tools, not all 600. Fewer, more relevant tools; lower cost; better choices.
  • Hierarchical tools. Expose high-level categories (read, write, log_incident) instead of hundreds of flat tools; the specific sub-tool is reached by passing an argument. The agent navigates down to what it needs.
  • A proxy server. When you don't control the underlying MCP servers, put your own MCP server in front of them. It decides which tools to expose and passes only those through.
  • Semantic discovery. Instead of returning all tools, the server (or gateway) runs a discover_tools(query) that matches the request against tool descriptions and returns only the closest ones.

That last one deserves a hands-on look, because you already know how it works.

The Semantic Router: RAG for MCPs

Slimming the toolset by meaning is just RAG pointed at tools instead of documents. Embed the user's query, embed every tool description, and keep the closest matches. Same cosine-similarity trick from Chapter V — new target.

Walk the code, then run it:

{{visual:semantic-router-walkthrough}}

{{cell:l36-router}}

Notice it surfaces check_order for "where's my parcel?" over book_flight — with no shared keywords. That's the point: meaning, not string matching. A gateway doing this at scale can turn 600 tools into the 10 that matter for this request.

The MCP Gateway

Now zoom out. When you have not just many tools but many servers — 10, 15, internal and external — you need a single place to govern them all. That place is an MCP gateway: one hub every agent talks to, sitting in front of every server.

flowchart TD
  U["User query"] --> G["🛡️ MCP Gateway"]
  G -->|"semantic filter + auth"| S1["MCP server 1"]
  G --> S2["MCP server 2"]
  G --> S3["MCP server n"]
  S1 --> T1["tools"]
  S2 --> T2["tools"]
  S3 --> T3["tools"]

A gateway is a genuine control plane, and it earns that name by doing four jobs at once:

  • Filtering — semantic matching (exactly the router above) so each agent sees only relevant tools.
  • Authentication & authorization — one place to validate an agent's identity and decide which servers and tools its role may touch. Agent in Finance? It never even sees the HR server.
  • Resilience — if a server (say, in the UK region) goes down, the gateway reroutes the request to a healthy equivalent (in the US). This is far easier now that MCP 2.0 is stateless — there's no sticky session to drag along, so failover is seamless.
  • A registry — a central catalogue of every available server. Register a new server once and the gateway (and the whole org) can see it.

💡 One clean way to think about it: the gateway is "RAG for MCP servers." Instead of retrieving the most relevant document chunks, it retrieves the most relevant tools and servers for a query — from a vast collection — and hands the agent just those. AWS's AgentCore ships exactly this kind of MCP gateway.

What Nova Learns Next

Nova can now reach hundreds of tools across many servers, governed by a gateway that keeps her focused. But some jobs aren't a tool call at all — they need her to hand a whole task to another agent. In Episode 37, tools give way to teammates: agent-to-agent communication.