Open source / Guides / Local tool routing

AI agent architecture guide

Route large tool catalogs before model inference

An agent with hundreds or thousands of functions does not need every JSON Schema on every turn. A local retrieval layer can preserve the original definitions while selecting a small, deterministic working set.

1. Tool definitions consume context before work begins

A tool definition carries a name, description, and input schema. A catalog of similar CRUD tools can become a large fixed prefix on every request. This increases payload size and gives the model more near-duplicate choices, even when the task clearly belongs to one namespace.

There are three broad strategies:

A local router is especially effective when tools have descriptive names, namespaces, verbs, tags, and schemas. It can also be a first stage before a more expensive semantic fallback.

2. Separate retrieval from provider execution

The router should normalize definitions only for search. The selected payload sent to the model should remain the original object, preserving its provider-specific schema dialect and metadata.

import { createToolRouter } from '@stackline/tool-router';

const router = createToolRouter(tools);
const prompt = 'Open an issue for the checkout regression';

const routed = router.route(prompt, {
  maxTools: 4,
  maxEstimatedTokens: 4_000,
  fallback: 'none'
});

const response = await openai.responses.create({
  model: 'your-model',
  input: prompt,
  tools: routed.tools
});

A BM25F-style local index can weight names, namespaces, aliases, tags, descriptions, and schema text differently. Prefix matches and bounded typo tolerance help identifiers; action synonyms connect common verbs such as “open/create” and “post/send.” Literal identifiers should remain stronger than broad synonym matches.

3. Keep one provider-native catalog per request

MCP and model providers place JSON Schema under different keys and envelopes. A routing layer can recognize those shapes without translating them.

const openaiRouter = createToolRouter({ tools: openaiTools });
const geminiRouter = createToolRouter({
  functionDeclarations: geminiFunctions
});

Do not combine incompatible provider definitions and expect the router to convert them. Keep a catalog that matches the outbound request, and store search-only metadata on the outer definition when the provider allows extra local fields.

4. Apply count, token, and policy controls independently

A maximum tool count is easy to understand but ignores schema size. A token estimate provides a second budget, even though the final provider tokenizer can differ. Record both selected count and estimated reduction rather than treating the estimate as billing truth.

const result = router.route(userMessage, {
  maxTools: 6,
  maxEstimatedTokens: 4_000,
  namespaces: ['github', 'slack'],
  tags: ['write'],
  pinned: ['auth_get_current_user'],
  fallback: 'none'
});

console.log({
  selected: result.selectedCount,
  estimatedTokens: result.estimatedTokens,
  reduction: result.tokenReduction,
  budgetExceeded: result.budgetExceeded
});

Pinned tools are useful for identity, policy, escalation, or discovery operations that must remain available regardless of lexical score. If pinned definitions exceed a token budget, surface that condition; silently dropping an explicit policy tool is the wrong optimization.

Choose fallback behavior from measured recall. A low-risk read assistant might send a small default set. A privileged agent may ask for clarification or run a second-stage search instead of guessing.

5. Evaluate with real intents and negative cases

Retrieval quality is a product metric, not a generic benchmark. Build a checked-in corpus from representative user requests, including paraphrases, typos, namespace mentions, ambiguous verbs, and requests that should select no tool.

Track at least:

• recall@k: whether an expected tool appears in the first k results;
• precision: whether selected tools are actually relevant;
• no-match accuracy: whether unrelated requests avoid a confident false positive;
• estimated reduction: how much catalog payload was excluded;
• routing latency: build, incremental update, and query time by catalog size.

const cases = [
  {
    query: 'post the release note in engineering chat',
    expected: ['slack_send_message']
  },
  {
    query: 'what should we have for lunch?',
    expected: []
  }
];

for (const testCase of cases) {
  const matches = router.search(testCase.query, { limit: 5 });
  evaluate(testCase, matches);
}

Use the bundled synthetic benchmark to detect regressions, not to claim universal production quality. Tune field weights and aliases against the application corpus, then freeze that corpus in CI.

6. Update live catalogs without rebuilding everything

Tool catalogs change as MCP servers connect, tenants gain capabilities, or feature flags move. Incremental add, remove, and replace operations avoid rebuilding an index for every small update.

const router = createToolRouter([], { onDuplicate: 'replace' });

router.add(discoveredTool);
router.remove('legacy_export');
router.replace(currentTenantCatalog);

const stats = router.stats();

Keep one router per authorization-visible catalog or filter eligible definitions before ranking. Observe query latency, selected count, budget overflow, no-match rate, fallback rate, and eventual tool-call success without logging sensitive prompts by default.

For very large or multilingual catalogs, local lexical retrieval can remain the fast first stage. A measured semantic reranker can operate on the small candidate set instead of embedding and sending the entire catalog for every request.