langchain · difficulty ◆◆◆
ProviderToolSearchMiddleware: Auto-Routed Tool Selection
Stop hand-rolling tool routers. Let the provider pick the relevant tools per query.
Hundreds of tools, one query. Let the provider find the three that matter instead of prompting a brittle router.
$ pip install langchain==1.3.7 langchain-core==1.4.5What it does
ProviderToolSearchMiddleware (langchain==1.3.7, PR #37969) intercepts chat model invocations and routes queries through a provider-aware tool search step before the model generates a response. Every user message first triggers a tool lookup using the model’s own provider ecosystem (OpenAI, Anthropic, Google) to surface relevant tools from your registry. Only then does the model respond, armed with the retrieved tool definitions.
Why it matters
In production RAG and agent pipelines you often have dozens or hundreds of tools. Passing all of them to every request is expensive and slow; hand-coding a tool router prompt is brittle and model-specific. This middleware offloads selection to a provider-native search step, giving cost reduction, better accuracy, and cleaner code.
Example
$ Agent query with auto tool routing> Entering new AgentExecutor chain...
{'input': 'What's the weather in Paris?', 'output': 'The weather in Paris is sunny and 72°F.'}
> Chain finishedThe middleware surfaces only the relevant tools before the model responds, cutting token cost.
Common flags
- search_type="similarity"
- Similarity-based tool retrieval
- search_type="mmr"
- Max marginal relevance for diversity
- top_k=3
- Retrieve the top N tools per query
History
Origin
Added in langchain==1.3.7 (released June 10, 2026), PR #37969, with release notes in langchain==1.3.7 (PR #38024).
Contrast with tool injection
Unlike manual tool schema injection into every prompt, the middleware makes routing automatic and context-aware via the provider.
Fun facts
Pros & cons
pros
- + Fewer tokens per call
- + Provider-side relevance vs brittle heuristics
- + No manual router prompt
cons
- − Adds a search round-trip
- − Depends on provider ecosystem
Takeaways
- 1Attach ProviderToolSearchMiddleware to auto-route tools per query.
- 2Choose similarity or mmr search and set top_k for your registry.
- 3Measure token savings with usage_metadata to justify adoption.