ChatOpenAI Now Surfaces Gateway Metadata

What it does

When your OpenAI calls run through a LangSmith gateway — an LLM proxy that sits between your app and the provider for caching, routing, and tracing — the gateway embeds useful context into the response HTTP headers: cache hit or miss, the underlying provider or region that served the request, gateway latency, and trace identifiers. In langchain-openai 1.5.2a1, ChatOpenAI parses those headers after each response and attaches the parsed values to the result's response_metadata. You read them with result.response_metadata["gateway"], without writing your own header-parsing code. When no gateway is present, the field is simply absent — a strict superset of normal usage.

Why it matters

In production, teams increasingly route LLM traffic through a gateway to enforce budgets, cache expensive calls, and trace requests across providers. Today you can finally observe what the gateway actually did from inside LangChain — was this answer served from cache? which upstream region handled my request? — and log it alongside the response. That turns a black-box proxy into a measurable part of your pipeline: you can build cost dashboards, detect cache thrash, and verify gateway routing from a single line of code. Combined with the core change in langchain-core 1.5.6 that pushes the same metadata into LangSmith traces, you get end-to-end visibility without hand-rolling HTTP header parsing.

Example

$ Point ChatOpenAI at a LangSmith gateway and read the extracted gateway metadata off the response

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="gpt-4o-mini",
    base_url="http://localhost:8080",  # your gateway endpoint
    api_key="your-key",
    temperature=0,
)

result = llm.invoke("What is the capital of Austria?")
print("Answer:", result.content)

meta = result.response_metadata
print("Model:", meta.get("model"))
print("Gateway metadata:", meta.get("gateway", "no gateway header present"))

gw = meta.get("gateway") or {}
print("Cache hit?", gw.get("cache"))
print("Upstream region:", gw.get("region"))
print("Gateway latency ms:", gw.get("latency_ms"))

$ Same call with no gateway — the feature is a no-op

Answer: Vienna
Model: gpt-4o-mini
Gateway metadata: no gateway header present

Common flags

ChatOpenAI.invoke

Sends a prompt to the OpenAI-compatible endpoint and returns an AIMessage carrying response_metadata.

AIMessage.response_metadata

Dict where the gateway-extracted header metadata now lands.

get_num_tokens_from_messages

Fixed in this release to correctly support OpenAI o-series reasoning models.

with_structured_output

Builds typed, structured responses that also carry gateway metadata.