Analytics

Failover events

Path: Analytics → Event Explorer

Four events record how each request was routed to a model provider: which route served it, why a provider call failed, and whether the answer was lost. Use them to answer “why did the agent get worse last Tuesday?”, “is our API key working?”, and “did we lose any requests?”

High availability explains how the routing decision is made. This page covers what it records.

Routes and passes

Two terms run through every event.

A route is one combination of a model and a provider key. Every agent has an ordered list of routes: its configured model on its own key first, then alternatives. Moving down that list is failover.

A pass is one request-and-response exchange with a model. A single user message is often several passes, because the agent asks the model, runs a tool, then asks again. Each pass fails and recovers independently, so one message can produce more than one failover.

The four events

Event	What it means	When it fires
Agent Failover	The request didn’t run on the agent’s first-choice route	Once per pass, when a later route succeeded after an earlier one failed
LLM Provider Failed	A call to a provider failed	Once for every failed attempt, whether or not the request recovered
Agent Response Failed	The agent couldn’t answer at all	When every route is exhausted and the user is shown an error
LLM Failback	A key or model that was taken out of rotation is working again	When the resource is confirmed healthy, or its cooldown expires

Agent Failover is a record of a rescue: it only exists when the user did get an answer. It fires once per pass no matter how many routes were tried, and never fires when every route failed.

LLM Provider Failed is the honest count of trouble, including failures masked by a successful retry. Two failed routes in one pass produce two events. Background health checks are synthetic and never counted.

Agent Response Failed is the count of requests genuinely lost. It carries the id of the error message the user saw, so you can find the exact conversation. An agent hitting its consecutive tool-call limit is not a routing failure and does not produce this event.

LLM Failback fires when a resource returns to rotation, in one of three ways: a background health check succeeds, normal traffic succeeds after the cooldown ended, or the cooldown expires. Rate-limited and billing-capped keys are never health-checked, since testing them burns the exhausted quota, so they always return on expiry. The event fires for both your own keys and Foldspace platform keys.

Failure reason

The most useful property on these events, because it determines whether there is anything you can do.

Reason	What happened	What to do
`BILLING_CAP`	Spend or quota exhausted at the provider	Top up or raise the quota
`RATE_LIMITED`	The key hit its per-minute request or token limit	Raise the limit or reduce load
`PROVIDER_OVERLOADED`	The provider itself is saturated	Wait it out
`UNAVAILABLE`	Transient provider outage, or the request timed out	Wait it out
`AUTH`	The provider rejected the key	Rotate or fix the key
`OTHER`	Unclassified, or a bad request that won’t be retried	Contact support

The meaning shifts with the event. On LLM Provider Failed it describes the attempt that just failed. On Agent Failover it describes the failure that caused the move to another route, not the route that worked. On LLM Failback it describes why the resource was taken out of rotation.

Properties

Property	What it tells you
LLM Failure Reason	Why the provider call failed, in one of the six categories above.
LLM Failure Status Code	The code the provider returned, such as 429 for a rate limit or 401 for a bad key. Often empty: a timed-out request never got a response, so there is no code.
LLM Configured Model	The model the agent is set up to use.
LLM Effective Model	The model that actually answered, or on LLM Provider Failed, the one that failed.
LLM Key Owner	Who owns the key that served the request: `BYOK` for your own key, `FOLDSPACE` for a platform key. Shown in the admin UI as Key Owner, with the values Customer and Foldspace.
LLM Configured Key Id	A reference to the stored key record the agent was set up with, never the API key itself. Empty for platform keys, and for keys since deleted.
LLM Effective Key Id	A reference to the key that served, failed, or recovered. Empty for platform keys by design.
LLM Down Duration Seconds	How long a key or model was out of rotation, from the first failure that tripped it until it recovered.
LLM Cooldown Seconds	How long traffic was held back before the resource was checked again. Each failed health check roughly doubles it, up to one hour, so a large value means the resource had been failing for a while.
LLM Health Probe Interval Seconds	How often out-of-rotation resources are checked. A recovery is noticed at worst one interval after it happens. This is a system-wide setting, identical for every account.

Comparing LLM Configured Model against LLM Effective Model is the core check for “was my traffic silently moved to a different model?” If they differ, the answer came from something other than what you configured.

Every pass also carries an internal identifier that links these events to each other and to the message. It is not selectable in Analytics; joining events by pass is a support request.

Which properties appear on which event

Property	Agent Failover	LLM Provider Failed	Agent Response Failed	LLM Failback
LLM Failure Reason	Always	Always	Never	Always
LLM Failure Status Code	If sent	If sent	Never	If sent
LLM Configured Model	Always	Always	Usually	Always
LLM Effective Model	Always	Always	Usually	Always
LLM Key Owner	Always	Always	Usually	Always
LLM Configured Key Id	Your keys only	Your keys only	Your keys only	Never
LLM Effective Key Id	Your keys only	Your keys only	Your keys only	Your keys only
LLM Down Duration Seconds	Never	Never	Never	Always
LLM Cooldown Seconds	Never	Never	Never	Always
LLM Health Probe Interval Seconds	Never	Never	Never	Always
Pass identifier	Always	Always	Usually	Never
Conversation id	Always	Always	Always	Never
Message id	Never	Never	Always	Never

Four rows need an explanation.

Agent Response Failed has no failure reason. It tells you a request was lost, not why. For the cause, read the LLM Provider Failed events from the same conversation at the same moment. A key that had expired shows AUTH there and nothing here.
“Usually” on Agent Response Failed means the routing details are present whenever the failure happened while talking to a provider, which is the normal case. A request that broke before it reached a provider carries only the conversation and message, which tells you the problem was not a routing problem.
LLM Failback carries no conversation or pass. It is triggered by a resource recovering, not by a request. Match it to failures by key and time.
“Your keys only” means the property is filled in when the request ran on your own provider key, and empty on a Foldspace platform key.

When a property does not apply it is left out rather than shown as zero. Don’t read an empty duration as “recovered instantly”.

Failures with no failover

LLM Provider Failed events with no matching Agent Failover are usually correct, for one of three reasons:

Everything failed. There was no rescue to record, and you will see Agent Response Failed instead.
The answer had already started. Once the provider begins streaming, part of the answer is on screen and the request can’t restart elsewhere.
The failure wasn’t worth retrying. Some errors, such as a malformed request, fail identically on every route.

Common investigations

The agent got worse last Tuesday

Filter Agent Failover for that agent over the period. No events means routing wasn’t the cause.

If there are events, compare LLM Configured Model against LLM Effective Model on the affected messages. Where they differ, a different model answered, and the failover events say why. The span of messages where they disagree tells you how long it went on. Rule out a retirement remap first, then check LLM Key Owner: FOLDSPACE means the traffic ran on a platform key for that period.

Is our API key working?

Look for LLM Provider Failed or Agent Failover with LLM Key Owner set to BYOK, and read the failure reason. That value usually determines the whole fix: top up, raise a limit, rotate the key, or wait.

To confirm the key recovered, find an LLM Failback with the same LLM Effective Key Id shortly afterwards. If the reason was BILLING_CAP or RATE_LIMITED, recovery comes from the cooldown expiring rather than a health check, which is expected.

Did we lose any requests?

Count Agent Response Failed. Anything above zero means users saw errors, and the LLM Provider Failed events from the same conversation explain each one.

Things that can mislead you

A missing status code doesn’t mean success. Timeouts have no code. Read the failure reason.
Timeouts and provider outages look identical. Both report UNAVAILABLE.
Provider failure counts include invisible failures. Many LLM Provider Failed events can coexist with a perfectly good user experience, because every one recovered. Check Agent Response Failed before concluding users were affected.
Failbacks on platform keys aren’t per-account. Foldspace platform keys serve everyone, and the recovery event is attributed to whichever traffic first tripped the key.

High availability: how failover picks the next route.
AI Provider Keys: add and rotate the keys failover routes across.
Properties & filters: filter and break down any of these events.