# Failover events

The four analytics events that show how your agent was routed to a model, why a provider call failed, and whether any requests were lost.

**Path:** Analytics → Event Explorer

Four events record how each request was routed to a model provider: which route served it, why a provider call failed, and whether the answer was lost. Use them to answer "why did the agent get worse last Tuesday?", "is our API key working?", and "did we lose any requests?"

[High availability](/user-guides/high-availability/) explains how the routing decision is made. This page covers what it records.

## Routes and passes

Two terms run through every event.

A **route** is one combination of a model and a provider key. Every agent has an ordered list of routes: its configured model on its own key first, then alternatives. Moving down that list is **failover**.

A **pass** is one request-and-response exchange with a model. A single user message is often several passes, because the agent asks the model, runs a tool, then asks again. Each pass fails and recovers independently, so one message can produce more than one failover.

## The four events

| Event | What it means | When it fires |
| :--- | :--- | :--- |
| **Agent Failover** | The request didn't run on the agent's first-choice route | Once per pass, when a later route succeeded after an earlier one failed |
| **LLM Provider Failed** | A call to a provider failed | Once for every failed attempt, whether or not the request recovered |
| **Agent Response Failed** | The agent couldn't answer at all | When every route is exhausted and the user is shown an error |
| **LLM Failback** | A key or model that was taken out of rotation is working again | When the resource is confirmed healthy, or its cooldown expires |

**Agent Failover** is a record of a rescue: it only exists when the user did get an answer. It fires once per pass no matter how many routes were tried, and never fires when every route failed.

**LLM Provider Failed** is the honest count of trouble, including failures masked by a successful retry. Two failed routes in one pass produce two events. Background health checks are synthetic and never counted.

**Agent Response Failed** is the count of requests genuinely lost. It carries the id of the error message the user saw, so you can find the exact conversation. An agent hitting its consecutive tool-call limit is not a routing failure and does not produce this event.

**LLM Failback** fires when a resource returns to rotation, in one of three ways: a background health check succeeds, normal traffic succeeds after the cooldown ended, or the cooldown expires. Rate-limited and billing-capped keys are never health-checked, since testing them burns the exhausted quota, so they always return on expiry. The event fires for both your own keys and Foldspace platform keys.

## Failure reason

The most useful property on these events, because it determines whether there is anything you can do.

| Reason | What happened | What to do |
| :--- | :--- | :--- |
| `BILLING_CAP` | Spend or quota exhausted at the provider | Top up or raise the quota |
| `RATE_LIMITED` | The key hit its per-minute request or token limit | Raise the limit or reduce load |
| `PROVIDER_OVERLOADED` | The provider itself is saturated | Wait it out |
| `UNAVAILABLE` | Transient provider outage, or the request timed out | Wait it out |
| `AUTH` | The provider rejected the key | Rotate or fix the key |
| `OTHER` | Unclassified, or a bad request that won't be retried | Contact support |

The meaning shifts with the event. On **LLM Provider Failed** it describes the attempt that just failed. On **Agent Failover** it describes the failure that *caused* the move to another route, not the route that worked. On **LLM Failback** it describes why the resource was taken out of rotation.

## Properties

| Property | What it tells you |
| :--- | :--- |
| **LLM Failure Reason** | Why the provider call failed, in one of the six categories above. |
| **LLM Failure Status Code** | The code the provider returned, such as 429 for a rate limit or 401 for a bad key. Often empty: a timed-out request never got a response, so there is no code. |
| **LLM Configured Model** | The model the agent is set up to use. |
| **LLM Effective Model** | The model that actually answered, or on LLM Provider Failed, the one that failed. |
| **LLM Key Owner** | Who owns the key that served the request: `BYOK` for your own key, `FOLDSPACE` for a platform key. Shown in the admin UI as **Key Owner**, with the values **Customer** and **Foldspace**. |
| **LLM Configured Key Id** | A reference to the stored key record the agent was set up with, never the API key itself. Empty for platform keys, and for keys since deleted. |
| **LLM Effective Key Id** | A reference to the key that served, failed, or recovered. Empty for platform keys by design. |
| **LLM Down Duration Seconds** | How long a key or model was out of rotation, from the first failure that tripped it until it recovered. |
| **LLM Cooldown Seconds** | How long traffic was held back before the resource was checked again. Each failed health check roughly doubles it, up to one hour, so a large value means the resource had been failing for a while. |
| **LLM Health Probe Interval Seconds** | How often out-of-rotation resources are checked. A recovery is noticed at worst one interval after it happens. This is a system-wide setting, identical for every account. |

Comparing **LLM Configured Model** against **LLM Effective Model** is the core check for "was my traffic silently moved to a different model?" If they differ, the answer came from something other than what you configured.

:::caution
A model mismatch is not always a failure. When a provider retires a model, Foldspace remaps the agent to its replacement, which produces the same mismatch on every message even though nothing failed. A mismatch with **no** Agent Failover event anywhere is almost always a retirement remap.
:::

Every pass also carries an internal identifier that links these events to each other and to the message. It is not selectable in Analytics; joining events by pass is a support request.

## Which properties appear on which event

| Property | Agent Failover | LLM Provider Failed | Agent Response Failed | LLM Failback |
| :--- | :--- | :--- | :--- | :--- |
| LLM Failure Reason | Always | Always | Never | Always |
| LLM Failure Status Code | If sent | If sent | Never | If sent |
| LLM Configured Model | Always | Always | Usually | Always |
| LLM Effective Model | Always | Always | Usually | Always |
| LLM Key Owner | Always | Always | Usually | Always |
| LLM Configured Key Id | Your keys only | Your keys only | Your keys only | Never |
| LLM Effective Key Id | Your keys only | Your keys only | Your keys only | Your keys only |
| LLM Down Duration Seconds | Never | Never | Never | Always |
| LLM Cooldown Seconds | Never | Never | Never | Always |
| LLM Health Probe Interval Seconds | Never | Never | Never | Always |
| Pass identifier | Always | Always | Usually | Never |
| Conversation id | Always | Always | Always | Never |
| Message id | Never | Never | Always | Never |

Four rows need an explanation.

- **Agent Response Failed has no failure reason.** It tells you a request was lost, not why. For the cause, read the LLM Provider Failed events from the same conversation at the same moment. A key that had expired shows `AUTH` there and nothing here.
- **"Usually" on Agent Response Failed** means the routing details are present whenever the failure happened while talking to a provider, which is the normal case. A request that broke before it reached a provider carries only the conversation and message, which tells you the problem was not a routing problem.
- **LLM Failback carries no conversation or pass.** It is triggered by a resource recovering, not by a request. Match it to failures by key and time.
- **"Your keys only"** means the property is filled in when the request ran on your own provider key, and empty on a Foldspace platform key.

When a property does not apply it is left out rather than shown as zero. Don't read an empty duration as "recovered instantly".

## Failures with no failover

LLM Provider Failed events with no matching Agent Failover are usually correct, for one of three reasons:

- **Everything failed.** There was no rescue to record, and you will see Agent Response Failed instead.
- **The answer had already started.** Once the provider begins streaming, part of the answer is on screen and the request can't restart elsewhere.
- **The failure wasn't worth retrying.** Some errors, such as a malformed request, fail identically on every route.

## Common investigations

### The agent got worse last Tuesday

Filter **Agent Failover** for that agent over the period. No events means routing wasn't the cause.

If there are events, compare **LLM Configured Model** against **LLM Effective Model** on the affected messages. Where they differ, a different model answered, and the failover events say why. The span of messages where they disagree tells you how long it went on. Rule out a retirement remap first, then check **LLM Key Owner**: `FOLDSPACE` means the traffic ran on a platform key for that period.

### Is our API key working?

Look for **LLM Provider Failed** or **Agent Failover** with **LLM Key Owner** set to `BYOK`, and read the failure reason. That value usually determines the whole fix: top up, raise a limit, rotate the key, or wait.

To confirm the key recovered, find an **LLM Failback** with the same **LLM Effective Key Id** shortly afterwards. If the reason was `BILLING_CAP` or `RATE_LIMITED`, recovery comes from the cooldown expiring rather than a health check, which is expected.

### Did we lose any requests?

Count **Agent Response Failed**. Anything above zero means users saw errors, and the LLM Provider Failed events from the same conversation explain each one.

## Things that can mislead you

- **A missing status code doesn't mean success.** Timeouts have no code. Read the failure reason.
- **Timeouts and provider outages look identical.** Both report `UNAVAILABLE`.
- **Provider failure counts include invisible failures.** Many LLM Provider Failed events can coexist with a perfectly good user experience, because every one recovered. Check Agent Response Failed before concluding users were affected.
- **Failbacks on platform keys aren't per-account.** Foldspace platform keys serve everyone, and the recovery event is attributed to whichever traffic first tripped the key.

## Related

- [High availability](/user-guides/high-availability/): how failover picks the next route.
- [AI Provider Keys](/user-guides/settings/ai-api-keys/): add and rotate the keys failover routes across.
- [Properties & filters](/user-guides/analytics/properties-filters/): filter and break down any of these events.