Back to overview

Incident detail

ResolvedcriticalControlled test

Lead-to-opportunity lifecycle simulation journey failed

Controlled synthetic failure: upstream API returned 503

Incident duration

1m 30s

Start to verified recovery

What failed

Account

KODA

Location

KODA Sentinel Sandbox

Journey

Lead-to-opportunity lifecycle simulation

Failure fingerprint

bb037e1547536d28fc1d91a1

Preserved evidence

{
  "failedStep": "Inject controlled failure",
  "stepResults": [
    {
      "ok": true,
      "name": "Authenticate and load location",
      "detail": "Simulated dependency available",
      "durationMs": 42
    },
    {
      "ok": false,
      "name": "Inject controlled failure",
      "detail": "Controlled synthetic failure: upstream API returned 503",
      "durationMs": 18
    }
  ],
  "completedSteps": [
    "Authenticate and load location"
  ],
  "failedStepDetail": "Controlled synthetic failure: upstream API returned 503",
  "firstAttemptRunId": "09a9177f-726d-476b-9727-baa3ea633223"
}

Check and recovery timeline

Detection, confirmation, subsequent checks, alerts, and recovery

Failure Detected

Sep 16, 2026, 2:11:47 PM MST

Initial failure detected; confirmation retry started

Run 09a9177f-726d-476b-9727-baa3ea633223

View event evidence
{
  "confirmationRequired": true
}

Incident Opened

Sep 16, 2026, 2:11:47 PM MST

Failure confirmed after retry: Controlled synthetic failure: upstream API returned 503

Run 9432789b-dd96-4025-bf02-d7629cae9aa8

View event evidence
{
  "failedStep": "Inject controlled failure",
  "diagnostics": {
    "cycleId": "6691b058-e348-40a2-891c-d6eebb4fa59d",
    "controlled": true,
    "ownerAlertMode": "live",
    "confirmationAttempts": 2,
    "syntheticHighLevelWrites": 0,
    "syntheticJourneyExternalCalls": 0,
    "syntheticCustomerNotifications": 0
  },
  "fingerprint": "bb037e1547536d28fc1d91a1",
  "stepResults": [
    {
      "ok": true,
      "name": "Authenticate and load location",
      "detail": "Simulated dependency available",
      "durationMs": 42
    },
    {
      "ok": false,
      "name": "Inject controlled failure",
      "detail": "Controlled synthetic failure: upstream API returned 503",
      "durationMs": 18
    }
  ],
  "completedSteps": [
    "Authenticate and load location"
  ],
  "failedStepDetail": "Controlled synthetic failure: upstream API returned 503",
  "firstAttemptRunId": "09a9177f-726d-476b-9727-baa3ea633223"
}

Check Passed

Sep 16, 2026, 2:13:17 PM MST

Recovery check passed in 47ms

Run 53f6fb7e-aa0b-4a52-93bc-98632b27e81f

View event evidence
{
  "diagnostics": {
    "cycleId": "6691b058-e348-40a2-891c-d6eebb4fa59d",
    "controlled": true,
    "ownerAlertMode": "live",
    "syntheticJourneyExternalCalls": 0,
    "syntheticCustomerNotifications": 0
  },
  "stepResults": [
    {
      "ok": true,
      "name": "Controlled recovery check",
      "detail": "Simulated dependency recovered",
      "durationMs": 47
    }
  ]
}

Incident Resolved

Sep 16, 2026, 2:13:17 PM MST

Incident resolved after 90000ms

Run 53f6fb7e-aa0b-4a52-93bc-98632b27e81f

View event evidence
{
  "durationMs": 90000
}

Alert Sent

Sep 16, 2026, 2:13:18 PM MST

Incident email sent

View event evidence
{
  "provider": "resend",
  "recipient": "wearewarrioruniversity@gmail.com",
  "providerMessageId": "01a0ac11-1dc2-716a-8146-c8e80be09472"
}

Alert Sent

Sep 16, 2026, 2:13:18 PM MST

Recovery email sent

View event evidence
{
  "provider": "resend",
  "recipient": "wearewarrioruniversity@gmail.com",
  "providerMessageId": "01a0ac11-1e4c-74cd-9e37-ab5cf916a2d2"
}

Timing

Started
Sep 16, 2026, 2:11:47 PM MST
Last checked
Sep 16, 2026, 2:13:17 PM MST
Recovered
Sep 16, 2026, 2:13:17 PM MST
Confirmed failures
1

Owner alerts

Incident email

sent

we•••@gmail.com · 1 attempt

Recovery email

sent

we•••@gmail.com · 1 attempt

Diagnostics

{
  "cycleId": "6691b058-e348-40a2-891c-d6eebb4fa59d",
  "controlled": true,
  "ownerAlertMode": "live",
  "confirmationAttempts": 2,
  "syntheticHighLevelWrites": 0,
  "syntheticJourneyExternalCalls": 0,
  "syntheticCustomerNotifications": 0
}
America/Phoenix
Opening run 9432789b-dd96-4025-bf02-d7629cae9aa8
Recovery run 53f6fb7e-aa0b-4a52-93bc-98632b27e81f