Back to overview

Incident detail

ResolvedcriticalControlled test

Lead-to-opportunity lifecycle simulation journey failed

Controlled synthetic failure: upstream API returned 503

Incident duration

1m 30s

Start to verified recovery

What failed

Account

KODA

Location

KODA Sentinel Sandbox

Journey

Lead-to-opportunity lifecycle simulation

Failure fingerprint

bb037e1547536d28fc1d91a1

Preserved evidence

{
  "failedStep": "Inject controlled failure",
  "stepResults": [
    {
      "ok": true,
      "name": "Authenticate and load location",
      "detail": "Simulated dependency available",
      "durationMs": 42
    },
    {
      "ok": false,
      "name": "Inject controlled failure",
      "detail": "Controlled synthetic failure: upstream API returned 503",
      "durationMs": 18
    }
  ],
  "completedSteps": [
    "Authenticate and load location"
  ],
  "failedStepDetail": "Controlled synthetic failure: upstream API returned 503",
  "firstAttemptRunId": "edc61152-5e14-485c-b18a-982db944566c"
}

Check and recovery timeline

Detection, confirmation, subsequent checks, alerts, and recovery

Failure Detected

Sep 16, 2026, 1:26:37 PM MST

Initial failure detected; confirmation retry started

Run edc61152-5e14-485c-b18a-982db944566c

View event evidence
{
  "confirmationRequired": true
}

Incident Opened

Sep 16, 2026, 1:26:37 PM MST

Failure confirmed after retry: Controlled synthetic failure: upstream API returned 503

Run 8cd80ccc-e0f2-4908-abe7-8687e6b10ccc

View event evidence
{
  "failedStep": "Inject controlled failure",
  "diagnostics": {
    "cycleId": "129842e2-1766-4b53-93f7-9045bb3ab556",
    "controlled": true,
    "externalCalls": 0,
    "highLevelWrites": 0,
    "confirmationAttempts": 2,
    "externalNotifications": 0
  },
  "fingerprint": "bb037e1547536d28fc1d91a1",
  "stepResults": [
    {
      "ok": true,
      "name": "Authenticate and load location",
      "detail": "Simulated dependency available",
      "durationMs": 42
    },
    {
      "ok": false,
      "name": "Inject controlled failure",
      "detail": "Controlled synthetic failure: upstream API returned 503",
      "durationMs": 18
    }
  ],
  "completedSteps": [
    "Authenticate and load location"
  ],
  "failedStepDetail": "Controlled synthetic failure: upstream API returned 503",
  "firstAttemptRunId": "edc61152-5e14-485c-b18a-982db944566c"
}

Check Passed

Sep 16, 2026, 1:28:07 PM MST

Recovery check passed in 47ms

Run a2ea647d-0429-42b0-8373-613ad777b04a

View event evidence
{
  "diagnostics": {
    "cycleId": "129842e2-1766-4b53-93f7-9045bb3ab556",
    "controlled": true,
    "externalCalls": 0,
    "externalNotifications": 0
  },
  "stepResults": [
    {
      "ok": true,
      "name": "Controlled recovery check",
      "detail": "Simulated dependency recovered",
      "durationMs": 47
    }
  ]
}

Incident Resolved

Sep 16, 2026, 1:28:07 PM MST

Incident resolved after 90000ms

Run a2ea647d-0429-42b0-8373-613ad777b04a

View event evidence
{
  "durationMs": 90000
}

Alert Captured

Sep 16, 2026, 1:28:07 PM MST

Incident email captured safely

View event evidence
{
  "mode": "capture",
  "recipient": "not-configured",
  "idempotencyKey": "sentinel:incident_opened:a8bc2e98-ce83-4af4-9ff0-246f32421f4b"
}

Alert Captured

Sep 16, 2026, 1:28:07 PM MST

Recovery email captured safely

View event evidence
{
  "mode": "capture",
  "recipient": "not-configured",
  "idempotencyKey": "sentinel:incident_resolved:a8bc2e98-ce83-4af4-9ff0-246f32421f4b"
}

Timing

Started
Sep 16, 2026, 1:26:37 PM MST
Last checked
Sep 16, 2026, 1:28:07 PM MST
Recovered
Sep 16, 2026, 1:28:07 PM MST
Confirmed failures
1

Owner alerts

Incident email

captured

not-configured · 1 attempt

Recovery email

captured

not-configured · 1 attempt

Diagnostics

{
  "cycleId": "129842e2-1766-4b53-93f7-9045bb3ab556",
  "controlled": true,
  "externalCalls": 0,
  "highLevelWrites": 0,
  "confirmationAttempts": 2,
  "externalNotifications": 0
}
America/Phoenix
Opening run 8cd80ccc-e0f2-4908-abe7-8687e6b10ccc
Recovery run a2ea647d-0429-42b0-8373-613ad777b04a