# AI Workflow Failure & Recovery Runbook — Free Starter
**RELESONA TECHNOLOGIES · Store Open Resource · v1.0 · 25 October 2026**

Copy this template before automating work that can modify documents, projects, databases or customer records. It is meant to make **stop, retry, undo and human escalation** visible before anything important goes wrong. No script is run by this document.

## 1. Workflow identity

- Workflow name:
- Version / source commit:
- Business owner:
- Technical maintainer:
- Environment: development / test / approved production
- Allowed systems:
- Explicitly forbidden systems:
- Trigger and expected run frequency:
- What constitutes a successful result:

## 2. Control points

| Step | Operation | Can change data? | Proof of success | Idempotency key | Retry safe? |
| --- | --- | --- | --- | --- | --- |
| 01 | Read inputs | No | | | Usually |
| 02 | Validate permissions | No | | | Usually |
| 03 | Prepare requested change | Maybe | | | Verify |
| 04 | Require human approval | No | Approval record | | N/A |
| 05 | Commit authorized operation | Yes | | | Only if idempotent |
| 06 | Independent verification | No | | | Usually |

### Minimum rule
A single **success screen** or an API response of HTTP 200 does not prove that all intended records were updated correctly. Verify the postcondition from the authoritative source before reporting completion.

## 3. Define failure classes

| Code | Example | Action | Human reviewer |
| --- | --- | --- | --- |
| INPUT | Missing / malformed data | Stop and request clarification | |
| AUTH | Expired token / permission denied | Stop; never broaden privileges automatically | |
| NETWORK | Timeout / transient provider outage | Retry only within a bounded plan | |
| CONFLICT | Concurrent revision changed | Re-read state and ask for reapproval | |
| INTEGRITY | Checksum mismatch / bad manifest | Quarantine and refuse publication | |
| SCOPE | Wrong project, folder or environment | Halt and escalate | |
| DUPLICATE | Same event received twice | Reconcile by idempotency key | |
| PRIVACY | Restricted data enters public output | Quarantine output and notify owner | |

## 4. Retry policy

- Number of attempts:
- Maximum wall-clock time:
- Safe backoff or schedule:
- Idempotency key definition:
- What will be checked before retrying?
- When is manual reconciliation required?
- Where is the last trustworthy checkpoint?

**Never repeatedly retry an irreversible operation** if you cannot confirm whether it already succeeded.

## 5. Rollback and restore

- Name of the last verified backup or source revision:
- Restoration owner and access requirements:
- Affected tables, files or external accounts:
- How to prevent new writes during recovery:
- How to verify restoration:
- Customer communication owner if necessary:
- Retention rules for audit and incident evidence:

A “backup exists” checkbox is not sufficient. Record a **tested restore** and compare meaningful records, not only file counts.

## 6. Evidence ledger

| Request ID | Start / finish time | Exact target | Actor / role | Tool result | Verified postcondition | Reviewer |
| --- | --- | --- | --- | --- | --- | --- |

Do not store private keys, signed URLs, bearer tokens or unnecessary personal data in the ledger.

## 7. Recovery acceptance tests

- [ ] Invalid scope cannot trigger an action.
- [ ] Missing approval prevents a write.
- [ ] A duplicate trigger does not double-charge or double-create a record.
- [ ] A timeout after a successful write does not cause a harmful retry.
- [ ] A stale version cannot overwrite newer work.
- [ ] A withdrawn or expired credential stops the flow.
- [ ] Failure is reported as failure rather than a fabricated success.
- [ ] Real restore results are compared to preincident evidence.

## 8. Incident closeout

- Incident ID and severity:
- Root cause, with evidence:
- What data or actions were affected?
- User/customer impact:
- Fix deployed at which exact commit?
- Restore or corrective migration performed:
- Evidence confirming recovery:
- Preventive test added:
- Responsible person and review date:

**Free starter template.** Not a production automation, disaster-recovery service, certified incident process or guarantee that actions can be undone. Validate safety and recovery in a disposable environment first.