---
name: vastai-incident-runbook
description: >-
Analyze, diagnose, and recover a Vast.ai renter workload from outbid, exited, offline, scheduling, low-credit, or data-risk incidents while controlling billing and destructive actions. Use when a Vast.ai workload or account enters an incident state. Trigger with: "Vast.ai incident", "recover an offline Vast.ai job", "stop emergency Vast.ai spend".
allowed-tools: Read, Grep, Write, Edit
version: 2.0.0
argument-hint: '[severity-resource-ids-and-last-good-checkpoint]'
model: inherit
effort: high
license: MIT
author: Jeremy Longshore <[email protected]>
tags:
- saas
- vastai
- incident-response
- recovery
- billing
compatibility: 'Requires affected resource IDs, read access, external checkpoints, incident authority, and a named billing owner.'
---
# Vast.ai Workload Recovery Runbook
## Overview
Protect people and credentials first, then preserve recoverable data, stop uncontrolled spend, and restore service from an external checkpoint or last-known-good template. Provider states imply different actions and must not be collapsed into retry.
## Prerequisites
- Severity, start time, affected instances/endpoints, release, and user impact
- Last verified external checkpoint or Serverless template
- Incident commander, data owner, billing owner, and authority for stop/destroy/rollback
## Instructions
### Step 1: Stabilize identity and scope
Confirm account/team context, affected IDs, current state, balance, and whether a credential or data exposure is involved.
### Step 2: Classify the provider state
Outbid/stopped may retain disk and storage charges; exited is workload failure; scheduling may await a reclaimed GPU; unknown/offline is host heartbeat loss; low credit threatens availability and data.
### Step 3: Preserve recovery evidence
Copy reachable artifacts and logs, record template/image identity, and verify the newest external checkpoint before any destroy.
### Step 4: Contain cost or exposure
Stop new creates, revoke compromised keys, and use stop or destroy only under the incident's data-versus-cost decision.
### Step 5: Restore on a clean target
Resume from the verified checkpoint on a compliant different offer, or roll Serverless back to the last-known-good template.
### Step 6: Reconcile and close
Confirm service and output, destroy superseded resources, audit charges, rotate temporary access, and document the gap and prevention action.
## Authentication
Use an incident key with the minimum temporary permissions and a short revocation deadline. Keep billing-write, team administration, workload storage, and control-plane authority separated.
## Tool Discipline
Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval.
## Output
- State-based incident timeline and containment decision
- Checkpoint/template recovery and service-validation evidence
- Resource cleanup, charge reconciliation, access rotation, and postmortem receipt
Return severity, IDs, state class, last good checkpoint/template, containment, restored target, data gap, spend impact, and closed resources.
## Examples
After a host goes offline, the team avoids blind restarts, restores the last external checkpoint on a different verified offer, validates output, and retains the original instance ID for support and billing reconciliation.
## Error Handling
| Failure | Response |
| --- | --- |
| No external checkpoint exists | State the recovery gap explicitly and attempt data salvage only if the host becomes reachable. |
| Balance is zero or negative | Escalate immediately; instances stop and resources may be at risk of deletion. |
| Destruction would erase the only data | Require the incident commander and data owner to make the containment decision. |
| Replacement also fails | Stop churn, compare common image/data/config factors, and escalate with redacted evidence. |
## Resources
- [First-party source notes](references/official-docs.md)
- [Troubleshooting](https://docs.vast.ai/guides/reference/troubleshooting)
- [Billing and negative balances](https://docs.vast.ai/guides/reference/billing#negative-balances)
- [Manage instances](https://docs.vast.ai/guides/instances/manage-instances)
