juicebox-incident-runbook by jeremylongshore

---
name: juicebox-incident-runbook
description: |
  Execute Juicebox incident response procedures.
  Use when responding to production incidents, troubleshooting outages,
  or following incident management protocols.
  Trigger with phrases like "juicebox incident", "juicebox outage",
  "juicebox down", "juicebox emergency".
allowed-tools: Read, Write, Edit, Bash(kubectl:*), Bash(curl:*)
version: 1.0.0
license: MIT
author: Jeremy Longshore <[email protected]>
---

# Juicebox Incident Runbook

## Overview
Standardized incident response procedures for Juicebox integration issues.

## Incident Severity Levels

| Severity | Description | Response Time | Examples |
|----------|-------------|---------------|----------|
| P1 | Critical | < 15 min | Complete outage, data loss |
| P2 | High | < 1 hour | Major feature broken, degraded performance |
| P3 | Medium | < 4 hours | Minor feature issue, workaround exists |
| P4 | Low | < 24 hours | Cosmetic, non-blocking |

## Quick Diagnostics

### Step 1: Immediate Assessment
```bash
#!/bin/bash
# quick-diag.sh - Run immediately when incident detected

echo "=== Juicebox Quick Diagnostics ==="
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"

# Check Juicebox status page
echo ""
echo "=== Juicebox Status ==="
curl -s https://status.juicebox.ai/api/status | jq '.status'

# Check our API health
echo ""
echo "=== Our API Health ==="
curl -s http://localhost:8080/health/ready | jq '.'

# Check recent error logs
echo ""
echo "=== Recent Errors (last 5 min) ==="
kubectl logs -l app=juicebox-integration --since=5m | grep -i error | tail -20

# Check metrics
echo ""
echo "=== Error Rate ==="
curl -s http://localhost:9090/api/v1/query?query=rate\(juicebox_requests_total\{status=\"error\"\}\[5m\]\) | jq '.data.result[0].value[1]'
```

### Step 2: Identify Root Cause
```markdown
## Incident Triage Decision Tree

1. Is Juicebox status page showing issues?
   - YES → External outage, skip to "External Outage Response"
   - NO → Continue

2. Are we getting authentication errors (401)?
   - YES → Check API key validity, skip to "Auth Issues"
   - NO → Continue

3. Are we getting rate limited (429)?
   - YES → Skip to "Rate Limit Response"
   - NO → Continue

4. Are requests timing out?
   - YES → Skip to "Timeout Response"
   - NO → Continue

5. Are we getting unexpected errors?
   - YES → Skip to "Application Error Response"
   - NO → Gather more data
```

## Response Procedures

### External Outage Response
```markdown
## When Juicebox is Down

1. **Confirm Outage**
   - Check https://status.juicebox.ai
   - Verify with curl test to API

2. **Enable Fallback Mode**
   ```bash
   kubectl set env deployment/juicebox-integration JUICEBOX_FALLBACK=true
   ```

3. **Notify Stakeholders**
   - Post to #incidents channel
   - Update status page if customer-facing

4. **Monitor Recovery**
   - Set up alert for Juicebox status change
   - Prepare to disable fallback mode

5. **Post-Incident**
   - Disable fallback when Juicebox recovers
   - Document timeline and impact
```

### Auth Issues Response
```markdown
## When Authentication Fails

1. **Verify API Key**
   ```bash
   # Mask key for logging
   echo "Key prefix: ${JUICEBOX_API_KEY:0:10}..."

   # Test auth
   curl -H "Authorization: Bearer $JUICEBOX_API_KEY" \
     https://api.juicebox.ai/v1/auth/me
   ```

2. **Check Key Status in Dashboard**
   - Log into https://app.juicebox.ai
   - Verify key is active and not revoked

3. **Rotate Key if Compromised**
   - Generate new key in dashboard
   - Update secret manager
   - Restart pods
   ```bash
   kubectl rollout restart deployment/juicebox-integration
   ```

4. **Verify Recovery**
   - Check health endpoint
   - Monitor error rate
```

### Rate Limit Response
```markdown
## When Rate Limited

1. **Check Current Usage**
   ```bash
   curl -H "Authorization: Bearer $JUICEBOX_API_KEY" \
     https://api.juicebox.ai/v1/usage
   ```

2. **Immediate Mitigation**
   - Enable aggressive caching
   - Reduce request rate
   ```bash
   kubectl set env deployment/juicebox-integration JUICEBOX_RATE_LIMIT=10
   ```

3. **If Quota Exhausted**
   - Contact Juicebox support for temporary increase
   - Implement request queuing

4. **Long-term Fix**
   - Review usage patterns
   - Implement better caching
   - Consider plan upgrade
```

### Timeout Response
```markdown
## When Requests Timeout

1. **Check Network**
   ```bash
   # DNS resolution
   nslookup api.juicebox.ai

   # Connectivity
   curl -v --connect-timeout 5 https://api.juicebox.ai/v1/health
   ```

2. **Check Load**
   - Review query complexity
   - Check for unusually large requests

3. **Increase Timeout**
   ```bash
   kubectl set env deployment/juicebox-integration JUICEBOX_TIMEOUT=60000
   ```

4. **Implement Circuit Breaker**
   - Enable circuit breaker if repeated timeouts
   - Serve cached/fallback data
```

## Incident Communication Template

```markdown
## Incident Report Template

**Incident ID:** INC-YYYY-MM-DD-XXX
**Status:** Investigating | Identified | Monitoring | Resolved
**Severity:** P1 | P2 | P3 | P4
**Start Time:** YYYY-MM-DD HH:MM UTC
**End Time:** (when resolved)

### Summary
[Brief description of the incident]

### Impact
- Users affected: [number/percentage]
- Features affected: [list]
- Duration: [time]

### Timeline
- HH:MM - Incident detected
- HH:MM - Investigation started
- HH:MM - Root cause identified
- HH:MM - Mitigation applied
- HH:MM - Incident resolved

### Root Cause
[Description of what caused the incident]

### Resolution
[What was done to fix it]

### Action Items
- [ ] Action 1 (Owner, Due Date)
- [ ] Action 2 (Owner, Due Date)
```

## On-Call Checklist

```markdown
## On-Call Handoff Checklist

### Before Shift
- [ ] Access to all monitoring dashboards
- [ ] VPN/access to production systems
- [ ] Runbook bookmarked
- [ ] Escalation contacts available

### During Shift
- [ ] Check dashboards every 30 min
- [ ] Respond to alerts within SLA
- [ ] Document all incidents
- [ ] Escalate P1/P2 immediately

### End of Shift
- [ ] Handoff open incidents
- [ ] Update incident log
- [ ] Brief incoming on-call
```

## Output
- Diagnostic scripts
- Response procedures
- Communication templates
- On-call checklists

## Resources
- [Juicebox Status](https://status.juicebox.ai)
- [Support Portal](https://juicebox.ai/support)

## Next Steps
After incident, see `juicebox-data-handling` for data management.