Backup & Disaster Recovery

This page turns §12 of the architecture into runbooks. Targets come from the architecture and are not changed here.

1. Backup matrix

Asset	Mechanism	Frequency / retention	Copies	Encryption	Verified by
App-cell PostgreSQL (RDS)	Automated backups + PITR; Multi-AZ synchronous standby	Continuous WAL; PITR window 35 days	Daily snapshot copied to ap-south-2 (35 days); monthly snapshot kept 12 months in the spyrl-dr account	KMS (cell key; copies re-encrypted with a DR-region key)	Weekly automated restore test (section 5)
Control-plane PostgreSQL	Same as app cell + nightly logical dump (pg_dump, custom format) to S3	35 days PITR; dumps 90 days	ap-south-2	KMS	Weekly restore test
Object storage (tenant files, payslips, statutory files, exports)	S3 versioning; Object Lock compliance mode for locked payroll artefacts (payslips, run snapshots, statutory files); cross-region replication	Non-current versions 90 days; locked artefacts for the retention period	ap-south-2 replica bucket in spyrl-dr	SSE-KMS	Replication metrics + monthly object sample check
Redis	Ephemeral. Not backed up. Queues rebuild from the transactional outbox; caches warm lazily	—	—	In-transit and at-rest encryption on ElastiCache	DR drill confirms rebuild
Per-tenant logical export	Nightly export for Enterprise tenants; on demand for any tenant	30 days	Same-region bucket + replication	Envelope-encrypted with the tenant data key	Monthly import test into staging
Infrastructure definition	Terraform state (S3 + DynamoDB lock, versioned); Dokploy config export; infra/ in git	Every change; Dokploy export nightly	Git remote + S3 replica	—	Rebuild during the DR drill
Secrets	Secrets Manager with cross-region replication for production secrets	—	ap-south-2	KMS	DR drill
KMS keys	Multi-region keys for envelope encryption, so DR-region restores can decrypt	—	ap-south-2 replica key	—	DR drill decrypts sample fields
Audit log	Part of the app DB + daily export to an Object Lock bucket	Per retention schedule (privacy)	ap-south-2	KMS	Monthly checksum chain check

Backup access. Snapshot copies in the spyrl-dr account are protected by AWS Backup Vault Lock, so neither a compromised production account nor an engineer mistake can delete them before retention expires. Only two named people hold restore rights in that account.

2. RPO and RTO

From architecture §12; these are the targets drills are measured against.

Scenario	RPO	RTO	Mechanism
Single DB instance or AZ failure	0 (synchronous standby)	~2 minutes automatic; ≤ 1 h worst case	RDS Multi-AZ failover
Logical corruption, bad migration, mass deletion in a cell	≤ 5 minutes	≤ 1 h for a full-cell PITR	PITR to a new instance, cut over
One tenant's data damaged (mistake by the client or a bug)	≤ 5 minutes	≤ 2 h to restore into staging for review; import after approval	Single-tenant restore (section 4)
Object storage deletion or overwrite	≤ 15 minutes	≤ 4 h	Versioning, replica bucket
Regional disaster (ap-south-1 unavailable)	≤ 24 h	≤ 8 h	Restore from ap-south-2 snapshot copies + replicated S3
Control plane loss	≤ 5 minutes	≤ 1 h	Same as app cell. App cells keep serving with cached tenant lookups (TTL 60 s, extended to 24 h in degraded mode)

The regional RPO of 24 h comes from daily snapshot copies. If a client contract needs better, the upgrade path is a cross-region read replica for that cell (cost roughly doubles the DB line), not a change of design.

3. Runbook: full-cell point-in-time restore

Use when a cell's database is logically damaged (bad migration, runaway job) and the damage affects many tenants. Declare a SEV1 first.

1. Stop writes. Put the cell in maintenance mode through the control plane (API returns 503 with a status-page link; ingest buffers to Redis streams; mobile offline queues hold punches). Scale workers to 0.

2. Find the target time. Use the audit log, migration log and Sentry to find the last good moment. Pick a time just before the first bad write.

3. Restore to a new instance:

   aws rds restore-db-instance-to-point-in-time \
     --source-db-instance-identifier prod-p1-pg \
     --target-db-instance-identifier prod-p1-pg-restore-20261031t0915 \
     --restore-time 2026-10-31T09:14:00Z \
     --db-subnet-group-name prod-p1-data \
     --vpc-security-group-ids sg-p1-db \
     --multi-az --region ap-south-1

4. Verify the restored instance: run infra/dr/verify.sql (row counts per tenant for key tables, latest audit_event time, latest pay_run states, RLS policies present, migration version matches the expected one).

5. Decide what is lost between the restore time and the incident: export audit events and outbox entries after the restore time from the damaged instance, so lost writes can be replayed or communicated per tenant.

6. Cut over: update the cell's db_dsn_ref secret to the restored endpoint (or rename instances), restart api and workers, run synthetic checks.

7. Replay safe lost writes (punches from ingest buffers and device re-sends are idempotent) and notify affected tenants about anything that must be redone (for example a leave approval).

8. Keep the damaged instance stopped (not deleted) for 14 days for forensics; then snapshot and delete.

9. Post-incident review within 5 business days.

4. Runbook: single-tenant restore

Use when one tenant needs data back (for example an admin bulk-deleted employees or a buggy import overwrote balances) and other tenants in the cell must not be affected.

1. Ticket and approval. Request from the tenant owner (or hr_admin), stating the entities and the time window. Confirm scope in writing; the platform admin approves.

2. Restore a scratch copy of the cell database to a point just before the damage, in the staging account network (isolated, no app access), using PITR as in section 3, step 3.

3. Extract the tenant's rows with the tenant mobility tool:

   spyrl-tenant-tool export \
     --dsn "$SCRATCH_DSN" --tenant-id 5b0e...c21 \
     --tables emp_employee,emp_job_assignment,lve_ledger_entry \
     --out s3://spyrl-prod-restores/t-5b0e/20261031/ --encrypt-with-tenant-key

4. Diff against current production for the same tenant: the tool produces "rows to re-insert", "rows changed since" and "conflicts" (rows modified after the restore point).

5. Review with the tenant: share the diff summary (counts and identifiers, not sensitive values). The tenant decides how to treat conflicts.

6. Import with spyrl-tenant-tool import --mode merge --conflicts <decision-file> inside a single transaction per entity group, with app.tenant_id set, so RLS limits the blast radius to this tenant.

7. Recompute derived state: leave balances from the ledger, attendance days for affected dates. Locked payroll runs are never modified; corrections go through an off-cycle or next-month adjustment.

8. Confirm with the tenant, record the audit event plt.tenant.restored, destroy the scratch database and the export files.

Target: scratch copy reviewable within 2 hours. Rehearse this runbook at least twice a year on staging.

5. Automated restore verification

A weekly job (Sunday 02:00 IST) per cell:

1. Restores the latest automated snapshot into the non-prod account (shared via a KMS grant).
2. Runs verify.sql and a sample decryption of envelope-encrypted fields using the multi-region key.
3. Records restore duration and data age as metrics (dr_restore_seconds, dr_snapshot_age_seconds).
4. Deletes the instance.

A failed or skipped run raises a SEV3 alert (observability).

6. DR drill procedure (quarterly)

Scope rotates: Q1 regional restore of a pooled cell, Q2 single-tenant restore, Q3 regional restore including the control plane, Q4 full game-day (region loss simulated in staging with the whole team).

Phase	Steps
Plan (1 week before)	Pick the scenario and the cell; book a 4-hour slot outside the payroll window; name an observer who records times
Execute	1. Declare the drill in the incident channel (DRILL prefix). 2. In spyrl-dr (ap-south-2), restore the latest copied snapshot of the cell. 3. Deploy the cell's app tier from Terraform (Stage B) or the Dokploy compose definitions on a fresh EC2 host (Stage A), with the image digest currently in production. 4. Point a test hostname at it. 5. Run the synthetic suite and verify.sql. 6. Decrypt sample sensitive fields. 7. Rebuild queues from the outbox.
Measure	Actual RTO (declare → synthetic checks green) and actual RPO (incident time − last data timestamp). Compare with section 2
Close	Tear down; write the drill report; file tickets for every manual step that should be automated; update this runbook

Pass criteria: RTO and RPO within target, no undocumented manual step, all data decryptable. Two consecutive failed drills escalate to the architecture review.

7. Incident severity levels

Severity	Definition	Examples	Response	Comms
SEV1	Data loss, data exposure across tenants, or a full outage of a production cell; payroll cannot be run or paid on payday	Cross-tenant data leak; cell down; corrupted payroll run being paid	Page on-call immediately, 24×7; incident commander within 15 min; all hands as needed	Status page within 30 min; affected tenants every 60 min; postmortem within 5 business days; regulator and fiduciary notification assessment (privacy)
SEV2	Major feature degraded for many tenants, or any degradation of payroll during the payroll window	Punch acceptance failing for one cell; payslip emails delayed over 2 h; one large tenant's payroll run failing	Page on-call 24×7; response within 30 min	Status page; affected tenants every 2 h; postmortem
SEV3	Minor feature degraded, workaround exists, or a single tenant affected outside payroll	Report export slow; one connector failing for a provider-side reason; restore-verification job failed	Business hours; response within 4 h	Ticket updates; status page optional
SEV4	Cosmetic or low-impact issue	UI glitch, typo, minor log noise	Next sprint	None

Roles in an incident: incident commander (decides and coordinates; from the platform_admin rota), operations lead (hands on keyboard), communications lead (support), scribe. On a small team one person may hold two roles, but never commander and operations lead for a SEV1 lasting over an hour.

Postmortems are blameless, use the template in the runbook repository, and list actions with owners and dates. SEV1 and SEV2 actions are tracked weekly until closed.