Observability & On-call

This page implements §13 of the architecture: OpenTelemetry for traces, metrics and logs; Grafana (Tempo, Mimir/Prometheus, Loki) for storage, dashboards and alerting; Sentry for errors; synthetic checks per cell.

1. Telemetry design

1.1 Standard attributes

Set once in the tenant-context middleware and job wrapper, so every span, log line and error carries them.

Attribute	Example	On traces	On logs	On metrics
service.name, service.version	spyrl-api, git SHA	Yes	Yes	Yes
deployment.environment	prod	Yes	Yes	Yes
cell.id	p1	Yes	Yes	Yes
process.role	api, worker, ingest	Yes	Yes	Yes
tenant.id	UUID	Yes	Yes	Only on a small set of metrics, for the top-N tenants (cardinality control)
tenant.plan	standard	Yes	Yes	Yes (low cardinality)
spyrl.module	pay	Yes	Yes	Yes
user.role_hash	short hash	Yes	No	No
job.queue, job.name	payroll, pay.run.calculate	Yes	Yes	Yes
request.id	ULID	Yes	Yes	No

User IDs appear in traces as opaque UUIDs only; names, emails and phone numbers never do.

1.2 Tenant-aware metrics without cardinality explosions

• RED metrics (rate, errors, duration) per route template and module, labelled by cell.id and tenant.plan, not by tenant.id.
• A per-tenant aggregation job in the collector (or a recording rule) emits tenant_requests_total, tenant_errors_total and tenant_p95_seconds for the top 50 tenants by traffic per cell, plus any tenant on a watch list (a new Enterprise tenant, a tenant with an open SEV).
• For per-tenant questions about the long tail, query logs and traces (which carry tenant.id), not metrics.
• Business metrics per tenant (payroll run status, punches per day, ingest lag) are computed in the database and pushed to the control plane health scores, not into Prometheus.

1.3 PII redaction

Defence in depth, because a single leaked log line with salary or Aadhaar data is a reportable incident.

Layer	Control
Code	Domain objects with sensitive fields implement toLog() that masks them; a lint rule flags logging of whole request bodies
Logger (pino)	redact paths for password, token, authorization, cookie, pan, aadhaar, account_number, ifsc, salary, ctc, gross, net, bank, face, otp, lat, lng (coordinates rounded to 2 decimals if logged at all)
HTTP instrumentation	Request and response bodies are not captured; query strings are stripped of known sensitive parameters
Collector	attributes and transform processors drop or hash any attribute matching sensitive key patterns; a regex scrubber for PAN-like and Aadhaar-like values in log bodies, as a last line
Sentry	beforeSend scrubbing with the same key list; "send default PII" off; breadcrumbs exclude form values; session replay disabled
Retention	Application logs 30 days, traces 14 days, metrics 13 months (aggregates), Sentry 90 days. Security and access logs (auth events, audit exports, ALB/CloudFront/WAF access logs, CloudTrail, VPC flow logs) are kept 1 year in an ap-south-1 S3 bucket: at least 180 days for the CERT-In direction and one year for the DPDP Rules, verify (security, privacy)
Residency	Pick an India-hosted region for the observability backend if the vendor offers one, or self-host the Grafana stack in ap-south-1; redaction makes residual personal data minimal either way. Verify the vendor region before go-live
Access	Grafana and Sentry via SSO, MFA; production logs readable by engineers on the on-call rota and the security lead only

A weekly job samples logs for PAN/Aadhaar/IFSC patterns; any hit is a SEV3 security ticket, and a SEV2 if real data is found.

2. SLOs, SLIs and error budgets

Targets are from architecture §13. Windows are rolling 28 days.

SLO	SLI (how it is measured)	Target Stage A	Target Stage B	Monthly error budget (Stage A / B)
Availability (web + API)	Share of non-5xx responses for user-facing routes, plus synthetic login checks, per cell	99.5%	99.9%	~3.6 h / ~43 min
API read latency	Share of GET requests under 300 ms at p95 per route class	p95 under 300 ms	same	5% of 5-minute windows may breach
API write latency	Share of mutating requests under 800 ms at p95	p95 under 800 ms	same	5% of windows
Punch acceptance	Time from punch request to 202 Accepted and from device push to stored punch	p95 under 2 s	same	1% of punches above 5 s
Payroll compute	Duration of pay.run.calculate normalised to 1,000 employees	under 5 min	same	2 slow runs per month per cell
Notification latency	Outbox insert → provider accepted	p95 under 60 s	same	1% above 5 min
Payslip availability	Payslips published within 1 h of run lock	99%	99.5%	—

Error-budget policy

• Budget above 50%: normal release train.
• Budget 25–50%: releases continue; the next sprint includes at least one reliability item from the top error sources.
• Budget under 25%: wave 1 and wave 2 rollouts pause except fixes; release manager and engineering lead review.
• Budget exhausted: feature freeze for the affected cell until the budget recovers or a review accepts the risk.
• The rollout orchestrator reads burn rates directly and refuses to advance a wave if the cell burns faster than 2× the budget rate during the soak (upgrades).

3. Alert catalogue

Alerts page only for user-visible impact or imminent data risk. Everything else goes to the ops channel or a ticket. Severity matches the incident levels in backup and DR.

Alert	Condition	Severity	Runbook
AvailabilityFastBurn	5xx/availability burn rate over 14× budget for 5 min and over 14× for 1 h (multi-window)	SEV1/SEV2 page	rb/availability
AvailabilitySlowBurn	Burn rate over 3× for 6 h	SEV3 ticket	rb/availability
SyntheticLoginFailing	Login or punch synthetic fails 3 consecutive runs in a cell	SEV2 page	rb/synthetics
LatencyP95High	p95 over 2× SLO for 15 min on read or write class	SEV3, SEV2 if during payroll window	rb/latency
DBCPUHigh	RDS CPU over 85% for 15 min	SEV2 page	rb/db-cpu
DBConnectionsSaturated	PgBouncer waiting clients over 50 for 5 min, or RDS connections over 90% of max	SEV2 page	rb/db-connections
DBStorageLow	Free storage under 15% or projected full within 7 days	SEV3; SEV2 under 5%	rb/db-storage
DBReplicaLag	Read-replica lag over 60 s for 10 min	SEV3	rb/replica-lag
RLSDenied	rls_denied_total over 0 for a write path, or unexpected spike on reads	SEV2 page (possible isolation bug)	rb/rls-denied
CrossTenantGuard	App guard detects a payload tenant_id different from the context	SEV1 page	rb/cross-tenant
QueueBacklogOld	Oldest job age over 10 min in notifications, or over 30 min in payroll, attendance	SEV2 page in payroll window, else SEV3	rb/queues
DLQGrowing	Any DLQ over 100 jobs or growing for 30 min	SEV3	rb/dlq
PayrollRunFailed	pay.run.calculate failed after retries for any tenant	SEV2 page (payroll window), else SEV3	rb/payroll-run
IngestLag	Device or GPS ingest lag over 5 min	SEV2	rb/ingest
EmailBounceRateHigh	SES bounce rate over 4% or complaint rate over 0.08% (account-level risk)	SEV2	rb/ses-reputation
SmsProviderErrors	SMS/WhatsApp provider error rate over 10% for 10 min	SEV3, SEV2 if OTP	rb/sms-failover
FaceProviderDegraded	Face verify timeouts over 5% for 10 min	SEV3 (punches fall back to unverified)	rb/face-provider
CertificateExpiring	Any TLS certificate under 14 days to expiry	SEV3; SEV2 under 3 days	rb/certificates
BackupMissing	No successful snapshot or cross-region copy in 26 h	SEV2	rb/backups
RestoreVerifyFailed	Weekly automated restore test failed	SEV3	rb/restore-verify
ErrorSpikeNewRelease	New Sentry issue with over 50 events in 30 min after a deploy	SEV3; halts the rollout wave	rb/release-halt
ControlPlaneDown	Control-plane API synthetic fails 3 runs	SEV2 (cells continue on cached lookups)	rb/control-plane
WAFBlockSpike	WAF blocks over 10× baseline for 15 min	SEV3, security review	rb/waf
AuthFailureSpike	Failed logins over 10× baseline for a tenant or IP range	SEV3, SEV2 if credential stuffing confirmed	rb/auth-abuse
NodeDiskOrMemory (Stage A)	EC2 disk over 85% or memory over 90%	SEV3	rb/stage-a-node

Runbooks live in docs/runbooks/ in the main repository, one file per rb/ key, each with: symptoms, dashboards to open, likely causes, safe mitigations (for example a kill switch), escalation, and how to verify recovery. An alert without a runbook cannot be merged.

4. Dashboards

Dashboard	Audience	Panels
Platform overview	On-call, leadership	Availability and burn per cell, active incidents, release wave status, top errors
Cell health	On-call	RED by module, DB CPU/IOPS/connections, PgBouncer, Redis, queue depth and age per queue, ingest lag, node or task health
Payroll window	Payroll SME, on-call (25th–5th)	Runs in progress and failed by tenant, compute duration, payslip publishing, notification backlog, bank and statutory file generation errors
Tenant drill-down	Support, customer success	For one tenant: request rate, errors, slow routes, failed jobs, connector health, last logins (from logs and traces filtered by tenant.id)
Integrations	On-call	Provider latency and error rates (SES, MSG91, Expo, Rekognition, maps), connector failures by key
Security	Security lead	Auth failures, MFA challenges, WAF blocks, RLS denials, support sessions active, privileged actions
Mobile	Mobile team	Crash-free sessions per version, app versions in use vs the minimum, offline queue depth reported on sync, OTA adoption
Capacity	Engineering lead	Section 5 trigger metrics with distance to threshold

5. Capacity triggers

Reviewed monthly after the payroll window, and alerting at 80% of each threshold.

Trigger	Threshold (architecture §13 and §2.1)	Action
DB CPU at monthly peak	Over 60% sustained	Scale the instance class; if already large, plan a new pooled cell
DB size	Over 500 GB	New pooled cell for new tenants; archive old GPS partitions
Tenants per cell	Over 300	New pooled cell
Active employees per cell	Over 30k	New pooled cell
Neighbour impact	A tenant's p95 regression attributable to another tenant	Tighten that tenant's quotas; move the heavy tenant to its own or a lighter cell
GPS ingest	Over ~200 writes/s sustained	Extract ingest as a separate service (architecture §2.1)
Payroll compute	Largest tenant's run over 10 minutes	Extract the payroll-compute worker; parallelise by employee batches
Stage A host toil	Over ~4 h/week of host operations, or more than 50 tenants / 10k employees	Start the Stage A → B migration (deployment)

6. On-call process

Item	Policy
Rota	Weekly primary and secondary from engineers who have completed on-call onboarding (shadow one week, reverse-shadow one week). Hand-off Monday 11:00 IST with a written summary
Hours	24×7 paging for SEV1/SEV2. SEV3 handled in business hours (09:00–19:00 IST, Monday–Saturday, matching Indian SME working days)
Response	Acknowledge a page within 15 min (SEV1) or 30 min (SEV2). If primary does not acknowledge in 15 min, secondary is paged; after 30 min, the engineering lead
Payroll window	25th–5th: secondary is also on heightened alert during 09:00–21:00 IST, and a payroll SME is reachable
Tools	Grafana alerting → a paging tool (for example Grafana OnCall or PagerDuty) → phone and app; incident channel per incident; status page
Load	Target under 2 pages per week outside business hours. More than that for 2 weeks triggers an alert-quality review
Compensation	On-call allowance plus time off in lieu after night pages (team policy, set by leadership)
Access	On-call has production read access, the kill switches, and rollout pause. Data changes in production need a second person (four-eyes) except during a declared SEV1
After the shift	Review every page: was it actionable, was the runbook good enough? Fix or delete noisy alerts