Upgrades & Migrations

This page details §11 of the architecture (ADR-010). One release train serves every cell, pooled and dedicated; dedicated cells simply sit in a later wave.

1. Release train

Day (IST)	Activity
Continuous	Trunk-based merges to main; every merge deploys to dev and staging automatically
Tuesday 12:00	Release cut: the latest green staging digest becomes the release candidate rYYYY.WW (for example r2026.44). Release notes drafted from PR labels
Tuesday–Wednesday	UAT smoke by QA; payroll-labelled changes get a payroll SME review
Wednesday 21:00	Wave 0 (canary cell and canary tenants)
Thursday 21:00	Wave 0 exit gate (24 h) → wave 1 (pooled cells, smallest first)
Saturday 21:00	Wave 1 exit gate (48 h) → wave 2 (dedicated cells, each in its maintenance window over the following week)
Monday	Release retrospective if any wave was halted

Deploys happen at 21:00 IST because Indian SME traffic is lowest in the evening, while the team is still awake; night shifts still punch, so deploys must be zero-downtime regardless.

Change classes

Class	Examples	Path
Standard	Features behind flags, UI changes, additive API changes	Weekly train
Payroll-sensitive	Anything under packages/payroll-engine, pay module, statutory generators, attendance day computer, leave accrual	Weekly train outside the payroll window only; golden tests mandatory; payroll SME approval
Patch	Bug fix with a small diff, including in the payroll window	Patch release rYYYY.WW.N through all waves with shortened soak (wave 0: 2 h, wave 1: 4 h) and a second reviewer
Emergency	Security fix or SEV1 fix	Release manager may skip soaks; postmortem explains why
Configuration	Statutory packs, flags, entitlements	Not code; published through the control plane (sections 8 and 9)

2. Expand/contract migration playbook

Rule: at any moment, the running code version N and the previous version N−1 must both work against the current schema. No destructive change ships in the same release as the code that stops using the old structure.

Phase	Release	Schema	Code
1. Expand	N	Add new columns or tables (nullable or with defaults), new indexes CONCURRENTLY	Old code ignores them
2. Dual write	N (or N+1)	—	Write both old and new; read old
3. Backfill	Background job after N is fully rolled out	—	Batched job, idempotent, per tenant, throttled; progress visible in the admin console
4. Switch reads	N+1	—	Read new (behind a flag per cell); still write both
5. Contract	N+2 or later, after all cells run N+1 for at least one full week	Drop old columns, add NOT NULL constraints (validated with NOT VALID then VALIDATE CONSTRAINT)	Stop writing old

Migration safety checklist (CI enforces the first four)

• No DROP, RENAME or type change of a column still read by the previous release (a migration linter compares with N−1's Prisma schema).
• Indexes on large tables are created CONCURRENTLY (in a separate non-transactional migration step).
• New tenant-owned tables have tenant_id, an RLS policy, FORCE ROW LEVEL SECURITY, and composite foreign keys including tenant_id (the RLS coverage test fails otherwise).
• lock_timeout set (for example 5 s) so a migration fails fast instead of blocking traffic.
• Backfills run as jobs, never inside the migration.
• Rollback note: what happens if this release is rolled back after the migration ran (answer must be "nothing breaks").

Worked example: encrypting PAN at the application level

Today emp_employee.pan is plain text (protected by RDS encryption at rest only). Target: pan_enc (envelope-encrypted bytes), pan_last4 (for masked display and search) and pan_hash (keyed HMAC for duplicate detection). See security.

Release N: expand + dual write

-- migration 2026_11_04_01_pan_encryption_expand.sql
ALTER TABLE emp_employee
  ADD COLUMN pan_enc   bytea,
  ADD COLUMN pan_last4 char(4),
  ADD COLUMN pan_hash  bytea;

-- migration 2026_11_04_02_pan_hash_index.sql  (non-transactional)
CREATE INDEX CONCURRENTLY IF NOT EXISTS emp_employee_tenant_pan_hash_idx
  ON emp_employee (tenant_id, pan_hash);

Code in N writes pan, pan_enc, pan_last4 and pan_hash on every create or update; reads still use pan.

Backfill job (after N is on all cells)

// runs per tenant with tenant context set; 500 rows per batch; resumable by cursor
for await (const batch of employeesWithoutPanEnc({ tenantId, batchSize: 500 })) {
  const dek = await keyring.dataKey(tenantId);
  await db.$transaction(batch.map((e) => db.emp_employee.update({
    where: { tenant_id_id: { tenant_id: tenantId, id: e.id } },
    data: { pan_enc: encrypt(dek, e.pan), pan_last4: e.pan.slice(-4), pan_hash: hmac(tenantId, e.pan) },
  })));
}

Release N+1: switch reads. Reads use pan_enc (decrypt only when the caller holds emp.sensitive.read), lists use pan_last4. A verification job confirms every row with pan has a matching pan_enc (decrypt and compare a sample of 1% per tenant, all rows for small tenants). Writes still populate pan in case N+1 is rolled back.

Release N+2 (one week after N+1 is everywhere): contract

-- migration 2026_11_25_01_pan_encryption_contract.sql
UPDATE emp_employee SET pan = NULL WHERE pan_enc IS NOT NULL;   -- batched by job in practice
ALTER TABLE emp_employee DROP COLUMN pan;

Because PAN plaintext also exists in older backups, the contract release notes record the date after which backups no longer contain plaintext PAN (35 days later, after PITR and snapshot age-out).

3. Payroll-window freeze

Rule	Detail
Window	25th of the month 00:00 IST to the 5th of the next month 23:59 IST (architecture §11)
Frozen	Payroll-sensitive change class (section 1); statutory generator code; migrations touching pay_*, att_* day tables or lve_* ledgers
Allowed	Standard changes behind flags that are off for payroll paths; patches with payroll SME + release manager approval; statutory pack publishing (it is data with an effective date)
Tenant-specific windows	Tenants with a non-standard payroll calendar (for example payroll on the 15th) set their window in the control plane; the orchestrator respects the union for their cell's payroll-sensitive changes
Enforcement	The release job checks the change class labels; the orchestrator refuses payroll-sensitive digests in the window unless an override with two approvals is recorded
Also frozen	Infrastructure changes with downtime risk (instance class changes, major version upgrades, Stage A → B cutover), OS patching, DR drills

4. Per-cell rollout orchestration

The control plane's rollout orchestrator is the only component that deploys to production cells.

• Pre-checks: latest snapshot under 26 h old; no tenant payroll run in calculating state; cell error budget above 25%; maintenance window open (dedicated cells); no active SEV1/SEV2 in the cell.
• Concurrency: one cell at a time in wave 1 until there are more than 5 pooled cells, then up to 2 in parallel. A halt in any cell halts the wave.
• Payroll jobs during deploy: workers drain gracefully (finish current job, up to 10 minutes, then release the lease). Payroll calculation is deterministic and idempotent, so a retried job produces the same result.
• Record: the cell's version, digest and migration version are written to the cell registry; tenants see "Updated to r2026.44" with release notes in-app.

5. Rollback

Situation	Action	Data impact
Bad code, migration was expand-only	Redeploy previous digest through the orchestrator (rollback command), cell by cell	None
Bad backfill	Stop the job (flag), fix, re-run (idempotent); the old column is still authoritative until N+1	None
Bad contract migration	Should not happen if the checklist was followed. Restore the dropped data from PITR into a scratch DB and re-populate (backup and DR)	Possible; SEV1
Catastrophic data corruption in a cell	Full-cell PITR (backup and DR runbook)	Writes after the restore point must be replayed
Bad mobile OTA update	Republish the previous update group to the channel	None
Bad mobile store build	Halt staged rollout; ship a fix; raise the minimum version only after the fix is live	None
Bad statutory pack	Publish a corrected pack version with the same effective date; affected unlocked runs recalculate; locked runs get an adjustment in the next run	Communicated to payroll managers

Rule: we never roll back across a contract migration. Roll forward instead.

6. Mobile versioning

Item	Policy
Version scheme	major.minor.patch (store version) + build number; EAS runtime version from the native fingerprint
Support window	The API supports every app version released in the last 90 days and at least the last two minor versions
Minimum version	mobile.min_supported per platform in the control plane. Raised only after the replacement has been in stores for 14 days and adoption is over 90% (from telemetry)
Gate behaviour	Below minimum: 426 Upgrade Required with a store link, but offline punches already queued are still accepted if their payload schema version is supported. Between minimum and recommended: soft banner
OTA	JS-only fixes; never changes to permissions, background location, native modules, or the offline queue schema
Kiosk devices	Kiosk tablets are often unattended; MDM or Play managed updates are recommended. Kiosk builds get a 30-day longer support window

7. API deprecation

• The public API /v1 changes additively only: new endpoints, new optional fields, new enum values (clients must tolerate unknown values, as stated in the API overview).
• A breaking change means /v2 for that resource; /v1 keeps working for at least 6 months after /v2 is generally available (12 months for Enterprise contracts that require it).
• Deprecated endpoints return Deprecation and Sunset headers and a Link to the migration guide. The developer portal and a monthly email to API client owners list deprecated endpoints still in use by that tenant (from API logs).
• Webhook payload versions follow the same rule; a subscription pins a payload version.
• The internal web API ships with the SPA in the same deploy and has no external compatibility promise, but must still tolerate the previous SPA build for one release (users with an open tab).

8. Statutory-pack updates

Statutory changes (for example a new ESI wage ceiling, a state PT slab revision, a budget change to TDS slabs) are published as pack versions, not code releases. The publishing workflow is in administration.

Step	Detail
1. Monitor	The payroll SME tracks EPFO, ESIC and CBDT circulars, state gazettes and the Union Budget; customers can report changes
2. Draft	New pack version with effective date; values marked "verify" until reviewed
3. Golden tests	Add or update golden files for the change (for example an employee at the new ceiling) (testing strategy)
4. Staging	Recalculate sample tenants' last run with the new pack; compare outputs
5. Publish	Effective date set; tenants on "inherit" pick it up automatically from that date; tenants on "review" see a task
6. Retroactive changes	If the effective date is in the past (common with government notifications), the next payroll run computes arrears or adjustments; locked runs are never changed
7. Communicate	In-app announcement to payroll managers and the change log

When a statutory change needs code (a new kind of rule, not new values, for example a new file format), it is a payroll-sensitive release, planned outside the payroll window, and the pack schema version is bumped so old runs keep using old evaluators.