Deployment Architecture

This page turns §10, §11 and §17 of the architecture into an operating design. The platform comparison (Dokploy vs ECS vs EKS vs PaaS) and the Stage A/Stage B triggers are decided there and are not repeated here.

Fixed points (from the architecture)

• Everything runs in AWS ap-south-1 (Mumbai). DR copies go to ap-south-2 (Hyderabad).
• One API image runs as PROCESS_ROLE=api|worker|ingest. The control plane, web SPA and admin console are separate artefacts.
• The database tier is managed from day one (RDS PostgreSQL Multi-AZ with PITR, ElastiCache, S3) in every production stage. Only the application tier changes between Stage A and Stage B.

1. Environments

Environment	Purpose	Hosting	Database	Data	Deployed from	Lifetime
local	Developer machine	docker compose	Postgres 16 container	Seed script: 2 synthetic tenants	Working copy	—
preview	One per pull request; product review	Dokploy (non-prod host), Traefik host pr-<number>.preview.spyrl.app	Postgres container per preview	Seeded demo tenant	PR branch	Destroyed on PR close or after 5 days idle
dev	Integration of main, internal testing	Dokploy (non-prod host)	Postgres container	Synthetic, reset weekly	main, automatic	Long-lived
uat	Pilot clients and QA sign-off with realistic configuration	Dokploy (non-prod host)	Small RDS (single-AZ, PITR 7 days) so restore runbooks can be rehearsed	Synthetic plus client configuration (never real employee data)	Release candidate tag	Long-lived
staging	Production-like rehearsal: migrations, e2e, k6 smoke	Stage A: separate Dokploy host in the prod VPC layout; Stage B: a cell module instance in the non-prod account	RDS (single-AZ)	Synthetic at production volume	Release candidate tag	Long-lived
prod cells	Customer traffic	Stage A: Dokploy on 2× EC2; Stage B: ECS Fargate per cell	RDS Multi-AZ per cell	Customer data	Promoted image digest, by wave	Long-lived

Naming. Cells are p1, p2 (pooled) and d-<slug> (dedicated). Every AWS resource carries the tags env, cell, owner, cost-centre and, for dedicated cells, tenant.

AWS accounts. Stage A uses three accounts under AWS Organizations: spyrl-nonprod, spyrl-prod and spyrl-dr (backup vault copies in ap-south-2, separate credentials). Stage B adds spyrl-shared (ECR, CI OIDC role, DNS) and optionally one account per model-D dedicated cell. CloudTrail from all accounts goes to the spyrl-dr account's log bucket with Object Lock.

1.1 Local docker-compose

# infra/compose/docker-compose.yml (outline)
services:
  postgres:   # postgres:16, with init script creating app_owner (migrations) and app_user (NOBYPASSRLS)
  redis:      # redis:7
  minio:      # S3-compatible storage, bucket spyrl-local
  mailpit:    # SMTP sink with web UI on :8025
  api:        # PROCESS_ROLE=api, hot reload
  worker:     # PROCESS_ROLE=worker
  control-plane:
  web:        # vite dev server, proxies /api to api
  admin:
  otel-collector:  # optional profile "obs": collector + Grafana LGTM all-in-one image

npm run dev:up starts the stack, runs migrations as app_owner, then seeds two tenants (acme, globex) so tenant-isolation behaviour is visible from the first day.

2. Stage A: Dokploy on EC2 + managed data tier

2.1 Dokploy setup specifics

Item	Setting
Dokploy server	Separate small EC2 instance (2 vCPU, 4 GB) in a management subnet. No public IP; admins reach the UI through SSM port forwarding or the team VPN. It is not a workload node, so a Dokploy problem never takes down production
Workload nodes	2× Graviton EC2 (for example m7g.large, 2 vCPU, 8 GB) in two AZs, joined as a Docker Swarm (both managers plus a third lightweight manager on the Dokploy host for quorum)
Images	Pulled from GHCR or ECR by digest, never by mutable tag. Dokploy deploys are triggered by the CI pipeline through the Dokploy API, not by "deploy on push"
Services	api (2 replicas, one per node), worker (2 replicas), ingest (1–2 replicas, R2+), control-plane (2 replicas), otel-collector (global mode)
Ingress	ALB → Traefik on each node (port 80 inside the VPC; TLS terminates at the ALB). Traefik routes by host and path. Health check GET /healthz (liveness) and /readyz (DB and Redis reachable)
Secrets	Fetched at container start from AWS Secrets Manager using the EC2 instance role (a small entrypoint wrapper). Dokploy environment variables hold only non-secret settings
Logs	Containers log JSON to stdout; the OTel collector ships logs, metrics and traces to Grafana Cloud
OS patching	SSM Patch Manager monthly (outside the payroll window), one node at a time with Swarm drain
Backups of Dokploy itself	Dokploy configuration is exported nightly to S3. Everything Dokploy deploys is also described in infra/dokploy/*.yml, so the server can be rebuilt from git

Non-prod Dokploy host. One larger EC2 instance (for example m7g.xlarge) runs dev, uat (app tier), and previews, with Traefik and a wildcard certificate for *.preview.spyrl.app via DNS challenge. It lives in the spyrl-nonprod account and never has network access to production.

3. Stage B: ECS Fargate + Terraform

Network notes.

• One VPC per environment with three AZs; per-cell security groups so a cell's tasks can only reach that cell's RDS and Redis.
• Interface and gateway VPC endpoints for S3, ECR, Secrets Manager, KMS and CloudWatch keep image pulls and file traffic off the NAT gateway (NAT data charges are a common surprise cost).
• Model D dedicated cells (own app tier, optionally own account) get their own VPC, peered or connected through Transit Gateway only to the shared services they need (control-plane API, observability).
• Connection pooling: PgBouncer in transaction mode per pooled cell. The tenant context is set with SELECT set_config('app.tenant_id', $1, true) inside each transaction, which is transaction-local and safe with transaction pooling (system design).

3.1 Terraform cell module

module "cell_p1" {
  source              = "../modules/cell"
  cell_id             = "p1"
  cell_type           = "pooled"            # pooled | dedicated
  isolation_model     = "shared_app"        # shared_app (model C) | full_stack (model D)
  environment         = "prod"
  vpc_id              = module.network.vpc_id
  private_subnet_ids  = module.network.private_subnet_ids
  alb_listener_arn    = module.edge.https_listener_arn
  host_rules          = ["*.spyrl.app"]     # dedicated: ["acme.spyrl.app", "hr.acme.com"]
  image_digest        = var.api_image_digest
  api_task            = { cpu = 1024, memory = 2048, min = 2, max = 8 }
  worker_task         = { cpu = 2048, memory = 4096, min = 1, max = 10 }
  ingest_task         = { cpu = 512,  memory = 1024, min = 1, max = 4, enabled = true }
  db_instance_class   = "db.r7g.large"
  db_multi_az         = true
  db_read_replica     = true
  db_backup_retention = 35
  db_kms_key_arn      = null                # null = cell key created by module; dedicated may pass a tenant CMK
  redis_node_type     = "cache.t4g.medium"
  s3_bucket           = "spyrl-prod-files"
  s3_prefix           = "cells/p1/"
  rollout_wave        = 1
  alarm_sns_topic_arn = module.alerting.sns_topic_arn
  tags                = { cell = "p1", env = "prod" }
}

Inputs (required)	Description
cell_id, cell_type, isolation_model, environment	Identity and isolation model
vpc_id, private_subnet_ids, alb_listener_arn	Network attachment (ignored for full_stack, which creates its own VPC)
image_digest	The API image digest to run in all three roles
api_task, worker_task, ingest_task	Sizing and autoscaling bounds
db_*, redis_node_type	Data-tier sizing and protection
s3_bucket, s3_prefix	File storage location (tenant keys are prefixed further by t/<tenant_id>/)
rollout_wave, alarm_sns_topic_arn, tags	Operations

Outputs	Used by
cell_id, api_url, ingest_url	Control-plane cell registry
db_endpoint, db_reader_endpoint, db_secret_arn	Control plane (db_dsn_ref), migration runner
redis_endpoint	App configuration
ecs_cluster_arn, service_arns (api/worker/ingest)	Rollout orchestrator (deploys, scaling)
target_group_arns	Edge routing
kms_key_arn	Envelope encryption configuration
security_group_ids	Network policies
dashboard_url, alarm_arns	Observability

The module also creates the standard alarm set for the cell (observability), the per-cell IAM task roles (least privilege: its own S3 prefix, its own secrets, its KMS key), and the CloudWatch log groups (14-day retention; logs are forwarded to Grafana).

4. Container topology

Artefact	Image / location	Runs as	Scales on	Notes
API	ghcr.io/spyrl/api@sha256:... (distroless Node 22)	api	CPU, p95 latency	Stateless; sticky sessions not needed
Worker	same image	worker	BullMQ queue depth and oldest-job age	Queues: payroll, attendance, notifications, reports, integrations, imports. Per-tenant group concurrency
Ingest	same image	ingest	Requests per second	Biometric push gateway and (R3) GPS points; writes to Redis streams, drained by workers
Migrator	same image, PROCESS_ROLE=migrate	One-off task per cell	—	Runs Prisma migrations as app_owner, then exits
Control plane	ghcr.io/spyrl/control-plane@sha256:...	api + scheduler	CPU	One per environment, not per cell
Web SPA	Static build in S3	—	CloudFront	Versioned path web/<git-sha>/; index.html short cache
Admin console	Static build in S3, separate distribution admin.spyrl.app	—	—	Access only through the company IdP plus WAF IP allow-list
PgBouncer	edoburu/pgbouncer or similar, pinned digest	Per pooled cell	Fixed	Transaction pooling
OTel collector	otel/opentelemetry-collector-contrib	Sidecar (Fargate) or global service (Swarm)	—	PII redaction processors live here too

5. CI/CD pipeline

Trunk-based development; every merge to main produces a signed image that can be promoted unchanged through every environment.

# .github/workflows/ci-cd.yml (outline)
name: ci-cd
on:
  pull_request:
  push:
    branches: [main]
  workflow_dispatch:
    inputs:
      promote_digest: { description: "Image digest to promote", required: false }
      wave: { description: "0, 1 or 2", required: false }

permissions: { contents: read, id-token: write, packages: write }

jobs:
  verify:                       # every PR and main
    steps:
      - checkout, setup-node 22, npm ci (cache)
      - lint + typecheck + dependency-cruiser module boundaries
      - unit tests (vitest) incl. packages/payroll-engine golden files
      - property tests (fast-check) for policy evaluators
      - contract tests: zod ↔ OpenAPI diff, breaking-change check vs main
      - integration tests with Postgres service container: RLS + tenant-isolation suite, RLS coverage check
      - SAST (CodeQL or Semgrep), secret scan (gitleaks), npm audit / OSV
  build:
    needs: verify
    steps:
      - docker buildx (arm64) api, control-plane; vite build web, admin
      - Trivy scan (fail on HIGH/CRITICAL with a fix available)
      - SBOM (syft) attach; cosign sign with GitHub OIDC (keyless)
      - push to GHCR/ECR by digest
  preview:
    if: github.event_name == 'pull_request'
    needs: build
    steps:
      - Dokploy API: create/update app pr-<number>, seed demo tenant
      - Playwright smoke against preview URL; comment URL on the PR
  staging:
    if: github.ref == 'refs/heads/main'
    needs: build
    environment: staging
    steps:
      - control-plane rollout API: migrate staging cell (expand only)
      - deploy digest to staging
      - Playwright e2e (web), Maestro smoke (mobile, against staging API)
      - k6 smoke (5 min), ZAP baseline scan
      - mark digest as "release candidate" in the control plane
  release:
    if: github.event_name == 'workflow_dispatch'
    environment: production       # requires release_manager approval
    steps:
      - verify cosign signature and that digest passed staging
      - check payroll-window freeze policy (blocks non-patch releases 25th–5th)
      - control-plane rollout API: start wave N with digest
      - poll rollout status; fail the job if the orchestrator halts the wave

GitHub Actions authenticates to AWS with OIDC (no long-lived keys). The production environment requires approval from a release_manager, and only the control-plane orchestrator, not CI, touches production cells directly.

6. Wave rollouts

Wave	Cells	Entry condition	Soak	Exit gate
0	p0-canary: internal tenants, the demo tenant, and opted-in canary tenants (early in Stage A this is a set of canary tenants in p1 routed to a canary task set)	Staging green	24 h	SLOs within budget; no new Sentry issue above 10 events; no migration errors
1	All pooled cells, in order of size (smallest first)	Wave 0 exit	48 h	Same, per cell
2	Dedicated cells, each in its client maintenance window	Wave 1 exit	—	Client smoke test, notification sent

The orchestrator in the control plane runs, for each cell: pre-checks (backups fresh, no payroll run in progress, cell healthy) → migrate (expand step) → deploy (rolling for Swarm; ECS rolling with circuit breaker, or blue/green for risky releases) → verify (synthetic login, punch and leave flows; error rate) → record the cell version. Any failure halts the wave and pages the release manager. Detail: upgrades and migrations.

7. Feature flags

• Flags and entitlements live in the control plane (OpenFeature-compatible SDK, cached in the app for 60 s, invalidated by an event).
• Naming: <module>.<feature> for release flags (for example pay.new_run_ui), kill.<module>.<feature> for kill switches, integration.<connector>.enabled for connectors.
• Targeting: by tenant, cohort, plan, cell, or percentage of tenants (never a percentage of users inside a tenant for payroll features, because two payroll managers must see the same behaviour).
• Every flag has an owner and an expiry date. Release flags older than 90 days after full rollout raise a tech-debt ticket (maintenance).
• Flag changes are audited and shown to tenants on the change log when they alter behaviour.

8. Mobile release (Expo EAS)

Step	Tooling	Notes
Build	EAS Build profiles development, preview (internal distribution, points at UAT), production	Credentials (keystore, APNs) managed by EAS
Store release	EAS Submit to Play internal → closed testing → production (staged rollout 10% → 50% → 100%); TestFlight → App Store with phased release	Play listing data-safety form must declare location and photo collection accurately (the reference product's listings contradict themselves, see research §21)
OTA updates	EAS Update channels preview, production, with runtime versions tied to the native fingerprint	Only JS/asset fixes go OTA; anything touching native modules, permissions or the punch queue format needs a store build
Minimum-version gate	The app sends X-App-Version and X-App-Build. The API compares them with mobile.min_supported (per platform, set in the control plane). Below the minimum it answers 426 Upgrade Required with the store link; between minimum and "recommended" it returns a soft-upgrade header	Offline punches queued by an old version are still accepted if their payload schema version is supported
Rollback	Republish the previous update group to the channel (OTA), or halt the staged store rollout	Store builds cannot be rolled back, so keep the previous build's API compatibility for at least 90 days

9. Monthly cost estimates

All figures are rough estimates in INR at approximately ₹85 per USD, using on-demand ap-south-1 list prices as understood in late 2026, before taxes. They exclude people, support tooling, and third-party per-message or per-check fees (SMS, WhatsApp, face, maps), which are passed through as metered add-ons. Re-price with the AWS calculator before committing to a budget.

9.1 Stage A (launch, one pooled cell, up to roughly 50 tenants / 10k employees)

Item	Sizing	₹ / month (estimate)
EC2 workload nodes	2× m7g.large	10,000
Dokploy management server	1× t4g.medium	2,000
RDS PostgreSQL (app cell)	db.t4g.large Multi-AZ, 100 GB gp3	17,000
RDS control plane	db.t4g.small Multi-AZ	4,000
Backups + cross-region snapshot copies	35-day PITR, daily copy to ap-south-2	2,000
ElastiCache Redis	cache.t4g.small, 1 primary + 1 replica	4,000
ALB + NAT gateway	1 ALB, 1 NAT	6,500
S3 + CloudFront + WAF	Under 200 GB, modest traffic	3,500
Secrets Manager, KMS, CloudWatch, SSM	—	1,500
Infrastructure subtotal		≈ 50,000 on-demand; ≈ 38,000–42,000 with 1-year reservations / savings plans
Observability and error tracking	Grafana Cloud (paid tier) + Sentry Team	5,000–10,000
Non-production (dev, uat, previews, staging)	1× m7g.xlarge Dokploy host, small UAT and staging RDS	15,000–20,000
Mobile and store accounts	EAS paid plan, Apple and Google developer fees amortised	2,000–17,000 depending on EAS tier

The production infrastructure line is consistent with the ₹25–45k range in the architecture once reservations are applied; it is not consistent with on-demand pricing, so buy the reservations at go-live.

9.2 Stage B (growth, ECS Fargate)

Item	Sizing (one pooled cell, ~150 tenants / 25k employees)	₹ / month (estimate)
Fargate api	3 tasks × 1 vCPU / 2 GB (Graviton)	7,500
Fargate worker	2–4 tasks × 2 vCPU / 4 GB, scaling at month end	10,000–18,000
Fargate ingest	2 tasks × 0.5 vCPU / 1 GB	2,500
RDS	db.r7g.large Multi-AZ + 1 read replica	40,000
ElastiCache	2× cache.t4g.medium	6,000
PgBouncer, ALB share, NAT (2 AZ), VPC endpoints	—	12,000
S3, CloudFront, WAF, backups, DR copies	—	6,000
Per pooled cell		≈ 85,000–95,000
Shared: control plane (Fargate + RDS), ECR, CloudTrail, GuardDuty, Security Hub	Per environment	15,000–20,000
Observability (higher volume)	—	15,000–30,000

Dedicated cell type	Incremental cost (estimate)	Implication
Model C: own RDS on the shared app tier	₹20,000–25,000 / month	Sets the minimum Enterprise dedicated-cell surcharge
Model D: own app tier and VPC	₹50,000–70,000 / month	Only for contracts that demand it; price accordingly

10. Stage A → Stage B migration runbook

Because the data tier is already managed, this is an application-tier cutover, not a data migration. RDS, ElastiCache and S3 stay where they are and are imported into Terraform state.

Preconditions

• Trigger met (architecture §10.2) and the decision recorded in the decision log.
• Not inside the payroll window (25th to 5th) and no tenant payroll run in progress.
• Terraform cell module tested in staging, including a full rollout through the orchestrator.
• Latest DR drill passed in the last 90 days.

Steps

1. Import the existing RDS, ElastiCache, S3 bucket, ALB and security groups into Terraform state (terraform import); run plan until it shows no changes to data resources. Add prevent_destroy on data resources.
2. Create the ECS cluster and the p1 services through the cell module, pointing at the existing RDS and Redis, with worker desired count = 0.
3. Register the ECS api target group on the existing ALB listener with weight 0 alongside the Dokploy/Traefik target group.
4. Verify the ECS api with a header-based ALB rule used only by synthetic checks (X-Spyrl-Canary: ecs): login, punch, leave apply, payslip view, a report export.
5. Shift traffic with weighted target groups: 10% for 2 hours → 50% for 12 hours → 100%. Watch error rate and p95 per target group.
6. Move workers: scale ECS workers to the target count, then scale Swarm workers to 0. BullMQ jobs are idempotent and leases expire, so any job interrupted mid-flight is retried by the ECS workers. Move cron-like repeatable jobs last and confirm each fires once.
7. Move ingest (if enabled) by updating the ingest.spyrl.app rule. Devices retry and buffer, so a brief gap is safe.
8. Keep Dokploy nodes warm (scaled to 1 api replica, out of the ALB) for 7 days as the rollback path.
9. Decommission the EC2 workload nodes after 7 clean days. Keep Dokploy for dev, uat and previews.

Rollback (any step up to 9): set the ALB weights back to the Dokploy target group and scale Swarm workers up, ECS workers to 0. No data rollback is involved.

Expected customer impact: none planned; announce a maintenance window anyway (in-app banner 7 days ahead) in case rollback is needed.