Testing
This page explains how the product is tested: which layers exist, what each one runs against and what it proves. It is part of section 9 of the arc42 template.
The problem
Section titled “The problem”The rules that matter most in Ochema are not visible in a screen. A company must never read the data of another; a message must never go out without consent; a write must never happen without an audit row. A test that replaces the database with a stand-in cannot prove any of them, because the rule lives in the database. And the product depends on providers of SMS, email and calendars that a test run cannot call.
The rule
Section titled “The rule”- A rule the database enforces is tested against a real PostgreSQL, connected as the same role the application uses.
- Every area has a test that proves company A can neither read nor change the data of company B.
- Everything outside the system is replaced by a local stand-in, so the whole suite runs with no provider account and no network.
- A change ships with its tests. Generated files have a test that fails when they are out of date.
- A query behind a list, a panel or a search has its execution plan checked against a seed of realistic volume.
The layers
Section titled “The layers”The counts are test cases declared in the code today (it(...) and test(...) blocks; a parameterized block counts once and runs more cases than that).
| Layer | What it runs against, and where | Files | Test cases |
|---|---|---|---|
| Database | A real PostgreSQL with every migration applied. In packages/db/test. |
49 | 858 |
| API | The API in process, a real PostgreSQL and a real Redis, with simulators for the providers. In apps/api/test. |
162 | 2,086 |
| Worker | The storage setup of the worker process. The queue consumers are tested in the API layer, where they live. In apps/worker/test. |
1 | 2 |
| Office system | The React screens in jsdom, with the generated client replaced at its boundary. In apps/web/test. |
81 | 737 |
| Components | The components of the design system in jsdom. In packages/ui/test. |
9 | 125 |
| Careers site | The pages as the built server answers them, against a stand-in for the API. In apps/careers/test. |
9 | 111 |
| Landing, driver app, documentation | The built pages and the screens. In the test folder of apps/landing, apps/driver and apps/docs. |
14 | 76 |
| Shared packages | The request helper, the generated client and the lint rules. In the test folder of packages/app-kit, packages/contracts and packages/config. |
9 | 20 |
| End to end | The built office system, careers site and landing site in Chromium, against the real API, worker, database and Redis. In e2e/tests. |
44 | 63 |
Vitest runs every layer except the last, which is Playwright. In total: 334 files and 4,015 declared cases in Vitest, and 44 files and 63 scenarios in Playwright.
Database tests
Section titled “Database tests”packages/db/test connects as ochema_app, the role of the application, and as the other roles when the test is about them. It covers:
- Isolation by area (
isolation*.test.ts, nine files): for each table, reading, writing and foreign keys across two companies, the exact privileges of each role per table and per column, and the policies. - Roles and privileges (
privileges.test.ts): no runtime role is a superuser or can bypass row level security, and every table has row level security enabled and forced. - The functions that cross the boundary (
gate.test.ts,public-context.test.ts,wave*-functions.test.ts,staff-*.test.ts): what each definer function returns, to whom, and what it refuses. - Upgrades (
upgrade*.test.ts, 14 files): each migration that changes existing data is applied over a database that already has companies, and the result is checked. - Plans (
volume-plans.test.ts): the search functions keep every company to itself and use a plan per call.
See Tenant isolation.
API tests
Section titled “API tests”apps/api/test starts the real application and calls it over HTTP. global-setup.ts recreates a dedicated database with the migrations before each run, so the development database is never touched. Each test file creates its own companies and uses its own Redis key prefix, which lets the files run in parallel. Six files that change rows shared by every company, such as the catalog of compliance rules, run afterwards, one at a time (apps/api/vitest.config.ts).
Besides the behaviour of each route, the suite has tests of a specific kind:
| Kind | What it proves, with examples |
|---|---|
| Isolation | A session of one company gets nothing of another through the API. isolation.test.ts, and the cases with two companies in the test files of the modules. |
| Access | Every route declares who may call it, and each role gets exactly its permissions. route-boundary.test.ts, role-access.test.ts, permissions.test.ts. |
| Concurrency | Simultaneous requests end in a correct state, never in an error 500. scheduling-concurrency.test.ts (200 bookings of one time: one interview, 199 refusals) and lead-distribution-concurrency.test.ts (300 leads among three recruiters: 100 each). |
| Volume | A routine keeps its totals over thousands of rows. import-volume.test.ts, broadcast-volume.test.ts, automation-volume.test.ts, compliance-engine-volume.test.ts. |
| Timing | A promised time is kept on the real worker. first-response-timing.test.ts, domain-events.test.ts. |
| Provider adapters | The adapter of each real provider, against local stand-ins that answer what the documentation of the provider shows. No test calls the network. twilio-send.test.ts, twilio-webhook.test.ts, ses-send.test.ts, smtp-send.test.ts, calendar-adapters.test.ts. |
| Architecture | The structure of the code. lint-boundary.test.ts and module-cycles.test.ts for the module boundaries, openapi.test.ts for the contract. |
Front end tests
Section titled “Front end tests”The screens of the office system run in jsdom with Testing Library. The tests replace the functions of @ochema/contracts, the generated client, and nothing below or above it (apps/web/test/api-mock.ts), so a screen is tested with the real types of the contract. Elements are found by role and accessible name, the way a screen reader finds them.
End to end tests
Section titled “End to end tests”e2e/tests drives a browser against the built apps and the real API and worker, with a database recreated for the run. journey.spec.ts follows a new customer through the screens a person uses, from the landing page to a driver who is eligible to drive and the audit export of the driver file. Other scenarios cover one area each, the top bar at 390 pixels wide (narrow.spec.ts) and two tabs in different companies (two-tabs.spec.ts). Every scenario also fails when the browser console shows an error or a warning (console-guard.spec.ts proves the guard itself).
Stand-ins for outside systems
Section titled “Stand-ins for outside systems”| Outside system | Stand-in in tests and in development |
|---|---|
| SMS provider | A simulator that keeps the message and marks it delivered; test routes inject replies. In apps/api/src/modules/platform-messaging. |
| Email provider | An outbox that writes each email to a local folder. In the same module and in platform-mail. |
| Calendar providers | An in-memory gateway, fake.gateway.ts in apps/api/src/modules/platform/calendar, which production refuses. |
| Object storage | An S3-compatible server in development (docker-compose.yml); an in-process fake in the API tests (apps/api/test/fake-s3.ts). |
See ADR-0013.
Volume and plans
Section titled “Volume and plans”pnpm --filter @ochema/db seed:volume fills a separate database with synthetic companies at a chosen scale. The scripts explain:wave01 and explain:wave04 of the same package run the SQL of the critical queries of the API against that database, as the application role with a company in context, and report the plan and the time of each one. The measurements taken this way, with their conditions, are in Quality requirements.
How the checks run
Section titled “How the checks run”scripts/check.sh runs the same steps on a developer machine and in continuous integration, in this order: format, lint, type check, tests, build, and the end to end suite when asked. The workflow .github/workflows/ci.yml runs them and also builds the container images.
Lint is part of the test strategy. Three rules described elsewhere are enforced by lint and not by review: module boundaries (Building blocks), the generated client (API contract) and the design tokens (Design system).
Known limits
Section titled “Known limits”- No real provider is called. The adapters for Twilio, Amazon SES, SMTP, Google Calendar and Microsoft Graph are tested against local stand-ins. Sending through a real account is Not verified.
- No load test. The suite proves correctness under concurrency and the plans of the critical queries. It does not measure the API under many simultaneous users, and it never runs more than one instance of the API or of the worker.
- Code coverage is not measured. No coverage tool is configured.
- One browser. The end to end suite runs in Chromium only.
- Accessibility is tested by role and name, not audited. No automated accessibility scanner runs, and no screen reader pass is recorded. See Quality requirements.
- Timing tests depend on the machine. The measurements come from a development machine and are stated with that condition.
Related
Section titled “Related”- Quality requirements: the requirements these tests are the evidence for.
- ADR-0013: providers behind adapters, with a simulator.
- ADR-0004: versioned migrations, and the upgrade tests.
- Tenant isolation: what the isolation tests prove.