Background work
This page describes how Ochema does the work that does not fit inside a request: sending messages, reacting to events, running reminders and daily routines, generating PDFs. It is part of section 9 of the arc42 template. The decisions are in ADR-0006 and ADR-0025.
The problem
Section titled “The problem”A request must answer quickly, and much of what the product does is slow, scheduled or depends on a system outside it. Moving that work to a background process raises three questions:
- What if the process stops? Work that lives only in memory or in a queue can be lost.
- Whose data is it? The API always acts for one company, chosen by the session. A background process has no session, and under tenant isolation a transaction without a company reads nothing.
- What if it runs twice? Retries and restarts deliver the same work again.
The rule
Section titled “The rule”- A request commits rows and nothing else. It never calls a provider and never waits for slow work.
- The database is the record of what is pending. A queue only triggers and carries work.
- The worker reads business data inside one company at a time, under the same policies as the API, with no user.
- A business event is written in the transaction of the change it describes. It exists if and only if the change was committed.
- Every piece of background work tolerates being run twice.
How the code applies it
Section titled “How the code applies it”The worker process
Section titled “The worker process”apps/worker has two source files. It starts the WorkerModule that the API code base exports as @ochema/api/worker (apps/api/src/worker.ts), so each queue consumer and each routine lives in the module it belongs to, next to the services it calls. The worker is a NestJS application without HTTP.
It connects to PostgreSQL as the same role as the API, ochema_app, and to Redis for the queues, which are BullMQ queues. It has no privilege the API lacks.
Three ways work reaches the worker
Section titled “Three ways work reaches the worker”%%{init: {"flowchart": {"wrappingWidth": 170}}}%%
flowchart LR
subgraph api["API request, one transaction"]
row["<b>Business row</b><br/>with a due time"]
event["<b>Outbox row</b><br/>domain_events"]
job["<b>Job after commit</b><br/>with the company id"]
end
pass["<b>Scheduled pass</b><br/>Which companies have<br/>work due?"]
dispatch["<b>Dispatcher pass</b><br/>One delivery<br/>per consumer"]
queue[("<b>Queue</b><br/>BullMQ on Redis")]
work["<b>Work</b><br/>One transaction per<br/>company, no user"]
row -.->|"Found in the<br/>database"| pass
event -.->|"Found in the<br/>database"| dispatch
dispatch -->|"Delivery job"| queue
job -->|"Job"| queue
pass --> work
queue --> work
Legend. Large box: what a request writes before it answers. Dotted arrow: nothing is sent; a later pass finds the committed row by asking the database. Solid arrow: a job put on a queue. Cylinder: the queues. Every path ends in a transaction that carries one company.
| Path | Used for | What survives a lost queue |
|---|---|---|
| Scheduled pass | Messages due, automation actions, sequence steps, broadcasts, reminders, expirations, daily compliance | Everything: the rows are still due, and the next pass finds them. |
| Outbox | Reactions to a business event | The event: it stays pending until a pass marks it dispatched. |
| Job after commit | Work asked for by a user that should start at once: a PDF, an import, an export, a score recalculation, account email | The job itself is in the queue. Exports and signed PDFs also have a pass that picks up a row still waiting. |
Scheduled passes and work discovery
Section titled “Scheduled passes and work discovery”A routine is a BullMQ job scheduler that fires a pass at a fixed interval. BullMQ gives each pass to one instance of the worker. A pass does two things.
It asks which companies have work. DueWorkTenants.find(kind) (apps/api/src/core/events/due-work.ts) calls the database function gate.tenants_with_due_work(kind) outside any company context. The function returns only the ids of active companies that have work of that kind due now. It reads only tenant_id and the status and due-time columns of the tables involved, sets no context, and refuses to run inside a transaction that already has a company. There are 21 kinds (dueWorkKinds in packages/db/src/gate.ts).
It opens each company. For each id, the routine opens withContext({ tenantId, userId: null }) and does the work under row level security. The sending pass and the event dispatcher take their rows with FOR UPDATE SKIP LOCKED, so a second instance of the worker takes the next rows instead of waiting or repeating.
| Routine | Interval | Source |
|---|---|---|
| Dispatch of priority events | 1 second | core/events/events.processors.ts |
| Dispatch of other events | 2 seconds | core/events/events.processors.ts |
| Sending of messages | 1 second | modules/platform-messaging/send-queue.processor.ts |
| Broadcast recipients | 2 seconds | modules/attract/broadcasts/broadcast-send.routine.ts |
| Automation actions | 10 seconds | modules/platform-automations/automations.module.ts |
| Sequence steps | 10 seconds | modules/recruit/sequences/sequence-steps.routine.ts |
| Interview reminders, overdue tasks, calendar reads, signing expirations, export pickup | 1 minute | The routine file of each module |
| Safety history reminders and deadlines | 10 minutes | modules/compliance/safety-history/safety-history.routine.ts |
| Daily compliance recalculation, abandoned application reminders, time-based automation triggers | 1 hour | The routine file of each module |
All paths are under apps/api/src. The interval is the longest a piece of work waits before a pass sees it.
The outbox of domain events
Section titled “The outbox of domain events”emitDomainEvent (apps/api/src/core/events/emit.ts) inserts one row into domain_events through the transaction of the caller. The company comes from the transaction context, never from a parameter.
- A typed catalog. Each event type is an entry of
domainEventCatalog(catalog.ts) with a schema for its body. A body outside the schema throws at the emitter. - Small bodies without personal data. A body carries what belongs to the instant of the event, such as the stage an application left and the one it entered. It holds no name, email, phone or text typed by a person, and at most 8 KB. A consumer reads the current state from the database.
- Two lanes.
application.createdfrom a live applicant travels in theprioritylane, with its own pass and its own queue, so the first message to a new applicant never waits behind an import or a broadcast. Everything else uses thedefaultlane. - Dispatch. The dispatcher (
dispatcher.ts) opens each company with pending events, reads them in order of time, queues one delivery per registered consumer, and marks the events dispatched in the same transaction. - At least once. A stop between queuing and the commit delivers again. Each delivery carries the key
<consumer>:<event id>, and a consumer makes its work unique by that key. - Retries. A consumer that throws is retried up to 8 times, waiting 5 seconds and then twice as long each time. After that the failure is recorded in
domain_event_failuresand the following events are not held. - No loops. An event emitted by an automation action carries a chain depth. The automation engine stops running rules at a depth of 3 (
CHAIN_LIMITinapps/api/src/modules/platform-automations/automation.types.ts), so rules cannot trigger each other without end; a constraint of the table caps the stored depth at 10.
A consumer registers itself by name in DomainEventConsumers (consumers.ts). The consumers today are the automation engine, the exit of sequences, the compliance engine, onboarding, expiration notices and the recovery of abandoned applications. Adding a reaction is adding a consumer: the module that emits the event does not change.
Jobs put on a queue by the API
Section titled “Jobs put on a queue by the API”Some work starts from a click and should not wait for a pass. The route registers the job with RequestContext.afterCommit, so the job is queued only after the transaction commits and never when it rolls back. The job body carries the company id of the request, and the consumer opens its transaction with it.
These jobs are retried by the queue with a growing wait. The score recalculation, for example, writes nothing when the result did not change, so a job that failed halfway runs again from the start. Sending a message is not one of these jobs: a message is a row, found by the sending pass. See Messaging.
Maintenance
Section titled “Maintenance”| Routine | When | What it does |
|---|---|---|
| Cleanup | Every hour | Removes expired sessions of companies and of the platform panel, old password reset tokens and expired application drafts, through database functions that return only counts. |
| Audit partitions | Daily, and at start | Creates the monthly partitions of audit_log and platform_audit_log three months ahead. |
| Event maintenance | Daily | Creates the monthly partitions of messages, domain_events and file_access_log three months ahead; logs a warning for events pending too long; logs the dispatch lag of each lane against its goal of 5 seconds at the 95th percentile. |
| Messaging usage | Daily | Compares the monthly usage counters of each company with its messages and corrects a counter that drifted. |
When something fails
Section titled “When something fails”- A pass fails for one company. The error is logged, the rows stay due, and the company is opened again in the next pass. Other companies in the same pass are not affected.
- The worker stops mid-work. A reserved message becomes visible to another pass after 2 minutes. An event not yet marked dispatched is dispatched again. A stalled delivery is put back by the queue.
- Redis is unavailable. Passes stop while it is down. Nothing decides in Redis whether a message leaves or an event is delivered, so work resumes from the database when it is back.
- The database drops a connection. Only the transaction using it fails. The process stays up and the pool opens a new connection.
What guarantees it
Section titled “What guarantees it”| Test file | What it proves |
|---|---|
apps/api/test/domain-events.test.ts |
An event exists only when its transaction commits; a body outside the catalog is refused; each event reaches each consumer with its key and company; a worker interrupted mid-dispatch delivers again and an idempotent consumer processes once; a consumer that always fails is recorded after 8 attempts without holding later events or another company; the priority lane does not wait behind the default lane and stays under 5 seconds at the 95th percentile in the test. |
packages/db/test/wave03-automation-functions.test.ts, wave04-public-functions.test.ts |
gate.tenants_with_due_work per kind: what counts as due, only active companies, no context set, refusal inside a context and for an unknown kind, and the list of kinds in the code equal to the one in the database. |
apps/api/test/messaging-queue.test.ts |
The sending pass: limit per second, priorities, the re-read at the reservation, and each provider outcome. |
apps/api/test/automation-engine.test.ts, sequences.test.ts, broadcasts.test.ts, interview-reminders.test.ts, draft-recovery.test.ts |
The routines of each feature. |
apps/api/test/application-drafts-purge.test.ts, packages/db/test/audit-partitions.test.ts |
The cleanup of drafts and the creation of partitions. |
apps/api/test/connection-loss.test.ts |
A dropped database connection fails one request and not the process. |
Known limits
Section titled “Known limits”- Polling, not push. Background work starts at the next pass, so its delay is bounded by the interval in the table above: about a second for a message, up to an hour for a daily recalculation.
- Reactions are not immediate and not strictly ordered. A reaction to an event runs shortly after the action, not inside it, and two events of the same record can be handled out of order. Consumers read the current state instead of trusting the order.
- Very old pending work is set aside. A message or an event still pending after 35 days is no longer picked up. The daily routine reports events in that state.
- Scale is by adding workers. There is one kind of worker, which runs every routine. Dedicating instances to one kind of work is not available.
- A new kind of routine needs a database migration. The discovery function has one branch per kind, by design.
Related
Section titled “Related”- ADR-0006: worker, queues and work discovery by company. ADR-0025: domain events with a transactional outbox. ADR-0028: monthly partitions.
- Runtime view: the passes and the outbox inside each flow.
- Tenant isolation: how the worker gets a company.
- Messaging: the largest user of scheduled passes.
- Building blocks: what each module runs in the worker.
- Deployment view: where the worker and Redis run.