# Deployment operations

## Backups and recovery

Production creates a verified SQLite snapshot before migrations and automatically
when the latest snapshot is older than `BACKUP_INTERVAL_HOURS` (24 by default).
The server checks once a minute. Snapshot files are private (0600), created with
SQLite `VACUUM INTO`, and checked for integrity and foreign-key errors before being
published. Seven snapshots are retained by default; `BACKUP_RETAIN` accepts 2–365.
Pre-migration snapshots are mandatory; a failure prevents migration. Set
`BACKUP_ENABLED=false` only to disable the periodic runner if an external backup
system already handles it. Manual `bun run db:backup` always takes a snapshot.

`BACKUP_DIR` defaults to `backups` beside the database. Relative backup paths are
resolved relative to the database directory, consistently in CLI and production.
Use a durable volume for both the database and backups. Configure
`BACKUP_REPLICA_DIR` to an existing mounted directory on a separate machine/storage
service to keep an off-machine copy. The app does not provision or authenticate
that storage. If the directory is unavailable, backup reports failure and does
not prune old snapshots. Local-only backups do not protect against machine loss.
Snapshots run synchronously; for larger databases use the backup CLI from an
external scheduler and disable the in-process periodic runner.

To rehearse a restore without touching the live database:

```sh
bun run db:backup
bun run db:restore /absolute/path/to/backup.sqlite /absolute/path/to/recovered.sqlite
```

Restore requires a new destination, verifies integrity, and revokes all sessions
in the restored copy. It never overwrites a database or its WAL/SHM sidecars.
Test the recovered copy with the matching release and an isolated local port.
For an actual recovery, stop the service, set `DATABASE_PATH` to the recovered
file, then start the service and sign in again. Keep the original file for rollback.
Do not copy only the live `.sqlite` file: recent commits may be in its WAL.

Backups include account data and session hashes. Protect backup storage and expire
copies according to your published retention period. Restoring an older backup
can restore deleted accounts; reapply deletions performed since the snapshot
before reopening access. Discord images already posted are separate from app data.

Verification: `bun test src/ops/backups.test.ts` exercises WAL progress, restored
content, revoked sessions, replicas, retention, missing storage, and refusal to
overwrite an existing destination.

## Monitoring

Every API response includes a server-generated `X-Request-ID`. Unexpected errors
return that reference to the browser and write a JSON `request_failed` event to
stderr with the request ID, method, route template, error type, and stack locations.
Error messages, SQL values, request bodies, cookies, and query strings are omitted
so credentials and habit content do not enter logs. Production source maps in
`dist/` help map stack locations back to the source. Backup outcomes also emit
`backup_completed` and `backup_failed` events.

`GET /api/health` checks SQLite and returns 503 if enabled automatic backups are
failing or have never succeeded. It returns no account data, paths, or credentials.
Run the health monitor independently of the application process:

```sh
bun run monitor --once
bun run monitor
```

`MONITOR_URL` defaults to `http://127.0.0.1:$PORT/api/health`. For an outside-in
check, run the monitor on another machine and set it to the public HTTPS endpoint.
The one-shot command exits nonzero on failure. Continuous mode checks every minute,
times out after ten seconds, reports `health_failed` after three consecutive
failures, and emits one `health_recovered` event on recovery. It stays quiet while
state is unchanged. Run it under a process supervisor and route these JSON events
to your deployment platform's alerting/log collection. No external alert provider
or off-machine monitor is provisioned automatically by this repository.

## Reminder worker

The production process checks opt-in reminders every minute, with a lock preventing
overlapping runs within the process and transactional delivery reservations across
processes. A crash after reservation may leave a reminder unconfirmed; the worker
chooses not to resend it that day. Known rate-limit rejections defer until Discord's
retry deadline. Quiet hours and completion are checked again immediately before
posting. Recovery disables saved reminder opt-ins to avoid replaying notifications
from an old backup. Use the disposable preview and stubbed transport tests for QA;
real Discord delivery depends on the bot token, shared server, and user DM permissions.