4.6 KiB
Deployment operations
Backups and recovery
Production creates a verified SQLite snapshot before migrations and automatically
when the latest snapshot is older than BACKUP_INTERVAL_HOURS (24 by default).
The server checks once a minute. Snapshot files are private (0600), created with
SQLite VACUUM INTO, and checked for integrity and foreign-key errors before being
published. Seven snapshots are retained by default; BACKUP_RETAIN accepts 2–365.
Pre-migration snapshots are mandatory; a failure prevents migration. Set
BACKUP_ENABLED=false only to disable the periodic runner if an external backup
system already handles it. Manual bun run db:backup always takes a snapshot.
BACKUP_DIR defaults to backups beside the database. Relative backup paths are
resolved relative to the database directory, consistently in CLI and production.
Use a durable volume for both the database and backups. Configure
BACKUP_REPLICA_DIR to an existing mounted directory on a separate machine/storage
service to keep an off-machine copy. The app does not provision or authenticate
that storage. If the directory is unavailable, backup reports failure and does
not prune old snapshots. Local-only backups do not protect against machine loss.
Snapshots run synchronously; for larger databases use the backup CLI from an
external scheduler and disable the in-process periodic runner.
To rehearse a restore without touching the live database:
bun run db:backup
bun run db:restore /absolute/path/to/backup.sqlite /absolute/path/to/recovered.sqlite
Restore requires a new destination, verifies integrity, and revokes all sessions
in the restored copy. It never overwrites a database or its WAL/SHM sidecars.
Test the recovered copy with the matching release and an isolated local port.
For an actual recovery, stop the service, set DATABASE_PATH to the recovered
file, then start the service and sign in again. Keep the original file for rollback.
Do not copy only the live .sqlite file: recent commits may be in its WAL.
Backups include account data and session hashes. Protect backup storage and expire copies according to your published retention period. Restoring an older backup can restore deleted accounts; reapply deletions performed since the snapshot before reopening access. Discord images already posted are separate from app data.
Verification: bun test src/ops/backups.test.ts exercises WAL progress, restored
content, revoked sessions, replicas, retention, missing storage, and refusal to
overwrite an existing destination.
Monitoring
Every API response includes a server-generated X-Request-ID. Unexpected errors
return that reference to the browser and write a JSON request_failed event to
stderr with the request ID, method, route template, error type, and stack locations.
Error messages, SQL values, request bodies, cookies, and query strings are omitted
so credentials and habit content do not enter logs. Production source maps in
dist/ help map stack locations back to the source. Backup outcomes also emit
backup_completed and backup_failed events.
GET /api/health checks SQLite and returns 503 if enabled automatic backups are
failing or have never succeeded. It returns no account data, paths, or credentials.
Run the health monitor independently of the application process:
bun run monitor --once
bun run monitor
MONITOR_URL defaults to http://127.0.0.1:$PORT/api/health. For an outside-in
check, run the monitor on another machine and set it to the public HTTPS endpoint.
The one-shot command exits nonzero on failure. Continuous mode checks every minute,
times out after ten seconds, reports health_failed after three consecutive
failures, and emits one health_recovered event on recovery. It stays quiet while
state is unchanged. Run it under a process supervisor and route these JSON events
to your deployment platform's alerting/log collection. No external alert provider
or off-machine monitor is provisioned automatically by this repository.
Reminder worker
The production process checks opt-in reminders every minute, with a lock preventing overlapping runs within the process and transactional delivery reservations across processes. A crash after reservation may leave a reminder unconfirmed; the worker chooses not to resend it that day. Known rate-limit rejections defer until Discord's retry deadline. Quiet hours and completion are checked again immediately before posting. Recovery disables saved reminder opt-ins to avoid replaying notifications from an old backup. Use the disposable preview and stubbed transport tests for QA; real Discord delivery depends on the bot token, shared server, and user DM permissions.