74 lines
3.9 KiB
Markdown
74 lines
3.9 KiB
Markdown
# Deployment operations
|
||
|
||
## Backups and recovery
|
||
|
||
Production creates a verified SQLite snapshot before migrations and automatically
|
||
when the latest snapshot is older than `BACKUP_INTERVAL_HOURS` (24 by default).
|
||
The server checks once a minute. Snapshot files are private (0600), created with
|
||
SQLite `VACUUM INTO`, and checked for integrity and foreign-key errors before being
|
||
published. Seven snapshots are retained by default; `BACKUP_RETAIN` accepts 2–365.
|
||
Pre-migration snapshots are mandatory; a failure prevents migration. Set
|
||
`BACKUP_ENABLED=false` only to disable the periodic runner if an external backup
|
||
system already handles it. Manual `bun run db:backup` always takes a snapshot.
|
||
|
||
`BACKUP_DIR` defaults to `backups` beside the database. Relative backup paths are
|
||
resolved relative to the database directory, consistently in CLI and production.
|
||
Use a durable volume for both the database and backups. Configure
|
||
`BACKUP_REPLICA_DIR` to an existing mounted directory on a separate machine/storage
|
||
service to keep an off-machine copy. The app does not provision or authenticate
|
||
that storage. If the directory is unavailable, backup reports failure and does
|
||
not prune old snapshots. Local-only backups do not protect against machine loss.
|
||
Snapshots run synchronously; for larger databases use the backup CLI from an
|
||
external scheduler and disable the in-process periodic runner.
|
||
|
||
To rehearse a restore without touching the live database:
|
||
|
||
```sh
|
||
bun run db:backup
|
||
bun run db:restore /absolute/path/to/backup.sqlite /absolute/path/to/recovered.sqlite
|
||
```
|
||
|
||
Restore requires a new destination, verifies integrity, and revokes all sessions
|
||
in the restored copy. It never overwrites a database or its WAL/SHM sidecars.
|
||
Test the recovered copy with the matching release and an isolated local port.
|
||
For an actual recovery, stop the service, set `DATABASE_PATH` to the recovered
|
||
file, then start the service and sign in again. Keep the original file for rollback.
|
||
Do not copy only the live `.sqlite` file: recent commits may be in its WAL.
|
||
|
||
Backups include account data and session hashes. Protect backup storage and expire
|
||
copies according to your published retention period. Restoring an older backup
|
||
can restore deleted accounts; reapply deletions performed since the snapshot
|
||
before reopening access. Discord images already posted are separate from app data.
|
||
|
||
Verification: `bun test src/ops/backups.test.ts` exercises WAL progress, restored
|
||
content, revoked sessions, replicas, retention, missing storage, and refusal to
|
||
overwrite an existing destination.
|
||
|
||
## Monitoring
|
||
|
||
Every API response includes a server-generated `X-Request-ID`. Unexpected errors
|
||
return that reference to the browser and write a JSON `request_failed` event to
|
||
stderr with the request ID, method, route template, error type, and stack locations.
|
||
Error messages, SQL values, request bodies, cookies, and query strings are omitted
|
||
so credentials and habit content do not enter logs. Production source maps in
|
||
`dist/` help map stack locations back to the source. Backup outcomes also emit
|
||
`backup_completed` and `backup_failed` events.
|
||
|
||
`GET /api/health` checks SQLite and returns 503 if enabled automatic backups are
|
||
failing or have never succeeded. It returns no account data, paths, or credentials.
|
||
Run the health monitor independently of the application process:
|
||
|
||
```sh
|
||
bun run monitor --once
|
||
bun run monitor
|
||
```
|
||
|
||
`MONITOR_URL` defaults to `http://127.0.0.1:$PORT/api/health`. For an outside-in
|
||
check, run the monitor on another machine and set it to the public HTTPS endpoint.
|
||
The one-shot command exits nonzero on failure. Continuous mode checks every minute,
|
||
times out after ten seconds, reports `health_failed` after three consecutive
|
||
failures, and emits one `health_recovered` event on recovery. It stays quiet while
|
||
state is unchanged. Run it under a process supervisor and route these JSON events
|
||
to your deployment platform's alerting/log collection. No external alert provider
|
||
or off-machine monitor is provisioned automatically by this repository.
|