feat: automate verified SQLite backups and safe recovery

This commit is contained in:
syntaxbullet
2026-09-04 18:28:28 +02:00
parent 32002151e0
commit 661bc6c346
10 changed files with 227 additions and 2 deletions

45
docs/OPERATIONS.md Normal file
View File

@@ -0,0 +1,45 @@
# Deployment operations
## Backups and recovery
Production creates a verified SQLite snapshot before migrations and automatically
when the latest snapshot is older than `BACKUP_INTERVAL_HOURS` (24 by default).
The server checks once a minute. Snapshot files are private (0600), created with
SQLite `VACUUM INTO`, and checked for integrity and foreign-key errors before being
published. Seven snapshots are retained by default; `BACKUP_RETAIN` accepts 2365.
Pre-migration snapshots are mandatory; a failure prevents migration. Set
`BACKUP_ENABLED=false` only to disable the periodic runner if an external backup
system already handles it. Manual `bun run db:backup` always takes a snapshot.
`BACKUP_DIR` defaults to `backups` beside the database. Relative backup paths are
resolved relative to the database directory, consistently in CLI and production.
Use a durable volume for both the database and backups. Configure
`BACKUP_REPLICA_DIR` to an existing mounted directory on a separate machine/storage
service to keep an off-machine copy. The app does not provision or authenticate
that storage. If the directory is unavailable, backup reports failure and does
not prune old snapshots. Local-only backups do not protect against machine loss.
Snapshots run synchronously; for larger databases use the backup CLI from an
external scheduler and disable the in-process periodic runner.
To rehearse a restore without touching the live database:
```sh
bun run db:backup
bun run db:restore /absolute/path/to/backup.sqlite /absolute/path/to/recovered.sqlite
```
Restore requires a new destination, verifies integrity, and revokes all sessions
in the restored copy. It never overwrites a database or its WAL/SHM sidecars.
Test the recovered copy with the matching release and an isolated local port.
For an actual recovery, stop the service, set `DATABASE_PATH` to the recovered
file, then start the service and sign in again. Keep the original file for rollback.
Do not copy only the live `.sqlite` file: recent commits may be in its WAL.
Backups include account data and session hashes. Protect backup storage and expire
copies according to your published retention period. Restoring an older backup
can restore deleted accounts; reapply deletions performed since the snapshot
before reopening access. Discord images already posted are separate from app data.
Verification: `bun test src/ops/backups.test.ts` exercises WAL progress, restored
content, revoked sessions, replicas, retention, missing storage, and refusal to
overwrite an existing destination.