deploy: automatically test and release pushed main commits

2026-09-07 12:57:42 +02:00
parent 5f36358709
commit b20af6d95c
6 changed files with 298 additions and 0 deletions
--- a/deploy/README.md
+++ b/deploy/README.md
@@ -0,0 +1,71 @@
+# Automatic production deployment
+
+The server checks `git@git.ayau.me:syntaxbullet/minabot.git`, branch `main`, every
+minute after the last check finishes. A local commit is deployed after it is
+pushed to Gitea. Other branches are not deployed. If several commits arrive
+between checks, the latest fetched main commit is built.
+
+```sh
+git push origin main
+```
+
+The server uses a repository-only read key under `/opt/minabot/git-auth`, registered
+as `minabot-production-read-only` in Gitea. The personal Mac SSH key is not copied
+to the server. Host verification uses the existing trusted Gitea host keys.
+
+The pipeline fetches into a bare repository and archives an exact commit into
+`/opt/minabot/releases/<sha>`. It builds a test image, runs typecheck and the full
+test suite, builds the runtime image, runs a disposable production smoke test,
+and validates Compose/Caddy. Secrets and production SQLite stay on the server;
+they are never synced from Git or development during a release.
+
+After checks pass, Caddy and the old app stop briefly, a verified database snapshot
+is taken, and the new app starts before Caddy reopens public access. Discord workers
+never overlap. A lock prevents concurrent manual/timer deployments. Successful
+commits are recorded in `/opt/minabot/deployed-sha` and skipped on later checks.
+
+A build/test failure leaves the old service running. Failed revisions are recorded
+in `failed-sha` and skipped until a new commit or an explicit retry. On startup
+failure, the old configuration/image is restored only if the schema and migration
+fingerprint is unchanged. Otherwise services remain stopped for operator recovery;
+the script never overwrites the production database with an old snapshot.
+Changes to `.env.production` or Caddy's pinned image remain explicit server operations.
+
+## Server operations
+
+```sh
+# Check status and recent deployment output
+systemctl status minabot-deploy.timer minabot-deploy.service
+journalctl -u minabot-deploy.service -n 100 --no-pager
+cat /opt/minabot/deployed-sha
+
+# Run a check now / retry a previously failed revision
+sudo systemctl start minabot-deploy.service
+/usr/local/bin/minabot-deploy --retry
+
+# Pause / resume automatic deployment
+sudo systemctl stop minabot-deploy.timer
+sudo systemctl start minabot-deploy.timer
+```
+
+Build/test logs are inside each release directory. Retained release images and
+Docker build cache need occasional housekeeping; the script refuses a build when
+less than 2 GiB is free. Keep the current and previous known-good image and backups.
+Failures are logged locally; external alerts are not configured.
+
+To install or update the controller from a reviewed checkout on the server:
+
+```sh
+sudo install -m 755 deploy/deploy.sh /usr/local/bin/minabot-deploy
+sudo install -m 644 deploy/minabot-deploy.service deploy/minabot-deploy.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl enable --now minabot-deploy.timer
+```
+
+Controller/service changes require this explicit installation; normal app,
+Dockerfile, Compose, and Caddyfile changes deploy automatically. Do not force-reset
+`main` to roll back: prefer a revert commit so the pipeline tests the result.
+
+`bun test deploy/deploy.test.ts` verifies success, idempotency, failure before
+cutover, safe rollback, and refusal to replace data after a schema change using
+isolated fake Git/Docker commands. It never contacts production.
--- a/deploy/deploy.sh
+++ b/deploy/deploy.sh
@@ -0,0 +1,106 @@
+#!/usr/bin/env bash
+# Installed on the server as /usr/local/bin/minabot-deploy.
+set -Eeuo pipefail
+umask 077
+root=${MINABOT_DEPLOY_ROOT:-/opt/minabot}
+cd "$root"
+exec 9>deploy.lock
+flock -n 9 || exit 0
+export GIT_SSH_COMMAND="ssh -i $root/git-auth/id_ed25519 -o IdentitiesOnly=yes -o BatchMode=yes -o StrictHostKeyChecking=yes -o UserKnownHostsFile=$root/git-auth/known_hosts -o ConnectTimeout=10 -o ServerAliveInterval=15 -o ServerAliveCountMax=2"
+repo="$root/git-sync.git"
+target=''
+switching=false
+schema_before=''
+previous_image=''
+docker_cmd() { sudo -n docker "$@" </dev/null; }
+compose() { docker_cmd compose --project-directory "$root" -f "$root/compose.yaml" --env-file "$root/.env" "$@"; }
+schema_fingerprint() {
+  docker_cmd run --rm --network none -v /srv/minabot/data:/data:ro "$1" bun -e '
+    import {Database} from "bun:sqlite";
+    const db=new Database("/data/minabot.sqlite",{readonly:true});
+    const schema=db.query("SELECT type,name,sql FROM sqlite_master WHERE sql IS NOT NULL ORDER BY type,name").all();
+    const migrations=db.query("SELECT * FROM __drizzle_migrations ORDER BY id").all();
+    console.log(new Bun.CryptoHasher("sha256").update(JSON.stringify({schema,migrations})).digest("hex"));
+    db.close();'
+}
+failed() {
+  local result=$?
+  trap - ERR
+  set +e
+  printf 'Deployment failed: %s (exit %s)\n' "${target:-fetch}" "$result" >&2
+  printf 'Build and test logs: %s\n' "${release:-$root}" >&2
+  if [[ -n "$target" ]]; then printf '%s\n' "$target" > failed-sha; fi
+  if $switching; then
+    compose stop caddy app monitor
+    local schema_after
+    schema_after=$(schema_fingerprint "$previous_image")
+    if [[ -n "$schema_before" && "$schema_before" == "$schema_after" ]]; then
+      cp "$rollback/.env" .env
+      cp "$rollback/compose.yaml" compose.yaml
+      cp "$rollback/Caddyfile" Caddyfile
+      if compose up -d --wait --wait-timeout 120; then
+        echo 'Previous release restored; database was not replaced.' >&2
+      else
+        echo 'Rollback failed; inspect the journal and saved release configuration.' >&2
+      fi
+    else
+      echo "Database migrations changed or could not be verified. Services remain stopped; inspect $rollback and /srv/minabot/data/backups before recovery. No database was overwritten." >&2
+    fi
+  fi
+  exit "$result"
+}
+trap failed ERR
+
+if [[ ! -d "$repo" ]]; then
+  git init --bare "$repo"
+  git --git-dir="$repo" remote add origin git@git.ayau.me:syntaxbullet/minabot.git
+fi
+timeout 90 git --git-dir="$repo" fetch --quiet origin +refs/heads/main:refs/remotes/origin/main
+target=$(git --git-dir="$repo" rev-parse refs/remotes/origin/main)
+[[ "$target" =~ ^[0-9a-f]{40}$ ]]
+[[ "$target" != "$(cat deployed-sha 2>/dev/null || true)" ]] || exit 0
+if [[ "${1:-}" != '--retry' && "$target" == "$(cat failed-sha 2>/dev/null || true)" ]]; then exit 0; fi
+
+echo "Testing main at $target"
+available_kb=$(df -Pk "$root" | awk 'NR==2 {print $4}')
+if (( available_kb < 2097152 )); then echo 'Less than 2 GiB free; refusing to build.' >&2; false; fi
+release="$root/releases/$target"
+mkdir -p "$release"
+git --git-dir="$repo" archive "$target" | tar -x -C "$release"
+image="minabot:git-${target:0:12}"
+checks="$image-checks"
+docker_cmd build --target build -t "$checks" "$release" > "$release/build-checks.log" 2>&1
+docker_cmd run --rm "$checks" sh -c 'bun run typecheck && bun test' > "$release/tests.log" 2>&1
+docker_cmd build -t "$image" "$release" > "$release/build-runtime.log" 2>&1
+docker_cmd run --rm "$image" bun scripts/api-smoke.ts > "$release/smoke.log" 2>&1
+docker_cmd image rm "$checks" > /dev/null
+
+previous_image=$(sed -n 's/^APP_IMAGE=//p' .env)
+[[ -n "$previous_image" ]]
+awk -v image="$image" '/^APP_IMAGE=/ {$0="APP_IMAGE=" image} {print}' .env > next.env
+docker_cmd compose --project-directory "$root" -f "$release/compose.yaml" --env-file "$root/next.env" config --quiet
+caddy_image=$(sed -n 's/^CADDY_IMAGE=//p' .env)
+docker_cmd run --rm -v "$release/Caddyfile:/etc/caddy/Caddyfile:ro" "$caddy_image" caddy validate --config /etc/caddy/Caddyfile > "$release/caddy-check.log" 2>&1
+
+rollback="$release/previous"
+mkdir -p "$rollback"
+cp .env compose.yaml Caddyfile "$rollback/"
+echo "Switching to $target"
+# Prevent user writes during migration and never overlap Discord workers.
+# Obtain the fingerprint before stopping so failure here leaves production running.
+schema_before=$(schema_fingerprint "$previous_image")
+switching=true
+compose stop caddy app
+compose run --rm --no-deps --interactive=false --entrypoint bun app scripts/backup.ts > "$rollback/backup-path.txt"
+cp "$release/compose.yaml" compose.yaml
+cp "$release/Caddyfile" Caddyfile
+mv next.env .env
+compose up -d --wait --wait-timeout 120 app monitor
+compose up -d --wait --wait-timeout 120 caddy
+curl --fail --silent --show-error --retry 5 --retry-all-errors --retry-delay 2 --max-time 15 https://mina.teppelinlabs.com/api/health > "$release/public-health.json"
+printf '%s\n' "$target" > deployed-sha.tmp
+mv deployed-sha.tmp deployed-sha
+rm -f failed-sha
+docker_cmd image inspect "$image" --format '{{.Id}}' > release-image-id
+switching=false
+echo "Deployed $target"
--- a/deploy/deploy.test.ts
+++ b/deploy/deploy.test.ts
@@ -0,0 +1,92 @@
+import { expect, test } from 'bun:test';
+import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, existsSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join, dirname } from 'node:path';
+
+const revision = 'b'.repeat(40);
+const stub = `#!${process.execPath}
+import {basename} from 'node:path';
+import {appendFileSync,mkdirSync,existsSync,writeFileSync} from 'node:fs';
+const tool=basename(process.argv[1]), args=process.argv.slice(2), root=process.env.STUB_ROOT, mode=process.env.STUB_MODE;
+appendFileSync(root+'/calls.jsonl',JSON.stringify({tool,args})+'\\n');
+if(tool==='flock') process.exit(0);
+if(tool==='timeout') { const r=Bun.spawnSync(args.slice(1),{env:process.env});process.stdout.write(r.stdout);process.stderr.write(r.stderr);process.exit(r.exitCode); }
+if(tool==='git') {
+ if(args.includes('init')) mkdirSync(args.at(-1),{recursive:true});
+ if(args.includes('rev-parse')) console.log('${revision}');
+ if(args.includes('archive')) { const r=Bun.spawnSync(['tar','-cf','-','-C',process.env.STUB_SOURCE,'.']);process.stdout.write(r.stdout);process.exit(r.exitCode); }
+ process.exit(0);
+}
+if(tool==='curl') { console.log('{"status":"ok"}');process.exit(0); }
+if(tool==='sudo') {
+ if(args.includes('build') && mode==='build-failure') process.exit(1);
+ if(args.some(a=>a.includes('SELECT type,name,sql'))) { console.log(mode==='migration-failure' && existsSync(root+'/switched')?'schema-new':'schema-old');process.exit(0); }
+ if(args.includes('compose') && args.includes('up') && args.includes('app')) {
+  writeFileSync(root+'/switched','yes');
+  if(['switch-failure','migration-failure'].includes(mode)) process.exit(1);
+ }
+ if(args.includes('scripts/backup.ts')) console.log('/data/backups/safety.sqlite');
+ if(args.includes('inspect')) console.log('sha256:test-image');
+ process.exit(0);
+}
+process.exit(2);
+`;
+
+async function scenario(mode: string, initial?: 'unchanged' | 'failed') {
+  const root = mkdtempSync(join(tmpdir(), 'minabot-deploy-test-'));
+  const bin = join(root, 'bin'), source = join(root, 'source');
+  mkdirSync(bin); mkdirSync(source);
+  for (const name of ['git', 'sudo', 'timeout', 'flock', 'curl']) writeFileSync(join(bin, name), stub, { mode: 0o755 });
+  writeFileSync(join(root, '.env'), 'APP_IMAGE=minabot:previous\nCADDY_IMAGE=caddy:pinned\n');
+  writeFileSync(join(root, '.env.production'), 'SECRET=must-stay-server-side\n');
+  writeFileSync(join(root, 'compose.yaml'), 'old-compose');
+  writeFileSync(join(root, 'Caddyfile'), 'old-caddy');
+  writeFileSync(join(root, 'deployed-sha'), initial === 'unchanged' ? revision : 'a'.repeat(40));
+  if (initial === 'failed') writeFileSync(join(root, 'failed-sha'), revision);
+  writeFileSync(join(source, 'compose.yaml'), 'new-compose');
+  writeFileSync(join(source, 'Caddyfile'), 'new-caddy');
+  try {
+    const child = Bun.spawn(['bash', new URL('./deploy.sh', import.meta.url).pathname], {
+      env: { ...process.env, PATH: `${bin}:${dirname(process.execPath)}:${process.env.PATH}`, MINABOT_DEPLOY_ROOT: root, STUB_ROOT: root, STUB_SOURCE: source, STUB_MODE: mode },
+      stdout: 'pipe', stderr: 'pipe',
+    });
+    const [code, stdout, stderr] = await Promise.all([child.exited, new Response(child.stdout).text(), new Response(child.stderr).text()]);
+    const calls = readFileSync(join(root, 'calls.jsonl'), 'utf8').trim().split('\n').map(line => JSON.parse(line));
+    return { code, stdout, stderr, calls, env: readFileSync(join(root, '.env'), 'utf8'), secret: readFileSync(join(root, '.env.production'), 'utf8'), deployed: readFileSync(join(root, 'deployed-sha'), 'utf8').trim(), failed: existsSync(join(root, 'failed-sha')) };
+  } finally { rmSync(root, { recursive: true, force: true }); }
+}
+
+test('deploys tested main, preserves secrets, and stops old workers before starting new ones', async () => {
+  const r = await scenario('success');
+  expect(r.code).toBe(0); expect(r.deployed).toBe(revision);
+  expect(r.env).toContain('APP_IMAGE=minabot:git-bbbbbbbbbbbb');
+  expect(r.secret).toBe('SECRET=must-stay-server-side\n');
+  const stop = r.calls.findIndex(c => c.tool === 'sudo' && c.args.includes('stop'));
+  const up = r.calls.findIndex(c => c.tool === 'sudo' && c.args.includes('up'));
+  expect(stop).toBeGreaterThan(0); expect(up).toBeGreaterThan(stop);
+});
+test('unchanged and previously failed revisions do no Docker work', async () => {
+  for (const initial of ['unchanged', 'failed'] as const) {
+    const r = await scenario('success', initial);
+    expect(r.code).toBe(0); expect(r.calls.some(c => c.tool === 'sudo')).toBe(false);
+  }
+});
+test('build failure leaves the current deployment running', async () => {
+  const r = await scenario('build-failure');
+  expect(r.code).not.toBe(0); expect(r.failed).toBe(true);
+  expect(r.env).toContain('APP_IMAGE=minabot:previous');
+  expect(r.calls.some(c => c.args.includes('stop'))).toBe(false);
+});
+test('startup failure rolls back configuration without restoring the database', async () => {
+  const r = await scenario('switch-failure');
+  expect(r.code).not.toBe(0); expect(r.env).toContain('APP_IMAGE=minabot:previous');
+  expect(r.stderr).toContain('database was not replaced');
+  expect(r.deployed).toBe('a'.repeat(40));
+});
+test('a changed migration fingerprint stops instead of overwriting production data', async () => {
+  const r = await scenario('migration-failure');
+  expect(r.code).not.toBe(0); expect(r.failed).toBe(true);
+  expect(r.stderr).toContain('No database was overwritten');
+  expect(r.deployed).toBe('a'.repeat(40));
+  expect(r.calls.filter(c => c.tool === 'sudo' && c.args.includes('up'))).toHaveLength(1);
+});
--- a/deploy/minabot-deploy.service
+++ b/deploy/minabot-deploy.service
@@ -0,0 +1,13 @@
+[Unit]
+Description=Test and deploy Minabot main from Gitea
+Wants=network-online.target
+After=network-online.target docker.service
+Requires=docker.service
+
+[Service]
+Type=oneshot
+User=syntaxbullet
+WorkingDirectory=/opt/minabot
+ExecStart=/usr/local/bin/minabot-deploy
+TimeoutStartSec=30min
+UMask=0077
--- a/deploy/minabot-deploy.timer
+++ b/deploy/minabot-deploy.timer
@@ -0,0 +1,11 @@
+[Unit]
+Description=Check Gitea main for Minabot updates every minute
+
+[Timer]
+OnBootSec=60s
+OnUnitInactiveSec=60s
+AccuracySec=5s
+Unit=minabot-deploy.service
+
+[Install]
+WantedBy=timers.target
--- a/docs/PRODUCTION.md
+++ b/docs/PRODUCTION.md
@@ -6,6 +6,11 @@ SSH: `ssh syntaxbullet@46.62.156.93` (key authentication; passwordless sudo).
 Deployment root: `/opt/minabot`. Compose manages `app`, `caddy`, and `monitor`.
 Docker starts at boot; containers use `restart: unless-stopped`.

+New pushes to Gitea `main` are now checked and deployed by a systemd timer.
+See [automatic deployment](../deploy/README.md) for checks, rollback, and controls.
+The current deployed commit is `/opt/minabot/deployed-sha`; the initial image below
+is the baseline before automatic deployment was enabled.
+
 ## Release and configuration

 Current image: `minabot:20260907-fonts-1`, built on the Linux amd64 server from