Operations

Health checks

curl -fsS https://account.example.com/healthz

{
  "status": "ok",
  "service": "universal-gaming-accounts",
  "version": "1.4.0",
  "database": "postgres"
}

The endpoint performs a database query. A failure returns a non-2xx status.

Docker health check:

docker compose ps
docker inspect --format '{{json .State.Health}}' \
  "$(docker compose ps -q accounts)"

Logs

docker compose logs -f accounts
docker compose logs --tail=200 db

Request log format

[http] 200 GET /api/v1/leaderboards 12.4ms ip=203.0.113.10 peer=10.0.0.2 ua="UniversalGaming/2.9.0"

• Query strings are stripped.
• Bodies, tokens, passwords and e-mail addresses are never logged.
• ip is the resolved client IP; peer is the direct socket peer.
• ua is JSON-escaped and length-limited.

Failed logins

[auth] login failed ip=203.0.113.10

Only the IP is logged.

Database

[db] postgres connected
[db] migration 2 (password reset tokens and session versioning) applied

PostgreSQL also logs connections and disconnections because Compose starts it with log_connections=on and log_disconnections=on.

Mail

[mail] SMTP host=mail.example.com port=587 secure=false requireTLS=true

If you see SMTP_SECURE=true is not valid for STARTTLS port 587, the mode was auto-corrected. Fix .env to remove the warning.

Management panel

Enabled when MANAGEMENT_PASSWORD is non-empty:

https://account.example.com/management

• HTTP Basic auth.
• Username MANAGEMENT_USER (default admin).
• Read-only: account, score and match counts plus recent activity.
• Cache-Control: no-store.

Test it:

curl -u admin:"$(cat /root/.universal-gaming-management-password)" \
  https://account.example.com/management | head

Keep the password in a root-only file (chmod 600) outside the repository.

SMTP verification

docker compose exec accounts node -e "
  import('./src/config.js').then(async ({loadConfig}) => {
    const {createMailer} = await import('./src/mailer.js');
    const m = createMailer(loadConfig());
    if (!m.enabled) throw new Error('SMTP is not configured');
    await m.verify();
    console.log('SMTP verify ok');
  });
"

Then test a real flow:

1. Register an account with an e-mail.
2. Confirm the address from the mail.
3. Request a password reset.
4. Open the reset link and choose a new password.
5. Confirm old sessions now return 401.

Backups

Automated pre-update backup

./update.sh writes:

backups/pre-update-YYYYMMDDTHHMMSSZ.dump

Scheduled backup

Run from cron or a systemd timer:

#!/usr/bin/env bash
set -euo pipefail
cd /srv/universal-gaming-backend
mkdir -p backups
out="backups/scheduled-$(date -u +%Y%m%dT%H%M%SZ).dump"
docker compose exec -T db \
  sh -c 'pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" -Fc' > "$out"
chmod 600 "$out"
find backups -name 'scheduled-*.dump' -mtime +30 -delete

Copy dumps off-host. A backup on the same disk does not survive disk loss.

Verify a dump

pg_restore --list backups/pre-update-YYYYMMDDTHHMMSSZ.dump | head

Restore

docker compose stop accounts

docker compose exec -T db \
  sh -c 'pg_restore -U "$POSTGRES_USER" -d "$POSTGRES_DB" --clean --if-exists' \
  < backups/pre-update-YYYYMMDDTHHMMSSZ.dump

docker compose start accounts
curl -fsS http://127.0.0.1:6767/healthz

After restoring, verify:

• A known account can sign in.
• Leaderboard rows are present.
• Chess matches load.
• schema_migrations contains the expected versions.

Data durability

• Base Compose uses the named volume postgres-data.
• Production may use a bind mount through a gitignored docker-compose.override.yml.
• Never delete the data directory or volume. It contains all accounts, scores and matches.
• Changing POSTGRES_PASSWORD does not change an existing role. Use ALTER ROLE.
• Keep the database and backups on a monitored disk.

Disk and images

df -h /
docker system df
docker image prune -f

Old application images accumulate after each release. Prune them after confirming the current deployment is healthy. Do not prune volumes.

Monitoring checklist

• /healthz returns ok and the expected version.
• docker compose ps shows healthy services.
• Request logs show real client IPs.
• Failed-login volume is normal.
• SMTP verify succeeds.
• Disk usage is below the alert threshold.
• Latest backup exists and is readable.
• Backups are copied off-host.
• schema_migrations matches the release.
• Management panel is reachable only as intended.

Graceful shutdown

The server handles SIGINT and SIGTERM:

1. Stops accepting connections.
2. Closes the database pool.
3. Exits after at most 10 seconds.

Compose sets stop_grace_period: 15s and init: true.

Incident checklist

1. Check /healthz, container status and logs.
2. Do not delete the database volume.
3. Take a fresh dump before changing anything.
4. Roll back the image or restore the pre-update dump.
5. Rotate JWT_SECRET if session integrity is in doubt; this signs everyone out.
6. Rotate SMTP and management passwords if they may have leaked.
7. Verify account access, leaderboard data and chess matches after recovery.