When something does not work

The Server will not start

The database has N pending migration(s)

The schema is behind the image and this Server has not been told it may fix that. Set MIGRATE_ON_START=true in .env — after taking a backup — or apply the migrations yourself before starting.

This is also what a fresh installation says if the switch is off: an empty database has every migration pending.

It exits complaining about Forwarded

TRUSTED_PROXY_NETWORKS is unset. The Server refuses to start without being told whose word to take for a visitor's address. none is a complete answer for a Server reached directly.

It names a network you did not write

You wrote a CIDR with host bits set — 172.28.0.5/24. .NET normalises that to 172.28.0.0/24 without a word, which turns "one machine" into "a whole laboratory", so the Server refuses and names the address you should have written. preflight.sh catches it first.

No storage is configured

STORAGE_KIND is empty or misspelled. It is postgres, filesystem or s3.

It names a storage setting that is empty

filesystem needs STORAGE_PATH; s3 needs STORAGE_ENDPOINT, STORAGE_BUCKET, STORAGE_ACCESS_KEY and STORAGE_SECRET_KEY, all four. The Server refuses at startup naming the one it wanted, and preflight.sh refuses before that. None of them is read while STORAGE_KIND is postgres, which is why they ship empty.

The API answers 404 and you are sure the path is right

/health is not the health endpoint. /api/v1/health is.

Every installation serves the API at /api/v1, and the Server answers 404 at the bare root on purpose — so that a Client pointed at the wrong path is corrected rather than answered. A monitor written against /health reports the installation down while it is serving perfectly.

/admin answers 404 with the right token

It is supposed to. /api/v1/admin/** needs the request to arrive on the Server's own loopback interface, and:

• through nginx it arrives as the bridge gateway — 404;
• through the published 127.0.0.1:8080 it also arrives as the bridge gateway, because that is what a published port does — 404.

Every refusal is the same 404, deliberately, so nothing distinguishes a wrong token from a wrong route. Not 403: a 403 would confirm the endpoint is there and that the caller got one of the two halves right.

The way in:

docker compose exec -T server aj-admin status

If that says AJ_Admin__Token is not set in this container, the token was empty when the container started. Set it in .env and recreate the Server.

Nobody can administer the installation

Restart the Server. At every start it looks for an active installation-wide grant carrying system:administrator — anybody's, not only the seeded account's — and where the installation has none, it gives the admin account the admin template back:

docker compose restart server

The log then says Nobody administered this installation, so admin has been granted system:administrator again. Its password is unchanged. That last clause is the whole of the caveat: a restart returns the rights, not the password. If nobody knows that either, set one with aj-admin password.

A grant somebody made by hand is never rewritten. Where admin already holds an installation-wide grant of its own that does not administer — trimmed to a shorter list, or set to invited — the row is left alone and the log says grant system:administrator by hand instead. That is the one case a restart does not repair.

Reaching this state at all now takes a database older than the refusal, or one edited directly: taking the installation's last system:administrator away — revoking the grant, rewriting it without the key, or demoting it to invited — is answered 403 with the code grant.administrator.last. aj-admin has no command for grants, which is why the door is a restart rather than a command.

Nothing is being judged

In order of how often it is each one:

1. The Runner has not been approved. Its log says waiting: this Runner has not been approved yet, and it will say that for ever. Approve it in the panel under Runners.
2. The sandbox images do not exist. Compiled in, the Runner looks for algojudge/lang-*:local, which only a development host has. compose.yaml sets all four AJ_Sandbox__Image__* variables from REGISTRY and RUNNER_TAG; if you overrode them, check docker images.
3. RUNNER_WORK_DIR is wrong. This is the nastiest one, because it fails silently: the Docker daemon is handed this path directly, and a path it cannot open produces an empty directory rather than an error. Every job then runs against nothing. It must be absolute, and it must be a path the daemon — not only your shell — can open.
4. Tags. A Runner with RUNNER_TAGS set is out of the general pool, and work with no tags goes to the general pool. Empty means the general pool on both sides. Tags are read once, at the first registration — changing the variable later does nothing, and re-registering is what changes them.

Every job fails with carries no /usr/local/bin/aj-shim

The Runner says it of one of the four language images, and means it: aj-shim is what makes a judged run measurable, so an image from before it existed cannot judge anything.

The host has an old copy. Two things keep that quiet, and both are by design: docker compose pull does not fetch the language images — they are values the Runner is handed, not services — and the Runner pulls one only when the host has none at all, never because a newer one exists.

docker pull ghcr.io/algojudge/lang-gcc:0     # and clang, python, pypy
docker compose restart runner-1              # every runner-N you run

scripts/update.sh pulls all four, so an installation kept up to date with it does not meet this.

Nothing external is being judged

A different list, because the ordinary causes are not the causes here. The first two are on An external judge and leave the queue empty with the container looking healthy: external judging is off for this installation, and the external Runner has not been approved.

The other three:

1. The problem was not created as an external one. It has to be typed for the archive and marked external; a locally typed problem is never offered to a forwarding Runner.
2. The version's props does not name the archive's problem number. That is refused by name, before anything leaves this installation.
3. Tags, exactly as for any Runner, and read once at the first registration.

A Runner restarts every few seconds

restart: unless-stopped turns any refusal at startup into a loop, and preflight.sh cannot see most of these because nothing is wrong with the configuration. Read the log:

docker compose logs external-runner | tail -20
docker compose logs runner-1 | tail -5

The account at the judging system is empty or wrong. The external Runner refuses while reading its configuration — before the identity key, before registration — and names the setting. preflight.sh refuses ahead of it when the profile is on, so a stack brought up through the scripts never gets this far. A leading or trailing space in the password is not the cause: it is passed through exactly as written, deliberately, because trimming one broke a sign-in once.

the Server refused with 403: runner.nonce.unknown is a different fault with the same symptom, and it applies to both Runners: this image is older than the Server it is registering against. It appears only from the second start onwards, because the identity volume is what makes the Server recognise the key, and a refused registration is not one a Runner retries. Pull both images from the same release rather than letting the moving tags drift — Update and rollback says why they have to move together.

A named poll or lease setting is the third: a Runner checks its intervals against one another before it does anything. None of these can be reached with the values this stack ships; they are what a .env of your own can produce. AJ_Poll__WaitSeconds above 300 is refused because 300 is the Server's own ceiling, and a Runner tells a held claim from an immediate answer by how long it took — so it would read every held claim as an empty one. To poll less often, the setting is AJ_Poll__MaxSeconds.

The archive says the account does not exist

Check the username at the archive itself — it is the same one a person signs in with. The lookup refuses a user id of 0 by name, because the archive answers 0 for a username it does not know and 0 parses as a number: before that refusal existed, every poll silently returned nothing.

Permission denied (os error 13) in the Runner

Two different causes produce this one sentence, and it names neither: it comes from deep inside an HTTP client or the sandbox layer, with no path and no number in it. preflight.sh checks both.

One: the daemon's socket. DOCKER_GID does not match the group that owns it.

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock alpine stat -c '%g' /var/run/docker.sock

On Docker Desktop that is 0; on a Linux host it is the docker group. preflight.sh runs exactly this and warns if they disagree — asked of a container rather than of your shell, because under Docker Desktop and WSL the socket this shell can see is not the one the daemon serves. Since the runner service runs as root this is only a cause where compose.yaml has been edited to drop that, and then the Runner fails at startup and never reaches a job.

Two: a job container cannot read RUNNER_WORK_DIR. The Runner writes a submission's files there as root, and every job container mounts the directory read-only and reads it as uid 65534. A directory locked down by hand passes for the Runner and fails for the job, so the socket works perfectly, the Runner registers, claims a job — and every single one fails at once, which is how to tell the two apart.

sudo chmod 755 /srv/algojudge/runner-work

Every job fails, and the Runner said something about cgroups

A time limit is decided on processor time, read from a cgroup the sandbox is started under. A Runner that cannot read one refuses to judge without it, so this is a startup message rather than a per-job one:

docker compose logs runner-1 | head -40

Both cgroup drivers work, and neither needs the daemon reconfigured. A native.cgroupdriver line in /etc/docker/daemon.json does no harm and buys nothing.

The verdicts are right but no memory is reported

Only on a host using the systemd cgroup driver, and the Runner says so at ERROR on every start. One slice serves every run there, so a peak is taken by resetting memory.peak — a kernel interface that arrived in Linux 6.12, which excludes Ubuntu 24.04 and includes Debian 13. A time limit is processor time, so every verdict stands, and a judged submission's own peak is unaffected: that one is read from a cgroup made fresh for the run. What is missing is the peak of the runs that are nobody's submission — a build, a checker, an interactor.

uname -r

On an older kernel, either accept it or give the daemon the cgroupfs driver, where every peak comes from a fresh cgroup and needs nothing past Linux 5.19:

// /etc/docker/daemon.json
{ "exec-opts": ["native.cgroupdriver=cgroupfs"] }

nginx will not start

Run the same test CI runs:

docker run --rm --add-host server:127.0.0.1 --add-host client:127.0.0.1 \
  -v "$PWD/nginx/algojudge.conf:/etc/nginx/conf.d/default.conf:ro" \
  -v "$PWD/nginx/snippets:/etc/nginx/snippets:ro" \
  -v "$PWD/certs:/etc/nginx/certs:ro" \
  nginx:1.30-alpine nginx -t

--add-host is needed because nginx resolves every upstream while parsing, so outside the Compose network it fails with host not found in upstream "server:8080" on a configuration that is perfectly correct.

Two real ones seen here: a duplicate proxy_read_timeout, because include puts a directive in the including block and nginx refuses duplicates outright rather than overriding; and a missing certificate, because certs/ was empty.

PostgreSQL will not start after an upgrade

18 moved where the data lives. PGDATA is /var/lib/postgresql/18/docker and the volume belongs one level up at /var/lib/postgresql. Every guide written before 18 says /var/lib/postgresql/data, and mounting that makes the container refuse to start.

compose.yaml gets it right. The major is pinned on purpose — an unpinned postgres:latest rolling over to 18 is how this was found.

docker compose ps shows no health for a Runner

Both Runners show an empty health column, for ever, and neither is a fault. The Runner's image declares no health check, and the external Runner's cannot have one — it has no shell, no port and nothing to ask. docker compose up --wait therefore calls both ready as soon as they are running, which they are even while one of them is failing to sign in. Read their logs instead.

The stack is up but the browser shows nothing

• A certificate warning is expected if you ran render-tls.sh: it is self-signed.
• A blank page and 404s for /assets/…: the Client image was replaced without the Server, or a proxy is caching index.html. It is served no-store for exactly this.
• The maintenance page: the Server is in a window. Run ./scripts/maintenance.sh status, and off if it should not be.

A script says another one is running

another AlgoJudge maintenance script is running (pid 1234, lock: …/state/algojudge.lock.d)

If pid 1234 is genuinely running, wait. If it is not, the next run takes the lock over by itself and says so — a lock is never permanently stuck. To clear one by hand, remove state/algojudge.lock.d.

The backup is not a complete backup

backup.sh warns on every run when STORAGE_KIND is not postgres, and the .meta beside each dump records it as coverage=INCOMPLETE. Restoring such a dump alone gives an installation whose rows point at bytes that are not there — which presents as a 503 on every download, with nothing naming the cause.

A restore appeared to work and changed nothing

Read state/restore.log.

This exact failure happened while the deployment repository was being written: a Compose invocation failed, its one-line error was reported as a pg_restore warning, and the restore did nothing at all while every check afterwards passed on data that had never been touched at all.

restore.sh takes the identity of every table before and compares it after: --clean --if-exists gives every restored table a new one, and a restore that did nothing leaves the old ones where they were. But the real test is the one that caught it: change something first, then restore, then check the change is gone.