Forwarding to an external judge

AlgoJudge-External-Runner is a second Runner implementation that evaluates nothing. It claims jobs of an external problem type, hands the solution to the judging system that owns that type, waits for that system to decide, and reports its verdict back to the installation.

It runs no code, has no sandbox, and measures nothing.

In an installation it is the external-runner profile, outside the default set because it signs in to that judging system under an account there. An external judge is the operator's side of this page: what to put in .env, and the two switches that leave its queue empty while everything looks healthy.

One judging system is built into it: UVa Online Judge, serving the problem type uva@1 against onlinejudge.org. Everything below that is not marked as UVa's applies to the Runner whichever judge it is pointed at.

How it differs from the sandboxing Runner

Because it starts nothing, it needs no socket, no cgroups and no scratch directory. The image is smaller for that reason rather than by tuning.

There is no package cache, and there is a source cache — the two are worth keeping apart. An external problem has no package, because its whole configuration travels on the job; but the participant's own file is fetched on every job and needs somewhere to live. Conflating the two shipped an image without the cache directory, and every job failed on it.

So there are two directories, and they differ in what losing them costs. The identity key is in /var/lib/algojudge-external-runner and is meant to be a volume: losing it costs a re-registration and an administrator's approval. The source cache is in /var/cache/algojudge-external-runner, and losing it costs one download.

Stopping one, and what it costs the archive

It hands back every job it is holding, exactly as the sandboxing Runner hands back its one. On SIGTERM it stops asking for work, tells the Server each held job is free, and exits; each of those is claimable immediately and none of them spends a delivery. See Runners for what that means on the Server's side, which is the same for both Runners.

What is different is what it cannot take back. The submission is already on somebody else's service under this installation's account there, and the set of what is outstanding lives in memory. So the answer that archive is still computing arrives with nowhere to land, and whoever claims the job next sends the same solution again. That cost is not new — it is what any restart has always cost — and it is not avoided by leaving the Runner to be killed. What a polite stop saves is the waiting: without it every held job sits out its lease first, up to ten minutes each, on a Runner that is already gone.

So give it longer than the sandboxing one. Twenty jobs is twenty calls to the Server, and a grace period shorter than they take turns a stop back into a kill. AlgoJudge-Ops ships stop_grace_period: 60s for this service against 30s for the other.

external: true is not a detail

The Server pairs a problem with a Runner on two booleans — the problem's external flag and the Runner's — by equality, in both directions.

• An external problem goes only to a Runner that forwards.
• A local problem goes only to a Runner that does not. That half is the one worth stating: a local problem reaching a forwarding Runner would send somebody's work out of the building with nobody having chosen that.

A Runner that forwards and does not declare it is handed nothing at all, and from a log that is indistinguishable from an empty queue. It is one of the three filters on routing; the Server reads no problem type to decide any of them.

The two switches that stop the queue while everything looks healthy

Neither produces an error. Both produce an empty queue.

One process, one judge

The configuration below holds one endpoint and one account. Two judging systems are two deployments of this Runner, each with its own problem types and its own pools — which is how Runners already scale, and the same answer as running two sandboxing Runners on two machines.

Configuration

The same AJ_ convention as everywhere else, in two halves. AJ_Server__*, AJ_Runner__* and AJ_Lease__* are the Runner's own and mean the same thing whatever it forwards to; AJ_External__* is the judging system it forwards to. Two values are secrets and have no defaults.

The two addresses both default to UVa's, because uva is the judge the Runner selects when nothing says otherwise. They are settings rather than constants so that a judge which moves does not need a release.

AJ_External__PollMinSeconds cannot go below twenty. The Runner refuses to start if it does: onlinejudge.org publishes no rate limit at all, and a judging system may publish none, so this one is not lowered. A politeness limit a deployment can silently disable is not a limit.

Every submission is made under one robot account, and every real submission stays on that account for ever. That is a decision for whoever owns it.

Three numbers checked against each other at start-up

All three failures happen far from their cause, and all three end the same way: the same solution is sent to somebody else's site twice.

AJ_Lease__RequestSeconds may not exceed 3600. The Server clamps what it grants, so a larger request is a deadline of this Runner's own invention: it would renew against a lease it does not have and hold a job past the real one. This is checked first, because it bounds the variable the next rule tells you to raise — pushing AJ_External__PendingTimeoutSeconds up eventually leaves no legal lease above it, and the refusal says so.

AJ_Lease__RequestSeconds must exceed AJ_External__PendingTimeoutSeconds. Otherwise the Server reclaims the job while this Runner is still waiting on the judge, the job returns to the queue, and the next Runner to claim it submits again. The defaults — a twenty-minute lease against a fifteen-minute wait — satisfy this; a lease left at the sandboxing Runner's ten-minute default would not.

AJ_External__PollMaxSeconds must fit four times inside the lease. A held lease is renewed at the top of the same cycle that asks the judge, so slowing the polling down to be polite to somebody else's service slows the renewing down with it. Four, so that three renewals fit inside every lease and two may fail in a row with the deadline still ahead.

The cycle is the judge's interval plus the claim. A lease is renewed once per loop, and the loop's other blocking call is the claim the Server may hold open for AJ_Poll__WaitSeconds — so both waits are inside one renewal gap, and what has to fit four times inside the lease is AJ_External__PollMaxSeconds + AJ_Poll__WaitSeconds.

That last one is the well-meant change that would have cost a double submission, which is exactly why it is refused at start-up rather than discovered an hour later. Note the ceiling it implies: with a lease clamped at 3600 seconds the cycle cannot exceed 900, so the poll interval cannot exceed 875 at the shipped wait — 900 only if the wait is set to 0.

These are the three an operator meets while tuning. The start-up check refuses seven configurations in all — the other four are a poll floor below twenty, a poll ceiling below the floor, AJ_External__MaxPending at zero, which would claim no job at all, and an AJ_Poll__WaitSeconds above the three hundred the Server will hold.

UVa Online Judge

The one integration there is. The two headings under this one are its own; the sections after them are the Runner's again.

It is two services rather than one: submissions are posted to onlinejudge.org as a web form, and verdicts are read from uHunt as an API. That is why the configuration above has both a BaseUrl and an ApiBaseUrl.

The six languages

The archive's own list, defined by the problem type rather than by each problem.

Three of these ids are the same ids standard-io@1 uses — deliberately, so one screen can resolve a label whichever type produced a submission. The three that are not name toolchains this project does not run itself; that is the point of forwarding.

The labels are not standard-io@1's, and must not be. cpp11-gcc here is GCC 5.3.0 with the archive's flags, and in the sandboxing Runner it is a compiler this project pins with its own. Showing the same label in both places would tell a participant the two were built by the same compiler.

What a uva@1 problem carries

There is no package. The whole of the configuration is documents the Server already carries and does not read:

• the version's props says which problem this is — the archive's number. Identity rather than settings, written once at import, inherited by every assignment. Without it the Runner refuses the job before anything leaves, and says so by name.
• the assignment's config says how this course judges it — which of the archive's verdicts count as solved, and which of the six languages are allowed. A course can accept a presentation error where a contest counts only an accepted answer, on the same imported problem.

Statements are fetched through the Server rather than by the manager's browser, because the archive sends no cross-origin header. That fetch is governed by an allowlist of hosts the installation configures.

Not judged, and not a wrong answer

A judge can fail to form an opinion — a submission error, or a problem it holds no tests for. Neither is a verdict and neither is scored. They are reported as infrastructure failures, and the Server refuses to score one, because marking somebody down for a third party's outage would be a lie about their solution.

A verdict the Runner does not recognise is treated as not judged yet rather than guessed at: an answer nobody has seen before is not evidence that a person was wrong.

A submission the activity's rules refuse — a language the manager excluded — is reported as PolicyViolation, the same word standard-io@1 uses, and nothing is sent to the judge at all.

How often the judge is asked, and the stream that runs anyway

The live stream is not a setting. Whenever anything is outstanding, this Runner holds a request open against the archive's event stream and asks for answers the moment an event about its own account arrives. Nothing turns that off. AJ_External__LongPollEnabled decides one thing: how often the judge is asked when no event has arrived.

So turning it on slows the safety net down — one request a minute instead of three — on the promise that the stream is what makes a verdict prompt. Measured against onlinejudge.org on 2026-09-08, a verdict came back in about 25 s either way: the archive's own judging is what that time is, and what the setting buys is the requests.

Which is why it is off unless an operator turns it on. Behind a proxy that cuts a long-held request the stream never fires, the flat minute is all that is left, and a verdict then waits a minute rather than twenty seconds. It is a value rather than a presence: true, 1, yes or on against false, 0, no or off, and anything else is refused at start-up by name.

The stream is a trigger and never a source of truth, whatever this is set to. It is global to the judge and keeps only its last hundred events, so it can drop one under load; what arrives is read for one thing — whether this account appears — and never for a verdict, because a missed event read as a verdict would be a submission that hangs until its timeout. The position in it is taken before the first submission of a batch, when nothing is outstanding and nothing can be lost by moving it.

Two more properties of the same loop: the archive is asked nothing at all while nothing is outstanding, and one request covers every outstanding submission, however many there are.

Only the UVa integration has a stream. An integration without one waits out the interval instead, and this setting then decides only how long that is.