Benchmarks

One real measurement, disclosed in full, instead of a "hundreds of milliseconds" hand-wave. Run the same commands yourself — nothing here depends on trusting us.

Cold start: sandbin vs Docker

Same machine, same moment, sequential runs, image pre-pulled for both so neither pays a network tax:

17.6× faster by median, 17.6× by mean — the distributions don't overlap at all; the slowest sandbin run (27 ms) still beat the fastest Docker run (343 ms) by more than 12×.

Method

Workload: run python3 -c "print(1)" and measure until the process exits.

# sandbin — via run() directly, durationMs from the result
{ language: 'python', code: 'print(1)' }

# Docker — wall-clock around the whole command
docker run --rm python:3.12-alpine python3 -c "print(1)"

20 sequential runs each, back to back, on the same otherwise-idle machine. No warm-up runs excluded — the numbers above include whatever cold-cache effects a real first request would also hit.

Environment

Raw samples

sandbin (ms), sorted:

13, 14, 17, 18, 20, 20, 20, 20, 20, 21, 21, 22, 22, 22, 23, 23, 24, 25, 26, 27

Docker (ms), sorted:

343, 347, 348, 349, 350, 356, 359, 359, 362, 363, 369, 369, 369, 377, 381, 382, 387, 393, 399, 404

What this does and doesn't show

This measures exactly one thing: the latency of going from "here is some code" to "it's running," for the smallest possible workload, on one machine, once. It does not measure sustained throughput or Docker's much larger ecosystem and orchestration tooling — sandbin's own concurrency behavior is measured separately, below, not here. The gap exists because Docker pays for a daemon round-trip, image layer resolution and full container lifecycle machinery that sandbin's direct bwrap + cgroup approach simply doesn't have to.

Reproduce it

git clone https://github.com/ayazdoruck/sandbin.git
cd sandbin
npm install

# sandbin
node -e "
import('./src/sandbox.mjs').then(async ({run}) => {
  const t = [];
  for (let i = 0; i < 20; i++) t.push((await run({ language: 'python', code: 'print(1)' })).durationMs);
  console.log(t.sort((a,b)=>a-b));
});"

# Docker, for comparison
docker pull python:3.12-alpine
for i in $(seq 1 20); do
  s=$(date +%s%N)
  docker run --rm python:3.12-alpine python3 -c "print(1)" >/dev/null
  e=$(date +%s%N)
  echo $(( (e - s) / 1000000 ))
done

Concurrency and throughput

The cold-start numbers above are one request at a time. This is the queue under real concurrent load instead — many requests arriving together, against a real spawned sandbox pipeline on an ephemeral port, not a mock.

Sustained, at the default concurrency limit

100 requests, client concurrency held at 4 — the queue's own default maxConcurrency — so every request starts running as soon as a slot frees up and none of them ever sit in the backlog:

Stable across repeated runs: 119–132 req/s, p50 27–32 ms, p95 46–48 ms.

Overload, past capacity

60 requests fired at once, no client-side throttling, against a queue deliberately shrunk to a 20-slot capacity (maxConcurrency=4 + maxQueueLength=16) — specifically to force the global queue_full path rather than the per-key key_limit one, isolated by raising maxPerKey so it can't interfere:

Exactly 20 accepted on every single run — the queue's own accounting is precise, not approximate. All 40 rejections landed as queue_full, never key_limit, confirming the two really are independent paths rather than one silently masking the other. And the rejections resolve in 36–62 ms — a caller finds out it was turned away about as fast as one that got accepted starts running, instead of being kept waiting only to be told no.

Method

Both scenarios spin up a real createServer() instance on an ephemeral port and drive it exactly the way a real client would: POST /runs, then one WebSocket connection per accepted run waiting for its finished message. Rate limiting is raised to a no-op ceiling for the run itself — every request here shares one loopback IP, which would otherwise trip the real 20/hour anonymous quota long before either scenario finished — since it's already covered on its own in the rate limit tests.

Reproduce it

git clone https://github.com/ayazdoruck/sandbin.git
cd sandbin
npm install
npm run loadtest