Testing
Every case is real code trying to break out, checked against evidence the guest can't fake — not mocks, and not the guest's own claims about what happened.
The rule: evidence, not narration
A guest process can print whatever it wants and exit with any code it likes. None of that is trusted. Every adversarial case asserts on something the guest doesn't control:
- cgroup counters —
pids.events'smaxfield,memory.peak,oom_kill— read directly from the kernel, not from anything the guest reported - host-side filesystem checks — did a file the guest claims to have written actually appear outside its tmpfs?
- the specific errno a blocked syscall returned, via a raw
ctypes/libc call rather than a higher-level wrapper that might swallow or reinterpret it - real wall-clock gaps between events, for anything claiming to stream incrementally rather than dump output at the end
Two concrete examples where this distinction actually mattered:
A classic :(){ :|:& };: exits in about 15 ms with
pidsMaxHits: 0 — the shell is PID 1 in its own
namespace, so it exits and the kernel reaps the whole bomb with it
before the process-count ceiling is ever touched. A sustained variant
(:(){ :|:& };: ; sleep 10) keeps the parent alive and
shows pidsMaxHits in the dozens. Both are correctly
contained; asserting on exit code alone would have called the first
one a false negative.
The Python output-buffering bug (see Languages)
was only visible because the test recorded the actual arrival
timestamp of each WebSocket chunk message and asserted a
real gap existed between them. A test that only checked the final
concatenated output would have passed either way — buffered or
truly streamed, the end result looks identical.
Eight suites
| Suite | Command | Covers |
|---|---|---|
| Sandbox | npm run test:sandbox | 40 adversarial cases against real spawned sandboxes: namespace escapes, resource ceilings, seccomp denials, per-language checks, malformed-limits injection attempts, both directions of the limits clamp (4 of the 40 are Go's, and only run on a host with a Go toolchain) |
| Queue | npm run test:queue | 6 cases against real spawned runs: bounded concurrency observed under load, backpressure firing exactly at capacity, a crashing job still freeing its slot |
| Permalinks | npm run test:permalinks | 5 cases against a real temp directory: save/load roundtrip, TTL expiry and sweep driven by an injected clock, not a real 30-day wait |
| Rate limit | npm run test:ratelimit | 6 cases against the fixed-window counter directly: independent ids, window reset driven by an injected clock, sweep removing only buckets past their own reset — proven by an exact bucket-count check, not just by re-peeking the id |
| API keys | npm run test:apikeys | 7 cases against the on-disk key store: issuance shape, custom quotas, unknown-key handling, TTL expiry enforced at load time (not just by the periodic sweep), fully clock-injected rather than racing real disk I/O against the TTL |
| Metrics | npm run test:metrics | 6 cases against the in-memory counters directly: zeroed start, independent accepted/rejected/finished tallies, real mean averages, uptime advancing with wall-clock time |
| Server | npm run test:server | 28 cases against a real HTTP+WebSocket server on an ephemeral port: incremental delivery, live resource stats, permalink persistence with no save/read race, API key issuance and usage, per-tier rate limiting, interactive stdin, backpressure over HTTP, reconnect-after-finish replay, metrics reflecting real runs, metrics Basic Auth, a non-conforming id rejected even when a real file exists under that name, CSRF-shaped requests rejected, a heartbeat-reaped zombie WebSocket, maxPerKey immune to header rotation |
| CLI | npm run test:cli | 10 cases spawning the real bin/sandbin.mjs binary as a subprocess, both running locally and against a real server |
npm test runs all eight in sequence — the same command CI runs on every push, with a 5-minute job timeout so a genuine hang fails fast instead of consuming CI budget silently.
Run it yourself
git clone https://github.com/ayazdoruck/sandbin.git
cd sandbin
npm install
npm test