Skip to content
Hariom Pandey
Back to projects

Completed, 2026 · Designed and built the system

Koder

Online judge executing untrusted JavaScript, Java, and Python in isolated Docker sandboxes.

A full-stack online judge for function-signature algorithmic problems. Authenticated users submit code evaluated across hidden test batches by language-specific BullMQ workers running hardened Docker containers with tight resource and watchdog caps.

  • TypeScript
  • Next.js
  • Express
  • MongoDB
  • Redis
  • BullMQ
  • Docker
Source code for Koder (opens in new tab)

Problem & context

Online judges must execute arbitrary, untrusted user code without compromising host security or blocking web services. Evaluating submissions directly in request-response cycles risks Denial of Service from infinite loops, malicious syscalls, network exfiltration, and memory exhaustion. The platform requires asynchronous job dispatching, non-root sandbox isolation, strict CPU and memory cgroups, multi-layer watchdogs, and deterministic verdict evaluation.

Target users

Developers solving algorithmic challenges and competing in timed contests, alongside administrators managing question sets, reference test suites, and contest leaderboards.

My contribution

Architected and implemented the entire platform: Express REST API, shared @koder/shared workspace contract, language-specific BullMQ workers, hardened DockerSandbox container isolation, Base64 Line Streaming Protocol (BLSP), multi-tier execution watchdogs, and MongoDB persistence.

Core capabilities

  • Function-signature problem judging for JavaScript (Node 20), Java (Temurin 17 JDK), and Python (3.11)
  • Cookie-authenticated REST API under /api/v1 with role-based admin endpoints for problem curation and contest management
  • Dedicated BullMQ worker queues per language (js-queue, java-queue, python-queue) bounded by Redis-backed host execution budgets
  • Detached per-submission Docker sandboxes with zero networking, dropped capabilities, tmpfs mounts, and read-only root filesystems
  • Base64 Line Streaming Protocol (BLSP) streaming batches of up to 50 test cases to a single runner process via stdin/stdout
  • Fail-fast verdict engine recording Accepted, Wrong Answer, Time Limit Exceeded, Runtime Error, or Compilation Error upon first failure
  • Shared @koder/shared package unifying models, queue settings, language configs, protocols, and database operations

Architecture & data flow

Distributed service architecture decoupling web ingestion, queue orchestration, and containerized grading. The Next.js client interacts with an Express API over HTTP; Redis carries BullMQ job queues; dedicated Node.js worker daemons supervise isolated Docker containers; and MongoDB stores problems, test cases, and persistent submission outcomes.

System components

Express API Gateway

Express, Node.js, JWT

Handles session authentication, validates submission parameters against supported languages, records initial pending submissions in MongoDB, and enqueues jobs.

Queue Broker

Redis, BullMQ

Distributes execution workloads into language-partitioned FIFO queues (js-queue, java-queue, python-queue) to prevent compute starvation.

Language Workers

TypeScript, Node.js

Independent daemons that consume jobs, generate runner harnesses, mount namespaced temporary directories, and supervise container lifecycles.

DockerSandbox Engine

Docker, Alpine Linux

Spawns detached, hardened containers with dropped Linux capabilities, zero networking, read-only root filesystems, and strict CPU/memory cgroups.

BLSP Protocol Streamer

Node.js Streams, Base64

Streams test case inputs and captures outputs over docker exec stdin/stdout using line-framed Base64 payloads to preserve data integrity.

Verdict & Persistence Store

MongoDB, Mongoose

Stores question definitions, hidden test suites, user profiles, and completed submission verdicts with runtime and memory diagnostics.

Execution lifecycle

  1. Submission Ingestion: Authenticated user submits { language, code } via POST /api/v1/submissions/:questionId.
  2. Validation & Queueing: The API validates language against SUPPORTED_LANGUAGES, creates a submission record marked 'pending' in MongoDB, and pushes an execute job to the matching BullMQ queue.
  3. Worker Dispatch & Sandbox Setup: The language worker dequeues the job, loads the question specification, and creates a namespaced sandbox directory (${language}-${jobId}) under the worker temp directory.
  4. Container Spawning: A detached Docker container starts with --network none, --cap-drop ALL, --security-opt no-new-privileges, --user 1000:1000, --read-only, --tmpfs /tmp:size=64m, --memory=256m, and --cpus=1.
  5. Compilation (Java only): Java code is compiled using javac Main.java with a 25-second compilation watchdog. The JVM boots with a restricted SecurityManager blocking process creation, env lookups, and socket operations.
  6. Test Streaming (BLSP): The worker streams batches of up to 50 test cases via docker exec -i. Requests format as <caseId> <base64(input)>; responses format as <caseId> <OK|ERROR|FATAL_ERROR> <base64(payload)>.
  7. Fail-Fast Judging: Each test case is monitored by a 2,000 ms per-test watchdog, bounded by an overall 45,000 ms submission deadline. Execution halts at the first failing test case, assigning Accepted, Wrong Answer, Runtime Error, or Time Limit Exceeded.
  8. Cleanup & Delivery: The worker force-terminates runner processes, removes the Docker container, purges the sandbox directory, and persists the verdict to MongoDB. The frontend polls GET /api/v1/submissions/:submissionId for completion.

Technical decisions

Asynchronous Queue-Based Decoupling

Placing Redis and BullMQ between HTTP ingestion and code evaluation ensures slow, hung, or resource-heavy submissions never block the Express API event loop or hold open HTTP connections.

One Container per Submission with Streaming Test Cases

Starting and stopping a fresh Docker container for every individual test case would introduce 500-1,000 ms overhead per case. Instead, one container is launched per submission, and test cases stream over the Base64 Line Streaming Protocol (BLSP) to a continuous runner process, reducing multi-test suite runtimes by over 80%.

Namespaced Sandbox Directory Isolation

Because BullMQ generates sequential job IDs per queue (e.g. job #1 on js-queue and job #1 on python-queue concurrently), sandbox paths are strictly keyed by ${language}-${jobId} and validated against path-traversal attacks to prevent concurrent cross-queue filesystem collisions.

Fail-Fast Evaluation Strategy

Submissions halt immediately on the first failed test case rather than evaluating remaining hidden cases, minimizing unnecessary CPU and memory consumption across the container fleet.

Multi-Layered Defense in Depth

Primary security is enforced at the Linux kernel level via Docker isolation (--network none, --cap-drop ALL, seccomp, non-root user, tmpfs, read-only root). For Java submissions, a secondary JVM SecurityManager is installed at runtime to block process forks, system property manipulation, and host reflection.

Challenges & solutions

Concurrent Cross-Language Sandbox Collisions

Discovered that independent BullMQ queues assign identical numeric job IDs. Solved by namespacing worker directories by language and job ID with canonical path resolution checks, verified through a dedicated multi-language collision test.

Unresponsive or Hostile User Submissions

Implemented layered watchdogs: a 2,000 ms per-test watchdog, a 25,000 ms Java compilation watchdog, and an overall 45,000 ms submission timeout. On timeout expiry, the engine sends SIGKILL to runner child processes and force-removes the Docker container via CLI cleanup.

Test Case Output Normalization

Output comparisons normalize carriage returns (\r\n vs \n) and surrounding whitespace, while supporting JSON-equivalent structural equality, case-insensitive boolean values, and sequence formatting.

Testing strategy

  • CI-safe Node test suites for backend API routes and language workers using built-in assert (npm test)
  • Protocol unit tests verifying Base64 Line Streaming Protocol framing, delimiter handling, and error states
  • Manual multi-language Docker sandbox collision test (test_sandbox_collision_docker.js) validating isolation under concurrent load

Verified outcomes

  • Completed issues ISSUE-001 through ISSUE-015 in repository tracking, validating full authentication, sandbox hardening, and multi-language execution
  • Production-ready execution contract supporting Node 20, Java 17, and Python 3.11 with identical verdict and resource semantics

Trade-offs & limitations

  • Out-of-memory verdicts: Docker cgroups enforce the 256 MiB memory ceiling, but because the engine does not parse cgroup OOM event notifications directly, OOM kills surface as generic Runtime Errors rather than a dedicated Memory Limit Exceeded verdict
  • Host-specific concurrency tuning: Worker concurrency is bounded by a Redis-backed host budget, but capacity limits are host-dependent and require benchmarking for specific server hardware
  • Docker daemon trust boundary: The Docker daemon is part of the trusted computing base; workers must operate on secured hosts where untrusted users cannot access the Docker socket or daemon
  • Authentication security: Rate limiting is implemented on submission routes, but automated account lockout after consecutive failed login attempts is not yet implemented
  • No live cloud deployment: The platform is tested and verified locally using Docker Compose; automated cloud CI/CD deployment is not currently maintained