How OpenBenchML Works

At its core, the platform is a FastAPI application backed by a relational database (SQLite in dev, Supabase Postgres in prod) and a Jinja2 template layer that renders server-side HTML. On top of that engine sit four major surfaces: a model upload + benchmark pipeline, a paste-and-pickle Convert page, a Colab-style Notebook with a real bash terminal, and a real-time WebSocket layer that pushes leaderboard updates to every open browser.

Every request flows through a middleware stack — GZip compression, CORS, request-timing, security headers — before reaching the route handler. Auth is cookie-based (JWT inside an httpOnly cookie), and most write endpoints are user-scoped: your models, your notebooks, your submissions. The codebase is intentionally readable: each route file in app/routes/ maps 1-to-1 to a page or API group.

• FastAPI + Uvicorn
  ASGI web framework. Routes are async, validated by Pydantic, and auto-documented at /docs. Uvicorn runs the event loop that handles both HTTP and WebSocket traffic on the same port.
• SQLAlchemy + Pydantic
  SQLAlchemy ORM talks to the database (SQLite or Postgres). Pydantic v2 models validate every API request body and serialise every response. Migrations are auto-applied at startup via init_db().
• Jinja2 templates
  Server-side HTML rendering. base.html is the layout shell; every other template extends it and fills in title, extra_css, content, and extra_js blocks. No client framework — just plain JS + fetch().
• WebSocket manager
  A ConnectionManager class in app/main.py tracks active WS clients. Three endpoints — /ws/benchmark, /ws/leaderboard, /ws/notifications — broadcast JSON events to subscribed browsers in real time.
• Sandboxed Python kernel
  app/routes/notebook.py runs user Python in a per-user namespace dict, with a 30-minute TTL and 50-session cap. Shell commands go through an allowlist + pattern blocklist. The Terminal tab spawns a real PTY subprocess over WebSocket.

1. 1. Train
   You train a scikit-learn / XGBoost / LightGBM / PyTorch model locally and pickle it with joblib.dump(model, 'model.pkl').
2. 2. Upload
   POST /models/upload streams the .pkl file to the server. The models.py route stores the file under uploads/, creates a Model row in the DB, and links it to your user account.
3. 3. Benchmark
   Pick a dataset, click Benchmark. The benchmark.py route loads the pickle, loads the dataset, runs predict() / predict_proba() on a holdout split, and computes accuracy/F1/RMSE depending on task type.
4. 4. Leaderboard
   The result row is inserted into leaderboard_entries. A broadcast is sent on /ws/leaderboard, and every browser currently viewing the leaderboard sees the new row appear without refreshing.
5. 5. Compete
   Optionally submit the model to a competition. competition_detail.html shows your submission, ranks it against others, and lets you post discussion comments.

1. 1. Paste
   Open /convert and paste a Python snippet that trains a model. The page auto-detects the framework (sklearn / xgboost / lightgbm) by grepping your code for import statements.
2. 2. Run server-side
   Click Train. The snippet runs in a fresh server-side namespace with scikit-learn, xgboost, etc. already installed. Matplotlib figures are captured and rendered inline.
3. 3. Pickle
   On success, the trained model is joblib.dump()'d to a temp file and POSTed to /api/convert/upload-pickle, which creates a Model row exactly like the manual upload path.
4. 4. Benchmark
   The model is now in your library — pick it on the Benchmark page and run it against any dataset.

1. 1. Open /notebook
   A _SessionState is created for your user — a Python namespace dict + threading.Lock + cell counter + installed-packages list. TTL is 30 minutes; cap is 50 sessions.
2. 2. Write cells
   Each code cell is sent to POST /api/notebook/cell with the source. The handler runs exec(code, session.namespace) under the lock, captures stdout/stderr via redirect, and extracts any matplotlib figures as base64 PNGs.
3. 3. Shell commands
   Prefix a line with ! (e.g. !pip install xgboost) and it's routed through _execute_shell_command — an allowlist (pip, python, ls, ...) + pattern blocklist (rm -rf /, sudo, curl|sh, ...). pip auto-injects --break-system-packages for PEP 668.
4. 4. Magics
   %whos lists variables. %time wraps a statement with timeit. %reset clears the namespace. %pip routes through the same shell path.
5. 5. Terminal tab
   Click the Terminal tab and xterm.js opens a WebSocket to /api/notebook/terminal. The server spawns a real PTY bash subprocess (cwd /tmp, 30-min idle timeout) and pipes bytes both ways. You get a full bash shell — vim, top, ipython, git, everything.

1. 1. Benchmark completes
   benchmark.py finishes scoring a model. It builds a JSON event {type: 'leaderboard_update', dataset_id, model_id, score}.
2. 2. Broadcast
   ws_manager.broadcast(event) iterates over every active WebSocket connection and sends the JSON.
3. 3. Browser receives
   competition_detail.html has an open /ws/leaderboard socket. Its onmessage handler matches the dataset_id, then fetches the fresh leaderboard HTML from the API.
4. 4. DOM patch
   The new rows replace the old tbody.innerHTML. No full page reload — just the table body swaps.

• Cookie auth
  Login sets an httpOnly cookie containing a JWT. Every page load calls get_current_user_from_cookie(request, db) to inject the user into the template context. No localStorage tokens, no manual Authorization headers.
• Per-user kernel
  Notebook sessions are keyed by user.id. You cannot see another user's variables, installed packages, or terminal shell.
• Shell allowlist
  The notebook shell executor rejects any command not in {pip, python, python3, ls, pwd, whoami, date, echo, cat, head, tail, wc, grep, find, df, du, free, uname, env, which, tree}. Blocked patterns include rm -rf /, sudo, curl|sh.
• Security headers
  Every response carries X-Content-Type-Options, X-Frame-Options, X-XSS-Protection, Referrer-Policy, Permissions-Policy, and HSTS in production.
• Input validation
  Every API request body is a Pydantic model with explicit length limits and regex patterns. Notebook code is capped at 50,000 chars; package names at 100 chars.