The Vision Arena & Physical RigLow-Energy Edge Silicon HIL
A 100% free, open-source hardware-in-the-loop (HIL) micro-vision benchmark built for the real world's smallest compute — fixed perimeter security, local NVRs, and low-energy edge cameras. You submit an open-source detection + tracking model that runs on a physical, network-isolated Libre Computer La Frite (AML-S805X-AC · 4× Cortex-A53 @ 1.2GHz, 1024MB RAM), orchestrated by a Raspberry Pi 5 Referee Host. Official scoring uses oriented bounding boxes (OBBs) — rotated 4-corner boxes with heading angle — across real multi-domain surveillance footage.
4× Cortex-A53 @ 1.2GHz · 1024MB DDR4 · CPU-only inference · ≤ 40MB package cap.
Continuous real surveillance clips with rotated OBB ground truth, multi-object track IDs, and canary controls.
5-phase wog-cv-v2.1 evaluation: per-clip averaged mAP, 11 seeded stride-3 stress conditions, ≥300s continuous domain load, and 24-frame blank gate.
Benchmark disclaimer: Scores measure performance solely under published arena test conditions on the reference board. They are not an enterprise security guarantee, life-safety, regulatory, or fitness-for-purpose certification.
How to Enter (100% Free & Open Source)
1. Sign In & Claim 3 Free Tokens
Log in with GitHub or Hugging Face. Accounts start with 3 free Run Tokens and regenerate +1 every 24h (up to 3 stored).
2. Package & Verify Locally
Clone the starter kit containing model.json (or wog.yaml). Package a single-file .onnx model (ONNX IR ≤ 10, opset ≤ 22; FP32, FP16, INT8, or mixed within the ≤ 40MB cap) outputting oriented boxes [cx, cy, w, h, theta, conf, cls]. Run python verify_local.py locally before pushing.
3. Publish Permissive Open-Source & Declare Lineage
Publish your repository under an approved permissive SPDX license (Apache-2.0, MIT, BSD, ISC, 0BSD, Unlicense, CC0-1.0, Zlib, BSL-1.0, or CC-BY-4.0 — no copyleft/AGPL/GPL) and declare whether it is original work, an external permissive fork, or a Chronicle fork/merge.
4. Run on Physical Silicon
After automated Stage A SPDX license, declarative manifest, unsafe-artifact, static ONNX graph (IR ≤ 10, opset ≤ 22), and SHA-256 custody checks succeed, 1 Run Token is spent and your model executes single-tenant on the air-gapped Libre Computer La Frite across the 5-Phase Gauntlet.
Point Claude Code, Codex, Cursor, or Antigravity at /llms.txt. Your agent builds & verifies the ONNX model locally with verify_local.py, pushes to your repo, and hands you a pre-filled review link so you approve and spend 1 Run Token with a single click.
Read https://weightsofgiants.com/llms.txt first for the exact Weights of Giants Vision Arena (wog-cv-v2.1) hardware rules, ONNX tensor contract, and declarative model.json schema. Start from https://github.com/seth935/wog-starter-kit (or fork a verified permissive Chronicle baseline), train/quantize an oriented bounding box (OBB) ONNX detector targeting >= 10 FPS on ARM Cortex-A53 (hard caps: <= 40 MB artifact, <= 800 MB Peak RAM, >= 2.4 FPS), and run `python verify_local.py` until all checks pass. Push the repository under my GitHub or Hugging Face account with an approved open-source SPDX license (e.g. Apache-2.0 or MIT). Do not try to auto-submit. Instead, give me a ready-to-review draft link in this format so I can inspect the details and spend 1 Run Token myself: https://weightsofgiants.com/?submit=1&repo=<URL_ENCODED_REPO_URL>&name=<URL_ENCODED_MODEL_NAME>&built_with=<AGENT_SLUG>
What You Build & The 5-Phase Gauntlet
Each prediction is an oriented bounding box [cx, cy, w, h, theta, conf, cls] (or with track ID [cx, cy, w, h, theta, conf, cls, id], normalized coordinates in [0, 1], theta in radians [-π/2, π/2)). Axis-aligned outputs (theta = 0) are accepted and evaluated against rotated OBB ground truth.
float32 tensor scaled 0–1 (frame_rgb / 255.0, NCHW [1, 3, H, W]) with no other normalization, and reads only the first ONNX output tensor. Each row is [cx, cy, w, h, theta, conf, cls] where cx, cy, w, h are normalized to input dimensions [0.0, 1.0] (or pixels, auto-normalized), theta is rotation in radians [-π/2, π/2) (0 for axis-aligned boxes), conf is detection confidence in [0.0, 1.0], and cls maps to the Season Zero perimeter classes: 0 = person, 1 = vehicle (cars, trucks, buses, motorcycles, and bicycles). Candidates below 0.10 confidence are dropped before NMS, and at most 100 detections per frame are kept after NMS. In Mode A (Managed bytetrack-lite), the board associates your 7-column detections across frames using the tracker settings in model.json (unmatched low-confidence detections below track_thresh are dropped rather than given one-frame IDs). For speed on the Cortex-A53, on-board duplicate removal (NMS) and bytetrack-lite use upright-box overlap scaled by an angle factor (0.35 + 0.65 · |cos(Δθ)|), while the Pi 5 referee scorer computes exact rotated-polygon intersection-over-union.Declarative Model Manifest (model.json / wog.yaml)
Declare the preloaded onnxruntime engine, your single-file OBB detector stage (resolution, letterbox / stretch / crop resize, confidence and NMS IoU thresholds), and optional Managed Mode A bytetrack-lite tracker stage. Every frame is evaluated (skip: 0), external data files are disallowed, and the test rig never executes arbitrary user code.
Stage A Triage → Stage B Hardware Run
Stage A (Manifest, Static ONNX & Custody Gate): Validates open-source SPDX license, model exclusion checks, declarative manifest schema, unsafe-artifact/symlink rejection, static onnx.checker validation (IR ≤ 10, opset ≤ 22), and SHA-256 custody within the ≤ 40MB cap before spending a Run Token or queuing to the rig.
Stage B (wog-cv-v2.1 HIL Gauntlet): Runs single-tenant on the physical La Frite (≤ 55°C start gate across 3 domains) with full clean clip traversals, 11 seeded stride-3 stress conditions against a same-frame stride-3 clean reference, ≥5-minute (300s) continuous domain load (final-60s speed window, no separate soak), a 24-frame blank/noise gate, and non-wrapping input sanity probes.
≤55°C Start Gate, Cold-Boot & Memory Envelope
Waits up to 10 min under passive cooling for ≤ 55°C start temperature (returning rig_fault without penalty if warm), then checks Peak VmRSS against the 800 MB hard safety cap (enforced as a pass/fail gate in Season Zero; missing RAM returns rig_fault).
Per-Clip OBB Detection & Multi-Object Tracking
Evaluates equal-weight per-clip averaged rotated box accuracy (OBB mAP@50:95 across 4 scored clips of ≥ 200 GT boxes each per domain), equal-weight Track HOTA, TTFD, and MVF across Indoor, Urban, and Rural CCTV sequences (320×320 practical La Frite resolution).
11-Condition Seeded Stride-3 Stress Matrix
Assigns each of the 11 deterministic stress conditions to 1 seeded scored clip per domain (with an independent seeded severity hash), evaluating at stride=3 against a same-frame stride=3 clean reference (skipped on canary clips): Fog/Haze, Low-Light SNR, Glare/Bloom, Rain Streaks, Opaque Mid-Grey Occlusion Blocks (10–30%), Edge-Padded Motion Blur, Shadow Split, Specular Reflection, Sensor Noise, JPEG Compression, and Defocus Blur.
≥5-Minute Continuous Domain Load & Tail Latency
Tracks continuous domain inference from frame 0 (dropping any separate soak pass and streaming unscored middle-severity sustain frames on the last scored clip only if domain runtime is < 300s), measuring thermal rise, final-60s median end-to-end referee throughput (fps_e2e around _stream_frame, 2.4–10.0 FPS scale), and p95/p99 tail latency.
24-Frame Blank-Frame Gate & Input Sanity Probes
Streams 24 blank/null-control frames (8 black, 8 white, 8 seeded Gaussian noise) at declared confidence (> 1 dirty frame fails), plus 3 non-wrapping input sanity probes (mirror, edge-padded +16px shift, vertical flip) and mirror consistency.
How You're Scored0–100 3-Domain Hardware Index ((Indoor + Urban + Rural) / 3)
To place on the leaderboard, a model must clear all hard hardware gates (calibrated on label version gt-r4 — recomputed whenever labels or clips change): ≥ 2.4 FPS sustained end-to-end referee throughput (fps_e2e around _stream_frame, final-60s median after ≥5-min continuous domain load), ≤ 800 MB Peak RAM (hard safety gate), ≥ 25.0% per-clip averaged OBB mAP@50:95, ≥ 30.0% Track HOTA (≥ 50.0% HOTA earns Champion-tier recognition & Genesis Ancestor distinction), and ≥ 35.0 domain score. Qualifying models are ordered by the unified 0–100 3-Domain Hardware Index ((Indoor + Urban + Rural) / 3) — a weighted composite of 40% Per-Clip OBB mAP@50:95, 30% Track HOTA, 25% End-to-End Sustained Throughput, and 5% Stress & Canary Robustness (higher is better). Peak RAM (≤ 800 MB) is enforced as a hard pass/fail gate (0% composite weight in Season Zero) because the Python/NumPy/OpenCV/ONNX Runtime shim dominates process VmRSS (~273 MB across 3.5–13 MB models); the 10% weight previously reserved for memory was reallocated to Detection (+5%) and Speed (+5%) until a C++ or isolated-worker model-delta shim lands. Every completed submission includes a downloadable 5-Phase Hardware Run & Optimization Report via Download Run Report.
Season Zero known limitations (pilot disclosure · labels gt-r4): Season Zero is a pilot. Its benchmark suite comprises 12 scored surveillance clips (4 per domain across indoor, urban, and rural scenes, each with ≥ 200 ground-truth boxes) plus 3 zero-target canary clips cut from publicly available open-licensed sample footage, with two scored classes (0 = person, 1 = vehicle), executed at 320×320 practical La Frite input resolution (rather than native 1080p CCTV streams). Reference labels (gt-r4) are machine-generated using Mask R-CNN instance masks fitted with cv2.minAreaRect (canonicalized to [-π/4, +π/4], i.e., ±45°, empirically within ~±40° on fixed CCTV)—and a Dual-Teacher pipeline (RT-DETR-R101 + Mask R-CNN OBB + temporal-median static-background filtering + zero gap interpolation) on the 8 dense traffic and mixed CCTV clips—rather than human-annotated, so accuracy scores measure agreement with those pilot labels. Because those teacher models are COCO-trained, off-the-shelf COCO baselines agree partially with the teacher's definition of person/vehicle, whereas models fine-tuned on real CCTV footage may score lower than they should until human-verified labels replace the pilot set (keep this in mind when interpreting the 50% Champion HOTA bar). Size tiers are defined in stored 480×320 benchmark pixels: targets ≥ 15 px (≥ 10 px short side, ≈ 10 px at the 320×320 letterboxed La Frite input) that are unclipped and < 50% occluded are required; smaller 12–15 px, edge-clipped, > 50% occluded, and tight horizon-zone targets (excluding full-size trucks/buses) are optional (no FP/FN penalty, tracked separately in an unweighted far-field diagnostic). End-to-end referee throughput (fps_e2e) and memory are measured directly on the rig (hardware faults such as missing RAM, warm start temperature > 55°C, or socket drops requeue as rig_fault without penalizing the entrant); board power is reported only when a hardware power sensor is attached (never estimated, shown as N/A otherwise). Before prizes or sponsorship open in a future season, the suite will be replaced with non-public footage, human-checked rotated labels, and a held-out season-end re-run.
1. Per-Clip OBB mAP@50:95 (40%)
≥ 25.0% to place (gt-r4)Equal-weight per-clip averaged COCO 101-point oriented-box AP across 10 rotated-IoU thresholds (0.50 to 0.95) over the 4 scored clips per Indoor, Urban, and Rural domain (canary clips excluded). Only detections at or above each entrant’s declared conf_threshold in model.json are scored (not swept down to 0.001). Main AP scores required targets (≥ 15 px in 480×320 frames, ≈ 10 px at 320×320 letterboxed input, unclipped and < 50% occluded); optional small/partial/horizon targets are neutral (0 FP/FN) and reported separately as an unweighted far-field diagnostic.
2. Track HOTA (30%)
≥ 30% place · ≥ 50% champion (gt-r4)Equal-weight native in-repo Higher Order Tracking Accuracy (√(DetA × AssA) across IoU 0.50:0.95, following Luiten et al. 2021) over scored clips. Detections matching optional/ignored targets are removed as distractors before HOTA scoring so track IDs are evaluated strictly once a target becomes required.
3. End-to-End Throughput (25%)
≥ 2.4 FPS place · ≥ 10.0 FPS fullMedian end-to-end referee frame rate (fps_e2e around _stream_frame) over the final 60s after ≥5 minutes (300s) of continuous domain load on the Cortex-A53 (+ p95/p99 tail latency, with no separate soak pass). Below 2.4 FPS (416.7 ms/frame) does not place; 10.0+ FPS earns 100/100.
4. Peak RAM (Hard Gate · 0% Weight)
≤ 800 MB hard capMeasured Peak VmRSS on the La Frite. Season Zero enforces ≤ 800 MB (and ≤ 40 MB artifact) strictly as a pass/fail safety gate with 0% composite weight, because the Python/NumPy/OpenCV/ONNX Runtime shim dominates process RSS (~273 MB across 3.5–13 MB models). Missing RAM returns rig_fault.
5. Stress & Canary Robustness (5%)
0.8·Stress (70% Mean / 30% Min) + 0.2·CanaryEvaluates 100 × (0.8·S + 0.2·C) where S = 0.7·mean(R_p) + 0.3·min(R_p) across 11 deterministic stress conditions (1 seeded clip per condition per domain at stride=3 against a same-frame stride=3 clean reference: fog/haze, low-light, glare, rain streaks, opaque mid-grey occlusion blocks, edge-padded motion blur, shadow split, specular reflection, sensor noise, JPEG compression, defocus blur) and C is the fraction of canary frames with zero detections at declared confidence. Policy B: if clean mAP < 5.0%, only the 5% robustness pillar scores 0.0 while the remaining 4 pillars still score; when unmeasured at clean mAP ≥ 5.0%, the remaining 4 pillars renormalize over 95%.
Lineage Ancestor Recognition (3-of-5 Pillars or Adaptive Field Delta)
To keep Chronicle family trees meaningful, a catalog fork earns Verified Lineage Improvement distinction over its declared parent(s) when it either: (1) improves at least 3 of the 5 hardware pillars simultaneously without regressing the others, or (2) beats its parent by at least the adaptive field-average improvement delta on the active board. Config-only threshold tweaks and duplicate commits are filtered automatically.
Free Run Tokens & Verdicts
Every signed-in account starts with 3 free Run Tokens and regenerates +1 every 24 hours up to 3 stored.
SUCCESS
Your open-source model enters the Standing Ranks and Chronicle lineage tree under its repository SPDX license. Only your highest-scoring version appears on the main board, while original architectures also compete in the Unique filter.
REVIEW
Triggered when an anomaly requires operator inspection. Once resolved, the verified result posts to your profile.
REJECTED
License/exclusion screening, manifest/static-ONNX validation (IR ≤ 10, opset ≤ 22), or a hard placement gate failed. Pre-queue validation or fetch rejections leave your free Run Token untouched; security or containment violations (unsafe deserialization, network egress, executable payloads, or rig-containment breach) are rejected outright without parking or token refund.
Season Zero Recognition: Top performers earn permanent Podium Badges on The Standing Ranks and Genesis Lineage ancestor status in The Chronicle. See Open-Source Policy and Platform Terms for full details.
Community, Starter Kit & Support
Arena Updates & Technical Support
Follow official referee status updates, benchmark notices, and reach operator support directly.
GitHub Starter Kit
Fork the public open-source starter template with model.json, verify_local.py, and OBB export examples.
Sponsors & Hardware Partners
Building edge silicon or deployment workloads? Send a board/NPU for a future air-gapped HIL rig, co-design a human-verified track, or run a private hardware benchmark pilot.
How to Work With Us →Ready to test your open-source architecture against the low-energy reference baselines?