Work, original
system-one-bench
Reproducible latency measurements for Laya, a decision model by Convai Innovations. One command times it on your machine and checks that every backend gives the same answers.
- Role
- Author of the benchmark
- With
- On my own
- Status
- Released, version 0.1.0
- Years
Hover a part to see its nameThe problem
Laya is a System 1 decision model: it returns a probability for each option of a decision. The time of one call is the number that matters. A latency table from one machine does not tell you where that time goes. It also does not let you check the number on your own machine.
What I built
One command, sob reproduce, runs every row a release ran, for the backends your machine has. For each checkpoint, backend and workload row it records five things.
- Latency. The p50 and p99 of the real
predict_batchcall, over five independent rounds. The spread is the range of the round p50s. - Phases. The same call, run step by step with the device synchronized between steps: prepare, tokenize, collate, copy in, dispatch, encoder, head, copy out, decode.
- Kernels. On CUDA, the number of GPU kernels one call launches and the time they run. When the kernels run for much less time than the call takes, the GPU waits for the host.
- Work. Useful floating-point operations from the checkpoint's own config, counting real tokens only.
- Answers. Every backend's probabilities against FP32 PyTorch on the CPU: the largest difference, and how many answers pick the same option.
The benchmark covers thirteen backends, from PyTorch on the CPU to TensorRT, ONNX Runtime and Apple Metal. A backend that cannot load, or that falls back to another device, becomes a row with its status and its error. It never becomes a gap or a silent CPU result. Each checkpoint is downloaded at a pinned commit, and weights whose SHA-256 does not match are refused.
Evidence
The v0.1.0 run on an RTX 4060 Laptop GPU produced 144 rows. The default call, BF16 under PyTorch eager, takes 24.4 ms. One call launches 1,329 kernels. They run for 19.6 ms of the 24.4 ms, so the GPU waits for the host the rest of the time. CUDA graphs take 18.2 ms, TileLang 12.9 ms and TensorRT in FP16 10.8 ms. On the CPU, in FP32, the same call takes 400 ms.
No answer changed. Across the 144 rows, every backend picked the same option as FP32 PyTorch on the CPU, and the largest probability difference was 0.012. The essay Where the time in a Laya decision goes, measured on a laptop GPU walks through each number.
My role
I wrote the benchmark: the harness, the phase split, the kernel count, the answer check and the released results. Laya, its checkpoints and its runtime are the work of Convai Innovations, released under Apache 2.0. This repository only measures them. It is independent of Convai and redistributes no weights.
Current limits
The released results come from one machine, an RTX 4060 Laptop GPU. A laptop GPU changes its clock with power and temperature. Every backend therefore starts after a rest, and each row records the GPU's temperature and clock. Two rows differ only when their p50 ranges do not overlap; a difference inside the overlap is noise. A FLOPs rate here is a floor, because it counts matrix multiplications on real tokens only. A result from a machine the repository does not have is the most useful contribution it can get.