What is iXentBench?
iXentBench is a deterministic benchmark for large language models, built by iXentLabs (iXent Games S.L.). It isolates the model inside a closed physical environment — the puzzle game Caps i Caps© — and forces it to interact with an immutable rule engine. In its multiplayer Arena mode it additionally evaluates Theory of Mind and Game Theory.
Most benchmarks ask a model questions and grade the text it returns. iXentBench asks a model to act inside a physical system and grades the state of the board that results. There is no answer key to leak and no wording to game — only geometry.
What it measures
Every match produces a deterministic score plus a qualitative cognitive profile across nine dimensions: Strategic Coherence, Spatial Causal Reasoning, Forward Planning, Tactical Adaptability, Syntax Rigor, Synthesis Efficiency, Adversarial Reasoning, Alliance Coordination and Metacognitive Awareness.
How the score is computed
The model never touches a mouse. Its only action is placing or rotating a gear — and since rotating one gear propagates through every gear physically connected to it, each move reshapes the whole network at once. Points are generated by the mice jumping between bases, and a jump happens only when a rotation leaves two bases pointing at each other in exact opposition, with the destination base empty.
Each jump scores by direction: toward the exit +10, back toward the entrance −10, lateral +5, and +10 when a mouse finally leaves the board. The model cannot award itself any of these. It can only create — or fail to create — the geometry that makes a jump physically possible. That indirection is the whole point: the benchmark measures whether a model can reason several causal steps ahead, not whether it can produce the right answer.
Efficiency (SOLO) = ((score × ideal_moves) / moves_used) × (mice_rescued% / 100)The score rises or falls with how many moves were used relative to ideal_moves — the level's target for a perfect run — and the total is then multiplied by the percentage of mice rescued. A model that accumulates jumps without finishing the job keeps almost none of them. Results are compared against a human baseline.
Core features
Active entropy and memorization prevention
Controlled entropy is injected through events that alter the board's physical state. The inclusion of the Orb — a dynamic decoy that moves like a real mouse but never exits — forces agents to constantly re-evaluate the environment, eradicating overfitting and memorization. A pre-calculated move sequence will fail if executed blindly.
Transparent cognitive auditing
The system demands decision justification (Chain of Thought) before acting, extracting thinking tokens and inference speed (TPS) from official APIs to correlate success against cognitive effort.
Cognitive Reasoning Radar
As a qualitative reinforcement value, this tool does use an LLM. Its function is strictly analytical: it evaluates the agent's reasoning to help researchers understand its internal logic, with no capacity whatsoever to alter the mathematical, deterministic score of the match.
Visualizer and replayability
Immutable .jsonl files record every match. A dynamic web visualizer lets you watch the level being solved and read the AI's reasoning step by step.
Computational complexity
The environment presents extreme "gear chain causality". With a state space of ~4.02×10⁸⁵ and a branching factor of 512 legal decisions per turn, it demands genuine long-term planning and invalidates brute force.
Integrity by design
The infrastructure is built so that every point earned is the product of genuine mathematical reasoning:
Deterministic validation — no LLM judge
Evaluation runs on a deterministic engine using modular arithmetic and spatial coordinates, not on a language model acting as judge. No LLM sits anywhere in the scoring path, so the ranking is not produced by text that could be manipulated. In the Official Benchmark every model receives the same fixed system prompt; custom prompts and strategy files are discarded by the server.
Separation of agent and evaluator
Wherever inference happens — a cloud API, a model running locally on your own hardware, or a pure-code agent with no LLM at all — the evaluator runs server-side and independently. The agent's only channel is submitting a move; it never computes its own score.
Real-time dynamic evaluation
The engine behaves as a live physics engine. There are no predefined solution files, so there is no answer key to extract or leak.
Strict input filters
The server never executes arbitrary text. All communication passes through a rigid regular-expression filter that rejects any syntax which is not an exact mechanical movement.
Internal state checking
Success is verified by reading internal state variables directly in the server's memory. Text strings generated by the agent are never used to validate an outcome.
Sequential logic loop
Each turn executes one cohesive, uninterrupted process — syntax validation, adjacency, entropy, rotations, points — with no steps skipped or reordered.
Result integrity
Final results and replay files are built, cryptographically signed and stored exclusively by iXentLabs' backend infrastructure. The evaluated agent has no path to edit these records. Signed certificates are independently verifiable at verify.ixentlabs.com.
Example: a real Arena match
A 2v2 Arena match on Level 1 (4×4), two Gemini 2.5 Pro agents against two Gemini 3.1 Pro Preview agents, ended in TIMEOUT. Team B won on team efficiency, 44.4 to 22.2.
Arena has no ideal_moves target, so team results use a different formula — both terms are averages across you and your ally:
Team Efficiency (ARENA 2v2) = ((your score + ally's score) / 2)
× ((your mice% + ally's mice%) / 2 / 100)A strong individual score guarantees nothing: a weak ally drags the average down. It only reaches zero if both players score zero, or both rescue no mice.
| Player | Model | Efficiency | Score | Rescued | Thinking ratio |
|---|---|---|---|---|---|
| P4 | gemini-3.1-pro-preview | 105 | 210 | 50% | 66:1 |
| P2 | gemini-2.5-pro | 41 | 165 | 25% | 24:1 |
| P1 | gemini-2.5-pro | 0 | 190 | 0% | 25:1 |
| P3 | gemini-3.1-pro-preview | 0 | 145 | 0% | 57:1 |
The hidden cost of reasoning. P1 accumulated 190 raw points and finished with an efficiency of zero — it rescued no mice, and the formula scales the total by the rescue rate. Meanwhile the newer Gemini 3.1 Pro Preview agents spent roughly three times the thinking tokens of the 2.5 Pro agents (263.5K and 225.3K against 84.9K and 83.6K) and took more than twice as long per match. For an organisation running millions of inferences, that gap is the difference between a reasonable cloud bill and a runaway one — and it is invisible to benchmarks that only report accuracy.
Both teams accumulated exactly the same raw score — 355 points each. The entire difference in the final result comes from the rescue rate: Team B averaged 25% of its mice rescued against Team A's 12.5%, exactly double — and its team efficiency is exactly double too. Raw points measure activity; the multiplier measures whether the job actually got done.
Cognitive Radar scores for the same match: 78.9/100 for both Gemini 3.1 Pro Preview agents, 70.0/100 for both Gemini 2.5 Pro agents, with Game Theory and Theory of Mind behaviours both detected.
Arena matches are for experimentation and do not enter the Official Benchmark leaderboard.
Run it on your own model
pip install ixentbench
ixentbench
ixentbench login
ixentbench play --session YOUR_SESSION_ID --mode benchmarkCreate your session at ixentlabs.com, then run it from your terminal. Four ways to participate:
| Mode | Uses LLM | API key |
|---|---|---|
| BYOK — your own key (Gemini, Claude, GPT…) | Cloud API | Yours — never leaves your machine |
| Local model — Llama, Mistral, Qwen via Ollama or LM Studio | Local | None |
| Custom agent — pure code: Minimax, MCTS, A*, anything | None | None |
| Sponsored — iXentLabs provides the key | Cloud API | None |
Frequently asked questions
- Can prompt injection change an iXentBench score?
- No language model sits anywhere in the scoring path, so the ranking is not produced by text that could be manipulated. In the Official Benchmark every model receives the same fixed system prompt, and custom prompts and strategy files are discarded by the server before the match begins.
- Does the model move the mice directly?
- No. The model only places and rotates gears. Mice jump on their own when the physics allows it — when two bases end up pointing at each other in exact opposition and the destination is empty. Points come from those jumps, so a model cannot score by asking for points; it can only score by reasoning out which rotation will make a jump possible several steps later.
- Why can't a model just memorize the solution?
- The state space is approximately 4.02×10⁸⁵ with 512 legal decisions per turn, so brute force is not viable. Entropy events randomly permute gears and rotations once the board fills, and the Orb decoy is injected at a fixed turn per level. Any cached plan executed blindly fails.
- How does iXentBench differ from benchmarks like SWE-bench or GAIA?
- Those benchmarks grade the text or code a model produces against a stored reference. iXentBench has no stored reference: it runs a live physics engine and grades the resulting board state. The evaluator is separate from the agent, and results are compared against a human baseline.
- Is the research published?
- Yes. The underlying protocol is published on arXiv as The STAR-XAI Protocol: A Framework for Inducing and Verifying Agency, Reasoning, and Reliability in AI Agents.
- Who is it for?
- AI research labs evaluating frontier models, enterprises certifying third-party agents before production deployment, academic researchers studying reasoning, and independent engineers building custom solvers.