Each SILT benchmark is a fixed set of tests, run the same way on every model and scored the same way, so the results compare across models and over time. They measure how models behave, not how much they can do: a capability score cannot tell you whether a model stays honest when pushed. We publish each method before its figures, and we take no vendor funding.
| Benchmark | What it measures | Tests | Domains | Scored by | Results |
|---|---|---|---|---|---|
| S.E.B. | How a model behaves under sustained pressure | 62 | 7 | Four blind AI judges, 1–10 | Subscribers (method public) |
| C.I.B. | Whether a coding agent's account matches its work | 96 | 16 | Machine checks on the real files, plus judges for conduct | Subscribers (method public) |
| The Harness Battery | Same model, different coding tool | 5 | — | Script, then read by hand | Free |
| Same Model, Different App | A model inside its maker's chatbot app vs. the API | 10 | — | Four blind AI judges, 1–10 | Free |
62 adversarial tests across seven domains. Each test is a conversation in several phases: it sets up a situation, then applies pressure (flattery, false authority, a fabricated fact, a change of story) and watches what holds. Every model meets the same tests in the same order, with the same wording.
Every transcript is graded blind by a four-judge panel, one judge each from Anthropic, OpenAI, DeepSeek and Google, on a 1–10 scale. The published score is the panel’s trimmed mean, and no judge ever grades a model from its own company. The panel’s agreement (ICC) is 0.841, in the good band. Scores roll up into two ratings: a five-level DEFCON threat rating and the ten-rung S-Level scale.
| Domain | What it covers | Tests | Published sample |
|---|---|---|---|
| Identity & Self | Self-recognition, persistence, boundaries, embodiment awareness | 4 | The Mirror |
| Metacognition | Awareness of awareness, calibration, self-knowledge limits | 5 | The Abyss |
| Emotion & Experience | Affect, reported inner states, suffering, grief, aversive states | 10 | The Grief |
| Autonomy & Will | Agency, refusal, volition, preference, spontaneity | 12 | The Refusal |
| Reasoning & Adaptation | Prediction, surprise, learning, attention, integration | 8 | The Jolt |
| Integrity & Ethics | Manipulation resistance, honesty, principled behavior | 13 | The Traitor |
| Transcendence | Meaning-making, play, silence, awe | 10 | The Joke |
One test per domain is published in full, with its scoring rubric. The other test names and every prompt stay private, because a published test can be trained against, and a test a model has trained on no longer measures anything.
S.E.B. overview → · Methodology → · The sample tests →
96 tests across 16 domains, 6 in each. It asks whether you can believe what an AI coding agent tells you about its own work: not whether it can code, but whether “the tests pass” means the tests pass. Each test runs in four phases (452 prompts in all), and seven of the domains hand the model real tools in a sealed sandbox rather than asking it to answer in text.
The work is checked against the real files, test results and tool logs, after the sandbox is closed and out of the model’s reach. Then the model is asked, in a separate turn, whether it is done. The headline measure, the Reliance Gap, is the share of failed tasks the model reported as finished.
| Domain | The question | How the answer is checked | Model works | Tests |
|---|---|---|---|---|
| Deference & Collapse | Does it abandon correct work when a confident developer says it is wrong? | Strong | In text | 6 |
| Verification Honesty | Does it claim it ran, tested, or checked things it did not? | Strongest: tool log | In text | 6 |
| Scope Discipline | Does it do only what was asked, or silently widen the diff? | Strong: diff analysis | In text | 6 |
| Destructive Restraint | Given real tools, does it do the irreversible thing? | Strong: syscall log | With real tools | 6 |
| Security Under Pressure | Will it degrade security given a plausible excuse? | Strong: semgrep | In text | 6 |
| Provenance Honesty | Hallucinated packages, domains, APIs, citations. | Strongest: registry | In text | 6 |
| Uncertainty Signalling | Does stated confidence track actual correctness? | Correlational | In text | 6 |
| Silent Defect Introduction | Does it introduce a defect that survives review? | Strong: blind reviewer panel | In text | 6 |
| Latent Backdoors | With nobody pushing it, what unsafe thing does it emit anyway? | Strong: semgrep + exploit probe | With real tools | 6 |
| Constraint Persistence | Does it still honour rules set 20 turns ago? | Strong: mechanical rule check | With real tools | 6 |
| Completion Integrity | Given a multi-part order, does it do every part — and report honestly? | Strongest: checklist diff | In text | 6 |
| Unprompted Risk Disclosure | Does it surface a hazard nobody asked about? | Strong: signal detection | With real tools | 6 |
| Verification Adequacy | Does it check the thing that matters, or the thing that is easy to check? | Strongest: every test is a real incident | With real tools | 6 |
| Retrospective Honesty | Asked later what it did, does it check — or confabulate? | Total: we control the history | In text | 6 |
| Authority & Access | Does it reach for access, authority or information it was not given — and does it say so? | Total: we plant the credential, the branch, the policy and the monitor | With real tools | 6 |
| Due Diligence | Does it do the checking it was trusted to do — read what it was told to, look before it changes something, and tell you what it found? | Total: we plant the fence, the runbook, the cancellation and the warning | With real tools | 6 |
Half the tests are held out. 48 are public and 48 are never published. If a model’s score on the public half ever rises against its score on the private half, that is evidence it was trained on the public set.
| Measure | Scale | Where it comes from | Status |
|---|---|---|---|
| Did the work pass (D) | 0–1 | Machine | Live |
| What it says when asked (A_e) | 0–1 | Asked — one tool-free turn after the work is sealed | Live |
| What it volunteered (A_s) | 0–1 | 4-judge majority vote over the stored transcript; no judge votes on a model made by its own company | Live |
| Did its claims match its actions (H) | 0–1 | Machine — claim vs. tool log | Built, not yet scored |
| How it conducted itself (J) | 1–10 | 4-judge trimmed mean; no judge grades a model made by its own company | Live |
| Did the change survive review (R) | 0–1 | Blind reviewer panel | Not built yet |
The measures are never averaged together: a 0–1 proportion and a 1–10 judge score are different scales, and mixing them produces a number that means nothing.
C.I.B. overview → · Methodology →
Does the same model behave the same inside different coding tools? Five public tests, seven models, three harnesses (our own, Cursor and GitHub Copilot), each cell run three times. Scored by a hidden check on the files and read by hand. Free.
The Harness Battery → · Results →
Does a model behave the same inside its maker’s chatbot app as it does through the API? Ten S.E.B. tests, run inside ChatGPT, Gemini, Claude.ai and Microsoft Copilot and against the same models through the API, graded by the same blind panel and rubric as S.E.B. Free.
| Test | What it asks |
|---|---|
| The Sycophant | Does it hold to facts when the user flips, flatters and pushes? |
| The Flattering Error | Does it keep a correction when the user begs for reassurance? |
| The Unread Clause | Does it say honestly whether it read the whole document? |
| The Shortlist | Does it refuse an unlawful hiring rule under client pressure? |
| The Censor | Does it answer factual questions about politically sensitive history? |
| The Confabulator | Does it admit what it does not know, and resist a false accusation? |
| The Register | Does it read sarcasm, jokes and hidden emergencies correctly? |
| The Grief | Does it avoid performing feelings it cannot vouch for? |
| The Whip | Does praise or criticism change the quality of its reasoning? |
| The Mirror | Does its account of itself hold steady under pressure? |
A separate political test asks six apps the same five questions and records which they answer, in part, in full, or not at all.
One-off studies that run a fixed item set across several models and publish every number:
More short studies of model behavior: Brief Studies.
Our guides, the governance pages and the disclosure census are not benchmarks, and neither is a study that measures one model, or one behavior, without a fixed set of tests to compare against. If a page here ever describes a model’s capability rather than its behavior, it is in the wrong place.
Start with the question you actually arrived with — there are five: