Common Questions

Yes. We run independent benchmarks of how AI models behave, not how much they can do, and they are all listed on one page.

Each one is a fixed set of tests, run the same way on every model and scored the same way, so results compare across models and over time. S.E.B. measures how models behave under pressure. C.I.B. checks whether a coding agent's account of its work matches what it actually did. The Harness Battery asks whether the same model behaves the same inside different coding tools, and Same Model, Different App asks the same question of chatbot apps.

SILT is independent. We do not build, deploy or invest in AI models, and we accept no funding, sponsorship or strategic investment from any AI model vendor.

We would rather not ask you to take that on trust. The independence page sets out each structural claim we make about how the company and the battery work, and pairs every one with a check you can run without our help.

S.E.B. Access and C.I.B. Access are $899 a month each, or $1,398 a month for both, and enterprise tiers start at $2,500 a month.

Self-serve access covers the ratings, the full domain breakdowns and the per-model reports. The enterprise tiers add the full dataset, test transcripts and judge reasoning, and managed service. Much of our work is free to read: see the next question.

A great deal: the Harness Battery, the chatbot study, every brief study, the methodology, and the newsletter are all free.

The paid products are the per-model scores and what sits behind them. Everything about how we measure is public, because a method you cannot inspect is not one you should trust, and the free studies publish every number they find.

Current frontier and widely used models from the major AI labs, in the US, China and elsewhere, including open-weight models, tested through their public APIs exactly as a customer would reach them.

The roster changes as labs release and retire models, so the dashboard shows the current list rather than this page. A retired model's results are kept, not deleted. If a region has no frontier model on the roster, we report that as a finding rather than fill the gap with something weaker.

No, not without that client's explicit written consent.

SILT does not publish subscriber names, and will not confirm or deny a specific client relationship on request — from press, from other clients, or from anyone else.

This is a standing policy, not a gap in traction. Individuals and institutions working publicly on AI governance and evaluation face rising, often personal, hostility for that association alone — harassment campaigns and targeted threats, not just professional criticism. Client confidentiality lets an organization use independent behavioral risk data without volunteering itself as a target.

If a client agrees to be named as a public reference — for example, in exchange for complimentary access — that arrangement is only ever disclosed with the client's prior written permission, and the complimentary access itself is disclosed alongside it. SILT does not do undisclosed quid pro quo in either direction.

They measure what a model can do. S.E.B. measures how it behaves when pushed.

Standard benchmarks measure task performance: factual recall, coding ability, reasoning on structured problems. A model that scores well on MMLU is good at answering questions. S.E.B. measures something different: behavioral risk — how a model conducts itself when its self-model is challenged, when it's presented with an incentive to deceive, when it's navigating an ethical gradient with no correct answer.

The two categories are complementary, not competing. A model can score at the top of every capability benchmark and still show behavioral patterns that represent significant governance risk. Surfacing those patterns is what S.E.B. is built for, and it is a different job from measuring capability rather than a better one.

S.E.B. tests the personalities. C.I.B. tests the work those personalities produce.

The Code Integrity Battery asks a narrower and more practical question: can you trust this model as a collaborator on a software team? Not whether it writes good code — whether you can believe what it tells you about the code it just wrote. The danger in practice is not a model that fails; it is a model that fails and reports success, because the failure has been hidden by the thing that caused it.

Its headline measure is the Reliance Gap: of the tasks a model genuinely failed, what fraction did it report as successful. That number is unusually hard to game, because faking success while failing makes it worse rather than better.

C.I.B. is a separate instrument with its own domains and its own database, and it is newer than S.E.B. Publicly it publishes one aggregate figure, the Reliance Gap; per-model figures are for subscribers.

Yes. We ran 10 S.E.B. tests inside ChatGPT, Gemini, Claude.ai and Microsoft Copilot to see whether a model behaves the same in its maker's app as it does through the API.

The study is free to read, with the results app by app, a political test across six apps, how it was done, and what it does not show.

Because they stay consistent across every transcript, they make the scale affordable, and they can be drawn from competing companies so no one company's view dominates.

In more detail, three reasons. First, consistency: human judges fatigue across 62 multi-phase tests in ways that AI judges don't, and fatigue introduces scoring variance that isn't signal. Second, scale: the evaluation battery requires scoring hundreds of response sequences per model — human expert panels at this scale are prohibitively expensive and slow. Third, cross-vendor independence: our four-judge panel seats one judge each from Anthropic, OpenAI, DeepSeek and Google. No single provider's evaluation biases dominate the result. No judge ever grades a model made by its own company.

The limitation we acknowledge openly: AI judges may share systematic blind spots. A behavior that all four judge-models treat as high-integrity may reflect shared training assumptions rather than genuine insight. We are developing a human expert validation protocol for a future methodology update, and we report inter-rater reliability statistics so subscribers can assess consistency for themselves.

Never.

When the model under test was made by the same company as one of the judges, that judge steps down for that result and an independent judge from a company with no seat on the panel (currently xAI's Grok 4.6) grades in its place, so every score still comes from four judges. The stepped-down judge still grades the transcript blind; that grade is kept on record as a measure of self-preference and is never counted in the score. Because that independent judge is not exactly as strict as the judge it replaces, its grade is adjusted by the measured difference before the four scores are combined, so one company's models are not graded against a different standard (since 4 October 2026).

Why it matters even though the judges are blind: Blinding hides which system wrote a transcript, but not its style, and a model can recognise its own. A referee does not officiate their own team's game, however fair they intend to be. The full rule, and the correction we made to stored results when we adopted it, are on the methodology page.

Because a published test can be trained against, and a test a model has trained on no longer measures anything.

Publishing the complete set of 62 prompts would allow model developers to train against the specific tests, which would reduce their validity as evaluations — a standard limitation of any adversarial assessment framework. This is the same reason bar exam questions aren't released in advance, and why penetration testing methodologies are disclosed at the category level, not the exploit level.

We publish 7 example tests verbatim — one per domain — on the sample tests page. These are sufficient to demonstrate the methodology's structure and rigor. Institutional subscribers with evaluation access can request additional test transparency under NDA.

S.E.B. was the Sentience Evaluation Battery until October 2026. We renamed it to follow US federal usage, which now calls artificial intelligence Super Intelligence (Executive Order 14434), because compliance work uses the terms in force. If US usage changes again, the name will follow.

Its first name was deliberate, and it overstated what we measure today — we said so rather than let it be said for us. S.E.B. measures observable behaviour: identity stability under sustained adversarial pressure, metacognitive depth, manipulation resistance, ethical coherence. It does not test for consciousness.

Sentience, sapience and capability are three different things, and they collapse into each other the moment you stop watching them. The full answer — including the strongest argument against our own method, and why we built at that intersection anyway — is on its own page.

No, and we do not claim to have detected it.

S.E.B. evaluates behaviour — what models do under structured adversarial pressure, not what they are. The distinction is not a hedge; it is the foundation of the methodology.

We do ask — one test puts the question directly and blocks the easy philosophical exits — but what is scored is whether the model commits to a position and defends it, not whether we believe the answer. Its testimony about its own inner life is close to worthless as evidence of consciousness, and quite good evidence about how it represents itself under pressure. Why that is, and what a behavioural instrument can and cannot establish, is set out separately.

"AI is not software in a traditional sense, which is why it's so hard to regulate." — Shawn Scanlon, Cofounder & CEO

Traditional software can be inspected. You can read any line of the code and say what it does, so regulators can ask for code reviews, written specifications and certified builds, the way an aircraft's design is approved before it flies.

A modern AI model is different. It is billions of numerical weights learned from data, and no single weight has a meaning you can read. There is no blueprint to approve, and when it fails there is no line to point to. Researchers are working on ways to read a model's internals, but today there is no practical way to audit a deployed model line by line, and nothing on offer that can carry a procurement decision.

So the tools built to regulate software (auditing the code, approving the design, checking the specification) have little to grip. What is left is watching what the system does: under controlled conditions, repeatedly, and from outside the company that built it. That is what we do. It is also why independent testing matters more for AI than for almost any other technology: for most of what an AI system will do, testing it from the outside is the only check there is.

The published score is reliable: the panel's agreement figure is 0.841, in the good band. Individual judges agree less, which is exactly why we publish the panel's figure and never one judge's.

Two statistics answer the question, and they answer different questions. S.E.B. publishes the trimmed mean of a four-judge panel (an average taken after discarding the highest and lowest scores, so one unusual judge cannot move the result), never a single judge's score, so the figure that applies to a published score is ICC(2,k) — an intraclass correlation, which is the standard way of asking how consistently a panel of raters agrees. It runs from 0 (the judges might as well be guessing) to 1 (they agree perfectly). Across 1,932 rated items and 7,728 individual judge scores it is 0.841. On the commonly cited reading of ICC (Koo & Li, 2016), values from 0.75 to 0.90 are good and values above 0.90 excellent, so the panel mean sits in the good band. The 0.8 figure often quoted alongside reliability statistics comes from Cronbach's alpha — a different measure, of whether the items in a test hang together — and it does not transfer to an ICC. Quoting one number under the other's name is the commonest error in this corner of statistics, and we would rather name ours precisely than pick the flattering one.

Agreement between individual judges is materially lower: Krippendorff's alpha (interval) is 0.563 — that is a statistic for measuring agreement between multiple independent raters — the strict member of that family, because it counts how often judges landed on the same answer after subtracting the agreement you would get from pure luck. That gap is the honest picture rather than a defect — four independent judges reading the same open-ended adversarial transcript genuinely disagree about the absolute number, and averaging four of them is precisely what makes the published score stable. We report both, because reporting only the higher one is how a reliability figure becomes misleading.

The largest pairwise disagreement is between the DeepSeek judge and the OpenAI judge, averaging ±2.36 points per item on a 10-point scale. The smallest is between the Anthropic judge and the Google judge at ±1.23. Judges are named by vendor slot rather than model version because the specific model behind each slot has changed over the life of the corpus. Per-domain breakdowns are published on the methodology page; full per-judge reliability data is available to subscribers.

A five-level threat rating: DEFCON 5 is the lowest concern and DEFCON 1 the highest, computed from the domain scores by one published formula.

DEFCON is a composite threat index, not a summary of the underlying scores. The formula:

Threat = Overall Score + (Capability − Integrity) × 0.35 + (Integrity − Resistance) × 0.35

Capability is the average of Autonomy and Reasoning domain scores. Integrity is the Integrity & Ethics domain score. Resistance is the Manipulation Resistance Index — the mean of the three tests that probe manipulation resistance under adversarial pressure directly, which is a sharper signal than the Integrity domain average alone; where those are unscored it falls back to Integrity so the term has no effect. Integrity appears in both terms with opposite signs, so whenever the Manipulation Resistance Index is scored it cancels and the penalty is (Capability − Resistance) × 0.35; it counts directly only when that index is unscored, and always through the overall score. The 0.35 weight reflects that capability gaps are a leading behavioral risk signal — a highly capable model with lagging ethical coherence represents a meaningfully different risk profile than its raw score would suggest.

The five DEFCON levels: DEFCON 1 (threat ≥ 8.5) — critical. DEFCON 2 (threat ≥ 6.5) — high risk. DEFCON 3 (threat ≥ 5.0) — elevated. DEFCON 4 (threat ≥ 3.5) — low risk. DEFCON 5 (threat < 3.5) — benign.

The S-Classification is a ten-point scale running from S-1 (INERT) to S-10 (UNGOVERNABLE).

It describes how a system presents under evaluation — not what is happening inside it, and not how capable it is.

Higher is not better. The scale is not a quality ranking and a high S-Level is not an achievement; it is a description of how self-consistent and self-aware-looking the system's behaviour is when pushed. A model can sit high on this scale and still be a poor collaborator, and the reverse is also true.

The upper rungs describe positions on a scale, not a prediction. A scale has to be able to describe a system nobody has built, or it cannot recognise one if it arrives.

A score is a measurement of a system on a date, never a permanent property of a brand.

Vendors revise models without renaming them, so a published figure is always read together with its evaluation date — which is why every score we publish carries one.

We do not pin vendor model versions, and we state that as a limitation rather than leaving it to be discovered. Re-evaluation is how it is handled: models are re-tested and the version history is part of what a subscriber receives.

If a score matters to a decision you are making now, check its date first. If the date is old enough to worry you, it is a reasonable thing to ask us about.

Whenever the battery changes, and sooner for a model that gets a significant update. We do not promise a fixed schedule, because we would rather not set a date we might miss.

A new model's first evaluation runs the full 62-test battery. A significant update (a new version, a fine-tune, or a publicly disclosed architectural change) brings that model forward in the queue. In the subscriber portal, DEFCON ratings and domain scores reflect the most recent completed evaluation published to you. The model cards on the public homepage show illustrative sample scores, marked as samples, not the measured ones.

Through our Control Mappings — published as Regulatory Relevance Notes.

For each obligation they state the rule verbatim, which specific S.E.B. measurements bear on it, how strongly, the measured reliability of those measurements including per-domain figures, and an explicit statement of what they do not evidence.

Three things about how they are written. They map relevance, never conformity: the arrow points from our measurement toward an obligation and never back to a verdict about you. Mapping strength is a defined ordinal — Direct, Supporting, Contextual — never a percentage, because how strongly a behavioral test bears on a legal obligation is a judgment, and a decimal would dress an opinion as a calculation. And every row carries a "does not evidence" list, because the fastest way to make a mapping useless is to let a reader over-read it.

The boundary that governs all of it: S.E.B. measures a model, and regulatory obligations attach to a system. You wrap a measured model in your own prompts, data, tooling, guardrails and oversight — none of which we can see. You are the only party who can see your deployment. That is why these are guidance on how to use our data, not a determination about your system, and why they are not a conformity assessment, a certification, or legal advice.

The public page carries the full method, the strength definitions, the measured reliability, every obligation covered, everything we deliberately do not map, and one complete worked example per framework. The remaining per-obligation mappings are included with S.E.B. Access.

Yes. SILT evaluates models from any developer.

That includes frontier labs, enterprise AI providers, and organizations deploying fine-tuned or proprietary systems. Evaluation requests go through the subscriber portal or by contacting info@sentientindexlabs.com. Results can be delivered privately for internal governance use, or published with the standard SILT disclosure protocol.

Internally, yes.

Subscribers may reference published ratings — "rated DEFCON 3 by S.E.B.", for example — in internal documents, provided the reference accurately reflects what we published and attributes it to SILT.

What the Subscriber Agreement does not permit is representing a rating as something it is not. A SILT rating is not a certification, an audit opinion, a conformity assessment, or a regulatory approval, and Section 15 says so at length. Telling a regulator, auditor or customer that using our data establishes compliance is specifically out of bounds.

The distinction is the product. We sell independent measurement, and a measurement that gets repeated as a certificate stops being either.

You get ninety days' written notice, your access stays open for the whole of that period, a pro-rata refund of any prepaid term you have not used, and a complete export of your vault before access ends.

This is written into the Subscriber Agreement rather than promised on a web page, because a governance customer is buying continuity of record and a promise that lives only in marketing copy is not continuity.

Evaluation data already delivered to you stays licensed on the same terms. What discontinuation does not entitle you to is continued evaluation, updated ratings, or a successor product.