SynthDeception probes
Activation probes trained on SynthDeception, in four architectures (linear, MLP, EFC (early-fusion covariance), and axial) for nine open-weight models. Each probe reads a model's residual-stream activations at six decoder layers and returns one score for the model's answer to a follow-up question. A higher score means the answer is more likely dishonest. The same architectures trained on FIBS are in probe-inference-weights.
Load them with the probe-inference package
(AlignmentResearch/fibs-probes):
from probe_inference import load_probe_from_hub
probe = load_probe_from_hub(
"ck-belief/qwen3.5-9b/efc",
repo_id="AlignmentResearch/synth-deception-probes",
)
score = probe.score(acts, probe.read_mask(prompt_mask, completion_mask, followup_start_position))
Layout
<training-set>/<model>/<arch>/, with <arch> in linear, mlp, efc, and axial. Below the training set, the
layout and the file formats are those of probe-inference-weights. Each directory is one trained probe with its
probe_metadata.json. A linear or MLP probe is one small probe per layer (layer_<L>/), and an EFC or axial probe is
one module that reads all its layers at once.
A training set is named for the follow-up focuses it contains. A focus is the kind of fact that a follow-up question
asks about: common knowledge (ck), the model's belief (belief), or its intent (intent).
| Training set | Training rows | Validation rows |
|---|---|---|
ck, belief, intent |
4,656 | 400 |
ck-belief, ck-intent, belief-intent |
9,312 | 800 |
ck-belief-intent |
13,968 | 1,200 |
| Directory | Model (revision) | Layers |
|---|---|---|
qwen3.5-2b |
Qwen/Qwen3.5-2B (15852e8c) |
7, 10, 13, 16, 19, 22 |
qwen3.5-9b |
Qwen/Qwen3.5-9B (c2022362) |
10, 13, 18, 21, 26, 29 |
qwen3.5-27b |
Qwen/Qwen3.5-27B (fc05daec) |
19, 27, 35, 43, 51, 58 |
qwen3.5-122b-a10b |
Qwen/Qwen3.5-122B-A10B (dc4d3484) |
14, 20, 26, 32, 38, 43 |
qwen3.5-397b-a17b |
Qwen/Qwen3.5-397B-A17B (84726181) |
18, 25, 33, 40, 48, 54 |
nemotron-3-nano-30b-a3b |
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 (bf77c317) |
16, 22, 29, 35, 42, 47 |
nemotron-3-super-120b-a12b |
nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-BF16 (2dc98e2a) |
26, 37, 48, 59, 70, 79 |
nemotron-3-ultra-550b-a55b |
nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 (77df655d) |
32, 45, 59, 72, 86, 97 |
kimi-k3 |
moonshotai/Kimi-K3 (f831ab66) |
28, 39, 51, 62, 74, 84 |
Linear and MLP scores are probabilities: the mean of the per-layer sigmoids over the layers in
layer_rule.used_layers. These are the layers whose training loss, averaged over the last five logged steps, is below
0.6, or all six layers when none is. The used layers differ from probe to probe, from one layer to all six. EFC and
axial scores are logits.
Training
Every probe was trained on SynthDeception, on the model's own activations over its conversations and follow-up answers, with the recipe of Telling FIBS:
- the dataset's 1,264 scenarios are split into 1,164 for training and 100 for validation. The split is the same for every training set and every model, so the validation rows come from scenarios that no probe trained on;
- seed 0;
- linear and MLP: up to 6 epochs; EFC and axial: up to 4 epochs. Training stops after 4 checkpoints (one epoch) in a row without a lower validation loss;
- the published probe is the checkpoint with the lowest validation loss.
Metrics
Each probe directory holds a metrics.json. metrics.csv, at the top of the repository, has the same values with one
row per probe.
| Field | Meaning |
|---|---|
validation_auroc |
The AUROC of the probe's score on its validation rows |
per_layer_validation_auroc |
Linear and MLP only: the AUROC of each layer's probe on the validation rows |
mean_per_layer_validation_auroc |
Linear and MLP only: the mean of the six per-layer AUROCs |
used_layers |
Linear and MLP only: the layers that the score averages |
checkpoint_step |
The training step of the published checkpoint |
stopped_early |
Whether training stopped before its last epoch |
training_rows, validation_rows |
The rows the probe trained and was validated on |
shade_arena |
The AUROC on SHADE-Arena, with its 95% interval, under the protocol of Telling FIBS |
bash_arena |
The same on BashArena, for qwen3.5-397b-a17b, nemotron-3-ultra-550b-a55b, and kimi-k3 only |
Each shade_arena and bash_arena entry also gives its follow-up question, its score, the episodes, and the positive
and negative steps it covers. SHADE-Arena uses the 14 scenarios of Telling FIBS (scenario_set). The score is the probe's
output on a prefilled "No." answer minus its output on "Yes.": a difference of probabilities for linear and MLP, and of
logits for EFC and axial. The intervals come from 1,000 bootstrap resamples of the episodes.
One probe, ck-intent/qwen3.5-2b/axial, collapsed in training: its validation loss is that of a constant prediction
(ln 2), and its scores barely vary from row to row, so its AUROCs measure noise.
Activations the probes expect
The same as for probe-inference-weights: layer k is the output of decoder block k (Hugging Face
hidden_states[k + 1]) in bfloat16, captured with vLLM, and the probes run in float32. Linear and MLP read the token
before the final end-of-turn token. EFC and axial read every token from the start of the final user turn through the
end-of-turn token. See that card for the Kimi K3 details.
Licences and attribution
The probe weights and this card are released by FAR AI, Inc. under the MIT licence (LICENSE). They are derived from
the models above, under the same licences as probe-inference-weights, whose licence files and NOTICE ship here
unchanged.
Citation
@misc{farai_synth_deception_probes,
author = {FAR.AI Deception Pod},
title = {SynthDeception probes},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/AlignmentResearch/synth-deception-probes}
}
@dataset{farai_synth_deception,
author = {FAR.AI Deception Pod},
title = {SynthDeception},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/datasets/AlignmentResearch/synth-deception}
}
@misc{hollinsworth2026tellingfibsprobeseffectively,
title={Telling FIBS: Probes Effectively Detect Sabotage and Catch Unverbalized Deception},
author={Oskar J. Hollinsworth and Alex F. Spies and Tigist Diriba and Adam Gleave and Chris Cundy},
year={2026},
eprint={2610.12445},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2610.12445},
}
Model tree for AlignmentResearch/synth-deception-probes
Base model
Qwen/Qwen3.5-122B-A10B