Instructions to use cbert33/Agnes-3.0-Flash-FP8-Calibrated with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use cbert33/Agnes-3.0-Flash-FP8-Calibrated with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="cbert33/Agnes-3.0-Flash-FP8-Calibrated", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("cbert33/Agnes-3.0-Flash-FP8-Calibrated", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use cbert33/Agnes-3.0-Flash-FP8-Calibrated with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cbert33/Agnes-3.0-Flash-FP8-Calibrated"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cbert33/Agnes-3.0-Flash-FP8-Calibrated",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/cbert33/Agnes-3.0-Flash-FP8-Calibrated

SGLang

How to use cbert33/Agnes-3.0-Flash-FP8-Calibrated with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "cbert33/Agnes-3.0-Flash-FP8-Calibrated" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cbert33/Agnes-3.0-Flash-FP8-Calibrated",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "cbert33/Agnes-3.0-Flash-FP8-Calibrated" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cbert33/Agnes-3.0-Flash-FP8-Calibrated",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use cbert33/Agnes-3.0-Flash-FP8-Calibrated with Docker Model Runner:
```
docker model run hf.co/cbert33/Agnes-3.0-Flash-FP8-Calibrated
```

Agnes 3.0 Flash FP8 (W8A8, calibrated)

Full FP8 quantization of Agnes-3.0-Flash Preview (33B parameters, multimodal). Weights and activations are FP8 e4m3 with static per-tensor scales, calibrated with LLM Compressor. This gives you both the model and context at FP8. Text, image, and video inputs are preserved, and the chat template is upstream's.

The Sglang patch from the original model checkpoint is included here as well.

Note that this model seems to be naturally uncensored. I ran Heretic's assessment and it gave 0/300 refusals. So no need to abliterate this model. And with that, my usual warning here:

Uncensored model: the language checkpoint has undergone abliteration to reduce refusal behavior. Treat outputs as untrusted, apply application-level safeguards, and do not assume the model will decline harmful requests.

User responsibility: this model is provided without warranty. The creators, uploaders, and maintainers are not responsible or liable for what others generate, publish, deploy, or otherwise do with this abliterated model. Users must operate it responsibly, apply appropriate safeguards, comply with applicable law, and respect third-party rights. This model is for research purposes only and is not intended for production use.

Quantization recipe

Field	Value
Format	`compressed-tensors`, `float-quantized`
Weights	FP8 (e4m3), per-tensor symmetric, static
Activations	FP8 (e4m3), per-tensor symmetric, static, calibrated
Targets	all `Linear` modules except the protected list below
Toolkit	llm-compressor 0.13.0, compressed-tensors 0.18.0
Calibration	`HuggingFaceH4/ultrachat_200k`, split `train_sft`, revision `8049631c405ae6576f93f445c6b8166f76f5505a`
Calibration size	512 samples, max sequence length 2048, batch 1, seed 42

Kept at original precision: lm_head, embed_tokens, the full model.visual tower, the recurrent-state layers of delta attention (conv1d, in_proj_a, in_proj_b), and every MTP weight (shipped unquantized in model-mtp.safetensors).

agnes_quantization_manifest.json records the full build, including the SHA-256 of the calibration prompt ids used, so the run is reproducible piece for piece. recipe.yaml is the machine-readable form of the table above.

Why the quantization runs on a fused checkpoint

The upstream FFN carries two branches per layer: a main branch (width 17,408) and a parallel branch (width 2,048). This checkpoint folds them into a single set of projections per layer (intermediate_size: 19456, parallel_ffn_intermediate_size: 0).

Fusing in bf16 is concatenation and it is exact: gate/up join along the output dimension, down along the input dimension. Static per-tensor FP8 scales cannot be merged after the fact, because each branch carries its own scale and a single fused matrix has room for exactly one. Quantizing the fused layout means every serving tensor is quantized once, from the values the model will actually run, and calibration observes the same matrix geometry as inference.

Numerics

The fused layout measures 5.9e-4 full-vocabulary KL against the two-branch layout at identical weights. The residual comes from reduction order in the fused projections, and it sits below the 6.5e-4 that independent inference stacks produce from the same unquantized checkpoint. Perplexity moves 17.05 to 17.07 across the fusion.

The FP8 conversion itself has no formal before/after eval on this card. Run your own benchmarks before trusting it for a workload.

Files

Group	Contents
Weights	`model-00001-of-00002.safetensors`, `model-00002-of-00002.safetensors`, `model-mtp.safetensors`, `model.safetensors.index.json`
Config	`config.json`, `generation_config.json`, custom `*_agnes.py` modeling and processor code
Tokenizer	`tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`
Provenance	`recipe.yaml`, `agnes_quantization_manifest.json`
Extras	`sglang_patch/`, `serve.sh`

The checkpoint is standard compressed-tensors float-quantized, so any engine that reads that format can load it.

Upstream and license

Derived from the Agnes-AI/Agnes-3.0-Flash Preview checkpoint at revision 891ce4f9ffb89b22888aa7fcc2bb2f3618867684, Apache 2.0, unchanged. This is a community quantization and is not affiliated with Agnes AI. The open-weight Preview differs from the production/API model, and the upstream benchmark figures below describe the Preview checkpoint before quantization.

The original upstream model card follows, unchanged.

Agnes-3.0-Flash Preview

Model version clarification

This repository contains an earlier open-weight Preview checkpoint of Agnes 3.0 Flash. It is distinct from the newer production/API checkpoint listed on Artificial Analysis. The Preview release has 33B parameters and a context window of 262,144 tokens. The production/API model uses a different checkpoint and configuration, with a 1M-token context window. Its benchmark results should not be attributed to the Preview weights released here. This repository was initially published as Agnes-3.0-Flash without the Preview suffix. The model card now explicitly identifies this release as Agnes-3.0-Flash Preview to clarify the distinction between the open-weight release and the production/API model. The specifications and Agnes benchmark results below refer to the Preview checkpoint. Hello! 👋 Today we are introducing Agnes-3.0-Flash Preview, an open-weights multimodal preview model built for people who want flagship-class reasoning without flagship-class hardware. Highlights:

Competitive across core capabilities. Agnes-3.0-Flash Preview posts competitive results across reasoning, coding, and instruction-following evaluations.
Built for demanding work. A 262 144-token context window, adjustable reasoning effort, tool calling, and text, image and video understanding.

Benchmarks

Benchmark scope: The Agnes results in the chart and table below belong to the Agnes-3.0-Flash Preview open-weight checkpoint released in this repository. They are not results for the production/API Agnes 3.0 Flash model listed on Artificial Analysis.

Agnes-3.0-Flash Preview benchmark reference results

The Agnes-3.0-Flash Preview scores in the chart correspond to the open-weight checkpoint released in this repository. Reference results across contemporary models are shown below. The figures were compiled from different sources, harnesses, and model snapshots and do not constitute a controlled head-to-head comparison.

Benchmark	Agnes-3.0-Flash Preview	Qwen3.6-35B-A3B 35B / 3B active	Kimi K2.5 1T / 32B active	Muse Glimmer 30B	Qwen3.5 27B	DeepSeek V4 Flash 0731 284B / 13B active	Qwen3.8 27B	Gemini 3.5 Flash undisclosed	Qwen3.8 Flash Next 125B / 6B active	MiniMax M3 428B / 23B active
IFBench	74.20	64.4	43.7	77.0	75.6	75.8	79.5	76.3	81.3	82.9
SciCode	38.08	35.8	39.6	43.6	39.5	50.3	46.6	53.1	50.6	45.4
GPQA Diamond	85.05	84.1	78.9	83.5	85.8	90.8	90.5	92.2	92.3	92.9
AA-LCR	68.33	66.7	59.0	80.0	72.3	79.7	82.0	81.0	79.7	74.0
AA-Omniscience Accuracy	23.00	18.8	22.9	27.0	20.7	40.4	18.4	51.4	24.5	16.7

Higher is better for every row. Header parameter figures mix total and active counts, and harnesses and snapshot dates differ across sources, so treat cross-column comparisons as reference values rather than a controlled head-to-head evaluation.

Architecture

Agnes-3.0-Flash Preview is a hybrid-attention decoder: three of every four layers run a gated delta rule (recurrent, with per-layer state independent of sequence length), and the fourth runs standard global attention. Only 18 of the 72 layers therefore hold a KV cache that grows with context.


Context length	262 144 tokens
Decoder layers	72 = 54 delta-rule recurrent + 18 global attention, alternating 3 : 1
Hidden size	5120
Global attention	24 query heads / 4 KV heads (6 : 1 GQA), head dim 256; RMS-norm on q and k, sigmoid-gated output
Delta-rule layers	16 key heads / 48 value heads, head dim 128; causal conv (kernel 4) in front, gated RMS-norm; recurrent state in fp32
Feed-forward	SwiGLU, intermediate size 17408; plus a parallel SwiGLU 2048 branch in every layer
Positions	3-axis rotary (text / height / width), interleaved mrope sections 11 : 11 : 10, base 1e7, applied to the first 25 % of each head dim (64 dims)
Vocabulary	248 320
Vision tower	27 layers, hidden 1152, patch 16, 2 × 2 spatial merge, projected to 5120

Quickstart

REMOTE CODE REQUIRED

Agnes-3.0-Flash Preview ships its own model implementation. Always load it with trust_remote_code=True.

Requirements

pip install "transformers>=5.12" torch torchvision accelerate

Tested on transformers 5.12.1. Image and video inputs go through the bundled processor, which needs torchvision.

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
path = "Agnes-AI/Agnes-3.0-Flash"
tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(
    path, dtype="bfloat16", device_map="auto", trust_remote_code=True
)
msgs = [{"role": "user", "content": "请用三句话解释什么是人工智能。"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

Images and video

Image and video inputs go through the bundled processor (also remote code):

from transformers import AutoProcessor
proc = AutoProcessor.from_pretrained(path, trust_remote_code=True)
msgs = [{"role": "user", "content": [{"type": "image", "image": "photo.jpg"},
                                     {"type": "text", "text": "描述这张图。"}]}]
inputs = proc.apply_chat_template(msgs, add_generation_prompt=True, tokenize=True,
                                  return_dict=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256)
print(proc.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Reasoning effort

The chat template exposes three reasoning levels — high (default), medium, low — plus a thinking-off switch:

ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt",
                              reasoning_effort="medium")   # or enable_thinking=False

Tool calling

The chat template renders tool definitions for you. The model emits calls as <tool_call><function=…><parameter=…>, and you feed results back as a tool role message:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Look up current weather for a city",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string", "description": "City name"}},
            "required": ["city"],
        },
    },
}]
msgs = [{"role": "user", "content": "What's the weather in Beijing right now?"}]
ids = tok.apply_chat_template(msgs, tools=tools, add_generation_prompt=True,
                              return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=256)
reply = tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True)

# <tool_call>

# <function=get_weather>

# <parameter=city>

# Beijing

# </parameter>

# </function>

# </tool_call>

# run the tool, append the result, generate the final answer
msgs += [{"role": "assistant", "content": reply},
         {"role": "tool", "content": "Clear, 26°C, light northeasterly wind"}]

Over the OpenAI API pass tools= the same way. The server returns the text above verbatim by default; to get structured tool_calls, configure sglang with a tool-call parser matching this format (likewise a reasoning parser, if you want the thinking span in reasoning_content).

SGLang

serve.sh starts a server from a stock public image, overlaying three files onto the image's sglang package and nothing else. See sglang_patch/README.md.

docker run --gpus all --shm-size 64g -p 30001:8080 \
    -v /path/to/agnes-3.0-flash:/model \
    lmsysorg/sglang:nightly-dev-20260908-20ca564b \
    bash /agnes-3.0-flash/serve.sh --served-model-name Agnes-3.0-Flash

serve.sh forwards extra command-line arguments to sglang, which is how --served-model-name takes effect; --tp 2 works the same way. The server listens on port 8080 inside the container:

from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:30001/v1")
response = client.chat.completions.create(
    model="Agnes-3.0-Flash",
    messages=[{"role": "user", "content": "Design a fault-tolerant event processing architecture."}],
    temperature=1.0,
    max_tokens=2000,
)
print(response.choices[0].message.content)

Pass stream=True for streaming; tools= and reasoning_effort= are accepted the same way.

Hardware Requirements

Resource	Recommendation
GPUs	1 × NVIDIA H200 141 GB or NVIDIA H100 80 GB (or equivalent) at bf16
Tensor parallel	`--tp 1`; `--tp 2` for maximum context and concurrency
Weights on disk	Approximately 66 GB for the bf16 checkpoint
Host memory	128 GB or more recommended

Actual context length and concurrency depend on KV-cache allocation, runtime overhead, and tensor-parallel configuration; validate the target workload on the intended hardware.

Recommended Inference Settings

Setting	Recommended
`temperature`	1.0
`top_p`	0.95
`top_k`	20
`reasoning_effort`	`high` for hard reasoning, `low` for latency-sensitive traffic
`max_tokens`	2000 or higher

These are the checkpoint's own generation_config.json defaults.

Model Capabilities

Capability	Support
Advanced reasoning	Yes, with `high` / `medium` / `low` effort levels
Coding and debugging	Yes
Long-context analysis	262 144 tokens
Image understanding	Yes
Video understanding	Yes
Tool calling	Yes (`<tool_call>` / `<tool_response>`)
Streaming	Yes
OpenAI-compatible APIs	Chat Completions via sglang

License

Released under the Apache License 2.0.

Citation

@misc{agnes30flash2026,
  title        = {Agnes-3.0-Flash Preview},
  author       = {{Agnes AI}},
  year         = {2026},
  month        = sep,
  howpublished = {Open-weights preview checkpoint},
  url          = {https://agnes-ai.com/}
}

Downloads last month: 17

Safetensors

Model size

33B params

Tensor type

BF16

F8_E4M3