DiffusionOPSD: On-Policy Self-Distillation in Diffusion Models

Reward-guided diffusion post-training through explicit, continually refreshed intermediate targets

Overview

DiffusionOPSD is an on-policy self-distillation framework for reward-guided diffusion post-training. A frozen behavior policy collects on-policy denoising states and clean-output anchors; differentiable reward gradients construct bounded positive and negative targets around each anchor; and the trainable policy fits these detached targets before an EMA update refreshes the behavior policy.

By turning image-level rewards into explicit, continually refreshed intermediate supervision, DiffusionOPSD makes target construction and finite realization separately observable. Across SD3.5-M and Z-Image-Turbo, it achieves the best final held-out score in 19 of 20 reward-matched settings and reduces training GPU-hours relative to DiffusionNFT by 40% and 63%, respectively.

Released Checkpoints

DiffusionOPSD* (v2): new.zip

new.zip contains the checkpoints for DiffusionOPSD*, the accelerated v2 recipe. It keeps the same target-construction principle and uses one reward-ascent step, broader prompt coverage, and rollouts that stop at the query. See the v2 blog for the update.

The archive contains eight rank-32 LoRA adapters, covering SD3.5-M and Z-Image-Turbo × HPSv3, HPSv2.1, CLIPScore, and PickScore. All are the final checkpoint-100 EMA inference weights. The archive is 1.46 GB.

hf download WeiChow/DiffusionOPSD new.zip --local-dir checkpoints/diffusionopsd-star
unzip checkpoints/diffusionopsd-star/new.zip -d checkpoints/diffusionopsd-star

The extracted opsd_v2_2x4_20261012/ directory contains:

Backbone	Adapter folders
SD3.5-M	sd35-m-hpsv3, sd35-m-hpsv2, sd35-m-clipscore, sd35-m-pickscore
Z-Image-Turbo	z-image-turbo-hpsv3, z-image-turbo-hpsv2, z-image-turbo-clipscore, z-image-turbo-pickscore

Each folder contains adapter_config.json and adapter_model.safetensors.

Original DiffusionOPSD adapters

The nine top-level adapter folders below contain the original DiffusionOPSD checkpoints. The DiffusionOPSD* checkpoints are packaged separately in new.zip.

Checkpoint	Backbone	Training objective
sd35-m-hpsv2	Stable Diffusion 3.5 Medium	HPSv2.1
sd35-m-pickscore	Stable Diffusion 3.5 Medium	PickScore
sd35-m-clipscore	Stable Diffusion 3.5 Medium	CLIPScore
sd35-m-hpsv3	Stable Diffusion 3.5 Medium	HPSv3
z-image-turbo-hpsv2	Z-Image-Turbo	HPSv2.1
z-image-turbo-pickscore	Z-Image-Turbo	PickScore
z-image-turbo-clipscore	Z-Image-Turbo	CLIPScore
z-image-turbo-hpsv3	Z-Image-Turbo	HPSv3
z-image-turbo-pointwise	Z-Image-Turbo	Pointwise reward

Download the repository, including new.zip, with:

hf download WeiChow/DiffusionOPSD --local-dir checkpoints/diffusionopsd

Resources

• Paper: On-Policy Self-Distillation in Diffusion Models
• Code: worldbench/DiffusionOPSD
• Project page: diffusionopsd.github.io
• V2 blog: DiffusionOPSD v2

Please refer to the GitHub repository for installation, inference, evaluation, and training instructions.

Citation

@article{zhou2026policy,
  title={On-Policy Self-Distillation in Diffusion Models},
  author={Zhou, Wei and Zhu, Xiongwei and Kong, Lingdong and Chen, Bo and Zhang, Lei and Liang, Yongyuan and Hou, Xiaoxia and Tian, Ye and Sun, Xian and Wang, Yingshuo and others},
  journal={arXiv preprint arXiv:2608.24646},
  year={2026}
}

License

The released adapters are provided under the Apache License 2.0. Users must also comply with the licenses of the corresponding base models.