Rebuilt with wider output-side initialization (std=0.2 on embed_tokens, embed_tokens_per_layer and the vision/audio embedding projections) so the logits no longer sit near a constant vector.
Measured on an image+text prompt, previous revision -> this one:
logit std 0.113 -> 1.130
min per-position std 0.113 -> 1.126
distinct argmax positions 24/273 -> 65/273
sensitivity to the image 0.060 -> 4.66
NaN / Inf 0 / 0 -> 0 / 0
Config fixes toward the original architecture: num_key_value_heads 2->1 (the original is multi-query), text global_head_dim 8->16 (the original keeps head_dim != global_head_dim), vision global_head_dim 64->8 (was left at the full-size value while head_dim was 8), hidden_size_per_layer_input 1->4, vision default_output_length 70->280.
model_type, architectures, sub-config types, special token ids, vocab_size and the layer counts (text 3 / vision 1 / audio 1) are unchanged, so the expected INT8 node counts in te…
- Align patch_size between image and video processing (b3d10b7f64b43ffec4a67c3adb04e2a70b5f34ef)
Co-authored-by: Iaroslav Tarkan <yatarkan@users.noreply.huggingface.co>
