Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text
+assets/benchmark-streaming.png filter=lfs diff=lfs merge=lfs -text
+assets/architecture.png filter=lfs diff=lfs merge=lfs -text
+assets/logo.png filter=lfs diff=lfs merge=lfs -text

KV_QUANTIZATION.md

@@ -0,0 +1,8 @@
+# MOSS-VL quantized KV-cache variant
+
+- Weights: bitsandbytes NF4 W4A16
+- KV cache: HQQ 8-bit
+- Quantization group size: 64
+- BF16 residual window: 128 tokens per layer
+- Vision/cross-attention weights remain BF16.
+- Requires the quant venv with the selected KV backend installed.

QUANTIZATION.md

@@ -0,0 +1,7 @@
+# MOSS-VL Streaming quantized checkpoint
+
+- Variant: `w4_keep_ends`
+- Method: `bitsandbytes NF4 W4A16 keep-first-last-4`
+- Compute dtype: `bfloat16`
+- Quantized Linear modules: `240`
+- Vision tower, cross-attention, embeddings, lm_head and norms remain BF16.

README.md

@@ -1,3 +1,196 @@
 ---
 license: apache-2.0
+language:
+- en
+- zh
+library_name: transformers
+pipeline_tag: video-text-to-text
+base_model: OpenMOSS-Team/MOSS-VL-Realtime
+tags:
+- MOSS-VL
+- realtime
+- streaming
+- video-understanding
+- bitsandbytes
+- NF4
+- quantized
+- custom_code
 ---
+
+# MOSS-VL-Realtime W4A16 NF4 + KV8 HQQ
+
+This is the 24 GiB quantized release of MOSS-VL-Realtime. It keeps the original
+timestamp-aware streaming interface and can also use the offline image/video
+helpers from the standard checkpoint.
+
+## Quantization profile
+
+| Component | Format |
+| --- | --- |
+| Most language layers | bitsandbytes NF4 4-bit weights with double quantization |
+| First/last language layers and multimodal modules | BF16 |
+| Activations and compute | BF16 |
+| Transformers KV cache | HQQ INT8 |
+| Attention backend | FlashAttention 2 |
+
+The checkpoint carries its bitsandbytes configuration, HQQ cache configuration,
+and MOSS-VL remote modeling code. Load the directory directly; do not add a
+second runtime quantization configuration.
+
+## Hardware requirements
+
+The model is designed to run on a single NVIDIA GPU with 24 GB of VRAM. Use
+FlashAttention 2 and `frame_queue_size=1` for the 24 GB realtime profile.
+
+## Environment
+
+### Installation
+
+Use the standard MOSS-VL repository requirements, then add the two quantization
+backends required by this checkpoint:
+
+```bash
+git clone https://github.com/OpenMOSS/MOSS-VL.git
+cd MOSS-VL
+
+conda create -n moss_vl_quant python=3.12 pip -y
+conda activate moss_vl_quant
+pip install -i https://pypi.org/simple --no-build-isolation -r requirements.txt
+pip install -i https://pypi.org/simple \
+  bitsandbytes==0.49.2 \
+  hqq==0.2.8.post1
+python -m pip check
+```
+
+The standard release environment uses the following core stack:
+
+| Package | Version |
+| --- | --- |
+| Python | 3.12 |
+| PyTorch | 2.8.0 + CUDA 12.8 |
+| Transformers | 4.57.1 |
+| Accelerate | 1.12.0 |
+| FlashAttention | 2.8.1 |
+| bitsandbytes | 0.49.2 |
+| HQQ | 0.2.8.post1 |
+
+Video decoding also requires FFmpeg to be available in `PATH`.
+
+## Load the model
+
+Keep `attn_implementation` set to `flash_attention_2` for the 24 GB profile.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoProcessor
+
+checkpoint = "/path/to/mossvl_streaming_w4a16_nf4_keep_first4_last4_kv8_hqq"
+
+processor = AutoProcessor.from_pretrained(
+    checkpoint,
+    trust_remote_code=True,
+    frame_extract_num_threads=1,
+)
+model = AutoModelForCausalLM.from_pretrained(
+    checkpoint,
+    trust_remote_code=True,
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+    attn_implementation="flash_attention_2",
+)
+model.eval()
+```
+
+`generation_config.json` automatically enables the HQQ INT8 KV cache. Do not
+override it with a BF16/dynamic cache when using the 24 GB profile.
+
+## Realtime inference
+
+The application supplies PIL-compatible frames with non-decreasing timestamps.
+Use `frame_queue_size=1` for the 24 GB realtime profile.
+
+```python
+import time
+from PIL import Image
+
+session = model.create_realtime_session(
+    processor,
+    initial_prompt=(
+        "Describe important changes in the video as they happen. "
+        "Stay silent when there is no meaningful update."
+    ),
+    frame_queue_size=1,
+    max_tokens_per_turn=12,
+    max_new_tokens=4096,
+    do_sample=False,
+)
+
+frame_paths = [
+    "data/frame_0001.jpg",
+    "data/frame_0002.jpg",
+    "data/frame_0003.jpg",
+]
+
+try:
+    session.start()
+    for index, frame_path in enumerate(frame_paths):
+        image = Image.open(frame_path).convert("RGB")
+        session.push_frame(image, timestamp=float(index))
+
+        while True:
+            chunk = session.poll_output(timeout=0.0)
+            if chunk is None:
+                break
+            print(chunk, end="", flush=True)
+
+        time.sleep(1.0)
+
+    session.push_prompt("What changed in the latest frames?")
+    deadline = time.monotonic() + 5.0
+    while time.monotonic() < deadline:
+        chunk = session.poll_output(timeout=0.1)
+        if chunk is not None:
+            print(chunk, end="", flush=True)
+finally:
+    session.close()
+```
+
+One model instance supports one active realtime session. The model may emit
+control tokens such as `<|silence|>`, `<|round_start|>`, and `<|round_end|>`;
+applications should filter or render them according to their protocol.
+
+## Offline video inference
+
+```python
+text = model.offline_video_generate(
+    processor,
+    prompt="Describe this video.",
+    video="data/example_video.mp4",
+    shortest_edge=4096,
+    longest_edge=16777216,
+    video_max_pixels=201326592,
+    patch_size=16,
+    temporal_patch_size=1,
+    merge_size=2,
+    video_fps=1.0,
+    min_frames=1,
+    max_frames=256,
+    num_extract_threads=4,
+    image_mean=[0.5, 0.5, 0.5],
+    image_std=[0.5, 0.5, 0.5],
+    max_new_tokens=256,
+    temperature=1.0,
+    top_k=50,
+    top_p=1.0,
+    repetition_penalty=1.0,
+    do_sample=False,
+    vision_chunked_length=64,
+)
+print(text)
+```
+
+## Configuration files
+
+- `config.json`: model and NF4 weight configuration.
+- `generation_config.json`: HQQ KV8 configuration.
+- `modeling_moss_vl.py`: checkpoint-local MOSS-VL and QuantizedCache code.

added_tokens.json

@@ -0,0 +1,32 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|response|>": 151672,
+  "<|silence|>": 151671,
+  "<|time_end|>": 151670,
+  "<|time_start|>": 151669,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.jinja

@@ -0,0 +1,120 @@
+{%- if tools %}
+    {{- '<|im_start|>system\n' }}
+    {%- if messages[0].role == 'system' %}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n\n' }}
+    {%- endif %}
+    {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {{- '<|im_start|>system\n' }}
+        {%- if messages[0].content is string %}
+            {{- messages[0].content }}
+        {%- else %}
+            {%- for content in messages[0].content %}
+                {%- if 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- for message in messages %}
+    {%- if message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|image|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|video|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "assistant" %}
+        {{- '<|im_start|>' + message.role + '\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content_item in message.content %}
+                {%- if 'text' in content_item %}
+                    {{- content_item.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {%- if message.tool_calls %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if (loop.first and message.content) or (not loop.first) %}
+                    {{- '\n' }}
+                {%- endif %}
+                {%- if tool_call.function %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {{- '<tool_call>\n{"name": "' }}
+                {{- tool_call.name }}
+                {{- '", "arguments": ' }}
+                {%- if tool_call.arguments is string %}
+                    {{- tool_call.arguments }}
+                {%- else %}
+                    {{- tool_call.arguments | tojson }}
+                {%- endif %}
+                {{- '}\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {%- if message.content is string %}
+            {{- message.content }}
+        {%- else %}
+            {%- for content in message.content %}
+                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}
+                    <|image|>
+                {%- elif content.type == 'video' or 'video' in content %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}
+                    <|video|>
+                {%- elif 'text' in content %}
+                    {{- content.text }}
+                {%- endif %}
+            {%- endfor %}
+        {%- endif %}
+        {{- '\n</tool_response>' }}
+        {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+{%- endif %}

chat_template.json

@@ -0,0 +1,3 @@
+{
+    "chat_template": "{%- if tools %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if messages[0].role == 'system' %}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within <tools></tools> XML tags:\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\\n\\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\\n<tool_call>\\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-json-object>}\\n</tool_call><|im_end|>\\n\" }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {{- '<|im_start|>system\\n' }}\n        {%- if messages[0].content is string %}\n            {{- messages[0].content }}\n        {%- else %}\n            {%- for content in messages[0].content %}\n                {%- if 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- for message in messages %}\n    {%- if message.role == \"user\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|image|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|video|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {{- '<|im_start|>' + message.role + '\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content_item in message.content %}\n                {%- if 'text' in content_item %}\n                    {{- content_item.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {%- if message.tool_calls %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if (loop.first and message.content) or (not loop.first) %}\n                    {{- '\\n' }}\n                {%- endif %}\n                {%- if tool_call.function %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {{- '<tool_call>\\n{\"name\": \"' }}\n                {{- tool_call.name }}\n                {{- '\", \"arguments\": ' }}\n                {%- if tool_call.arguments is string %}\n                    {{- tool_call.arguments }}\n                {%- else %}\n                    {{- tool_call.arguments | tojson }}\n                {%- endif %}\n                {{- '}\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {%- if message.content is string %}\n            {{- message.content }}\n        {%- else %}\n            {%- for content in message.content %}\n                {%- if content.type == 'image' or 'image' in content or 'image_url' in content %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                    {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif -%}\n                    <|image|>\n                {%- elif content.type == 'video' or 'video' in content %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                    {%- if add_vision_id %}Video {{ video_count.value }}: {% endif -%}\n                    <|video|>\n                {%- elif 'text' in content %}\n                    {{- content.text }}\n                {%- endif %}\n            {%- endfor %}\n        {%- endif %}\n        {{- '\\n</tool_response>' }}\n        {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n{%- endif %}\n"
+}

config.json

@@ -0,0 +1,111 @@
+{
+  "architectures": [
+    "MossVLForConditionalGeneration"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_moss_vl.MossVLConfig",
+    "AutoModel": "modeling_moss_vl.MossVLForConditionalGeneration",
+    "AutoModelForCausalLM": "modeling_moss_vl.MossVLForConditionalGeneration"
+  },
+  "dtype": "bfloat16",
+  "image_token_id": 151655,
+  "model_type": "moss_vl",
+  "quantization_config": {
+    "_load_in_4bit": true,
+    "_load_in_8bit": false,
+    "bnb_4bit_compute_dtype": "bfloat16",
+    "bnb_4bit_quant_storage": "uint8",
+    "bnb_4bit_quant_type": "nf4",
+    "bnb_4bit_use_double_quant": true,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": [
+      "model.visual",
+      "cross_attn",
+      "lm_head",
+      "model.language_model.layers.0",
+      "model.language_model.layers.1",
+      "model.language_model.layers.2",
+      "model.language_model.layers.3",
+      "model.language_model.layers.44",
+      "model.language_model.layers.45",
+      "model.language_model.layers.46",
+      "model.language_model.layers.47"
+    ],
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": true,
+    "load_in_8bit": false,
+    "quant_method": "bitsandbytes"
+  },
+  "text_config": {
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "bos_token_id": 151643,
+    "cross_attention_layers": [
+      2,
+      6,
+      10,
+      14,
+      18,
+      22,
+      26,
+      30,
+      34,
+      38,
+      42,
+      46
+    ],
+    "dtype": "bfloat16",
+    "eos_token_id": 151645,
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 4096,
+    "initializer_range": 0.02,
+    "intermediate_size": 12288,
+    "max_position_embeddings": 262144,
+    "model_type": "moss_vl_text",
+    "num_attention_heads": 32,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 8,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        24,
+        20,
+        20
+      ],
+      "rope_type": "default"
+    },
+    "rope_theta": 5000000,
+    "use_cache": true,
+    "vocab_size": 151936
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "4.57.1",
+  "video_token_id": 151656,
+  "vision_config": {
+    "deepstack_visual_indexes": [
+      8,
+      16,
+      24
+    ],
+    "depth": 27,
+    "dtype": "bfloat16",
+    "hidden_act": "gelu_pytorch_tanh",
+    "hidden_size": 1152,
+    "in_channels": 3,
+    "initializer_range": 0.02,
+    "intermediate_size": 4304,
+    "model_type": "moss_vl_vision",
+    "num_heads": 16,
+    "num_position_embeddings": 2304,
+    "out_hidden_size": 4096,
+    "patch_size": 16,
+    "spatial_merge_size": 2,
+    "temporal_patch_size": 1
+  },
+  "vision_end_token_id": 151653,
+  "vision_seq_pad_multiple": 1,
+  "vision_start_token_id": 151652
+}

configuration_moss_vl.py

@@ -0,0 +1,164 @@
+# coding=utf-8
+# Copyright 2025 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""MossVL model configuration"""
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.modeling_rope_utils import rope_config_validation
+from transformers.utils import logging
+
+logger = logging.get_logger(__name__)
+
+
+class MossVLVisionConfig(PretrainedConfig):
+    """
+    Configuration for MossVL Vision Model
+    """
+
+    model_type = "moss_vl_vision"
+    base_config_key = "vision_config"
+
+    def __init__(
+        self,
+        depth=27,
+        hidden_size=1152,
+        hidden_act="gelu_pytorch_tanh",
+        intermediate_size=4304,
+        num_heads=16,
+        in_channels=3,
+        patch_size=16,
+        spatial_merge_size=2,
+        temporal_patch_size=1,
+        out_hidden_size=3584,
+        num_position_embeddings=2304,
+        deepstack_visual_indexes=[8, 16, 24],
+        initializer_range=0.02,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.depth = depth
+        self.hidden_size = hidden_size
+        self.hidden_act = hidden_act
+        self.intermediate_size = intermediate_size
+        self.num_heads = num_heads
+        self.in_channels = in_channels
+        self.patch_size = patch_size
+        self.spatial_merge_size = spatial_merge_size
+        self.temporal_patch_size = temporal_patch_size
+        self.out_hidden_size = out_hidden_size
+        self.num_position_embeddings = num_position_embeddings
+        self.initializer_range = initializer_range
+        self.deepstack_visual_indexes = deepstack_visual_indexes
+
+
+class MossVLTextConfig(PretrainedConfig):
+    """
+    Configuration for MossVL Text Model
+    """
+
+    model_type = "moss_vl_text"
+    base_config_key = "text_config"
+
+    def __init__(
+        self,
+        vocab_size=151936,
+        hidden_size=4096,
+        intermediate_size=22016,
+        num_hidden_layers=32,
+        num_attention_heads=32,
+        num_key_value_heads=32,
+        head_dim=128,
+        hidden_act="silu",
+        max_position_embeddings=128000,
+        initializer_range=0.02,
+        rms_norm_eps=1e-6,
+        use_cache=True,
+        tie_word_embeddings=False,
+        rope_theta=5000000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        # Cross attention specific
+        cross_attention_layers=None,  # List of layer indices to insert cross attention
+        **kwargs,
+    ):
+
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+
+        # for backward compatibility
+        if num_key_value_heads is None:
+            num_key_value_heads = num_attention_heads
+
+        self.num_key_value_heads = num_key_value_heads
+        self.head_dim = head_dim
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+
+        rope_config_validation(self, ignore_keys={"mrope_section", "mrope_interleaved"})
+        self.cross_attention_layers = cross_attention_layers or [2, 6, 10, 14, 18, 22, 26, 30, 34, 38, 42, 46]
+        super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)
+
+class MossVLConfig(PretrainedConfig):
+    """
+    Configuration for MossVL Model
+    """
+
+    model_type = "moss_vl"
+    sub_configs = {"vision_config": MossVLVisionConfig, "text_config": MossVLTextConfig}
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    def __init__(
+        self,
+        text_config=None,
+        vision_config=None,
+        image_token_id=151655,
+        video_token_id=151656,
+        vision_start_token_id=151652,
+        vision_end_token_id=151653,
+        vision_seq_pad_multiple=8,
+        tie_word_embeddings=False,
+        **kwargs,
+    ):
+        if isinstance(vision_config, dict):
+            self.vision_config = self.sub_configs["vision_config"](**vision_config)
+        elif vision_config is None:
+            self.vision_config = self.sub_configs["vision_config"]()
+
+        if isinstance(text_config, dict):
+            self.text_config = self.sub_configs["text_config"](**text_config)
+        elif text_config is None:
+            self.text_config = self.sub_configs["text_config"]()
+
+        self.image_token_id = image_token_id
+        self.video_token_id = video_token_id
+        self.vision_start_token_id = vision_start_token_id
+
+
+        self.vision_end_token_id = vision_end_token_id
+        self.vision_seq_pad_multiple = vision_seq_pad_multiple
+        super().__init__(**kwargs, tie_word_embeddings=tie_word_embeddings)
+        
+
+__all__ = ["MossVLConfig", "MossVLTextConfig"]

generation_config.json

@@ -0,0 +1,15 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "transformers_version": "4.57.1",
+  "cache_implementation": "quantized",
+  "cache_config": {
+    "backend": "hqq",
+    "nbits": 8,
+    "axis_key": 0,
+    "axis_value": 0,
+    "q_group_size": 64,
+    "residual_length": 128
+  }
+}

kv_quantization_report.json

@@ -0,0 +1,15 @@
+{
+  "created_at": "2026-07-26T06:34:21.056464+00:00",
+  "source_weight_checkpoint": "/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/mossvl_streaming_w4a16_nf4_keep_first4_last4",
+  "weight_quantization": "bitsandbytes NF4 W4A16",
+  "kv_cache_quantization": {
+    "backend": "hqq",
+    "nbits": 8,
+    "axis_key": 0,
+    "axis_value": 0,
+    "q_group_size": 64,
+    "residual_length": 128
+  },
+  "storage": "Model shards are hard-linked to the source checkpoint when the filesystem supports hard links.",
+  "compatibility_patch": "Cross-attention materializes the quantized history one layer at a time because the original MOSS-VL code directly reads cache_layer.keys."
+}

model-00001-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0219c7d3d132e72d80cc6f0d04aa20e98edce88dea6d282ace84539c81e8c12
+size 3962258708

model-00002-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12114b03fc2c0735059a5617610848d480b5bfb04fe13569fa7e24776b8b4ee1
+size 3993030849

model-00003-of-00003.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:db7736eefc60465956ece6b3748a5175b60b764375d50d8c13bf67f947a18704
+size 3886708579

preprocessor_config.json

@@ -0,0 +1,44 @@
+{
+  "auto_map": {
+    "AutoImageProcessor": "processing_moss_vl.MossVLImageProcessorFast",
+    "AutoProcessor": "processing_moss_vl.MossVLProcessor"
+  },
+  "crop_size": null,
+  "data_format": "channels_first",
+  "default_to_square": true,
+  "device": null,
+  "disable_grouping": null,
+  "do_center_crop": null,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_pad": null,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "MossVLImageProcessorFast",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "input_data_format": null,
+  "max_pixels": null,
+  "merge_size": 2,
+  "min_pixels": null,
+  "multi_image_max_pixels": 65536000,
+  "pad_size": null,
+  "patch_size": 16,
+  "processor_class": "MossVLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "return_tensors": null,
+  "size": {
+    "longest_edge": 16777216,
+    "shortest_edge": 4096
+  },
+  "temporal_patch_size": 1
+}

processing_moss_vl.py

@@ -0,0 +1,1143 @@
+# coding=utf-8
+# Copyright 2025 The FNLP Vision Team and The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Processor class for Moss-VL.
+"""
+
+from typing import Any, Dict, List, Optional, Union
+
+import numpy as np
+import torch
+from torchvision.transforms.v2 import functional as F
+from PIL import Image
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.image_utils import ImageInput, SizeDict
+from transformers.image_processing_utils_fast import group_images_by_shape, reorder_images
+from transformers.utils import TensorType
+from transformers.processing_utils import (
+    ImagesKwargs,
+    ProcessingKwargs,
+    ProcessorMixin,
+    Unpack,
+    VideosKwargs,
+)
+from transformers.tokenization_utils_base import PreTokenizedInput, TextInput
+from transformers.utils import logging
+from transformers.models.qwen2_vl.image_processing_qwen2_vl_fast import Qwen2VLImageProcessorFast
+from transformers.models.qwen2_vl.image_processing_qwen2_vl import smart_resize
+
+
+logger = logging.get_logger(__name__)
+
+
+class MossVLImageProcessorFast(Qwen2VLImageProcessorFast):
+    """
+    Custom image processor that overrides _preprocess to support multi_image_max_pixels.
+    Inherits from Qwen2VLImageProcessorFast.
+    """
+    # Multi-image batch total pixels limit (read from config)
+    multi_image_max_pixels = None
+    
+
+    def _preprocess(
+        self,
+        images: list["torch.Tensor"],
+        do_resize: bool,
+        size: SizeDict,
+        interpolation: Optional["F.InterpolationMode"],
+        do_rescale: bool,
+        rescale_factor: float,
+        do_normalize: bool,
+        image_mean: Optional[Union[float, list[float]]],
+        image_std: Optional[Union[float, list[float]]],
+        patch_size: int,
+        temporal_patch_size: int,
+        merge_size: int,
+        disable_grouping: Optional[bool],
+        return_tensors: Optional[Union[str, TensorType]],
+        **kwargs,
+    ):
+        """Override _preprocess to use custom smart_resize with batch-level max_pixels.
+        
+        multi_image_max_pixels is treated as a batch-level total budget, proportionally allocated
+        to each image based on its original pixel count. min_pixels remains a per-image
+        constraint. multi_image_max_pixels can be configured separately from longest_edge.
+        """
+        min_pixels = size["shortest_edge"]
+        max_pixels = size["longest_edge"]  # Per-image upper limit
+        # Use multi_image_max_pixels if configured, otherwise fall back to longest_edge
+        multi_image_max_pixels = getattr(self, "multi_image_max_pixels", None) or max_pixels
+        
+        # Calculate total original pixels across all images in the batch
+        # This is used to proportionally allocate max_pixels to each image
+        total_original_pixels = sum(img.shape[-2] * img.shape[-1] for img in images)
+
+        # Group images by size for batched resizing
+        grouped_images, grouped_images_index = group_images_by_shape(images, disable_grouping=disable_grouping)
+        resized_images_grouped = {}
+        for shape, stacked_images in grouped_images.items():
+            height, width = stacked_images.shape[-2:]
+            if do_resize:
+                # Calculate proportional max_pixels for images with this shape
+                # Each image's max_pixels is allocated based on its proportion of total pixels
+                original_pixels = height * width
+                if total_original_pixels > 0:
+                    proportion = original_pixels / total_original_pixels
+                    proportional_max_pixels = int(multi_image_max_pixels * proportion)
+                else:
+                    proportional_max_pixels = multi_image_max_pixels
+                
+                # Ensure proportional max_pixels is within [min_pixels, max_pixels] range
+                # min_pixels: per-image lower limit (shortest_edge)
+                # max_pixels: per-image upper limit (longest_edge)
+                proportional_max_pixels = max(proportional_max_pixels, min_pixels)
+                proportional_max_pixels = min(proportional_max_pixels, max_pixels)
+                
+                resized_height, resized_width = smart_resize(
+                    height,
+                    width,
+                    factor=patch_size * merge_size,
+                    min_pixels=min_pixels,
+                    max_pixels=proportional_max_pixels,
+                )
+                stacked_images = self.resize(
+                    image=stacked_images,
+                    size=SizeDict(height=resized_height, width=resized_width),
+                    interpolation=interpolation,
+                )
+            resized_images_grouped[shape] = stacked_images
+        resized_images = reorder_images(resized_images_grouped, grouped_images_index)
+        
+        # Warn if multi-image batch exceeds multi_image_max_pixels due to min_pixels constraint
+        if len(images) > 1:
+            total_resized_pixels = sum(img.shape[-2] * img.shape[-1] for img in resized_images)
+            if total_resized_pixels > multi_image_max_pixels:
+                logger.warning_once(
+                    f"Multi-image batch total pixels ({total_resized_pixels}) exceeds multi_image_max_pixels ({multi_image_max_pixels}). "
+                    f"This may happen when image_count * min_pixels > multi_image_max_pixels."
+                )
+
+        # Group images by size for further processing
+        # Needed in case do_resize is False, or resize returns images with different sizes
+        grouped_images, grouped_images_index = group_images_by_shape(resized_images, disable_grouping=disable_grouping)
+        processed_images_grouped = {}
+        processed_grids = {}
+        for shape, stacked_images in grouped_images.items():
+            resized_height, resized_width = stacked_images.shape[-2:]
+            # Fused rescale and normalize
+            patches = self.rescale_and_normalize(
+                stacked_images, do_rescale, rescale_factor, do_normalize, image_mean, image_std
+            )
+            if patches.ndim == 4:
+                # add a temporal dimension if we have images
+                patches = patches.unsqueeze(1)
+            if patches.shape[1] % temporal_patch_size != 0:
+                repeats = patches[:, -1:].repeat(1, temporal_patch_size - 1, 1, 1, 1)
+                patches = torch.cat([patches, repeats], dim=1)
+            batch_size, grid_t, channel = patches.shape[:3]
+            grid_t = grid_t // temporal_patch_size
+            grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+
+            patches = patches.view(
+                batch_size,
+                grid_t,
+                temporal_patch_size,
+                channel,
+                grid_h // merge_size,
+                merge_size,
+                patch_size,
+                grid_w // merge_size,
+                merge_size,
+                patch_size,
+            )
+            # Reorder dimensions to group grid and patch information for subsequent flattening.
+            # (batch, grid_t, grid_h, grid_w, merge_h, merge_w, channel, temp_patch_size, patch_h, patch_w)
+            patches = patches.permute(0, 1, 4, 7, 5, 8, 3, 2, 6, 9)
+            flatten_patches = patches.reshape(
+                batch_size,
+                grid_t * grid_h * grid_w,
+                channel * temporal_patch_size * patch_size * patch_size,
+            )
+
+            processed_images_grouped[shape] = flatten_patches
+            processed_grids[shape] = [[grid_t, grid_h, grid_w]] * batch_size
+
+        processed_images = reorder_images(processed_images_grouped, grouped_images_index)
+        processed_grids = reorder_images(processed_grids, grouped_images_index)
+        pixel_values = torch.cat(processed_images, dim=0)
+        image_grid_thw = torch.tensor(processed_grids)
+
+        return BatchFeature(
+            data={"pixel_values": pixel_values, "image_grid_thw": image_grid_thw}, tensor_type=return_tensors
+        )
+
+def _to_numpy(x):
+    """
+    Convert various tensor types to numpy array.
+    Supports torch.Tensor, tf.Tensor, jax.Array, np.ndarray, lists, and primitives.
+    
+    Args:
+        x: Input value that can be a tensor from various frameworks or a Python primitive
+        
+    Returns:
+        np.ndarray: NumPy array representation of the input
+    """
+    # Already numpy
+    if isinstance(x, np.ndarray):
+        return x
+    
+    # Torch tensor or TensorFlow tensor (both have .numpy() method)
+    if hasattr(x, 'numpy'):
+        # For torch tensors on CUDA, need to move to CPU first
+        if hasattr(x, 'cpu'):
+            return x.cpu().numpy()
+        # For TensorFlow or already on CPU
+        return x.numpy()
+    
+    # JAX arrays and other array-like objects that support __array__ protocol
+    if hasattr(x, '__array__'):
+        return np.asarray(x)
+    
+    # Python primitives (list, tuple, int, float)
+    return np.array(x)
+
+
+def _split_array_or_tensor(x, split_indices):
+    """Split along the first dimension while preserving tensor/array type."""
+    split_indices = [int(idx) for idx in split_indices]
+    if isinstance(x, torch.Tensor):
+        if not split_indices:
+            return [x]
+        chunks = []
+        start = 0
+        for end in split_indices:
+            chunks.append(x[start:end])
+            start = end
+        chunks.append(x[start:])
+        return chunks
+    return np.split(x, split_indices)
+
+
+def _concat_array_or_tensor(items, axis=0):
+    """Concatenate while preserving tensor/array type and device."""
+    if not items:
+        return None
+
+    if any(isinstance(item, torch.Tensor) for item in items):
+        ref = next(item for item in items if isinstance(item, torch.Tensor))
+        tensor_items = [
+            item
+            if isinstance(item, torch.Tensor)
+            else torch.as_tensor(item, device=ref.device, dtype=ref.dtype)
+            for item in items
+        ]
+        return torch.cat(tensor_items, dim=axis)
+
+    return np.concatenate(items, axis=axis)
+
+
+def _stack_array_or_tensor(items, axis=0):
+    """Stack while preserving tensor/array type and device."""
+    if not items:
+        return None
+
+    if any(isinstance(item, torch.Tensor) for item in items):
+        ref = next(item for item in items if isinstance(item, torch.Tensor))
+        tensor_items = [
+            item
+            if isinstance(item, torch.Tensor)
+            else torch.as_tensor(item, device=ref.device, dtype=ref.dtype)
+            for item in items
+        ]
+        return torch.stack(tensor_items, dim=axis)
+
+    return np.stack(items, axis=axis)
+
+
+class MossVLImagesKwargs(ImagesKwargs):
+    min_pixels: Optional[int]
+    max_pixels: Optional[int]
+    patch_size: Optional[int]
+    temporal_patch_size: Optional[int]
+    merge_size: Optional[int]
+
+
+
+class MossVLVideosKwargs(VideosKwargs, total=False):
+    video_fps: Optional[Union[int, float]]
+    min_frames: Optional[int]
+    max_frames: Optional[int]
+    num_extract_threads: Optional[int]
+
+
+class MossVLProcessorKwargs(ProcessingKwargs, total=False):
+    images_kwargs: MossVLImagesKwargs
+    videos_kwargs: MossVLVideosKwargs
+    # _defaults = {
+    #     "text_kwargs": {
+    #         "padding": True,                    # 👈 启用 padding
+    #         "padding_side": "left",            # 👈 左 padding
+    #         "pad_to_multiple_of": 8,           # 👈 pad 到 8 的倍数
+    #         "return_token_type_ids": False,
+    #         "return_mm_token_type_ids": False,
+    #     },
+    #     "videos_kwargs": {"return_metadata": True},
+    # }
+    _defaults = {
+        "text_kwargs": {
+            "padding": False,
+            "return_token_type_ids": False,
+            "return_mm_token_type_ids": False,
+        },
+        "videos_kwargs": {"return_metadata": True},
+    }
+
+class MossVLProcessor(ProcessorMixin):
+    r"""
+    Constructs a Moss-VL processor which wraps a Qwen2VL image processor, Moss-VL video processor and a Qwen2 tokenizer
+    into a single processor.
+
+    [`MossVLProcessor`] offers all the functionalities of [`Qwen2VLImageProcessor`], [`MossVLVideoProcessor`] and [`Qwen2TokenizerFast`].
+    See the [`~MossVLProcessor.__call__`] and [`~MossVLProcessor.decode`] for more information.
+
+    Args:
+        image_processor ([`Qwen2VLImageProcessor`], *optional*):
+            The image processor is a required input.
+        tokenizer ([`Qwen2TokenizerFast`], *optional*):
+            The tokenizer is a required input.
+        video_processor ([`MossVLVideoProcessor`], *optional*):
+            The video processor is a required input.
+        chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages
+            in a chat into a tokenizable string.
+    """
+
+    attributes = ["image_processor", "tokenizer", "video_processor"]
+    image_processor_class = "AutoImageProcessor"
+    video_processor_class = "AutoVideoProcessor"
+    tokenizer_class = ("Qwen2Tokenizer", "Qwen2TokenizerFast")
+
+    def __init__(
+        self, 
+        image_processor=None, 
+        tokenizer=None, 
+        video_processor=None, 
+        chat_template=None,
+        **kwargs
+    ):
+        super().__init__(image_processor, tokenizer, video_processor, chat_template=chat_template)
+        
+
+        self.image_token = "<|image_pad|>" if not hasattr(tokenizer, "image_token") else tokenizer.image_token
+        self.video_token = "<|video_pad|>" if not hasattr(tokenizer, "video_token") else tokenizer.video_token
+
+        
+        self.image_token_id = (
+            tokenizer.image_token_id
+            if getattr(tokenizer, "image_token_id", None)
+            else tokenizer.convert_tokens_to_ids(self.image_token)
+        )
+        self.video_token_id = (
+            tokenizer.video_token_id
+            if getattr(tokenizer, "video_token_id", None)
+            else tokenizer.convert_tokens_to_ids(self.video_token)
+        )
+        
+        self.vision_start_token = (
+            "<|vision_start|>" if not hasattr(tokenizer, "vision_start_token") else tokenizer.vision_start_token
+        )
+        self.vision_end_token = (
+            "<|vision_end|>" if not hasattr(tokenizer, "vision_end_token") else tokenizer.vision_end_token
+        )
+
+        # Placeholders used in input text
+        self.image_placeholder = "<|image|>"
+        self.video_placeholder = "<|video|>"
+
+        self.time_start_token = "<|time_start|>"
+        self.time_end_token = "<|time_end|>"
+        
+        # EOS token for labels generation (assistant's response should end with this)
+        self.im_end_token = "<|im_end|>"
+        self.im_end_token_id = tokenizer.convert_tokens_to_ids(self.im_end_token)
+        
+        # Vision-related token ids (all should be masked in labels)
+        self.vision_start_token_id = tokenizer.convert_tokens_to_ids(self.vision_start_token)
+        self.vision_end_token_id = tokenizer.convert_tokens_to_ids(self.vision_end_token)
+        
+        # Token ids that should always be masked in labels (e.g. <|image_pad|>)
+        self.mask_token_ids = {self.image_token_id}
+
+    def __call__(
+        self,
+        text: Union[TextInput, PreTokenizedInput, List[TextInput], List[PreTokenizedInput]] = None,
+        images: ImageInput = None,
+        videos: Union[str, Dict[str, Any], List[Union[str, Dict[str, Any]]]] = None,
+        labels_spans: Optional[Union[List[tuple], List[List[tuple]]]] = None,
+        ignore_index: int = -100,
+        **kwargs: Unpack[MossVLProcessorKwargs],
+    ) -> BatchFeature:
+        """
+        Main method to prepare for the model one or several sequences(s) and image(s)/video(s).
+
+        Args:
+            text (`str`, `list[str]`, `list[list[str]]`):
+                The sequence or batch of sequences to be encoded.
+            images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`):
+                The image or batch of images to be prepared.
+            videos (`str`, `Dict`, `list[str]`, `list[Dict]`):
+                The video or batch of videos to be prepared. Each video can be:
+                - A string path to a video file
+                - A dict with keys:
+                    - "video_path": str, path to the video file
+                    - "segments": list of segments, where each segment is:
+                        - [start, end]: a time segment (left-closed, right-open interval in seconds)
+                        - [time]: a single frame at the specified time (in seconds)
+                  The number of segments should match the number of video placeholders in the text.
+            labels_spans (`list[list[int]]`, `list[list[list[int]]]`, *optional*):
+                Character-level spans indicating assistant regions in original text.
+                Each span is a [start, end] list with inclusive start and exclusive end.
+                Example: [[10, 50], [100, 150]] means characters [10:50) and [100:150) are assistant.
+                Note: Use list (not tuple) for spans as they will be modified in place during processing.
+                When provided, the processor will generate `labels` in the output, where:
+                - Non-assistant tokens have value `ignore_index` (-100 by default)
+                - Image tokens always have value `ignore_index` even in assistant part
+                - Other assistant tokens have their token id as label
+            ignore_index (`int`, *optional*, defaults to -100):
+                Value for masked positions in labels.
+            return_tensors (`str` or [`~utils.TensorType`], *optional*):
+                If set, will return tensors of a particular framework. Acceptable values are:
+                - `'tf'`: Return TensorFlow `tf.constant` objects.
+                - `'pt'`: Return PyTorch `torch.Tensor` objects.
+                - `'np'`: Return NumPy `np.ndarray` objects.
+                - `'jax'`: Return JAX `jnp.ndarray` objects.
+
+
+        Returns:
+            [`BatchFeature`]: A [`BatchFeature`] with the following fields:
+            - **input_ids** -- List of token ids to be fed to a model.
+            - **attention_mask** -- List of indices specifying which tokens should be attended to by the model.
+            - **pixel_values** -- Pixel values to be fed to a model (concatenation of images and videos).
+            - **grid_thw** -- List of grid sizes (t, h, w) for each media item.
+            - **media_nums_per_sample** -- List of number of media items per sample.
+            - **labels** -- (Optional) Labels for training, only present when `labels_spans` is provided.
+        """
+        # Merge kwargs with defaults
+        output_kwargs = self._merge_kwargs(
+            MossVLProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+        
+        # Step 1: Process images if provided
+        if images is not None:
+            images_kwargs = output_kwargs["images_kwargs"].copy()
+            images_kwargs["return_tensors"] = None
+            image_inputs = self.image_processor(images=images, **images_kwargs)
+            image_grid_thw = image_inputs["image_grid_thw"]
+        else:
+            image_inputs = {}
+            image_grid_thw = None
+
+        # Step 2: Process videos if provided
+        if videos is not None:
+            videos_kwargs = output_kwargs["videos_kwargs"].copy()
+            videos_kwargs["return_tensors"] = None
+            videos_inputs = self.video_processor(videos=videos, **videos_kwargs)
+            video_grid_thw = videos_inputs["video_grid_thw"]
+            # If user has not requested video metadata, pop it
+            if "return_metadata" not in kwargs:
+                video_metadata = videos_inputs.pop("video_metadata")
+            else:
+                video_metadata = videos_inputs["video_metadata"]
+        else:
+            videos_inputs = {}
+            video_grid_thw = None
+            video_metadata = None
+
+        # Step 3: Process text with placeholder replacement
+        if text is None or (isinstance(text, str) and len(text.strip()) == 0):
+            raise ValueError("Text input is required for MossVL processor and cannot be empty.")
+        
+        if not isinstance(text, list):
+            text = [text]
+        
+        text = text.copy()  # Copy to avoid in-place modifications
+        
+        # Prepare labels_spans if provided
+        # labels_spans format: List[List[List[int]]] - batch of samples, each sample has multiple spans
+        # Each span is [start, end] (list, not tuple) so it can be modified in place
+        should_create_labels = labels_spans is not None
+        if should_create_labels:
+            # Ensure batch format: convert single sample spans to batch format
+            # Single sample: [[start, end], [start, end], ...]
+            # Batch: [[[start, end], ...], [[start, end], ...], ...]
+            if labels_spans and isinstance(labels_spans[0], list) and len(labels_spans[0]) == 2 and isinstance(labels_spans[0][0], int):
+                labels_spans = [labels_spans]
+
+        # Step 3.0-pre: Check if we need to reorder (when both images and videos exist)
+        # If only one media type exists, we can skip the expensive split+reorder+concat
+        has_images = images is not None and "pixel_values" in image_inputs
+        has_videos = videos is not None and "pixel_values_videos" in videos_inputs
+        needs_reorder = has_images and has_videos
+        
+        image_pixel_values_list = []
+        video_pixel_values_list = []
+        
+        # Step 3.0: Record the order of media in original text (before replacement)
+        # This will be used later to correctly order pixel_values and grid_thw
+        media_order_per_sample = []
+        for i in range(len(text)):
+            media_order = []
+            temp_text = text[i]
+            pos = 0
+            while pos < len(temp_text):
+                img_pos = temp_text.find(self.image_placeholder, pos)
+                vid_pos = temp_text.find(self.video_placeholder, pos)
+                
+                if img_pos == -1 and vid_pos == -1:
+                    break
+                
+                if img_pos != -1 and (vid_pos == -1 or img_pos < vid_pos):
+                    media_order.append(("image", img_pos))
+                    pos = img_pos + len(self.image_placeholder)
+                elif vid_pos != -1:
+                    media_order.append(("video", vid_pos))
+                    pos = vid_pos + len(self.video_placeholder)
+            
+            media_order_per_sample.append(media_order)
+        
+        # Step 3.0.1: Check if any sample has no media (empty samples need blank image)
+        # If there are empty samples, we need to enter slow path to handle them properly
+        has_empty_samples = any(len(order) == 0 for order in media_order_per_sample)
+        if has_empty_samples:
+            needs_reorder = True
+        
+        # Split pixel values for reordering if needed
+        if needs_reorder:
+            if has_images:
+                flat_pixel_values = image_inputs["pixel_values"]
+                flat_grid_thw = image_inputs["image_grid_thw"]
+                # grid_thw is (t, h, w), num_patches = t * h * w
+                patch_counts = [int(np.prod(_to_numpy(grid))) for grid in flat_grid_thw]
+                if len(patch_counts) == 1:
+                    # Single image case: no need to split
+                    image_pixel_values_list = [flat_pixel_values]
+                elif len(patch_counts) > 1:
+                    # Multiple images: split by cumulative counts
+                    split_indices = np.cumsum(patch_counts)[:-1]
+                    image_pixel_values_list = _split_array_or_tensor(
+                        flat_pixel_values, split_indices
+                    )
+
+            if has_videos:
+                flat_video_values = videos_inputs["pixel_values_videos"]
+                flat_video_grid = videos_inputs["video_grid_thw"]
+                video_patch_counts = [int(np.prod(_to_numpy(grid))) for grid in flat_video_grid]
+                if len(video_patch_counts) == 1:
+                    # Single video case: no need to split
+                    video_pixel_values_list = [flat_video_values]
+                elif len(video_patch_counts) > 1:
+                    # Multiple videos: split by cumulative counts
+                    split_indices = np.cumsum(video_patch_counts)[:-1]
+                    video_pixel_values_list = _split_array_or_tensor(
+                        flat_video_values, split_indices
+                    )
+        
+        # Step 3.1: Replace placeholders (simple replacement, no expansion yet)
+        # In MossVL, one image placeholder = one image token
+        # One video placeholder = one video token (will be expanded later)
+        for i in range(len(text)):
+            if should_create_labels:
+                # Replace and update spans for image placeholders
+                text[i], labels_spans[i] = self._replace_and_update_spans(
+                    text[i], self.image_placeholder, self.image_token, labels_spans[i]
+                )
+                # Replace and update spans for video placeholders
+                text[i], labels_spans[i] = self._replace_and_update_spans(
+                    text[i], self.video_placeholder, self.video_token, labels_spans[i]
+                )
+            else:
+                text[i] = text[i].replace(self.image_placeholder, self.image_token)
+                text[i] = text[i].replace(self.video_placeholder, self.video_token)
+        
+        # Step 3.2: Validate token counts 
+        n_images_in_text = [t.count(self.image_token) for t in text]
+        n_videos_in_text = [t.count(self.video_token) for t in text]
+        
+        # Count placeholders in text
+        total_images_in_text = sum(n_images_in_text)
+        total_videos_in_text = sum(n_videos_in_text)
+        
+        # Count actual images and videos provided
+        total_images_provided = len(image_grid_thw) if image_grid_thw is not None else 0
+        total_videos_provided = len(video_grid_thw) if video_grid_thw is not None else 0
+        
+        # Validate image counts
+        if total_images_in_text != total_images_provided:
+            raise ValueError(
+                "Number of image tokens does not match number of images provided. "
+                f"Found {total_images_in_text} image tokens in text and {total_images_provided} images."
+            )
+        
+        # Validate video counts
+        if total_videos_in_text != total_videos_provided:
+            raise ValueError(
+                "Number of video tokens does not match number of videos provided. "
+                f"Found {total_videos_in_text} video tokens in text and {total_videos_provided} videos."
+            )
+        
+        # Step 3.3: Expand video tokens with timestamps
+        # Now expand each video token to multiple tokens (one per frame) with timestamps
+        if video_grid_thw is not None:
+            index = 0
+            for i in range(len(text)):
+                while self.video_token in text[i]:
+                    metadata = video_metadata[index]
+                    if metadata.fps is None:
+                        logger.warning_once(
+                            "MossVL requires frame timestamps to construct prompts, but the `fps` of the input video could not be inferred. "
+                            "Probably `video_metadata` was missing from inputs and you passed pre-sampled frames. "
+                            "Defaulting to `fps=24`. Please provide `video_metadata` for more accurate results."
+                        )
+                        metadata.fps = 24 if metadata.fps is None else metadata.fps
+
+                    # Calculate timestamps
+                    # Use actual_timestamps if available (for segments), otherwise use frames_indices
+                    actual_timestamps = getattr(metadata, 'actual_timestamps', None)
+                    curr_timestamp = self._calculate_timestamps(
+                        metadata.frames_indices,
+                        metadata.total_num_frames,
+                        metadata.fps,
+                        metadata.duration,
+                        self.video_processor.temporal_patch_size,
+                        actual_timestamps=actual_timestamps,
+                    )
+
+                    # Build video placeholder: one video token per frame with timestamp
+                    # video_grid_thw[index][0] is the temporal dimension (number of frames after merging)
+
+                    video_tokens = []
+                    for frame_idx in range(video_grid_thw[index][0]):
+                        curr_time = curr_timestamp[frame_idx]
+                        # Format: <|time_start|>X.X seconds<|time_end|><|image_pad|>
+                        video_tokens.append(
+                            f"{self.time_start_token}{curr_time:.1f} seconds{self.time_end_token}{self.image_token}"
+                        )
+                    
+                    # Wrap the entire video sequence with vision_start and vision_end tokens
+                    video_placeholder = f"{self.vision_start_token}{''.join(video_tokens)}{self.vision_end_token}"
+                    
+                    # Replace the video token with expanded sequence and update spans if needed
+                    if should_create_labels:
+                        text[i], labels_spans[i] = self._replace_and_update_spans(
+                            text[i], self.video_token, video_placeholder, labels_spans[i], replace_count=1
+                        )
+                    else:
+                        text[i] = text[i].replace(self.video_token, video_placeholder, 1)
+                    index += 1
+                
+
+
+        # Step 4: Tokenize text
+        return_tensors = output_kwargs["text_kwargs"].pop("return_tensors", None)
+        return_mm_token_type_ids = output_kwargs["text_kwargs"].pop("return_mm_token_type_ids", None)
+        
+        # Request offset_mapping if we need to create labels
+        if should_create_labels:
+            output_kwargs["text_kwargs"]["return_offsets_mapping"] = True
+        
+        text_inputs = self.tokenizer(text, **output_kwargs["text_kwargs"])
+        
+        # ignore check_special_mm_tokens nums in test and input ids.
+        # self._check_special_mm_tokens(text, text_inputs, modalities=["image", "video"])
+        
+        # Create labels if labels_spans was provided
+        if should_create_labels:
+            offset_mapping = text_inputs.pop("offset_mapping")
+            labels = self._create_labels_from_spans(
+                text_inputs["input_ids"],
+                offset_mapping,
+                labels_spans,
+                ignore_index
+            )
+
+        if return_mm_token_type_ids:
+            array_ids = np.array(text_inputs["input_ids"])
+            mm_token_type_ids = np.zeros_like(text_inputs["input_ids"])
+            mm_token_type_ids[array_ids == self.image_token_id] = 1
+            text_inputs["mm_token_type_ids"] = mm_token_type_ids.tolist()
+
+        # Step 5: Concatenate pixel_values and grid_thw in sequence order
+        # Prepare output
+        output_data = {**text_inputs}
+        
+        if not needs_reorder:
+            # Fast path: only one media type, no reordering needed
+            final_pixel_values = []
+            final_grid_thw = []
+            
+            if has_images:
+                final_pixel_values.append(image_inputs["pixel_values"])
+                final_grid_thw.extend(image_grid_thw)
+            
+            if has_videos:
+                final_pixel_values.append(videos_inputs["pixel_values_videos"])
+                final_grid_thw.extend(video_grid_thw)
+            
+            if final_pixel_values:
+                output_data["pixel_values"] = np.concatenate(final_pixel_values, axis=0) if len(final_pixel_values) > 1 else final_pixel_values[0]
+            
+            if final_grid_thw:
+                output_data["grid_thw"] = np.stack(final_grid_thw, axis=0)
+            
+            # Calculate media_nums_per_sample
+            media_nums_per_sample = []
+            for batch_idx in range(len(text)):
+                media_order = media_order_per_sample[batch_idx]
+                media_nums_per_sample.append(len(media_order) if len(media_order) > 0 else 1)
+            
+            # Don't add media_nums_per_sample to output_data yet
+            # Will add it after BatchFeature to keep it as list
+            
+        else:
+            # Slow path: both images and videos exist, need reordering
+            final_pixel_values = []
+            final_grid_thw = []
+            media_nums_per_sample = []
+            
+            # Global indices to track position in flattened image/video arrays
+            global_image_idx = 0
+            global_video_idx = 0
+            
+            for batch_idx in range(len(text)):
+                # Use the recorded media order from Step 3.0
+                media_order = media_order_per_sample[batch_idx]
+                
+                if len(media_order) == 0:
+                    # If no media provided for this sample, add a blank image
+                    media_nums_per_sample.append(1)
+                    min_pixels = 128 * 128
+                    patch_size = getattr(self.image_processor, "patch_size", None) or 16
+                    temporal_patch_size = getattr(self.image_processor, "temporal_patch_size", None) or 1
+                    merge_size = getattr(self.image_processor, "merge_size", None) or 2
+                    
+                    factor = patch_size * merge_size
+                    side = int(np.ceil(np.sqrt(min_pixels) / factor) * factor)
+                    grid_h = side // patch_size
+                    grid_w = side // patch_size
+                    grid_t = 1
+                    
+                    # Channel = 3 (RGB)
+                    channel = 3
+                    dim = channel * temporal_patch_size * patch_size * patch_size
+                    num_patches = grid_t * grid_h * grid_w
+                    
+                    blank_pixel_values = np.zeros((num_patches, dim), dtype=np.float32)
+                    blank_grid_thw = np.array([grid_t, grid_h, grid_w], dtype=np.int64)
+                    
+                    final_pixel_values.append(blank_pixel_values)
+                    final_grid_thw.append(blank_grid_thw)
+                else:
+                    media_nums_per_sample.append(len(media_order))
+                    
+                    # Collect media data according to the recorded order
+                    for media_type, _ in media_order:
+                        if media_type == "image" and image_grid_thw is not None:
+                            # Get image data
+                            if image_pixel_values_list:
+                                final_pixel_values.append(image_pixel_values_list[global_image_idx])
+                            final_grid_thw.append(image_grid_thw[global_image_idx])
+                            global_image_idx += 1
+                        elif media_type == "video" and video_grid_thw is not None:
+                            # Get video data
+                            if video_pixel_values_list:
+                                final_pixel_values.append(video_pixel_values_list[global_video_idx])
+                            final_grid_thw.append(video_grid_thw[global_video_idx])
+                            global_video_idx += 1
+            
+            # Concatenate/stack to unified format
+            if final_pixel_values:
+                output_data["pixel_values"] = _concat_array_or_tensor(
+                    final_pixel_values, axis=0
+                )
+            
+            if final_grid_thw:
+                output_data["grid_thw"] = _stack_array_or_tensor(
+                    final_grid_thw, axis=0
+                )
+            
+            # Don't add media_nums_per_sample to output_data yet
+            # Will add it after BatchFeature to keep it as list
+
+        # Create cross_attention_mask using media_nums_per_sample
+        if "input_ids" in output_data and "grid_thw" in output_data and media_nums_per_sample:
+            cross_attention_mask = self._create_cross_attention_mask(
+                output_data["input_ids"],
+                output_data["grid_thw"],
+                media_nums_per_sample,
+                output_data.get("attention_mask", None)
+            )
+            output_data["cross_attention_mask"] = cross_attention_mask
+        
+        # Add labels to output if created
+        if should_create_labels:
+            output_data["labels"] = labels
+
+        # BatchFeature will handle conversion to pt/tf/jax/np based on tensor_type
+        batch_feature = BatchFeature(data=output_data, tensor_type=return_tensors)
+        
+        # Add media_nums_per_sample after BatchFeature to keep it as list (not tensor)
+        if media_nums_per_sample:
+            batch_feature["media_nums_per_sample"] = media_nums_per_sample
+        
+        return batch_feature
+
+    def _create_cross_attention_mask(self, input_ids, grid_thw, media_nums_per_sample, attention_mask=None):
+        """
+        Create cross_attention_mask of shape (batch_size, 1, text_len, num_images).
+        Video frames are treated as individual images.
+        Mask values: True for masked, False for visible.
+        Causal masking: text can see images that appear at or before the text position.
+        
+        Args:
+            input_ids: List of token ids
+            grid_thw: Grid sizes for each media item
+            media_nums_per_sample: Number of media items per sample
+            attention_mask: Optional attention mask to filter out padding positions
+        """
+        batch_size = len(input_ids)
+        max_text_len = max(len(ids) for ids in input_ids)
+        
+        # Calculate total frames per sample to find max_num_frames
+        total_frames_per_sample = []
+        media_idx = 0
+        for b in range(batch_size):
+            num_media = media_nums_per_sample[b]
+            if num_media == 0:
+                total_frames_per_sample.append(0)
+                continue
+                
+            sample_frames = 0
+            for _ in range(num_media):
+                # grid_thw is (N, 3) where first dim is t (num_frames)
+                t = grid_thw[media_idx][0]
+                if isinstance(t, torch.Tensor):
+                    t = int(t.item())
+                else:
+                    t = int(t)
+                sample_frames += t
+                media_idx += 1
+            total_frames_per_sample.append(sample_frames)
+            
+        max_num_frames = max(total_frames_per_sample) if total_frames_per_sample else 0
+        
+        if max_num_frames == 0:
+            return None
+            
+        # Vectorized implementation for speed
+        
+        # 1. Pad input_ids to create a tensor
+        # We use -1 as pad value since token ids are positive
+        input_ids_tensor = torch.full((batch_size, max_text_len), -1, dtype=torch.long)
+        for b, ids in enumerate(input_ids):
+            l = len(ids)
+            input_ids_tensor[b, :l] = torch.tensor(ids, dtype=torch.long)
+            
+        # 2. Identify image tokens
+        is_image_token = (input_ids_tensor == self.image_token_id)
+        
+        # 3. Compute cumulative image tokens (how many image tokens appeared up to position t)
+        # shape: (batch_size, text_len)
+        cum_image_tokens = is_image_token.cumsum(dim=1)
+        
+        # 4. Create frame indices
+        # shape: (1, 1, max_num_frames)
+        frame_indices = torch.arange(max_num_frames).reshape(1, 1, -1)
+        
+        # 5. Determine visibility based on causal relationship
+        # Text at `t` sees frame `i` if `cum_image_tokens[t] > i`
+        # Because if frame `i` is the (i+1)-th image token, it becomes visible when count reaches i+1
+        # shape: (batch_size, text_len, max_num_frames)
+        visible_mask = cum_image_tokens.unsqueeze(-1) > frame_indices
+        
+        # 6. Apply attention_mask if provided
+        if attention_mask is not None:
+            # Convert to tensor if needed
+            if isinstance(attention_mask, torch.Tensor):
+                 attn_mask_tensor = attention_mask
+            else:
+                 # List of lists
+                 attn_mask_tensor = torch.zeros((batch_size, max_text_len), dtype=torch.long)
+                 for b, mask_row in enumerate(attention_mask):
+                     l = len(mask_row)
+                     attn_mask_tensor[b, :l] = torch.tensor(mask_row, dtype=torch.long)
+            
+            # shape: (batch_size, text_len, 1)
+            valid_text = (attn_mask_tensor.unsqueeze(-1) == 1)
+            visible_mask = visible_mask & valid_text
+            
+        # 7. Mask out frames that don't exist for a sample
+        # shape: (batch_size, 1, 1)
+        total_frames_tensor = torch.tensor(total_frames_per_sample).reshape(batch_size, 1, 1)
+        # shape: (batch_size, 1, max_num_frames)
+        valid_frames = frame_indices < total_frames_tensor
+        
+        visible_mask = visible_mask & valid_frames
+        
+        # 8. Create final mask (True for masked, False for visible)
+        mask = ~visible_mask
+        
+        # 9. Add channel dimension: (batch_size, 1, text_len, max_num_frames)
+        mask = mask.unsqueeze(1)
+        
+        return mask
+
+    def _replace_and_update_spans(
+        self,
+        text: str,
+        old_str: str,
+        new_str: str,
+        spans: List[List[int]],
+        replace_count: int = -1
+    ) -> tuple:
+        """
+        Replace occurrences of old_str with new_str and update spans accordingly.
+        
+        Args:
+            text: The text to perform replacement on
+            old_str: String to be replaced
+            new_str: String to replace with
+            spans: List of [start, end] spans to update (modified in place)
+            replace_count: Maximum number of replacements (-1 for all)
+            
+        Returns:
+            Tuple of (new_text, updated_spans)
+        """
+        delta = len(new_str) - len(old_str)
+        result_text = text
+        count = 0
+        search_start = 0
+        
+        while True:
+            pos = result_text.find(old_str, search_start)
+            if pos == -1:
+                break
+            if replace_count != -1 and count >= replace_count:
+                break
+            
+            # Update all spans that come after this position
+            for span in spans:
+                if span[0] > pos:
+                    # Span starts after replacement point
+                    span[0] += delta
+                    span[1] += delta
+                elif span[1] > pos:
+                    # Span ends after replacement point (spans the replacement)
+                    span[1] += delta
+            
+            # Perform the replacement
+            result_text = result_text[:pos] + new_str + result_text[pos + len(old_str):]
+            search_start = pos + len(new_str)
+            count += 1
+        
+        return result_text, spans
+
+    def _create_labels_from_spans(
+        self,
+        input_ids: List[List[int]],
+        offset_mapping: List[List[tuple]],
+        labels_spans: List[List[List[int]]],
+        ignore_index: int = -100,
+        mask_token_ids: Optional[set] = None
+    ) -> List[List[int]]:
+        """
+        Create labels from spans and offset_mapping.
+        
+        Args:
+            input_ids: Tokenized input ids
+            offset_mapping: Character offsets for each token from tokenizer (special tokens included)
+            labels_spans: Updated spans indicating assistant regions (after text transformations)
+            ignore_index: Value for masked positions
+            mask_token_ids: Set of token ids that should always be masked (set to ignore_index)
+                in labels, regardless of whether they fall inside a span.
+                Defaults to self.mask_token_ids if not provided.
+            
+        Returns:
+            labels: List of label ids, same shape as input_ids
+        
+        Note:
+            - Tokenizer's offset_mapping already includes correct offsets for special tokens in text
+            - Only need to mask tokens inside <|vision_start|>...<|vision_end|>
+            - Tokens whose id is in mask_token_ids are always masked
+            - All other tokens in spans (including special tokens like <|im_end|>) get labels
+        """
+        if mask_token_ids is None:
+            mask_token_ids = self.mask_token_ids
+        
+        batch_labels = []
+        
+        for batch_idx in range(len(input_ids)):
+            ids = input_ids[batch_idx]
+            offsets = offset_mapping[batch_idx]
+            spans = labels_spans[batch_idx]
+            
+            labels = [ignore_index] * len(ids)
+            
+            # Process each span: find token range and set labels
+            for span_start, span_end in spans:
+                in_vision = False
+                
+                # Find tokens that overlap with this span
+                for token_idx, (token_id, (char_start, char_end)) in enumerate(zip(ids, offsets)):
+                    # Skip tokens completely before this span
+                    if char_end <= span_start:
+                        continue
+                    # Stop when tokens are completely after this span
+                    if char_start >= span_end:
+                        break
+                    
+                    # Token overlaps with span, process it
+                    # Track vision region: <|vision_start|> ... <|vision_end|>
+                    if token_id == self.vision_start_token_id:
+                        in_vision = True
+                        continue
+                    if token_id == self.vision_end_token_id:
+                        in_vision = False
+                        continue
+                    
+                    # Skip tokens inside vision region
+                    if in_vision:
+                        continue
+                    
+                    # Always mask special tokens that should never have labels
+                    if token_id in mask_token_ids:
+                        continue
+                    
+                    # Set label for this token
+                    labels[token_idx] = token_id
+            
+            batch_labels.append(labels)
+        
+        return batch_labels
+
+    def _calculate_timestamps(
+        self, 
+        frames_indices: Optional[Union[List[int], np.ndarray]], 
+        total_num_frames: int, 
+        video_fps: float, 
+        duration: float, 
+        merge_size: int = 1,
+        actual_timestamps: Optional[List[float]] = None
+    ):
+        """
+        Calculate timestamps for video frames.
+        
+        Args:
+            frames_indices: Actual frame indices extracted (if available)
+            total_num_frames: Total number of sampled frames
+            video_fps: Video frames per second
+            duration: Video duration in seconds
+            merge_size: Temporal merge size
+            actual_timestamps: Pre-calculated actual timestamps (for segments)
+            
+        Returns:
+            List of timestamps (one per merged temporal patch)
+        """
+        # If actual timestamps are provided (from segment), use them directly
+        if actual_timestamps is not None:
+            timestamps = list(actual_timestamps)
+            
+            # Pad timestamps to be multiple of merge_size
+            if len(timestamps) % merge_size != 0:
+                timestamps.extend([timestamps[-1]] * (merge_size - len(timestamps) % merge_size))
+            
+            # Frames are merged by merge_size, so we average the timestamps within each temporal patch
+            timestamps = [
+                (timestamps[i] + timestamps[i + merge_size - 1]) / 2 
+                for i in range(0, len(timestamps), merge_size)
+            ]
+            return timestamps
+        
+        # Use frames_indices if available, otherwise generate uniformly sampled indices
+        if frames_indices is not None:
+            if isinstance(frames_indices, np.ndarray):
+                indices = frames_indices.tolist()
+            else:
+                indices = list(frames_indices)
+        else:
+            # Generate uniformly sampled frame indices
+            if total_num_frames <= 1:
+                indices = [0]
+            else:
+                # Uniformly sample frames across the video duration
+                indices = np.linspace(0, duration * video_fps - 1, total_num_frames).astype(np.int32).tolist()
+        
+        # Pad indices to be multiple of merge_size
+        if len(indices) % merge_size != 0:
+            indices.extend([indices[-1]] * (merge_size - len(indices) % merge_size))
+        
+        # Convert frame indices to timestamps
+        timestamps = [idx / video_fps for idx in indices]
+        
+        # Frames are merged by merge_size, so we average the timestamps within each temporal patch
+        timestamps = [
+            (timestamps[i] + timestamps[i + merge_size - 1]) / 2 
+            for i in range(0, len(timestamps), merge_size)
+        ]
+        return timestamps
+
+    def batch_decode(self, *args, **kwargs):
+        """
+        This method forwards all its arguments to the tokenizer's batch_decode. 
+        Please refer to the docstring of this method for more information.
+        """
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def decode(self, *args, **kwargs):
+        """
+        This method forwards all its arguments to the tokenizer's decode.
+        Please refer to the docstring of this method for more information.
+        """
+        return self.tokenizer.decode(*args, **kwargs)
+
+    def post_process_image_text_to_text(
+        self, generated_outputs, skip_special_tokens=True, clean_up_tokenization_spaces=False, **kwargs
+    ):
+        """
+        Post-process the output of the model to decode the text.
+
+        Args:
+            generated_outputs (`torch.Tensor` or `np.ndarray`):
+                The output of the model `generate` function. The output is expected to be a tensor 
+                of shape `(batch_size, sequence_length)` or `(sequence_length,)`.
+            skip_special_tokens (`bool`, *optional*, defaults to `True`):
+                Whether or not to remove special tokens in the output.
+            clean_up_tokenization_spaces (`bool`, *optional*, defaults to `False`):
+                Whether or not to clean up the tokenization spaces.
+            **kwargs:
+                Additional arguments to be passed to the tokenizer's `batch_decode` method.
+
+        Returns:
+            `list[str]`: The decoded text.
+        """
+        return self.tokenizer.batch_decode(
+            generated_outputs,
+            skip_special_tokens=skip_special_tokens,
+            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
+            **kwargs,
+        )
+
+
+__all__ = ["MossVLProcessor", "MossVLImageProcessorFast"]

processor_config.json

@@ -0,0 +1,6 @@
+{
+  "auto_map": {
+    "AutoProcessor": "processing_moss_vl.MossVLProcessor"
+  },
+  "processor_class": "MossVLProcessor"
+}

quantization_report.json

@@ -0,0 +1,57 @@
+{
+  "created_at": "2026-07-26T06:19:47.245758+00:00",
+  "source": "/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/final_release/mossvl_streaming_final_release",
+  "output": "/inspire/qb-ilm/project/video-understanding/public/train/moss_vl_streaming/8B/quant/mossvl_streaming_w4a16_nf4_keep_first4_last4",
+  "variant": "w4_keep_ends",
+  "method": "bitsandbytes NF4 W4A16 keep-first-last-4",
+  "compute_dtype": "bfloat16",
+  "skip_modules": [
+    "model.visual",
+    "cross_attn",
+    "lm_head",
+    "model.language_model.layers.0",
+    "model.language_model.layers.1",
+    "model.language_model.layers.2",
+    "model.language_model.layers.3",
+    "model.language_model.layers.44",
+    "model.language_model.layers.45",
+    "model.language_model.layers.46",
+    "model.language_model.layers.47"
+  ],
+  "quantized_kind": "4bit",
+  "quantized_linear_count": 240,
+  "quantized_module_samples": [
+    "model.language_model.layers.4.self_attn.q_proj",
+    "model.language_model.layers.4.self_attn.k_proj",
+    "model.language_model.layers.4.self_attn.v_proj",
+    "model.language_model.layers.4.self_attn.o_proj",
+    "model.language_model.layers.4.mlp.gate_proj",
+    "model.language_model.layers.4.mlp.up_proj",
+    "model.language_model.layers.4.mlp.down_proj",
+    "model.language_model.layers.5.self_attn.q_proj",
+    "model.language_model.layers.5.self_attn.k_proj",
+    "model.language_model.layers.5.self_attn.v_proj",
+    "model.language_model.layers.5.self_attn.o_proj",
+    "model.language_model.layers.5.mlp.gate_proj",
+    "model.language_model.layers.5.mlp.up_proj",
+    "model.language_model.layers.5.mlp.down_proj",
+    "model.language_model.layers.6.mlp.gate_proj",
+    "model.language_model.layers.6.mlp.up_proj",
+    "model.language_model.layers.6.mlp.down_proj",
+    "model.language_model.layers.7.self_attn.q_proj",
+    "model.language_model.layers.7.self_attn.k_proj",
+    "model.language_model.layers.7.self_attn.v_proj"
+  ],
+  "peak_gpu_memory_gib_during_conversion": 11.038,
+  "gpu": "NVIDIA H200",
+  "versions": {
+    "torch": "2.8.0+cu128",
+    "transformers": "4.57.1",
+    "accelerate": "1.12.0",
+    "bitsandbytes": "0.49.2"
+  },
+  "reload_verified": true,
+  "reload_quantized_kind": "4bit",
+  "reload_quantized_linear_count": 240,
+  "peak_gpu_memory_gib_after_reload": 14.549
+}

special_tokens_map.json

@@ -0,0 +1,35 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<|time_start|>",
+    "<|time_end|>",
+    "<|silence|>",
+    "<|response|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e7bbd0f9784004df51aca562befc3c7a8f294b4045aa8685536c35804c9aa493
+size 11423411

tokenizer_config.json

@@ -0,0 +1,279 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<|time_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151670": {
+      "content": "<|time_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151671": {
+      "content": "<|silence|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151672": {
+      "content": "<|response|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>",
+    "<|time_start|>",
+    "<|time_end|>",
+    "<|silence|>",
+    "<|response|>"
+  ],
+  "auto_map": {
+    "AutoProcessor": "processing_moss_vl.MossVLProcessor"
+  },
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 262144,
+  "pad_token": "<|endoftext|>",
+  "processor_class": "MossVLProcessor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}

video_preprocessor_config.json

@@ -0,0 +1,48 @@
+{
+  "auto_map": {
+    "AutoProcessor": "processing_moss_vl.MossVLProcessor",
+    "AutoVideoProcessor": "video_processing_moss_vl.MossVLVideoProcessor"
+  },
+  "crop_size": null,
+  "data_format": "channels_first",
+  "default_to_square": true,
+  "device": null,
+  "do_center_crop": null,
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "do_sample_frames": true,
+  "fps": null,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "input_data_format": null,
+  "max_frames": 256,
+  "merge_size": 2,
+  "min_frames": 1,
+  "num_extract_threads": 4,
+  "num_frames": null,
+  "pad_size": null,
+  "patch_size": 16,
+  "processor_class": "MossVLProcessor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098,
+  "return_metadata": false,
+  "size": {
+    "longest_edge": 16777216,
+    "shortest_edge": 4096
+  },
+  "temporal_patch_size": 1,
+  "video_fps": 1.0,
+  "video_max_pixels": 65536000,
+  "video_metadata": null,
+  "video_processor_type": "MossVLVideoProcessor"
+}

video_processing_moss_vl.py

@@ -0,0 +1,1248 @@
+# coding=utf-8
+# Copyright 2025 The FNLP Vision Team and The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""video processor class for Moss-VL."""
+
+import json
+import logging as system_logging
+import math
+import os
+import re
+import subprocess
+import traceback
+from functools import lru_cache
+from typing import Any, Dict, List, Optional, Union
+
+import numpy as np
+import torch
+from joblib import Parallel, delayed
+from torchcodec.decoders import VideoDecoder
+
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.image_utils import ChannelDimension, PILImageResampling, SizeDict, get_image_size, validate_kwargs
+from transformers.processing_utils import Unpack, VideosKwargs
+from transformers.utils import TensorType, add_start_docstrings, logging
+from transformers.video_processing_utils import BASE_VIDEO_PROCESSOR_DOCSTRING, BaseVideoProcessor
+from transformers.video_utils import VideoMetadata, group_videos_by_shape, reorder_videos
+
+
+logger = logging.get_logger(__name__)
+
+TORCHCODEC_TIMESTAMP_EPSILON = 1e-6
+
+
+# -----------------------------------------------------------------------------
+# Torchcodec video frame extraction utilities
+# -----------------------------------------------------------------------------
+
+def check_video_for_extra_streams_and_errors(video_path: str) -> dict:
+    """
+    Check if video file has abnormal streams or errors reported by ffprobe.
+    
+    Args:
+        video_path: Path to the video file.
+        
+    Returns:
+        A dictionary containing:
+        - 'has_extra_streams': bool, whether there are streams other than video and audio.
+        - 'unsupported_codec_errors': list, all "Unsupported codec" error messages.
+        - 'ffprobe_output_error': str, other errors/warnings from ffprobe stderr.
+        - 'ffprobe_successful': bool, whether ffprobe command executed successfully (return code 0).
+        - 'stream_details': list, codec_type and index for each stream.
+        - 'num_streams': int, total number of streams identified in the video file.
+    """
+    result = {
+        'has_extra_streams': False,
+        'unsupported_codec_errors': [],
+        'ffprobe_output_error': '',
+        'ffprobe_successful': False,
+        'stream_details': [],
+        'num_streams': 0
+    }
+    
+    command = [
+        "ffprobe",
+        "-v", "error",
+        "-show_streams",
+        "-show_format",
+        "-of", "json",
+        video_path
+    ]
+    
+    try:
+        process = subprocess.run(
+            command,
+            capture_output=True,
+            text=True,
+            check=False
+        )
+        result['ffprobe_successful'] = (process.returncode == 0)
+        
+        if process.stderr:
+            result['ffprobe_output_error'] = process.stderr
+            unsupported_codec_pattern = re.compile(r"Unsupported codec with id \d+ for input stream \d+")
+            result['unsupported_codec_errors'] = unsupported_codec_pattern.findall(process.stderr)
+        
+        if process.stdout:
+            ffprobe_data = json.loads(process.stdout)
+            if 'streams' in ffprobe_data:
+                result['num_streams'] = len(ffprobe_data['streams'])
+                for stream in ffprobe_data['streams']:
+                    stream_type = stream.get('codec_type')
+                    stream_index = stream.get('index')
+                    result['stream_details'].append({'index': stream_index, 'codec_type': stream_type})
+                    if stream_type not in ['video', 'audio']:
+                        result['has_extra_streams'] = True
+                        
+            if 'format' in ffprobe_data and 'nb_streams' in ffprobe_data['format']:
+                if result['num_streams'] == 0:
+                    result['num_streams'] = ffprobe_data['format']['nb_streams']
+                elif result['num_streams'] != ffprobe_data['format']['nb_streams']:
+                    logger.warning(
+                        f"Number of streams in 'streams' list ({result['num_streams']}) "
+                        f"differs from 'nb_streams' in 'format' ({ffprobe_data['format']['nb_streams']})."
+                    )
+    except FileNotFoundError:
+        result['ffprobe_output_error'] = "ffprobe command not found. Please ensure FFmpeg is installed and in your PATH."
+        result['ffprobe_successful'] = False
+    except json.JSONDecodeError:
+        result['ffprobe_output_error'] = "Failed to parse ffprobe JSON output. Check ffprobe installation or video file."
+        result['ffprobe_successful'] = False
+    except Exception as e:
+        result['ffprobe_output_error'] = f"An unexpected error occurred: {e}"
+        result['ffprobe_successful'] = False
+        
+    return result
+
+
+def remove_video_extra_stream_ffmpeg(input_video: str, output_video: str) -> bool:
+    """
+    Remove extra streams from video using ffmpeg.
+    
+    Args:
+        input_video: Path to input video.
+        output_video: Path to output video.
+        
+    Returns:
+        bool: True if successful, False otherwise.
+    """
+    command_list = [
+        "ffmpeg", "-y", "-i", input_video,
+        "-map", "0:v:0",
+        "-c", "copy",
+        "-an",
+        "-sn",
+        "-dn",
+        "-map_metadata", "-1",
+        "-map_chapters", "-1",
+        "-movflags", "faststart",
+        output_video,
+    ]
+    
+    try:
+        subprocess.run(command_list, shell=False, check=True, capture_output=True)
+        return True
+    except subprocess.CalledProcessError as e:
+        system_logging.error(f"Command execution failed with return code: {e.returncode}, video: {input_video}")
+        system_logging.error(f"Error output:\n{e.stderr}")
+        return False
+    except FileNotFoundError:
+        system_logging.error("Error: ffmpeg command not found. Please ensure ffmpeg is installed and in PATH.")
+        return False
+    except Exception as e:
+        system_logging.error(f"Unexpected error executing command: {e}, video: {input_video}", exc_info=True)
+        return False
+
+
+def clean_video_streams(video_path: str) -> str:
+    """
+    Clean video streams if extra streams are detected.
+    
+    Args:
+        video_path: Path to the video file.
+        
+    Returns:
+        str: Path to cleaned video (or original if no cleaning needed).
+    """
+    ffprobe_res = check_video_for_extra_streams_and_errors(video_path)
+    if ffprobe_res['has_extra_streams']:
+        base_name = os.path.basename(video_path)
+        output_folder = os.path.dirname(video_path)
+        file_name_without_ext, file_ext = os.path.splitext(base_name)
+        new_base_name = f"{file_name_without_ext}_fix{file_ext}"
+        video_path_output = os.path.join(output_folder, new_base_name)
+        
+        process_flag = remove_video_extra_stream_ffmpeg(video_path, video_path_output)
+        if not process_flag:
+            logger.warning("Failed to remove extra streams with ffmpeg")
+            return video_path
+        return video_path_output
+    return video_path
+
+
+@lru_cache(maxsize=8192)
+def cached_clean_video_streams(video_path: str) -> str:
+    return clean_video_streams(video_path)
+
+
+def clamp_timestamps_for_torchcodec(timestamps: List[float], torchcodec_metadata) -> List[float]:
+    if not timestamps:
+        return timestamps
+
+    min_pts = torchcodec_metadata.begin_stream_seconds_from_content
+    if min_pts is None:
+        min_pts = 0.0
+
+    max_pts_candidates = []
+    if torchcodec_metadata.num_frames_from_content and torchcodec_metadata.average_fps:
+        max_pts_candidates.append(
+            (torchcodec_metadata.num_frames_from_content - 1) / torchcodec_metadata.average_fps + min_pts
+        )
+    if torchcodec_metadata.end_stream_seconds_from_content is not None:
+        # TorchCodec requires requested PTS to be strictly smaller than the content end.
+        max_pts_candidates.append(torchcodec_metadata.end_stream_seconds_from_content - TORCHCODEC_TIMESTAMP_EPSILON)
+    if not max_pts_candidates and torchcodec_metadata.duration_seconds is not None:
+        max_pts_candidates.append(torchcodec_metadata.duration_seconds - TORCHCODEC_TIMESTAMP_EPSILON)
+
+    if max_pts_candidates:
+        max_pts = max(min_pts, min(max_pts_candidates))
+        return [max(min_pts, min(float(t), max_pts)) for t in timestamps]
+    if min_pts > 0:
+        return [max(min_pts, float(t)) for t in timestamps]
+    return [float(t) for t in timestamps]
+
+
+def split_indices(indices: List[Union[int, float]], num_chunks: int) -> List[List[Union[int, float]]]:
+    """
+    Split an index list into roughly equal chunks.
+    
+    Args:
+        indices: List of indices to split.
+        num_chunks: Number of chunks to create.
+        
+    Returns:
+        List of index chunks.
+    """
+    chunk_size = len(indices) // num_chunks
+    chunks = []
+    for i in range(num_chunks - 1):
+        chunks.append(indices[i * chunk_size:(i + 1) * chunk_size])
+    chunks.append(indices[(num_chunks - 1) * chunk_size:])
+    return chunks
+
+
+def decode_sequentially(indices: List[int], video_path: str, ffmpeg_threads: int = 0):
+    """
+    Decode frames sequentially from a video.
+    
+    Args:
+        indices: List of frame indices to decode.
+        video_path: Path to the video file.
+        ffmpeg_threads: Number of ffmpeg threads to use.
+        
+    Returns:
+        FrameBatch from torchcodec.
+    """
+    decoder = VideoDecoder(video_path, num_ffmpeg_threads=ffmpeg_threads)
+    try:
+        return decoder.get_frames_at(indices)
+    finally:
+        del decoder
+
+
+def decode_with_multithreading(indices: List[int], num_threads: int, video_path: str) -> dict:
+    """
+    Decode frames using multithreading with joblib.
+    
+    Args:
+        indices: List of frame indices to decode.
+        num_threads: Number of threads to use.
+        video_path: Path to the video file.
+        
+    Returns:
+        dict: Contains 'data', 'duration_seconds', 'pts_seconds' tensors.
+    """
+    chunks = split_indices(indices, num_chunks=num_threads)
+    results = Parallel(n_jobs=num_threads, prefer="threads", verbose=0)(
+        delayed(decode_sequentially)(chunk, video_path) for chunk in chunks
+    )
+    
+    return {
+        "data": torch.cat([frame_batch.data for frame_batch in results], dim=0),
+        "duration_seconds": torch.cat([frame_batch.duration_seconds for frame_batch in results], dim=0),
+        "pts_seconds": torch.cat([frame_batch.pts_seconds for frame_batch in results], dim=0)
+    }
+
+
+def decode_sequentially_timestamp(timestamp_list: List[float], video_path: str, ffmpeg_threads: int = 0):
+    """
+    Decode frames sequentially from a video based on timestamps.
+    
+    Args:
+        timestamp_list: List of timestamps (in seconds) to decode.
+        video_path: Path to the video file.
+        ffmpeg_threads: Number of ffmpeg threads to use.
+        
+    Returns:
+        FrameBatch from torchcodec.
+    """
+    decoder = VideoDecoder(video_path, num_ffmpeg_threads=ffmpeg_threads)
+    try:
+        metadata = decoder.metadata
+        timestamp_list = clamp_timestamps_for_torchcodec(timestamp_list, metadata)
+
+        return decoder.get_frames_played_at(timestamp_list)
+    finally:
+        del decoder
+
+
+def timestamp_decode_with_multithreading(timestamp_list: List[float], num_threads: int, video_path: str) -> dict:
+    """
+    Decode frames using multithreading based on timestamps.
+    
+    Args:
+        timestamp_list: List of timestamps (in seconds) to decode.
+        num_threads: Number of threads to use.
+        video_path: Path to the video file.
+        
+    Returns:
+        dict: Contains 'data', 'duration_seconds', 'pts_seconds' tensors.
+    """
+    chunks = split_indices(timestamp_list, num_chunks=num_threads)
+    results = Parallel(n_jobs=num_threads, prefer="threads", verbose=0)(
+        delayed(decode_sequentially_timestamp)(chunk, video_path) for chunk in chunks
+    )
+    
+    # Concatenate results from all threads
+    data_list = [frame_batch.data for frame_batch in results]
+    duration_list = [frame_batch.duration_seconds for frame_batch in results]
+    pts_list = [frame_batch.pts_seconds for frame_batch in results]
+    
+    if not data_list:
+        logger.warning("No frames were successfully decoded.")
+        return {"data": torch.empty(0), "duration_seconds": torch.empty(0), "pts_seconds": torch.empty(0)}
+    
+    return {
+        "data": torch.cat(data_list, dim=0),
+        "duration_seconds": torch.cat(duration_list, dim=0),
+        "pts_seconds": torch.cat(pts_list, dim=0)
+    }
+
+
+def extract_frames_with_torchcodec(
+    video_path: str,
+    sample_frames_count: int,
+    num_threads: int = 4,
+
+) -> Optional[dict]:
+    """
+    Extract frames from video using torchcodec with multithreading.
+    
+    Args:
+        video_path: Path to the video file.
+        sample_frames_count: Number of frames to sample.
+        num_threads: Number of threads to use for extraction.
+        sampling_method: Sampling method, either "index" (uniform frame indices) or "timestamp" (uniform timestamps).
+        
+    Returns:
+        dict: Contains 'data' (N, C, H, W), 'duration_seconds' (N,), 'pts_seconds' (N,) tensors.
+              Returns None if extraction fails.
+    """
+    try:
+        video_path = cached_clean_video_streams(video_path)
+        decoder = VideoDecoder(video_path, num_ffmpeg_threads=0)
+        metadata = decoder.metadata
+
+
+        total_frames_in_video = metadata.num_frames_from_content
+        
+        effective_sample_count = min(sample_frames_count, total_frames_in_video)
+        if effective_sample_count == 0:
+            logger.error("Cannot extract frames: video has 0 frames or specified frame count is 0")
+            return None
+        
+        # Generate uniform frame indices
+        frame_indices = np.linspace(0, total_frames_in_video - 1, effective_sample_count).astype(np.int32)
+        # Ensure indices are valid and remove duplicates
+        frame_indices = np.unique(np.clip(frame_indices, 0, total_frames_in_video - 1))
+        
+        result = decode_with_multithreading(frame_indices.tolist(), num_threads=num_threads, video_path=video_path)
+        # Add frame_indices to the result for later use
+        result["frame_indices"] = frame_indices
+        return result
+
+
+            
+    except Exception:
+        traceback.print_exc()
+        return None
+
+
+def smart_resize(
+    num_frames: int,
+    height: int,
+    width: int,
+    temporal_factor: int = 1,
+    factor: int = 32,
+    min_pixels: int = 128 * 128,
+    max_pixels: int = 16 * 16 * 2 * 2 * 2 * 6144,
+    per_frame_min_pixels: int = None,
+    per_frame_max_pixels: int = None,
+):
+    if num_frames < temporal_factor:
+        raise ValueError(f"t:{num_frames} must be larger than temporal_factor:{temporal_factor}")
+    if height < factor or width < factor:
+        raise ValueError(f"height:{height} or width:{width} must be larger than factor:{factor}")
+    elif max(height, width) / min(height, width) > 200:
+        raise ValueError(
+            f"absolute aspect ratio must be smaller than 200, got {max(height, width) / min(height, width)}"
+        )
+    h_bar = round(height / factor) * factor
+    w_bar = round(width / factor) * factor
+    t_bar = round(num_frames / temporal_factor) * temporal_factor
+
+    # Step 1: Apply per-frame upper limit constraint
+    if per_frame_max_pixels is not None and h_bar * w_bar > per_frame_max_pixels:
+        beta = math.sqrt((height * width) / per_frame_max_pixels)
+        h_bar = max(factor, math.floor(height / beta / factor) * factor)
+        w_bar = max(factor, math.floor(width / beta / factor) * factor)
+
+    # Step 2: Apply 3D volume constraints (frames * height * width)
+    if t_bar * h_bar * w_bar > max_pixels:
+        beta = math.sqrt((num_frames * height * width) / max_pixels)
+        h_bar = max(factor, math.floor(height / beta / factor) * factor)
+        w_bar = max(factor, math.floor(width / beta / factor) * factor)
+    elif t_bar * h_bar * w_bar < min_pixels:
+        beta = math.sqrt(min_pixels / (num_frames * height * width))
+        h_bar = math.ceil(height * beta / factor) * factor
+        w_bar = math.ceil(width * beta / factor) * factor
+
+    # Step 3: Ensure per-frame lower limit is respected (after volume constraint)
+    # This guarantees single frame stays within [per_frame_min_pixels, per_frame_max_pixels]
+    if per_frame_min_pixels is not None and h_bar * w_bar < per_frame_min_pixels:
+        beta = math.sqrt(per_frame_min_pixels / (height * width))
+        h_bar = math.ceil(height * beta / factor) * factor
+        w_bar = math.ceil(width * beta / factor) * factor
+
+    return h_bar, w_bar
+
+
+class MossVLVideoProcessorInitKwargs(VideosKwargs):
+    patch_size: Optional[int]
+    temporal_patch_size: Optional[int]
+    merge_size: Optional[int]
+    min_frames: Optional[int]
+    max_frames: Optional[int]
+    video_fps: Optional[Union[int, float]]
+    num_extract_threads: Optional[int]
+    # Total 3D volume budget across all videos; distributed proportionally per video by T*H*W
+    video_max_pixels: Optional[int]
+
+
+@add_start_docstrings(
+    "Constructs a fast Moss-VL video processor that dynamically resizes videos based on the original videos.",
+    BASE_VIDEO_PROCESSOR_DOCSTRING,
+    """
+        patch_size (`int`, *optional*, defaults to 16):
+            The spacial patch size of the vision encoder.
+        temporal_patch_size (`int`, *optional*, defaults to 1):
+            The temporal patch size of the vision encoder.
+        merge_size (`int`, *optional*, defaults to 2):
+            The merge size of the vision encoder to llm encoder.
+        video_fps (`float`, *optional*, defaults to 1.0):
+            Target frames per second for video sampling.
+        min_frames (`int`, *optional*, defaults to 1):
+            Minimum number of frames to sample from a video.
+        max_frames (`int`, *optional*, defaults to 256):
+            Maximum number of frames to sample from a video.
+        num_extract_threads (`int`, *optional*, defaults to 4):
+            Number of threads to use for frame extraction.
+    """,
+)
+class MossVLVideoProcessor(BaseVideoProcessor):
+    resample = PILImageResampling.BICUBIC
+    size = {"shortest_edge": 128 * 32 * 32, "longest_edge": 32 * 32 * 768}
+    image_mean = [0.5, 0.5, 0.5]
+    image_std = [0.5, 0.5, 0.5]
+    do_resize = True
+    do_rescale = True
+    do_normalize = True
+    do_convert_rgb = True
+    patch_size = 16
+    temporal_patch_size = 1
+    merge_size = 2
+    video_fps = 1.0
+    min_frames = 1
+    max_frames = 256
+    num_extract_threads = 4
+    do_sample_frames = True
+    # Total 3D volume budget across all videos; distributed proportionally per video by T*H*W
+    video_max_pixels = None  # read from config
+    valid_kwargs = MossVLVideoProcessorInitKwargs
+    model_input_names = ["pixel_values_videos", "video_grid_thw"]
+
+    def __init__(self, **kwargs: Unpack[MossVLVideoProcessorInitKwargs]):
+        super().__init__(**kwargs)
+        if self.size is not None and (
+            self.size.get("shortest_edge", None) is None or self.size.get("longest_edge", None) is None
+        ):
+            raise ValueError("size must contain 'shortest_edge' and 'longest_edge' keys.")
+
+    def _further_process_kwargs(
+        self,
+        size: Optional[SizeDict] = None,
+        **kwargs,
+    ) -> dict:
+        """
+        Update kwargs that need further processing before being validated
+        Can be overridden by subclasses to customize the processing of kwargs.
+        """
+        if size is not None and ("shortest_edge" not in size or "longest_edge" not in size):
+            raise ValueError("size must contain 'shortest_edge' and 'longest_edge' keys.")
+
+        return super()._further_process_kwargs(size=size, **kwargs)
+
+    def _get_video_path_from_input(self, video_input: Union[str, Dict[str, Any]]) -> str:
+        """Normalize a video input into a video path."""
+        if isinstance(video_input, dict):
+            return video_input["video_path"]
+        return video_input
+
+    def _get_video_duration_seconds(self, video_input: Union[str, Dict[str, Any]]) -> float:
+        """Get video duration in seconds for weighted frame-budget allocation."""
+        video_path = cached_clean_video_streams(self._get_video_path_from_input(video_input))
+        decoder = VideoDecoder(video_path, num_ffmpeg_threads=0)
+        try:
+            metadata = decoder.metadata
+            duration = None
+            if (
+                metadata.end_stream_seconds_from_content is not None
+                and metadata.begin_stream_seconds_from_content is not None
+            ):
+                duration = metadata.end_stream_seconds_from_content - metadata.begin_stream_seconds_from_content
+            if duration is None or duration <= 0:
+                duration = metadata.duration_seconds
+            return max(0.0, float(duration or 0.0))
+        finally:
+            del decoder
+
+    def _allocate_max_frames_for_multiple_videos(
+        self,
+        video_inputs: List[Union[str, Dict[str, Any]]],
+        total_max_frames: Optional[int],
+    ) -> List[Optional[int]]:
+        """
+        Treat max_frames as a total budget for multi-video input and allocate it by duration.
+
+        The returned values are per-video max_frames. Segment dict inputs still keep their
+        existing per-segment weighting logic after receiving the video-level allocation.
+        """
+        if not video_inputs:
+            return []
+        if total_max_frames is None or len(video_inputs) == 1:
+            return [total_max_frames] * len(video_inputs)
+
+        total_max_frames = int(total_max_frames)
+        num_videos = len(video_inputs)
+        if total_max_frames < num_videos:
+            logger.warning(
+                "Received max_frames=%s for %s videos. At least one frame per video is required, "
+                "so falling back to 1 frame per video.",
+                total_max_frames,
+                num_videos,
+            )
+            return [1] * num_videos
+
+        video_durations = [self._get_video_duration_seconds(video_input) for video_input in video_inputs]
+        total_duration = sum(video_durations)
+
+        # Reserve one frame per video first, then distribute the remaining budget by duration.
+        allocations = [1] * num_videos
+        remaining_budget = total_max_frames - num_videos
+        if remaining_budget == 0:
+            return allocations
+
+        if total_duration <= 0:
+            raw_extra_allocations = [remaining_budget / num_videos] * num_videos
+        else:
+            raw_extra_allocations = [
+                remaining_budget * (duration / total_duration) for duration in video_durations
+            ]
+
+        base_extra_allocations = [int(math.floor(value)) for value in raw_extra_allocations]
+        allocations = [base + extra for base, extra in zip(allocations, base_extra_allocations)]
+
+        remainder = remaining_budget - sum(base_extra_allocations)
+        if remainder > 0:
+            fractional_parts = [
+                (raw_value - base_value, index)
+                for index, (raw_value, base_value) in enumerate(zip(raw_extra_allocations, base_extra_allocations))
+            ]
+            fractional_parts.sort(key=lambda item: (-item[0], item[1]))
+            for _, index in fractional_parts[:remainder]:
+                allocations[index] += 1
+
+        return allocations
+
+    def calculate_num_frames(
+        self,
+        metadata: VideoMetadata,
+        num_frames: Optional[int] = None,
+        fps: Optional[Union[int, float]] = None,
+        min_frames: Optional[int] = None,
+        max_frames: Optional[int] = None,
+        **kwargs,
+    ) -> int:
+        """
+        Calculate the number of frames to sample using fps-based logic with min/max constraints.
+        
+        Logic:
+        1. Calculate target_frames based on fps and video duration
+        2. Apply min_frames and max_frames constraints
+        3. Apply max_allowed_frames protection (rough cap from total video_max_pixels budget)
+        4. Return the number of frames to sample
+        
+        Args:
+            metadata (`VideoMetadata`):
+                Metadata of the video containing information about total duration, fps and total number of frames.
+            num_frames (`int`, *optional*):
+                Maximum number of frames to sample. If provided, overrides fps-based calculation.
+            fps (`int` or `float`, *optional*):
+                Target frames to sample per second. Defaults to `self.video_fps`.
+            min_frames (`int`, *optional*):
+                Minimum number of frames to sample. If None, uses self.min_frames.
+            max_frames (`int`, *optional*):
+                Maximum number of frames to sample. If None, uses self.max_frames.
+        Returns:
+            int:
+                Number of frames to sample.
+        """
+        if fps is not None and num_frames is not None:
+            raise ValueError("`num_frames` and `fps` are mutually exclusive arguments, please use only one!")
+
+        total_num_frames = metadata.total_num_frames
+        
+        # Use provided min/max or fall back to defaults
+        effective_min_frames = min_frames if min_frames is not None else self.min_frames
+        effective_max_frames = max_frames if max_frames is not None else self.max_frames
+        
+        # Rough per-video frame cap derived from the multi-video total budget
+        # (exact allocation happens later in _preprocess via weighted distribution)
+        per_frame_min_pixels = self.size.get("shortest_edge", None) if self.size else None
+        video_max_pixels = getattr(self, "video_max_pixels", None)
+        if per_frame_min_pixels is not None and video_max_pixels is not None and per_frame_min_pixels > 0:
+            max_allowed_frames = video_max_pixels // per_frame_min_pixels
+            effective_max_frames = min(effective_max_frames, max_allowed_frames)
+        
+        # Get video duration
+        if hasattr(metadata, 'duration') and metadata.duration is not None:
+            duration = metadata.duration
+        else:
+            video_fps = metadata.fps
+            if video_fps is not None and video_fps > 0:
+                duration = total_num_frames / video_fps
+            else:
+                # Fallback: assume 24 fps
+                video_fps = 24.0
+                duration = total_num_frames / video_fps
+                logger.warning_once(
+                    "Could not determine video fps from metadata, defaulting to 24 fps for duration calculation."
+                )
+
+        # Use provided fps or default
+        target_fps = fps if fps is not None else self.video_fps
+        
+        # Calculate target frames based on fps and duration
+        if num_frames is None:
+            # Calculate how many frames we should sample based on target fps
+            target_total_frames = int(math.ceil(duration * target_fps - 1e-6))
+            
+            # Apply min/max constraints
+            sample_frames = max(target_total_frames, effective_min_frames)
+            sample_frames = min(sample_frames, effective_max_frames, total_num_frames)
+        else:
+            # If num_frames is explicitly provided, use it directly with constraints
+            sample_frames = min(max(num_frames, effective_min_frames), effective_max_frames, total_num_frames)
+
+        return sample_frames
+
+
+    def _decode_timestamps_with_decoder(
+        self,
+        decoder: VideoDecoder,
+        timestamps: List[float],
+        chunk_size: int = 128,
+    ) -> torch.Tensor:
+        if not timestamps:
+            return torch.empty(0)
+
+        frame_chunks = []
+        for start in range(0, len(timestamps), chunk_size):
+            frame_batch = decoder.get_frames_played_at(timestamps[start:start + chunk_size])
+            frame_chunks.append(frame_batch.data)
+
+        if len(frame_chunks) == 1:
+            return frame_chunks[0]
+        return torch.cat(frame_chunks, dim=0)
+
+    def _clamp_timestamps_for_decoder(
+        self,
+        timestamps: List[float],
+        torchcodec_metadata,
+    ) -> List[float]:
+        return clamp_timestamps_for_torchcodec(timestamps, torchcodec_metadata)
+
+    def _fetch_video_segments_batched(
+        self,
+        video_path: str,
+        segments: List[List[float]],
+        min_frames: Optional[int] = None,
+        max_frames: Optional[int] = None,
+        video_fps: Optional[float] = None,
+    ):
+        min_frames = max(1, min_frames if min_frames is not None else self.min_frames)
+        max_frames = max(1, max_frames if max_frames is not None else self.max_frames)
+        target_video_fps = video_fps if video_fps is not None else self.video_fps
+
+        video_path = cached_clean_video_streams(video_path)
+        decoder = VideoDecoder(video_path, num_ffmpeg_threads=0)
+        try:
+            torchcodec_metadata = decoder.metadata
+            source_video_fps = torchcodec_metadata.average_fps
+
+            duration = None
+            if (
+                torchcodec_metadata.end_stream_seconds_from_content is not None
+                and torchcodec_metadata.begin_stream_seconds_from_content is not None
+            ):
+                duration = (
+                    torchcodec_metadata.end_stream_seconds_from_content
+                    - torchcodec_metadata.begin_stream_seconds_from_content
+                )
+            if duration is None or duration <= 0:
+                duration = torchcodec_metadata.duration_seconds
+
+            segment_durations = [
+                segment[1] - segment[0] if len(segment) == 2 else None
+                for segment in segments
+            ]
+            total_segment_duration = sum(d for d in segment_durations if d is not None)
+            num_range_segments = sum(1 for d in segment_durations if d is not None)
+
+            segment_timestamps = []
+            decode_timestamps = []
+            for i, segment in enumerate(segments):
+                if len(segment) == 1:
+                    actual_timestamps = self._clamp_timestamps_for_decoder([segment[0]], torchcodec_metadata)
+                    segment_timestamps.append(actual_timestamps)
+                    decode_timestamps.extend(actual_timestamps)
+                    continue
+
+                start_time, end_time = segment
+                segment_duration = end_time - start_time
+                target_frames = int(math.ceil(segment_duration * target_video_fps))
+
+                if total_segment_duration > 0:
+                    weight = segment_durations[i] / total_segment_duration
+                else:
+                    weight = 1.0 / num_range_segments if num_range_segments > 0 else 1.0
+
+                weighted_min_frames = max(1, int(round(min_frames * weight)))
+                weighted_max_frames = max(1, int(round(max_frames * weight)))
+                target_frames = max(target_frames, weighted_min_frames)
+                target_frames = min(target_frames, weighted_max_frames)
+
+                if target_frames == 1:
+                    actual_timestamps = [start_time]
+                else:
+                    actual_timestamps = np.linspace(
+                        start_time,
+                        end_time,
+                        target_frames,
+                        endpoint=False,
+                    ).tolist()
+
+                actual_timestamps = self._clamp_timestamps_for_decoder(actual_timestamps, torchcodec_metadata)
+                segment_timestamps.append(actual_timestamps)
+                decode_timestamps.extend(actual_timestamps)
+
+            flat_frames = self._decode_timestamps_with_decoder(decoder, decode_timestamps)
+
+            videos = []
+            metadata = []
+            frame_offset = 0
+            for actual_timestamps in segment_timestamps:
+                sample_count = len(actual_timestamps)
+                video_tensor = flat_frames[frame_offset:frame_offset + sample_count]
+                frame_offset += sample_count
+
+                video_metadata = VideoMetadata(
+                    total_num_frames=sample_count,
+                    fps=source_video_fps,
+                    duration=duration,
+                    video_backend="torchcodec",
+                    height=torchcodec_metadata.height,
+                    width=torchcodec_metadata.width,
+                    frames_indices=None,
+                )
+                video_metadata.actual_timestamps = actual_timestamps
+
+                videos.append(video_tensor)
+                metadata.append(video_metadata)
+
+            return videos, metadata
+        finally:
+            del decoder
+
+
+    def _fetch_video_segment(
+        self,
+        video_path: str,
+        segment: List[float],
+        min_frames: Optional[int] = None,
+        max_frames: Optional[int] = None,
+        video_fps: Optional[float] = None,
+    ):
+        """
+        Fetch video frames for a specific segment.
+        
+        Args:
+            video_path: Path to the video file
+            segment: [start, end] for a segment (left-closed, right-open) or [time] for a single frame
+            min_frames: Minimum frames for this segment (weighted). Defaults to self.min_frames. Must be >= 1.
+            max_frames: Maximum frames for this segment (weighted). Defaults to self.max_frames. Must be >= 1.
+            video_fps: Target frames per second for video sampling. If None, uses self.video_fps.
+        
+        Returns:
+            Tuple of (video_tensor, video_metadata)
+        """
+        # Use provided min/max or fall back to defaults, ensure >= 1
+        min_frames = max(1, min_frames if min_frames is not None else self.min_frames)
+        max_frames = max(1, max_frames if max_frames is not None else self.max_frames)
+        # Use provided video_fps or fall back to self.video_fps
+        target_video_fps = video_fps if video_fps is not None else self.video_fps
+        
+        video_path = clean_video_streams(video_path)
+        decoder = VideoDecoder(video_path, num_ffmpeg_threads=0)
+        try:
+            torchcodec_metadata = decoder.metadata
+            
+            video_fps = torchcodec_metadata.average_fps
+            
+            # Calculate duration
+            duration = None
+            if torchcodec_metadata.end_stream_seconds_from_content is not None and torchcodec_metadata.begin_stream_seconds_from_content is not None:
+                duration = torchcodec_metadata.end_stream_seconds_from_content - torchcodec_metadata.begin_stream_seconds_from_content
+            if duration is None or duration <= 0:
+                duration = torchcodec_metadata.duration_seconds
+            
+            if len(segment) == 1:
+                # Single frame at specified time
+                actual_timestamps = self._clamp_timestamps_for_decoder([segment[0]], torchcodec_metadata)
+                frame_batch = decoder.get_frames_played_at(actual_timestamps)
+                video_tensor = frame_batch.data
+                sample_count = 1
+            else:
+                # Segment [start, end) - left-closed, right-open interval
+                start_time, end_time = segment
+                segment_duration = end_time - start_time
+                
+                # Calculate number of frames to sample for this segment
+                target_frames = int(math.ceil(segment_duration * target_video_fps))
+                target_frames = max(target_frames, min_frames)
+                target_frames = min(target_frames, max_frames)
+                
+                # Generate timestamps for uniform sampling within segment
+                if target_frames == 1:
+                    actual_timestamps = [start_time]  # Use start_time for single frame
+                else:
+                    # Sample uniformly within [start, end), endpoint=False for left-closed right-open
+                    actual_timestamps = np.linspace(start_time, end_time, target_frames, endpoint=False).tolist()
+                
+                actual_timestamps = self._clamp_timestamps_for_decoder(actual_timestamps, torchcodec_metadata)
+
+                # Use multithreading for extraction
+                result = timestamp_decode_with_multithreading(actual_timestamps, self.num_extract_threads, video_path)
+                video_tensor = result["data"]
+                sample_count = len(actual_timestamps)
+            
+            # Create VideoMetadata
+            video_metadata = VideoMetadata(
+                total_num_frames=sample_count,
+                fps=video_fps,
+                duration=duration,
+                video_backend="torchcodec",
+                height=torchcodec_metadata.height,
+                width=torchcodec_metadata.width,
+                frames_indices=None
+            )
+            
+            # Store actual timestamps as a custom attribute for _calculate_timestamps to use
+            video_metadata.actual_timestamps = actual_timestamps
+            
+            return video_tensor, video_metadata
+        finally:
+            del decoder
+
+    def fetch_videos(
+        self, 
+        video_url_or_urls: Union[str, Dict[str, Any], List[Union[str, Dict[str, Any]]]], 
+        sample_indices_fn=None, 
+        video_fps: Optional[float] = None,
+        min_frames: Optional[int] = None,
+        max_frames: Optional[int] = None,
+    ):
+        """
+        Override fetch_videos to use torchcodec for frame extraction.
+        
+        This method uses torchcodec with multithreading for efficient frame extraction.
+        Frame count is calculated by the calculate_num_frames method
+        (fps-based with min/max constraints).
+        
+        Args:
+            video_url_or_urls: Can be one of:
+                - str: Single video path
+                - Dict: Video with segments {"video_path": str, "segments": List[List[float]]}
+                - List[Union[str, Dict]]: List of video paths or segment dicts
+            sample_indices_fn: (Not used) Kept for compatibility with base class signature.
+            video_fps: Target frames per second for video sampling. If None, uses self.video_fps.
+            min_frames: Minimum number of frames to sample. If None, uses self.min_frames.
+            max_frames: Maximum number of frames to sample. If None, uses self.max_frames.
+        
+        Returns:
+            Tuple of (videos, metadata) where videos are torch.Tensors and metadata are VideoMetadata objects.
+        """
+        # Use provided values or fall back to self defaults
+        effective_video_fps = video_fps if video_fps is not None else self.video_fps
+        effective_min_frames = min_frames if min_frames is not None else self.min_frames
+        effective_max_frames = max_frames if max_frames is not None else self.max_frames
+        # Handle recursive calls for lists
+        if isinstance(video_url_or_urls, list):
+            all_videos = []
+            all_metadata = []
+            if len(video_url_or_urls) == 1:
+                per_video_max_frames = [effective_max_frames]
+            else:
+                per_video_max_frames = self._allocate_max_frames_for_multiple_videos(
+                    video_url_or_urls,
+                    effective_max_frames,
+                )
+            for x, allocated_max_frames in zip(video_url_or_urls, per_video_max_frames):
+                result = self.fetch_videos(
+                    x, 
+                    video_fps=effective_video_fps,
+                    min_frames=effective_min_frames,
+                    max_frames=allocated_max_frames,
+                )
+                # Check if result is from segment expansion (returns lists) or single item
+                if isinstance(result[0], list):
+                    all_videos.extend(result[0])
+                    all_metadata.extend(result[1])
+                else:
+                    all_videos.append(result[0])
+                    all_metadata.append(result[1])
+            return all_videos, all_metadata
+        
+        # Handle dict with segments - returns lists (one per segment)
+        if isinstance(video_url_or_urls, dict):
+            video_path = video_url_or_urls["video_path"]
+            segments = video_url_or_urls["segments"]
+            
+            return self._fetch_video_segments_batched(
+                video_path,
+                segments,
+                min_frames=effective_min_frames,
+                max_frames=effective_max_frames,
+                video_fps=effective_video_fps,
+            )
+        
+        # Single video path
+        video_path = video_url_or_urls
+        
+        # Clean video streams first (remove extra streams if needed)
+        video_path = cached_clean_video_streams(video_path)
+
+        decoder = None
+        try:
+            # Create VideoDecoder only once for both metadata and frame extraction
+            decoder = VideoDecoder(video_path, num_ffmpeg_threads=0)
+            torchcodec_metadata = decoder.metadata
+
+            duration = None
+            if torchcodec_metadata.end_stream_seconds_from_content is not None and torchcodec_metadata.begin_stream_seconds_from_content is not None:
+                duration = torchcodec_metadata.end_stream_seconds_from_content - torchcodec_metadata.begin_stream_seconds_from_content
+            
+            if duration is None or duration <= 0:
+                duration = torchcodec_metadata.duration_seconds
+            
+            # Use num_frames_from_content for accurate frame count (consistent with extraction)
+            total_frames_in_video = torchcodec_metadata.num_frames_from_content
+            
+            # Create VideoMetadata object for sample_frames method
+            temp_metadata = VideoMetadata(
+                total_num_frames=total_frames_in_video,
+                fps=torchcodec_metadata.average_fps,
+                duration=duration,
+                video_backend="torchcodec",
+                height=torchcodec_metadata.height,
+                width=torchcodec_metadata.width,
+                frames_indices=None
+            )
+            
+            # Use calculate_num_frames method to get the number of frames to sample
+            sample_frames_count = self.calculate_num_frames(
+                temp_metadata, 
+                fps=effective_video_fps,
+                min_frames=effective_min_frames,
+                max_frames=effective_max_frames,
+            )
+            
+            # Ensure sample count is valid
+            effective_sample_count = min(sample_frames_count, total_frames_in_video)
+            if effective_sample_count == 0:
+                raise ValueError(f"Cannot extract frames: video has 0 frames or specified frame count is 0")
+            
+            # Generate uniform frame indices
+            frame_indices = np.linspace(0, total_frames_in_video - 1, effective_sample_count).astype(np.int32)
+            # Ensure indices are valid and remove duplicates
+            frame_indices = np.unique(np.clip(frame_indices, 0, total_frames_in_video - 1))
+            
+            # Extract frames using multithreading (decoder is created inside each thread for thread safety)
+            result = decode_with_multithreading(frame_indices.tolist(), num_threads=self.num_extract_threads, video_path=video_path)
+            
+            # Extract frame tensor (N, C, H, W)
+            frames_tensor = result["data"]
+            
+            # Create final VideoMetadata object
+            video_metadata = VideoMetadata(
+                total_num_frames=len(frame_indices),
+                fps=torchcodec_metadata.average_fps,
+                duration=duration,
+                video_backend="torchcodec",
+                height=torchcodec_metadata.height,
+                width=torchcodec_metadata.width,
+                frames_indices=frame_indices
+            )
+            
+            # Ensure frames are in (T, C, H, W) format
+            if frames_tensor.dim() == 4:  # (N, C, H, W)
+                video_tensor = frames_tensor
+            else:
+                raise ValueError(f"Unexpected frame tensor shape: {frames_tensor.shape}")
+            
+            return video_tensor, video_metadata
+            
+        except Exception as e:
+            logger.error(f"Error loading video {video_path}: {e}")
+            traceback.print_exc()
+            raise ValueError(f"Failed to load video {video_path}: {e}")
+        finally:
+            if decoder is not None:
+                del decoder
+
+    def _preprocess(
+        self,
+        videos: list[torch.Tensor],
+        do_convert_rgb: bool = True,
+        do_resize: bool = True,
+        size: Optional[SizeDict] = None,
+        interpolation: PILImageResampling = PILImageResampling.BICUBIC,
+        do_rescale: bool = True,
+        rescale_factor: float = 1 / 255.0,
+        do_normalize: bool = True,
+        image_mean: Optional[Union[float, list[float]]] = None,
+        image_std: Optional[Union[float, list[float]]] = None,
+        patch_size: Optional[int] = None,
+        temporal_patch_size: Optional[int] = None,
+        merge_size: Optional[int] = None,
+        return_tensors: Optional[Union[str, TensorType]] = None,
+        **kwargs,
+    ):
+        grouped_videos, grouped_videos_index = group_videos_by_shape(videos)
+        resized_videos_grouped = {}
+
+        video_max_pixels = getattr(self, "video_max_pixels", None)
+        if video_max_pixels is not None:
+            total_volume = sum(
+                sv.shape[0] * sv.shape[1] * sv.shape[3] * sv.shape[4]
+                for sv in grouped_videos.values()
+            )
+        else:
+            total_volume = 0
+
+        for shape, stacked_videos in grouped_videos.items():
+            B, T, C, H, W = stacked_videos.shape
+            num_frames, height, width = T, H, W
+            # Convert to RGB if needed (reuse from base class)
+            if do_convert_rgb:
+                stacked_videos = self.convert_to_rgb(stacked_videos)
+            if do_resize:
+                if video_max_pixels is not None and total_volume > 0:
+                    allocated_max_pixels = int(video_max_pixels * (T * H * W) / total_volume)
+                else:
+                    allocated_max_pixels = size.longest_edge
+                resized_height, resized_width = smart_resize(
+                    num_frames=num_frames,
+                    height=height,
+                    width=width,
+                    temporal_factor=temporal_patch_size,
+                    factor=patch_size * merge_size,
+                    min_pixels=size.shortest_edge,
+                    max_pixels=allocated_max_pixels,
+                    per_frame_min_pixels=size.shortest_edge,
+                    per_frame_max_pixels=size.longest_edge,
+                )
+                stacked_videos = stacked_videos.view(B * T, C, H, W)
+                stacked_videos = self.resize(
+                    stacked_videos,
+                    size=SizeDict(height=resized_height, width=resized_width),
+                    interpolation=interpolation,
+                )
+                stacked_videos = stacked_videos.view(B, T, C, resized_height, resized_width)
+            resized_videos_grouped[shape] = stacked_videos
+        resized_videos = reorder_videos(resized_videos_grouped, grouped_videos_index)
+
+        # Group videos by size for further processing
+        # Needed in case do_resize is False, or resize returns videos with different sizes
+        grouped_videos, grouped_videos_index = group_videos_by_shape(resized_videos)
+        processed_videos_grouped = {}
+        processed_grids = {}
+        for shape, stacked_videos in grouped_videos.items():
+            resized_height, resized_width = get_image_size(stacked_videos[0], channel_dim=ChannelDimension.FIRST)
+
+            # Fused rescale and normalize
+            stacked_videos = self.rescale_and_normalize(
+                stacked_videos, do_rescale, rescale_factor, do_normalize, image_mean, image_std
+            )
+            patches = stacked_videos
+
+            # Check that videos have `num_frames` divisible by `temporal_patch_size`
+            if patches.shape[1] % temporal_patch_size != 0:
+                repeats = patches[:, -1:].repeat(1, temporal_patch_size - 1, 1, 1, 1)
+                patches = torch.cat([patches, repeats], dim=1)
+            batch_size, grid_t, channel = patches.shape[:3]
+            grid_t = grid_t // temporal_patch_size
+            grid_h, grid_w = resized_height // patch_size, resized_width // patch_size
+
+            patches = patches.view(
+                batch_size,
+                grid_t,
+                temporal_patch_size,
+                channel,
+                grid_h // merge_size,
+                merge_size,
+                patch_size,
+                grid_w // merge_size,
+                merge_size,
+                patch_size,
+            )
+            patches = patches.permute(0, 1, 4, 7, 5, 8, 3, 2, 6, 9)
+            flatten_patches = patches.reshape(
+                batch_size,
+                grid_t * grid_h * grid_w,
+                channel * temporal_patch_size * patch_size * patch_size,
+            )
+
+            processed_videos_grouped[shape] = flatten_patches
+            processed_grids[shape] = [[grid_t, grid_h, grid_w]] * batch_size
+
+        processed_videos = reorder_videos(processed_videos_grouped, grouped_videos_index)
+        processed_grids = reorder_videos(processed_grids, grouped_videos_index)
+        pixel_values_videos = torch.cat(processed_videos, dim=0)
+        video_grid_thw = torch.tensor(processed_grids)
+        data = {
+            "pixel_values_videos": pixel_values_videos,
+            "video_grid_thw": video_grid_thw,
+        }
+
+        return BatchFeature(data=data, tensor_type=return_tensors)
+
+    def preprocess(
+        self,
+        videos: Union[str, Dict[str, Any], List[Union[str, Dict[str, Any]]]],
+        **kwargs,
+    ) -> BatchFeature:
+        """
+        Preprocess videos for the model.
+        
+        This method overrides the base class to handle two video input formats:
+        1. String path: "path/to/video.mp4"
+        2. Dict with segments: {"video_path": "...", "segment": [[start, end], [time], ...]}
+        
+        Args:
+            videos: Video input(s) in one of the supported formats.
+            **kwargs: Additional arguments passed to _preprocess.
+            
+        Returns:
+            BatchFeature with pixel_values_videos, video_grid_thw, and optionally video_metadata.
+        """
+        # Validate kwargs
+        validate_kwargs(
+            captured_kwargs=kwargs.keys(),
+            valid_processor_keys=list(self.valid_kwargs.__annotations__.keys()) + ["return_tensors"],
+        )
+        
+        # Set default kwargs from self
+        for kwarg_name in self.valid_kwargs.__annotations__:
+            kwargs.setdefault(kwarg_name, getattr(self, kwarg_name, None))
+        
+        # Pop kwargs that are handled separately
+        return_tensors = kwargs.pop("return_tensors", None)
+        return_metadata = kwargs.pop("return_metadata", False)
+        input_data_format = kwargs.pop("input_data_format", None)
+        device = kwargs.pop("device", None)
+        kwargs.pop("video_metadata", None)  # We generate our own metadata
+        kwargs.pop("do_sample_frames", None)  # We handle sampling ourselves
+        kwargs.pop("data_format", None)  # Not used
+        
+        # Normalize input to list format
+        if not isinstance(videos, list):
+            videos = [videos]
+        
+        # Get video processing params from kwargs (may be passed explicitly for per-batch configuration)
+        video_fps = kwargs.pop("video_fps", None)
+        min_frames = kwargs.pop("min_frames", None)
+        max_frames = kwargs.pop("max_frames", None)
+        
+        # Use fetch_videos to handle both string and dict formats
+        video_tensors, video_metadata = self.fetch_videos(
+            videos, 
+            video_fps=video_fps,
+            min_frames=min_frames,
+            max_frames=max_frames,
+        )
+        
+        # Prepare video tensors using _prepare_input_videos
+        prepared_videos = self._prepare_input_videos(
+            videos=video_tensors,
+            input_data_format=input_data_format,
+            device=device,
+        )
+        
+        # Process kwargs for _preprocess
+        kwargs = self._further_process_kwargs(**kwargs)
+        self._validate_preprocess_kwargs(**kwargs)
+        
+        # Call _preprocess with prepared videos
+        result = self._preprocess(videos=prepared_videos, return_tensors=return_tensors, **kwargs)
+        
+        # Add metadata if requested
+        if return_metadata:
+            result["video_metadata"] = video_metadata
+        
+        return result
+
+
+__all__ = ["MossVLVideoProcessor"]
+