Transformers documentation

SarvamMLA

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.23.0).
Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

This model was contributed to Hugging Face Transformers on 2026-11-19.

SarvamMLA

Overview

SarvamMLA is a 105B parameter Mixture of Experts (MoE) language model developed by Sarvam AI. It uses Multi-head Latent Attention (MLA) combined with sparse MoE routing, architecturally identical to DeepSeek-V3.

Key architectural features:

  • Multi-head Latent Attention (MLA): Low-rank KV compression with decoupled RoPE, reducing KV cache memory while maintaining performance.
  • Sparse Mixture of Experts: 128 routed experts with 8 active per token, plus 1 shared expert. The first layer uses a dense MLP.
  • DeepSeek YaRN RoPE: Extended context support up to 131K tokens via YaRN rotary position embeddings.
  • Sigmoid routing with group-based top-k: Token-choice routing using sigmoid scores with expert bias correction and group-aware selection.

This model uses the DeepSeek-V3 architecture with a custom configuration. See the DeepSeek-V3 documentation for model and forward reference.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "sarvamai/sarvam-105b",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("sarvamai/sarvam-105b")

inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

SarvamMLAConfig

class transformers.SarvamMLAConfig

< >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonevocab_size: int = 262144hidden_size: int = 4096intermediate_size: int = 16384moe_intermediate_size: int = 2048num_hidden_layers: int = 32num_attention_heads: int = 64num_key_value_heads: int | None = Nonen_shared_experts: int = 1n_routed_experts: int = 128output_router_logits: bool = Falserouted_scaling_factor: float = 2.5kv_lora_rank: int = 512q_lora_rank: int | None = Noneqk_rope_head_dim: int = 64v_head_dim: int | None = 128qk_nope_head_dim: int = 128n_group: int | None = 1topk_group: int | None = 1num_experts_per_tok: int | None = 8first_k_dense_replace: int | None = 1norm_topk_prob: bool | None = Truehidden_act: str = 'silu'max_position_embeddings: int = 4096initializer_range: float = 0.006rms_norm_eps: float = 1e-06use_cache: bool = Truepad_token_id: int | None = Nonebos_token_id: int | None = 0eos_token_id: int | list[int] | None = 1pretraining_tp: int | None = 1tie_word_embeddings: bool = Falserope_parameters: transformers.modeling_rope_utils.RopeParameters | dict | None = Nonerope_interleave: bool | None = Trueattention_bias: bool = Falseattention_dropout: float | int | None = 0.0num_mtp_layers: int = 0 )

Parameters

  • vocab_size (int, optional, defaults to 262144) — Vocabulary size of the model. Defines the number of different tokens that can be represented by the input_ids.
  • hidden_size (int, optional, defaults to 4096) — Dimension of the hidden representations.
  • intermediate_size (int, optional, defaults to 16384) — Dimension of the MLP representations.
  • moe_intermediate_size (int, optional, defaults to 2048) — Intermediate size of the routed expert MLPs.
  • num_hidden_layers (int, optional, defaults to 32) — Number of hidden layers in the Transformer decoder.
  • num_attention_heads (int, optional, defaults to 64) — Number of attention heads for each attention layer in the Transformer decoder.
  • num_key_value_heads (int, optional) — This is the number of key_value heads that should be used to implement Grouped Query Attention. If num_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), if num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out this paper. If it is not specified, will default to num_attention_heads.
  • n_shared_experts (int, optional, defaults to 1) — Number of shared experts.
  • n_routed_experts (int, optional, defaults to 128) — Number of routed experts.
  • output_router_logits (bool, optional, defaults to False) — Whether or not the router logits should be returned by the model. Enabling this will also allow the model to output the auxiliary loss, including load balancing loss and router z-loss.
  • routed_scaling_factor (float, optional, defaults to 2.5) — Scaling factor or routed experts.
  • kv_lora_rank (int, optional, defaults to 512) — Rank of the LoRA matrices for key and value projections.
  • q_lora_rank (int, optional) — Rank of the LoRA matrices for query projections.
  • qk_rope_head_dim (int, optional, defaults to 64) — Dimension of the query/key heads that use rotary position embeddings.
  • v_head_dim (int, optional, defaults to 128) — Dimension of the value heads.
  • qk_nope_head_dim (int, optional, defaults to 128) — Dimension of the query/key heads that don’t use rotary position embeddings.
  • n_group (int, optional, defaults to 1) — Number of groups for routed experts. The released checkpoints route over all experts at once.
  • topk_group (int, optional, defaults to 1) — Number of selected groups for each token.
  • num_experts_per_tok (int, optional, defaults to 8) — Number of experts to route each token to. This is the top-k value for the token-choice routing.
  • first_k_dense_replace (int, optional, defaults to 1) — Number of dense layers in shallow layers(embed->dense->moe->moe…->lm_head). --k dense layers—/
  • norm_topk_prob (bool, optional, defaults to True) — Whether to normalize the weights of the routed experts.
  • hidden_act (str, optional, defaults to silu) — The non-linear activation function (function or string) in the decoder. For example, "gelu", "relu", "silu", etc.
  • max_position_embeddings (int, optional, defaults to 4096) — The maximum sequence length that this model might ever be used with.
  • initializer_range (float, optional, defaults to 0.006) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
  • rms_norm_eps (float, optional, defaults to 1e-06) — The epsilon used by the rms normalization layers.
  • use_cache (bool, optional, defaults to True) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if config.is_decoder=True or when the model is a decoder-only generative model.
  • pad_token_id (int, optional) — Token id used for padding in the vocabulary.
  • bos_token_id (int, optional, defaults to 0) — Token id used for beginning-of-stream in the vocabulary.
  • eos_token_id (Union[int, list[int]], optional, defaults to 1) — Token id used for end-of-stream in the vocabulary.
  • pretraining_tp (int, optional, defaults to 1) — Experimental feature. Tensor parallelism rank used during pretraining. Please refer to this document to understand more about it. This value is necessary to ensure exact reproducibility of the pretraining results. Please refer to this issue.
  • tie_word_embeddings (bool, optional, defaults to False) — Whether to tie weight embeddings according to model’s tied_weights_keys mapping.
  • rope_parameters (Union[~modeling_rope_utils.RopeParameters, dict], optional) — Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for rope_theta and optionally parameters used for scaling in case you want to use RoPE with longer max_position_embeddings.
  • rope_interleave (bool, optional, defaults to True) — Whether to interleave the rotary position embeddings.
  • attention_bias (bool, optional, defaults to False) — Whether to use a bias in the query, key, value and output projection layers during self-attention.
  • attention_dropout (Union[float, int], optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
  • num_mtp_layers (int, optional, defaults to 0) — Number of Multi-Token Prediction (MTP) modules available to append after the base transformer model. The released checkpoints do not include MTP weights.

This is the configuration class to store the configuration of a DeepseekV3Model. It is used to instantiate a Sarvam Mla model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the sarvamai/sarvam-105b

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

Example:

>>> from transformers import SarvamMLAConfig

>>> # Initializing a SarvamMLA style configuration
>>> configuration = SarvamMLAConfig()

>>> # Accessing the model configuration
>>> configuration = model.config
Update on GitHub