Transformers documentation
SarvamMLA
This model was contributed to Hugging Face Transformers on 2026-11-19.
SarvamMLA
Overview
SarvamMLA is a 105B parameter Mixture of Experts (MoE) language model developed by Sarvam AI. It uses Multi-head Latent Attention (MLA) combined with sparse MoE routing, architecturally identical to DeepSeek-V3.
Key architectural features:
- Multi-head Latent Attention (MLA): Low-rank KV compression with decoupled RoPE, reducing KV cache memory while maintaining performance.
- Sparse Mixture of Experts: 128 routed experts with 8 active per token, plus 1 shared expert. The first layer uses a dense MLP.
- DeepSeek YaRN RoPE: Extended context support up to 131K tokens via YaRN rotary position embeddings.
- Sigmoid routing with group-based top-k: Token-choice routing using sigmoid scores with expert bias correction and group-aware selection.
This model uses the DeepSeek-V3 architecture with a custom configuration. See the DeepSeek-V3 documentation for model and forward reference.
Usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"sarvamai/sarvam-105b",
device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("sarvamai/sarvam-105b")
inputs = tokenizer("Hello, how are you?", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))SarvamMLAConfig
class transformers.SarvamMLAConfig
< source >( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonevocab_size: int = 262144hidden_size: int = 4096intermediate_size: int = 16384moe_intermediate_size: int = 2048num_hidden_layers: int = 32num_attention_heads: int = 64num_key_value_heads: int | None = Nonen_shared_experts: int = 1n_routed_experts: int = 128output_router_logits: bool = Falserouted_scaling_factor: float = 2.5kv_lora_rank: int = 512q_lora_rank: int | None = Noneqk_rope_head_dim: int = 64v_head_dim: int | None = 128qk_nope_head_dim: int = 128n_group: int | None = 1topk_group: int | None = 1num_experts_per_tok: int | None = 8first_k_dense_replace: int | None = 1norm_topk_prob: bool | None = Truehidden_act: str = 'silu'max_position_embeddings: int = 4096initializer_range: float = 0.006rms_norm_eps: float = 1e-06use_cache: bool = Truepad_token_id: int | None = Nonebos_token_id: int | None = 0eos_token_id: int | list[int] | None = 1pretraining_tp: int | None = 1tie_word_embeddings: bool = Falserope_parameters: transformers.modeling_rope_utils.RopeParameters | dict | None = Nonerope_interleave: bool | None = Trueattention_bias: bool = Falseattention_dropout: float | int | None = 0.0num_mtp_layers: int = 0 )
Parameters
- vocab_size (
int, optional, defaults to262144) — Vocabulary size of the model. Defines the number of different tokens that can be represented by theinput_ids. - hidden_size (
int, optional, defaults to4096) — Dimension of the hidden representations. - intermediate_size (
int, optional, defaults to16384) — Dimension of the MLP representations. - moe_intermediate_size (
int, optional, defaults to2048) — Intermediate size of the routed expert MLPs. - num_hidden_layers (
int, optional, defaults to32) — Number of hidden layers in the Transformer decoder. - num_attention_heads (
int, optional, defaults to64) — Number of attention heads for each attention layer in the Transformer decoder. - num_key_value_heads (
int, optional) — This is the number of key_value heads that should be used to implement Grouped Query Attention. Ifnum_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), ifnum_key_value_heads=1the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out this paper. If it is not specified, will default tonum_attention_heads. - n_shared_experts (
int, optional, defaults to1) — Number of shared experts. - n_routed_experts (
int, optional, defaults to128) — Number of routed experts. - output_router_logits (
bool, optional, defaults toFalse) — Whether or not the router logits should be returned by the model. Enabling this will also allow the model to output the auxiliary loss, including load balancing loss and router z-loss. - routed_scaling_factor (
float, optional, defaults to2.5) — Scaling factor or routed experts. - kv_lora_rank (
int, optional, defaults to512) — Rank of the LoRA matrices for key and value projections. - q_lora_rank (
int, optional) — Rank of the LoRA matrices for query projections. - qk_rope_head_dim (
int, optional, defaults to64) — Dimension of the query/key heads that use rotary position embeddings. - v_head_dim (
int, optional, defaults to128) — Dimension of the value heads. - qk_nope_head_dim (
int, optional, defaults to128) — Dimension of the query/key heads that don’t use rotary position embeddings. - n_group (
int, optional, defaults to 1) — Number of groups for routed experts. The released checkpoints route over all experts at once. - topk_group (
int, optional, defaults to 1) — Number of selected groups for each token. - num_experts_per_tok (
int, optional, defaults to8) — Number of experts to route each token to. This is the top-k value for the token-choice routing. - first_k_dense_replace (
int, optional, defaults to 1) — Number of dense layers in shallow layers(embed->dense->moe->moe…->lm_head). --k dense layers—/ - norm_topk_prob (
bool, optional, defaults toTrue) — Whether to normalize the weights of the routed experts. - hidden_act (
str, optional, defaults tosilu) — The non-linear activation function (function or string) in the decoder. For example,"gelu","relu","silu", etc. - max_position_embeddings (
int, optional, defaults to4096) — The maximum sequence length that this model might ever be used with. - initializer_range (
float, optional, defaults to0.006) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices. - rms_norm_eps (
float, optional, defaults to1e-06) — The epsilon used by the rms normalization layers. - use_cache (
bool, optional, defaults toTrue) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant ifconfig.is_decoder=Trueor when the model is a decoder-only generative model. - pad_token_id (
int, optional) — Token id used for padding in the vocabulary. - bos_token_id (
int, optional, defaults to0) — Token id used for beginning-of-stream in the vocabulary. - eos_token_id (
Union[int, list[int]], optional, defaults to1) — Token id used for end-of-stream in the vocabulary. - pretraining_tp (
int, optional, defaults to1) — Experimental feature. Tensor parallelism rank used during pretraining. Please refer to this document to understand more about it. This value is necessary to ensure exact reproducibility of the pretraining results. Please refer to this issue. - tie_word_embeddings (
bool, optional, defaults toFalse) — Whether to tie weight embeddings according to model’stied_weights_keysmapping. - rope_parameters (
Union[~modeling_rope_utils.RopeParameters, dict], optional) — Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value forrope_thetaand optionally parameters used for scaling in case you want to use RoPE with longermax_position_embeddings. - rope_interleave (
bool, optional, defaults toTrue) — Whether to interleave the rotary position embeddings. - attention_bias (
bool, optional, defaults toFalse) — Whether to use a bias in the query, key, value and output projection layers during self-attention. - attention_dropout (
Union[float, int], optional, defaults to0.0) — The dropout ratio for the attention probabilities. - num_mtp_layers (
int, optional, defaults to 0) — Number of Multi-Token Prediction (MTP) modules available to append after the base transformer model. The released checkpoints do not include MTP weights.
This is the configuration class to store the configuration of a DeepseekV3Model. It is used to instantiate a Sarvam Mla model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the sarvamai/sarvam-105b
Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.