Baidu has released the weights for ERNIE 5.5 Flash Base, a sparse mixture-of-experts language model intended to give developers a starting point for specialized training. The model contains 280 billion parameters but activates 14 billion for each token, an architecture designed to limit computation while preserving the capacity of a much larger network.

The company says the checkpoint was trained from scratch on 33.5 trillion tokens. Pretraining used sequences of up to 128,000 tokens, while the architecture supports contexts of as many as one million tokens during fine-tuning and inference. The released weights represent the end of pretraining, before instruction tuning, reinforcement learning or other post-training stages.

That distinction makes the release a foundation for researchers rather than a finished conversational product. Developers can adapt it for particular domains, build instruction-following behavior or test new reinforcement-learning methods without paying the full cost of large-scale pretraining. Baidu says its training corpus emphasized capabilities that are difficult to add later, including repository-scale code and long-context understanding.

ERNIE 5.5 Flash Base uses 43 transformer layers, a hidden dimension of 4,096 and 64 attention heads. Its main efficiency technique is a hybrid of full-context and compressed attention. One layer in seven retains token-level access to the complete history using multi-head latent attention. Most other layers compress each block of 128 tokens into one cached entry while retaining a short window of recent tokens.

This design reduces the memory required for the key-value cache, which can become a major cost when models process long documents or maintain extended agent conversations. The full-context layers also use sparse attention: an indexer selects the most relevant previous positions for each query, preventing attention computation from rising directly with the entire context length.

Baidu introduced several other experimental components. A multi-router system combines scores from independently initialized routers when choosing experts, aiming to make selection less sensitive to noisy inputs. Four parallel residual streams replace a conventional single residual path in most of the network. The output layer uses a learned piecewise temperature intended to preserve several plausible next tokens while suppressing unlikely ones.

The model also replaces a separate multi-token prediction module with a method called ECHO, which reuses the backbone’s final layer to predict future tokens. Baidu says this directs more of the future-token training signal into the main model rather than an attached module.

The release gives the open-model community a large base checkpoint with unusually long-context ambitions, but its practical value will depend on independent testing and the quality of downstream training. Baidu’s performance and efficiency descriptions are the developer’s own claims; the base weights have not yet been shaped into a general-purpose assistant.