Dots.llm1.inst: The Instruction-Tuned Mixture-of-Experts Language Model You Need to Know About
Overview
dots.llm1.inst is an instruction-tuned mixture-of-experts language model built by dots-studio that activates 14 billion parameters out of a total of 142 billion parameters during inference, delivering performance comparable to Qwen2.5-72B while requiring significantly less computational resources. The model uses a 62-layer architecture with 32 attention heads, multi-head attention with QK-Norm in each attention layer, and fine-grained MoE routing that selects top-6 experts out of 128 routed experts plus 2 shared experts. It supports a 32,768 token context window, handles both English and Chinese text, and uses the transformers library for inference.
The model was pretrained using a three-stage data processing framework on high-quality, non-synthetic corpora without synthetic data generation, and then instruction-tuned through supervised fine-tuning. All training happens under the MIT license, enabling both academic and commercial use.
Best Use Cases
Chat and instruction-following tasks.This model is specifically instruction-tuned, making it ideal for conversational AI applications, task completion,...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
