Duplicate from rednote-hilab/dots.llm1.base

Co-authored-by: redmoe-ai-v1 <[email protected]>

.gitattributes

@@ -0,0 +1,38 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+figures/performance.png filter=lfs diff=lfs merge=lfs -text
+figures/new_logo.png filter=lfs diff=lfs merge=lfs -text
+figures/wechat.png filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 rednote-hilab
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,192 @@
+---
+license: mit
+license_link: https://huggingface.co/rednote-hilab/dots.llm1.base/blob/main/LICENSE
+library_name: transformers
+language:
+- en
+- zh
+---
+
+# dots1
+
+<p align="center">
+    <img src="figures/new_logo2.png" width="300"/>
+<p>
+
+<p align="center">
+    &nbsp&nbsp🤗 <a href="https://huggingface.co/rednote-hilab">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp 📑 <a href="https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf">Paper</a> &nbsp&nbsp 
+<br>
+🖥️ <a href="https://huggingface.co/spaces/rednote-hilab/dots-demo">Demo</a>&nbsp&nbsp | &nbsp&nbsp💬 <a href="figures/wechat.png">WeChat (微信)</a>&nbsp&nbsp | &nbsp&nbsp📕 <a href="https://www.xiaohongshu.com/user/profile/683ffe42000000001d021a4c">rednote</a>&nbsp&nbsp
+</p>
+
+
+Visit our Hugging Face (click links above), search checkpoints with names starting with `dots.llm1` or visit the [dots1 collection](https://huggingface.co/collections/rednote-hilab/dotsllm1-68246aaaaba3363374a8aa7c), and you will find all you need! Enjoy!
+
+
+## News
+
+- 2025.06.06: We released the `dots.llm1` series. Check our [report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf) for more details!
+
+
+## 1. Introduction
+
+
+The `dots.llm1` model is a large-scale MoE model that activates 14B parameters out of a total of 142B parameters, delivering performance on par with state-of-the-art models. 
+Leveraging our meticulously crafted and efficient data processing pipeline, `dots.llm1` achieves performance comparable to Qwen2.5-72B after pretrained on 11.2T high-quality tokens without synthetic data. To foster further research, we open-source intermediate training checkpoints at every one trillion tokens, providing valuable insights into the learning dynamics of large language models.
+
+
+<p align="center">
+  <img width="90%" src="./figures/performance.png">
+</p>
+
+## 2. Model Summary
+
+**This repo contains the base and instruction-tuned `dots.llm1` model**. which has the following features:
+
+- Type: A MoE model with 14B activated and 142B total parameters trained on 11.2T tokens.
+- Training Stages: Pretraining and SFT.
+- Architecture: Multi-head Attention with QK-Norm in attention Layer, fine-grained MoE utilizing top-6 out of 128 routed experts, plus 2 shared experts.
+- Number of Layers: 62
+- Number of Attention Heads: 32
+- Supported Languages: English, Chinese
+- Context Length: 32,768 tokens
+- License: MIT
+
+The highlights from `dots.llm1` include:
+
+- **Enhanced Data Processing**: We propose a scalable and fine-grained *three-stage* data processing framework designed to generate large-scale, high-quality and diverse data for pretraining.
+- **No Synthetic Data during Pretraining**: *11.2 trillion* high-quality non-synthetic tokens was used in base model pretraining.
+- **Performance and Cost Efficiency**: `dots.llm1` is an open-source model that activates only *14B* parameters at inference, delivering both comprehensive capabilities and high computational efficiency.
+- **Infrastructure**: We introduce an innovative MoE all-to-all communication and computation overlapping recipe based on interleaved 1F1B pipeline scheduling and an efficient grouped GEMM implementation to boost computational efficiency.
+- **Open Accessibility to Model Dynamics**: Intermediate model checkpoints for *every 1T tokens* trained are released, facilitating future research into the learning dynamics of large language models.
+
+## 3. Example Usage
+
+### Model Downloads
+
+<div align="center">
+
+| **Model** | **#Total Params** | **#Activated Params** | **Context Length** | **Download Link** |
+| :------------: | :------------: | :------------: | :------------: | :------------: |
+| dots.llm1.base | 142B | 14B | 32K   | [🤗 Hugging Face](https://huggingface.co/rednote-hilab/dots.llm1.base)   |
+| dots.llm1.inst  | 142B | 14B |  32K   | [🤗 Hugging Face](https://huggingface.co/rednote-hilab/dots.llm1.inst)   |
+
+</div>
+
+### Docker (recommended)
+
+
+The docker images are available on [Docker Hub](https://hub.docker.com/repository/docker/rednotehilab/dots1/tags), based on the official images.
+
+You can start a server via vllm.
+
+```shell
+docker run --gpus all \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    -p 8000:8000 \
+    --ipc=host \
+    rednotehilab/dots1:vllm-openai-v0.9.0.1 \
+    --model rednote-hilab/dots.llm1.inst \
+    --tensor-parallel-size 8 \
+    --trust-remote-code \
+    --served-model-name dots1
+```
+
+Then you can verify whether the model is running successfully in the following way.
+
+```shell
+curl http://localhost:8000/v1/chat/completions \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "dots1",
+        "messages": [
+            {"role": "system", "content": "You are a helpful assistant."},
+            {"role": "user", "content": "Who won the world series in 2020?"}
+        ],
+        "max_tokens": 32,
+        "temperature": 0
+    }'
+```
+
+
+### Inference with huggingface
+
+We are working to merge it into Transformers ([PR #38143](https://github.com/huggingface/transformers/pull/38143)).
+
+#### Text Completion
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
+
+model_name = "rednote-hilab/dots.llm1.base"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)
+
+text = "An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is"
+inputs = tokenizer(text, return_tensors="pt")
+outputs = model.generate(**inputs.to(model.device), max_new_tokens=100)
+result = tokenizer.decode(outputs[0], skip_special_tokens=True)
+print(result)
+```
+
+#### Chat Completion
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig
+
+model_name = "rednote-hilab/dots.llm1.inst"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+
+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.bfloat16)
+
+messages = [
+    {"role": "user", "content": "Write a piece of quicksort code in C++"}
+]
+input_tensor = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
+outputs = model.generate(input_tensor.to(model.device), max_new_tokens=200)
+
+result = tokenizer.decode(outputs[0][input_tensor.shape[1]:], skip_special_tokens=True)
+print(result)
+```
+
+### Inference with vllm
+
+[vLLM](https://github.com/vllm-project/vllm) is a high-throughput and memory-efficient inference and serving engine for LLMs. Official support for this feature is covered in [PR #18254](https://github.com/vllm-project/vllm/pull/18254).
+
+```shell
+vllm serve dots.llm1.inst --port 8000 --tensor-parallel-size 8
+```
+
+An OpenAI-compatible API will be available at `http://localhost:8000/v1`.
+
+### Inference with sglang
+
+[SGLang](https://github.com/sgl-project/sglang) is a fast serving framework for large language models and vision language models. SGLang could be used to launch a server with OpenAI-compatible API service. Official support for this feature is covered in [PR #6471](https://github.com/sgl-project/sglang/pull/6471).
+
+Getting started is as simple as running:
+
+```shell
+python -m sglang.launch_server --model-path dots.llm1.inst --tp 8 --host 0.0.0.0 --port 8000
+```
+
+An OpenAI-compatible API will be available at `http://localhost:8000/v1`.
+
+## 4. Evaluation Results
+
+Detailed evaluation results are reported in this [📑 report](https://github.com/rednote-hilab/dots.llm1/blob/main/dots1_tech_report.pdf).
+
+## Citation
+
+If you find `dots.llm1` is useful or want to use in your projects, please kindly cite our paper:
+
+```
+@article{dots1,
+      title={dots.llm1 Technical Report}, 
+      author={rednote-hilab},
+      journal={arXiv preprint arXiv:TBD},
+      year={2025}
+}
+```

config.json

@@ -0,0 +1,38 @@
+{
+  "architectures": [
+    "Dots1ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": null,
+  "eos_token_id": 151643,
+  "first_k_dense_replace": 1,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 10944,
+  "max_position_embeddings": 32768,
+  "model_type": "dots1",
+  "moe_intermediate_size": 1408,
+  "moe_layer_freq": 1,
+  "n_routed_experts": 128,
+  "n_shared_experts": 2,
+  "norm_topk_prob": true,
+  "num_attention_heads": 32,
+  "num_experts_per_tok": 6,
+  "num_hidden_layers": 62,
+  "num_key_value_heads": 32,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000000,
+  "routed_scaling_factor": 2.5,
+  "sliding_window": null,
+  "scoring_func": "noaux_tc",
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.46.3",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vocab_size": 152064
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 151643,
+  "eos_token_id": 151643,
+  "transformers_version": "4.46.3"
+}

model-00001-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9d49fdd6709c3d0ce0dea361861380df8931b844c47380a686cc2bbe2f6c6a49
+size 4290290528

model-00002-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c4406c15b74a1bfda6b810945834e456313bb651b4e80928ebaff259b0da16d7
+size 4287689952

model-00003-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b5681d9f1d4f4e7712f1390e5781ecf5c715cdfe083df667fa70f4bb6a6da0ad
+size 4287690208

model-00004-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc6e994dc951aa15c0473c92660c5caa25af4a02e8a3ff93c06fcaa2b61039dc
+size 4287690208

model-00005-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b41977295314b2a0d022068cbda68f0e89d5e5bf5d70e79f8830ac938515b9d4
+size 4287690216

model-00006-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6bf8c31cff99600fce52e7500b892a8851cbbb41ca16c4adcc318eef9d47edf6
+size 4290819520

model-00007-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b98f276b4eb94bb6e6beef33176f5dcdb07b3d1706f1af75a493d0485b83eb4b
+size 4287690248

model-00008-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1e7aa2e2b2c443edff36587803b20bb3dfec08c1c1d3a1b98c81e3817a0defe4
+size 4287690248

model-00009-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bdfcf2c2d5e7f664b51defb6160f4877bca129ee1b8401418283f9a33a12964c
+size 4287690240

model-00010-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:67f2c1d0f29a708c024341550ae32ccc1a9e9a1171cf1322aefdbcbb90f0e350
+size 4287690232

model-00011-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bc1adf542d3853f5c0ec52c4c01bff2d00138f3b6081de0b7b868b5f7cc07c41
+size 4287690232

model-00012-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca9d244f137a1532b1e16751126a5fa09c87ba99499d34a7d355d7f63a44ffd3
+size 4287690232

model-00013-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11242ffebd90a3445d399e0328eeddf34cfa3bdcdfffc3ddaa307aacf91afd3c
+size 4287690072

model-00014-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:520acecbd3265af0cdf3644121ca056b20df606a3ab7fc8121a60fecbdf9ed99
+size 4287690000

model-00015-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:abe2d1f90b0989498b0d646762364cc8d86176674aae06679b2145ac65c7c86e
+size 4287690232

model-00016-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ea87941b1e85a5d8137a3c6fce4b756a1ffe502440c166fec9f510c72dbb56f0
+size 4287690208

model-00017-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:52540eba653a20f9d491b2aa30425fece044ab153720de7009e53b8fee59d28b
+size 4287690200

model-00018-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0a9959428b6d8756204d23a5bf319eac9b963148a2ff45e8bca9f774a2ce839
+size 4287690208

model-00019-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dfc5ed9f7ba6cfbe997deb9cd5566509b937ae10ff12349a3b378a641984d3fb
+size 4280334120

model-00020-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:26d50d67c9195dc5db9ee4619ea4d4e4ccc18b859e8f44aff47a91dab6ad5a5a
+size 4286641184

model-00021-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94246a1b2bc662d84622e79121c1308e7a8a7518ad118e1448f057543b7fe72c
+size 4287690248

model-00022-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a9f1b05307d49d8b4dcd47fdc21414fdbd9b7ca03c2e7d147b98b11538d01227
+size 4287690248

model-00023-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2b8903626d00cc89609571a57700db263b178f6af3745804f0acc79854867d2
+size 4287690232

model-00024-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89384643591a0ba8a39aa01ecb5f542bd981e3aa886c2e0b2577ea5b224a2558
+size 4287690232

model-00025-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:73040aeda4fce0a83d772149314d79b28b97358559a6cef1542b2f7088fd175e
+size 4287690032

model-00026-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:102a06dadca9106fb4a21150154b019512ff65f5173e3cbcc9b75a30510e7619
+size 4287690040

model-00027-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74a0323d50e7d98b4ba79ed45503b9fbc6a3cc359046a8593a4aecb3e69ee512
+size 4287690232

model-00028-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f9f11bc0fdacc010614211f494b95b980ca43fb2475894e066bb0176b9f42db6
+size 4287690232

model-00029-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:812d937410397a7f5a651c81397521ef67e2223a3d01e39a31dba59df14c0924
+size 4287690216

model-00030-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63ae456aec8e210e6430e18e3bd4f986e6eb9f0f65a9708cd5f3112cd19fdce6
+size 4287690208

model-00031-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fce896c941352359c60fd765316d6d6dd0a283d13b2c4abd501fbd01a95b283c
+size 4287690208

model-00032-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:91a9a9a0465f602e33a98dbb211ad396a13449c9faf55879b6a783aaa7a40421
+size 4289779584

model-00033-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce843941e1fd7591ca10e2d5725dfc0cd575ed38c238b2f5d6be6e761b24b2c5
+size 4288730168

model-00034-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ca34bf5695346747c6795c03b1d01d8f7158b291b0c3555bb87c15c180b211f3
+size 4287690248

model-00035-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:57e51388ffc129840e1d3cd21b2e1a13fe9b93da77ecb756d80a9d39114d1a17
+size 4287690248

model-00036-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:83d99c7a1d9daea01217c5432aff47ada42fad3c9d7183ec7d8f76525b81f955
+size 4287690240

model-00037-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d8fd47f7486afcc47293209e17278b4371a6c819b510e11401a633c19232e79
+size 4287689984

model-00038-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:28ccc69f954f3305993644738349cf64991770f2e17cdcb3aa71acf87e57d827
+size 4287690088

model-00039-of-00067.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cd3585b10b1ac4a49f8f1fd778a98234e5b6870c4021dc49af99b3362e9ef84a
+size 4287690232