aplux/MiniCPM-1B-sft

MiniCPM-1B-sft

MiniCPM is an End-Size LLM developed by ModelBest Inc. and TsinghuaNLP, with only 1.2B parameters excluding embeddings.

Model Details

• MiniCPM has very close performance compared with Mistral-7B on open-sourced general benchmarks with better ability on Chinese, Mathmetics and Coding after SFT. The overall performance exceeds Llama2-13B, MPT-30B, Falcon-40B, etc.
• After DPO, MiniCPM outperforms Llama2-70B-Chat, Vicuna-33B, Mistral-7B-Instruct-v0.1, Zephyr-7B-alpha, etc. on MTBench.
• MiniCPM-V, based on MiniCPM-2B, achieves the best overall performance among multimodel models of the same scale, surpassing existing multimodal large models built on Phi-2 and achieving performance comparable to or even better than 9.6B Qwen-VL-Chat on some tasks.
• MiniCPM can be deployed and infer on smartphones, and the speed of streaming output is relatively higher than the verbal speed of human. MiniCPM-V is the first multi-modal models that can be deployed on smartphones.
• The cost of developing based on MiniCPM is low. Parameter efficient finetuning can be conducted with a single 1080/2080 GPU and full parameter finetuning can be conducted with a 3090/4090 GPU.

Source Model Evaluation

Note: This table showed source model instead of quantized model evaluation. Source Model Evaluation refer to MiniCPM-1B-sft Evaluation Results

• Code Generation：Average pass@1 score of HumanEval(0-shot) and MBPP(3-shot).
• Commonsense Reasoning: Average 0-shot accuracy of PIQA, SIQA, HellaSwag, WinoGrande and COPA.
• Reading Comprehension: Average 0-shot accuracy of BoolQ, LAMBADA and TyDi-QA.
• Other Benchmarks: average performance of GSM8K(8-shot)、MMLU(5-shot)、BBH(3-shot) and AGI-Eval(0-shot).

Setting	Average Sparsity	Average Performance	Code Generation	Commonsense Reasoning	Reading Comprehension	GSM8K	MMLU	BBH	AGI-Eval
LLaMA2-7B	-	37.96	16.37	69.59	61.87	12.96	44.45	32.96	27.53
ReluLLaMA-7B	66.98	37.62	15.85	69.64	70.54	5.84	38.64	35.07	27.73
ProSparse-7B*	88.11	38.31	19.47	66.29	63.33	12.74	45.21	33.59	27.55
ProSparse-7B	89.32	38.46	19.42	66.27	63.50	12.13	45.48	34.99	27.46
LLaMA2-13B	-	44.06	20.19	72.58	71.55	22.21	54.69	37.89	29.33
ReluLLaMA-13B	71.56	42.74	20.19	70.44	73.29	18.50	50.58	37.97	28.22
ProSparse-13B*	87.97	45.07	29.03	69.75	67.54	25.40	54.78	40.20	28.76
ProSparse-13B	88.80	44.90	28.42	69.76	66.91	26.31	54.35	39.90	28.67
MiniCPM-1B	-	44.44	36.85	63.67	60.90	35.48	50.44	35.03	28.71
MiniCPM-S-1B*	86.25	44.72	41.38	64.55	60.69	34.72	49.36	34.04	28.27
MiniCPM-S-1B	87.89	44.72	42.04	64.37	60.73	34.57	49.51	34.08	27.77

Performance Reference

Please search model by model name in Model Farm

Inference & Model Conversion

Please search model by model name in Model Farm

License

• Source Model: OpenBMB-General-Model-License

• Deployable Model: OpenBMB-General-Model-License