Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM

基本信息

• 📝 原文链接 : https://arxiv.org/abs/2502.06635
• 👥 作者 : Qingshui Gu, Shu Li, Tianyu Zheng, Zhaoxiang Zhang
• 🏷️ 关键词 : Soft Mixture of Experts, open-source, resource-efficient, Chinese-centric LLM, Flash Attention
• 📚 分类 : 自然语言处理, 机器学习

摘要

中文摘要

Steel-LLM 是一个以中文为中心的从零开始开发的语言模型，旨在在有限的计算资源下创建一个高质量、开源的模型。该项目于2024年3月启动，目标是基于大规模数据集训练一个拥有10亿参数的模型，优先考虑透明度和实用见解的共享，以帮助社区中的其他人。训练过程主要侧重于中文数据，包含一小部分英文数据，通过提供更详细和实用的模型构建历程来弥补现有开源LLM的不足。Steel-LLM在CEVAL和CMMLU等基准测试中表现出色，超越了来自更大机构的早期模型。本文对项目的关键贡献进行了全面总结，包括数据收集、模型设计、训练方法以及过程中遇到的挑战，为希望开发自己LLM的研究人员和从业者提供了宝贵的资源。模型检查点和训练脚本可在 https://github.com/zhanshijinwat/Steel-LLM 获取。

原文摘要

Steel-LLM is a Chinese-centric language model developed from scratch with the goal of creating a high-quality, open-source model despite limited computational resources. Launched in March 2024, the project aimed to train a 1-billion-parameter model on a large-scale dataset, prioritizing transparency and the sharing of practical insights to assist others in the community. The training process primarily focused on Chinese data, with a small proportion of English data included, addressing gaps in existing open-source LLMs by providing a more detailed and practical account of the model-building journey. Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. This paper provides a comprehensive summary of the project’s key contributions, including data collection, model design, training methodologies, and the challenges encountered along the way, offering a valuable resource for researchers and practitioners looking to develop their own LLMs. The model checkpoints and training script are available at https://github.com/zhanshijinwat/Steel-LLM.

论文解读

一句话总结

本文介绍了Steel-LLM，一个基于有限计算资源开发的、以中文为中心的开源语言模型，通过提供透明度和实用见解，帮助社区其他成员进行模型构建。

…

阅读全文 ：https://www.llamafactory.cn/daily-paper/detail/?id=1160