MiniLLM

community

https://github.com/microsoft/LMOps/tree/main/minillm

t1101675

AI & ML interests

Training efficient language models (MiniLLM, MiniPLM)

Organization Card

Community About org cards

Training Small Language Models with Knowledge Distillation

Official pre-trained models and baselines in

MiniLLM: Knowledge distillation of LLMs during instruction tuning.
MiniPLM: Knowledge distillation of LLMs during pre-training.

Collections 2

models 50

MiniLLM/MiniLLM-gpt2-340M

Text Generation • Updated Apr 11 • 859 • 4

MiniLLM/SFT-gpt2-120M

Text Generation • 0.1B • Updated Mar 25 • 16

MiniLLM/SFT-gpt2-760M

Text Generation • 0.8B • Updated Mar 25 • 12

MiniLLM/MiniPLM-Qwen-500M

Text Generation • 0.5B • Updated Mar 25 • 21 • 7

MiniLLM/MiniPLM-llama3.1-212M

Text Generation • 0.2B • Updated Mar 25 • 16 • 6

MiniLLM/MiniPLM-Mamba-130M

Text Generation • 0.1B • Updated Mar 25 • 50 • 3

MiniLLM/MiniPLM-Qwen-1.2B

Text Generation • 1B • Updated Mar 25 • 21 • 4

MiniLLM/Ref-Pretrain-Qwen-104M

Text Generation • 0.1B • Updated Mar 25 • 13 • 2

MiniLLM/Pretrain-Qwen-1.2B

Text Generation • 1B • Updated Mar 25 • 22

MiniLLM/Pretrain-Qwen-500M

Text Generation • 0.5B • Updated Mar 25 • 20

datasets 10

MiniLLM/pile-diff_samp-qwen_1.8B-qwen_104M-r0.5

Updated Mar 25 • 482

MiniLLM/pile-tokenized

Updated Nov 14, 2024 • 260 • 2

MiniLLM/roberta-corpus-processed

Updated Oct 22, 2024 • 84

MiniLLM/openwebtext-processed

Updated Sep 27, 2024 • 69

MiniLLM/dolly-processed

Viewer • Updated Sep 26, 2024 • 110k • 276 • 1

MiniLLM/sinst

Viewer • Updated Sep 26, 2024 • 8.35k • 115 • 1

MiniLLM/uinst

Viewer • Updated Sep 26, 2024 • 64.8k • 153 • 1

MiniLLM/self-inst

Viewer • Updated Sep 26, 2024 • 242 • 74 • 2

MiniLLM/Vicuna

Viewer • Updated Sep 26, 2024 • 80 • 66 • 1

MiniLLM/dolly

Viewer • Updated Sep 26, 2024 • 500 • 115