Research Papers Institution
Company behind the open-source machine-learning hub of the same name, known for the Transformers library, the model and dataset Hub, and fully open releases like the SmolLM small-model family.
Small Language Models · Hugging Face
DistilBERT distills BERT during pretraining into a 6-layer model with 40% fewer parameters that runs about 60% faster and keeps roughly 97% of BERT-base's GLUE score.
Small Language Models · Hugging Face
SmolLM2 is a 1.7B model overtrained on ~11T tokens through four data stages. It scores 68.7 on HellaSwag and 19.4 on MMLU-Pro, beating Llama3.2-1B — and ships every dataset, not just the weights.