GlossiaSLM Leaderboard Discord

Small language models

Zero-shot · ~350M parameters and below

Best by benchmark

Leaderboard

Model benchmark results and weighted index.

Best <100M Highest index among models below that size · ✦ Best measured result for a benchmark

About

This leaderboard compares language models with roughly 350 million parameters or fewer. We run evaluations with lm-eval, without examples in the prompt (zero-shot).

Reading the results

The table shows measured results. The index combines them into one score from 0 to 100 using the weights you choose. Each benchmark has its own scale, described below. Open data indicates published training datasets; open mixes indicates the data mix and splits can be reproduced; open code indicates an open training codebase. Partial means some material or details are public, but the full release is incomplete. Unknown means insufficient evidence. Openness does not affect the index.

“Best by benchmark” shows the highest accuracy or lowest bits per byte (BPB). These winners do not depend on the weights.

If a result is missing, its weight is shared among the remaining benchmarks. That model’s index is therefore based on a different mix of results.

How the index is calculated

Evaluation setup

The current results were collected with lm-eval 0.4.13. LAMBADA scores the final word using the preceding passage as context. MMLU-redux scores each answer’s text and chooses the answer with the highest log-likelihood per character. The BPB beneath its accuracy is the question-weighted mean of the subjects’ correct-answer continuation BPBs; it does not affect the index.

Task definitions and run instructions · Leaderboard data

Rules

  1. No cheating.

    No training on test or validation splits of any of the benchmarks on this leaderboard is allowed. If we catch wind of it, we will remove all models from the leaderboard and ban you from further participation. For the purposes of this leaderboard, it also includes finetuning on training splits of any datasets here with training splits available. (My justification for this is that this is supposed to be a leaderboard for abilities garnered from pretraining, not finetuning; both serve as reasonable proxies for the quality of a base model however)

  2. No vibe-submitting.

    Your model must be documented and submitted by the human responsible for its training, *not* an AI agent or anything otherwise automated or not authored by you. Having an LLM-trained model is fine by itself, but you should be knowledgeable enough about it to write about it yourself.

  3. The HF community tab is not a forum.

    Don't have extended conversations under model request/model PR threads, especially about drama.

Submission guide

Run these evaluations on your model, then share the results and model details for review. Read the rules before submitting.

1. Install lm-eval

The current leaderboard runs use lm-eval 0.4.13. Create a Python environment and install the Hugging Face backend:

python -m venv .venv
source .venv/bin/activate
python -m pip install "lm_eval[hf]==0.4.13"

On Windows, activate the environment with .venv\Scripts\activate. Install a PyTorch build appropriate for your hardware if needed.

2. Get the tasks

Download the leaderboard Space repository using the Hugging Face CLI, then enter the downloaded directory. The custom LAMBADA, GoldenSwag, and MMLU-redux definitions are in tasks/; the built-in ARC tasks come with lm-eval.

hf download glossia/SLM-leaderboard --repo-type space --local-dir slm-leaderboard
cd slm-leaderboard

The hf command is provided by huggingface_hub, installed with the backend above.

3. Run the benchmarks

Replace OWNER/MODEL with your Hugging Face model ID or a local model path. This example uses a CUDA GPU:

lm_eval --model hf \
  --model_args pretrained=OWNER/MODEL,add_bos_token=False \
  --tasks arc_easy,arc_challenge,lambada_bpb,goldenswag,mmlu_redux_mcqa,mmlu_redux_continuation \
  --include_path ./tasks \
  --num_fewshot 0 \
  --device cuda:0 \
  --batch_size 32 \
  --output_path ./results/my-model

Use --device cpu or --device mps if appropriate. Reduce the batch size if you run out of memory; it does not change the intended evaluation. Do not use --limit for submitted results. If your model requires custom Hugging Face code, add ,trust_remote_code=True to --model_args.

4. Update the leaderboard data

Your results will be saved under results/my-model/. Add your model to data.js by editing it or using the importer below. Replace the example path with your result file:

node scripts/import-results.js --name "Model name" --family "Model family" --license "License" results/my-model/OWNER__MODEL/results_TIMESTAMP.json

The importer requires Node.js and reads the model ID and parameter count from the output when available. See the importer instructions for overrides and runs split across files. Check the resulting entry and keep its source path pointed at your results file.

5. Open a pull request

Open a pull request to the leaderboard Space with your data.js change and result JSON. Tell us a little about your model and mention anything unusual about the run.

You can create the PR from the downloaded directory with the Hugging Face CLI. Sign in with hf auth login, then run:

hf upload glossia/SLM-leaderboard . . --repo-type space --create-pr \
  --include "data.js" --include "results/my-model/**" \
  --commit-message "Add model results"

Replace my-model with the directory you used for your results. That’s it!

For reference, the leaderboard uses acc_norm for both ARC tasks, GoldenSwag, and MMLU-redux, and bits_per_byte for LAMBADA. MMLU-redux continuation BPB is supplementary. Report the original output; there is no need to calculate the weighted index yourself.