Benchmarks
Light
Benchmarks
MBPP
Coding
unit % · normalized over [10, 96]
Mostly Basic Python Problems, pass@1.
#
Model
Source
Score
Normalized
1
Llama-3.3-Nemotron-Super-49B v1 (Reasoning On)
OPEN
self-reported
91.3%
2
Granite 4.1 8B Instruct
OPEN
self-reported
87.3%
3
Claude 3 Opus
CLOSED
self-reported
86.4%
4
Llama-3.3-Nemotron-Super-49B v1 (Reasoning Off)
OPEN
self-reported
84.9%
5
Qwen2.5-72B
OPEN
self-reported
84.7%
6
Qwen2.5-32B
OPEN
self-reported
84.5%
7
Granite 4.0 H Small
OPEN
self-reported
84.0%
8
YandexGPT 5 Pro
CLOSED
independent
84.0%
9
Falcon-H1 34B
OPEN
self-reported
83.9%
10
Qwen2.5-Coder-32B
OPEN
self-reported
83.0%
11
LG EXAONE 3.5 32B Instruct
OPEN
self-reported
81.8%
12
Phi-3.5-MoE (16x3.8B, 6.6B active)
OPEN
self-reported
80.8%
13
Claude 3 Haiku
CLOSED
self-reported
80.4%
14
Codestral 25.01
CLOSED
self-reported
80.2%
15
Mistral Large 2 (24.07)
OPEN
self-reported
80.0%
16
Claude 3 Sonnet
CLOSED
self-reported
79.4%
17
Llama 3.1 405B
OPEN
self-reported
78.8%
18
Codestral 22B
OPEN
self-reported
78.2%
19
Skywork-o1-Open-Llama-3.1-8B
OPEN
self-reported
77.9%
20
Nemotron-H 56B
OPEN
self-reported
77.8%
21
Llama 4 Maverick
OPEN
self-reported
77.6%
22
Qwen2-72B
OPEN
self-reported
76.9%
23
Qwen2.5-14B
OPEN
self-reported
76.7%
24
Phi-3-medium (14B)
OPEN
self-reported
75.2%
25
Qwen2.5-7B
OPEN
self-reported
74.9%
26
Mistral Small 3.1 (24B)
OPEN
self-reported
74.7%
27
Inflection-2.5
CLOSED
self-reported
73.9%
28
WizardCoder-Python-34B V1.0
OPEN
self-reported
73.2%
29
Mistral Large
CLOSED
self-reported
73.1%
30
Gemma 3 12B
OPEN
self-reported
73.0%
31
Hunyuan-Large (A52B)
OPEN
self-reported
72.6%
32
CodeQwen1.5-7B
OPEN
self-reported
72.2%
33
DeepSeek-V2 236B
OPEN
self-reported
72.0%
34
Qwen2-57B-A14B
OPEN
self-reported
71.9%
35
Phi-3-small (7B)
OPEN
self-reported
71.7%
36
Aya Vision 32B
OPEN
self-reported
70.4%
37
Soofi S
OPEN
self-reported
70.2%
38
DeepSeek Coder 33B
OPEN
self-reported
70.0%
39
Ministral 8B
OPEN
self-reported
70.0%
40
Phi-3-mini (3.8B)
OPEN
self-reported
69.8%
41
Phi-3.5-mini (3.8B)
OPEN
self-reported
69.6%
42
Codestral Mamba (7B)
OPEN
self-reported
68.5%
43
Llama 4 Scout
OPEN
self-reported
67.8%
44
Ministral 3B
CLOSED
self-reported
67.7%
45
StarCoder2 15B
OPEN
curated
66.2%
46
Qwen2-7B
OPEN
self-reported
65.9%
47
InternLM2-20B
OPEN
self-reported
65.8%
48
Yi-1.5 34B
OPEN
self-reported
65.5%
49
Phi-4-mini (3.8B)
OPEN
self-reported
65.3%
50
Phi-4-mini-instruct
OPEN
self-reported
65.3%
51
Gemma 3 4B
OPEN
self-reported
63.2%
52
Gemma 2 27B
OPEN
self-reported
62.6%
53
Code Llama 70B
OPEN
self-reported
62.4%
54
Code Llama 70B Instruct
OPEN
self-reported
62.2%
55
Upstage Solar Pro
OPEN
self-reported
61.6%
56
DeepSeek LLM 67B
OPEN
self-reported
61.4%
57
Yi-1.5 9B
OPEN
self-reported
61.1%
58
Mixtral 8x7B
OPEN
self-reported
60.7%
59
Phi-2 (2.7B)
OPEN
self-reported
60.6%
60
Qwen1.5-110B
OPEN
self-reported
58.1%
61
StarCoder2 3B
OPEN
curated
57.4%
62
Code Llama 34B Python
OPEN
self-reported
56.2%
63
CodeGemma 7B
OPEN
self-reported
56.2%
64
Phi-1 (1.3B)
OPEN
self-reported
55.5%
65
Code Llama 34B
OPEN
self-reported
55.0%
66
Qwen1.5-72B
OPEN
self-reported
53.4%
67
Inflection-2
CLOSED
self-reported
53.0%
68
StarCoder (15.5B)
OPEN
self-reported
52.7%
69
ChatGLM3-6B
OPEN
self-reported
52.4%
70
Gemma 2 9B
OPEN
self-reported
52.4%
71
Qwen-72B
OPEN
self-reported
52.2%
72
Granite 3.0 8B Instruct
OPEN
self-reported
49.6%
73
Mistral 7B
OPEN
self-reported
47.5%
74
LG EXAONE 3.0 7.8B Instruct
OPEN
self-reported
47.4%
75
Poro 34B
OPEN
self-reported
47.4%
76
MiniCPM-2B
OPEN
self-reported
47.3%
77
Code Llama 13B
OPEN
self-reported
47.0%
78
Llama 2 70B
OPEN
self-reported
45.0%
79
Gemma 7B
OPEN
self-reported
44.4%
80
Inflection-1
CLOSED
self-reported
43.8%
81
Comma v0.1-2T
OPEN
self-reported
41.5%
82
Code Llama 7B
OPEN
self-reported
41.4%
83
Granite 3.0 2B Instruct
OPEN
self-reported
41.0%
84
Yi-34B
OPEN
self-reported
41.0%
85
Qwen-14B
OPEN
self-reported
40.8%
86
Nemotron-4 15B
OPEN
self-reported
40.6%
87
DeepSeekMoE 16B
OPEN
self-reported
39.2%
88
LLaMA 65B
OPEN
self-reported
37.7%
89
Phi-1.5 (1.3B)
OPEN
self-reported
37.7%
90
CodeGen 16B (Mono)
OPEN
self-reported
35.3%
91
DeepSeek-VL 7B
OPEN
self-reported
35.2%
92
SantaCoder 1.1B
OPEN
self-reported
35.0%
93
NAVER HyperCLOVA X
CLOSED
self-reported
33.4%
94
Llama-3.1-Minitron 4B
OPEN
self-reported
32.4%
95
Qwen-7B
OPEN
self-reported
31.6%
96
Llama 2 13B
OPEN
self-reported
30.6%
97
Baichuan2-13B
OPEN
self-reported
30.2%
98
LLaMA 33B
OPEN
self-reported
30.2%
99
Gemma 2B
OPEN
self-reported
29.2%
100
Gemini 1.0 Nano
CLOSED
self-reported
27.2%
101
Yi-6B
OPEN
self-reported
26.3%
102
Baichuan2-7B
OPEN
self-reported
24.2%
103
MPT-7B
OPEN
independent
22.8%
104
CodeGeeX (13B)
OPEN
self-reported
22.4%
105
LLaMA 13B
OPEN
self-reported
22.0%
106
Llama 2 7B
OPEN
self-reported
20.8%
107
LLaMA 7B
OPEN
self-reported
17.7%
108
LaMDA
CLOSED
self-reported
14.8%
109
Qwen-1.8B
OPEN
self-reported
14.2%
Score distribution
109 tracked results across the normalization window
10
96
Score vs. parameters
Open-weights models, log-x params
1B
10B
100B
1000B