Model

Qwen1.5-MoE-A2.7B

OPEN

Alibaba · Qwen1.5 family · released Mar 28, 2024

First Qwen MoE model: 14.3B total/2.7B active params match dense Qwen1.5-7B quality at ~25% of its training cost and 1.74x inference throughput.

ReasoningCodingVisionFunction callingTool useAgentic

94.1

Elo · rank #386

Parameters

14.3B

Active params

2.7B (MoE)

Context

32K tokens

Architecture

Fine-grained MoE, decoder-only, upcycled from Qwen-1.8B: 64 experts (4 always-on shared + 60 routed, 4 routed activated per token)

License

Tongyi Qianwen LICENSE AGREEMENT (custom, free commercial use below usage threshold)

Languages

—

API price (in/out)

No hosted API

Modalities

text

Benchmark results

Bar shows position within the tracked field; marker = field best

ARC-ChallengeReasoning44.0%#140

best: Llama 3.1 405B · 96.9%

ARC-EasyReasoning69.5%#50

best: Phi-3-medium (14B) · 97.7%

GSM8KMath61.5%#112

best: Llama 3.1 405B · 96.8%

HellaSwagReasoning77.3%#96

best: Claude 3 Opus · 95.4%

HumanEvalCoding34.2%#143

best: Claude Opus 4.5 · 99.4%

LAMBADAReasoning71.3%#15

best: GPT-3 175B · 86.4%

MMLUKnowledge62.5%#194

best: OpenAI o3 · 92.9%

MT-BenchHuman preference7.17#57

best: Hunyuan-Large (A52B) · 9.4

PIQAReasoning80.5%#38

best: GPT-4o mini · 93.1%

WinoGrandeReasoning69.3%#105

best: PaLM 2 · 90.9%

Run it locally

VRAM @ Q4

9 GB

VRAM @ FP16

29 GB

Fits on (Q4)

RTX 3060 12GBRTX 4070 Ti 16GBRTX 3090 24GBRTX 4090 24GBRTX 5090 32GBM4 Pro 48GBM3 Max 128GBM3 Ultra 512GBA100 80GBH100 80GBH200 141GBB200 192GB

VRAM sized on the 14.3B total weights (all experts must stay resident even though only 2.7B are active per token); at release only HF transformers and vLLM were supported, llama.cpp/GGUF support came later.

Quantizations

—

Qwen1.5 family

Elo progression across releases

Qwen1.5-72BFeb 2024545.1 Qwen1.5-MoE-A2.7BMar 202494.1 CodeQwen1.5-7BApr 2024526.5 Qwen1.5-110BApr 2024665.5

Resources

HFQwen/Qwen1.5-MoE-A2.7B DOCSqwenlm.github.io/blog/qwen-moe

API price weights · each benchmark row carries its own source badge (see methodology)