Apex-2
Our first mixture-of-experts model, trained from scratch and fine-tuned for chat.
- Repository
- YOON1v/Apex-2
- Parameters
- 3.87B total · 1.45B active
- Architecture
- MoE, 16 experts, top-4 · 32 layers
- Training
- 86.5B pretraining tokens · 2-stage SFT
- Context
- 4,096 tokens
- Runs with
- transformers · vLLM
We’re introducing Apex-2, our first mixture-of-experts model. It has 3.87B parameters, 1.45B of them active for each token, and was pretrained from scratch on 86.5B tokens before being fine-tuned for chat.
Apex-2 is a separate line from Apex-1, our 1.1B dense model, and no external pretrained weights were used at any stage. It maps onto the Qwen3-MoE architecture, so it loads directly with transformers and vLLM.
Here is what Apex-2 brings:
Performance. Supervised fine-tuning lifted code, math and instruction following: HumanEval 43.9, MBPP 56.3, GSM8K 32.4 and IFEval 44.7. Commonsense stayed at the pretraining level.
Token efficiency. With 1/23 to 1/400 of the pretraining data of comparable models, the pretrained model already matches Qwen2.5-1.5B, trained on 18T tokens, on HumanEval+ (32.9). The largest gap is math, which tracks pretraining tokens.
Training. Pretraining started on one GH200 and moved to two GH200s through DiLoCo, ending with a linear learning-rate decay. Two SFT stages followed on one H100.
What we dropped. We also tried DPO. Answers grew 2.3× longer, and code, math and instruction following all fell, so the release is the SFT model.
Prefer to do your own post-training? Apex-2-Base is the pretrained model before SFT.
On code, Apex-2 is ahead of Gemma-3-1B, trained on 23 times more data, and behind models trained on 60 to 200 times more. On math the gap is wide, and we report it as it is.
| Apex-23.87B · 1.45B active | Qwen2.5-1.5B-Instruct1.5B | Gemma-3-1B-it1B | Llama-3.2-1B-Instruct1B | OLMoE-1B-7B7B · 1.3B active | |
|---|---|---|---|---|---|
| Pretraining dataTokens | 0.087T | 18T | 2T | 9T | 5.1T |
| CodeHumanEval | 43.9 | 61.6 | 41.5 | — | 62.3 |
| CodeHumanEval+ | 41.5 | — | — | — | 54.4 |
| CodeMBPP | 56.3 | 63.2 | 35.2 | — | — |
| CodeMBPP+ | 48.9 | — | — | — | — |
| MathGSM8K | 32.40-shot CoT | 73.2 | 62.8 | 44.4 | 72.4 |
| Instruction followingIFEval | 44.7prompt strict | 42.5prompt strict | 80.2other metric | 59.5other metric | 66.4other metric |
Apex-2 was measured by us (vLLM greedy, chat template; EvalPlus for code, lm-eval for IFEval). Other numbers come from official model cards and technical reports, and shots, CoT and IFEval metrics differ between models, so treat this as a rough comparison. Sources: Qwen2.5 report, Gemma 3 report, Llama 3.2, OLMoE.
All results by training stage
| Apex-2SFT · released | Apex-2-Basebefore SFT | DPOdropped | |
|---|---|---|---|
| CodeHumanEval | 43.9 | 36.6 | 36.0 |
| CodeHumanEval+ | 41.5 | 32.9 | 32.3 |
| CodeMBPP | 56.3 | 54.8 | 45.2 |
| CodeMBPP+ | 48.9 | 46.3 | 37.3 |
| CodeMultiPL-E HumanEval C++ | 36.0 | — | 8.1 |
| CodeMultiPL-E MBPP C++ | 41.6 | — | 3.8 |
| CodeLiveCodeBench v5–v6 | 3.2 | — | 3.5 |
| CodeCRUXEval-O | 8.5 | — | 12.1 |
| CodeCRUXEval-I | 3.2 | — | 8.9 |
| MathGSM8K | 32.40-shot CoT | 15.18-shot | 14.3 |
| MathMATH-500 | 21.0 | — | 14.8 |
| Instruction followingIFEval prompt strict | 44.7 | — | 35.7 |
| Instruction followingIFEval instruction strict | 56.6 | — | 49.2 |
| CommonsenseHellaSwag | 62.2 | 60.8 | 63.1 |
| CommonsenseARC-e | 64.1 | 66.7 | 64.6 |
| CommonsenseARC-c | 38.4 | 39.7 | 39.2 |
| CommonsensePIQA | 74.8 | 73.9 | 74.4 |
| CommonsenseWinoGrande | 61.8 | 60.1 | 63.3 |
| CommonsenseLAMBADA | 52.8 | 54.5 | 51.2 |
The Apex-2-Base GSM8K score is 8-shot and the SFT score 0-shot CoT, so the two are not the same protocol. Code runs in a sandbox with no network; every scorer was checked with gold answers first.
Quick start
from transformers import AutoModelForCausalLM, AutoTokenizer model_id = "YOON1v/Apex-2" tok = AutoTokenizer.from_pretrained(model_id) model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype="auto", device_map="auto") msgs = [{"role": "user", "content": "Write a Python function that reverses a string."}] ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device) out = model.generate(ids, max_new_tokens=256) print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))
Before you download
Apex-2 or Apex-2-Base?
Use Apex-2 for chat, code and instruction following. Use Apex-2-Base, the pretrained model before SFT, if you want to run your own post-training.
What are its limits?
It is English-centric; Korean and other languages barely work. Its knowledge is limited, so factual errors are common. It is weak at competitive programming and code-execution reasoning, and its context is 4,096 tokens.
Is it free to use?
The weights are free to download from Hugging Face. The license and intended use are on the model card; please read it before deploying.