Mathematics serves as a fundamental field for evaluating LLM reasoning intelligence. However, the in-depth relationship between “language” and “reasoning” remains underexplored. Current popular multilingual mathematical datasets are too simple to evaluate the reasoning capabilities of advanced LLMs effectively. This suggests that multilingual mathematical reasoning benchmarks have not kept pace with the progress in LLM reasoning abilities, as most advanced and challenging benchmarks are available only in English. Therefore, creating a challenging multilingual mathematical reasoning benchmark is crucial for studying multilingual reasoning abilities in the current era of LLMs.

To bridge this gap, we build PolyMath, a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels, with 9,000 samples in total. Our benchmark ensures difficulty comprehensiveness, language diversity, and high-quality translation, making it a highly discriminative multilingual mathematical benchmark in the era of reasoning LLMs.

With PolyMath, we conduct extensive experiments on advanced non-reasoning and reasoning LLMs, finding that even Qwen-3-235B-A22B-Thinking and Gemini-2.5-pro-preview achieve benchmark scores of only 54.6 and 52.2. Crucially, reasoning performance varies widely across languages, with differences of up to 10 points even at low accuracy settings. Beyond performance, we further explore the consistency between input and output languages, the variation in thinking length across different languages, and the impact of controlling the model's response language on performance. These offer insights into the slow-thinking pattern in multilingual contexts. We hope PolyMath serves as a strong benchmark to advance the research of multilingual mathematical reasoning.

We design Difficulty-Weighted Accuracy (DW-ACC) for the benchmark score of PolyMath.

PolyMath includes four difficulty levels: low, medium, high, and top. To effectively distinguish the contribution of correctly answered problems at each level to the final score, we set difficulty weights. Specifically, let the accuracies at the four levels be denoted as \( \{a_i\}_{i=1}^4 \). We define the weights as \( w_1 = 1 \), \( w_i = 2w_{i-1} \) for \( i = 2, 3, 4 \). The benchmark score is then given by: $$\text{DW-ACC} = \frac{\sum_{i=1}^4 w_i a_i}{\sum_{i=1}^4 w_i} = \sum_{i=1}^4 \left( \frac{2^{i-1}}{15}a_i \right).$$

PolyMath is designed with the following key principles:

• Broad Difficulty Range: We define and partition difficulty levels in the mathematical field through two key dimensions: Thought Depth and Knowledge Breadth. Thought Depth corresponds to human IQ, while Knowledge Breadth represents the extent of a person’s mathematical knowledge.
• Language Diversity: Each problem in PolyMath is available in 18 parallel language versions, encompassing over 75% of the world’s native speakers and major language families, ensuring diversity across both high-resource and low-resource languages.
• High-Quality Translations: Each translation is calibrated by language experts, avoiding direct use of LLM-generated outputs and ensuring precise term and logical clarity.

You can download the dataset on Hugging Face Dataset.