Benchmarks provide the foundation for comparisons across models and providers. This page explains how data enters the platform and how we maintain consistency.
Data sources
- Official releases — Provider announcements, technical blogs, and press materials.
- Academic papers — Peer-reviewed publications and arXiv preprints with reproducible results.
- Community submissions — Verified contributions from the Phaseo community and partners.
Every score must include a public citation or reproducible evaluation before it is accepted.
Normalisation
We standardise benchmark names, metrics, and splits to make cross-model comparisons accurate:
- Map each submission to an internal benchmark ID.
- Convert reported metrics to a canonical scale (for example percentages).
- Note any deviations, such as few-shot variants or custom evaluation harnesses.
- Run automated validation to ensure fields and ranges are sensible.
Update cadence
- Weekly for popular suites such as MMLU, GSM8K, and HumanEval.
- Monthly for niche or emerging benchmarks.
- On-demand when providers or community members submit high-quality evaluations.
Large updates trigger changelog entries so you can audit historical movements.
Transparency and reproducibility
Feedback loop
If you notice discrepancies or missing data:
- Open a GitHub issue with evidence or references.
- Submit a pull request using the contribution guides.
- Reach out in Discord to coordinate larger data imports.
We review every report and provide status updates as the benchmark is verified. Last modified on July 26, 2026
