Harness design matters for AI capabilities in mathematics.

To assess the ability of current AI systems to correctly solve research-level mathematics problems, we tested four AI systems on a set of ten problems in a broad range of mathematical fields; these problems arose naturally in the research process of the contributors. This document includes our methodology, and the results of our testing, including the assessment of human reviewers and a report of costs. We provide links to supplementary documents including the human solutions, the AI-generated s
To assess the ability of current AI systems to correctly solve research-level mathematics problems, we tested four AI systems on a set of ten problems in a broad range of mathematical fields; these problems arose naturally in the research process of the contributors. This document includes our methodology, and the results of our testing, including the assessment of human reviewers and a report of costs. We provide links to supplementary documents including the human solutions, the AI-generated solutions, and the referee reports and logs for the AI-generated solutions.




