Forget AGI—Top AI Models Still Struggle With Math

New benchmark tests reveal that even advanced AI models like GPT-4 Vision struggle to match human capabilities in multimodal mathematical reasoning, highlighting a significant gap in the pursuit of artificial general intelligence (AGI). The MATHVISTA benchmark, developed by researchers from Microsoft Research, Sahara AI, and Emory University, evaluates AI's ability to interpret visual information such as charts, graphs, and diagrams to solve complex math problems.

GPT-4 Vision Leads, But Lags Behind Humans

Among the 12 foundation models tested, GPT-4 Vision achieved the highest score with 49.9%. However, this performance is still considerably lower than the 60.3% average achieved by human participants. This disparity underscores the challenges AI faces in replicating the broad reasoning abilities associated with AGI, which aims for systems capable of performing diverse tasks like humans.

MATHVISTA Benchmark Addresses Limitations

The MATHVISTA benchmark was created to address limitations in existing evaluation datasets, where models often solved problems using text alone rather than true visual reasoning. The benchmark requires models to interpret visual data and perform multi-step mathematical and logical reasoning. The dataset, launched in October 2023, has seen significant adoption, with over 275,000 downloads, including more than 13,000 in the past month.

Developing the benchmark involved meticulous annotation, with over 6,000 multimodal examples created by trained annotators from Sahara AI. These examples cover various mathematical domains, including arithmetic, algebra, geometry, and statistics, with a focus on distinguishing deep reasoning from simpler tasks.

Challenges in AI Development and Evaluation

Researchers emphasize the difficulty in accurately measuring AI progress toward general intelligence. Issues like data contamination, where benchmark answers might be inadvertently included in training data, can lead to inflated scores that reflect memorization rather than genuine reasoning capabilities. This makes it challenging to ascertain if AI systems are truly improving.

Limitations in current training data, which largely consists of existing internet content, also pose a hurdle. Experts suggest the need for new methods to inject novel knowledge into AI models, potentially through high-quality, specialized data. Simulated environments where AI can learn through interaction and feedback are also being explored as a way to overcome the boundaries of existing internet data.

Despite AI's rapid content generation, human oversight remains crucial for evaluating AI performance. The distinct strengths and weaknesses of humans and AI can be leveraged to drive future improvements in AI systems.