GPT Can Solve Mathematical Problems Without a Calculator

Published on Sep 5

Authors:Zhen Yang,Ming Ding,Qingsong Lv,Zhihuan Jiang,Zehai He,Yuyi Guo,Jinfeng Bai,Jie Tang

Abstract

Previous studies have typically assumed that large language models are unable to accurately perform arithmetic operations, particularly multiplication of >8 digits, and operations involving decimals and fractions, without the use of calculator tools. This paper aims to challenge this misconception. With sufficient training data, a 2 billion-parameter language model can accurately perform multi-digit arithmetic operations with almost 100% accuracy without data leakage, significantly surpassing GPT-4 (whose multi-digit multiplication accuracy is only 4.3%). We also demonstrate that our MathGLM, fine-tuned from GLM-10B on a dataset with additional multi-step arithmetic operations and math problems described in text, achieves similar performance to GPT-4 on a 5,000-samples Chinese math problem test set.

View arXiv pageView PDF

Commentary

Based on the information provided, let's evaluate the potential impact of this paper:

1. Challenging Established Assumptions: The paper directly challenges a prevailing notion that large language models (LLMs) cannot perform arithmetic operations, especially complex ones, without calculator tools. Revising this assumption could influence how future LLMs are developed and applied.

2. Accuracy: Achieving almost 100% accuracy in multi-digit arithmetic operations without data leakage is a significant advancement. This level of accuracy means that for tasks requiring arithmetic computations, such an LLM could be directly employed without an external calculator.

3. Comparison with Previous Models: Demonstrating that a 2 billion-parameter model significantly surpasses GPT-4 (with only 4.3% accuracy in multi-digit multiplication) is a key contribution. This shows that it's not just about the size but also the quality and nature of the training data.

4. Fine-tuning on Math Problems: Their model, MathGLM, when fine-tuned, achieves performance comparable to GPT-4 on a Chinese math problem test set. This suggests potential for global applications, considering they've demonstrated its efficacy on a non-English dataset.

5. Potential Real-world Applications: The model's capability can be beneficial for applications like tutoring, where step-by-step arithmetic problem solving is needed. It might also find uses in industries requiring quick arithmetic checks or where integrating a calculator tool is cumbersome.

6. Scope of Research: While the research demonstrates the model's arithmetic prowess, its application might be limited if it only excels at arithmetic operations. For broad real-world impact, LLMs often need to be versatile across various tasks.

Considering the above factors:

I'd rate the real-world impact of this paper as a 7 out of 10.

While the paper does showcase a commendable achievement in the realm of arithmetic computations by LLMs, the broader applications beyond mathematical computations would determine its widespread impact. The paper can be seen as a significant step towards enhancing the capabilities of LLMs in arithmetic domains.