Enhancing coding training boosts LLM efficiency on non-programming assignments
A groundbreaking study has shed light on the impact of programming code data on the performance of large language models (LLMs) in non-coding tasks. The research, outlined in the paper "To code or not to code," conducted ablation studies on decoder-only autoregressive Transformer models ranging from 470M to 2.8B parameters.
The study found that incorporating small amounts of high-quality code and mathematical data during model annealing boosts performance on key benchmarks that are not explicitly coding-related. This suggests that exposure to structured and logical content like code can enhance general reasoning abilities in language models.
Empirical evidence supports the claim that training on programming code alongside natural language data can enhance LLMs' overall performance, including on tasks unrelated to coding. Improvements were observed in reasoning and instruction-following benchmarks after code exposure in training.
One of the study's key findings is that including code data during pre-training improves AI language models' performance on non-coding tasks like answering questions and reasoning about topics. Interestingly, the researchers found that including code examples during AI training improves performance on non-coding tasks, which is surprising and not intuitive.
Incorporating code data during the final pre-training "cooldown" phase provided additional gains in performance. The "balanced→text" approach, which involves initializing from a model pre-trained on a balanced mix of code and text, then continuing pre-training primarily on text, led to the best overall performance across tasks.
The study also highlighted that improvements from fine-tuning strategies, including on code generation tasks, also led to better performance in commonsense reasoning, arithmetic reasoning, instruction following, and visual recognition—all non-coding domains. This implies that skills gained from code data can transfer and improve capabilities in broader tasks.
However, the study did not fully explain why code data helps with non-coding tasks, and further analysis of model internals could shed light on the transfer mechanisms. The researchers tested model initialization from models pre-trained on different mixtures of code and text data.
The findings of the study have implications for optimizing pre-training approaches to develop more capable and generalizable language models. Including 25% code data in pre-training achieved optimal performance on non-coding tasks. Using high-quality, carefully crafted code examples is especially helpful in improving the performance of AI language models on non-coding tasks.
In conclusion, the study sheds light on how code data contributes to models' reasoning and knowledge capabilities beyond just coding tasks, offering valuable insights for the future development of AI language models.
The study shows that integrating code data during pre-training and annealing significantly enhances large language model (LLM) performance on non-coding tasks, like reasoning and question answering, presumably due to the structured and logical nature of code data. Furthermore, the study suggests that the skills gained from code data can transfer and improve capabilities in broader tasks, such as commonsense reasoning, arithmetic reasoning, and visual recognition.
