Large language models (LLMs) have proven to be adept at handling simple questions but struggle with more complex tasks that require higher-level reasoning and planning. This limitation has led researchers to explore various prompting techniques, often referred to as “System 2” techniques, to enhance the reasoning capabilities of LLMs. These techniques involve prompting the model to generate intermediate steps toward problem-solving, mimicking the deliberate and analytical thinking of System 2.
In cognitive science, System 1 and System 2 refer to two distinct modes of thinking in humans. System 1 is fast, intuitive, and automatic, while System 2 is slow, analytical, and deliberate. LLMs are typically likened to System 1 thinking due to their ability to generate text rapidly. However, they often struggle with tasks that require conscious reasoning and planning, akin to System 2 thinking.
Inspired by the concept of ingraining System 2 thinking into System 1 through repeated task performance in humans, researchers at Meta FAIR developed “System 2 distillation” for LLMs. This technique aimed to distill the knowledge gained from System 2 reasoning capabilities into the fast-paced and computationally efficient System 1 generation of LLMs.
Unlike traditional distillation techniques that involve a separate teacher model, System 2 distillation leverages the model’s own System 2 reasoning capabilities. The process begins by prompting the LLM to solve a problem using System 2 techniques, followed by verifying the correctness of responses through an unsupervised mechanism. Correct responses are selected based on factors like self-consistency and discarded if inconsistent. The model is then fine-tuned on the initial question and answer, enabling it to skip reasoning steps and directly provide answers.
The researchers evaluated the effectiveness of System 2 distillation on a variety of reasoning tasks using different System 2 prompting techniques. Results indicated a significant improvement in LLM performance on complex tasks, rivaling or surpassing the accuracy of original System 2 methods. Moreover, distilled models exhibited faster response generation and reduced computational requirements by eliminating intermediate reasoning steps.
While System 2 distillation showed promise in enhancing LLM performance, researchers encountered challenges in distilling certain types of reasoning skills, such as complex math tasks requiring Chain-of-Thought prompting. Further research is needed to explore the applicability of distillation on smaller models and its impact on broader task performance. Additionally, the susceptibility of LLM benchmarks to contamination and the potential for distillation to optimize mature LLM pipelines warrant further investigation.
The evolution of System 2 distillation in large language models represents a significant advancement in enhancing their reasoning capabilities. By distilling System 2 knowledge into System 1 generation, LLMs can tackle complex tasks more effectively while streamlining the inference process. As research in this field progresses, the potential for distillation to optimize LLM pipelines and facilitate improved task reasoning holds promise for the future of artificial intelligence.
Leave a Reply