<rt id="kyeiu"></rt>

萬字長文解讀Scaling Law的一切，洞見LLM的未來

AIGC動態3個月前發布機器之心

452 0 0

LLM 還將繼續 scaling，但可能會是一種范式

原標題：萬字長文解讀Scaling Law的一切，洞見LLM的未來
文章來源：機器之心
內容字數：35098字

LLM Scaling Laws: Hitting a Wall?

This article explores the current state of Large Language Model (LLM) scaling,a cornerstone of recent AI advancements. While scaling—training larger models on more data—has driven progress,questions arise about its future viability. The article delves into scaling laws,their practical applications,and the factors potentially hindering further scaling.

1. Understanding Scaling Laws

LLM scaling laws describe the relationship between a model’s performance (e.g.,test loss) and factors like model size,dataset size,and training compute. This relationship often follows a power law,meaning a change in one factor leads to a predictable,relative change in performance. Early research demonstrated consistent performance improvements with increased scale across several orders of magnitude. However,this improvement is not exponential; it’s more akin to exponential decay,making further gains increasingly challenging.

2. The Pre-Training Era and GPT Models

The GPT series exemplifies scaling’s impact. From GPT’s 117M parameters to GPT-3’s 175B,scaling consistently improved performance. GPT-3’s success,achieved through in-context learning (few-shot learning),highlighted the potential of massive pre-training. Subsequent models like InstructGPT and GPT-4 incorporated further techniques beyond scaling,like reinforcement learning from human feedback (RLHF),to enhance model quality and alignment.

3. Chinchilla and Compute-Optimal Scaling

Research on Chinchilla challenged the initial scaling laws,emphasizing the importance of balancing model size and dataset size. Chinchilla,a 70B parameter model trained on a significantly larger dataset than previous models,demonstrated superior performance despite being smaller. This highlighted the potential for “compute-optimal” scaling,where both model and data size are scaled proportionally.

4. The Slowdown and its Interpretations

Recent reports suggest a slowdown in LLM improvements. This slowdown is complex and multifaceted. While technically scaling might still work,the rate of user-perceived progress is slowing. This is partly due to the inherent nature of scaling laws,which naturally flatten over time. The challenge is defining “improvement”—lower test loss doesn’t automatically translate to better performance on all tasks or user expectations.

5. Data Limitations and Future Directions

A significant obstacle is the potential “data death”—the scarcity of new,high-quality data sources for pre-training. This has led to explorations of alternative approaches: synthetic data generation,improved data curation techniques (like curriculum learning and continued pre-training),and refining scaling laws to focus on more meaningful downstream performance metrics.

6. Beyond Pre-training: Reasoning Models and LLM Systems

The limitations of solely relying on pre-training have pushed research towards enhancing LLM reasoning capabilities and building more complex LLM systems. Techniques like chain-of-thought prompting and models like OpenAI‘s o1 and o3 demonstrate significant progress in complex reasoning tasks. These models highlight a new scaling paradigm—scaling the compute dedicated to reasoning during both training and inference,yielding impressive results.

7. Conclusion: Scaling Continues,but in New Ways

While scaling pre-training might face limitations,the fundamental concept of scaling remains crucial. The focus is shifting towards scaling different aspects of LLM development: constructing robust LLM systems,improving reasoning abilities,and exploring new scaling paradigms beyond simply increasing model and data size during pre-training. The question isn’t *if* scaling will continue,but rather *what* we will scale next.