Tang Jie, founder of Zhi Spectrum: Big model scaling is not just heap parameters; the core of future competition will shift to post-training and reasoning capabilities
Comparing news, Tang Jie, founder of Smart Spectrum, published a thought article on the big model Scaling Law on the X platform, saying that currently, the artificial intelligence industry's understanding of improving model capabilities is shifting from “expanding the scale of parameters” to multi-dimensional expansion. The number of parameters is not the only indicator for measuring the model's ability; it also requires a comprehensive evaluation based on data scale, computational resource allocation methods, and actual model operation scenarios.
Tang Jie pointed out that early research had driven the industry to rapidly expand the scale of model parameters. Kaplan et al.'s research in 2020 suggests that the growth rate of model parameters should be higher than the data growth rate, driving the development of large-scale models such as GPT-3, Gopher, and MT-NLG. However, by analyzing hundreds of models in 2022, Hoffmann and others discovered that the optimal calculation solution is closer to “about 20 training tokens for each parameter,” and that model parameters and data scale should continue to grow at the same time. In the past, the path of pursuing a trillion parameter model was actually a “yaw” experienced by the industry. As model application scenarios change, inference costs have gradually become an important part of life cycle costs, and optimization goals have also shifted from simply reducing training costs to improving long-term operation efficiency. Therefore, the “small model+fuller training” route has begun to receive attention.
Tang Jie said that the sparse hybrid expert (MoE) architecture has further changed the scaling logic. In MoE models, the total number of parameters determines how much knowledge the model can store, and activation parameters and effective depth affect the model's ability to complete complex inference tasks. For tasks that require a long chain of reasoning, such as finding bugs, the ability does not come from simply memorizing more information, but rather requires models to maintain the continuity of the multi-step inference process. Recent research shows that there is no uniform answer for the optimal “token/parameter ratio”: memory-oriented tasks require more parameters, while biased reasoning tasks rely more on data and computational depth. At a fixed scale of training data, blindly increasing total parameters may even weaken reasoning ability, while increasing the number of active experts is more helpful in improving model performance.
Regarding the latest development of the smart spectrum model, Tang Jie revealed that GLM-5.3 is an experiment in the direction of scaling. This model uses the same basic model, architecture, and total parameters and activation parameter scales as GLM-5.2, but post-training optimization is performed through a month of large-scale long-term environmental training and reinforcement learning (RL). The performance improvement does not come from an increase in parameters, but from an expansion in the post-training phase. He concluded that competition for large models has moved from simply competing for parameter sizes to the stage of exploring “multi-dimensional scaling”. Future model capability improvements will rely more on continuous optimization of training strategies, inference depth, and post-training capabilities.
