AI Capabilities Can Be Significantly Improved Without Expensive Retraining | Epoch AI

epoch.ai · 922 words · saved by 1 readers

While scaling compute is key to improving LLMs, post-training enhancements can offer gains equivalent to 5-20x more compute at less than 1% of the cost.

AI capabilities can be significantly improved without expensive retraining | Epoch AI The massive computation used to train LLMs and similar foundation models has been one of the main drivers of AI progress in recent years, which has led to the recognition of the “Bitter Lesson”: that general methods that better leverage computational power are ultimately the most effective ( Sutton, 2019 ). The cost of training frontier models has now become so high that only a handful of actors can afford it ( Epoch AI, 2023 ). Our study explores methods of improving performance after training that don’t rel

AI Capabilities Can Be Significantly Improved Without Expensive Retraining | Epoch AI

related reading