flâneur

Marketplace: my first attempt at training without backprop on GPU efficiently – Fang-Pen's coding note

fangpenlin.com · 8,234 words · saved by 1 readers

If you’ve read my previous articles, you know I’m a big fan of first-principles thinking. I’ve mentioned many times that I want to eliminate backpropagation. Many people think I’m crazy and assume I must be joking. But no, I’m serious. I thought about the problem from time to time. Recently, I came up with an idea that could potentially work. I spent two weeks implementing it and running experiments, and it worked! While this is just a baby step, there are still many things to improve, but at least I think it’s an interesting idea that could be worth exploring and sharing. Today, I would like to share my approach to training without backpropagation on GPUs efficiently. A diagram shows the validation accuracy of a small MNIST CNN model training process without using backpropagation. A diagram shows the loss of a small MNIST CNN model training process without using backpropagation. Just because a solution exists and is widely used doesn’t mean it’s the best one. From this perspective, we

Update: Please read the second article for the details of the V2 algorithm. Also the third article, Continual Learning with Marketplace: Model Learns New Data with Mostly Inference , introduces the continual learning with the Marketplace algorithm. If you’ve read my previous articles, you know I’m a big fan of first-principles thinking. I’ve mentioned many times that I want to eliminate backpropagation. Many people think I’m crazy and assume I must be joking. But no, I’m serious. I thought about the problem from time to time. Recently, I came up with an idea that could potentially work. I spen

saved by

related reading