flâneur

Creepy crawlies — Konstantin Ryabitsev

people.kernel.org · 1,460 words · saved by 1 readers

You've probably heard me complain about the "AI crawlers" before, but now I actually have some hard numbers I can put up to show their im...

You've probably heard me complain about the “AI crawlers” before, but now I actually have some hard numbers I can put up to show their impact. In a few words, it's bad enough to create a constant “background radiation” of system load, permanently tying up a chunk of capacity spent on producing output that is only useful for a single purpose — feeding a learning model. TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones. At any one time, across 5 geo-distributed nodes, there are 14 CPU cores doing nothing…

saved by

related reading