flâneur

Using Multiprocessing to speed up web scraping - David Ten

xang1234.github.io · 906 words · saved by 1 readers

Webscraping or calling APIs to collect data can take significant amounts of time due to the quantities of data involved. We previously saw how we could build a simple web scraper to collect data from Finviz. Depending on the number of tickers that we want to scan, this can take a significant amount of time.

Introduction Webscraping or calling APIs to collect data can take significant amounts of time due to the quantities of data involved. We previously saw how we could build a simple web scraper to collect data from Finviz . Depending on the number of tickers that we want to scan, this can take a significant amount of time. Multiprocessing One way to speed up the web scraping or API calls is to run multiple processes in parallel. The caveat is that the website that we are scraping has to allow us to query it frequently as we will be sending multiple requests from the same IP address. For API call

related reading