✳flâneur — a map of the web's best reading

AWS Lambda: Parallelizing Invocations with S3 Batch Operations | by David Larsen | Medium

lilneutron.medium.com · 610 words · saved by 1 readers

How to Run 1000s of Lambdas in Parallel

AWS Lambda AWS Aws S3 AWS Lambda: Parallelizing Invocations with S3 Batch Operations David Larsen 4 min read · Oct 20, 2020 -- Listen Share Press enter or click to view image in full size The Problem I have a simple Lambda function that downloads historical equity trades for a given symbol (“AAPL”) and date (“2020–01–01”) and saves the output as a parquet file to S3 for analysis but how can I parallelize the invocations when my input list is very large? If I use Boto3 and Python on my local machine, the downloads of symbol+date are sequentially invoked which results in O(N) performance. Search

Explore this link on the map →

related reading