✳flâneur — a map of the web's best reading

What are Blocking Rules? - Splink

moj-analytical-services.github.io · 933 words · saved by 1 readers

The number of pairs of records to compare grows using the formula 𝑛 ( 𝑛 − 1 ) 2 , i.e. with (approximately) the square of the number of records, as shown in the following chart: For example, a dataset of 1 million input records would generate around 500 billion pairwise record comparisons. So, when datasets get bigger the amount of computational resource gets extremely large (and costly). In reality, we try and reduce the amount of computation required using blocking. Blocking is a technique for reducing the number of record pairs that are considered by a model. Considering a dataset of 1 million records, comparing each record against all of the other records in the dataset generates ~500 billion pairwise comparisons. However, we know the vast majority of these record comparisons won't be matches, so processing the full ~500 billion comparisons would be largely pointless (as well as costly and time-consuming). Instead, we can define a subset of potential comparisons using Blocking R

What are Blocking Rules? ¶ The primary driver of Splink's run time is the number of record pairs that the Splink model has to process. This is controlled by the blocking rules. This guide explains what blocking rules are, and how they can be used. Introduction ¶ One of the main challenges to overcome in record linkage is the scale of the problem. The number of pairs of records to compare grows using the formula \(\frac{n\left(n-1\right)}2\) , i.e. with (approximately) the square of the number of records, as shown in the following chart: For example, a dataset of 1 million input records would g

Explore this link on the map →

related reading