✳flâneur — a map of the web's best reading

3. Blocking - Splink

moj-analytical-services.github.io · 1,457 words · saved by 1 readers

To link records, we need to compare pairs of records, and decide which pairs are matches and non matches. For most large datasets, it is computationally intractable to compare every row with every other row, since the number of comparisons rises quadratically with the number of records. Instead we rely on blocking rules, which specify which pairwise comparisons to generate. For example, we could generate the subset of pairwise comparisons where either first name or surname matches. This is part of a two step process to link data: Use blocking rules to generate candidate pairwise record comparisons Use a probabilistic linkage model to score these candidate pairs, to determine which ones should be linked Blocking rules are the most important determinant of the performance of your linkage job. When deciding on your blocking rules, you're trading off accuracy for performance: This tutorial clarifies what blocking rules are, and how to choose good rules. In Splink, blocking rules are specif

Choosing blocking rules to optimise runtime ¶ To link records, we need to compare pairs of records and decide which pairs are matches. For example consider the following two records: first_name surname dob city email Robert Allen 1971-05-24 nan roberta25@smith.net Rob Allen 1971-06-24 London roberta25@smith.net These can be represented as a pairwise comparison as follows: first_name_l first_name_r surname_l surname_r dob_l dob_r city_l city_r email_l email_r Robert Rob Allen Allen 1971-05-24 1971-06-24 nan London roberta25@smith.net roberta25@smith.net For most large datasets, it is computatio

Explore this link on the map →

related reading