Estimating the Performance of Entity Resolution Algorithms: Lessons Learned Through PatentsView.org

Published in

American Statistician(2022)

External Links:

Generate Graph DownloadPDF

Abstract

Abstract This article introduces a novel evaluation methodology for entity resolution algorithms. It is motivated by PatentsView.org, a public-use patent data exploration platform that disambiguates patent inventors using an entity resolution algorithm. We provide a data collection methodology and tailored performance estimators that account for sampling biases. Our approach is simple, practical, and principled—key characteristics that allow us to paint the first representative picture of PatentsView’s disambiguation performance. The results are used to inform PatentsView’s users of the reliability of the data and to allow the comparison of competing disambiguation algorithms.

Authors

Olivier Binette

3 papers

Sarvothaman Madhavan

2 papers

Christina Jones

3 papers

References59 items

PatentsView-Evaluation: Evaluation Datasets and Tools to Advance Research on Inventor Name Disambiguation

Bridging the Gap between Reality and Ideality of Entity Matching: A Revisiting and Benchmark Re-Construction

PatentsView: An Open Data Platform to Advance Science and Technology Policy

S2AND: A Benchmark and Evaluation System for Author Name Disambiguation

Disambiguation by Namesake Risk Assessment

Estimating the Performance of Entity Resolution Algorithms: Lessons Learned Through PatentsView.org

Published in

American Statistician(2022)

External Links:

Generate Graph DownloadPDF

Abstract

Authors

Olivier Binette

3 papers

Sarvothaman Madhavan

2 papers

Christina Jones

3 papers

References59 items

PatentsView-Evaluation: Evaluation Datasets and Tools to Advance Research on Inventor Name Disambiguation

Bridging the Gap between Reality and Ideality of Entity Matching: A Revisiting and Benchmark Re-Construction

PatentsView: An Open Data Platform to Advance Science and Technology Policy

S2AND: A Benchmark and Evaluation System for Author Name Disambiguation

Disambiguation by Namesake Risk Assessment

Sokhna A. York

1 papers

E. Hickerson

1 papers

Youngsoo Baek

2 papers

An Overview of End-to-End Entity Resolution for Big Data

(Almost) all of entity resolution

Data Linkage: The Big Picture

Scalable Hierarchical Clustering with Tree Grafting

Optimal Transport-based Alignment of Learned Character Representations for String Similarity

Scaling Bayesian Probabilistic Record Linkage with Post-Hoc Blocking: An Application to the California Great Registers

Disambiguating USPTO inventor names with semantic fingerprinting and DBSCAN clustering

The ethnic migrant inventor effect: Codification and recombination of knowledge across borders

Exploring the Effect of Household Structure in Historical Record Linkage of Early 1900s Ireland Census Records

Machine Learning and Natural Language Processing on the Patent Corpus: Data, Tools, and New Measures

Using a Probabilistic Model to Assist Merging of Large-Scale Administrative Records

How Well Do Automated Methods Perform in Historical Samples? Evidence from New Ground Truth

Semantic Author Name Disambiguation with Word Embeddings

A Mixture Record Linkage Approach for US Patent Inventor Disambiguation

A note on using the F-measure for evaluating record linkage algorithms

In Search of an Entity Resolution OASIS: Optimal Asymptotic Sequential Importance Sampling

Random Forest DBSCAN for USPTO Inventor Name Disambiguation

A Practioner's Guide to Evaluating Entity Resolution Results

Disambiguation and co-authorship networks of the U.S. patent inventor database (1975–2010)

Big data integration

A brief survey of automatic methods for author name disambiguation

Evaluating entity resolution results

Data quality and record linkage techniques

Email data cleaning

Model Assisted Survey Sampling

A method for calibrating false-match rates in record linkage

A Theory for Record Linkage

Automatic linkage of vital records.

A Generalization of Sampling Without Replacement from a Finite Universe

Bias of Precision Computed on Benchmark Datasets This section provides more information on example 1. For this example, we considered the RLdata10000 dataset from Sariyar and Borg (2022)

RecordLinkage: Record Linkage Functions for Linking and Deduplicating Data Sets . R package version 0.4-12

The Four Generations of Entity Resolution

“PatentsView: Disam-biguating Inventors, Assigness, and Locations,”

Descriptor : Disambiguation of patent inventors and assignees using high-resolution geolocation data

Learning String Alignments for Entity Aliases

Seeing the non-stars: (Some) sources of bias in past disambiguation approaches and a new public tool leveraging labeled records

Methods Matter: Rethinking Inventor Disambiguation with Classification & Labeled Inventor Records

On choosing thresholds for duplicate detection

Entity Resolution Evaluation Measures

Record Linkage Measures in an Entity Centric World

THE PINHAS SAPIR CENTER FOR DEVELOPMENT TEL AVIV UNIVERSITY Identification and Mobility of Israeli Patenting Inventors

“Matching Accuracy of the Lee-Kim-Marschke Computer Matching Program.”

Quality and Complexity Measures for Data Linkage and Deduplication

On Evaluation and Training-Set Construction for Duplicate Detection

Algorithms for Scoring Coreference Chains

Sampling techniques.

These are the precision and recall estimators corresponding to uniformly sampling records in lemma 1 ( p i ∝ 1) with the bias adjustment given in

It can easily be generalized to estimate other clustering metrics, such as cluster precision, cluster recall, cluster homogeneity and completeness, and other generalized merge distances

This is the “naive” precision (respectively recall) estimator obtained by computing precision (respectively recall) when only looking at records that appear in the sampled clusters

trivial precision estimate

200 records were sampled and the ground truth clusters associated with them were recovered

Field of Study

Computer ScienceMathematics

Journal Information

Name

The American Statistician

Page

370 - 380

Volume

Venue Information

Name

American Statistician

Type

journal

URL

http://openurl.ingenta.com/content?genre=journal&issn=0003-1305

Alternate Names

Am Stat
The American Statistician

Abstract

Authors

References59 items

PatentsView-Evaluation: Evaluation Datasets and Tools to Advance Research on Inventor Name Disambiguation

Bridging the Gap between Reality and Ideality of Entity Matching: A Revisiting and Benchmark Re-Construction

PatentsView: An Open Data Platform to Advance Science and Technology Policy

S2AND: A Benchmark and Evaluation System for Author Name Disambiguation

Disambiguation by Namesake Risk Assessment

Abstract

Authors

References59 items

PatentsView-Evaluation: Evaluation Datasets and Tools to Advance Research on Inventor Name Disambiguation

Bridging the Gap between Reality and Ideality of Entity Matching: A Revisiting and Benchmark Re-Construction

PatentsView: An Open Data Platform to Advance Science and Technology Policy

S2AND: A Benchmark and Evaluation System for Author Name Disambiguation

Disambiguation by Namesake Risk Assessment

An Overview of End-to-End Entity Resolution for Big Data

(Almost) all of entity resolution

Block

Data Linkage: The Big Picture

Scalable Hierarchical Clustering with Tree Grafting

Optimal Transport-based Alignment of Learned Character Representations for String Similarity

Scaling Bayesian Probabilistic Record Linkage with Post-Hoc Blocking: An Application to the California Great Registers

Disambiguating USPTO inventor names with semantic fingerprinting and DBSCAN clustering

The ethnic migrant inventor effect: Codification and recombination of knowledge across borders

Exploring the Effect of Household Structure in Historical Record Linkage of Early 1900s Ireland Census Records

Machine Learning and Natural Language Processing on the Patent Corpus: Data, Tools, and New Measures

Using a Probabilistic Model to Assist Merging of Large-Scale Administrative Records

How Well Do Automated Methods Perform in Historical Samples? Evidence from New Ground Truth

Semantic Author Name Disambiguation with Word Embeddings

A Mixture Record Linkage Approach for US Patent Inventor Disambiguation

A note on using the F-measure for evaluating record linkage algorithms

In Search of an Entity Resolution OASIS: Optimal Asymptotic Sequential Importance Sampling

Random Forest DBSCAN for USPTO Inventor Name Disambiguation

A Practioner's Guide to Evaluating Entity Resolution Results

Disambiguation and co-authorship networks of the U.S. patent inventor database (1975–2010)

Big data integration

A brief survey of automatic methods for author name disambiguation

Data Matching

Evaluating entity resolution results

Data quality and record linkage techniques

Email data cleaning

Model Assisted Survey Sampling

A method for calibrating false-match rates in record linkage

A Theory for Record Linkage

Automatic linkage of vital records.

A Generalization of Sampling Without Replacement from a Finite Universe

Bias of Precision Computed on Benchmark Datasets This section provides more information on example 1. For this example, we considered the RLdata10000 dataset from Sariyar and Borg (2022)

RecordLinkage: Record Linkage Functions for Linking and Deduplicating Data Sets . R package version 0.4-12

The Four Generations of Entity Resolution

“PatentsView: Disam-biguating Inventors, Assigness, and Locations,”

Descriptor : Disambiguation of patent inventors and assignees using high-resolution geolocation data

Learning String Alignments for Entity Aliases

Data Matching

Seeing the non-stars: (Some) sources of bias in past disambiguation approaches and a new public tool leveraging labeled records

Methods Matter: Rethinking Inventor Disambiguation with Classification & Labeled Inventor Records

On choosing thresholds for duplicate detection

Entity Resolution Evaluation Measures

Record Linkage Measures in an Entity Centric World

THE PINHAS SAPIR CENTER FOR DEVELOPMENT TEL AVIV UNIVERSITY Identification and Mobility of Israeli Patenting Inventors

“Matching Accuracy of the Lee-Kim-Marschke Computer Matching Program.”

Quality and Complexity Measures for Data Linkage and Deduplication

On Evaluation and Training-Set Construction for Duplicate Detection

Algorithms for Scoring Coreference Chains

Sampling techniques.

These are the precision and recall estimators corresponding to uniformly sampling records in lemma 1 ( p i ∝ 1) with the bias adjustment given in

It can easily be generalized to estimate other clustering metrics, such as cluster precision, cluster recall, cluster homogeneity and completeness, and other generalized merge distances

This is the “naive” precision (respectively recall) estimator obtained by computing precision (respectively recall) when only looking at records that appear in the sampled clusters

trivial precision estimate

200 records were sampled and the ground truth clusters associated with them were recovered

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names