Noise-Robust De-Duplication at Scale

arXiv:2210.04261v2 Announce Type: replace
Abstract: Identifying near duplicates within large, noisy text corpora has a myriad of applications that range from de-duplicating training datasets, reducing privacy risk, and evaluating test set leakage, to identifying reproduced news articles and literature within large corpora. Across these diverse applications, the overwhelming majority of work relies on N-grams. Limited efforts have been made to evaluate how well N-gram methods perform, in part because it is unclear how one could create an unbiased evaluation dataset …

abstract applications articles arxiv cs.cl datasets diverse diverse applications literature near noise privacy risk robust scale set test text training training datasets type work