Data deduplication is critical for preventing training data contamination and ensuring evaluation integrity. Our deduplication engine operates at three levels: exact match, near-duplicate, and semantic similarity. Each level uses different algorithms optimized for its detection threshold.
Exact deduplication uses content hashing with normalization to catch trivially identical records regardless of whitespace, encoding, or formatting differences. This stage processes data at millions of records per second on standard hardware.
Near-duplicate detection employs MinHash with locality-sensitive hashing to identify documents that share substantial content but differ in minor ways. The similarity threshold is configurable, and the system reports cluster sizes to help identify content that has been copied and lightly modified across sources.
Semantic deduplication uses embedding-based similarity to find records that express the same information in different words. This catches paraphrased content, translations, and reformulations that escape surface-level detection.
Cross-split deduplication ensures that training, validation, and test sets contain no overlapping content, preventing data leakage that would inflate evaluation metrics.