BigQuery deduplication strategies

Problem statement

Context

• Lets assume that we have data pipeline(s) dumping messages into Google BigQuery tables (lets call them raw tables).


• There maybe duplicate messages being stored in the raw table due to reasons like:
  
  
  
  
  • Duplicate messages sent from source
  
  
  • Message inserted multiple times due to network issues and retries between the data pipeline and big query (although this can be addressed to some extent by using unique request ids while loading the data into BQ)
  
  
  
  


• BigQuery doesn't have unique …

big big query bigquery call context data data pipeline deduplication duplicate google googlecloud messages multiple network pipeline query raw sql strategies table tables