Loading a large, messy csv using data.table fread with cli tools

Setup

library(data.table)
library(here)

## here() starts at /Users/davidlucey/Desktop/David/Projects/redwall-analytics

library(glue)

## Warning: package 'glue' was built under R version 4.1.2

library(tictoc)
setDTthreads(percent = 90)
path_to_data <- "~/Desktop/David/Projects/uscompanies/data"
path_to_original <- here::here(path_to_data, "uscompanieslist.csv")

Introduction

On a recent side project, we encountered a large (7GB) csv of 30+ million US business names and addresses, which couldn’t be loaded into R, because of corrupted records. While not widely discussed, we have known for some time that it was possible ...

cli csv data loading r bloggers tools