How to handle extreme large datasets

I want to learn techniques to handle extremely large datasets say tens of millions of rows.
If I have to create a pipeline of how I can preprocess the data, do inference write outputs to table in minimum time. What are some best practices which can speed up the pipeline/workflow ?

best practices data datasets deeplearning inference large datasets learn pipeline practices preprocess speed table workflow