Few new things in Python which I learned last week.

Chunkify Huge List into Smaller N equal size lists.

In order to backfill data for one of our machine learning pipeline I have to divide the date list into small n list of equal length and distribute them at n GPU cluster.

from datetime import timedelta,date,datetime
start_dt = date(2023,1,1)
end_dt = date(2023,12,31)
cdays = []

while start_dt < end_dt:
    cdays.append(start_dt)
    start_dt += timedelta(days=3)

#print(cdays)

def split(a, n):
    k, m = divmod(len(a), n)
    return (a[i*k+min(i, m):(i+1)*k+min(i+1, m)] for i in range(n)) …

cluster data dataengineering datetime equal gpu import list lists machine machine learning machinelearning pipeline python small them