Web: https://www.reddit.com/r/MachineLearning/comments/xqfi92/d_shrink_or_compress_data_size/

Sept. 28, 2022, 3:20 p.m. | /u/Beneficial_Law_5613

Machine Learning reddit.com

I have the following data structure:
ID, type x_coorde, y_coord, velocity, ... , speed, NG1,NG2.. NG9, lane.
the format looks like a two 2D Array

[
 [
  [1,1,0.9832,0.55,...,1,...0],[1,1,0.9832,0.55,...,2,...0],..., [1,1,0.9832,0.55,...,3,...0] (here are 9 arrays with 31 elements)
 ],
 [...],...[...] (39 million)
]
in total, I have 39 million arrays, each containing 9 arrays with 31 elements.
When I try to load this dataset into the GPU to train I have CUDA out of memory RuntimeError. My question is:
Is there any …

data machinelearning

DATA ANALYST /- CONTROLE DE GESTION ET FINANCE H/F

@ METRO/MAKRO | Nanterre, France

Data Analyst

@ Netcentric | Barcelona, Spain

Power BI Developer

@ Lendi Group | Sydney, Australia

Staff Data Scientist - Merchant Services (Remote, North America)

@ Shopify | Dallas, TX, United States

Machine Learning / Data Engineer

@ WATI | Vietnam - Remote

F/H Data Manager

@ Bosch Group | Saint-Ouen-sur-Seine, France

[Fixed-term contract until July 2023] Data Quality Controller - Space Industry Luxembourg (m/f/o)

@ LuxSpace Sarl | Betzdorf, Luxembourg

Senior Data Engineer (Azure DataBricks/datalake)

@ SpectraMedix | East Windsor, NJ, United States

Abschlussarbeit im Bereich Data Analytics (w/m/div.)

@ Bosch Group | Rülzheim, Germany

Data Engineer - Marketing

@ Publicis Groupe | London, United Kingdom

Data Engineer (Consulting division)

@ Starschema | Budapest, Hungary

Team Leader, Master Data Management - Support CN, HK & TW

@ Publicis Groupe | Kuala Lumpur, Malaysia