Dataset too big

Hi guys!

I need some input on how to work around this:

I'm trying to create a Real-ESRGAN Compact upscaling model with NeoSR, but my dataset it quite too big:

\~28mio 256x256 pictures as gt and the same amount as 128x128 lq for a 2x upscaling model.

This dataset has about 600GB - i tried both io options: disk and lmdb
But i always get an memory error, not VRAM, but normal memory - it seems to try to load …

big compact dataset deeplearning upscaling work