Training on multi-gpu

I’m planning to rent a machine with two A100-SXM4-80G but I’m a bit confused on the multi-gpu setup. I hear that it can combine the VRAM so that it becomes one GPU of 160G where we can double the batch size. But when I checked out Pytorch multi-gpu code samples, the closest I can find is DataParallel, which splits the data to x amount of small batches to train on each gpu and then merge the result. Am I looking …

deeplearning gpu multi-gpu training