Implement Multi-GPU Training on a single GPU

Multi-GPU Training on a single GPU System in 3 Minutes

An Advanced Guide for TensorFlow

I want to share with you a neat little trick on how I test my multi GPU training code on a single GPU.

Motivation

I guess the problem is obvious and you probably experienced it yourself. You want to train a deep learning model and you want to take advantage of multiple GPUs, a TPU or even multiple workers …

deep learning distributed systems gpu python tensorflow