NanoBatch Privacy: Enabling fast Differentially Private learning on the IPU. (arXiv:2109.12191v2 [cs.LG] UPDATED)

Differentially private SGD (DPSGD) has recently shown promise in deep
learning. However, compared to non-private SGD, the DPSGD algorithm places
computational overheads that can undo the benefit of batching in GPUs.
Micro-batching is a common method to alleviate this and is fully supported in
the TensorFlow Privacy library (TFDP). However, it degrades accuracy. We
propose NanoBatch Privacy, a lightweight add-on to TFDP to be used on Graphcore
IPUs by leveraging batch size of 1 (without microbatching) and gradient
accumulation. This …

arxiv enabling ipu learning privacy