LookupFFN: Making Transformers Compute-lite for CPU inference

arXiv:2403.07221v1 Announce Type: new
Abstract: While GPU clusters are the de facto choice for training large deep neural network (DNN) models today, several reasons including ease of workflow, security and cost have led to efforts investigating whether CPUs may be viable for inference in routine use in many sectors of the industry. But the imbalance between the compute capabilities of GPUs and CPUs is huge. Motivated by these considerations, we study a module which is a workhorse within modern DNN …

arxiv compute cpu cs.lg inference making transformers type