Equall & Apple’s Revolutionizing Transformers: One Wide Feedforward for Unprecedented Efficiency and Accuracy

A collaborative research effort from Equall and Apple delves into the role of the FFN and uncovers a surprising revelation: despite consuming a significant portion of the model's parameters, the FFN exhibits high redundancy. As a result, the researchers propose sharing a single FFN across both the encoder and decoder, thereby reducing the parameter count while causing only a modest drop in accuracy.

accuracy ai apple artificial intelligence collaborative deep-neural-networks efficiency machine learning machine learning & data science ml redundancy research researchers role technology transformers