Stochastic Two Points Method for Deep Model Zeroth-order Optimization

Large foundation models, such as large language models, have performed exceptionally well in various application scenarios. Building or fully fine-tuning such large models is usually prohibitive due to either hardware budget or lack of access to backpropagation. The zeroth-order methods offer a promising direction for tackling this challenge, where only forward passes are needed to update the model. This paper introduces an efficient Stochastic Two-Point (S2P) approach within the gradient-free regime. We present the theoretical convergence properties of S2P under …

application backpropagation budget building challenge cs.lg fine-tuning foundation hardware language language models large language large language models large models optimization stochastic