[N] PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation

The core concept of PRILoRA involves departing from the conventional practice of assigning a uniform low rank to each layer in a model. Instead, they propose a dynamic assignment that linearly increases across the layers. This ensures that layers closer to the input receive lower ranks, while deeper layers are assigned higher ranks. For instance, in the DeBERTaV3-base model, instead of uniformly assigning a rank of 8 to each layer, they start with the first layer at a rank of …

concept core dynamic layer low low-rank adaptation machinelearning practice uniform