Common PEFT method principles

2026/08/285080 words11 min read

"TLDR: This article discusses LLM model fine-tuning methods trained on large-scale general data sets, especially how to adapt to specific tasks through methods such as adapters, prefixes, and prompts. In particular, the low-rank adaptive (LoRA) technology proposed by Microsoft is introduced. This method reduces the amount of calculations by decomposing the update matrix into the product of two low-rank matrices, and achieves similar effects to full fine-tuning. The article also discusses the specific principles, applications and challenges of LoRA."

#Common PEFT method principles

Why PEFT?

At present, LLM models are very large and are usually trained on large-scale general data sets. If they want to be transferred to a specific domain to achieve certain tasks, they need to be fine-tuned. In the Bert era, it can be fully fine-tuned, add an output layer at the end of the model's output, and then retrain with vertical domain data sets. However, models in the LLM era have too many parameters, and full fine-tuning is time-consuming and expensive. Therefore, the weight parameters of the LLM as a base model are no longer updated, but a small number of parameters are updated through other methods to adapt to specific tasks.

What are the methods?

The SFT method of LLM is usually as follows:

Adapter Tuning: Design an Adapter structure to embed into each layer of the Transformer, freeze the parameters of the original Transformer, and only update the parameters of the Adapter structure. The general structure of the Adapter is to first reduce the dimension, then use the nonlinear activation function, and finally increase the dimension, $Adapter(x)=W_2⋅ReLU(W_1⋅x)$
Prefix Tuning: Add prefixed "guide words" to the input sentences to guide the model to adapt to new tasks, but the "guide words" here are a set of randomly trainable vectors that are inserted into the input of each layer of the model. Specifically, the input added to each layer is [天, 气, 真, 好], then after inserting the Prefix vector, it becomes [P1, P2, P3, 天, 气, 真, 好], and this set of vectors is used as the input of each layer.
Prompt Tuning: Similar to Prefix Tuning, a set of randomly trainable vectors are used as "guide words", but the Prompt guide word vector only acts on the first input of the entire model, while Prefix Tuning is used as an input in every layer within the model.
LoRA: Decomposing the update matrix of the model into the product of two low-rank matrices can greatly reduce the amount of calculations, which will be described in detail later.

The most popular and common among the above methods is LoRA (Low-Rank Adaptation) proposed by Microsoft. Experiments have proven that LoRA can achieve almost the same effect as full fine-tuning, which can be said to be a huge difference.

The specific principles of LoRA

The essence of fine-tuning is that the parameters are updated along the gradient direction, that is, $W_1 = W_0 - \alpha \nabla W$ . We record the update matrix as $\Delta W$ . As long as the $\Delta W$ of each backpropagation can be calculated, then for an LLM of size 6B, the parameter amount of $\Delta W$ is also 6B, which is unacceptable.

LoRA adopts an approximation scheme and selects two matrices $A_{d\times r}, B_{r\times k} (r << {d, k})$ to satisfy $\Delta W = A_{d\times r} * B_{r \times k}$ . At this time, the sum of the parameters of $A and B$ is $(d + k) * r$ . The original $\Delta The parameter amount of W$ is $d*k$ , and the parameter amount is greatly reduced.

But we know that there are only a few matrices that can perform low-rank decomposition, so the disadvantage of LoRA is its poor expressive ability.

Specific application

Since the parameters trained by Lora are very few, perhaps less than 10MB in size, it is often used for personalized adaptation of the model.

In the image generation diffusion model, the community will share different styles of lora parameters, so that users can get different image generation styles according to their needs.
In the production environment of large models, we often want to provide users with personalized models, so we can try to train lora parameters separately for different users, so that the large model can be equipped with its unique lora parameters when different users ask questions.

Code implementation



from transformers import AutoModelForCausalLM


from peft import get_peft_config, get_peft_model, LoraConfig, TaskType





model_name_or_path = "facebook/opt-350m"





peft_config = LoraConfig(


    task_type=TaskType.CAUSAL_LM,


    r=8, # low rank


    lora_alpha=32, # scaling factor


    lora_dropout=0.1,


)





model = AutoModelForCausalLM.from_pretrained(model_name_or_path)


model = get_peft_model(model, peft_config)


model.print_trainable_parameters()

$r$ and $lora\_alpha$ in LoraConfig are in the formula $W_1 = W_0 + \frac{lora\_alpha}{r} \Delta W = W_0 + \frac{lora\_alpha}{r} AB$

Selection of lora_alpha:

If you want more aggressive adaptation, set α > r
If you want a more conservative adaptation, set α < r
If unsure, you can set α = r as a starting point

KV Cache

This article introduces the importance of KV Cache in LLM (Large Language Model). It is a key step in accelerating operations to ensure that the inference speed is not affected when the input text grows during the conversation. The article explains the Self Attention mechanism and the concept of causal mask in detail, and discusses how KV Cache optimizes the prediction process by caching calculation results.

2026/08/28·6 min read