Token Diversity Generation Strategies for LLMs
"TLDR: This article explores three strategies used by LLMs (Language Models) when generating predicted tokens: greedy strategy, Beam Search, and Top-K sampling. The greedy strategy tends to produce repetitive content, while Beam Search introduces diversity by retaining the token sequences with the highest probabilities, but it may still cause the model to fall into local optima. Top-K sampling, on the other hand, increases diversity by randomly sampling from the top k tokens with the highest probabilities while discarding low-probability words. Additionally, the article discusses the impact of the temperature parameter on the normalization effect of the sigmoid function, as well as how adjusting temperature can control the creativity and diversity of generated text."
Greedy Strategy
When an LLM predicts the next token, it always selects the token with the highest probability. This approach tends to make the model generate repetitive content.
Beam Search
Beam Search employs a more lenient sampling strategy, and the greedy strategy is a special case of Beam Search where Beam Size = 1.
When the LLM generates the next token, we retain the Beam Size tokens with the highest probabilities. In the figure above, Beam Size = 2, so the 2 tokens with the highest probabilities (A, C) are retained. When predicting the next token, generation continues from each of the current sequences (A, C), with each sequence capable of generating 5 possibilities, resulting in 10 possible sequences in total (AA, AB, AC, AD, AE, CA, CB, CC, CD, CE). We then select the 2 sequences with the highest probabilities to retain.
By analogy, at each step, only the Beam Size token sequences with the highest probabilities are retained.
Beam Search introduces richer diversity to a certain extent, alleviating the model's tendency to fall into local optima, but there is still a certain probability that the model generates repetitive content.
Top-K Sampling
When the LLM generates the next token, it only retains the k tokens with the highest probabilities, and then performs random sampling among these k tokens.
This approach discards tokens with excessively low generation probabilities while greatly increasing generation diversity.
Temperature
The temperature parameter is mainly used in the softmax normalization function, where the parameter is set to adjust the smoothness of the normalized distribution.
Effects:
- : The distribution becomes flatter, increasing randomness. Low-probability words are more likely to be sampled.
- : The distribution becomes sharper, and the model tends to favor high-probability words.
- : The distribution is not adjusted, and sampling follows the original probabilities.
Advantages:
- It can control the creativity and diversity of generated text.
Disadvantages:
- Excessively high temperatures may lead to nonsensical outputs.
- Excessively low temperatures may lead to repetitive or monotonous outputs.