Posts

This article reviews the content of the Fall 2024 software testing final exam, including multiple-choice questions, short-answer questions, and application questions. The multiple-choice section mainly covers concepts from the PPT, such as identifying implicit language and explicit language. The short-answer section requires introducing and providing examples of mutation testing, differential testing, as well as text-based and image-based software testing methods. The application questions involve the greedy algorithm and its related calculations, designing metamorphic testing relations and test cases for LCS, and designing fuzzing strategies, rules, and evaluation metrics for a speech recognition system.

This article introduces the application of the Byte Pair Encoding (BPE) algorithm in tokenization scenarios. Traditional text tokenization methods include word-based and character-based approaches, while for Chinese, there is also a common-word-based tokenization method. In the era of large language models (LLMs), the BPE algorithm has been adopted due to its suitability for massive datasets. By comparing tokenization methods at different levels, such as char level, word level, and subword level, the article highlights their respective advantages, disadvantages, and applicable scenarios.

This article explores three strategies used by LLMs (Language Models) when generating predicted tokens: greedy strategy, Beam Search, and Top-K sampling. The greedy strategy tends to produce repetitive content, while Beam Search introduces diversity by retaining the token sequences with the highest probabilities, but it may still cause the model to fall into local optima. Top-K sampling, on the other hand, increases diversity by randomly sampling from the top k tokens with the highest probabilities while discarding low-probability words. Additionally, the article discusses the impact of the temperature parameter on the normalization effect of the sigmoid function, as well as how adjusting temperature can control the creativity and diversity of generated text.

This article explores the shortcomings of KV Cache in LLMs (Language Models), including redundant computations during the inference process, issues with GPU memory utilization, and memory fragmentation problems. The article points out that although KV Cache can cache intermediate results to improve inference speed, it has several deficiencies in GPU memory utilization, such as underutilization of pre-allocated memory, memory fragmentation issues, and memory waste caused by the inability to effectively handle prompts of varying lengths. Next, the article introduces the Page Attention technique from operating systems, which optimizes process memory allocation by dividing memory into multiple pages and using virtual memory, providing a similar solution for vLLM. Finally, the article discusses the concept of shared KV blocks, aiming to reduce GPU memory usage and improve throughput.

This article introduces the implementation principles of MySQL's ACID, including atomicity, consistency, isolation, and durability. As the most commonly used storage engine in MySQL, InnoDB's internal architecture includes the buffer pool, redo log, and undo log. The implementation principle of atomicity primarily relies on the undo log to ensure transactional atomicity. Consistency is achieved through business logic or constraints. Isolation employs MVCC to enhance transactional concurrency performance. Durability is ensured via the redo log to prevent data loss caused by database crashes.