1 of 61

會進行�「深度思考」的�大型語言模型

Todo:: Foundation 對 RL 很重要 O

Todo: fix rejection sampling O

Todo: NV namotron O X

Todo: 從概念上說明 reasoning O

Todo: fixing deepseek r1 O done

Todo: Supervised Chain of Thought (good to read) O

Todo: 要不要講模型學習改自己的錯誤: Self-rewarding correction for mathematical reasoning X

Todo: 要不要講甚麼是真正的 RL X

Todo: 要不要講 GRPO X

Todo:人類一輩子只有四 GB 的知識? X

Todo: 從理論上說明 reasoning -> Next time

Todo: 要不要講 MCTS X

===

Limitations of attention mechanism, with implications in generalization and optimization:

https://www.youtube.com/watch?v=EMUsZ0OqYCA

Why chain of thought: https://www.youtube.com/watch?v=_I2yEhxRJlc -> about removing coT

Video:

https://www.youtube.com/watch?app=desktop&v=fZNNqcN_UQM (not sure good or not)

=====

檢查作業 3 的問題

Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling, https://arxiv. org/pdf/2502.06703

The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models

https://arxiv.org/abs/2503.02875

要不要講對於 self-correction 的質疑?

(1) When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMs

(2) Large Language Models Cannot Self-Correct Reasoning Yet

(3) Chain of Thoughtlessness? An Analysis of CoT in Planning

要不要講 sampling 的時候需要 diversity:

Forest of thought , https://arxiv.org/abs/2412.09078

Diverse Verifier Tree Search (DVTS)

PRM: The Lessons of Developing Process Reward Models in Mathematical Reasoning

要不要講 large language monkey:

可能一定需要講:

為什麼列出思考過程會比較好

=====

要不要講 AI agent vs reasoning 之間的關係?

要不要講 ARC-AGI

要不要講 last exam?

====

要不要講 Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiaqi Wang, Mengkang Hu, Zhi Chen, Wanxiang Che, and Ting Liu. Ecm: A unified electronic circuit model for explaining the emergence of in-context learning and chain-of-thought in large language model. arXiv preprint arXiv:2502.03325, 2025 (sampling vs reasoning limitation) with formulation

要不要講 DeltaBench: https://arxiv.org/pdf/2502.19361 對於 long CoT 內容的分析

====

一堂課搞懂

Or lean to reasoning

Denny Zhou: LLM Reasoning: Key Ideas and Limitations

https://www.youtube.com/watch?v=-SZAciVbswk

Good video: Speculations on Test-Time Scaling (o1)

https://www.youtube.com/watch?v=6PEJ96k1kiw&t=2162s

Overview paper:

A Survey on LLM Complex Reasoning through the Lens of Self-Evolution

Thinking Machines: A Survey of LLM based Reasoning Strategies 1

Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models 7 (large group)

LLM Post-Training: A Deep Dive into Reasoning Large Language Models 13

Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models 12

Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models - A Survey 31 (but old)

Some blog post:

https://sebastianraschka.com/blog/2025/understanding-reasoning-llms.html (done)

https://sebastianraschka.com/blog/2025/first-look-at-reasoning-from-scratch.html (need subcribe???)

https://sebastianraschka.com/blog/2025/state-of-llm-reasoning-and-inference-scaling.html -> 14 papers

Good blog:

https://towardsdatascience.com/advanced-prompt-engineering-chain-of-thought-cot-8d8b090bf699/

Clear reference:

https://huggingface.co/blog/Kseniase/testtimecompute (focus on images)

https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-reasoning-llms (this one is pretty clear)

https://kili-technology.com/large-language-models-llms/llm-reasoning-guide

Reference (not special):

https://cloudsecurityalliance.org/blog/2024/12/13/test-time-compute

https://www.youtube.com/watch?v=pi7LF-OpO6k

2 of 61

等等，這裡會不會有什麼陷阱？ …… 讓我想想，有沒有什麼情況下1+1不等於2？在二進位中，1+1等於10

使用者只是寫了「1+1=」，沒有提供額外資訊。他們可能是在測試我會不會想得太複雜。

ChatGPT o1/o3/o4、DeepSeek r1、Gemini 2 Flash Thinking、Claude 3.7 Sonnet (Extended Thinking) ……

3 of 61

「深度思考」語言模型的特色

問題

給一個「思考」過程

答案

<think>

</think>

Verification

“Let me check the answer …”

Planning

“Let’s first try to ……”

Explore

“Let’s try a different approach …”

……

(「Inference」字面翻譯類似，但意思完全不同)

Test-Time Compute

第一堂課：“深度不夠，長度來湊”

「推理」(Reasoning)

4 of 61

https://www.nature.com/articles/nature16961

AlphaGo

Training

Time

Testing

Time

Test-Time Compute

5 of 61

「思考」越多結果越好

https://arxiv.org/abs/2104.03113

Scaling Scaling Laws with Board Games

Test-Time Scaling

6 of 61

打造「推理」語言模型的方法

更強的思維鏈 (Chain-of-Thought, CoT)

給模型推論工作流程

教模型推理過程 (Imitation Learning)

以結果為導向學習推理 (Reinforcement Learning, RL)

不用微調參數

需要微調參數

7 of 61

打造「推理」語言模型的方法

更強的思維鏈 (Chain-of-Thought, CoT)

給模型推論工作流程

教模型推理過程 (Imitation Learning)

以結果為導向學習推理 (Reinforcement Learning, RL)

8 of 61

Chain-of-Though �(CoT)

https://arxiv.org/abs/2201.11903

Few-shot CoT

Zero-shot CoT

https://arxiv.org/abs/2205.11916

推理模型在做的事情叫做 “Long CoT”

https://arxiv.org/abs/2503.09567

“Short CoT”

9 of 61

gpt-4o

Supervised CoT

https://arxiv.org/abs/2410.14198

10 of 61

11 of 61

不是所有模型都有能力根據複雜指令做 Long CoT

12 of 61

打造「推理」語言模型的方法

更強的思維鏈 (Chain-of-Thought, CoT)

給模型推論工作流程

教模型推理過程 (Imitation Learning)

以結果為導向學習推理 (Reinforcement Learning, RL)

13 of 61

如何 Explore？同一個問題多試幾次

input

output 1

output 2

output 3

14 of 61

“Large Language Monkeys”

https://arxiv.org/abs/2407.21787

15 of 61

如何 Explore？同一個問題多試幾次

input

output 1

output 2

output 3

Majority Vote

(Self-consistency)

https://arxiv.org/abs/2203.11171

Confidence

(used in CoT decoding)

https://arxiv.org/abs/2402.10200

把答案放到 <answer></answer>

如何選出正確答案來？

16 of 61

如何 Explore？同一個問題多試幾次

https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute

17 of 61

加上 Verification

input

output 1

output 2

output 3

Verifier

output

score

Verifier

0.1

Verifier

0.9

Verifier

0.2

Best-of-N

https://arxiv.org/abs/2110.14168

也是語言模型

18 of 61

加上 Verification

input

output 1

output 2

output 3

ground truth

Verifier

1.0

output 1

Verifier

1.0

output 3

Verifier

0.0

output 2

input

Training Data:

19 of 61

Parallel vs. Sequential

input

output 1

output 2

output 3

input

output 2

output 3

output 1

Parallel

Sequential

20 of 61

Parallel vs. Sequential

input

output 1-1

output 2-1

output 3-1

Parallel + Sequential

output 1-2

output 2-2

output 3-2

https://arxiv.org/abs/2408.03314

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

21 of 61

會對中間的步驟進行驗證

planning

Verification (for a step)

123 x 456 =?

22 of 61

input

step 1

Process

Verifier

score

step 1

Let's Verify Step by Step

https://arxiv.org/abs/2305.20050

23 of 61

24 of 61

input

step 1

Process

Verifier

score

step 1

Let's Verify Step by Step

https://arxiv.org/abs/2305.20050

</step>

25 of 61

input

step 1

step 2

step 3

ans

step 2

step 3

ans

step 2

step 3

step 4

ans

2/3

input

step 1

step 2

step 3

ans

step 3

ans

1/3

step 3

ground truth

input

Training Data:

26 of 61

input

step 1

2/3

input

step 1

step 2

1/3

Process

Verifier

Process

Verifier

2/3

1/3

Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations

https://arxiv.org/abs/2312.08935

27 of 61

</step>

input

step 1

</step>

Process

Verifier

score

step 1

step 2

Beam Search

https://arxiv.org/abs/2305.00633

https://arxiv.org/abs/2401.17686

每次保留 N 條路徑

28 of 61

https://huggingface.co/spaces/HuggingFaceH4/blogpost-scaling-test-time-compute

29 of 61

Heuristic Search Algorithm

ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search

https://arxiv.org/abs/2406.03816

Source of image: Wikipedia

Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning

https://arxiv.org/abs/2405.00451

Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solvers

https://arxiv.org/abs/2408.06195

e.g. Monte Carlo Tree Search (MCTS)

30 of 61

打造「推理」語言模型的方法

更強的思維鏈 (Chain-of-Thought, CoT)

給模型推論工作流程

教模型推理過程 (Imitation Learning)

以結果為導向學習推理 (Reinforcement Learning, RL)

需要微調參數

(Foundation Model)

Post-Training

LLM without Reasoning

Learn to Reasoning

(Fine-tuned Model)

LLM with

Reasoning

31 of 61

打造「推理」語言模型的方法

更強的思維鏈 (Chain-of-Thought, CoT)

給模型推論工作流程

教模型推理過程 (Imitation Learning)

以結果為導向學習推理 (Reinforcement Learning, RL)

32 of 61

教模型推理過程

reasoning process

Input

ground truth

Training data:

Input

???

哪裡來？

33 of 61

想辦法生成推論過程的訓練資料

input

Verifier

reasoning process

ans

Training data

ground truth

input

Training Data:

CoT

怎麼確保推理過程都是對的？

34 of 61

input

step 1

step 2

step 3

ans

step 3

ans

https://arxiv.org/abs/2501.04519

rStar-Math

input

step 1

step 2

step 3

ans

Training data:

Reasoning Processing

35 of 61

input

step 1

step 2

step 3

ans

step 3

ans

input

step 1

input

step 1

step 2

step 3

step 1

step 2

step 3

https://arxiv.org/abs/2501.04519

rStar-Math

36 of 61

找找自己的問題!

(正確答案為 9)

推理過程需要每一步都是對的嗎？

只要最後答案正確就好

37 of 61

input

step 1

step 2

step 3

ans

step 3

ans

input

step 1

step 2

step 3

ans

Training data:

Reasoning Processing

推理過程全是正確的，不知道找找自己的問題!

38 of 61

Stream of search (SoS)

https://arxiv.org/abs/2404.03683

input

step 1

step 2

step 3

ans

step 3

ans

input

step 1

step 2

step 1

step 2

step 3

ans

[插入 Verifier 的回饋]

“重新來過”

39 of 61

https://arxiv.org/abs/2410.18982

40 of 61

現在可以做 Knowledge Distillation

Sky-T1: https://novasky-ai.github.io/posts/sky-t1/

s1:https://arxiv.org/abs/2501.19393

reasoning process

Input

answer

Input

?????

Reasoning Model

41 of 61

現在可以做 Knowledge Distillation

https://arxiv.org/abs/2501.12948

Foundation Model

42 of 61

打造「推理」語言模型的方法

更強的思維鏈 (Chain-of-Thought, CoT)

給模型推論工作流程

教模型推理過程 (Imitation Learning)

以結果為導向學習推理 (Reinforcement Learning, RL)

DeepSeek-R1 系列的作法

43 of 61

input

Reasoning Process

answer

Reasoning Process

answer

ground truth

input

Training Data:

(Foundation Model)

RL

DeepSeek-v3-base

DeepSeek-R1-Zero

https://arxiv.org/abs/2501.12948

Accuracy as reward

推理過程內容不重要，最後答案是對的就好

Reinforcement Learning (RL)

44 of 61

Source of image: https://arxiv.org/abs/2501.12948

Majority vote

45 of 61

Aha Moment

Source of image: https://arxiv.org/abs/2501.12948

46 of 61

input

Reasoning Process

answer

Reasoning Process

answer

ground truth

input

Training Data:

推理過程內容不重要，最後答案是對的就好

https://arxiv.org/abs/2501.12948

Poor readability &

Language Mixing

(Foundation Model)

RL

DeepSeek-v3-base

DeepSeek-R1-Zero

Accuracy as reward

47 of 61

Imitation Learning

Model A?

RL

DeepSeek-v3-base

DeepSeek-R1-Zero

Accuracy as reward

DeepSeek-v3-base

reasoning process

Input

ground truth

Generated data

+ human annotation

using few-shot prompting with a long CoT as an example

directly prompting models to generate detailed answers with reflection and verification

Model A?

RL

Accuracy / Language coherence as reward

Model B?

(Thousands of examples)

48 of 61

Model B?

input

Reasoning Process

Including tasks without standard answers

ans

Reasoning Process

ans

Reasoning Process

ans

Imitation Learning

Model C?

DeepSeek-v3-base

RL

DeepSeek-R1

Model C?

Safety / Helpfulness

DeepSeek-v3

As verifier

Based on the Deepseek-R1 paper, both the process verifier and MTCS were tried but ultimately not used.

600k examples

“filtered out chain-of-thought with mixed languages, long parapraphs, and code blocks”

49 of 61

50 of 61

Foundation Model 很重要

透過 RL 無法大幅強化 Qwen-32B-Base 的能力

以 Qwen-32B-Base 作為 Foundation Model

Imitation Learning (向 DeepSeek-R1學習) 對 Qwen-32B-Base 反而比較有效

RL 是強化模型原有的能力

51 of 61

Foundation Model

本來就會 Aha

Source of image:

https://arxiv.org/abs/2503.20783

DeepSeek-v3

52 of 61

打造「推理」語言模型的方法

更強的思維鏈 (Chain-of-Thought, CoT)

給模型推論工作流程

教模型推理過程 (Imitation Learning)

以結果為導向學習推理 (Reinforcement Learning, RL)

不用微調參數

需要微調參數

53 of 61

下回預告：推理模型的挑戰與未來發展

reasoning process

Input

answer

Reasoning Model

花錢、花算力

54 of 61

planning

Verification (for a step)

123 x 456 =?

55 of 61

(sub)planning

verification

56 of 61

verification + explore

57 of 61

verification + explore

58 of 61

verification + explore

59 of 61

verification

60 of 61

verification + explore

……

61 of 61

下回預告：推理模型的挑戰與未來發展

reasoning process

Input

answer

Reasoning Model

真的需要產生這麼長的推論過程嗎？有沒有辦法縮短推論過程

花錢、花算力