Lecture 4��DQN and Its Extensions

1

Instructor: Ercan Atam

Course: DSAI 642- Advanced Reinforcement Learning

2

List of contents for this lecture

​

• DQN: Deep Q-Networks

​

• DQN with experience replay

​

• Double DQN

​

• Duelling DQN

​

• Prioritized experience replay

​

3

Relevant readings for this lecture

​

• Chapter 8 of Shiyu Zhao, “Mathematical Foundations of Reinforcement learning”, Springer, 2025.

(Some slides at the beginning are modified/improved versions from the presentation slides of Shiyu Zhao)

​

​

• Chapter 2 of Laura Graesser and Wah Loon Keng, “Foundations of Deep Reinforcement Learning:

Theory and Practice in Python”, Addison-Wesley Professional, 2019.

​

​

• Chapter 13 of Richard S. Sutton and Andrew G. Barto, “Reinforcement Learning: An Introduction”,

Second Edition, MIT Press, Cambridge, MA, 2018.

​

​

• Chapter 7 of Nimish Sanghi, “Deep Reinforcement Learning with Python”, Second Edition, APress,

2024.

​

• https://cse.buffalo.edu/~avereshc/rl_fall19/lecture_14_1_Dueling_DQN_and_PER.pdf

(Some slides at the end are modified/improved versions from the presentation slides of Alina Vereshchaka)

4

A convention on notation

5

Deep Q-Learning (1)

6

Deep Q-Learning (2)

7

Deep Q-Learning (3)

8

Deep Q-Learning (4)

9

Deep Q-Learning (5)

10

Deep Q-Learning (6)

11

Deep Q-Learning: Two networks (1)

12

Deep Q-Learning: Two networks (2)

13

Deep Q-Learning: Experience replay (1)

14

Deep Q-Learning: Experience replay (2)

15

Deep Q-Learning: Experience replay (3)

16

Deep Q-Learning: Experience replay (4)

17

Polyak averaging in DQN (1)

Polyak averaging (also called soft target updates) is a technique used in DQN to improve training stability

and convergence by gradually updating the target network parameters, rather than updating them periodically

—which can sometimes cause abrupt changes.

18

Polyak averaging in DQN (2)

Why is it useful?

• Smoother target changes, leading to more stable Q-value estimates.
• Reduced risk of divergence due to abrupt parameter shifts.
• Better performance in complex or noisy environments.
• Faster convergence in many cases due to more consistent learning signals.

Advantages over periodic (hard) updates:

19

DQN Algorithm

20

Double DQN (DDQN) (1)

Remember “Double Q-Learning”:

21

Double DQN (DDQN) (2)

22

Double DQN (DDQN) (3)

23

Summary: DQN versus Double DQN

24

Double DQN Algorithm

25

Duelling DQN (1)

26

Duelling DQN (2)

27

Duelling DQN (3)

Example:

28

Duelling DQN (4)

Example (continue):

29

Duelling DQN (5)

30

Duelling DQN (6)

31

Example 1 for understanding duelling DQN motivation (1)

32

Example 1 for understanding duelling DQN motivation (2)

33

Example 2 for understanding duelling DQN motivation

34

Why can duelling DQN be more sample efficient?

35

Improvements provided by the duelling DQN

36

Duelling DQN: summary

37

Prioritized Experience Replay (PER)

38

PER: TD Error

39

PER: How to get priorities? (1)

40

PER: How to get priorities? (2)

41

PER: Adjustment through use of importance sampling (1)

42

PER: Adjustment through use of importance sampling (2)

43

PER: Adjustment through use of importance sampling (3)

44

PER: Adjustment through use of importance sampling (4)

45

Double DQN with PER

46

PER: Summary

References �(utilized for preparation of lecture notes or Matlab code)

​

• Shiyu Zhao, “Mathematical Foundations of Reinforcement learning”, Springer, 2025.
• Laura Graesser and Wah Loon Keng, “Foundations of Deep Reinforcement Learning: Theory and Practice in Python”, Addison-Wesley Professional, 2019.
• Richard S. Sutton and Andrew G. Barto, “Reinforcement Learning: An Introduction”, Second Edition, MIT Press, Cambridge, MA, 2018.
• https://cse.buffalo.edu/~avereshc/rl_fall19/lecture_14_1_Dueling_DQN_and_PER.pdf

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

47