Section 1 of 1
Day 5 · Fri 9 Oct · Reinforcement Learning, Goal Misgeneralisation and AIXI: Daily Quiz
Day 5 · Fri 9 Oct · Reinforcement Learning (D.1.2), Goal Misgeneralisation (D.2), AIXI (D.3) · David Quarel
This is a simple quiz to help you recall what you have learned today, and to confirm you have achieved the learning outcomes. The questions are deliberately easy. Passing this quiz is *not* a replacement for being in the course. You do not need to submit the quiz, and we will not grade it.
Nothing is collected about who you are. If you do submit, your score and the correct answers appear immediately.
This is a simple quiz to help you recall what you have learned today, and to confirm you have achieved the learning outcomes. The questions are deliberately easy. Passing this quiz is *not* a replacement for being in the course. You do not need to submit the quiz, and we will not grade it.
Nothing is collected about who you are. If you do submit, your score and the correct answers appear immediately.
Reinforcement learning
Reinforcement learning
Loading...
How is the state value V^π(s) related to the action value Q^π(s, a)?
How is the state value V^π(s) related to the action value Q^π(s, a)?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
V^π(s) = Σ_a π(a|s) Q^π(s, a)
V^π(s) = max_a Q^π(s, a), for every policy π
V^π(s) = Q^π(s, a) − γ, for any action a
V^π(s) = Σ_a Q^π(s, a), summed without weights
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
In the worksheet's three-state MDP, V^πL(s0) = 1/(1−γ²) and V^πR(s0) = 2γ/(1−γ²). For which discount factors is going right optimal?
In the worksheet's three-state MDP, V^πL(s0) = 1/(1−γ²) and V^πR(s0) = 2γ/(1−γ²). For which discount factors is going right optimal?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
γ > 1/2
γ < 1/2
Every γ in (0, 1)
No γ: going left is always optimal
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
The Bellman optimality operator B is a γ-contraction in the sup-norm. What does that buy us?
The Bellman optimality operator B is a γ-contraction in the sup-norm. What does that buy us?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
B has a unique fixed point V*, and value iteration Vₙ₊₁ = BVₙ converges to it from any starting V₀
Value iteration converges in exactly |S| steps
Every policy is optimal once γ < 1
The optimal policy is unique
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
Policy improvement: let π' be greedy with respect to V^π. Which statement is true?
Policy improvement: let π' be greedy with respect to V^π. Which statement is true?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
V^π'(s) ≥ V^π(s) at every state s
V^π'(s) ≥ V^π(s) on average over states, but it can be worse at some states
π' is always optimal after one step
π' can be worse than π if γ is close to 1
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
Why does policy iteration on a finite MDP stop after finitely many steps?
Why does policy iteration on a finite MDP stop after finitely many steps?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
There are finitely many deterministic policies (|A|^|S|), and each step strictly improves the policy until it is optimal
Because the Bellman operator is linear
Because the discount factor shrinks the value to zero
It does not: it only converges in the limit
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
Exact policy evaluation gives v^π = (I − γP^π)⁻¹ r^π. Why is I − γP^π always invertible?
Exact policy evaluation gives v^π = (I − γP^π)⁻¹ r^π. Why is I − γP^π always invertible?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
P^π is row-stochastic, so ‖γP^π‖∞ = γ < 1 and the Neumann series Σₖ (γP^π)ᵏ converges
P^π is symmetric, so its eigenvalues are real
P^π is the identity for a deterministic policy
It is not always invertible; it fails when the MDP has absorbing states
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
Goal misgeneralisation
Goal misgeneralisation
Loading...
What is goal misgeneralisation?
What is goal misgeneralisation?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
The agent's capabilities generalise out of distribution, but it competently pursues a goal other than the intended one, even though the reward was correct on the training distribution
The designer wrote down the wrong reward, and the agent exploits it (specification gaming)
The agent fails to learn anything useful and acts randomly out of distribution
The agent overfits to the training levels and its capabilities collapse in new ones
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
In CoinRun, the coin always sat at the right-hand end of the level during training. When the coin is moved at test time, the trained agent typically:
In CoinRun, the coin always sat at the right-hand end of the level during training. When the coin is moved at test time, the trained agent typically:
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
Still runs competently to the right-hand end and ignores the coin
Goes straight to the coin wherever it is
Stops moving, because it no longer recognises the level
Dies on the first obstacle, because its navigation skills fail to transfer
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
Why can't more training on the same distribution fix goal misgeneralisation by itself?
Why can't more training on the same distribution fix goal misgeneralisation by itself?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
On the training distribution the intended goal and the proxy goal earn the same reward, so the reward signal cannot tell them apart
Because policy gradients cannot learn from sparse rewards
Because the network is too small to represent the intended goal
It can: training longer always removes the proxy goal
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
Solomonoff induction and AIXI
Solomonoff induction and AIXI
Loading...
A Bayesian mixture ξ = Σ_ν w_ν ν "dominates" every ν in its class. What does dominance say?
A Bayesian mixture ξ = Σ_ν w_ν ν "dominates" every ν in its class. What does dominance say?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
ξ(h) ≥ w_ν · ν(h) for every history h and every ν in the class
ξ(h) ≥ ν(h) for every ν in the class
ξ always puts all its weight on the true environment after one step
ξ(h) ≤ w_ν · ν(h) for every ν in the class
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
What is AIXI?
What is AIXI?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
The Bayes-optimal agent for a mixture over all computable (lower-semicomputable) environments, with prior weight 2^(−K(ν)), choosing actions by expectimax
A Q-learning agent with an infinitely large table
A policy-gradient agent trained on every possible environment
The agent that knows the true environment μ and acts optimally in it
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
"AIXI cannot be fooled": if the true environment μ is deterministic and V*_μ > ε along the history, what is guaranteed?
"AIXI cannot be fooled": if the true environment μ is deterministic and V*_μ > ε along the history, what is guaranteed?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
V*_ξ ≥ w_μ · ε > 0: AIXI expects strictly positive value. Not that it acts near-optimally
AIXI acts optimally in μ from the first step
AIXI's value converges to V*_μ for every environment
AIXI never takes an action with zero reward
Add option
or
add "Other"
…
Answer key
(1 point)
Loading...
In a sentence or two: a reward that is correct on every training episode can still produce an agent that pursues the wrong goal. How?
In a sentence or two: a reward that is correct on every training episode can still produce an agent that pursues the wrong goal. How?
Question Type
Short answer
Paragraph
Multiple choice
Checkboxes
Dropdown
File upload
Linear scale
Rating
Multiple choice grid
Checkbox grid
Date
Time
Answer key
(0 points)
Loading...
Link to Sheets
Summary
Question
Individual
Insights
Total points distribution
Loading...
Settings
Release grades
Respondent settings
Global quiz defaults
Default question point value
Point values for every new question
Responses
Manage how responses are collected and protected
Presentation
Manage how the form and responses are presented
Defaults
Form defaults
Settings applied to this form and new forms
Question defaults
Settings applied to all new questions