The version of the browser you are using is no longer supported. Please upgrade to a supported browser.Dismiss
Day 5 · Fri 9 Oct · Reinforcement Learning, Goal Misgeneralisation and AIXI: Daily Quiz
11 responses
Loading...
Loading responses…
This form is approaching the response limit. Once surpassed, the summary of responses will no longer be displayed. Learn more
Close
This form has exceeded 50,000 responses and the summary of responses can no longer be displayed. You can still download responses as a .csv file. Learn more
Reinforcement learning
How is the state value V^π(s) related to the action value Q^π(s, a)?
Copy chart
No responses yet for this question.
In the worksheet's three-state MDP, V^πL(s0) = 1/(1−γ²) and V^πR(s0) = 2γ/(1−γ²). For which discount factors is going right optimal?
Copy chart
No responses yet for this question.
The Bellman optimality operator B is a γ-contraction in the sup-norm. What does that buy us?
Copy chart
No responses yet for this question.
Policy improvement: let π' be greedy with respect to V^π. Which statement is true?
Copy chart
No responses yet for this question.
Why does policy iteration on a finite MDP stop after finitely many steps?
Copy chart
No responses yet for this question.
Exact policy evaluation gives v^π = (I − γP^π)⁻¹ r^π. Why is I − γP^π always invertible?
Copy chart
No responses yet for this question.
Goal misgeneralisation
What is goal misgeneralisation?
Copy chart
No responses yet for this question.
In CoinRun, the coin always sat at the right-hand end of the level during training. When the coin is moved at test time, the trained agent typically:
Copy chart
No responses yet for this question.
Why can't more training on the same distribution fix goal misgeneralisation by itself?
Copy chart
No responses yet for this question.
Solomonoff induction and AIXI
A Bayesian mixture ξ = Σ_ν w_ν ν "dominates" every ν in its class. What does dominance say?
Copy chart
No responses yet for this question.
What is AIXI?
Copy chart
No responses yet for this question.
"AIXI cannot be fooled": if the true environment μ is deterministic and V*_μ > ε along the history, what is guaranteed?
Copy chart
No responses yet for this question.
In a sentence or two: a reward that is correct on every training episode can still produce an agent that pursues the wrong goal. How?