Laboratory of Computational Neuroscience
LCN - Brain Mind - I&C - Life Sciences - EPFL

Video Lectures of Wulfram Gerstner:

Reinforcement Learning

Part A covers classical methods of Reinforcement Learning whereas Part B covers modern topics such as deep reinforcment learning, intrinsic motivation and reward-based learning in the brain.

A: Foundations of Reinforcement Learning

This part A covers classical methods of Reinforcement Learning.

Reinforcement Learning Lecture 0. General Introduction to Artificial Neural Networks: Biological background and recent breakthroughs (26 min)
Part 1 - Neural Networks: from biological to artificial neurons (14 min)
Part 2 - Introduction to Deep Networks for Classification (4 min)
Part 3 - Artificial neural networks for Action Learning (8 min)

slides for Introduction - Lecture 0: General Introduction to Artificial Neural Networks

Reinforcement Learning Lecture 1. Reinforcement Learning and SARSA (134 min - 11 short videos)
Part 1 - Examples of Reward-based Learning (9 min)
Part 2 - Elements of Reinforcement Learning (13 min)
Part 2B - Quiz (2 min)
Part 3 - One-step Horizon (Bandit Problems) (22 min)
Part 3A - Convergence of iterative one-step algorithm (5 min)
Part 3B - Detour: Batch, Online, and Expected Online updates in Machine Learning (15 min)
Part 3C - Apply Batch, Online, and Expected Online to Q-values (9 min)
Part 4 - Exploration versus Exploitation (9 min)
Part 4A - Quiz: Exploration versus Exploitation with SoftMax Policy (6 min)
Part 5 - Bellman Equation (18 min)
Part 6 - SARSA algorithm (19 min)
Part 6B - Relation of SARSA to Bellman equation (9 min)

slides for Reinforcement Learning- Lecture 1: Q-values and SARSA
Reinforcement Learning Lecture 2. TD learning methods and Eligbility Traces (65 min - 9 short videos)
Part 1 - SARSA and BackUp diagrams (6 min)
Part 2 - Q-learning and Variations of SARSA (4 min)
Part 3 - TD-learning (Temporal Difference Learning) (7 min)
Part 4 - Monte-Carlo Methods (15 min)
Part 4A - Quiz: Monte-Carlo methods (2 min)
Part 4B - Example: Monte-Carlo versus Batch SARSA (6 min)
Part 5 - Eligibility traces (12 min)
Part 6 - n-step TD methods (8 min)
Part 6B - Quiz: comparing eligibility traces and n-step TD methods (5 min)

slides for Reinforcement Learning- Lecture 2: Variations of TD-learning methods and eligibility traces
Reinforcement Learning Lecture 3. Continuous State Space and Function Approximation (82 min - 9 short videos)
Part 1 - Continuous input space in Reinforcement Learning (13 min)
Part 2 - Learning parameters for function approximation in TD Learning (10 min)
Part 3 - Implementing Reinforcement Learning in Artificial Neural Networks (7 min)
Part 4 - Deep Q-Learning (basic idea - a longer study follows in a later video) (4 min)
Part 4B - Batch Semi-Gradient: Slow and Fast network updates in DQN. (7 min)
Part 5 - No Free Lunch Theorem (8 min)
Part 6 - Inductive Bias in Machine Learning (11 min)
Part 7 - Inductive Bias in Reinforcement Learning (Example 1) (4 min)
Part 8 - Inductive Bias (Example 2): Self-localization and goal-finding (18 min)

slides for Reinforcement Learning - Lecture 3: Continuous State Space and Function Approximation in Temporal Difference Learning
Reinforcement Learning Lecture 4. Policy Gradient Methods for Reinforcement Learning (96 min - 9 short videos)
Part 1 - Introduction: TD Methods versus Policy Gradient Methods (4 min)
Part 2 - Basic idea of policy gradient. (6 min)
Part 3 - Example: Binary actor with 1-step horizon (22 min)
Part 4A - From batch to online:Log-likelihood trick (7 min)
Part 4B Example (1-step horizon) revisited (17 min)
Part 4* Quiz - Policy Gradient Methods (3 min)
Part 5 - Policity gradient over Multiple time steps (23 min)
Part 6 - Subtracting the mean reward via the value function (13 min)
Part6* - Quiz (1 min)

slides for Reinforcement Learning - Lecture 4: Policy Gradient Methods for Reinforcement Learning
Reinforcement Learning Lecture 5. Actor-Critic Networks for Reinforcement Learning (69 min)
Part 1 - Introduction (3 min)
Part 2 - From Policy Gradeint to Deep Reinforcement Learning (14 min)
Part 3 - Actor-Critic Architecture and Advantage-Actor Critic (10 min)
Part 4 - Eligibility traces for Policy-Gradient and Actor-Critic (11 min)
Part 5 - How do eligibility traces arise in Policy-Gradient? (20 min)
Part 6 - Model-based versus Model-free Reinforcement Learning (11 min)

slides for Actor-Critic Networks for Reinforcement Learning

B: Reinforcement Learning, Intrinsic Rewards, and the Brain

This part B covers modern methods of 'Reinforcement Learning' with applications to intrinsically motivated agents, and the brain.

Reinforcement Learning and the Brain: 3-factor rules and brain-style computing (67 min)
Part 1 - Introduction: Roots of Reinforcement Learning in Brain Science (10 min)
Part 2 - Coarse Brain Anatomy and Reinforcement Learning (8 min)
Part 3 - Synaptic Plasticity and 'Learning rules' (17 min)
Part 4 - What Learning Rules to expect for Reinforcement Learning? (4 min)
Part 5 - Three-factor Learning Rules (14 min)
Part 6 - Policy Gradient with eligibility traces revisited (5 min)
Part 7 - Dopamine as a third factor is a TD-like signal (9 min)
slides for Reinforcement Learning and the Brain: 3-factor rules
Intrinsically motivated agents 1: Exploration, Novelty, and Surprise in Reinforcement Learning (54 min)
Part 1 - Introduction: Novelty is not Surprise (10 min)
Part 2 - Why is Surprise Useful? (6 min)
Part 3 - Change Point Detection with the Bayes-Factor Surprise (7 min)
Part 4 - Why is Novelty Useful? (14 min)
Part 5 - A Hybrid Model with Surprise, Novelty, and Reward (3 min)
Part 6 - Exploration of a Markov environment by Humans and Artificial Agents (14 min)

slides for Exploration, Novelty, and Surprise in Reinforcement Learning
Intrinsically motivated Agents 2: Curiosity-Driven Exploration in Reinforcement Learning (71 min, lectured by Alireza Modirshanechi)
Part 1 - Exploration Bonus in Tabular Reinforcement Learning (23 min)
Part 2 - Curiosity-driven Reinforcement Learning (48 min)
Intrinsically motivated agents 3: Noisy-TV problem and search strategies in human and artificial agents (45 min, lectured by Alireza Modirshanechi)
All parts - Noisy-TV Problem and Drawbacks of Curiosity-driven Reinforcement Learning

Recommended Textbook
Reinforcement Learning: an Introduction by R. Sutton and A. Barto (MIT Press)