Laboratory of Computational Neuroscience
LCN - Brain Mind - I&C - Life Sciences - EPFL

Video Lectures of Wulfram Gerstner:

Reinforcement Learning

A: Foundations of Reinforcement Learning

• Reinforcement Learning Lecture 0. General Introduction to Artificial Neural Networks: Biological background and recent breakthroughs (26 min)
  Part 1 - Neural Networks: from biological to artificial neurons (14 min)
  Part 2 - Introduction to Deep Networks for Classification (4 min)
  Part 3 - Artificial neural networks for Action Learning (8 min)
  

  slides for Introduction - Lecture 0: General Introduction to Artificial Neural Networks

• Reinforcement Learning Lecture 1. Reinforcement Learning and SARSA (134 min - 11 short videos)
  Part 1 - Examples of Reward-based Learning (9 min)
  Part 2 - Elements of Reinforcement Learning (13 min)
  Part 2B - Quiz (2 min)
  Part 3 - One-step Horizon (Bandit Problems) (22 min)
  Part 3A - Convergence of iterative one-step algorithm (5 min)
  Part 3B - Detour: Batch, Online, and Expected Online updates in Machine Learning (15 min)
  Part 3C - Apply Batch, Online, and Expected Online to Q-values (9 min)
  Part 4 - Exploration versus Exploitation (9 min)
  Part 4A - Quiz: Exploration versus Exploitation with SoftMax Policy (6 min)
  Part 5 - Bellman Equation (18 min)
  Part 6 - SARSA algorithm (19 min)
  Part 6B - Relation of SARSA to Bellman equation (9 min)
  

  slides for Reinforcement Learning- Lecture 1: Q-values and SARSA

• Reinforcement Learning Lecture 2. TD learning methods and Eligbility Traces (65 min - 9 short videos)
  Part 1 - SARSA and BackUp diagrams (6 min)
  Part 2 - Q-learning and Variations of SARSA (4 min)
  Part 3 - TD-learning (Temporal Difference Learning) (7 min)
  Part 4 - Monte-Carlo Methods (15 min)
  Part 4A - Quiz: Monte-Carlo methods (2 min)
  Part 4B - Example: Monte-Carlo versus Batch SARSA (6 min)
  Part 5 - Eligibility traces (12 min)
  Part 6 - n-step TD methods (8 min)
  Part 6B - Quiz: comparing eligibility traces and n-step TD methods (5 min)
  

  slides for Reinforcement Learning- Lecture 2: Variations of TD-learning methods and eligibility traces

• Reinforcement Learning Lecture 3. Continuous State Space and Function Approximation (82 min - 9 short videos)

  Part 1 - Continuous input space in Reinforcement Learning (13 min)
  Part 2 - Learning parameters for function approximation in TD Learning (10 min)
  Part 3 - Implementing Reinforcement Learning in Artificial Neural Networks (7 min)
  Part 4 - Deep Q-Learning (basic idea - a longer study follows in a later video) (4 min)
  Part 4B - Batch Semi-Gradient: Slow and Fast network updates in DQN. (7 min)
  Part 5 - No Free Lunch Theorem (8 min)
  Part 6 - Inductive Bias in Machine Learning (11 min)
  Part 7 - Inductive Bias in Reinforcement Learning (Example 1) (4 min)
  Part 8 - Inductive Bias (Example 2): Self-localization and goal-finding (18 min)
  

  slides for Reinforcement Learning - Lecture 3: Continuous State Space and Function Approximation in Temporal Difference Learning

• Reinforcement Learning Lecture 4. Policy Gradient Methods for Reinforcement Learning (96 min - 9 short videos)

  Part 1 - Introduction: TD Methods versus Policy Gradient Methods (4 min)
  Part 2 - Basic idea of policy gradient. (6 min)
  Part 3 - Example: Binary actor with 1-step horizon (22 min)
  Part 4A - From batch to online:Log-likelihood trick (7 min)
  Part 4B Example (1-step horizon) revisited (17 min)
  Part 4* Quiz - Policy Gradient Methods (3 min)
  Part 5 - Policity gradient over Multiple time steps (23 min)
  Part 6 - Subtracting the mean reward via the value function (13 min)
  Part6* - Quiz (1 min)
  

  slides for Reinforcement Learning - Lecture 4: Policy Gradient Methods for Reinforcement Learning

• Reinforcement Learning Lecture 5. Actor-Critic Networks for Reinforcement Learning (69 min)

  Part 1 - Introduction (3 min)
  Part 2 - From Policy Gradeint to Deep Reinforcement Learning (14 min)
  Part 3 - Actor-Critic Architecture and Advantage-Actor Critic (10 min)
  Part 4 - Eligibility traces for Policy-Gradient and Actor-Critic (11 min)
  Part 5 - How do eligibility traces arise in Policy-Gradient? (20 min)
  Part 6 - Model-based versus Model-free Reinforcement Learning (11 min)
  

  slides for Actor-Critic Networks for Reinforcement Learning

• Reinforcement Learning Lecture 6 (Johanni Brea) Markov Decision Processes (91 min - 3 videos)

  Part 1 - Definition and Examples (50 min)
  Part 2 - Solving MDPs with Dynamic Programming and Value Iteration (17 min)
  Part 3 - Policy Improvement Theorem and Solving MDPs with Policy Iteration and Linear Programming (24 min)

B: Deep Reinforcement Learning, Applications, Intrinsic Rewards, and the Brain

• Deep Reinforcement Learning Lecture 1. Applications in Video Games and Simulated Robotics (53 min, lectured by Johanni Brea)

  Part 1 (Lecture recording 2024) Model-free Deep Reinforcement Learning for Video Games: DQN, A2C and Importance Sampling (Lecture recording starts at timer 6:30)

  Part 2 (Lecture recording 2024) Unfortunately only the sound track was recorded but not the lecture slides. Please look Version from 2023

  Part 1 (version 2023)- Mini-Batches in On- and Off-Policy Deep Reinforcement Learning (18 min)
  Part 2 (version 2023) - Proximal Policy Optimization for Continuous Control (25 min)
  Part 3 (version 2023) - Deep Deterministic Policy Gradient for Continuous Control (10 min)
  

• Deep Reinforcement Learning Lecture 2. Applications of Model-Based RL in Board Games (Chess, Go, others) (55 min, lectured by Johanni Brea)

  Part 1 (Lecture recording 2024) Basics of Model-Based Deep RL and Variational State Tabulation. (video starts at Min 0:40, Model-based RL starts at Min 14:00. Unfortunately, the sound track stops after 34 minutes, please refer to version from 2023)

  Part 2 (Lecture recording 2024) alpha-zero,mu-zero, agent 57,

  Part 1 (Version 2023) - Background Planning and Variational State Tabulation (16 min)
  Part 2 (Version 2023) - Monte Carlo Tree Search and Alpha Zero (25 min)
  Part 3 (Version 2023) - MuZero (14 min)
  

  slides for Deep Reinforcement Learning - Lectures 1+ 2: Deep Reinforcement Learning

• Reinforcement Learning and AI: Ethics and Regulation
  Lecture recording 2024 The need of regulation

• Large Language Models and Reinforcement Learning (C. Gulcehre)
  Lecture recording 2024 Human-AI alignment

• Reinforcement Learning and the Brain: 3-factor rules and brain-style computing (67 min)
  Part 1 - Introduction: Roots of Reinforcement Learning in Brain Science (10 min)
  Part 2 - Coarse Brain Anatomy and Reinforcement Learning (8 min)
  Part 3 - Synaptic Plasticity and 'Learning rules' (17 min)
  Part 4 - What Learning Rules to expect for Reinforcement Learning? (4 min)
  Part 5 - Three-factor Learning Rules (14 min)
  Part 6 - Policy Gradient with eligibility traces revisited (5 min)
  Part 7 - Dopamine as a third factor is a TD-like signal (9 min)

  slides for Reinforcement Learning and the Brain: 3-factor rules

• Intrinsically motivated agents 1: Exploration, Novelty, and Surprise in Reinforcement Learning (54 min)
  Part 1 - Introduction: Novelty is not Surprise (10 min)
  Part 2 - Why is Surprise Useful? (6 min)
  Part 3 - Change Point Detection with the Bayes-Factor Surprise (7 min)
  Part 4 - Why is Novelty Useful? (14 min)
  Part 5 - A Hybrid Model with Surprise, Novelty, and Reward (3 min)
  Part 6 - Exploration of a Markov environment by Humans and Artificial Agents (14 min)
  

  slides for Exploration, Novelty, and Surprise in Reinforcement Learning

• Intrinsically motivated Agents 2: Curiosity-Driven Exploration in Reinforcement Learning (71 min, lectured by Alireza Modirshanechi)
  Part 1 - Exploration Bonus in Tabular Reinforcement Learning (23 min)
  Part 2 - Curiosity-driven Reinforcement Learning (48 min)
  

• Intrinsically motivated agents 3: Noisy-TV problem and search strategies in human and artificial agents (45 min, lectured by Alireza Modirshanechi)
  All parts - Noisy-TV Problem and Drawbacks of Curiosity-driven Reinforcement Learning

Recommended Textbook
Reinforcement Learning: an Introduction by R. Sutton and A. Barto (MIT Press)