Learning-based Control for Robotics

EN.530.696: Learning-Based Control for Robotics

Course Description:

Model-based methods provide a powerful framework for controlling challenging robotic systems; however, imperfect models often lead to poor performance during real-world deployment. Machine learning methods provide one means of addressing deficient models, either through explicitly learning a model of the system dynamics or computing a control policy directly from data. In this course, we will explore the intersection between optimal control and machine learning, covering both model-free and model-based methods for learning-based control. We first will start with a review of dynamic programming and other model-based approaches to controller synthesis. We will then explore the three primary means of incorporating learning into controller designs: learning value functions, control policies, and dynamics models. The course will culminate in a discussion of model-based reinforcement learning and adaptive optimal control. We will also discuss advanced topics such as learning Lyapunov functions and contraction metrics from data, iterative learning control, and techniques for adaptive nonlinear model predictive control.

Recommended Prerequisites:

Prior introductory graduate-level course in optimal control, reinforcement learning, or nonlinear control.

Grading:

40% Problem Sets

30% Midterm

30% Final Project

“Reinforcement Learning and Optimal Control,” by Dimitri Bertsekas
“Underactuated Robotics,” by Russ Tedrake
“Modern Adaptive Control and Reinforcement Learning,” by Byron Boots and Drew Bagnell
“Reinforcement Learning: An Introduction,” by Andrew Barto and Richard S. Sutton
“Deep Learning,” by Ian Goodfellow and Yoshua Bengio and Aaron Courville
“Applied Nonlinear Control,” by Jean-Jacques E. Slotine and Weiping Li.
“Optimal Control and Estimation” by Robert F. Stengel.

Course Schedule:

Date	Topics	Readings
Week 1	Dynamic Programming	Lectures 1 & 2
	Value Iteration and Policy Iteration	Lecture 3; Bertsekas 4
Week 2	LQR	Lecture 4; Tedrake 7,8
	Iterative LQR	Lecture 5; [1]
Week 3	Lyapunov-based Design	Lecture 6; Tedrake 9; [2]
	Contraction-based Design	Lecture 7; [3], [4]
Week 4	Model-Based Policy Search (Gradient-based)	Lecture 8; Bertsekas 2.2-2.5
	Model Based Policy Search (Gradient-free)	Lecture 9; [5], [6]
Week 5	Linear Function Approximation	Lecture 10; Bertsekas 3, Tedrake 18
	Nonlinear Function Approximation	Lecture 11; Bertsekas 3
Week 6	Offline Approximate Dynamic Programming	Lecture 14; Bertsekas 2.1, 5.1, 5.2; [7]
	Online Approximate Dynamic Programming	Lecture 15; Bertsekas 5.4 & 5.5
Week 7	Learning Lyapunov Functions	Lecture 16; [8]
	Learning Contraction Metrics	Lecture 17; [9]
Week 8	Midterm

Week 9	Spring Break

Week 10	Model-Free Policy Search	Lecture 18; Bertsekas 5.7, Tedrake 20; [10]
	Guest Lecture
Week 11	Advanced Nonlinear Function Approximation	Lectures 12 & 13; Goodfellow et al. 6
	Model-Based Reinforcement Learning	Lectures 19; Bertsekas 5.3; [11], [12], [13]
Week 12	Actor-Critic Methods	Lecture 20; [14], [15]
	Guest Lecture
Week 13	Learning with Perception (Part I)	Lecture 21; [16]
	Learning with Perception (Part II)	Lecture 22
Week 14	Adaptive Control	Lecture 23
	Learning-based Model Predictive Control	Lecture 24

Problem Sets:

There will be 5 Problem Sets for this course.

Problem Sets will be due approximately every two weeks.

All Problem Sets must be typeset in Latex.

Collaboration:

Collaboration on assignments is permitted, however, everyone is responsible for submitting their own work.

This means you must understand completely everything you submit and write down/code your own solutions.

If you collaborate substantially on a problem set, include the names of your collaborators on your submission.

Late Assignments:

Everyone will be allowed a single one-week extension.

10% will be deducted for a late problem set and an additional 10% will be deducted every 24 hrs thereafter.

AI Policy

AI can be a useful tool for learning. However, it can also be used in a way that is unethical and/or harmful to learning.

In this class, you should feel free to ask AI general questions about concepts to clarify your understanding. However, be aware that AI often gives subtly incorrect answers and can lead an incorrect overall understanding of the material. You should cross check EVERYTHING AI provides with peer-reviewed textbooks and/or publications.

It is also considered unethical, for this class, to submit homework questions in any form directly to AI. This is viewed as cheating. Keep in mind, that you will not have access to AI on the exam.

Final Project

The final project is an opportunity to more deeply explore learning-based control applied to a robotics problem of your choice. The problem must involve a dynamical robotic system where learning is necessary to achieve the required performance. The project is to be done as an individual. An initial project proposal, as well as a final report (typeset in Latex) will be required. There will also be a final presentation to be scheduled during the final exam period. You are free to leverage problems and applications relevant to your graduate research. However, you may NOT recycle your previous research results. Your project idea should be conceived following the start of this class. It is fine to implement/investigate methods found in the literature, so long as you explore/extend them in some new way. All projects will, at the very least, include a simulation component. Hardware experiments will be viewed very favorably.

Acknowledgements:

The assignments and lectures for this course are partially adapted with permission from Russ Tedrake’s course (https://underactuated.csail.mit.edu/) at MIT and Byron Boots’ course (https://courses.cs.washington.edu/courses/cse579/22au/) at University of Washington.