Course description
Understanding human intelligence and how to replicate it in machines is arguably one of the greatest problems in science. Learning, its principles and computational implementations, is at the very core of intelligence. During the last two decades, for the first time, artificial intelligence systems have been developed that begin to solve complex tasks, until recently the exclusive domain of biological organisms, such as computer vision, speech recognition or natural language understanding: cameras recognize faces, smart phones understand voice commands, smart speakers/assistants answer questions and cars can see and avoid obstacles. The machine learning algorithms that are at the roots of these success stories are trained with examples rather than programmed to solve a task. However, a comprehensive theory of learning is still incomplete, as shown by the puzzles of deep learning. An eventual theory of learning that explains why and how deep networks work and what their limitations are, may thus enable the development of even much more powerful learning approaches and even inform our understanding of human intelligence.
In this spirit, the course covers foundations and recent advances in statistical machine learning theory, with the dual goal a) of providing students with the theoretical knowledge and the intuitions needed to use effective machine learning solutions and b) to prepare more advanced students to contribute to progress in the field. This year the emphasis is again on b).
The course is organized about the core idea of supervised learning as an inverse problem, with stability as the key property required for good generalization performance of an algorithm.
The content is roughly divided into three parts. The first part is about classical regularization (regularized least squares, kernel machines, SVM, logistic regression, square and exponential loss) large margin/minimum norm, stochastic gradient methods, overparametrization, implicit regularization and also approximation/estimation errors. The second part is about deep networks: approximation theory -- which functions can be represented more efficiently by deep networks than shallow networks -- optimization theory -- why can stochastic gradient descent easily find global minima -- and estimation error -- how generalization in deep networks can be explained in terms of the stability implied by the complexity control implicit in gradient descent. The third part is about the connections between learning theory and the brain, which was the original inspiration for modern networks and may provide ideas for future developments and breakthroughs in the theory and the algorithms of leaning. Throughout the course, and especially in the final classes, we will have occasional talks by leading researchers on selected advanced research topics. This class is the first step of a NSF funded project in which a team of deep learning researchers from GeorgiaTech (Vempala), Columbia (Papadimitriou, Blei), Princeton (Hazan) and MIT (Madry, Daskalakis, Jegelka, Poggio) plans to create over the next 3 years courses that leverages all the recent advances in our understanding of deep learning as well and crystallizes the corresponding “modern” outlook on the field. Eventually it will include 1) deep learning: representations and generalization; (2) deep learning from an optimization perspective; (3) robustness and reliability in machine learning; (4) the brain connection; (5) (robust) reinforcement learning; and (6) societal impact of machine learning, in addition to laying out the theoretical foundations of the field.
Apart for the first part on regularization, which is an essential part of any introduction to the field of machine learning, this year course is designed for students with a background in ML to foster conjectures and exploratory projects on ongoing research.
Prerequisites
We will make extensive use of basic notions of calculus, linear algebra and probability. The essentials are covered in class and in the math camp material. We will introduce a few concepts in functional/convex analysis and optimization. Note that this is an advanced graduate course and some exposure on introductory Machine Learning concepts or courses is expected: for course 6 students prerequisites are 6.041 and 18.06 and (6.036 or 6.401 or 6.867). Students are also expected to have basic familiarity with MATLAB/Octave.
Grading
Requirements for grading are attending lectures/participation (10%), three problem sets (45%) and a final project (45%).
Classes will be conducted in-person this year (Fall 2021), until MIT policy changes.
Grading policies, pset and project tentative dates: (slides)
Problem Sets
Problem Set 1, out: Tue. Sept. 21, due: Sun. Oct. 03
Problem Set 2, out: Tue. Oct. 12, due: Sun. Oct. 24
Problem Set 3, out: Tue. Nov. 02, due: Sun. Nov. 14
Submission instructions: Follow the instructions included with the problem set. Use the latex template for the report. Submit your report online through canvas by the due date/time.
Projects
Guidelines and key dates. Online form for project proposal (complete by Thu. Oct. 28).
Final project reports (5 pages for individuals, 8 pages for teams, NeurIPS style) are due on Thu. Dec. 9.
Projects archive
List of Wikipedia entries, created or edited as part of projects during previous course offerings.
Navigating Student Resources at MIT
This document has more information about navigating student resources at MIT