This class meets Tuesday and Thursday from 10:30 - 11:50 AM in Huang Engineering Center 18.

Teaching Assistants

Nathan Zhang
Office Hours TBA

Class Information

Funding for this research/activity was partially provided by the National Science Foundation Division of Computing and Communication Foundations under award number 1563113.

Schedule

Lecture

Date

Topic

Reading Assignments Lecture Slides

Spatial Assignment

1

1/07/2020

Introduction,

Software 2.0

Role of hardware accelerators in post Dennard and Moore era

 

 

2

1/09/2020

Kian Katan:Classical ML algorithms: Regression, SVMs

Is Dark silicon useful?
Hennessy Patterson Chapter 7.1-7.2

 

3

1/14/2020

Linear algebra fundamentals and accelerating linear algebra
BLAS operations
20th century techniques: Systolic arrays and MIMDs, CGRAs

Why Systolic Architectures?
Anatomy of high performance GEMM

Dark Memory

Linear Algebra
Accelerators

4

1/16/2020

Introduction to Spatial: Analyzing Performance and Energy with Spatial

Spatial
Aladdin

Codesign Tradeoffs

5

1/21/2020

MLPs and CNNs Inference

Efficient Processing of DNNs

NVIDIA Tesla V100

6

1/23/2020

Evaluating Performance, Energy efficiency, Parallelism, Locality,
Memory hierarchy, Roofline model

Luigi Nardi: Design Space Optimization with Spatial

Roofline Model

Google TPU

7

1/28/2020

Boris Ginsburg:Generalization and Regularization of Training

 

Caterpillar
Optimizing Gradient Descent

 

8

1/30/2020

Azalia Mirhosseini: Reinforcement Learning for Hardware Design

A Beginner's Guide to RL
Resource Management w DRL

9

2/04/2020

Fanny Nina Paravecino: Catapult Brainwave

Catapult
Brainwave

 

10

2/06/2018

Amir Gholami: Quantized Deep Learning

 

SqueezNext

CNN Inference
Accelerators

11

2/11/2020

Tze Meng Low: Fast Implementation of Deep Learning Kernels

 

Systematic Approach to Blocking

High Performance Zero-Memory Overhead Direct Convolutions

12

2/13/2020

Guest Lecture: Paulius Micikevicius

GPU Design Tradeoffs for Deeplearning and MLPerf

Nvidia Volta

Mixed Precision Training Nvidia
Mixed Precision Training With 8-bit Floating Point

13

2/18/2020

Guest Lecture: Cliff Young

Neural Networks Have Rebooting Computing: What Should We Reboot Next?

DawnBench
MLPerf

14

2/20/2020

Mohammad Shoeybi

Accelerating Natural Language Processing

GNMT
BERT

 

15

2/25/2020

Mikhail Smelyanskiv:AI at Facebook Datacenter Scale 

ML @ Facebook

 

16

2/27/2020

Assignment 1 Feedback and Discussion

Midterm

17

3/03/2020

Boris Ginsburg: Large Scale Training

Revisiting Small Batch Training for Neural Networks
Large Batch Training of Convolutional Networks
Deep Learning At Supercomputer Scale
Deep Gradient Compression

 

18

3/05/2020

Sparsity in Deep Learning

EIE
Campfire

 

19

3/10/2020

Machine Learning Systems and Software Stack

Taxonomy of Accelerator Architectures
ML Systems Stuck in a Rut

 

 

20

3/12/2020

TBD

 

 

Guest Lectures

Kian Katanforoosh, deeplearning.ai and Stanford University
From Machine Learning to Deep Learning: a computational transition
Thursday January 9, 2020

Luigi Nardi, Lund University and Stanford University
Design Space Optimization with Spatial
Thursday January 23, 2020

Boris Ginsburg, NVIDIA
Generalization and Regularization of Training
Tuesday January 28, 2018

Azalia MirHosseini, Google Brain
Reinforcement Learning and Hardware Design
Thursday January 30, 2020

Fanny Nina Paravecino, Microsoft Research
Real-Time AI at Cloud Scale with Project Brainwave
Tuesday February 4, 2020

Amir Gholami, UC Berkeley
Precision and Quantized Training for Deep Learning
Thursday February 6, 2020

Tze Meng Low, Carnegie Melon University
Fast Implementation of Deep Learning Kernels
Tuesday February 11, 2020

Paulius Micikevicius, NVIDIA
GPU Design Tradeoffs for Deeplearning and MLPerf
Thursday February 13, 2020

Cliff Young, Google
Neural Networks Have Rebooting Computing: What Should We Reboot Next?
Tuesday February 18, 2020

Mohammad Shoeybi, NVIDIA
Natural Language Processing
Thursday February 20, 2020

Mikhail Smelyanskiy, Facebook
AI at Facebook Datacenter Scale
Tuesday February 25, 2020

Boris Ginsburg, NVIDIA
Large Batch Training of Convolution Networks
Tuesday March 3, 2020

Lecture Notes (Fall 2018)

Reading list and other resources

Basic information about deep learning

Cheat sheet – things that everyone needs to know

Blogs

Grading