This class meets Tuesday and Thursday from 10:30 - 11:50 AM in Packard, Room 101.

Teaching Assistant

Nathan Zhang
Office Hours TBA
Nandita Bhaskhar
Office Hours TBA

Schedule

Lecture

Date

Topic

Reading

Spatial Assignment

1

9/25/2018

Introduction,

Softwore 2.0

Role of hardware accelerators in post Dennard and Moore era

 

 

2

9/27/2018

Guest Lecture: Kian Katan

Classical ML algorithms: Regression, SVMs

(What is the building block?)

Is Dark silicon useful?
Hennessy Patterson Chapter 7.1-7.2

 

3

10/2/2018

Linear algebra fundamentals and accelerating linear algebra
BLAS operations
20th century techniques: Systolic arrays and MIMDs, CGRAs

Why Systolic Architectures?
Anatomy of high performance GEMM

Dark Memory

4

10/4/2018

Guest Lecture: Hadi Esmaeilzadeh

Tabla & Cosmic

TABLA

Codesign Tradeoffs

5

10/9/2018

David Koeplinger & Tian: Modeling neural networks with Spatial, Analyzing 
performance and energy with Spatial

Spatial
Aladdin

Linear Algebra
Accelerators

6

10/11/2018

Evaluating Performance, Energy efficiency, Parallelism, Locality,
Memory hierarchy, Roofline model

Real-World Architectures: Putting it into practice
Accelerating GEMM:
Custom, GPU, TPU1 architectures and their GEMM performance

Roofline Model

Google TPU
NVIDIA Tesla V100

7

10/16/2018

Neural networks: MLPs and CNNs Inference

Accelerating Inference for CNNs

Blocking and parallelism

DianNao, TPUs

Efficient Processing of DNNs

Systematic Approach to Blocking

8

10/18/2018

Guest Lecture: Yu-Hsin

CNN Inference

Eyeriss
Google TPU (see lecture 5)
Brooks' Book, Chapter 5

 

9

10/23/2018

Accelerating CNN cont.

 

High Performance Zero-Memory Overhead Direct Convolutions
Fast algorithms for convolution

10

10/25/2018

Guest Lecture: Robert Schreiber

 

 

CNN Inference
Accelerators

11

10/30/2018

Training 1: SGD, back propagation, statistical efficiency, batch size

Caterpillar
ScaleDeep
GraphCore Talk

 

12

11/1/2018

Guest Lecture: Boris Ginsburg

Generalization and Regularization of Training

 

Optimizing Gradient Descent
Large Batch Training of Convolutional Networks
Dense-Sparse-Dense

 

13

11/6/2018

Guest Lecture: Paulius Micikevicius

Low Precision of Training

HALP
EIE
GEMV + Project Proposals due 11/6
GEMM due 11/10

 

14

11/8/2018

Guest Lecture: Eric Chung Catapult, Brainwave

Catapult
Brainwave

 

15

11/13/2018

Guest Lecture: Cliff Young:

ML benchmarks MLPerf

DawnBench
MLPerf

Staff meetings with student project groups
CNN assignment due 11/17

 

16

11/15/2018

Scaling Training

Revisiting Small Batch Training for Neural Networks
Deep Learning At Supercomputer Scale
Deep Gradient Compression

 

-

11/20/2018

Thanksgiving

 

 

-

11/22/2018

Thanksgiving

 

 

17

11/27/2018

FPGAs and CGRAs, Plasticine

Plasticine

 

18

11/29/2018

Guest Lecture: Mikhail Smelyanskiv

AI at Facebook Datacenter Scale 

ML @ Facebook , Due Friday

 

19

12/4/2018

NIPS Keynote Preview

Programming Assignment Optimization and Performance

 

 

20

12/6/2018

No Lecture

 

 

Guest Lectures

Kian Katanforoosh, deeplearning.ai and Stanford University
From Machine Learning to Deep Learning: a computational transition
Thursday September 27, 2018

Hadi Esmaeilzadeh, UC San Diego
TABLA
Thursday October 4, 2018

Yu-Hsin Chen, MIT
Accelerating Inference for CNNs & Eyeriss
Thursday October 18, 2018

Robert Schreiber, Cerebras Systems
Understanding Numerical Errors
Thursday October 25, 2018

Boris Ginsburg, NVIDIA
Low Precision Training of DNNs
Thursday November 1, 2018

Paulius Micikevicius, NVIDIA
Low Precision Training
Tuesday November 6, 2018

Eric Chung, Microsoft Research
Real-Time AI at Cloud Scale with Project Brainwave
Thursday November 8, 2018

Cliff Young, Google
MLPerf
Tuesday November 13, 2018

Mikhail Smelyanskiy, Facebook
AI at Facebook Datacenter Scale
Thursday November 29, 2018

Reading list and other resources

Lecture slides

Basic information about deep learning

Cheat sheet – things that everyone needs to know

Blogs

Grading