Tiny Machine Learning (COMPSCI 249R)

Tiny Machine Learning (COMPSCI 249R)

Fall 2023 | Harvard University

Vijay Janapa Reddi

Tiny machine learning (TinyML) is defined as a fast-growing field of machine learning technologies and applications including hardware (dedicated integrated circuits), algorithms and software capable of performing on-device sensor (vision, audio, IMU, biomedical, etc.) data analytics at extremely low power, typically in the mW range and below, and hence enabling a variety of always-on use-cases and targeting battery-operated devices. The pervasiveness of ultra-low-power embedded devices, coupled with the introduction of embedded machine learning frameworks like TensorFlow Lite for Microcontrollers, will enable the mass proliferation of AI-powered IoT devices. The explosive growth in machine learning and the ease of use of platforms like TensorFlow (TF) make it an indispensable topic of study for modern computer science and electrical engineering students.

General

Course Overview

Course Topics

The course provides a sweeping overview of machine learning systems, from foundational concepts like the stages of machine learning to advanced topics such as hardware acceleration and on-edge generative AI. This includes a journey through data engineering, optimized model frameworks, and sustainability dimensions of ML, all tailored to embedded environments.

Overview and Introduction to Embedded Machine Learning

Data Engineering

Embedded Machine Learning Frameworks

Efficient Model Representation and Compression

Performance Metrics and Benchmarking of ML Systems

Learning on the Edge

Hardware Acceleration for Edge ML: GPUs, TPUs and FPGAs

Embedded MLOps

Secure and Privacy-Preserving On-Device ML

Responsible AI

Sustainability at the Edge

Generative AI at the Edge

Textbook

The field of embedded machine learning systems is rapidly evolving. While there is a TinyML textbook available, it has become somewhat dated. We'll primarily rely on the latest academic publications for our studies. You may use the TinyML textbook as a reference material when needed. We will be drawing some of the content from the textbook titled “AI at the Edge,” which contains more up-to-date methods and examples. We can make an online version available to you via Canvas that cannot be distributed outside of class. However, we encourage students to get their own copy so that it is easier to follow and work through the material.

Open-source Textbook Website

Machine Learning Systems Textbook

Reading and Lecture Notes

Lecture 1: Overview and Introduction to TinyML

Lecture 2: Data Engineering

Required Readings

Models Cards for Model Reporting

Multilingual Spoken Words Corpus

Optional Readings

Datasheets for Datasets

Visual Wake Words Dataset

Lecture 3: Machine Learning Frameworks

Required Readings

TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems

MCUNet: Tiny Deep Learning on IoT Devices

Optional Readings

TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

CMSIS-NN: Efficient Neural Network Kernels for Arm Cortex-M CPUs

Lecture 4: Model Optimizations

Required Readings

A Survey of Quantization Methods for Efficient Neural Network Inference

The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Optional Readings

Lost In Pruning: The effects of pruning neural networks beyond test accuracy

Distilling the Knowledge in a Neural Network

What is the state of Neural Network Pruning?

RELEQ: A Reinforcement Learning Approach for Automatic Deep Quantization of Neural Networks

Lecture 5: Learning on the Edge

Required Readings

Flower: A Friendly Federated Learning Framework

On-Device Training Under 256KB Memory

Optional Readings

TinyTL: Reduce Memory, Not Parameters for Efficient On-Device Learning

TinyTrain: Deep Neural Network Training at the Extreme Edge

Lecture 6: Acceleration for ML: GPUs, TPUs and FPGAs

Required Readings

CFU Playground: Full-Stack Open-Source Framework for Tiny Machine Learning (tinyML) Acceleration on FPGAs

An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks

Optional Readings

In-Datacenter Performance Analysis of a Tensor Processing Unit

A systematic Methodology for Analysis of Deep Learning Hardware and Software Platforms

Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning

Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks

Lecture 7: MLOps

Required Readings

Hidden Technical Debt in Machine Learning

Optional Readings

Data Cascades in High-Stakes AI

Secure and Privacy-Preserving On-Device ML

Required Readings

Machine Learning Sensors

Security of Neural Networks from Hardware Perspective: A Survey and Beyond

Optional Readings

Robust Machine Learning Systems: Reliability and Security for Deep Neural Networks

On Safeguarding Privacy and Security in the Framework of Federated Learning

Lecture 9: Responsible AI

Required Readings

Characterizing Bias In Compressed Models

Adversarial Nibbler: A Data-Centric Challenge for Improving the Safety of Text-to-Image Models

Optional Readings

A Framework for Responsible Innovation

Dollar Street Dataset

Responsible AI-by-Design: A Pattern Collection for Designing Responsible AI Systems

Lecture 10: Sustainability at the Edge

Required Readings

Is TinyML Sustainable? Assessing the Environmental Impacts of Machine Learning on Microcontrollers

ACT: Designing Sustainable Computer Systems With An Architectural Carbon Model Tool

Optional Readings

Data Centers on Wheels: Emissions from Computing Onboard Autonomous Vehicles

Lecture 11: Generative AI at the Edge

Required Readings

SensorGAN: A Novel Data Recovery Approach for Wearable Human Activity Recognition

LoRA: Low-Rank Adaptation of Large Language Models

Optional Readings

Generative Agents: Interactive Simulacra of Human Behavior

Social Simulacra: Creating Populated Prototypes for Social Computing Systems

Planning with Large Language Models for Code Generation

Additional Lecture Note

Diffusion Models & Prompt Engineering

Assignments

Assignment 1: Setup Arduino Nicla Vision + Data Engineering for a Person Detection Transfer Learning Model

Assignment 2: Keyword Spotting on Arduino Nicla Vision + Data Preprocessing and Model Optimizations