AI at the Edge: Solving Real World Problems with Embedded Machine Learning

Foreword
Preface
    About this Book
    What to Expect
    What You Need to Know Already
    Responsible, Ethical, and Effective AI
    Further Resources
    Acknowledgements
    Conventions Used in This Book
    Using Code Examples
    O’Reilly Online Learning
    How to Contact Us
    Dedications
    Acknowledgments
1. A Brief Introduction to Edge AI
    Defining key terms
        Embedded
        The edge (and the Internet of Things)
        Artificial intelligence
        Machine learning
        Edge AI
        Embedded machine learning and tiny machine learning
        Digital signal processing
    Why do we need edge AI?
        To understand the benefits of edge AI, just BLERP
            Bandwidth
            Latency
            Economics
            Reliability
            Privacy
            Using BLERP
        Edge AI for good
        Key differences between edge AI and regular AI
            Training on the edge is rare
            The focus of edge AI is on sensor data
            ML models can get very small
            Learning from feedback is limited
            Compute is diverse and heterogeneous
            “Good enough” is often the goal
            Tools and best practices are still evolving
    Summary
2. Edge AI in the Real World
    Common Use Cases for Edge AI
        Green and brown field projects
        Real world products
            Preventing forest fires using power line fault detection
            Protecting first responders with intelligent wearables
            Understanding elephant behavior with smart collars
    Types of applications
        Keeping track of objects
            Key benefits for object tracking
        Understanding and controlling systems
            Key benefits for understanding and controlling systems
        Understanding people and living things
            Key benefits for understanding people and living things
        Transforming signals
            Key benefits for transforming signals
    Building applications responsibly
        Responsible design and AI ethics
        Black boxes and bias
        Technology that harms, not helps
            The costs of negligence
                Violations due to negligence
                Deliberate ethical violations
            Mitigating societal harms
    Summary
3. The Hardware of Edge AI
    Sensors, signals, and sources of data
        Types of sensors and signals
        Acoustic and vibration
        Visual and scene
        Motion and position
        Force and tactile
        Optical, electromagnetic, and radiation
        Environmental, biological, and chemical
        Other signals
    Processors for edge AI
        Edge AI hardware architecture
        Microcontrollers and digital signal processors
            Low-end MCUs
            High-end MCUs
            Digital signal processors (DSPs)
        System on Chip
        Deep learning accelerators
        FPGAs and ASICs
        Edge servers
        Multi-device architectures
        Devices and workloads
    Summary
4. Algorithms for Edge AI
    Feature engineering
        Working with data streams
        Digital signal processing algorithms
            Resampling
            Filtering
            Spectral analysis
            Image feature detection
        Combining features and sensors
    Artificial intelligence algorithms
        Algorithm types by functionality
            Classification
            Regression
            Object detection and segmentation
            Anomaly detection
            Clustering
            Dimensionality reduction
            Transformation
        Algorithm types by implementation
            Conditionals and heuristics
            Classical machine learning
            Deep learning
            Combining algorithms
            Post-processing algorithms
            Fail-safe design
        Optimization for edge devices
            Choice of algorithm
            Compression and optimization
        On-device training
    Summary
5. Tools and Expertise
    Building a team for AI at the edge
        Domain expertise
        Diversity
        Stakeholders
        Roles and responsibilities
            Knowledge and understanding
            Planning and execution
            Algorithm development
            Product engineering
            Technical services
        Hiring for edge AI
        Learning edge AI skills
            Practice
            Theory
    Tools of the trade
        Software engineering
            Operating systems
            Programming and scripting languages
            Dependency management
            Containerization
            Distributed computing
            Cloud providers
        Working with data
            Data capture
            IoT device management
            Data storage and management
            Data pipelines
        Algorithm development
            Mathematical and scientific computing libraries
            Data visualization
            Interactive computing environments
            Digital signal processing
            Deep learning frameworks
            Model compression and optimization
            Experiment tracking
            Automated machine learning (AutoML)
            Machine learning operations (MLOps)
        Running algorithms on-device
            Math and DSP libraries
            Machine learning inference
            On-device learning
        Embedded software engineering and electronics
            Embedded hardware tools
            Development boards
            Embedded software tools
            Emulators and simulators
            Embedded Linux
            Automated hardware testing
        End-to-end platforms for edge AI
    Summary
6. Understanding and Framing Problems
    The edge AI workflow
        Responsible AI in the edge AI workflow
    Do I need edge AI?
        Describing a problem
        Do I need to deploy to the edge?
            Things that don’t work well on the edge
            Disadvantages of edge compute
        Do I need machine learning?
            Reasons to use ML
            The drawbacks of ML
            Knowing when to use ML
        Practical exercise
    Determining feasibility
        Moral feasibility
        Business feasibility
            Proving benefit
            Understanding constraints
        Dataset feasibility
        Technological feasibility
            Framing problems
            Device capabilities and solution choice
        Making a final decision
        Planning an edge AI project
            Defining acceptable performance
            Understanding time and resource constraints
    Summary
7. How to Build a Dataset
    What does a dataset look like?
    The ideal dataset
    Datasets and domain expertise
    Data, ethics, and responsible AI
        Minimizing unknowns
        Ensuring domain expertise
    Data-centric machine learning
    Estimating data requirements
        A practical workflow for estimating data requirements
    Getting your hands on data
        The unique challenges of capturing data at the edge
    Storing and retrieving data
        Getting data into stores
        Collecting metadata
    Ensuring data quality
        Ensuring representative datasets
        Reviewing data by sampling
        Label noise
            Avoiding label noise
        Common data errors
        Drift and shift
        The uneven distribution of errors
    Preparing data
        Labelling
            Not all problems require labels
            Semi-supervised and active learning algorithms
            Bias in labelling
            Labelling tools
                Annotation tools
                Crowdsourced labelling
                Assisted and automated labelling
                Semi-supervised and active learning
        Formatting
        Data cleaning
            Auditing your dataset
            Fixing issues
                Amending values
                Substituting values
                Excluding records
            Evaluation and automation
            Fixing balance issues
        Feature engineering
        Splitting your data
            How is data split?
            Pitfalls when splitting data
        Data augmentation
        Data pipelines
    Building a dataset over time
    Summary
8. Designing Edge AI Applications
    Product and experience design
        Design principles
        Scoping a solution
        Setting design goals
            Systemic goals
            Technical goals
            Values-based design goals
    Architectural design
        Hardware, software, and services
        Basic application architectures
            Basic flow
            Ensemble flow
            Parallel flow
            Series flow
            Cascading flow
            Sensor fusion flow
        Complex application architectures and design patterns
            Heterogeneous cascade
            Multi-device cascade
            Cascade to the cloud
            Intelligent gateway
            Human-in-the-loop
        Working with design patterns
    Accounting for choices in design
        Design deliverables
    Summary
9. Developing Edge AI Applications
    An iterative workflow for edge AI development
        Exploration
        Goal-setting
            Calling it quits
        Bootstrapping
            Why bootstrapping is helpful
            Developing a baseline algorithm
            Our first hardware
            Responsible AI review
        Test and iterate
            Feedback loops
            Iterations in practice
                Updating your plans
            Ethical AI review
        Deployment
        Support
    Summary
10. Evaluating, Deploying, and Supporting Edge AI Applications
    Evaluating edge AI systems
        Ways to evaluate a system
            Evaluating individual components
            Evaluating integrated systems
            Simulated real-world testing
            Real-world testing
                Quality assurance testing
                Usability testing
            Monitoring a deployed system
        Useful metrics
            Algorithmic performance
                Loss
                Accuracy
                Confusion matrix
                Precision and recall
                Positive and negative rates
                F1 score and MCC
                ROC & AUC
                Error metrics
                Mean average precision
            Computational and hardware performance
                Memory
                Floating point operations
                Latency
                Duty cycle
                Energy
                Thermal
        Techniques for evaluation
        Evaluation and responsible AI
    Deploying edge AI applications
        Pre-deployment tasks
        Mid-deployment tasks
        Post-deployment tasks
    Supporting edge AI applications
        Post-deployment monitoring
            Types of feedback from deployed systems
                Data samples
                Distribution changes
                Application metrics
                Outcomes
                User reports
        Improving a live application
            Solving problems using feedback
            Refining an algorithm over time
            Supporting multiple deployed algorithms
        Ethics and long term support
            Performance degradation
            New information
            Evolving cultural norms
            Changing legal standards
    What comes next
11. Use Case: Wildlife Monitoring
    Problem exploration
    Solution exploration
    Goal-setting
    Solution design
        What solutions already exist?
        Solution design approaches
        Design considerations
        Environmental impact
        Bootstrapping
        Define your machine learning classes
    Dataset gathering
        Edge Impulse
            Public project
        Choose your hardware & sensors
            Hardware configuration
        Data collection
            Connecting your device directly to Edge Impulse for data collection
        iNaturalist
        Dataset limitations
        Dataset licensing and legal obligations
        Cleaning your dataset
        Uploading data to Edge Impulse
    DSP & machine learning workflow
        Digital signal processing block
        Machine learning block
            Visual mode
            EON Tuner
    Testing the model
        Live classification
        Model testing
        Test your model locally
    Deployment
        Create library
        Mobile phone & computer
        Pre-built binary flashing
        Impulse runnner
        GitHub source code
    Iterate & feedback loops
    AI for Good
    Related works
        Datasets
        Research
12. Use Case: Food Quality Assurance
    Problem exploration
    Solution exploration
    Goal-setting
    Solution design
        What solutions already exist?
        Solution design approaches
        Design considerations
        Environmental & social impact
        Bootstrapping
        Define your machine learning classes
    Dataset gathering
        Edge Impulse
            Edge Impulse public project
        Choose your hardware & sensors
            Hardware configuration
        Data collection
        Data ingestion firmware
        Uploading data to Edge Impulse
        Cleaning your dataset
        Dataset licensing and legal obligations
    DSP & machine learning workflow
        Digital signal processing block
        Machine learning block
            Visual mode
    Testing the model
        Live classification
        Model testing
    Deployment
        Pre-built binary flashing
        GitHub source code
    Iterate & feedback loops
    Related works
        Research
        News & other articles
13. Use Case: Consumer Products
    Problem exploration
    Goal-setting
    Solution design
        What solutions already exist?
        Solution design approaches
        Design considerations
        Environmental & social impact
        Bootstrapping
        Define your machine learning classes
    Dataset gathering
        Edge Impulse
            Edge Impulse public project
        Choose your hardware & sensors
            Hardware configuration
        Data collection
        Data ingestion firmware
            Mobile phone
            nRF Edge Impulse mobile phone application
        Cleaning your dataset
        Dataset licensing and legal obligations
    DSP & machine learning workflow
        Digital signal processing block
        Machine learning blocks
            Visual mode
            Anomaly detection
    Testing the model
        Live classification
        Model testing
    Deployment
        Pre-built binary flashing
        GitHub source code
    Iterate & feedback loops
    Related works
        Research
        News & other articles