IoT System Testing: An IoT Journey from Devices to Analytics and the Edge

1
    Contents
    About the Author
    About the Technical Reviewer
    Acknowledgments
2
    Part 1: Getting Started
978-1-4842-8276-2.webp
978-1-4842-8276-2_1
    Chapter 1: The Internet of Things, V&V, and Testing
        IoT at a Glance
        Understanding Chaos, Security, and Other Product Qualities
        Why Test IoT Beyond the Device Itself
        Agile vs. Traditional vs. “Who Cares About It” Development
        This Book’s Audience
        How to Use This Book
            Reference Standards, Books, and the Internet: Context Matters
            Valuable References to Start Your IoT Test Library
        Summary
        References
            Figure Reference
978-1-4842-8276-2_2
    Chapter 2: IoT Technology in Time and Space
        IoT at a Glance
        IoT Market Segments – Where Is IoT Now and in the Future?
        A Sampling of IoT Challenges in Development and Testing
        IoT Test Team Responses to the Development History of “Testing Is Dead”
            Examples of How the IoT Devices Can Impact Testers
        Summary
        References
            Figure References
978-1-4842-8276-2_3
    Chapter 3: Big Picture Lessons Learned in IoT Project Test Planning
        High-Level Issues at a Glance
            #1: Creating and Releasing Software
            #2: Understanding the IoT Lifecycle
            #3: Test Tools for IoT
            #4: Avoid Impacts of Rushing in Competition
            #5: Advanced IoT Device Challenges
            #6: Testing in the Complex World of IoT Systems and Systems of Systems
        Project Test Risks: What Are They for IoT?
        Getting Started with IoT Test Plans and Strategy
            Introducing IoT Verification, Validation, and Testing Concepts and Standards
                What Is IoT V&V in a Nutshell
        Summary
        References
            Figure Reference
978-1-4842-8276-2_4
    Chapter 4: Factors Driving IoT Testing/V&V Selection and Planning
        Factor 1: Using Integrity Levels to Drive V&V/Test Planning and Strategy
        Factor 2: Risk-Based DevOps and Product Maturity Testing
            Risk Analysis Process for Testers
        Factor 3: Organizational Ability Impacts on IoT Test Planning
            “Newbie” Companies – Level 1
            Companies with Experience Moving into IoT – Level 2
            Companies with Hardware (Electronics) Experience – Level 3
            Companies with Software Experience – Level 3
            Company with Hardware-Software Experience – Level 4
            Companies with Systems, Hardware, and Software Experience – Level 5
            Government Organization – No Level but a Special Case
            Experienced Companies with Only Consumer Product History – Level 2
                Highly Experienced Company Including an IoT History – Level 5+
        Factor 4: IoT Project Size and Complexity Impacts on Testing
            Composite Total Scoring Factors (If You Need to Do That)
            How IoT Projects Should Mix and Match Factors
            Combining IoT Factors for Better Test Planning
            IoT Test/V&V Cost Estimation
        Summary
        References
            Figure References
978-1-4842-8276-2_5
    Chapter 5: Beginner Keys for Starting IoT Test Planning
        IoT Key 1: Have a Ubiquitous User Interface (UI)
        IoT Key 2: Learning from Data Analytics
        IoT Key 3: Unique and Specialized Hardware Working to Be a System with the Software
        IoT Key 4: Level of V&V/Test Need for “Good Enough” IoT
        IoT Key 5: Remaining Agile
        IoT Key 6: Testing IoT, Systems, and Large/Complex Software (LCS)
        Summary
        References
978-1-4842-8276-2_6
    Chapter 6: IoT Test Plan: Strategy and Architecture Introductions
        Establishing the Mindset
            Additional Considerations
        Planning
            Good Enough IoT Software and Devices
        IoT Test Planning Basics
        General Test Planning Outlined by Organization Classification
            The Pure Startup, Single Device, and Small Team Who Are Trying to Stay Alive
            Mature Groups or Growing Teams Targeting Growth of Sales
            IoT Test for a System with More Devices
            IoT Test Planning for Procuring Organizations: Governments, Large Corporations, and Others
        Impact of AI, Data, and Analytics on IoT Test Planning
            Product and Development Lifecycle Impacts on Test Planning – DevOps and Agile
        Summary
        References
            Figure Reference
978-1-4842-8276-2_7
    Chapter 7: IoT Test Planning and Strategy for Hardware and Software
        Traditional Testing Overview
        Planning in the Agile/DevOps IoT Test Lifecycle
            What Should Development and Test Do During Agile and DevOps?
            Ops Team People Skilled in Agile Test Thinking
            Planning Using a Hybrid Agile Test Lifecycle
        IoT Test Plan Budgeting, Estimating, and Scheduling
            Estimating Schedules – A Brief Introduction to Support Cost Estimates
            Estimation of Testing Size
            Quality, Verification, Validation, and Testing
            Test and Quality
            Verification and Validation Activity
                Wrap Up Test Planning: Regression Test Cases in IoT
        IoT Test Planning: OTS Hardware and Software
            Management Activities of the Coil
            Engineering Using the Coil
            Software Dev-Test-Sec-Ops Team in the Coil
        IoT Test Planning: Security and Critical Quality Factors
            IoT Test Planning: Introduction to Software Test Architecture (STA)
        IoT Test Planning: IoT Strategy
            Strategy – The Basics
            IoT Test Strategy
            What Is a Checklist for Strategy Selection?
            What Is a Strategy for the Individual IoT Tester?
            IoT System V&V Planning: Start with a Combination of Test Strategies
            Hardware V&V Strategy (Chapters 8, 12, and 19)
            Software V&V Strategy
            Ops V&V Strategy
        Hardware Test/V&V Planning (IEEE 1012)
        Test Planning for Software Systems
        IoT System Integration Test Planning
        IoT System and System of Systems V&V/Test Planning – A Conceptual Overview
        Summary
        References
            Figure Reference
978-1-4842-8276-2_8
    Chapter 8: Planning for the IoT Tester on Environments and Testing Details
        Assuring the Test Environment
            Selecting the Right Test Environment
            Planning for Automation in IoT Tool Environments
            Data Analytics with Tools
        IoT Detailed Project Test Planning (After the Master Test Plans)
            Hardware Planning a Tester Should Know
            Detailed Software Testing
            System Test Plan Patterns a Tester Should Know
            Planning Individual Tests (What All Testers Should Do Daily)
        Test Planning from Operations to the End Product Life
            Test Operations (Ops) Impacts on Test Planning
            How Does Ops and Test Planning Change Over Time?
            System Maintenance, Security, and Retirement
            Ops Maintenance in Test Planning
            Planning Retirement and Disposal of an IoT System
            Testing Integration Factors
            Planning Test IoT Operations and Maintenance (O&M) with Data Analytics and the Edge
            Test Planning for Release Deployment
            Factory Production: A Very Brief Introduction to Testing IoT Hardware
                Test and Quality
                Evolving As You Go
                Minimum Activities Needed to Release an IoT Device into the Wild First Pass
            Risk and Opportunity Management on an IoT Test Project
            Last but Not Least, Test Documentation
        Summary
        References
978-1-4842-8276-2_9
    Chapter 9: System Engineering Concepts in IoT Test Planning
        Reviewing Basic Software Engineering (SE) Concepts
        Critical Test Enabling Engineering Support Processes
            SQA/QA
            SCM and Testing
                Software (Test) Measurements
                Measurement and Status Supporting Systems Engineering and Testing
            System Architecture and Design – Test Top-Level Support
            System Planning Trade Study – Decision Analysis
        Designing with Safety for IoT
        Hardware Design Considerations
        Software Design Considerations
        IoT System Integration
        Agile and DevOps Development Impacts
        Summary
        References
            Figure Reference
978-1-4842-8276-2_10
    Chapter 10: IoT Test Design: Frameworks, Techniques, Attacks, Patterns, and Tours
        Test and Heuristics
        Test Patterns
        Example 1: Planning Pattern for IoT [8, 9]
        Example 2: Mind Maps – Test Patterning Tool for Process Selection for IoT
        Example 3: Attacks for IoT – High-Level Test Design Pattern
        Example 4: Test Meta-design Pattern – Tours for IoT
            Applying the Specific Tours to Project Factors
        Example 5: Frameworks – The Top Level of Test Design Planning
        Summary
        References
978-1-4842-8276-2_11
    Chapter 11: Classic IoT V&V/Test Concepts, Techniques, and Practices
        Moving from Simple Testing to Providing Information on Quality
        Techniques, Practices, Levels, and Types of Testing to Apply to IoT
        Functional Test Design Techniques
            Requirements Based
            Boundary Value Analysis (BVA)
            Scenario (a.k.a. Use Case and Stories) Testing
            Syntax of IoT Command Structure Testing
        Exploratory Testing
        Structural Testing
        Industrial Test Practices
            Math-Based Testing for IoT System and Software
            Model-Based Development and Testing/V&V – A Highly Automated and Integrated Test Practice
        IoT Levels of Testing Related to Lifecycle Phases
        Summary
        References
978-1-4842-8276-2_12
    Chapter 12: Test Approaches and Quality Assessments for IoT Agile/DevOps
        Working Without a Formal IoT Test Plan (or Any Other Plans)
        Test Planning for the Agile Teams
        Agile Grows and Evolves (and Will Continue to Change)
        Agile Hardware
        Agile Security Testing
        Agile Quality V&V Assessment
        Summary
        References
            Figure References
978-1-4842-8276-2_13
    Chapter 13: IoT Software Security Test Attacks and Designs
        Before We Get Started – Security Data Points and Warnings
            Definitions Used in This Chapter
        Using Attack Surfaces in Security Test Planning
        Planning Security Tests: IoT System Level Inspection and Assessment
        Key Activity: Risk and Vulnerability Analysis
            IoT Security Threat Risks and How to Find Them
        Cybersecurity Team: Test/V&V Supports Developer-Operations (DevOps) Security
            Key Activity: Cyber Kill Chain Model
            Key Activity: Using a Zero-Trust Security Architecture Model to Support IoT Testing
            Key Activity: Software Security Testing Cycle
                Social Engineering
                Penetration (Pen) Testing
                Access Sensitive Information
                Access Control and Password Cracking
                Elevate Access Privileges
                Denial of Service (DoS) Attack
                Tampering Attack
                Malware Attacks
                Security Attacking the Off-the-Shelf (OTS) Software for Malware
                    Using a Table for Detection of Malware in the IoT Off-the-Shelf (OTS) Software
                Security Taint Analysis
                Spoofing [11]
                Edge, Fog, and Cloud Attack: Security Testing Needed beyond a Single IoT Device
        Historic Security Books You Should Have and Know
        Security Test Tool Categories
            Tools for Software Security Testing
            IoT Testing Tools for Hardware
            Software-Defined Security and AI – An IoT Future
            Security Finding Reporting
        Summary
        References
978-1-4842-8276-2_14
    Chapter 14: Security OWASP IoT Information Pointer and Logging Events
        Intro to OWASP Top Ten Threats (As of 2022)
        Summary
978-1-4842-8276-2_15
    Chapter 15: Internal Security Team Penetration Test Process
        Pen Test Process: A Beginning
            Why Perform Pen Test?
        Pen Security Attack and Risk-Based Test Planning for Systems
        Full-Scale Penetration of the System
        Simplified Process to Perform a Pen Test
        Summary
        References
978-1-4842-8276-2_16
    Chapter 16: IoT Test Environment Introduction
        Test Lab Lifecycle
        Test Lab Refresher
        IoT Lab/SIL Planning and Requirements
        Preferred Test Environment with Full Integration of a Complex IoT System
        The Field Test Environment, Analytics, and SIL Working Together
        Deep Dive on Test Environment Cost and Schedule Introduction
        Test Tool Introduction for IoT
        Test Hardware Setup for SIL-Chaos Engineering Support with ZIF Connectors
        Summary
        References
            Figure Reference
978-1-4842-8276-2_17
    Chapter 17: Architectures Critical to Project Success
        Cyber-Physical Systems
        Environments and Architectures for IoT
        Architecture Definition of Terms for This Book
        Historical Reference: Architecture in Engineering and Literature
        Summary
        References
978-1-4842-8276-2_18
    Chapter 18: Overview of IoT Software Architectures: Products and Testing Support
        A Quick Look at IoT Architectures
        Overview of IoT Support Architectures
        Mind Maps of IoT Environments
        IoT Software Test Architecture (STA) Introduction
        IoT Software Test Architecture (STA) Details
            Major Element: Test Plan (a.k.a. STAp)
            Major Element: System/Software Under Test (SUT)
            Major Element: Test Environment (STAe)
            Major Element: Test Model (STAm)
            Major Architecture Element: Views, Viewpoints, and Containers (STAv)
        Summary
        References
978-1-4842-8276-2_19
    Chapter 19: IoT STA System: Software Integration Lab (SIL) Environments
        Environment: Development Team Testing and Integration Support
        Environment: Hardware Team Testing and Integration
        Environment: Full Hardware-Software-System Integration
            A Simple Integration STA SIL
            A More Advanced IoT STA SIL with Rapid Integration Reconfiguration
        Test Environment Supports: Simulation, Modeling, and Emulation
            Full System IoT STA SIL with Simulation and Modeling
        Environment: Real-World Full System Software Test
        Special Environment: Security Test Sandbox
            IoT Chaos Engineering “Live” in the Real World
            Level of SIL Environment vs. Project Factors
            Bringing STA to a Large-Scale Software Test Architecture/Environment
        Summary
        References
            Figure Reference
978-1-4842-8276-2_20
    Chapter 20: Tools for the Software System Integration Lab (SIL)
        Test Lab Needs
        Modeling and Requirements Management Tools
        Automate, Automate, Automate
        Simulation Test Tools Needed to Support IoT
            Evolve into Automation Tooling for IoT Success
            Evolve IoT Testing with AI Tools
        Summary
        References
978-1-4842-8276-2_21
    Chapter 21: Environments for Independent Testing and IV&V on Large IoT Systems
        Getting the Most Out of Independent Testing and IV&V
            Verification
            Validation
        N-Version Testing Supporting Independence and IV&V
        Lessons Learned in Testing, in Independence, and IV&V
        Tooling Example for Testing Large IoT Systems and IV&V
        When to Consider IoT Test Independence and IV&V Environments
        Summary
        References
978-1-4842-8276-2_22
    Chapter 22: Self-Organizing Data Analytics (SODA): IoT Data Analytics, AI, and Statistics
        SODA Model Examined
        Defining SODA
        SODA Implementation Options Using Stats, Taxonomies, and AI
        Use Case Example for SODA and Edge Data Flow
        Leveraging SODA for IoT Testing
        AI SODA – A Near Future for IoT/Edge/Cloud Data Analytics
        Summary
        References
978-1-4842-8276-2_23
    IoT Supporting Interface, Hardware, Platform, and Protocol Standards
    Careers in IoT Testing
    IoT Testing Startup Checklist
    Getting Started with IoT Testing
        Usability Testing
    Compatibility and Integration Testing
    App Localization Issue
    External Connectivity Issues
    Interrupt Testing
    Operational Testing
    Installation Test
    Miscellaneous
    Example of an IoT System of Systems
    Index