Creating Augmented and Virtual Realities: Theory and Practice for Next-Generation Spatial Computing

Foreword
Preface
    A Note on Terminology
    Why We Wrote This Book
    What We Didn’t Cover in This Book
    How This Book Is Organized
    Conventions Used in This Book
    Using Code Examples
    O’Reilly Online Learning
    How to Contact Us
    Acknowledgments
I. Design and Art Across Digital Realities
1. How Humans Interact with Computers
    Common Term Definition
    Introduction
    Modalities Through the Ages: Pre-Twentieth Century
    Modalities Through the Ages: Through World War II
    Modalities Through the Ages: Post-World War II
    Modalities Through the Ages: The Rise of Personal Computing
    Modalities Through the Ages: Computer Miniaturization
    Why Did We Just Go Over All of This?
        Types of Common HCI Modalities
    New Modalities
    The Current State of Modalities for Spatial Computing Devices
    Current Controllers for Immersive Computing Systems
        Body Tracking Technologies
    A Note on Hand Tracking and Hand Pose Recognition
        Voice, Hands, and Hardware Inputs over the Next Generation
2. Designing for Our Senses, Not Our Devices
    Envisioning a Future
    Sensory Technology Explained
    So, Who Are We Building This Future For?
        The Role of Women in AI
    Sensory Design
        An Introduction
    Five Sensory Principles
        1. Intuitive Experiences Are Multisensory
        2. 3D Will Be Normcore
        3. Designs Become Physical Nature
        4. Design for the Uncontrollable
        5. Unlock the Power of Spatial Collaboration
    Adobe’s AR Story
    Conclusion
II. How eXtended Reality Is Changing Digital Art
3. Virtual Reality for Art
    A More Natural Way of Making 3D Art
    VR for Animation
4. 3D Art Optimization
    Introduction
        Options to Consider
        Ideal Solution
        Topology
        Baking
    Draw Calls
    Using VR Tools for Creating 3D Art
    Acquiring 3D Models Versus Making Them from Scratch
    Summary
III. Hardware, SLAM, Tracking
5. How the Computer Vision That Makes Augmented Reality Possible Works
    Who Are We?
    A Brief History of AR
    How and Why to Select an AR Platform
        I’m a Developer, What Platform Should I Use and Why?
        Performance Is Statistics
        Integrating Hardware and Software
        Optical Calibration
        Inertial Calibration
        The Future of Tracking
        The Future of AR Computer Vision
    Mapping
        How Does Multiplayer AR Work?
        What’s the Difficult Part?
        How Does Relocalization Work?
        What’s the State of the Art in Research (and Coming Soon to Consumer)?
        Can the Relocalization Problem Really Be Solved for Consumers?
        Can’t Google or Apple Just Do This?
        Relocalization != Multiplayer; It’s Also Critical for…
        How Is Relocalization Really Being Done Today in Apps?
    Platforms
        Apple’s ARKit
        Some Mysteries Explained
        Isn’t ARCore Just Tango-Lite?
        So, Should I Build on ARCore Now?
        What About Tango, Hololens, Vuforia, and Others?
    Other Development Considerations
        Lighting
        Multiplayer AR—Why It’s Quite Difficult
        How Do People Connect Through AR?
        How Do AR Apps Connect to the World and Know Where They Really Are?
        How Do AR Apps Understand and Connect to Things in the Real World?
    The AR Cloud
        What I Envision When I Think About the AR Cloud
        How Big a Deal Is the AR Cloud?
        So, Just What Is This AR Cloud?
        Why All the “Meh” from the Press for ARKit and ARCore?
        What’s Missing to Make a Great AR App?
        Is Today’s Mobile Cloud up to the Job?
        Is ARKit (or ARCore) Useless Without the AR Cloud?
        The Dawn of the AR Cloud
        The Bigger Picture—Privacy and AR Cloud Data
    Glossary
IV. Creating Cross-Platform Augmented Reality and Virtual Reality
6. Virtual Reality and Augmented Reality: Cross-Platform Theory
    Why Cross-Platform?
    The Role of Game Engines
    Understanding 3D Graphics
        The Virtual Camera
        Degrees of Freedom
    Portability Lessons from Video Game Design
    Simplifying the Controller Input
        Development Step 1: Designing the Base Interface
        Development Step 2: Platform Integration
    Summary
7. Virtual Reality Toolkit: Open Source Framework for the Community
    What Is VRTK and Why People Use It?
    The History of VRTK
    Welcome to the SteamVR Unity Toolkit
    VRTK v4
    The Future of VRTK
    The Success of VRTK
        Getting Started with VRTK 4
8. Three Virtual Reality and Augmented Reality Development Best Practices
    Developing for Virtual Reality and Augmented Reality Is Difficult
    Handling Locomotion
        Locomotion in VR
        Locomotion in AR
    Effective Use of Audio
        Audio in VR
        Audio in AR
    Common Interactions Paradigms
        Inventory for VR
        Augmented Reality Raycasts
    Conclusion
V. Enhancing Data Representation: Data Visualization and Artificial Intelligence in Spatial Computing
9. Data and Machine Learning Visualization Design and Development in Spatial Computing
    Introduction
    Understanding Data Visualization
    Principles for Data and Machine Learning Visualization in Spatial Computing
    Why Data and Machine Learning Visualization Works in Spatial Computing
        The Evolution of Data Visualization Design with the Emergence of XR
        2D and 3D Data Represented in XR
    2D Data Visualizations versus 3D Data Visualization in Spatial Computing
    Interactivity in Data Visualizations in Spatial Computing
    Animation
    Failures in Data Visualization Design
    Good Data Visualization Design Optimize 3D Spaces
    “Savings in Time Feel Like Simplicity”
    Data Representations, Infographics, and Interactions
        What Qualifies as Data Visualization?
    Defining Distinctions in Data Visualization and Big Data or Machine Learning Visualizations
    How to Create Data Visualization: Data Visualization Creation Pipeline
    WebXR: Building Data Visualizations for the Web
    Data Visualization Challenges in XR
    Data Visualization Industry Use Case Examples of Data Visualizations
    3D Reconstruction and Direct Manipulation of Real-World Data: Anatomical Structures in XR
        A Closer Look at Glass Brain
        TVA Surg Medical Imaging VR Module
    Data Visualization Is for Everyone: Open Source–Based Data Visualization in XR
        Protein Data Visualization
    Hands-On Tutorials: How to Create Data Visualization in Spatial Computing
    How to Create Data Visualization: Resources
    Conclusion
    References
    Resources
        Data Visualization Tools
        Machine Learning Visualization Tools
        Data Journalism Visualizations
    Glossary
10. Character AI and Behaviors
    Introduction
    Behaviors
    Current Practice: Reactive AI
        Adaptability
        Complexity and Universality
        Feasibility
    More Intelligence in the System: Deliberative AI
    Machine Learning
        Reinforcement Learning
        Deep Reinforcement Learning
        Imitation Learning
        Combining Automated Planning and Machine Learning
        Applications
    Conclusion
    References
VI. Use Cases in Embodied Reality
11. The Virtual and Augmented Reality Health Technology Ecosystem
    VR/AR Health Technology Application Design
    Standard UX Isn’t Intuitive
        Pick a Calm Environment
        Convenience
    Tutorial: Insight Parkinson’s Experiment
        What Insight Does
        How It Was Built
    Companies
        Planning and Guidance
        Experiences Designed for Medical Education
        Experiences Designed for Use by Patients
        Proactive Health
    Case Studies from Leading Academic Institutions
12. The Fan Experience: SportsXR
    Introduction
    Part 1: Five Key Principles of AR and VR for Sports
        Nothing Is Live
    Part 2: The Next Evolution of Sports Experiences
    Part 3: Making the Future
        Ownership
        Final Thought
    Conclusion
13. Virtual Reality Enterprise Training Use Cases
    Introduction: The Importance of Enterprise Training
    Does VR Training Work?
    Use Case: Flood House Training
    What Is VR Training Good for? R.I.D.E.
    What Makes Good VR Training?
    Spherical Video
        The Benefits of Spherical Video
        The Challenges of Spherical Video
        Interactions with Spherical Video
    Use Case: Factory Floor Training
    The Role of Narrative
    Use Case: Store Robbery Training
    The Future of XR Training: Beyond Spherical Video
        Computer Graphics
    Use Case: Soft Skills Training
    The Future: Photogrammetry
    The Future: Light Fields
    The Future: AR Training
    The Future: Voice Recognition
    The Future: The Ideal Training Scenario
    References
Afterword
Index