Python for Data Analysis: Data Wrangling with pandas, NumPy, and Jupyter, 3rd Edition

Preface
    1. Conventions Used in This Book
    2. Using Code Examples
    3. O’Reilly Online Learning
    4. How to Contact Us
    5. Acknowledgments
        In Memoriam: John D. Hunter (1968–2012)
        Acknowledgments for the Third Edition (2022)
        Acknowledgments for the Second Edition (2017)
        Acknowledgments for the First Edition (2012)
1. Preliminaries
    1.1. What Is This Book About?
        What Kinds of Data?
    1.2. Why Python for Data Analysis?
        Python as Glue
        Solving the “Two-Language” Problem
        Why Not Python?
    1.3. Essential Python Libraries
        NumPy
        pandas
        matplotlib
        IPython and Jupyter
        SciPy
        scikit-learn
        statsmodels
        Other Packages
    1.4. Installation and Setup
        Miniconda on Windows
        GNU/Linux
        Miniconda on macOS
        Installing Necessary Packages
        Integrated Development Environments and Text       Editors
    1.5. Community and Conferences
    1.6. Navigating This Book
        Code Examples
        Data for Examples
        Import Conventions
2. Python Language Basics, IPython, and Jupyter Notebooks
    2.1. The Python Interpreter
    2.2. IPython Basics
        Running the IPython Shell
        Running the Jupyter Notebook
        Tab Completion
        Introspection
    2.3. Python Language Basics
        Language Semantics
            Indentation, not braces
            Everything is an object
            Comments
            Function and object method calls
            Variables and argument passing
            Dynamic references, strong types
            Attributes and methods
            Duck typing
            Imports
            Binary operators and comparisons
            Mutable and immutable objects
        Scalar Types
            Numeric types
            Strings
            Bytes and Unicode
            Booleans
            Type casting
            None
            Dates and times
        Control Flow
            if, elif, and else
            for loops
            while loops
            pass
            range
    2.4. Conclusion
3. Built-In Data Structures, Functions, and Files
    3.1. Data Structures and Sequences
        Tuple
            Unpacking tuples
            Tuple methods
        List
            Adding and removing elements
            Concatenating and combining lists
            Sorting
            Slicing
        Dictionary
            Creating dictionaries from sequences
            Default values
            Valid dictionary key types
        Set
        Built-In Sequence Functions
            enumerate
            sorted
            zip
            reversed
        List, Set, and Dictionary Comprehensions
            Nested list comprehensions
    3.2. Functions
        Namespaces, Scope, and Local Functions
        Returning Multiple Values
        Functions Are Objects
        Anonymous (Lambda) Functions
        Generators
            Generator expressions
            itertools module
        Errors and Exception Handling
            Exceptions in IPython
    3.3. Files and the Operating System
        Bytes and Unicode with Files
    3.4. Conclusion
4. NumPy Basics: Arrays and Vectorized   Computation
    4.1. The NumPy ndarray: A Multidimensional Array Object
        Creating ndarrays
        Data Types for ndarrays
        Arithmetic with NumPy Arrays
        Basic Indexing and Slicing
            Indexing with slices
        Boolean Indexing
        Fancy Indexing
        Transposing Arrays and Swapping Axes
    4.2. Pseudorandom Number Generation
    4.3. Universal Functions: Fast Element-Wise Array Functions
    4.4. Array-Oriented Programming with Arrays
        Expressing Conditional Logic as Array Operations
        Mathematical and Statistical Methods
        Methods for Boolean Arrays
        Sorting
        Unique and Other Set Logic
    4.5. File Input and Output with Arrays
    4.6. Linear Algebra
    4.7. Example: Random Walks
        Simulating Many Random Walks at Once
    4.8. Conclusion
5. Getting Started with pandas
    5.1. Introduction to pandas Data Structures
        Series
        DataFrame
        Index Objects
    5.2. Essential Functionality
        Reindexing
        Dropping Entries from an Axis
        Indexing, Selection, and Filtering
            Selection on DataFrame with loc and iloc
            Integer indexing pitfalls
            Pitfalls with chained indexing
        Arithmetic and Data Alignment
            Arithmetic methods with fill values
            Operations between DataFrame and Series
        Function Application and Mapping
        Sorting and Ranking
        Axis Indexes with Duplicate Labels
    5.3. Summarizing and Computing Descriptive Statistics
        Correlation and Covariance
        Unique Values, Value Counts, and Membership
    5.4. Conclusion
6. Data Loading, Storage, and File   Formats
    6.1. Reading and Writing Data in Text Format
        Reading Text Files in Pieces
        Writing Data to Text Format
        Working with Other Delimited Formats
        JSON Data
        XML and HTML: Web Scraping
            Parsing XML with lxml.objectify
    6.2. Binary Data Formats
        Reading Microsoft Excel Files
        Using HDF5 Format
    6.3. Interacting with Web APIs
    6.4. Interacting with Databases
    6.5. Conclusion
7. Data Cleaning and Preparation
    7.1. Handling Missing Data
        Filtering Out Missing Data
        Filling In Missing Data
    7.2. Data Transformation
        Removing Duplicates
        Transforming Data Using a Function or Mapping
        Replacing Values
        Renaming Axis Indexes
        Discretization and Binning
        Detecting and Filtering Outliers
        Permutation and Random Sampling
        Computing Indicator/Dummy Variables
    7.3. Extension Data Types
    7.4. String Manipulation
        Python Built-In String Object Methods
        Regular Expressions
        String Functions in pandas
    7.5. Categorical Data
        Background and Motivation
        Categorical Extension Type in pandas
        Computations with Categoricals
            Better performance with categoricals
        Categorical Methods
            Creating dummy variables for modeling
    7.6. Conclusion
8. Data Wrangling: Join, Combine, and Reshape
    8.1. Hierarchical Indexing
        Reordering and Sorting Levels
        Summary Statistics by Level
        Indexing with a DataFrame’s columns
    8.2. Combining and Merging Datasets
        Database-Style DataFrame Joins
        Merging on Index
        Concatenating Along an Axis
        Combining Data with Overlap
    8.3. Reshaping and Pivoting
        Reshaping with Hierarchical Indexing
        Pivoting “Long” to “Wide” Format
        Pivoting “Wide” to “Long” Format
    8.4. Conclusion
9. Plotting and Visualization
    9.1. A Brief matplotlib API Primer
        Figures and Subplots
            Adjusting the spacing around subplots
        Colors, Markers, and Line Styles
        Ticks, Labels, and Legends
            Setting the title, axis labels, ticks, and tick labels
            Adding legends
        Annotations and Drawing on a Subplot
        Saving Plots to File
        matplotlib Configuration
    9.2. Plotting with pandas and seaborn
        Line Plots
        Bar Plots
        Histograms and Density Plots
        Scatter or Point Plots
        Facet Grids and Categorical Data
    9.3. Other Python Visualization Tools
    9.4. Conclusion
10. Data Aggregation and Group   Operations
    10.1. How to Think About Group Operations
        Iterating over Groups
        Selecting a Column or Subset of Columns
        Grouping with Dictionaries and Series
        Grouping with Functions
        Grouping by Index Levels
    10.2. Data Aggregation
        Column-Wise and Multiple Function Application
        Returning Aggregated Data Without Row Indexes
    10.3. Apply: General split-apply-combine
        Suppressing the Group Keys
        Quantile and Bucket Analysis
        Example: Filling Missing Values with Group-Specific       Values
        Example: Random Sampling and Permutation
        Example: Group Weighted Average and Correlation
        Example: Group-Wise Linear Regression
    10.4. Group Transforms and “Unwrapped” GroupBys
    10.5. Pivot Tables and Cross-Tabulation
        Cross-Tabulations: Crosstab
    10.6. Conclusion
11. Time Series
    11.1. Date and Time Data Types and Tools
        Converting Between String and Datetime
    11.2. Time Series Basics
        Indexing, Selection, Subsetting
        Time Series with Duplicate Indices
    11.3. Date Ranges, Frequencies, and Shifting
        Generating Date Ranges
        Frequencies and Date Offsets
            Week of month dates
        Shifting (Leading and Lagging) Data
            Shifting dates with offsets
    11.4. Time Zone Handling
        Time Zone Localization and Conversion
        Operations with Time Zone-Aware Timestamp Objects
        Operations Between Different Time Zones
    11.5. Periods and Period Arithmetic
        Period Frequency Conversion
        Quarterly Period Frequencies
        Converting Timestamps to Periods (and Back)
        Creating a PeriodIndex from Arrays
    11.6. Resampling and Frequency Conversion
        Downsampling
            Open-high-low-close (OHLC) resampling
        Upsampling and Interpolation
        Resampling with Periods
        Grouped Time Resampling
    11.7. Moving Window Functions
        Exponentially Weighted Functions
        Binary Moving Window Functions
        User-Defined Moving Window Functions
    11.8. Conclusion
12. Introduction to Modeling Libraries in   Python
    12.1. Interfacing Between pandas and Model Code
    12.2. Creating Model Descriptions with Patsy
        Data Transformations in Patsy Formulas
        Categorical Data and Patsy
    12.3. Introduction to statsmodels
        Estimating Linear Models
        Estimating Time Series Processes
    12.4. Introduction to scikit-learn
    12.5. Conclusion
13. Data Analysis Examples
    13.1. Bitly Data from 1.USA.gov
        Counting Time Zones in Pure Python
        Counting Time Zones with pandas
    13.2. MovieLens 1M Dataset
        Measuring Rating Disagreement
    13.3. US Baby Names 1880–2010
        Analyzing Naming Trends
            Measuring the increase in naming diversity
            The “last letter” revolution
            Boy names that became girl names (and vice versa)
    13.4. USDA Food Database
    13.5. 2012 Federal Election Commission Database
        Donation Statistics by Occupation and Employer
        Bucketing Donation Amounts
        Donation Statistics by State
    13.6. Conclusion
A. Advanced NumPy
    A.1. ndarray Object Internals
        NumPy Data Type Hierarchy
    A.2. Advanced Array Manipulation
        Reshaping Arrays
        C Versus FORTRAN Order
        Concatenating and Splitting Arrays
            Stacking helpers: r_ and c_
        Repeating Elements: tile and repeat
        Fancy Indexing Equivalents: take and put
    A.3. Broadcasting
        Broadcasting over Other Axes
        Setting Array Values by Broadcasting
    A.4. Advanced ufunc Usage
        ufunc Instance Methods
        Writing New ufuncs in Python
    A.5. Structured and Record Arrays
        Nested Data Types and Multidimensional Fields
        Why Use Structured Arrays?
    A.6. More About Sorting
        Indirect Sorts: argsort and lexsort
        Alternative Sort Algorithms
        Partially Sorting Arrays
        numpy.searchsorted: Finding Elements in a Sorted Array
    A.7. Writing Fast NumPy Functions with Numba
        Creating Custom numpy.ufunc Objects with Numba
    A.8. Advanced Array Input and Output
        Memory-Mapped Files
        HDF5 and Other Array Storage Options
    A.9. Performance Tips
        The Importance of Contiguous Memory
B. More on the IPython System
    B.1. Terminal Keyboard Shortcuts
    B.2. About Magic Commands
        The %run Command
            Interrupting running code
        Executing Code from the Clipboard
    B.3. Using the Command History
        Searching and Reusing the Command History
        Input and Output Variables
    B.4. Interacting with the Operating System
        Shell Commands and Aliases
        Directory Bookmark System
    B.5. Software Development Tools
        Interactive Debugger
            Other ways to use the debugger
        Timing Code: %time and %timeit
        Basic Profiling: %prun and %run -p
        Profiling a Function Line by Line
    B.6. Tips for Productive Code Development Using IPython
        Reloading Module Dependencies
        Code Design Tips
            Keep relevant objects and data alive
            Flat is better than nested
            Overcome a fear of longer files
    B.7. Advanced IPython Features
        Profiles and Configuration
    B.8. Conclusion
Index