Data Science Bookcamp: Five real-world Python projects

Data Science Bookcamp
brief contents
contents
preface
acknowledgments
about this book
	Who should read this book
	How this book is organized
	About the code
about the author
about the cover illustration
Case study 1—Finding the winning strategy in a card game
	Section 1—Computing probabilities using Python
		1.1 Sample space analysis: An equation-free approach for measuring uncertainty in outcomes
			1.1.1 Analyzing a biased coin
		1.2 Computing nontrivial probabilities
			1.2.1 Problem 1: Analyzing a family with four children
			1.2.2 Problem 2: Analyzing multiple die rolls
			1.2.3 Problem 3: Computing die-roll probabilities using weighted sample spaces
		1.3 Computing probabilities over interval ranges
			1.3.1 Evaluating extremes using interval analysis
		Summary
	Section 2—Plotting probabilities using Matplotlib
		2.1 Basic Matplotlib plots
		2.2 Plotting coin-flip probabilities
			2.2.1 Comparing multiple coin-flip probability distributions
		Summary
	Section 3—Running random simulations in NumPy
		3.1 Simulating random coin flips and die rolls using NumPy
			3.1.1 Analyzing biased coin flips
		3.2 Computing confidence intervals using histograms and NumPy arrays
			3.2.1 Binning similar points in histogram plots
			3.2.2 Deriving probabilities from histograms
			3.2.3 Shrinking the range of a high confidence interval
			3.2.4 Computing histograms in NumPy
		3.3 Using confidence intervals to analyze a biased deck of cards
		3.4 Using permutations to shuffle cards
		Summary
	Section 4—Case study 1 solution
		4.1 Predicting red cards in a shuffled deck
			4.1.1 Estimating the probability of strategy success
		4.2 Optimizing strategies using the sample space for a 10-card deck
		Summary
Case study 2—Assessing online ad clicks for significance
	Section 5—Basic probability and statistical analysis using SciPy
		5.1 Exploring the relationships between data and probability using SciPy
		5.2 Mean as a measure of centrality
			5.2.1 Finding the mean of a probability distribution
		5.3 Variance as a measure of dispersion
			5.3.1 Finding the variance of a probability distribution
		Summary
	Section 6—Making predictions using the central limit theorem and SciPy
		6.1 Manipulating the normal distribution using SciPy
			6.1.1 Comparing two sampled normal curves
		6.2 Determining the mean and variance of a population through random sampling
		6.3 Making predictions using the mean and variance
			6.3.1 Computing the area beneath a normal curve
			6.3.2 Interpreting the computed probability
		Summary
	Section 7—Statistical hypothesis testing
		7.1 Assessing the divergence between sample mean and population mean
		7.2 Data dredging: Coming to false conclusions through oversampling
		7.3 Bootstrapping with replacement: Testing a hypothesis when the population variance is unknown
		7.4 Permutation testing: Comparing means of samples when the population parameters are unknown
		Summary
	Section 8—Analyzing tables using Pandas
		8.1 Storing tables using basic Python
		8.2 Exploring tables using Pandas
		8.3 Retrieving table columns
		8.4 Retrieving table rows
		8.5 Modifying table rows and columns
		8.6 Saving and loading table data
		8.7 Visualizing tables using Seaborn
		Summary
	Section 9—Case study 2 solution
		9.1 Processing the ad-click table in Pandas
		9.2 Computing p-values from differences in means
		9.3 Determining statistical significance
		9.4 41 shades of blue: A real-life cautionary tale
		Summary
Case study 3—Tracking disease outbreaks using news headlines
	Section 10—Clustering data into groups
		10.1 Using centrality to discover clusters
		10.2 K-means: A clustering algorithm for grouping data into K central groups
			10.2.1 K-means clustering using scikit-learn
			10.2.2 Selecting the optimal K using the elbow method
		10.3 Using density to discover clusters
		10.4 DBSCAN: A clustering algorithm for grouping data based on spatial density
			10.4.1 Comparing DBSCAN and K-means
			10.4.2 Clustering based on non-Euclidean distance
		10.5 Analyzing clusters using Pandas
		Summary
	Section 11—Geographic location visualization and analysis
		11.1 The great-circle distance: A metric for computing the distance between two global points
		11.2 Plotting maps using Cartopy
			11.2.1 Manually installing GEOS and Cartopy
			11.2.2 Utilizing the Conda package manager
			11.2.3 Visualizing maps
		11.3 Location tracking using GeoNamesCache
			11.3.1 Accessing country information
			11.3.2 Accessing city information
			11.3.3 Limitations of the GeoNamesCache library
		11.4 Matching location names in text
		Summary
	Section 12—Case study 3 solution
		12.1 Extracting locations from headline data
		12.2 Visualizing and clustering the extracted location data
		12.3 Extracting insights from location clusters
		Summary
Case study 4—Using online job postings to improve your data science resume
	Section 13—Measuring text similarities
		13.1 Simple text comparison
			13.1.1 Exploring the Jaccard similarity
			13.1.2 Replacing words with numeric values
		13.2 Vectorizing texts using word counts
			13.2.1 Using normalization to improve TF vector similarity
			13.2.2 Using unit vector dot products to convert between relevance metrics
		13.3 Matrix multiplication for efficient similarity calculation
			13.3.1 Basic matrix operations
			13.3.2 Computing all-by-all matrix similarities
		13.4 Computational limits of matrix multiplication
		Summary
	Section 14—Dimension reduction of matrix data
		14.1 Clustering 2D data in one dimension
			14.1.1 Reducing dimensions using rotation
		14.2 Dimension reduction using PCA and scikit-learn
		14.3 Clustering 4D data in two dimensions
			14.3.1 Limitations of PCA
		14.4 Computing principal components without rotation
			14.4.1 Extracting eigenvectors using power iteration
		14.5 Efficient dimension reduction using SVD and scikit-learn
		Summary
	Section 15—NLP analysis of large text datasets
		15.1 Loading online forum discussions using scikit-learn
		15.2 Vectorizing documents using scikit-learn
		15.3 Ranking words by both post frequency and count
			15.3.1 Computing TFIDF vectors with scikit-learn
		15.4 Computing similarities across large document datasets
		15.5 Clustering texts by topic
			15.5.1 Exploring a single text cluster
		15.6 Visualizing text clusters
			15.6.1 Using subplots to display multiple word clouds
		Summary
	Section 16—Extracting text from web pages
		16.1 The structure of HTML documents
		16.2 Parsing HTML using Beautiful Soup
		16.3 Downloading and parsing online data
		Summary
	Section 17—Case study 4 solution
		17.1 Extracting skill requirements from job posting data
			17.1.1 Exploring the HTML for skill descriptions
		17.2 Filtering jobs by relevance
		17.3 Clustering skills in relevant job postings
			17.3.1 Grouping the job skills into 15 clusters
			17.3.2 Investigating the technical skill clusters
			17.3.3 Investigating the soft-skill clusters
			17.3.4 Exploring clusters at alternative values of K
			17.3.5 Analyzing the 700 most relevant postings
		17.4 Conclusion
		Summary
Case study 5—Predicting future friendships from social network data
	Section 18—An introduction to graph theory and network analysis
		18.1 Using basic graph theory to rank websites by popularity
			18.1.1 Analyzing web networks using NetworkX
		18.2 Utilizing undirected graphs to optimize the travel time between towns
			18.2.1 Modeling a complex network of towns and counties
			18.2.2 Computing the fastest travel time between nodes
		Summary
	Section 19—Dynamic graph theory techniques for node ranking and social network analysis
		19.1 Uncovering central nodes based on expected traffic in a network
			19.1.1 Measuring centrality using traffic simulations
		19.2 Computing travel probabilities using matrix multiplication
			19.2.1 Deriving PageRank centrality from probability theory
			19.2.2 Computing PageRank centrality using NetworkX
		19.3 Community detection using Markov clustering
		19.4 Uncovering friend groups in social networks
		Summary
	Section 20—Network-driven supervised machine learning
		20.1 The basics of supervised machine learning
		20.2 Measuring predicted label accuracy
			20.2.1 Scikit-learn’s prediction measurement functions
		20.3 Optimizing KNN performance
		20.4 Running a grid search using scikit-learn
		20.5 Limitations of the KNN algorithm
		Summary
	Section 21—Training linear classifiers with logistic regression
		21.1 Linearly separating customers by size
		21.2 Training a linear classifier
			21.2.1 Improving perceptron performance through standardization
		21.3 Improving linear classification with logistic regression
			21.3.1 Running logistic regression on more than two features
		21.4 Training linear classifiers using scikit-learn
			21.4.1 Training multiclass linear models
		21.5 Measuring feature importance with coefficients
		21.6 Linear classifier limitations
		Summary
	Section 22—Training nonlinear classifiers with decision tree techniques
		22.1 Automated learning of logical rules
			22.1.1 Training a nested if/else model using two features
			22.1.2 Deciding which feature to split on
			22.1.3 Training if/else models with more than two features
		22.2 Training decision tree classifiers using scikit-learn
			22.2.1 Studying cancerous cells using feature importance
		22.3 Decision tree classifier limitations
		22.4 Improving performance using random forest classification
		22.5 Training random forest classifiers using scikit-learn
		Summary
	Section 23—Case study 5 solution
		23.1 Exploring the data
			23.1.1 Examining the profiles
			23.1.2 Exploring the experimental observations
			23.1.3 Exploring the Friendships linkage table
		23.2 Training a predictive model using network features
		23.3 Adding profile features to the model
		23.4 Optimizing performance across a steady set of features
		23.5 Interpreting the trained model
			23.5.1 Why are generalizable models so important?
		Summary
index
	Symbols
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	R
	S
	T
	U
	V
	W
	X
	Y