Using Python 3.9 and BeautifulSoup to Web Scrape Market, Economic, Financial and Historical Company Earnings Data

Length: 357 pages
Edition: 1
Language: English
Publication Date: 2020-05-04
ISBN-10: B0882LV9T2
Sales Rank: #1401760 (See Top 100 Books)

January 01, 2021, Added
Added section 5.4 Russell 3000 Index
December 18, 2021, Added
Added section 30 What is JSON
December 11, 2021, Added
Added section 16.4 High Dividend Yield Stocks
September 18, 2021, Update
Python code on how to retrieve historical company earnings
Three standard web technologies

We will use Python and the Python Library BeautifulSoup to retrieve many different types of market, financial and economic data. The types of market, financial and economic data we will retrieve are stock, market index, currency rates, bonds, futures, mutual funds, electronically traded funds (ETF), commodities, balance sheet, income statement, statement of cash flow, GDP, consumer price index, unemployment rate, household income and money supply.

Chapter 28 discusses how to retrieve historical company earnings.

There are two ways to retrieve market data; one of them is web scraping the data; the other is to use the REST web service or REST API that is provided by the website. Most REST API from government agencies are free. When web scraping, we will use the beautifulsoup Python library. The install instruction is located at https://pypi.org/project/beautifulsoup4.

1. 1. December 18, 2021, Added
2. Added section 30 What is JSON
  1. December 11, 2021, Added
3. Added section 16.4 High Dividend Yield Stocks
4. 1. September 18, 2021, Update
5. Python code on how to retrieve historical company earnings
6. Three standard web technologies
  1. January 23, 2021, Update
Added section 3, Standard & Poor’s Sectors and Industries
1. In this section, we will demonstrate the Yahoo! finance Sector and Industry API. We will discuss how to create the URL for the API.
1. 1. September 5, 2020, Update
Added section 21, Yahoo! Finance Ticker Info API
1. In this section, we will demonstrate the Yahoo! finance Ticker Info API. We will discuss how to create the URL for the API.
  1. August 21, 2020, Update
Added 2.3.3 NASDAQ 100
Updated 1.4.5 Yahoo! Finance Python Libraries
1. 1. August 13, 2020, Update

France, The United States and South Korea
1. 19.1 GDP
2. 19.2 Unemployment Rate
3. 19.3 Consumer Prices
  1. August 11, 2020, Update

Added 5.5 Rate of Return
1. 1. August 10, 2020, Update

Updated 1.4.1 Visual Studio Code
Added 13.7 API for Financial Statements

Introduction
    1.1Target Audience
    1.2Official Python Website
    1.3Python 3.9
    1.4Integrated Development Environment (IDE) for Python
        1.4.1Visual Studio Code
        1.4.2Sublime Text
        1.4.3Kite Python Addon
        1.4.4Python IDLE
        1.4.5Yahoo! Finance Python Libraries
    1.5Book Updates
        1.5.1December 18, 2021, Added
        1.5.2December 18, 2021, Added
        1.5.3December 11, 2021, Added
        1.5.4September 18, 2021, Update
        1.5.5January 23, 2021, Update
        1.5.6September 5, 2020, Update
        1.5.7August 21, 2020, Update
        1.5.8August 13, 2020, Update
        1.5.9August 11, 2020, Update
        1.5.10August 10, 2020, Update
2Three Standard Web Technologies
    2.1.1HTML
        2.1.1.1Element Syntax
        2.1.1.2Attributes
        2.1.1.3General Purpose Attributes
        2.1.1.4HTML Tables
    2.1.2CSS
    2.1.3JavaScript
3Google Chrome “Inspect” Tool
4XPath
5Market Indexes
    5.1Dow Jones Industrial Average
        5.1.1The Three Market Indexes
        5.1.2Dow Jones Industrial Average 30 Components
    5.2S&P 500 Index
        5.2.1S&P 500 Components
        5.2.2S&P 500 Sectors
    5.3Nasdaq Composite Index
        5.3.1Tickers on NASDAQ
        5.3.2NASDAQ Index Components
        5.3.3NASDAQ 100
    5.4Russell 3000 Index
        5.4.1Russell 1000 Index
        5.4.2Russell 2000 Index
6Standard & Poor’s Sectors and Industries
    6.1Yahoo! Finance Sector and Industry API
7Mutual Funds
    7.1Credit Rating Agencies and the SEC
    7.2Mutual Fund Python Code.
    7.325 Largest Mutual Funds
    7.4Risk Measure - Beta of 1.0
    7.5Risk Measure – Alpha
    7.6S&P 500
        7.6.1S&P 500 Total Returns from 1941 to 2020
        7.6.2Two Largest S&P 500 Mutual Fund
        7.6.3US News Mutual fund Sector Rankins
    7.7NASDAQ Index Mutual Funds
        7.7.1Fidelity NASDAQ Composite Index Fund (FNCMX)
8Electronic Traded Funds (ETF)
    8.1ETF Python Code
    8.2Top 6 S&P 500 ETF
        8.2.1iShare Core S&P 500 ETF (IVV)
        8.2.2Vanguard S&P 500 ETF (VOO)
        8.2.3SPDR S&P 500 ETF Trust (SPY)
        8.2.4Portfolio Plus S&P 500 ETF (PPLC)
        8.2.5Schwab U.S. Large-Cap ETF(SCHX)
        8.2.6iShares S&P 500 Growth ETF(IVW)
    8.3Dow Jones Industrial Average ETF Fund
        8.3.1Non-Leverage
        8.3.2Leverage
9Stocks
    9.1Stock Exchanges
        9.1.1Python Code
        9.1.2Microsoft Corporation (MSFT)
        9.1.3Walmart Inc (WMT)
        9.1.4Yahoo! Finance Financial Summary
    9.2Plotting historical/Time-series Data
        9.2.1Python Code
    9.3Microsoft (MSFT) and Market Indexes
        9.3.1Python Code
        9.3.2Python Output
    9.4High, Low, Negative Beta Stocks
        9.4.1Python Code
        9.4.2High Beta Stocks
        9.4.3Low Beta Stocks
        9.4.4Negative Beta Stocks
    9.5Rate of Return
        9.5.1Python Code
        9.5.2Python Output
10Call and Put Options on Stock
    10.1Python Code
    10.2Call Stock Options
        10.2.1Python Output
    10.3Put Stock Options
        10.3.1Python Output
11Currency
    11.1Latest FX Rates
    11.2Major Currencies
    11.3Minor Currencies
    11.4Exotic Currencies
    11.518 Python Code – Major, Minor, Exotic
    11.618 Python Output – Major, Minor, Exotic
    11.7Currency Pairs
        11.7.1Major Currency Pairs
    11.8Plotting Historical Currency Rates
        11.8.1Python Code
        11.8.2Python Output
12Bonds
    12.1U.S. Treasuries
        12.1.1Treasury.gov
        12.1.213 Week Treasury Bill
        12.1.35 Year Treasury
        12.1.410 Year Treasury
        12.1.530 Year Treasury
    12.2Corporate Bonds
        12.2.1Python Code
        12.2.25 Year High-Quality Market Corporate Bond
        12.2.310 Year High-Quality Market Corporate Bond
        12.2.430 Year High-Quality Market Corporate Bond
    12.3World Government Bonds
        12.3.1Yields
        12.3.2Spreads
        12.3.3Python Code
        12.3.4Python Output
    12.4Regional Government Bonds
        12.4.1Python Code
        12.4.2Americas
        12.4.3Europe, Middle East & Africa
        12.4.4Asia Pacific
13Credit Ratings
    13.1.1Credit Ratings
        13.1.1.1Credit Ratings History
14Central Bank Rates
15Sovereign Credit Default Swaps (CDS)
16Dividends
    16.1History
        16.1.1Python Code
        16.1.2Stocks
        16.1.3Electronic Transfer Funds (ETF)
        16.1.4Preferred Stock ETF
        16.1.5Mutual Funds
    16.2Dividend Payment Dates
        16.2.1Stocks
        16.2.2Electronic Transfer Funds (ETF)
        16.2.3Preferred Stock ETF
        16.2.4SPDR Wells Fargo Preferred Stock ETF
    16.3Mutual Funds
        16.3.1Vanguard 500 Index Fund Admiral Shares Inst
    16.4High Dividend Yield Stocks
        16.4.1Python Code
        16.4.2Python Output
17Financial Statements
    17.1Microsoft Peers
    17.2Balance Sheet
        17.2.1Microsoft Corporation (MSFT)
        17.2.2Apple Inc (AAPL)
        17.2.3Amazon.com Inc (AMZN)
        17.2.4Facebook, Inc. (FB)
        17.2.5Python Code
        17.2.6Python Output
    17.3Income Statement
        17.3.1Microsoft Corporation (MSFT)
        17.3.2Apple Inc (AAPL)
        17.3.3Amazon.com Inc (AMZN)
        17.3.4Facebook, Inc. (FB)
    17.4Statement of Cash Flows
        17.4.1Microsoft Corporation (MSFT)
        17.4.2Apple Inc (AAPL)
        17.4.3Amazon.com Inc (AMZN)
        17.4.4Facebook, Inc. (FB)
    17.5Securities and Exchange Commission (SEC)
    17.6SEC Form 10-K
        17.6.1Apple’s 10-K Form
    17.7API for Financial Statements
18Financial Ratios
    18.1Investment Valuation Ratios
    18.2Profitability Indicator Ratios
    18.3Operating Performance Ratio
    18.4Liquidity Measurement Ratios
    18.5Debt Ratios
    18.6Cash Flow Indicator Ratios
19Measuring Market Volatility
    19.1COBE Volatility Index (^VIX)
        19.1.1Python Output
        19.1.2Python Output Plotting Historical Data
    19.2S&P 500 VIX Futures Contracts
20NYSE Holidays
    20.1Python Code
    20.2Python Output
21S&P 500 Sector Performance
    21.1Python Code
    21.2Python Output
22Federal Reserve Economic Data (FRED)
    22.1QUANDL Data Provider
    22.2Gross Domestic Product
        22.2.1Python Code
        22.2.2Python Output
    22.3Consumer Price Index
        22.3.1Python Code
        22.3.2Python Output
    22.4M1 Money Stock
        22.4.1Python Code
        22.4.2Python Output
    22.5Unemployment Rate
        22.5.1Python Code
        22.5.2Python Output
    22.6Real Median Household Income
        22.6.1Python Code
        22.6.2Python Output
23Dbnomics – World Economic Data
    23.1Gross Domestic Product
        23.1.1GDP Report
        23.1.2DBnomics Python Library
        23.1.3Python Code – GDP Graph of France, The United States and Korea
        23.1.4Python Output
    23.2Unemployment Rate
        23.2.1Python Code – GDP Graph of France, The United States and Korea
        23.2.2Python Output
    23.3Consumer Prices
        23.3.1Python Code – Consumer Prices Graph of France, The United States and Korea
        23.3.2Python Output
24Commodities
    24.1Energy
    24.2Metals
    24.3Agriculture
    24.4Meat & Livestock
    24.5Consumer
    24.6Python Code
    24.7Python Output
    24.8Historical Commodity Prices
        24.8.1Energy – Brent Crude Oil
        24.8.2Metals – Gold
        24.8.3Agriculture – Corn
        24.8.4Meat & Live Stock – Live Cattle
        24.8.5Consumer – Coffee
25Yahoo! Finance Ticker Info API
    25.1Dow Jones Industrial Average
        25.1.1Price
        25.1.2Time Stamp.
    25.2S&P 500
        25.2.1Price
        25.2.2Time Stamp.
    25.3NASDAQ
        25.3.1Price
        25.3.2Time Stamp.
26What is A JSON API
27Zip Codes
    27.1City
        27.1.1Python Code
        27.1.2Python Output
    27.2Zip Code of City
        27.2.1Python Code
        27.2.2Python Output
28Weather
    28.1Current Forecast
        28.1.1XML Response
        28.1.2JSON Response
        28.1.3HTML Response
29Historical Corporate Earnings
    29.1Python Code
    29.2Excel Output
30What is JSON and Example of JSON
    30.1JSON Example
    30.2Python Code
    30.3Python Output
    30.4Analysis of JSON and Python Code
31References