Serverless ETL and Analytics with AWS Glue: Your comprehensive reference guide to learning about AWS Glue and its features

by Albert Quiroga, Ishan Gaur, Noritaka Sekiyama, Subramanya Vajiraya, Tomohiro Tanaka, Vishal Pathak

Length: 434 pages
Edition: 1
Language: English
Publisher: Packt Publishing
Publication Date: 2022-08-30
ISBN-10: 1800564988
ISBN-13: 9781800564985
Sales Rank: #1242651 (See Top 100 Books)

2 ratings

Print Book Look Inside

Build efficient data lakes that can scale to virtually unlimited size using AWS Glue

Key Features

Learn to work with AWS Glue to overcome typical implementation challenges in data lakes
Create and manage serverless ETL pipelines that can scale to manage big data
Written by AWS Glue community members, this practical guide shows you how to implement AWS Glue in no time

Book Description

Organizations these days have gravitated toward services such as AWS Glue that undertake undifferentiated heavy lifting and provide serverless Spark, enabling you to create and manage data lakes in a serverless fashion. This guide shows you how AWS Glue can be used to solve real-world problems along with helping you learn about data processing, data integration, and building data lakes.

Beginning with AWS Glue basics, this book teaches you how to perform various aspects of data analysis such as ad hoc queries, data visualization, and real-time analysis using this service. It also provides a walk-through of CI/CD for AWS Glue and how to shift left on quality using automated regression tests. You’ll find out how data security aspects such as access control, encryption, auditing, and networking are implemented, as well as getting to grips with useful techniques such as picking the right file format, compression, partitioning, and bucketing. As you advance, you’ll discover AWS Glue features such as crawlers, Lake Formation, governed tables, lineage, DataBrew, Glue Studio, and custom connectors. The concluding chapters help you to understand various performance tuning, troubleshooting, and monitoring options.

By the end of this AWS book, you’ll be able to create, manage, troubleshoot, and deploy ETL pipelines using AWS Glue.

What you will learn

Apply various AWS Glue features to manage and create data lakes
Use Glue DataBrew and Glue Studio for data preparation
Optimize data layout in cloud storage to accelerate analytics workloads
Manage metadata including database, table, and schema definitions
Secure your data during access control, encryption, auditing, and networking
Monitor AWS Glue jobs to detect delays and loss of data
Integrate Spark ML and SageMaker with AWS Glue to create machine learning models

Who this book is for

This book is for ETL developers, data engineers, and data analysts who want to understand how AWS Glue can help you solve your business problems. Basic knowledge of AWS data services is assumed.

Serverless ETL and Analytics with AWS Glue
Contributors
About the authors
About the reviewers
Preface
    Who this book is for
    What this book covers
    To get the most out of this book
    Download the example code files
    Download the color images
    Conventions used
    Get in touch
    Share Your Thoughts
Section 1 – Introduction, Concepts, and the Basics of AWS Glue
Chapter 1: Data Management – Introduction and Concepts
    Types of data processing – OLTP and OLAP
    Data warehouses and data marts
    Data lakes
    Data lakehouse
    Data mesh
    Distributed computing for big data
        Apache Spark
        Apache Spark on the AWS cloud
    AWS Glue 
        Querying data using AWS
    Summary
Chapter 2: Introduction to Important AWS Glue Features
    Data integration 
    Integrating data with AWS Glue
        Data discovery
        Data ingestion
        Data preparation
        Data replication
    Features of AWS Glue
        AWS Glue Data Catalog
        Glue connections 
        AWS Glue crawlers
        Custom classifiers
        AWS Glue Schema Registry
        AWS Glue ETL jobs
        Glue development endpoints 
        AWS Glue interactive sessions 
        Triggers
    Summary
Chapter 3: Data Ingestion
    Technical requirements
    Data ingestion from file/object stores
        Data ingestion from Amazon S3
        Data ingestion from HDFS data stores
    Data ingestion from JDBC data stores
        AWS Glue custom JDBC connectors
    Data ingestion from streaming data sources
        AWS Glue Schema Registry
    Data ingestion from SaaS data stores
    Summary
Section 2 – Data Preparation, Management, and Security
Chapter 4: Data Preparation
    Technical requirements
    Introduction to data preparation 
    Data preparation using AWS Glue
        Visual data preparation using AWS Glue DataBrew
        Source code-based approach to data preparation using AWS Glue
    Selecting the right service/tool
    Summary
Chapter 5: Data Layouts 
    Technical requirements
    Why do we need to pay attention to data layout?
    Key techniques to optimally storing data 
        Selecting a file format
        Compressing your data
        Splittable or unsplittable files
        Partitioning
        Bucketing
    Optimizing the number of files and each file size
        What is compaction?
        Compaction with AWS Glue ETL Spark jobs
        Automatic Compaction with AWS Lake Formation acceleration
    Optimizing your storage with Amazon S3
        Selecting suitable S3 storage classes for your data
        Using S3 Lifecycle for managing object lifecycles
    Summary
    Further reading
Chapter 6: Data Management
    Technical requirements
    Normalizing data
        Casting data types and map column names
        Inferring schemas
        Computing schemas on the fly
        Enforcing schemas
        Flattening nested schemas
        Normalizing scale
        Handling missing values and outliers
        Normalizing date and time values
        Handling error records
    Deduplicating records
    Denormalizing tables
    Securing data content
        Masking values
        Hashing values
    Managing data quality
        AWS Glue DataBrew data quality rules
        DeeQu
    Summary
Chapter 7: Metadata Management
    Technical requirements
    Populating metadata
        Glue Data Catalog API
        DDL statements
        Glue crawlers
        Crawler configuration
    Maintaining metadata
        Glue crawlers
        Updating Data Catalog tables from ETL jobs
    Partition management
        Partition indexes
    Versioning and rollback
        Table versioning
        Lake Formation-governed tables
    Lineage
        Glue DataBrew
    Summary
Chapter 8: Data Security
    Technical requirements
    Access control
        IAM permissions
        Glue dependencies on other AWS services
        S3 bucket policies
        S3 object ownership
        Lake Formation permissions
    Encryption
        Encryption at rest
        Encryption in transit
    Network
        Glue network architecture
        Glue connections
        Network configuration requirements and limitations
        Connecting to resources on the public internet
        Connecting to resources in your on-premise network
    Summary
Chapter 9: Data Sharing 
    Technical requirements
    Overview of data sharing strategies
        Single tenant
        Hub and spoke
        Data mesh
    Sharing data with multiple AWS accounts using S3 bucket policies and Glue catalog policies
        Scenario 1 – sharing data from one account with another using S3 bucket policies and Glue catalog policies
        Prerequisite – S3
        Prerequisite – Glue
        Configuring S3 bucket policies and Glue Catalog resource policies
    Sharing data with multiple AWS accounts using AWS Lake Formation permissions
        Lake Formation permission model
        Lake Formation cross-account sharing
        Lake Formation named resource-based access control
        Lake Formation tag-based access control
        Scenario 2 – sharing data from one account with another using Lake Formation Tag-based access control
        Prerequisite – S3
        Prerequisite – Glue
        Prerequisite – Lake Formation and IAM
        Step 1 – configuring Glue catalog policies
        Step 2 – configuring Lake Formation permissions (producer)
        Step 3 – configuring Lake Formation permissions (consumer)
    Summary
Chapter 10: Data Pipeline Management 
    Technical requirements
    What are data pipelines?
        Why do we need data pipelines?
        How do we build and manage data pipelines?
    Selecting the appropriate data processing services for your analysis
        AWS Batch
        Amazon ECS
        AWS Lambda
        AWS Glue ETL jobs
        Amazon EMR
    Orchestrating your pipelines with workflow tools
        Using AWS Glue workflows
        Using AWS Step Functions
        Using Amazon Managed Workflows for Apache Airflow 
    utomating how you provision your pipelines with provisioning tools
        Provisioning resources with AWS CloudFormation
        Provisioning AWS Glue workflows and resources with AWS Glue Blueprints
    Developing and maintaining your data pipelines
        Developing AWS Glue ETL jobs locally
        Deploying AWS Glue ETL jobs 
        Deploying workflows and pipelines using provisioning tools such as IaC 
    Summary
    Further reading
Section 3 – Tuning, Monitoring, Data Lake Common Scenarios, and Interesting Edge Cases
Chapter 11: Monitoring
    Defining an SLA for a data platform
    Monitoring the SLA of a data platform
    Monitoring the components of a data platform
        Monitoring state changes
        Monitoring delay
        Monitoring performance
        Monitoring common failures
        Monitoring log messages
    Analyzing usage
    Summary
Chapter 12: Tuning, Debugging, and Troubleshooting
    Tuning AWS Glue workloads
        Tuning AWS Glue crawlers
        Tuning the performance of AWS Glue Spark ETL jobs
    Troubleshooting and debugging common issues in AWS Glue ETL
        ETL job failures
    Summary
Chapter 13: Data Analysis
    Creating Marketplace connections
        Creating the Glue Hudi connection
        Creating a Delta Lake connection
        Creating an OpenSearch connection
    Creating the CloudFormation stack
        Prerequisites for creating the CloudFormation stack
    The benefit of ad hoc analysis and how a data lake enables it 
        Amazon Athena
        Amazon Redshift Spectrum
    Creating and updating Hudi tables using Glue
    Creating and updating Delta Lake tables using Glue
    Inserting data into Lake Formation governed tables
    Consuming streaming data using Glue
        Creating chapter-data-analysis-msk-connection
        Loading and consuming data from MSK using Glue
        Glue streaming job as a consumer of a Kafka topic
        Hudi DeltaStreamer streaming job as a consumer of a Kafka topic
        Creating and consuming CDC data through streaming jobs on Glue
    Glue’s integration with OpenSearch
    Cleaning up
    Summary
Chapter 14: Machine Learning Integration
    Technical requirements
    Glue ML transformations
        Creating an ML transform
        Training an ML transform
        Using an ML transform
    SageMaker integration
    Developing ML pipelines with Glue
    Summary
Chapter 15: Architecting Data Lakes for Real-World Scenarios and Edge Cases
    Technical requirements
    Running a highly selective query on a big fact table using AWS Glue
        Hands-on tutorial
    Dealing with Join performance issues with big fact and small dimension tables in ETL workloads
    Solving Join problems involving big fact and big dimension tables using AWS Glue
        Hands-on tutorial
        Solution
    Reducing time on read operations using AWS Glue grouping
    Solving S3 eventual consistency problems using AWS Glue
        Using glueparquet
        S3-optimized output committer
    Summary
    Why subscribe?
Other Books You May Enjoy
    Packt is searching for authors like you
    Share Your Thoughts

Data Processing Data Warehousing Database Storage & Design

Donate to keep this site alive

To access the Link, solve the captcha.

How to download source code?

1. Go to: https://github.com/PacktPublishing

2. In the Find a repository… box, search the book title: Serverless ETL and Analytics with AWS Glue: Your comprehensive reference guide to learning about AWS Glue and its features, sometime you may not get the results, please search the main title.