Natural Language Processing with TensorFlow: The definitive NLP book to implement the most sought-after machine learning models and tasks, 2nd Edition

Preface
    Who this book is for
    What this book covers
    To get the most out of this book
    Get in touch
Introduction to Natural Language Processing
    What is Natural Language Processing?
    Tasks of Natural Language Processing
    The traditional approach to Natural Language Processing
        Understanding the traditional approach
            Example – generating football game summaries
        Drawbacks of the traditional approach
    The deep learning approach to Natural Language Processing
        History of deep learning
        The current state of deep learning and NLP
        Understanding a simple deep model – a fully connected neural network
    Introduction to the technical tools
        Description of the tools
        Installing Anaconda and Python
            Creating a Conda environment
        TensorFlow (GPU) software requirements
        Accessing Jupyter Notebook
        Verifying the TensorFlow installation
    Summary
Understanding TensorFlow 2
    What is TensorFlow?
        Getting started with TensorFlow 2
        TensorFlow 2 architecture – What happens during graph build?
        TensorFlow architecture – what happens when you execute the graph?
        Café Le TensorFlow 2 – understanding TensorFlow 2 with an analogy
        Flashback: TensorFlow 1
    Inputs, variables, outputs, and operations
        Defining inputs in TensorFlow
            Feeding data as NumPy arrays
            Feeding data as tensors
            Building a data pipeline using the tf.data API
        Defining variables in TensorFlow
        Defining outputs in TensorFlow
        Defining operations in TensorFlow
            Comparison operations
            Mathematical operations
            Updating (scattering) values in tensors
            Collecting (gathering) values from a tensor
        Neural network-related operations
            Nonlinear activations used by neural networks
            The convolution operation
            The pooling operation
            Defining loss
    Keras: The model building API of TensorFlow
        Sequential API
        Functional API
        Sub-classing API
    Implementing our first neural network
        Preparing the data
        Implementing the neural network with Keras
            Training the model
            Testing the model
    Summary
Word2vec – Learning Word Embeddings
    What is a word representation or meaning?
    Classical approaches to learning word representation
        One-hot encoded representation
        The TF-IDF method
        Co-occurrence matrix
    An intuitive understanding of Word2vec – an approach to learning word representation
        Exercise: does queen = king – he + she?
    The skip-gram algorithm
        From raw text to semi-structured text
        Understanding the skip-gram algorithm
        Implementing and running the skip-gram algorithm with TensorFlow
            Implementing the data generators with TensorFlow
            Implementing the skip-gram architecture with TensorFlow
            Training and evaluating the model
    The Continuous Bag-of-Words algorithm
        Generating data for the CBOW algorithm
        Implementing CBOW in TensorFlow
        Training and evaluating the model
    Summary
Advanced Word Vector Algorithms
    GloVe – Global Vectors representation
        Understanding GloVe
        Implementing GloVe
        Generating data for GloVe
        Training and evaluating GloVe
    ELMo – Taking ambiguities out of word vectors
        Downloading ELMo from TensorFlow Hub
        Preparing inputs for ELMo
        Generating embeddings with ELMo
    Document classification with ELMo
        Dataset
        Generating document embeddings
        Classifying documents with document embeddings
    Summary
Sentence Classification with Convolutional Neural Networks
    Introducing CNNs
        CNN fundamentals
            The power of CNNs
    Understanding CNNs
        Convolution operation
            Standard convolution operation
            Convolving with stride
            Convolving with padding
            Transposed convolution
        Pooling operation
            Max pooling
            Max pooling with stride
            Average pooling
        Fully connected layers
        Putting everything together
    Exercise – image classification on Fashion-MNIST with CNN
        About the data
        Downloading and exploring the data
        Implementing the CNN
        Analyzing the predictions produced with a CNN
    Using CNNs for sentence classification
        How data is transformed for sentence classification
        Implementation – downloading and preparing data
            Implementation – building a tokenizer
        The sentence classification CNN model
            The convolution operation
            Pooling over time
        Implementation – sentence classification with CNNs
        Training the model
    Summary
Recurrent Neural Networks
    Understanding RNNs
        The problem with feed-forward neural networks
        Modeling with RNNs
        Technical description of an RNN
    Backpropagation Through Time
        How backpropagation works
        Why we cannot use BP directly for RNNs
        Backpropagation Through Time – training RNNs
        Truncated BPTT – training RNNs efficiently
        Limitations of BPTT – vanishing and exploding gradients
    Applications of RNNs
        One-to-one RNNs
        One-to-many RNNs
        Many-to-one RNNs
        Many-to-many RNNs
    Named Entity Recognition with RNNs
        Understanding the data
        Processing data
        Defining hyperparameters
        Defining the model
            Introduction to the TextVectorization layer
            Defining the rest of the model
        Evaluation metrics and the loss function
        Training and evaluating RNN on NER task
        Visually analyzing outputs
    NER with character and token embeddings
        Using convolution to generate token embeddings
        Implementing the new NER model
            Defining hyperparameters
            Defining the input layer
            Defining the token-based TextVectorization layer
            Defining the character-based TextVectorization layer
            Processing the inputs for the char_vectorize_layer
            Performing convolution on the character embeddings
        Model training and evaluation
        Other improvements you can make
    Summary
Understanding Long Short-Term Memory Networks
    Understanding Long Short-Term Memory Networks
        What is an LSTM?
        LSTMs in more detail
        How LSTMs differ from standard RNNs
    How LSTMs solve the vanishing gradient problem
    Improving LSTMs
        Greedy sampling
        Beam search
        Using word vectors
        Bidirectional LSTMs (BiLSTMs)
    Other variants of LSTMs
        Peephole connections
        Gated Recurrent Units
    Summary
Applications of LSTM – Generating Text
    Our data
        About the dataset
        Generating training, validation, and test sets
        Analyzing the vocabulary size
        Defining the tf.data pipeline
    Implementing the language model
        Defining the TextVectorization layer
        Defining the LSTM model
        Defining metrics and compiling the model
        Training the model
        Defining the inference model
        Generating new text with the model
    Comparing LSTMs to LSTMs with peephole connections and GRUs
        Standard LSTM
            Review
        Gated Recurrent Units (GRUs)
            Review
            The model
        LSTMs with peepholes
            Review
            The code
        Training and validation perplexities over time
    Improving sequential models – beam search
        Implementing beam search
        Generating text with beam search
    Improving LSTMs – generating text with words instead of n-grams
        The curse of dimensionality
        Word2vec to the rescue
        Generating text with Word2vec
    Summary
Sequence-to-Sequence Learning – Neural Machine Translation
    Machine translation
    A brief historical tour of machine translation
        Rule-based translation
        Statistical Machine Translation (SMT)
        Neural Machine Translation (NMT)
    Understanding neural machine translation
        Intuition behind NMT systems
        NMT architecture
            The embedding layer
            The encoder
            The context vector
            The decoder
    Preparing data for the NMT system
        The dataset
            Adding special tokens
            Splitting training, validation, and testing datasets
            Defining sequence lengths for the two languages
            Padding the sentences
    Defining the model
        Converting tokens to IDs
        Defining the encoder
        Defining the decoder
        Attention: Analyzing the encoder states
            Computing Attention
            Implementing Attention
        Defining the final model
    Training the NMT
    The BLEU score – evaluating the machine translation systems
        Modified precision
        Brevity penalty
        The final BLEU score
    Visualizing Attention patterns
    Inference with NMT
    Other applications of Seq2Seq models – chatbots
        Training a chatbot
        Evaluating chatbots – the Turing test
    Summary
Transformers
    Transformer architecture
        The encoder and the decoder
        Computing the output of the self-attention layer
        Embedding layers in the Transformer
        Residuals and normalization
    Understanding BERT
        Input processing for BERT
        Tasks solved by BERT
        How BERT is pre-trained
            Masked Language Modeling (MLM)
            Next Sentence Prediction (NSP)
    Use case: Using BERT to answer questions
        Introduction to the Hugging Face transformers library
        Exploring the data
        Implementing BERT
            Implementing and using the Tokenizer
            Defining a TensorFlow dataset
            BERT for answering questions
            Defining the config and the model
        Training and evaluating the model
        Answering questions with Bert
    Summary
Image Captioning with Transformers
    Getting to know the data
        ILSVRC ImageNet dataset
        The MS-COCO dataset
    Downloading the data
    Processing and tokenizing data
        Preprocessing data
        Tokenizing data
    Defining a tf.data.Dataset
    The machine learning pipeline for image caption generation
        Vision Transformer (ViT)
        Text-based decoder Transformer
        Putting everything together
    Implementing the model with TensorFlow
        Implementing the ViT model
        Implementing the text-based decoder
            Defining the self-attention layer
            Defining the Transformer layer
            Defining the full decoder
    Training the model
    Evaluating the results quantitatively
        BLEU
        ROUGE
        METEOR
        CIDEr
    Evaluating the model
    Captions generated for test images
    Summary
Appendix A: Mathematical Foundations and Advanced TensorFlow
    Basic data structures
        Scalar
        Vectors
        Matrices
        Indexing of a matrix
    Special types of matrices
        Identity matrix
        Square diagonal matrix
        Tensors
    Tensor/matrix operations
        Transpose
        Matrix multiplication
        Element-wise multiplication
        Inverse
        Finding the matrix inverse – Singular Value Decomposition (SVD)
        Norms
        Determinant
    Probability
        Random variables
            Discrete random variables
            Continuous random variables
        The probability mass/density function
        Conditional probability
        Joint probability
        Marginal probability
        Bayes’ rule
    Visualizing word embeddings with TensorBoard
        Starting TensorBoard
        Saving word embeddings and visualizing via TensorBoard
    Summary
    Other Books You May Enjoy
Index