Logging in Action: With Fluentd, Kubernetes and more

Logging in Action
contents
foreword
preface
acknowledgments
about this book
	Who should read this book
	How this book is organized
	About the code
	liveBook discussion forum
about the author
about the cover illustration
Part 1 From zero to “Hello World”
	1 Introduction to Fluentd
		1.1 Elevator pitch for Fluentd
			1.1.1 What is a log event?
			1.1.2 Fluentd compared to middleware
		1.2 Why do we produce logs?
		1.3 Evolving ideas
			1.3.1 Four golden signals
			1.3.2 Three pillars of observability
		1.4 Log unification
			1.4.1 Unifying logs vs. log analytics
		1.5 Software stacks
			1.5.1 ELK stack
			1.5.2 Comparing Fluentd and Logstash
			1.5.3 The relationship between Fluentd and Fluent Bit
			1.5.4 The relationship between Logstash and Beats
		1.6 Log routing as a vehicle for security
		1.7 Log event life cycle
		1.8 Evolution of Fluentd
			1.8.1 Treasure Data
			1.8.2 CNCF
			1.8.3 Relationship to major cloud vendors PaaS/IaaS
		1.9 Where can Fluentd and Fluent Bit be used?
			1.9.1 Platform constraints
		1.10 Fluentd UI-based editing
		1.11 Plugins
		1.12 How Fluentd can be used to make operational tasks easier
			1.12.1 Actionable log events
			1.12.2 Making logs more meaningful
			1.12.3 Polyglot environments
			1.12.4 Multiple targets
			1.12.5 Controlling log data costs
			1.12.6 Logs to metrics
			1.12.7 Rapid operational consolidation
		Summary
	2 Concepts, architecture, and deployment of Fluentd
		2.1 Architecture and core concepts
			2.1.1 The makeup of a log event
			2.1.2 Handling time
			2.1.3 Architecture of Fluentd
			2.1.4 Fluent configuration execution order
			2.1.5 Directives
			2.1.6 Putting timing requirements into action
		2.2 Deployment of Fluentd
			2.2.1 Deploying Fluentd for the book’s examples
			2.2.2 Deployment considerations for Fluentd
			2.2.3 Fluentd minimum footprint
			2.2.4 Simple deployment of Ruby
			2.2.5 Simple deployment of Fluentd
			2.2.6 Deploying a log generator
			2.2.7 Installing Postman
		2.3 Bringing Fluentd to life with “Hello World”
			2.3.1 “Hello World” scenario
			2.3.2 “Hello World” configuration
			2.3.3 Starting Fluentd
		2.4 “Hello World” with Fluent Bit
			2.4.1 Starting Fluent Bit
			2.4.2 Alternate Fluent Bit startup options
			2.4.3 Fluent Bit configuration file comparison
			2.4.4 Fluent Bit configuration file in detail
			2.4.5 Putting the dummy plugin into action
		2.5 Fluentd deployment with Kubernetes and containers
			2.5.1 Fluentd DaemonSet
			2.5.2 Dockerized Fluentd
		2.6 Using Fluentd UI
			2.6.1 Installing Fluentd with UI
		Summary
Part 2 Fluentd in depth
	3 Using Fluentd to capture log events
		3.1 Dry running to check a configuration
			3.1.1 Putting validating Fluentd configuration into action
		3.2 Reading log files
			3.2.1 Putting the adaption of a Fluentd configuration to Fluent Bit into action
			3.2.2 Rereading and resuming reading of log files
			3.2.3 Configuration considerations for tracking position
			3.2.4 Wildcards in the path attribute
			3.2.5 Expressing time
			3.2.6 Controlling the impact of wildcards in filenames
			3.2.7 Replacing wildcards with delimited lists in action
			3.2.8 Handling log rotation
		3.3 Self-monitoring
			3.3.1 HTTP interface check
		3.4 Imposing structure on log events
			3.4.1 Standard parsers
			3.4.2 Third-party parsers
			3.4.3 Applying a Regex parser to a complex log
			3.4.4 Putting parser configuration into action
		Summary
	4 Using Fluentd to output log events
		4.1 File output plugin
			4.1.1 Basic file output
			4.1.2 Basics of buffering
			4.1.3 Chunks and Controlling Buffering
			4.1.4 Retry and backoff
			4.1.5 Putting configuring buffering size settings into action
		4.2 Output formatting options
			4.2.1 out_file
			4.2.2 json
			4.2.3 ltsv
			4.2.4 csv
			4.2.5 msgpack
			4.2.6 Applying formatters
			4.2.7 Putting JSON formatter configuration into action
		4.3 Sending log events to MongoDB
			4.3.1 Deploying MongoDB Fluentd plugin
			4.3.2 Configuring the Mongo output plugin for Fluentd
			4.3.3 Putting MongoDB connection configuration strings into action
		4.4 Actionable log events
			4.4.1 Actionable log events through service invocation
			4.4.2 Actionable through user interaction tools
		4.5 Slack to demonstrate the social output
			4.5.1 Handling tokens and credentials more carefully
			4.5.2 Externalizing Slack configuration attributes in action
		4.6 The right tool for the right job
		Summary
	5 Routing log events
		5.1 Reaching multiple outputs by copying
			5.1.1 Copy by reference or by value
			5.1.2 Handling errors when copying
		5.2 Configuration reuse and extension through inclusion
			5.2.1 Place holding with null output
			5.2.2 Putting inclusions with a MongoDB output into action
		5.3 Injecting context into log events
			5.3.1 Extraction of values
		5.4 Tag-based routing
			5.4.1 Using exec output plugin
			5.4.2 Putting tag naming conventions into action
			5.4.3 Putting dynamic tagging with extract into action
		5.5 Tag plugins
		5.6 Labels: Taking tags to a new level
			5.6.1 Using a stdout filter to see what is happening
			5.6.2 Illustrating label and tag routing
			5.6.3 Connecting pipelines
			5.6.4 Label sequencing
			5.6.5 Special labels
			5.6.6 Putting a common pipeline into action
		Summary
	6 Filtering and extrapolation
		6.1 Application of filters
			6.1.1 All is well events do not need to be distributed
			6.1.2 Spotting the needle in a haystack
			6.1.3 False urgency
			6.1.4 Releveling
			6.1.5 Unimplemented housekeeping
		6.2 Why change log events?
			6.2.1 Easier to process meaning downstream
			6.2.2 Add context
			6.2.3 Record when we have reacted to a log event
			6.2.4 Data redaction/masking
		6.3 Applying filters and parsers
			6.3.1 Filter plugins
			6.3.2 Applying grep filters
			6.3.3 Changing log events with the record_transformer plugin
			6.3.4 Filter parser vs. record transformer
		6.4 Demonstrating change impact with stdout in action
			6.4.1 A solution demonstrating change impact with stdout in action
		6.5 Extract to set key values
		6.6 Deriving new data values with the record_transformer
			6.6.1 Putting the incorporation of calculations into a log event transformation into action
		6.7 Generating simple Fluentd metrics
			6.7.1 Putting log event counting into action
		Summary
Part 3 Beyond the basics
	7 Performance and scaling
		7.1 Threading and processes to scale with workers
			7.1.1 Seeing workers in action
			7.1.2 Worker constraints
			7.1.3 Controlling output plugin threads
			7.1.4 Memory management optimization
		7.2 Scaling and moving workloads
			7.2.1 Fan-in/log aggregation and consolidation
			7.2.2 Fan-out and workload distribution
			7.2.3 High availability
			7.2.4 Putting a high-availability comparison into action
		7.3 Fluentd scaling in containers vs. native and virtual environments
			7.3.1 Kubernetes worker node configuration
			7.3.2 Per-cluster configuration
			7.3.3 Container as virtualization
			7.3.4 Sidecar pattern
			7.3.5 Options comparison
		7.4 Securing traffic between Fluentd nodes
			7.4.1 TLS configuration
			7.4.2 TLS not just for encryption
			7.4.3 Certificate and private key storage
			7.4.4 Security is more than certificates
		7.5 Credentials management
			7.5.1 Simple credentials use case
			7.5.2 Putting certification into action
		Summary
	8 Driving logs with Docker and Kubernetes
		8.1 Fluentd out of the box from Docker Hub
			8.1.1 Official Docker images
			8.1.2 Docker log drivers
			8.1.3 Getting set up for Docker log drivers
		8.2 Using Docker log drivers
			8.2.1 Docker drivers via the command line
			8.2.2 A quick check of network connections
			8.2.3 Running Docker command line
			8.2.4 Switching to driver configuration through a configuration file
		8.3 Kubernetes components logging and the use of Fluentd
			8.3.1 Kubernetes components and structured logging
			8.3.2 Kubernetes default log retention and log rotation
			8.3.3 kubectl with logging
		8.4 Demonstrating logging with Kubernetes
			8.4.1 Kubernetes setup
			8.4.2 Creating logs to capture
			8.4.3 Understanding how Fluentd DaemonSets are put together
		8.5 Getting a peek at host logs
		8.6 Configuring a Kubernetes logging DaemonSet
			8.6.1 Getting the Fluentd configuration ready to be used
			8.6.2 Creating our Kubernetes deployment configuration
			8.6.3 Putting the implementation of a Fluentd for Kubernetes into action
			8.6.4 Deploying to minikube
			8.6.5 Tidying up
		8.7 Kubernetes configuration in action
			8.7.1 Answer
		8.8 More Kubernetes monitoring and logging to watch for
			8.8.1 Node monitoring
			8.8.2 Termination messages
		Summary
	9 Creating custom plugins
		9.1 Plugin source code
		9.2 What is Redis, and why build a plugin with the Redis list capability?
			9.2.1 Redis list over RedisTimeSeries
		9.3 Illustrating our objective using Redis CLI
		9.4 Preparing for development
		9.5 Plugin frameworks
			9.5.1 Creating the skeleton plugin
			9.5.2 Plugin life cycle
		9.6 Implementing the plugin core
			9.6.1 How configuration attributes work
			9.6.2 Starting up and shutting down
			9.6.3 Getting the plugin to work with our Fluentd installation
			9.6.4 Putting additional configuration validation into action
			9.6.5 Implementing the Redis output logic
			9.6.6 Putting the testing of synchronous output into action
		9.7 Implementing the Redis input plugin
			9.7.1 Testing input and output plugin execution
		9.8 Extending output with buffering
			9.8.1 Improving our scenario by putting maintainability into action
		9.9 Unit testing
		9.10 Putting the development of unit tests into action
			9.10.1 Answer
		9.11 Package and deployment
			9.11.1 Documentation
			9.11.2 Complete metadata aka manifest
			9.11.3 Building the gem package
			9.11.4 Rerun without the plugin paths
		9.12 Extending to be an enterprise-class solution
		Summary
Part 4 Good logging practices and frameworks to maximize log value
	10 Logging best practices
		10.1 Audit events vs. log events
		10.2 Log levels and severities
			10.2.1 Trace
			10.2.2 Debug
			10.2.3 Info(rmation)
			10.2.4 Warn(ing)
			10.2.5 Error
			10.2.6 Fatal
			10.2.7 Extending or creating your own log levels
		10.3 Clear language
		10.4 Human and machine-readable
		10.5 Context is key
			10.5.1 Context: What
			10.5.2 Context: When
			10.5.3 Context: where
			10.5.4 Context: Why
			10.5.5 Context: Who
			10.5.6 a practical checklist for capturing context
		10.6 Error codes
			10.6.1 Using standard errors
			10.6.2 Codes can be for more than errors
		10.7 Too little logging or too much?
			10.7.1 What qualifies as sensitive?
			10.7.2 GDPR is only the start
		10.8 Log structure and format
			10.8.1 Putting making log entries ready for application shipping into action
		10.9 Use frameworks if you can
		10.10 Development practices
			10.10.1 Rethrowing exceptions
			10.10.2 Using standard exceptions and error structures
			10.10.3 String construction as a reason not to log
		Summary
	11 Logging frameworks
		11.1 Value of logging frameworks
		11.2 Typical structure of a logging framework
			11.2.1 Logger context
			11.2.2 Appender
			11.2.3 Logger
			11.2.4 Filter
			11.2.5 Formatter
			11.2.6 Configuration
			11.2.7 Logger config
		11.3 Appender structures
		11.4 Logging framework landscape
		11.5 Choosing a framework
			11.5.1 Putting optimizing application logging into action
		11.6 Fluentd’s own logging and appenders
		11.7 Illustrations of an application logging directly to Fluentd
			11.7.1 Python with logging framework: Using the Fluentd library
			11.7.2 Invoking Fluentd appender directly
			11.7.3 Illustration with only Python’s logging
			11.7.4 Illustration without Python’s logging or Fluentd library
			11.7.5 Porting the Fluentd calls to another language into action
			11.7.6 Using generic appenders: The takeaways
		Summary
appendix A Processing times and dates, regular expressions, and other configuration values
	A.1 Tool installation overview
	A.2 Creating environment variables and amending PATH
		A.2.1 Windows
		A.2.2 Linux
	A.3 Java and Groovy
	A.4 Postman
	A.5 Elasticsearch
		A.5.1 Core Elasticsearch installation
		A.5.2 Elasticsearch UI installation
		A.5.3 Fluentd plugin for Elasticsearch
	A.6 Mongo database
		A.6.1 Mongo DB installation
		A.6.2 MongoDB configuration
	A.7 Slack
	A.8 Setting up Docker and Kubernetes
		A.8.1 Windows Prerequisites
		A.8.2 VirtualBox approach
		A.8.3 Ubuntu image preparation for working with Docker
		A.8.4 Kubernetes installation
	A.9 Support Ruby development libraries and tools
	A.10 Redis
		A.10.1 Redis gem
	A.11 Python
	A.12 Vault
appendix B Processing times and dates, regular expressions, and other configuration values
	B.1 Expressing relative time
	B.2 Expressing dates and times
	B.3 Expressing sizes
	B.4 Regular expressions
		B.4.1 Escape Codes
		B.4.2 Repetition/selection
		B.4.3 Anchors, groups, and alternates
	B.5 Docker tag customization
appendix C Plugins summary
	C.1 Formatter plugins
	C.2 Extract and inject plugin support
	C.3 Filter plugins
	C.4 Tag manipulation plugins
	C.5 Preventing alert storms
	C.6 Analytical and metrics plugins
	C.7 Plugin Interfaces
appendix D Real-world use case
	D.1 Fluentd use in a real-world use case
	D.2 Setting the scene
		D.2.1 The operational challenges
	D.3 Introducing monitoring
		D.3.1 Extending monitoring
		D.3.2 Finessing of monitoring
	D.4 Cloud dilemma
	D.5 Solution
	D.6 Conclusion
appendix E Useful resources
	E.1 Helpful Fluentd resources
	E.2 Helpful Fluentd third-party tools
	E.3 Helpful logging practices resources
	E.4 Common log formats and descriptions
	E.5 Helpful Ruby resources
	E.6 Docker and Kubernetes
	E.7 Elasticsearch
	E.8 Redis
	E.9 SSL/TLS and security
	E.10 Environment setup
	E.11 Logging frameworks
	E.12 Information portals on legislation
	E.13 Other handy sources of information
	E.14 Supporting Fluentd resources
	E.15 Related reading
index
	A
	B
	C
	D
	E
	F
	G
	H
	I
	J
	K
	L
	M
	N
	O
	P
	R
	S
	T
	U
	V
	W
	X
	Y
Logging in Action - back