Research Software Engineering with Python: Building software that makes research possible

by Charlotte Wickham, Damien Irving, Greg Wilson, Joel Ostblom, Kate Hertweck, Luke Johnston

Length: 528 pages
Edition: 1
Language: English
Publisher: Chapman and Hall/CRC
Publication Date: 2021-08-06
ISBN-10: 036769834X
ISBN-13: 9780367698348
Sales Rank: #0 (See Top 100 Books)

Writing and running software is now as much a part of science as telescopes and test tubes, but most researchers are never taught how to do either well. As a result, it takes them longer to accomplish simple tasks than it should, and it is harder for them to share their work with others than it needs to be.

This book introduces the concepts, tools, and skills that researchers need to get more done in less time and with less pain. Based on the practical experiences of its authors, who collectively have spent several decades teaching software skills to scientists, it covers everything graduate-level researchers need to automate their workflows, collaborate with colleagues, ensure that their results are trustworthy, and publish what they have built so that others can build on it. The book assumes only a basic knowledge of Python as a starting point, and shows readers how it, the Unix shell, Git, Make, and related tools can give them more time to focus on the research they actually want to do.

Research Software Engineering with Python can be used as the main text in a one-semester course or for self-guided study. A running example shows how to organize a small research project step by step; over a hundred exercises give readers a chance to practice these skills themselves, while a glossary defining over two hundred terms will help readers find their way through the terminology. All of the material can be re-used under a Creative Commons license, and all royalties from sales of the book will be donated to The Carpentries, an organization that teaches foundational coding and data science skills to researchers worldwide.

Cover
Half Title
Title Page
Copyright Page
Dedication
Contents
Welcome
	0.1. The Big Picture
	0.2. Intended Audience
	0.3. What You Will Learn
	0.4. Using this Book
	0.5. Contributing and Re-Use
	0.6. Acknowledgments
1. Getting Started
	1.1. Project Structure
	1.2. Downloading the Data
	1.3. Installing the Software
	1.4. Summary
	1.5. Exercises
	1.6. Key Points
2. The Basics of the Unix Shell
	2.1. Exploring Files and Directories
	2.2. Moving Around
	2.3. Creating New Files and Directories
	2.4. Moving Files and Directories
	2.5. Copying Files and Directories
	2.6. Deleting Files and Directories
	2.7. Wildcards
	2.8. Reading the Manual
	2.9. Summary
	2.10. Exercises
	2.11. Key Points
3. Building Tools with the Unix Shell
	3.1. Combining Commands
	3.2. How Pipes Work
	3.3. Repeating Commands on Many Files
	3.4. Variable Names
	3.5. Redoing Things
	3.6. Creating New Filenames Automatically
	3.7. Summary
	3.8. Exercises
	3.9. Key Points
4. Going Further with the Unix Shell
	4.1. Creating New Commands
	4.2. Making Scripts More Versatile
	4.3. Turning Interactive Work into a Script
	4.4. Finding Things in Files
	4.5. Finding Files
	4.6. Configuring the Shell
	4.7. Summary
	4.8. Exercises
	4.9. Key Points
5. Building Command-Line Tools with Python
	5.1. Programs and Modules
	5.2. Handling Command-Line Options
	5.3. Documentation
	5.4. Counting Words
	5.5. Pipelining
	5.6. Positional and Optional Arguments
	5.7. Collating Results
	5.8. Writing Our Own Modules
	5.9. Plotting
	5.10. Summary
	5.11. Exercises
	5.12. Key Points
6. Using Git at the Command Line
	6.1. Setting Up
	6.2. Creating a New Repository
	6.3. Adding Existing Work
	6.4. Describing Commits
	6.5. Saving and Tracking Changes
	6.6. Synchronizing with Other Repositories
	6.7. Exploring History
	6.8. Restoring Old Versions of Files
	6.9. Ignoring Files
	6.10. Summary
	6.11. Exercises
	6.12. Key Points
7. Going Further with Git
	7.1. What's a Branch?
	7.2. Creating a Branch
	7.3. What Curve Should We Fit?
	7.4. Verifying Zipf's Law
	7.5. Merging
	7.6. Handling Conflicts
	7.7. A Branch-Based Workflow
	7.8. Using Other People's Work
	7.9. Pull Requests
	7.10. Handling Conflicts in Pull Requests
	7.11. Summary
	7.12. Exercises
	7.13. Key Points
8. Working in Teams
	8.1. What Is a Project?
	8.2. Include Everyone
	8.3. Establish a Code of Conduct
	8.4. Include a License
	8.5. Planning
	8.6. Bug Reports
	8.7. Labeling Issues
	8.8. Prioritizing
	8.9. Meetings
	8.10. Making Decisions
	8.11. Make All This Obvious to Newcomers
	8.12. Handling Conflict
	8.13. Summary
	8.14. Exercises
	8.15. Key Points
9. Automating Analyses with Make
	9.1. Updating a Single File
	9.2. Managing Multiple Files
	9.3. Updating Files When Programs Change
	9.4. Reducing Repetition in a Makefile
	9.5. Automatic Variables
	9.6. Generic Rules
	9.7. Defining Sets of Files
	9.8. Documenting a Makefile
	9.9. Automating Entire Analyses
	9.10. Summary
	9.11. Exercises
	9.12. Key Points
10. Configuring Programs
	10.1. Configuration File Formats
	10.2. Matplotlib Configuration
	10.3. The Global Configuration File
	10.4. The User Configuration File
	10.5. Adding Command-Line Options
	10.6. A Job Control File
	10.7. Summary
	10.8. Exercises
	10.9. Key Points
11. Testing Software
	11.1. Assertions
	11.2. Unit Testing
	11.3. Testing Frameworks
	11.4. Testing Floating-Point Values
	11.5. Integration Testing
	11.6. Regression Testing
	11.7. Test Coverage
	11.8. Continuous Integration
	11.9. When to Write Tests
	11.10. Summary
	11.11. Exercises
	11.12. Key Points
12. Handling Errors
	12.1. Exceptions
	12.2. Writing Useful Error Messages
	12.3. Testing Error Handling
	12.4. Reporting Errors
	12.5. Summary
	12.6. Exercises
	12.7. Key Points
13. Tracking Provenance
	13.1. Data Provenance
	13.2. Code Provenance
	13.3. Summary
	13.4. Exercises
	13.5. Key Points
14. Creating Packages with Python
	14.1. Creating a Python Package
	14.2. Virtual Environments
	14.3. Installing a Development Package
	14.4. What Installation Does
	14.5. Distributing Packages
	14.6. Documenting Packages
	14.7. Software Journals
	14.8. Summary
	14.9. Exercises
	14.10. Key Points
15. Finale
	15.1. Why We Wrote This Book
Appendix
	A. Solutions
	B. Learning Objectives
		B.1. Getting Started
		B.2. The Basics of the Unix Shell
		B.3. Building Tools with the Unix Shell
		B.4. Going Further with the Unix Shell
		B.5. Building Command-Line Tools with Python
		B.6. Using Git at the Command Line
		B.7. Going Further with Git
		B.8. Working in Teams
		B.9. Automating Analyses with Make
		B.10. Configuring Programs
		B.11. Testing Software
		B.12. Handling Errors
		B.13. Tracking Provenance
		B.14. Creating Packages with Python
	C. Key Points
		C.1. Getting Started
		C.2. The Basics of the Unix Shell
		C.3. Building Tools with the Unix Shell
		C.4. Going Further with the Unix Shell
		C.5. Building Command-Line Programs in Python
		C.6. Using Git at the Command Line
		C.7. Going Further with Git
		C.8. Working in Teams
		C.9. Automating Analyses with Make
		C.10. Configuring Programs
		C.11. Testing Software
		C.12. Handling Errors
		C.13. Tracking Provenance
		C.14. Creating Packages with Python
	D. Project Tree
	E. Working Remotely
		E.1. Logging In
		E.2. Copying Files
		E.3. Running Commands
		E.4. Creating Keys
		E.5. Dependencies
	F. Writing Readable Code
		F.1. Python Style
		F.2. Order
		F.3. Checking Style
		F.4. Refactoring
		F.5. Code Reviews
		F.6. Python Features
		F.7. Summary
	G. Documenting Programs
		G.1. Writing Good Docstrings
		G.2. Defining Your Audience
		G.3. Creating an FAQ
	H. YAML
	I. Anaconda
		I.1. Package Management with conda
		I.2. Environment Management with conda
	J. Glossary
	K. References
Index