Pandas for Everyone

Daniel Chen is a graduate student in the interdisciplinary PhD program in Genetics, Bioinformatics & Computational Biology (GBCB) at Virginia Tech. He is involved with Software Carpentry as an instructor and lesson maintainer. He completed his master’s degree in public health at Columbia University Mailman School of Public Health in Epidemiology, and currently works at the Social and Decision Analytics Laboratory under the Biocomplexity Institute of Virginia Tech where he is working with data to inform policy decision-making. He is the author of Pandas for Everyone and Pandas Data Analysis with Python Fundamentals LiveLessons.

• Foreword xixPreface xxiAcknowledgments xxviiAbout the Author xxxi   Part I: Introduction 1  Chapter 1: Pandas DataFrame Basics 3 1.1 Introduction 31.2 Loading Your First Data Set 41.3 Looking at Columns, Rows, and Cells 71.4 Grouped and Aggregated Calculations 181.5 Basic Plot 231.6 Conclusion 24  Chapter 2: Pandas Data Structures 252.1 Introduction 252.2 Creating Your Own Data 262.3 The Series 282.4 The DataFrame 362.5 Making Changes to Series and DataFrames 382.6 Exporting and Importing Data 432.7 Conclusion 47  Chapter 3: Introduction to Plotting 493.1 Introduction 493.2 Matplotlib 513.3 Statistical Graphics Using matplotlib 563.4 Seaborn 613.5 Pandas Objects 833.6 Seaborn Themes and Styles 863.7 Conclusion 90   Part II: Data Manipulation 91  Chapter 4: Data Assembly 93 4.1 Introduction 934.2 Tidy Data 934.3 Concatenation 944.4 Merging Multiple Data Sets 1024.5 Conclusion 107  Chapter 5: Missing Data 1095.1 Introduction 1095.2 What Is a NaN Value? 1095.3 Where Do Missing Values Come From? 1115.4 Working with Missing Data 1165.5 Conclusion 121  Chapter 6: Tidy Data 1236.1 Introduction 1236.2 Columns Contain Values, Not Variables 1246.3 Columns Contain Multiple Variables 1286.4 Variables in Both Rows and Columns 1336.5 Multiple Observational Units in a Table (Normalization) 1346.6 Observational Units Across Multiple Tables 1376.7 Conclusion 141   Part III: Data Munging 143  Chapter 7: Data Types 145 7.1 Introduction 1457.2 Data Types 1457.3 Converting Types 1467.4 Categorical Data 1527.5 Conclusion 153  Chapter 8: Strings and Text Data 1558.1 Introduction 1558.2 Strings 1558.3 String Methods 1588.4 More String Methods 1608.5 String Formatting 1618.6 Regular Expressions (RegEx) 1648.7 The regex Library 1708.8 Conclusion 170  Chapter 9: Apply 1719.1 Introduction 1719.2 Functions 1719.3 Apply (Basics) 1729.4 Apply (More Advanced) 1779.5 Vectorized Functions 1829.6 Lambda Functions 1859.7 Conclusion 187  Chapter 10: Groupby Operations: Split–Apply–Combine 18910.1 Introduction 18910.2 Aggregate 19010.3 Transform 19710.4 Filter 20110.5 The pandas.core.groupby.DataFrameGroupBy Object 20210.6 Working with a MultiIndex 20710.7 Conclusion 211  Chapter 11: The datetime Data Type 21311.1 Introduction 21311.2 Python’s datetime Object 21311.3 Converting to datetime 21411.4 Loading Data That Include Dates 21711.5 Extracting Date Components 21711.6 Date Calculations and Timedeltas 22011.7 Datetime Methods 22111.8 Getting Stock Data 22411.9 Subsetting Data Based on Dates 22511.10 Date Ranges 22711.11 Shifting Values 23011.12 Resampling 23711.13 Time Zones 23811.14 Conclusion 240   Part IV: Data Modeling 241  Chapter 12: Linear Models 243 12.1 Introduction 24312.2 Simple Linear Regression 24312.3 Multiple Regression 24712.4 Keeping Index Labels From sklearn 25112.5 Conclusion 252  Chapter 13: Generalized Linear Models 25313.1 Introduction 25313.2 Logistic Regression 25313.3 Poisson Regression 25713.4 More Generalized Linear Models 26013.5 Survival Analysis 26013.6 Conclusion 264  Chapter 14: Model Diagnostics 26514.1 Introduction 26514.2 Residuals 26514.3 Comparing Multiple Models 27014.4 k-Fold Cross-Validation 27514.5 Conclusion 278  Chapter 15: Regularization 27915.1 Introduction 27915.2 Why Regularize? 27915.3 LASSO Regression 28115.4 Ridge Regression 28315.5 Elastic Net 28515.6 Cross-Validation 28715.7 Conclusion 289  Chapter 16: Clustering 29116.1 Introduction 29116.2 k-Means 29116.3 Hierarchical Clustering 29716.4 Conclusion 301   Part V: Conclusion 303  Chapter 17: Life Outside of Pandas 305 17.1 The (Scientific) Computing Stack 30517.2 Performance 30617.3 Going Bigger and Faster 307  Chapter 18: Toward a Self-Directed Learner 30918.1 It’s Dangerous to Go Alone! 30918.2 Local Meetups 30918.3 Conferences 30918.4 The Internet 31018.5 Podcasts 31018.6 Conclusion 311   Part VI: Appendixes 313  Appendix A: Installation 315 A.1 Installing Anaconda 315A.2 Uninstall Anaconda 316  Appendix B: Command Line 317B.1 Installation 317B.2 Basics 318   Appendix C: Project Templates 319  Appendix D: Using Python 321 D.1 Command Line and Text Editor 321D.2 Python and IPython 322D.3 Jupyter 322D.4 Integrated Development Environments (IDEs) 322   Appendix E: Working Directories 325  Appendix F: Environments 327  Appendix G: Install Packages 329 G.1 Updating Packages 330   Appendix H: Importing Libraries 331  Appendix I: Lists 333  Appendix J: Tuples 335  Appendix K: Dictionaries 337  Appendix L: Slicing Values 339  Appendix M: Loops 341   Appendix N: Comprehensions 343  Appendix O: Functions 345 O.1 Default Parameters 347O.2 Arbitrary Parameters 347   Appendix P: Ranges and Generators 349  Appendix Q: Multiple Assignment 351  Appendix R: numpy ndarray 353  Appendix S: Classes 355  Appendix T: Odo: The Shapeshifter 357  Index 359