Data Engineering for Beginners

CHISOM NWOKWU, is a Big-Data Engineer, Multi-Published Author, and Creator specialising in the design and development of scalable data platforms for teams. She’s an Azure Certified Data Engineer Associate who has worked with large international firms, including Microsoft and Bank of America.

• Foreword xxiIntroduction xxiiiChapter 1 Understanding Data 1A Brief History of Data 2Data in 19,000 bce: The Great Baboon and Abacus 2Data in the 1600s: Public Health Statistics 2Data in the 1800s: The U.S. Census 3Data in the 1900s: The Concept of Storage 3Data in the 1990s: Data and the Internet 4Types of Data 4Structured Data 4Unstructured Data 5Semi-structured Data 6Why Is Data Important? 7Healthcare 7Supply Chain 8Transportation and Logistics 8Artificial Intelligence 9Data and Information 9Summary 10Notes 11Chapter 2 Introduction to Data Engineering 13Data Engineering Explained Using an Oil Refinery Analogy 14An Overview of the Data Engineering Life Cycle 15Data Storage 16Data Ingestion 20Data Transformation 21Data Serving 22Navigating Project Requirements, Engaging Stakeholders, and Delivering Business Value 24Requirements Gathering 24Understanding Stakeholders 24Understanding System Requirements 26Delivering Business Value 28The Current State of Data Engineering 28The Importance of Data Engineering 29Summary 30Chapter 3 Database Fundamentals 33Key Concepts of Databases 34Rows 34Columns 34Schema 35Keys 35Types of Databases 35Relational Databases 36NoSQL Databases 47Choosing Between Relational and NoSQL Databases 55Start With Your Data’s Structure 55Think About the Relationships in Your Data 55How Fast Do You Need to Move? 55How Do You Need to Query Your Data? 55Scaling and Performance 56Transaction and Strong Consistency Needs 56Summary 56Chapter 4 SQL Fundamentals 59Introduction to SQL 60Basic SQL Clauses 60Comparison Operators 62LIKE Statement 63IN Statement 64BETWEEN Statement 64AND Statement 65OR Statement 65NOT Statement 66IS NULL and IS NOT NULL Statements 66Sorting and Limiting 67Aggregate Functions 68Sum() 69Avg() 69MAX() and MIN() 69Group by 70Having 71Understanding Joins 72Inner Join 72Left Join 73Right Join 74Full Outer Join 75Subqueries 76Common Table Expressions (CTEs) 77Set Operations 78Window Functions 80Lab: Setting Up SQL Server and Running SQL Queries 85Best Practices for Writing Efficient SQL Queries 87Summary 88Chapter 5 Database Design 91Data Modeling 92Why Do We Need to Model Data? 92Types of Data Modeling 93Normalization 100Rules of Normalization 102Downsides of Normalization 109Denormalization 110Data Modeling Best Practices 111Define the Grain 111Normalize Now, Denormalize Later 112Choose the Right Data Types 112Proper Naming Conventions 113Database Optimization 114Indexing 114Partitioning 115Sharding 116Views 118Summary 120Chapter 6 Data Warehouses, Data Lakes, and Data Lakehouses 123Data Warehouses 124Extract, Transform, and Load (ETL) 126Schema Design 127Snowflake Schema 132Slowly Changing Dimensions 134Data Marts 138Benefits of a Data Mart 138Challenges with Data Marts 138Data Lakes 139How Do Data Lakes Work? 139Challenges of Data Lakes 142Data Lakehouse 142Features of a Data Lakehouse 143Data Lakehouse Architecture 143The Key Differences Between a Database, Data Warehouse, Data Lake, and Data Lakehouse 144Summary 145Chapter 7 Data Pipelines 147Batch Pipelines 148Components of a Batch Pipeline 148ETL Pipelines vs. ELT Pipelines 151Stream Pipelines 152How Would This Work? 152Components of a Streaming Data Pipeline 153Lambda Architecture 164Components of the Lambda Architecture 165Advantages of the Lambda Architecture 166Challenges and Trade-offs 166Data Orchestration 167Directed Acyclic Graphs (DAGs) 168Scheduling and Automation 170Monitoring 171Alerts 172Lab: Building an ETL Pipeline and Automating with Apache Airflow 173Requirements 174Set Up Your Development Environment 174Extracting Data from CSV 176Transforming the Data 177Load the New CSV File into a Postgres Database Instance 181Schedule ETL Pipeline with Apache Airflow 182Summary 185Chapter 8 Data Quality 187Bad Data 188Dimensions of Data Quality 190Accuracy 191Completeness 191Consistency 194Validity 195Uniqueness 196Timeliness 198Accessibility 198Relevance 198Data Quality Hierarchy 199Data Quality Best Practices 200Summary 201Chapter 9 Data Security 203What Is Data Security? 204Common Threats to Data Security 205Core Principles of Data Security 206Confidentiality 206Integrity 207Availability 208Data Encryption 209Symmetric Encryption 209Asymmetric Encryption 210Data Masking 211Understanding Network Security 214Access Control 216Authentication 217Authorization 219The Principle of Least Privilege 222Access Levels 224Secrets Management 225Data Security and Data Privacy 225Summary 226Chapter 10 Data Governance 229How to Think About Data Governance 230Data Governance Framework 232Policies 233Regulatory Compliance Policy 234Data Classification Policy 238Data Retention and Disposal Policy 239Data Sharing Policy 240Processes 241Metadata Management 242Data Lineage 244Incident Management 244Master Data Management 246Roles in the Data Governance Framework 247Data Owner 248Data Steward 248Data Custodian 249Chief Data Officer (CDO) 249Data Management and Data Governance 250Summary 250Chapter 11 Big Data and Distributed Systems 253The Five V’s of Big Data 254Volume 255Velocity 255Variety 255Veracity 256Value 256Distributed Systems 256Scalability 258Fault Tolerance 259Reliability 260Concurrency 260Resource Management 260Consistency 261Availability 261Load Balancing 261Latency 262Distributed Data Processing 262Apache Hadoop 262Big Data File Types 272Avro 272Parquet 273Optimized Row Columnar (ORC) 274Choosing the File Type 275Summary 276Chapter 12 Data Engineering on the Cloud 279Cloud Computing 280On-Premises 281Cloud 281Making the Right Choice 282Core Cloud Concepts 282Storage 282Compute 286Networking 287Cloud Service Models 291Infrastructure as a Service 291Platform as a Service 292Software as a Service 293Choosing Between IaaS, PaaS, and SaaS 294A Hybrid Approach 298Cloud Management Models 298Serverless 299Managed 300Self-Managed 301Putting It All Together 302Cost Optimization 302Understanding Cloud Pricing Models 302Rightsizing Resources 303Smart Job Scheduling 304Storage Optimization 304Shutting Down Idle Resources 304Use Serverless Where Possible 304Monitoring and Alerting 305Summary 305Chapter 13 Building a Career in Data Engineering 307Types of Data Engineering Roles 308Types of Data Engineers 308Platform Data Engineer 308Analytics Data Engineer 310AI/ML Data Engineers 310Landing Your First Data Engineering Role 312A Typical Data Engineering Job Description 312How to Build a Winning Résumé 314Preparing for a Data Engineering Interview 316Thinking Like a Data Engineer 321Think in Systems 321Learn to Prioritize Data Quality 321Design for Failure 321Balance Business Context with Technical Choices 322Optimize for Clarity, Then Speed 322Think Beyond the Tool 322Master Automation 322Summary 323Appendix Sample Interview Questions 325SQL 325Data Modeling 328Data Pipelines 330Apache Spark 332System Design 333Data Engineering Glossary 335Index 347