​Scott Haines is a full stack engineer with a current focus on real-time, highly available, trustworthy analytics systems. He works at Twilio as a Principal Software Engineer on the Voice Insights team, where he helps drive Spark adoption, creates streaming pipeline architectures, and helps to architect and build out a massive stream and batch processing platform.Prior to Twilio, Scott worked writing the backend Java APIs for Yahoo Games as well as the real-time game ranking and ratings engine (built on Storm) to provide personalized recommendations and page views for 10 million customers. He finished his tenure at Yahoo working for Flurry Analytics where he wrote the alerts and notifications system for mobile devices.

• Part I. The Fundamentals of Data Engineering with Spark.- 1. Introduction to Modern Data Engineering.- 2. Getting Started with Apache Spark.- 3. Working with Data.- 4. Transforming Data with Spark SQL and the  DataFrame API.- 5. Bridging Spark SQL with JDBC.- 6. Data Discovery and the Spark SQL Catalog.- 7. Data Pipelines & Structured Spark Applications.- Part II. The Streaming Pipeline Ecosystem.- 8. Workflow Orchestration with Apache Airflow.- 9. A Gentle Introduction to Stream Processing.- 10. Patterns for Writing Structured Streaming Applications.- 11. Apache Kafka & Spark Structured Streaming.- 12. Analytical Processing & Insights.- Part III. Advanced Techniques.- 13. Advanced Analytics with Spark Stateful Structured Streaming.- 14. Deploying Mission Critical Spark Applications on Spark Standalone.- 15. Deploying Mission Critical Spark Applications on Kubernetes.

​Scott Haines is a full stack engineer with a current focus on real-time, highly available, trustworthy analytics systems. He works at Twilio as a Principal Software Engineer on the Voice Insights team, where he helps drive Spark adoption, creates streaming pipeline architectures, and helps to architect and build out a massive stream and batch processing platform.Prior to Twilio, Scott worked writing the backend Java APIs for Yahoo Games as well as the real-time game ranking and ratings engine (built on Storm) to provide personalized recommendations and page views for 10 million customers. He finished his tenure at Yahoo working for Flurry Analytics where he wrote the alerts and notifications system for mobile devices.

• Part I. The Fundamentals of Data Engineering with Spark.- 1. Introduction to Modern Data Engineering.- 2. Getting Started with Apache Spark.- 3. Working with Data.- 4. Transforming Data with Spark SQL and the  DataFrame API.- 5. Bridging Spark SQL with JDBC.- 6. Data Discovery and the Spark SQL Catalog.- 7. Data Pipelines & Structured Spark Applications.- Part II. The Streaming Pipeline Ecosystem.- 8. Workflow Orchestration with Apache Airflow.- 9. A Gentle Introduction to Stream Processing.- 10. Patterns for Writing Structured Streaming Applications.- 11. Apache Kafka & Spark Structured Streaming.- 12. Analytical Processing & Insights.- Part III. Advanced Techniques.- 13. Advanced Analytics with Spark Stateful Structured Streaming.- 14. Deploying Mission Critical Spark Applications on Spark Standalone.- 15. Deploying Mission Critical Spark Applications on Kubernetes.

​Scott Haines is a full stack engineer with a current focus on real-time, highly available, trustworthy analytics systems. He works at Twilio as a Principal Software Engineer on the Voice Insights team, where he helps drive Spark adoption, creates streaming pipeline architectures, and helps to architect and build out a massive stream and batch processing platform.Prior to Twilio, Scott worked writing the backend Java APIs for Yahoo Games as well as the real-time game ranking and ratings engine (built on Storm) to provide personalized recommendations and page views for 10 million customers. He finished his tenure at Yahoo working for Flurry Analytics where he wrote the alerts and notifications system for mobile devices.

• Part I. The Fundamentals of Data Engineering with Spark.- 1. Introduction to Modern Data Engineering.- 2. Getting Started with Apache Spark.- 3. Working with Data.- 4. Transforming Data with Spark SQL and the  DataFrame API.- 5. Bridging Spark SQL with JDBC.- 6. Data Discovery and the Spark SQL Catalog.- 7. Data Pipelines & Structured Spark Applications.- Part II. The Streaming Pipeline Ecosystem.- 8. Workflow Orchestration with Apache Airflow.- 9. A Gentle Introduction to Stream Processing.- 10. Patterns for Writing Structured Streaming Applications.- 11. Apache Kafka & Spark Structured Streaming.- 12. Analytical Processing & Insights.- Part III. Advanced Techniques.- 13. Advanced Analytics with Spark Stateful Structured Streaming.- 14. Deploying Mission Critical Spark Applications on Spark Standalone.- 15. Deploying Mission Critical Spark Applications on Kubernetes.