Data Science and Big Data Analytics

EMC is a global leader in enabling businesses and service providers to transform their operations and deliver IT as a service. Fundamental to this transformation is cloud computing. Through innovative products and services, EMC accelerates the journey to cloud computing, helping IT departments to store, manage, protect and analyze their most valuable asset — information — in a more agile, trusted and cost-efficient way. Additional information about EMC can be found at www.EMC.com

• Introduction xviiChapter 1 Introduction to Big Data Analytics 11.1 Big Data Overview 21.1.1 Data Structures 51.1.2 Analyst Perspective on Data Repositories 91.2 State of the Practice in Analytics 111.2.1 BI Versus Data Science 121.2.2 Current Analytical Architecture 131.2.3 Drivers of Big Data 151.2.4 Emerging Big Data Ecosystem and a New Approach to Analytics 161.3 Key Roles for the New Big Data Ecosystem 191.4 Examples of Big Data Analytics 22Summary 23Exercises 23Bibliography 24Chapter 2 Data Analytics Lifecycle 252.1 Data Analytics Lifecycle Overview 262.1.1 Key Roles for a Successful Analytics Project 262.1.2 Background and Overview of Data Analytics Lifecycle 282.2 Phase 1: Discovery 302.2.1 Learning the Business Domain 302.2.2 Resources 312.2.3 Framing the Problem 322.2.4 Identifying Key Stakeholders 332.2.5 Interviewing the Analytics Sponsor 332.2.6 Developing Initial Hypotheses 352.2.7 Identifying Potential Data Sources 352.3 Phase 2: Data Preparation 362.3.1 Preparing the Analytic Sandbox 372.3.2 Performing ETLT 382.3.3 Learning About the Data 392.3.4 Data Conditioning 402.3.5 Survey and Visualize 412.3.6 Common Tools for the Data Preparation Phase 422.4 Phase 3: Model Planning 422.4.1 Data Exploration and Variable Selection 442.4.2 Model Selection 452.4.3 Common Tools for the Model Planning Phase 452.5 Phase 4: Model Building 462.5.1 Common Tools for the Model Building Phase 482.6 Phase 5: Communicate Results 492.7 Phase 6: Operationalize 502.8 Case Study: Global Innovation Network and Analysis (GINA) 532.8.1 Phase 1: Discovery 542.8.2 Phase 2: Data Preparation 552.8.3 Phase 3: Model Planning 562.8.4 Phase 4: Model Building 562.8.5 Phase 5: Communicate Results 582.8.6 Phase 6: Operationalize 59Summary 60Exercises 61Bibliography 61Chapter 3 Review of Basic Data Analytic Methods Using R 633.1 Introduction to R 643.1.1 R Graphical User Interfaces 673.1.2 Data Import and Export 693.1.3 Attribute and Data Types 713.1.4 Descriptive Statistics 793.2 Exploratory Data Analysis 803.2.1 Visualization Before Analysis 823.2.2 Dirty Data 853.2.3 Visualizing a Single Variable 883.2.4 Examining Multiple Variables 913.2.5 Data Exploration Versus Presentation 993.3 Statistical Methods for Evaluation 1013.3.1 Hypothesis Testing 1023.3.2 Difference of Means 1043.3.3 Wilcoxon Rank-Sum Test 1083.3.4 Type I and Type II Errors 1093.3.5 Power and Sample Size 1103.3.6 ANOVA 110Summary 114Exercises 114Bibliography 115Chapter 4 Advanced Analytical Theory and Methods: Clustering 1174.1 Overview of Clustering 1184.2 K-means 1184.2.1 Use Cases 1194.2.2 Overview of the Method 1204.2.3 Determining the Number of Clusters 1234.2.4 Diagnostics 1284.2.5 Reasons to Choose and Cautions 1304.3 Additional Algorithms 134Summary 135Exercises 135Bibliography 136Chapter 5 Advanced Analytical Theory and Methods: Association Rules 1375.1 Overview 1385.2 Apriori Algorithm 1405.3 Evaluation of Candidate Rules 1415.4 Applications of Association Rules 1435.5 An Example: Transactions in a Grocery Store 1435.5.1 The Groceries Dataset 1445.5.2 Frequent Itemset Generation 1465.5.3 Rule Generation and Visualization 1525.6 Validation and Testing 1575.7 Diagnostics 158Summary 158Exercises 159Bibliography 160Chapter 6 Advanced Analytical Theory and Methods: Regression 1616.1 Linear Regression 1626.1.1 Use Cases 1626.1.2 Model Description 1636.1.3 Diagnostics 1736.2 Logistic Regression 1786.2.1 Use Cases 1796.2.2 Model Description 1796.2.3 Diagnostics 1816.3 Reasons to Choose and Cautions 1886.4 Additional Regression Models 189Summary 190Exercises 190Chapter 7 Advanced Analytical Theory and Methods: Classification 1917.1 Decision Trees 1927.1.1 Overview of a Decision Tree 1937.1.2 The General Algorithm 1977.1.3 Decision Tree Algorithms 2037.1.4 Evaluating a Decision Tree 2047.1.5 Decision Trees in R 2067.2 Naïve Bayes 2117.2.1 Bayes’ Theorem 2127.2.2 Naïve Bayes Classifier 2147.2.3 Smoothing 2177.2.4 Diagnostics 2177.2.5 Naïve Bayes in R 2187.3 Diagnostics of Classifiers 2247.4 Additional Classification Methods 228Summary 229Exercises 230Bibliography 231Chapter 8 Advanced Analytical Theory and Methods: Time Series Analysis 2338.1 Overview of Time Series Analysis 2348.1.1 Box-Jenkins Methodology 2358.2 ARIMA Model 2368.2.1 Autocorrelation Function (ACF) 2368.2.2 Autoregressive Models 2388.2.3 Moving Average Models 2398.2.4 ARMA and ARIMA Models 2418.2.5 Building and Evaluating an ARIMA Model 2448.2.6 Reasons to Choose and Cautions 2528.3 Additional Methods 253Summary 254Exercises 254Chapter 9 Advanced Analytical Theory and Methods: Text Analysis 2559.1 Text Analysis Steps 2579.2 A Text Analysis Example 2599.3 Collecting Raw Text 2609.4 Representing Text 2649.5 Term Frequency—Inverse Document Frequency (TFIDF) 2699.6 Categorizing Documents by Topics 2749.7 Determining Sentiments 2779.8 Gaining Insights 283Summary 290Exercises 290Bibliography 291Chapter 10 Advanced Analytics—Technology and Tools: MapReduce and Hadoop 29510.1 Analytics for Unstructured Data 29610.1.1 Use Cases 29610.1.2 MapReduce 29810.1.3 Apache Hadoop 30010.2 The Hadoop Ecosystem 30610.2.1 Pig 30610.2.2 Hive 30810.2.3 HBase 31110.2.4 Mahout 31910.3 NoSQL 322Summary 323Exercises 324Bibliography 324Chapter 11 Advanced Analytics—Technology and Tools: In-Database Analytics 32711.1 SQL Essentials 32811.1.1 Joins 33011.1.2 Set Operations 33211.1.3 Grouping Extensions 33411.2 In-Database Text Analysis 33811.3 Advanced SQL 34311.3.1 Window Functions 34311.3.2 User-Defined Functions and Aggregates 34711.3.3 Ordered Aggregates 35111.3.4 MADlib 352Summary 356Exercises 356Bibliography 357Chapter 12 The Endgame, or Putting It All Together 35912.1 Communicating and Operationalizing an Analytics Project 36012.2 Creating the Final Deliverables 36212.2.1 Developing Core Material for Multiple Audiences 36412.2.2 Project Goals 36512.2.3 Main Findings 36712.2.4 Approach 36912.2.5 Model Description 37112.2.6 Key Points Supported with Data 37212.2.7 Model Details 37212.2.8 Recommendations 37412.2.9 Additional Tips on Final Presentation 37512.2.10 Providing Technical Specifications and Code 37612.3 Data Visualization Basics 37712.3.1 Key Points Supported with Data 37812.3.2 Evolution of a Graph 38012.3.3 Common Representation Methods 38612.3.4 How to Clean Up a Graphic 38712.3.5 Additional Considerations 392Summary 393Exercises 394References and Further Reading 394Bibliography 394Index 397