Big Data Fundamentals

Thomas Erl is a top-selling IT author, founder of Arcitura Education and series editor of the Prentice Hall Service Technology Series from Thomas Erl. With more than 200,000 copies in print worldwide, his books have become international bestsellers and have been formally endorsed by senior members of major IT organizations, such as IBM, Microsoft, Oracle, Intel, Accenture, IEEE, HL7, MITRE, SAP, CISCO, HP and many others. As CEO of Arcitura Education Inc., Thomas has led the development of curricula for the internationally recognized Big Data Science Certified Professional (BDSCP), Cloud Certified Professional (CCP) and SOA Certified Professional (SOACP) accreditation programs, which have established a series of formal, vendor-neutral industry certifications obtained by thousands of IT professionals around the world. Thomas has toured more than 20 countries as a speaker and instructor. More than 100 articles and interviews by Thomas have been published in numerous publications, including The Wall Street Journal and CIO Magazine. Wajid Khattak is a Big Data researcher and trainer at Arcitura Education Inc. His areas of interest include Big Data engineering and architecture, data science, machine learning, analytics and SOA. He has extensive .NET software development experience in the domains of business intelligence reporting solutions and GIS. Wajid completed his MSc in Software Engineering and Security with distinction from Birmingham City University in 2008. Prior to that, in 2003, he earned his BSc (Hons) degree in Software Engineering from Birmingham City University with first-class recognition. He holds MCAD & MCTS (Microsoft), SOA Architect, Big Data Scientist, Big Data Engineer and Big Data Consultant (Arcitura) certifications. Dr. Paul Buhler is a seasoned professional who has worked in commercial, government and academic environments. He is a respected researcher, practitioner and educator of service-oriented computing concepts, technologies and implementation methodologies. His work in XaaS naturally extends to cloud, Big Data and IoE areas. Dr. Buhler’s more recent work has been focused on closing the gap between business strategy and process execution by leveraging responsive design principles and goal-based execution. As Chief Scientist at Modus21, Dr. Buhler is responsible for aligning corporate strategy with emerging trends in business architecture and process execution frameworks. He also holds an Affiliate Professorship at the College of Charleston, where he teaches both graduate and undergraduate computer science courses. Dr. Buhler earned his Ph.D. in Computer Engineering at the University of South Carolina. He also holds an MS degree in Computer Science from Johns Hopkins University and a BS in Computer Science from The Citadel.

• Acknowledgments     xviiReader Services     xviiiPART I: THE FUNDAMENTALS OF BIG DATAChapter 1: Understanding Big Data     3 Concepts and Terminology     5 Datasets     5 Data Analysis     6 Data Analytics     6 Descriptive Analytics     8 Diagnostic Analytics     9 Predictive Analytics     10 Prescriptive Analytics     11 Business Intelligence (BI)     12 Key Performance Indicators (KPI)     12 Big Data Characteristics     13 Volume     14 Velocity     14 Variety     15 Veracity     16 Value     16 Different Types of Data     17 Structured Data     18 Unstructured Data     19 Semi-structured Data     19 Metadata     20 Case Study Background     20 History     20 Technical Infrastructure and Automation Environment     21 Business Goals and Obstacles     22 Case Study Example     24 Identifying Data Characteristics     26 Volume     26 Velocity     26 Variety     26 Veracity     26 Value     27 Identifying Types of Data     27 Chapter 2: Business Motivations and Drivers for Big Data Adoption     29 Marketplace Dynamics     30 Business Architecture     33 Business Process Management     36 Information and Communications Technology     37 Data Analytics and Data Science     37 Digitization     38 Affordable Technology and Commodity Hardware     38 Social Media     39 Hyper-Connected Communities and Devices     40 Cloud Computing     40 Internet of Everything (IoE)     42 Case Study Example     43 Chapter 3: Big Data Adoption and Planning Considerations     47 Organization Prerequisites     49 Data Procurement     49 Privacy     49 Security     50 Provenance     51 Limited Realtime Support     52 Distinct Performance Challenges     53 Distinct Governance Requirements     53 Distinct Methodology     53 Clouds     54 Big Data Analytics Lifecycle     55 Business Case Evaluation     56 Data Identification     57 Data Acquisition and Filtering     58 Data Extraction     60 Data Validation and Cleansing     62 Data Aggregation and Representation     64 Data Analysis     66 Data Visualization     68 Utilization of Analysis Results     69 Case Study Example     71 Big Data Analytics Lifecycle     73 Business Case Evaluation     73 Data Identification     74 Data Acquisition and Filtering     74 Data Extraction     74 Data Validation and Cleansing     75 Data Aggregation and Representation     75 Data Analysis     75 Data Visualization     76 Utilization of Analysis Results     76 Chapter 4: Enterprise Technologies and Big Data Business Intelligence     77 Online Transaction Processing (OLTP)     78 Online Analytical Processing (OLAP)     79 Extract Transform Load (ETL)     79 Data Warehouses     80 Data Marts     81 Traditional BI     82 Ad-hoc Reports     82 Dashboards     82 Big Data BI     84 Traditional Data Visualization     84 Data Visualization for Big Data     85 Case Study Example     86 Enterprise Technology     86 Big Data Business Intelligence     87 PART II: STORING AND ANALYZING BIG DATAChapter 5: Big Data Storage Concepts     91 Clusters     93 File Systems and Distributed File Systems     93 NoSQL     94 Sharding     95 Replication     97 Master-Slave     98 Peer-to-Peer     100 Sharding and Replication     103 Combining Sharding and Master-Slave Replication     104 Combining Sharding and Peer-to-Peer Replication     105 CAP Theorem     106 ACID     108 BASE     113 Case Study Example     117 Chapter 6: Big Data Processing Concepts     119 Parallel Data Processing     120 Distributed Data Processing     121 Hadoop     122 Processing Workloads     122 Batch     123 Transactional     123 Cluster     124 Processing in Batch Mode     125 Batch Processing with MapReduce     125 Map and Reduce Tasks     126 Map     127 Combine     127 Partition     129 Shuffle and Sort     130 Reduce     131 A Simple MapReduce Example     133 Understanding MapReduce Algorithms     134 Processing in Realtime Mode     137 Speed Consistency Volume (SCV)     137 Event Stream Processing     140 Complex Event Processing     141 Realtime Big Data Processing and SCV     141 Realtime Big Data Processing and MapReduce     142 Case Study Example     143 Processing Workloads     143 Processing in Batch Mode     143 Processing in Realtime     144 Chapter 7: Big Data Storage Technology     145 On-Disk Storage Devices     147 Distributed File Systems     147 RDBMS Databases     149 NoSQL Databases     152 Characteristics     152 Rationale     153 Types     154 Key-Value     156 Document     157 Column-Family     159 Graph     160 NewSQL Databases     163 In-Memory Storage Devices     163 In-Memory Data Grids     166 Read-through     170 Write-through     170 Write-behind     172 Refresh-ahead     172 In-Memory Databases     175 Case Study Example     179 Chapter 8: Big Data Analysis Techniques     181 Quantitative Analysis     183 Qualitative Analysis     184 Data Mining     184 Statistical Analysis     184 A/B Testing     185 Correlation     186 Regression     188 Machine Learning     190 Classification (Supervised Machine Learning)     190 Clustering (Unsupervised Machine Learning)     191 Outlier Detection     192 Filtering     193 Semantic Analysis     195 Natural Language Processing     195 Text Analytics     196 Sentiment Analysis     197 Visual Analysis     198 Heat Maps     198 Time Series Plots     200 Network Graphs     201 Spatial Data Mapping     202 Case Study Example     204 Correlation     204 Regression     204 Time Series Plot     205 Clustering     205 Classification     205 Appendix A: Case Study Conclusion     207About the Authors     211 Thomas Erl     211 Wajid Khattak     211 Paul Buhler     212 Index     213