Big Data Revolution

Rob Thomas is Vice President of Product Development for Big Data and Information Management in IBM Software Group. Previously, he had responsibility for global sales and mergers & acquisitions. Patrick McSharry is a Senior Research Fellow at the Smith School of Enterprise and the Environment, Faculty Member of the Oxford Man Institute of Quantitative Finance at Oxford University and Visiting Professor at the Department of Electrical and Computer Engineering, Carnegie Mellon University.

• Prologue 1Berkeley, 1930s 1Pattern Recognition 2Nelson Peltz 3Committing to One Percent 5The Big Data Revolution 6Introduction 7Storytelling 7Objective 7Outline 8Part I “The Revolution Starts Now: 9 Industries Transforming with Data” 8Part II “Learning from Patterns in Big Data” 11Part III “Leading the Revolution” 11Storytelling (Continued) 13Part I: the Revolution Starts Now: 9 Industries Transforming With Data 15Chapter 1: Transforming Farms with Data 17California, 2013 17Brief History of Farming 18The Data Era 19Potato Farming 20Precision Farming 21Capturing Farm Data 22Deere & Company Versus Monsanto 24Integrated Farming Systems 25Data Prevails 26The Climate Corporation 26Growsafe Systems 27Farm of the Future 27California, 2013 (Continued) 29Chapter 2: Why Doctors Will Have Math Degrees 31United States, 2014 31The History of Medical Education 32Scientific Method 32Rise of Specialists 33We Have a Problem 34Ben Goldacre 35Vinod Khosla 35The Data Era 36Collecting Data 36Telemedicine 38Innovating with Data 40Implications of a Data-Driven Medical World 42The Future of Medical School 42A Typical Medical School 42A Medical School for the Data Era 43United States, 2030 44Chapter 3: Revolutionizing Insurance: Why Actuaries Will Become Data Scientists 45Middle of Somewhere, 2012 45Short History of Property & Casualty Insurance and Underwriting 46Actuarial Science In Insurance 47Pensions, Insurance, Leases 49Compound Interest 50Probability 50Mortality Data 50Modern-Day Insurance 51Eight Weeks to Eight Days 51Online Policies 52The Data Era 52Dynamic Risk Management 52Catastrophe Risk 54Open Access Modeling 55Opportunities 56Middle of Somewhere, 2012 (Continued) 58Chapter 4: Personalizing Retail and Fashion 59Karolina 59A Brief History of Retail 60Retail Eras 60Aristide Boucicaut 61The Shift 62The Data Era 63Stitch Fix 63Keaton Row 65Zara 66Karolina (Continued) 67Chapter 5: Transforming Customer Relationships with Data 69Buying a House 69Brief History of Customer Service 70Customer Service Over Time 70Boeing 72Financial Services 74The Data Era 75An Automobile Manufacturer 76Zendesk 76Buying a House (Continued) 77Chapter 6: Intelligent Machines 79Denmark 79Intelligent Machines 80Machine Data 81The Data Era 82General Electric 82Drones 84Tesla 86Networks of Data 87Denmark (Continued) 88Chapter 7: Government and Society 89Egypt, 2011 89Social Media 90Intelligence 90Snowden Effect 91Privacy Risk Versus Reward 91Observation or Surveillance 93Development Targets 93Open Data 95Hackathons 95Open Access 95Ensuring Personal Protection 96Private Clouds 97Sanitizing Data 97Evidence-Based Policy 97Public-Private Partnerships 98Impact Bonds 101Social Impact Bond 102Development Impact Bonds 103The Role of Big Data 104Egypt, 2011 (Continued) 105Chapter 8: Corporate Sustainability 107City of London 107Global Megaforces 109Population 109Carbon Footprint 110Water Scarcity 110Environmental Risk 111BP and Exxon Mobile 111Early Warning Systems 112Social Media 113Risk and Resilience 114Measuring Sustainability 115Long-Term Decision Making 116Stranded Assets 117City of London (Continued) 118Chapter 9: Weather and Energy 119India, 2012 119The Weather 120Forecasting the Weather 120When are Weather Forecasts Wrong? 121Chaos 122Ensemble Forecasts 122Communication 123Renewable Energy 124Solar, Hydro, and Wind Power 124Volatile or Intermittent Supply 125Energy Consumption 126Smart Meters 127Intelligent Demand-Side Management 128India, 2012 (Continued) 129Part II: Learning From Patterns in Big Data 131Chapter 10: Pattern Recognition 133Elements of Success Rhyme 133Pattern Recognition: A Gift or Trap? 134What Fish Teach Us About Pattern Recognition 135Bayes’ Theorem 135Tsukiji Market 135Pattern Recognition 137Rochester Institute of Technology 137A Method for Recognizing Patterns 137Elements of Success Rhyme (Continued) 140Chapter 11: Why Patterns in Big Data Have Emerged 141Meatpacking District 141Business Models in the Data Era 142Data as a Competitive Advantage 143Data Improves Existing Products or Services 145Data as the Product 145Dun & Bradstreet 146CoStar 148Ihs 149Meatpacking District (Continued) 151Chapter 12: Patterns in Big Data 153The Data Factor 154Summary of Big Data Patterns 155Redefining a Skilled Worker 155Creating and Utilizing New Sources of Data 156Building New Data Applications 157Transforming and Creating New Business Processes 157Data Collection for Competitive Advantage 158Exposing Opinion-Based Biases 159Real-Time Monitoring and Decision Making 159Social Networks Leveraging and Creating Data 160Deconstructing the Value Chain 161New Product Offerings 161Building for Customers Instead of Markets 162Tradeoff Between Privacy and Insight 163Changing the Definition of a Product 163Inverting the Search Paradigm for Data Discovery 164Data Security 165New Partnerships Founded on Data 165Shortening the Innovation Lifecycle 166Defining New Channels to Market 166New Economic Models 167Forecasting and Predicting Future Events 168Changing Incentives 168New Partnerships (Public/Private) 169Real-Time Monitoring and Decision Making (Early Warning Systems) 169A Framework for Big Data Patterns 170Part III: Leading the Revolution 171Chapter 13: The Data Opportunity 173What Oil Teaches Us About Data 173Bain Study 175Seizing the Opportunity 176Chapter 14: Porsche 177Rome 177Ferdinand Porsche 178The Birth of Porsche 178The Porsche Sports Car 179Porsche Today 180Rome (Continued) 180Chapter 15: Puma 181Herzogenaurach 181Advertising Wars 182Jochen Zeitz 182Environmental Profit and Loss 183Herzogenaurach (Continued) 184Chapter 16: A Methodology for Applying Big Data Patterns 185Introduction 185The Method 186Step 1: Understand Data Assets 187The Patterns 188Step 2: Explore Data 191Challenges 192Questions 192Hypotheses 193Data 193Models 193Statistical Significance 194Step 3: Design the Future 194The Patterns 195Step 4: Design a Data-Driven Business Model 197The Patterns 197Step 5: Transform Business Processes for the Data Era 199The Patterns 199Step 6: Design for Governance and Security 201The Patterns 201Step 7: Share Metrics and Incentives 202Chapter 17: Big Data Architecture 205Introduction 205Architect for the Future 206Lessons from Stuttgart 207Big Data Reference Architectures 207Leveraging Investments in Architecture 208Big Data Reference Architectures 211Business View 212Logical View 213Chapter 18: Business View Reference Architecture 215Introduction 215Men’s Trunk: A Retailer in the Data Era 216The Business View Reference Architecture 217Answer Fabric 218Data Virtualization 219Data Engines 220Management 221Data Governance 221User Interface, Applications, and Business Processes 222Summary 222Chapter 19: Logical View Reference Architecture 223Introduction 223Men’s Trunk: A Retailer in the Data Era (Continued) 224The Logical View Reference Architecture 226Data Ingest 227Analytics 227Discovery 228Landing 228Operational Warehouse 229Information Insight 230Operational Data 231Governance 231Men’s Trunk: A Retailer in the Data Era (Continued) 232Chapter 20: The Architecture of the Future 233Men’s Trunk: A Retailer in the Data Era (Continued) 233Men’s Trunk: Applying the Methodology 235Step 1: Understand Data Assets 235Step 2: Explore the Data 236Step 3: Design the Future 237Step 4: Design a Data-Driven Business Model 237Step 5: Transform Business Processes for the Data Era 237Step 6: Design for Governance and Security 237Step 7: Share Metrics and Incentives 238Men’s Trunk: The Business View Reference Architecture 239Answer Fabric 240Data Virtualization 241Data Engines 241Management 242Data Governance 242User Interface, Applications, and Business Processes 243Men’s Trunk: The Logical View Reference Architecture 244Approach 244Men’s Trunk: A Retailer in the Data Era (Continued) 248Epilogue 249The Time is Now 249Taking Action 250Fear not Usual Competitors 251The Future 252Index 255