Data Lakes

Anne Laurent is a Full Professor at the University of Montpellier, France, and teaches at the Polytech Montpellier Engineering School. She is also a member of the LIRMM laboratory at the University of Montpellier, where she works on the semantic web, data mining, data warehousing, data lakes and fuzzy logic. Dominique Laurent is Emeritus Professor at Cergy-Pontoise University, France. He is a member of the ETIS-CNRS laboratory and his main research interests include database theory, database updates, data mining and data warehousing. Cédrine Madera is an Executive Information Architect at IBM, France. She is a doctor in Data Science and, in close collaboration with the world of academics, she works on the evolution of information systems.

• Preface xiChapter 1. Introduction to Data Lakes: Definitions and Discussions 1Anne LAURENT, Dominique LAURENT and Cédrine MADERA1.1. Introduction to data lakes 11.2. Literature review and discussion 31.3. The data lake challenges 71.4. Data lakes versus decision-making systems 101.5. Urbanization for data lakes 131.6. Data lake functionalities 171.7. Summary and concluding remarks 20Chapter 2. Architecture of Data Lakes 21Houssem CHIHOUB, Cédrine MADERA, Christoph QUIX and Rihan HAI2.1. Introduction 212.2. State of the art and practice 252.2.1. Definition 252.2.2. Architecture 252.2.3. Metadata 262.2.4. Data quality 272.2.5. Schema-on-read 272.3. System architecture 282.3.1. Ingestion layer 292.3.2. Storage layer 312.3.3. Transformation layer 322.3.4. Interaction layer 332.4. Use case: the Constance system 332.4.1. System overview 332.4.2. Ingestion layer 352.4.3. Maintenance layer 352.4.4. Query layer 372.4.5. Data quality control 382.4.6. Extensibility and flexibility 382.5. Concluding remarks 39Chapter 3. Exploiting Software Product Lines and Formal Concept Analysis for the Design of Data Lake Architectures 41Marianne HUCHARD, Anne LAURENT, Thérèse LIBOUREL, Cédrine MADERA and André MIRALLES3.1. Our expectations 413.2. Modeling data lake functionalities 433.3. Building the knowledge base of industrial data lakes 463.4. Our formalization approach 493.5. Applying our approach 513.6. Analysis of our first results 533.7. Concluding remarks 55Chapter 4. Metadata in Data Lake Ecosystems 57Asma ZGOLLI, Christine COLLET† and Cédrine MADERA4.1. Definitions and concepts 574.2. Classification of metadata by NISO 584.2.1. Metadata schema 594.2.2. Knowledge base and catalog 604.3. Other categories of metadata 614.3.1. Business metadata 614.3.2. Navigational integration 634.3.3. Operational metadata 634.4. Sources of metadata 644.5. Metadata classification 654.6. Why metadata are needed 704.6.1. Selection of information (re)sources 704.6.2. Organization of information resources 704.6.3. Interoperability and integration 704.6.4. Unique digital identification 714.6.5. Data archiving and preservation 714.7. Business value of metadata 724.8. Metadata architecture 754.8.1. Architecture scenario 1: point-to-point metadata architecture 754.8.2. Architecture scenario 2: hub and spoke metadata architecture 764.8.3. Architecture scenario 3: tool of record metadata architecture 784.8.4. Architecture scenario 4: hybrid metadata architecture 794.8.5. Architecture scenario 5: federated metadata architecture 804.9. Metadata management 824.10. Metadata and data lakes 864.10.1. Application and workload layer 864.10.2. Data layer 884.10.3. System layer 904.10.4. Metadata types 904.11. Metadata management in data lakes 924.11.1. Metadata directory 934.11.2. Metadata storage 934.11.3. Metadata discovery 944.11.4. Metadata lineage 944.11.5. Metadata querying 954.11.6. Data source selection 954.12. Metadata and master data management 964.13. Conclusion 96Chapter 5. A Use Case of Data Lake Metadata Management 97Imen MEGDICHE, Franck RAVAT and Yan ZHAO5.1. Context 975.1.1. Data lake definition 985.1.2. Data lake functional architecture 1005.2. Related work 1035.2.1. Metadata classification 1045.2.2. Metadata management 1055.3. Metadata model 1065.3.1. Metadata classification 1065.3.2. Schema of metadata conceptual model 1105.4. Metadata implementation 1115.4.1. Relational database 1125.4.2. Graph database 1155.4.3. Comparison of the solutions 1195.5. Concluding remarks 121Chapter 6. Master Data and Reference Data in Data Lake Ecosystems 123Cédrine MADERA6.1. Introduction to master data management 1256.1.1. What is master data? 1256.1.2. Basic definitions 1256.2. Deciding what to manage 1266.2.1. Behavior 1266.2.2. Lifecycle 1276.2.3. Cardinality 1276.2.4. Lifetime 1286.2.5. Complexity 1286.2.6. Value 1286.2.7. Volatility 1296.2.8. Reuse 1296.3. Why should I manage master data? 1306.4. What is master data management? 1316.4.1. How do I create a master list? 1366.4.2. How do I maintain a master list? 1386.4.3. Versioning and auditing 1396.4.4. Hierarchy management 1406.5. Master data and the data lake 1416.6. Conclusion 143Chapter 7. Linked Data Principles for Data Lakes 145Alessandro ADAMOU and Mathieu D’AQUIN7.1. Basic principles 1457.2. Using Linked Data in data lakes 1487.2.1. Distributed data storage and querying with linked data graphs 1517.2.2. Describing and profiling data sources 1537.2.3. Integrating internal and external data 1567.3. Limitations and issues 1597.4. The smart cities use case 1627.4.1. The MK Data Hub 1637.4.2. Linked data in the MK Data Hub 1657.5. Take-home message 169Chapter 8. Fog Computing 171Arnault IOUALALEN8.1. Introduction 1718.2. A little bit of context 1718.3. Every machine talks 1728.4. The volume paradox 1738.5. The fog, a shift in paradigm 1748.6. Constraint environment challenges 1768.7. Calculations and local drift 1778.7.1. A short memo about computer arithmetic 1788.7.2. Instability from within 1798.7.3. Non-determinism from outside 1808.8. Quality is everything 1818.9. Fog computing versus cloud computing and edge computing 1848.10. Concluding remarks: fog computing and data lake 185Chapter 9. The Gravity Principle in Data Lakes 187Anne LAURENT, Thérèse LIBOUREL, Cédrine MADERA and André MIRALLES9.1. Applying the notion of gravitation to information systems 1879.1.1. Universal gravitation 1879.1.2. Gravitation in information systems 1899.2. Impact of gravitation on the architecture of data lakes 1939.2.1. The case where data are not moved 1959.2.2. The case where processes are not moved 1979.2.3. The case where the environment blocks the move 198Glossary 201References 207List of Authors 217Index 219