Big Data Systems

Jawwad A. Shamsi completed B.E. (Electrical Engineering) from NED University of Enginnering and Technology, Karachi in 1998. He completed his MS in Computer and Information Sciences from University of Michigan-Dearborn, MI, USA in 2002. In 2009, he completed his PhD. from Wayne State University, MI, USA. He has also worked as a Programmar Analyst in USA from 2000 to 2002. In 2009, he joined FAST- National Univesity of Computer and Emerging Sciences (NUCES), Karachi. He has served as the head of computer science department from 2012 to 2017. Currently, he is serving as a Professor of Computer Science and Director of the Karachi Campus. He also leads a research group - syslab (http://syslab.khi.nu.edu.pk). His research is focused on developing systems which can meet the growing needs of scalability, security, high performance, robustness, and agility. His research has been funded by different International and National agencies including NVIDIA and Higher Education Commission, Pakistan.Muhammad Ali Khojaye has more than decade of industrial experience ranging from the cloud-native side of things to distributed systems design, CI/CD, and infrastructure. His current technical interests revolve around big data, cloud, containers, and large scale systems design. He currently lives in the Glasgow suburbs with his wife and son. When he's not at work, Ali enjoys cycling, travelling, and spending time with family and friends.

• Preface Author Bios Acknowledgements List of Figures List of Tables Introduction to Big Data Systems 1.1 INTRODUCTION: REVIEW OF BIG DATA SYSTEMS1.2 UNDERSTANDING BIG DATA 1.3 TYPE OF DATA: TRANSACTIONAL OR ANALYTICAL1.4 REQUIREMENTS AND CHALLENGES OF BIG DATA 1.5 CONCLUDING REMARKS 1.6 FURTHER READING 1.7 EXERCISE QUESTIONS Architecture and Organization of Big Data Systems 2.1 ARCHITECTURE FOR BIG DATA SYSTEMS 2.2 ORGANIZATION OF BIG DATA SYSTEMS: CLUSTERS2.3 CLASSIFICATION OF CLUSTERS: DISTRIBUTED MEMORY VS. SHARED MEMORY2.4 CONCLUDING REMARKS 2.5 FURTHER READING 2.6 EXERCISE QUESTIONS Cloud Computing for Big Data 3.1 CLOUD COMPUTING 3.2 VIRTUALIZATION 3.3 PROCESSOR VIRTUALIZATION 3.4 CONTAINERIZATION 3.5 VIRTUALIZATION OR CONTAINERIZATION 3.6 FOG COMPUTING 3.7 EXAMPLES 3.8 CONCLUDING REMARKS 3.9 FURTHER READING 3.10 EXERCISE QUESTIONS HADOOP: An Efficient Platform for Storing and Processing Big Data 4.1 REQUIREMENTS FOR PROCESSING AND STORING BIG DATA 4.2 HADOOP - THE BIG PICTURE 4.3 HADOOP DISTRIBUTED FILE SYSTEM 4.4 MAPREDUCE 4.5 HBASE 4.6 CONCLUDING REMARKS 4.7 FURTHER READING 4.8 EXERCISE QUESTIONS Enhancements in Hadoop 5.1 ISSUES WITH HADOOP 5.2 YARN 5.3 PIG 5.4 HIVE 5.5 DREMEL 5.6 IMPALA 5.7 DRILL 5.8 DATA TRANSFER 5.9 AMBARI 5.10 CONCLUDING REMARKS 5.11 FURTHER READING 5.12 EXERCISE QUESTIONS Spark 6.1 LIMITATIONS OF MAPREDUCE 6.2 INTRODUCTION TO SPARK 6.3 SPARK CONCEPTS 6.4 SPARK SQL 6.5 SPARK MLLIB 6.6 STREAM BASED SYSTEM 6.7 SPARK STREAMING 6.8 CONCLUDING REMARKS 6.9 FURTHER READING 6.10 EXERCISE QUESTIONS NoSQL Systems 7.1 INTRODUCTION 7.2 HANDLING BIG DATA SYSTEMS - PARALLEL RDBMS 7.3 EMERGENCE OF NOSQL SYSTEMS 7.4 KEY-VALUE DATABASE 7.5 DOCUMENT-ORIENTED DATABASE 7.6 COLUMN-ORIENTED DATABASE 7.7 GRAPH DATABASE 7.8 CONCLUDING REMARKS 7.9 FURTHER READING 7.10 EXERCISE QUESTIONS NewSQL Systems 8.1 INTRODUCTION8.2 TYPES OF NEWSQL SYSTEMS 8.3 FEATURES 8.4 NEWSQL SYSTEMS: CASE STUDIES 8.5 CONCLUDING REMARKS 8.6 FURTHER READING8.7 EXERCISE QUESTIONS Networking for Big Data 9.1 NETWORK ARCHITECTURE FOR BIG DATA SYSTEMS9.2 CHALLENGES AND REQUIREMENTS 9.3 NETWORK PROGRAMMABILITY AND SOFTWARE DEFINED NETWORKING 9.4 LOW LATENCY AND HIGH SPEED DATA TRANSFER9.5 AVOIDING TCP INCAST - ACHIEVING LOW LATENCYAND HIGH THROUGHPUT 9.6 FAULT TOLERANCE9.7 CONCLUDING REMARKS 9.8 FURTHER READING 9.9 EXERCISE QUESTIONS Security for Big Data 10.1 INTRODUCTION 10.2 SECURITY REQUIREMENTS 10.3 SECURITY: ATTACK TYPES AND MECHANISMS 10.4 ATTACK DETECTION AND PREVENTION 10.5 CONCLUDING REMARKS 10.6 FURTHER READING 10.7 EXERCISE QUESTIONS Privacy for Big Data 11.1 INTRODUCTION 11.2 UNDERSTANDING BIG DATA AND PRIVACY 11.3 PRIVACY VIOLATIONS AND THEIR IMPACT 11.4 TYPES OF PRIVACY VIOLATIONS 11.5 PRIVACY PROTECTION SOLUTIONS AND THEIR LIMITATIONS 11.6 CONCLUDING REMARKS 11.7 FURTHER READING 11.8 EXERCISE QUESTIONS High Performance Computing for Big Data 12.1 INTRODUCTION 12.2 SCALABILITY: NEED FOR HPC 12.3 GRAPHIC PROCESSING UNIT 12.4 TENSOR PROCESSING UNIT 12.5 HIGH SPEED INTERCONNECTS 12.6 MESSAGE PASSING INTERFACE 12.7 OPENMP 12.8 OTHER FRAMEWORKS 12.9 CONCLUDING REMARKS 12.10 FURTHER READING 12.11 EXERCISE QUESTIONS Deep Learning with Big Data 13.1 INTRODUCTION 13.2 FUNDAMENTALS 13.3 NEURAL NETWORK 13.4 TYPES OF DEEP NEURAL NETWORK 13.5 BIG DATA APPLICATIONS USING DEEP LEARNING13.6 CONCLUDING REMARKS 13.7 FURTHER READING 13.8 EXERCISE QUESTIONS Big Data Case Studies 14.1 GOOGLE EARTH ENGINE 14.2 FACEBOOK MESSAGES APPLICATION 14.3 HADOOP FOR REAL-TIME ANALYTICS 14.4 BIG DATA PROCESSING AT UBER 14.5 BIG DATA PROCESSING AT LINKEDIN 14.6 DISTRIBUTED GRAPH PROCESSING AT GOOGLE 14.7 FUTURE TRENDS 14.8 CONCLUDING REMARKS 14.9 FURTHER READING 14.10 EXERCISE QUESTIONS Bibliography Index