Foundation Models for Natural Language Processing

Dr. Gerhard Paaß is a Lead Scientist at the Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS). With a background in Mathematics, he is a recognized expert in the field of Artificial Intelligence, particularly in the area of Natural Language Processing. Dr. Paaß has previously worked at UC Berkeley in California and the University of Technology in Brisbane. He has served as reviewer and conference chair at various international conferences, including NeurIPS, CIKM, ECML/PKDD, ICDM, and KDD, where he regularly is a member of the program committee. Dr. Paaß has received a “best paper” award on probabilistic logic and is the author of about 70 publications for international conferences and journals. Recently, he authored the book “Artificial Intelligence: What's Behind the Technology of the Future?” (in German). He is currently involved in the creation of a computer center for Foundation Models. Besides experimental research on Foundation Models, he holds lectures for Deep Learning and Natural Language Understanding at the University of Bonn and in industry. Sven Giesselbach is the leader of the Natural Language Understanding (NLU) team at the Fraunhofer Institute for Intelligent Analysis and Information Systems (IAIS), where he has specialized in Artificial Intelligence and Natural Language Processing. He and his team develop solutions in the areas of medical, legal and general document understanding, which in their core build upon Foundation Models. Sven Giesselbach is also part of the Competence Center for Machine Learning Rhine-Ruhr (ML2R), where he works as a research scientist and investigates Informed Machine Learning, a paradigm in which knowledge is injected into machine learning models, in conjunction with language modeling. He has published more than 10 papers on Natural Language Processing and Understanding, which focus on the creation of application-ready NLU systems and the integration of expert knowledge in various stages of the solution design. He led the development of the Natural Language Understanding Showroom, a platform for showcasing state-of-the-art Natural Language Understanding models. He regularly gives talks about NLU at summer schools, conferences and AI-Meetups.

• 1. Introduction.- 2. Pre-trained Language Models.- 3. Improving Pre-trained Language Models.- 4. Knowledge Acquired by Foundation Models.- 5. Foundation Models for Information Extraction.- 6. Foundation Models for Text Generation.- 7. Foundation Models for Speech, Images, Videos, and Control.- 8. Summary and Outlook.