Build a Text-to-Image Generator (from Scratch)

Mark Liu is a professor and program director known for translating cutting-edge AI into practical curricula. With years mentoring graduate students and professionals, Mark brings clarity, rigor, and enthusiasm to every page. He distills deep generative-model expertise into step-by-step guidance that empowers readers to build powerful visual AI systems.

This book stands out for its hands-on, no-fluff approach to text-to-image generation—perfect for practitioners who want to build rather than just theorize. The clear PyTorch implementations, Colab-friendly examples, and practical exercises make even advanced concepts like Diffusion Models feel achievable.Simeon Leyzerzon, President, Excelsior Software Ltd.  This book is a great hands-on intro to how text-to-image models like Stable Diffusion actually work under the hood. It explains the roles of transformers, VAEs, and denoising U-Nets in a super approachable way, with lots of code you can run yourself. If you’re curious about generative AI and want to build or tweak your own models, this is a solid place to start.Ravikumar Sanapala, Product Manager, Reality Labs, Meta

• PART 1: UNDERSTANDING ATTENTION AND TRANSFORMERS 1 A TALE OF TWO MODELS: TRANSFORMERS AND DIFFUSIONS 2 BUILD A TRANSFORMER 3 CLASSIFY IMAGES WITH A VISION TRANSFORMER (VIT)4 ADD CAPTIONS TO IMAGES PART 2: INTRODUCTION TO DIFFUSION MODELS 5 GENERATE IMAGES WITH DIFFUSION MODELS 6 CONTROL WHAT IMAGES TO GENERATE IN DIFFUSION MODELS 7 GENERATE HIGH-RESOLUTION IMAGES WITH DIFFUSION MODELS PART 3: TEXT-TO-IMAGE GENERATION WITH DIFFUSION MODELS 8 CLIP: A MODEL TO MEASURE THE SIMILARITY BETWEEN IMAGE AND TEXT 9 TEXT-TO-IMAGE GENERATION WITH LATENT DIFFUSION 10 A DEEP DIVE INTO STABLE DIFFUSION PART 4: TEXT-TO-IMAGE GENERATION WITH TRANSFORMERS 11 VQGAN: CONVERT IMAGES INTO SEQUENCES OF INTEGERS 12 A MINIMAL IMPLEMENTATION OF DALL-E PART 5: NEW DEVELOPMENTS AND CHALLENGES 13 NEW DEVELOPMENTS AND CHALLENGES IN TEXT-TO-IMAGE GENERATION APPENDIX INSTALL PYTORCH AND ENABLE GPU TRAINING LOCALLY AND IN COLAB