Reversing

Eldad Eilam is a consultant in the field of reverse engineering. He assists clients with operating system and in-depth software reverse engineering, and has devoted several years to developing advanced reverse engineering techniques.

• Foreword viiAcknowledgments xiIntroduction xxiiiPart I Reversing 101 1Chapter 1 Foundations 3What Is Reverse Engineering? 3Software Reverse Engineering: Reversing 4Reversing Applications 4Security-Related Reversing 5Malicious Software 5Reversing Cryptographic Algorithms 6Digital Rights Management 7Auditing Program Binaries 7Reversing in Software Development 8Achieving Interoperability with Proprietary Software 8Developing Competing Software 8Evaluating Software Quality and Robustness 9Low-Level Software 9Assembly Language 10Compilers 11Virtual Machines and Bytecodes 12Operating Systems 13The Reversing Process 13System-Level Reversing 14Code-Level Reversing 14The Tools 14System-Monitoring Tools 15Disassemblers 15Debuggers 15Decompilers 16Is Reversing Legal? 17Interoperability 17Competition 18Copyright Law 19Trade Secrets and Patents 20The Digital Millenium Copyright Act 20DMCA Cases 22License Agreement Considerations 23Code Samples & Tools 23Conclusion 23Chapter 2 Low-Level Software 25High-Level Perspectives 26Program Structure 26Modules 28Common Code Constructs 28Data Management 29Variables 30User-Defined Data Structures 30Lists 31Control Flow 32High-Level Languages 33C 34C++ 35Java 36C# 36Low-Level Perspectives 37Low-Level Data Management 37Registers 39The Stack 40Heaps 42Executable Data Sections 43Control Flow 43Assembly Language 101 44Registers 44Flags 46Instruction Format 47Basic Instructions 48Moving Data 49Arithmetic 49Comparing Operands 50Conditional Branches 51Function Calls 51Examples 52A Primer on Compilers and Compilation 53Defining a Compiler 54Compiler Architecture 55Front End 55Intermediate Representations 55Optimizer 56Back End 57Listing Files 58Specific Compilers 59Execution Environments 60Software Execution Environments (Virtual Machines) 60Bytecodes 61Interpreters 61Just-in-Time Compilers 62Reversing Strategies 62Hardware Execution Environments in Modern Processors 63Intel NetBurst 65µops (Micro-Ops) 65Pipelines 65Branch Prediction 67Conclusion 68Chapter 3 Windows Fundamentals 69Components and Basic Architecture 70Brief History 70Features 70Supported Hardware 71Memory Management 71Virtual Memory and Paging 72Paging 73Page Faults 73Working Sets 74Kernel Memory and User Memory 74The Kernel Memory Space 75Section Objects 77VAD Trees 78User-Mode Allocations 78Memory Management APIs 79Objects and Handles 80Named objects 81Processes and Threads 83Processes 84Threads 84Context Switching 85Synchronization Objects 86Process Initialization Sequence 87Application Programming Interfaces 88The Win32 API 88The Native API 90System Calling Mechanism 91Executable Formats 93Basic Concepts 93Image Sections 95Section Alignment 95Dynamically Linked Libraries 96Headers 97Imports and Exports 99Directories 99Input and Output 103The I/O System 103The Win32 Subsystem 104Object Management 105Structured Exception Handling 105Conclusion 107Chapter 4 Reversing Tools 109Different Reversing Approaches 110Offline Code Analysis (Dead-Listing) 110Live Code Analysis 110Disassemblers 110IDA Pro 112ILDasm 115Debuggers 116User-Mode Debuggers 118OllyDbg 118User Debugging in WinDbg 119IDA Pro 121PEBrowse Professional Interactive 122Kernel-Mode Debuggers 122Kernel Debugging in WinDbg 123Numega SoftICE 124Kernel Debugging on Virtual Machines 127Decompilers 129System-Monitoring Tools 129Patching Tools 131Hex Workshop 131Miscellaneous Reversing Tools 133Executable-Dumping Tools 133DUMPBIN 133PEView 137PEBrowse Professional 137Conclusion 138Part II Applied Reversing 139Chapter 5 Beyond the Documentation 141Reversing and Interoperability 142Laying the Ground Rules 142Locating Undocumented APIs 143What Are We Looking For? 144Case Study: The Generic Table API in NTDLL.DLL 145RtlInitializeGenericTable 146RtlNumberGenericTableElements 151RtlIsGenericTableEmpty 152RtlGetElementGenericTable 153Setup and Initialization 155Logic and Structure 159Search Loop 1 161Search Loop 2 163Search Loop 3 164Search Loop 4 165Reconstructing the Source Code 165RtlInsertElementGenericTable 168RtlLocateNodeGenericTable 170RtlRealInsertElementWorker 178Splay Trees 187RtlLookupElementGenericTable 188RtlDeleteElementGenericTable 193Putting the Pieces Together 194Conclusion 196Chapter 6 Deciphering File Formats 199Cryptex 200Using Cryptex 201Reversing Cryptex 202The Password Verification Process 207Catching the “Bad Password” Message 207The Password Transformation Algorithm 210Hashing the Password 213The Directory Layout 218Analyzing the Directory Processing Code 218Analyzing a File Entry 223Dumping the Directory Layout 227The File Extraction Process 228Scanning the File List 234Decrypting the File 235The Floating-Point Sequence 236The Decryption Loop 238Verifying the Hash Value 239The Big Picture 239Digging Deeper 241Conclusion 242Chapter 7 Auditing Program Binaries 243Defining the Problem 243Vulnerabilities 245Stack Overflows 245A Simple Stack Vulnerability 247Intrinsic Implementations 249Stack Checking 250Nonexecutable Memory 254Heap Overflows 255String Filters 256Integer Overflows 256Arithmetic Operations on User-Supplied Integers 258Type Conversion Errors 260Case-Study: The IIS Indexing Service Vulnerability 262CVariableSet::AddExtensionControlBlock 263DecodeURLEscapes 267Conclusion 271Chapter 8 Reversing Malware 273Types of Malware 274Viruses 274Worms 274Trojan Horses 275Backdoors 276Mobile Code 276Adware/Spyware 276Sticky Software 277Future Malware 278Information-Stealing Worms 278BIOS/Firmware Malware 279Uses of Malware 280Malware Vulnerability 281Polymorphism 282Metamorphism 283Establishing a Secure Environment 285The Backdoor.Hacarmy.D 285Unpacking the Executable 286Initial Impressions 290The Initial Installation 291Initializing Communications 294Connecting to the Server 296Joining the Channel 298Communicating with the Backdoor 299Running SOCKS4 Servers 303Clearing the Crime Scene 303The Backdoor.Hacarmy.D: A Command Reference 304Conclusion 306Part III Cracking 307Chapter 9 Piracy and Copy Protection 309Copyrights in the New World 309The Social Aspect 310Software Piracy 310Defining the Problem 311Class Breaks 312Requirements 313The Theoretically Uncrackable Model 314Types of Protection 314Media-Based Protections 314Serial Numbers 315Challenge Response and Online Activations 315Hardware-Based Protections 316Software as a Service 317Advanced Protection Concepts 318Crypto-Processors 318Digital Rights Management 319DRM Models 320The Windows Media Rights Manager 321Secure Audio Path 321Watermarking 321Trusted Computing 322Attacking Copy Protection Technologies 324Conclusion 324Chapter 10 Antireversing Techniques 327Why Antireversing? 327Basic Approaches to Antireversing 328Eliminating Symbolic Information 329Code Encryption 330Active Antidebugger Techniques 331Debugger Basics 331The IsDebuggerPresent API 332SystemKernelDebuggerInformation 333Detecting SoftICE Using the Single-Step Interrupt 334The Trap Flag 335Code Checksums 335Confusing Disassemblers 336Linear Sweep Disassemblers 337Recursive Traversal Disassemblers 338Applications 343Code Obfuscation 344Control Flow Transformations 346Opaque Predicates 346Confusing Decompilers 348Table Interpretation 348Inlining and Outlining 353Interleaving Code 354Ordering Transformations 355Data Transformations 355Modifying Variable Encoding 355Restructuring Arrays 356Conclusion 356Chapter 11 Breaking Protections 357Patching 358Keygenning 364Ripping Key-Generation Algorithms 365Advanced Cracking: Defender 370Reversing Defender’s Initialization Routine 377Analyzing the Decrypted Code 387SoftICE’s Disappearance 396Reversing the Secondary Thread 396Defeating the “Killer” Thread 399Loading KERNEL32.DLL 400Reencrypting the Function 401Back at the Entry Point 402Parsing the Program Parameters 404Processing the Username 406Validating User Information 407Unlocking the Code 409Brute-Forcing Your Way through Defender 409Protection Technologies in Defender 415Localized Function-Level Encryption 415Relatively Strong Cipher Block Chaining 415Reencrypting 416Obfuscated Application/Operating System Interface 416Processor Time-Stamp Verification Thread 417Runtime Generation of Decryption Keys 418Interdependent Keys 418User-Input-Based Decryption Keys 419Heavy Inlining 419Conclusion 419Part IV Beyond Disassembly 421Chapter 12 Reversing .NET 423Ground Rules 424.NET Basics 426Managed Code 426.NET Programming Languages 428Common Type System (CTS) 428Intermediate Language (IL) 429The Evaluation Stack 430Activation Records 430IL Instructions 430IL Code Samples 433Counting Items 433A Linked List Sample 436Decompilers 443Obfuscators 444Renaming Symbols 444Control Flow Obfuscation 444Breaking Decompilation and Disassembly 444Reversing Obfuscated Code 445XenoCode Obfuscator 446DotFuscator by Preemptive Solutions 448Remotesoft Obfuscator and Linker 451Remotesoft Protector 452Precompiled Assemblies 453Encrypted Assemblies 453Conclusion 455Chapter 13 Decompilation 457Native Code Decompilation: An Unsolvable Problem? 457Typical Decompiler Architecture 459Intermediate Representations 459Expressions and Expression Trees 461Control Flow Graphs 462The Front End 463Semantic Analysis 463Generating Control Flow Graphs 464Code Analysis 466Data-Flow Analysis 466Single Static Assignment (SSA) 467Data Propagation 468Register Variable Identification 470Data Type Propagation 471Type Analysis 472Primitive Data Types 472Complex Data Types 473Control Flow Analysis 475Finding Library Functions 475The Back End 476Real-World IA-32 Decompilation 477Conclusion 477Appendix A Deciphering Code Structures 479Appendix B Understanding Compiled Arithmetic 519Appendix C Deciphering Program Data 537Appendix D Citations 561Index 567