How to Measure Anything in Project Management

DOUGLAS W. HUBBARD has 35 years’ experience as a management consultant with a focus on the application of quantitative methods in decision making. He is the founder and president of Hubbard Decision Research and the creator of the “Applied Information Economics” method. He is also the author of the original How to Measure Anything: Finding the Value of Intangibles in Business as well as other books in measurement, risk analysis and decision making.ALEXANDER BUDZIER, PHD, is a Fellow at the University of Oxford’s Saïd Business School. He specializes in IT, infrastructure, energy, mega-events and change.ANDREAS BANG LEED is the Head of Data Science at Oxford Global Projects. He specializes in data-driven project planning and risk analysis for some of the world’s most ambitious mega-projects.

• Foreword xvPreface xixAcknowledgments xxiAbout the Authors xxiiiChapter 1 A World-scale Risk and a World-scale Opportunity 1The Size of Projects 2The Size of Project Problems 4Efforts to Fix Projects: The Emergence of Project Management 5A Path Forward: The Meta Project 8Notes 10Chapter 2 A Measurement Primer for Project Management 13The Concept of Measurement 14A Definition of Measurement 15Measurement and Probabilities for Practical Decision-making 16Are Scales Really Measurements? 18The Object of Measurement 21What Do You See When You See More of It? 21Why Do You Care? 23The Methods of Measurement 25Statistical Significance: What’s the Significance? 26Small Samples Tell You More Than You Think 28Other Sources of Measurement Aversion 30The Cost Objection 30Measurements Change What Is Being Measured 31Statistics Can Prove Anything 32Ethical Objections to Measurement 33Notes 34Chapter 3 How We Know What Works 35Skepticism for Project Managers 36The Analysis Placebo 36The Problem of Feedback and Learning 38How to Test Methods 40Controlled Experiments and Component Testing 40Evaluating Sources 41The Performance of Quantitative Methods 43Experts Versus Algorithms 43The Exsupero Ursus Fallacy: Algorithm Aversion 44Potential Reasons for Exsupero Ursus 45Improving the Human Expert 47Calibrating the Expert 48The Expert Consistency Component 49Collaboration on Estimates 50The Decomposition Component 52A Summary of Research on Other Project Planning and Management Methods 54Reference Class Forecasting 54Various Project Management Methods 55The Performance of Monte Carlo Simulations 58Notes 60Chapter 4 The Project Decision Model: The Reason for Measurements 63Two Types of Project Measurements 64Proto-purpose Discovery Measurements 64Decision-driven Measurements 66Unproductive Incentives vs. Measurements 69Decisions Before: Thinking Slow 70Exploration vs. Exploitation 71Tracking the Outside World 73Choosing How to Run the Project 74How Models Indicate What to Measure 77The Expected Value of Information: A Simple Introduction 77The Measurement Inversion: Measuring the Wrong Things 79The Value of Imperfect Measurements 80An Aspirational Model 82The Rise of Digital Twins 83Digital Twins in Project Management 84A Practical Path Forward 87Notes 88Chapter 5 Project Uncertainty and Risk: A Primer 91Basic Concepts and Definitions 92Uncertainty as a Probability Distribution 93Risk: A Special Case of Uncertainty 96The Problem with Current Methods 98Why Risk “Scores” Don’t Work 99How the Risk Matrix Makes Scores Worse 101A Quantitative Risk Model: Starting Very Simple 105The One-for-One Substitution 106Monte Carlo Mechanics: A Brief Introduction 108Supporting Decisions 111A Return on Mitigation 112How Much Risk Do You Tolerate? 113Risk Versus Return: The Powerful Theory of Utility 115Simple Tools for Measuring Uncertainty and Risk 117A First Estimate of a Discrete Probability 118A First Estimate of a Continuous Probability 119Final Clarifications 120Case Examples for What Probability Means 121Uncertainty Versus Risk Versus Opportunity 123Epistemic Versus Aleatory Uncertainty 124Even More Ordinal Scales 125Risk as Governance or Compliance 125The Problem of “Black Swans” 126Some Items That Aren’t Really Risks 127More Improvements to Come 128Notes 129Chapter 6 Calibrated Subjective Probability Estimates 131Introduction to Subjective Probability 132Calibration Exercise 135The Calibration Exercises 136Evaluating Performance and Typical Results 137Improving Calibration 140The Equivalent Bet 141More Techniques 142More Advanced Calibration Topics to Come 144The Effects of Calibration 146Conceptual Obstacles to Calibration 149Conflating Uncertainty with Knowing Nothing 149Hypotheses That Contradict the Data 152Objections Based on the Philosophical Debate in Statistics 153Notes 155Chapter 7 Cost and Schedule Measurements 157The Big Plan Versus Iteration: Meta-measurements of Common Estimation Methods 158Top-down Estimations: Reference Class Forecasting 162Bottom-up Forecasting with Monte Carlo 165A Deterministic View of Tasks 165Probability Distributions for Project Tasks 167Correlations 168Multiple Prerequisites and Stochastic Critical Paths 170Parade of Trades 171Comparing Top Down and Bottom Up: Case Examples 174The Swedish Nuclear Waste Program 175High-speed Rail 176How to Improve the Models 181The Granularity of the Monte Carlo Model 182Distributions and Biases 182Correlations 183Improving the RCF with Monte Carlo 184Notes 185Chapter 8 Betting on Benefits 187Meta-measurements of Benefits 189How Much Should Benefits Be to Justify a Project? 190Why This May Be Optimistic 192Why Measuring Benefits Is Rare 195Fermi Decompositions for Benefits 196Introduction to Fermi 197Some Example Decompositions 199Monetizing Benefits 201Forecasts of Monetary Impacts 201Preferences 202Quantifying Preferences 203The Use of Scores and Multiple Objectives 205An Example of Challenging Benefit Measurement: Biodiversity 206Measuring What Matters in Projects 206A (Slightly) More Realistic Information Value Calculation 207The High Information Values for Projects 209Getting Started on Measuring What Matters 211Considering Risk and Return 213A Risk Neutral Decision-maker for Projects 214Adding Utility Theory to Projects 215Some Alternatives within Utility Math 217Are Executives Too Risk Averse for Projects? 219A Framework and Its Consequences 221Findings from Quantitative Analysis of Past Projects 223How and When, Not Just Whether 223Benefits Are Not Just for Project Approval Decisions 224Notes 225Chapter 9 Measuring Progress 227The Progress Problem 227Simple Progress, Simple Interventions 228Earned Value Management 229EVM Basics 230The XRL Example 231Recovery vs. Performance 233Forecasting with EVM 235Progress in Information Projects 237Waterfall 237Agile and Measurement in Other Software Development Methods 237Summarizing Software Metric Difficulties 239Four Stories and Lessons 240Interfaces in a Global Bank Transformation 240An Energy Project Front End 241Construction Constraints 243Testing as Software Checkpoints 245Lessons 246The Remaining Project Simulation 247Conditional Reference Class Forecasting (CRCF) 247The Bottom-up Simulation for the Remaining Project 251Further Considerations for the RPA 252Notes 254Chapter 10 More Measurement Methods Made Easy 257Intuition for the Habitually Scientific 258A Jelly Bean Example 258A Little Probability Theory 260Consequences of Probability Theory 262Myths Exposed by Probability Theory 262Significant Points About Statistical Significance 265Basic Sampling Methods 266The “Mathless” Table for Medians 269Estimating a Population Proportion 270Project Cancellation Rates as a Function of Duration 274Measuring Population Size 274Measuring Some Things by Knowing Other Things 276Controlled Experiments 277Regression 277More Advanced Methods of Regression and Classification 283Estimating the Whole Distribution 285Summarizing Methods 289Brainstorming a Measurement Approach 289Data Gathering Methods 291A Review of Methods in This Chapter 292Notes on Surveys 293Notes 296Chapter 11 The Meta-project: Implementing Better Project Measurements 297Start with the End in Mind: The Continuous Improvement Process 299Measure What Matters 299(Real) Skepticism and Meta-measurements 301Measuring and Forecasting the Outside World 302AI: The Most Important Project Ecosystem Measurement? 304More Thinking, Fewer Projects, Bigger Wins 307Start Your Meta-project 307Examples of Meta-projects Deliverables: Continuous Improvement 308Develop an Initial Team 309Assess the Current State of the Project Portfolio 310Considerations for the Meta-project Plan 312The Pilot Project 312Scaling to the Final Deliverable 314Organizational Challenges 315Resistance to Change 315Addressing Organizational Objections to Measurement 316The Politics of Measurement 318Notes 319Chapter 12 A Call to Action for the Industry 321Call for Action for Project Software Vendors 321Put Decisions at the Center 322Deal in Uncertainties 324Build the User-buyer-builder Federation 325Be the Vendor That Measures Its Performance 325Be Forward-looking 326Call for Action for the Standard-setting Bodies 327Call to Action for Consultants, Researchers, and Advisory Firms 329Big Future Projects 331A Mars Mission 331Stopping Hurricanes 332The Meta-Project 333Notes 333Appendix 1 Analysis of Survey Responses on Project Management Practices 335Introduction and data overview 335Success Metrics: Cost and Schedule Overrun Ratios 337Overview of Project Management Practices Reported in the Survey 339Project Management Methodologies 339Cost and Schedule Estimation Methods 339Uncertainty and Risk Assessment Tools 340Certifications 341Results 341Project Management Methodologies 341Cost and Schedule Estimation Methods 343Uncertainty and Risk Assessment Tools 343Certifications 343Interpreting the (Mostly) Statistically Insignificant Results 344Appendix 2 Reference Class Data on Project Cost, Schedule, and Benefit Overruns 345Relevance of the Data and Reference Class Forecasting 346Using Historical Data to Improve Estimates – An Example 347Notes 351Appendix 3 Selected Distributions 353Uniform 354Beta 355Beta PERT 356Triangular 357Binary 358Normal 359Lognormal 360Power Law 361Truncated Power Law 362Quantile-parameterized 363Gamma Poisson 365Stochastic Information Packet 366Appendix 4 Chapter 6 Calibration Question Answers 369Answers to Confidence Interval Questions 369Answers to True/False Questions 371Appendix 5 Measuring Biodiversity 373The Benefits of Biodiversity 373Measuring Biodiversity 375Notes 376Index 377