FORTUNATO PESARIN is Emeritus Professor at the Department of Statistical Sciences, University of Padova, Italy. LUIGI SALMASO is Professor of Statistics at the Department of Management and Engineering, University of Padova, Italy.

• Preface xviiAcknowledgments xxiAcronyms xxiiiNotation xxvAbout the Companion Website xxix1 Introduction 11.1 On Permutation Analysis 11.2 Basic Notation 61.3 The Permutation – Conditional – Testing Principle 81.3.1 Nonparametric Family of Distributions 91.3.2 The Permutation – Conditional – Testing Principle 101.4 Permutation Approaches 131.5 When and Why Conditioning Is Appropriate 151.6 Randomization and Permutation 181.7 Computational Aspects 191.8 A Problem with Paired Observations 201.8.1 Modelling Responses 211.8.2 Symmetry Induced by Exchangeability 231.8.3 Further Aspects 241.8.4 Student’s t-Paired Solution 241.8.5 The Signed Rank Solution 261.8.6 The McNemar Solution 271.9 The Permutation Solution 281.9.1 General Aspects 281.9.2 The Permutation Sample Space 281.9.3 The Conditional Monte Carlo Method 291.9.3.1 An Algorithm for Inspecting Permutation Sample Spaces 291.9.3.2 Approximating the Permutation Distribution 311.9.4 Problems and Exercises 321.10 A Two-Sample Problem 331.10.1 Modelling Responses 341.10.2 The Student t Solution 361.10.3 The Permutation Solution 361.10.4 Rank Solutions 381.10.5 Fisher’s Exact Probability Solution 391.10.6 Problems and Exercises 391.11 One-Way ANOVA 401.11.1 Modelling Responses 411.11.2 Permutation Solutions 421.11.3 Problems and Exercises 432 Theory of One-Dimensional Permutation Tests 452.1 Introduction 452.1.1 Notation and Basic Assumptions 452.1.2 The Conditional Reference Space 492.1.3 Conditioning on a Set of Sufficient Statistics 552.2 Definition of Permutation Tests 592.2.1 General Aspects 592.2.2 Randomized Permutation Tests 602.2.3 Non-Randomized Permutation Tests 612.2.4 Unbiasedness of Permutation Tests 632.3 Some Useful Test Statistics 652.4 Equivalence of Permutation Statistics 692.4.1 Two Examples 702.4.2 Problems and Exercises 722.5 Further Relevant Properties 742.5.1 The p-Value-Like Statistic 742.5.2 Unbiasedness of p-Value-Like Test Statistics 752.5.3 Weak Consistency of Permutation Tests 772.5.4 A CMC Algorithm for Estimating the p-Value-Like Statistic 802.6 Some Arguments for Choosing Permutation Tests 822.7 Power Functions of Permutation Tests 872.7.1 Definition and Algorithm for the Conditional Power 872.7.2 The Conditional ROC Curve 902.7.3 Definition and Algorithm for the Unconditional Power: Fixed Effects 902.7.4 Unconditional Power: Random Effects 922.7.5 Comments on Power Functions 922.8 Permutation Confidence Interval for δ 932.8.1 Problems and Exercises 982.9 Generalizing Inference from Conditional to Unconditional 992.10 Optimal Properties 1032.10.1 Problems and Exercises 1052.11 Some Further Asymptotic Properties 1072.11.1 Introduction 1072.11.2 Two Basic Theorems 1072.12 Permutation Central Limit Theorems 1092.12.1 Basic Notions 1092.12.2 Permutation Central Limit Theorems 1102.12.3 Problems and Exercises 1133 Some Examples of Unidimensional Permutation Tests 1173.1 Examples of One-Sample Problems 1173.1.1 A Problem with Repeated Observations 1253.1.1.1 Friedman’s Rank Test 1263.1.1.2 A Permutation Solution 1263.1.1.3 An Example 1273.1.2 Problems and Exercises 1283.2 Some Examples of Multi-Sample Problems 1303.2.1 Synchronized Permutations 1393.3 Analysis of Ordered Categorical Variables 1453.3.1 General Aspects 1453.3.2 A Solution Based on Score Transformations 1473.3.3 Typical Goodness-of-Fit Solutions 1483.3.3.1 An Example 1493.3.4 Extension to Non-Dominance Alternatives and C Samples 1493.3.5 Problems and Exercises 1514 The Nonparametric Combination Methodology 1534.1 Introduction 1534.1.1 General Aspects 1534.1.2 Some Preliminary Notes 1554.1.3 Main Assumptions and Notation 1574.1.4 Some Comments 1594.2 The Nonparametric Combination Methodology 1604.2.1 Assumptions on Partial Tests 1604.2.2 Desirable Properties of Combining Functions 1624.2.3 A Two-Phase Algorithm for Nonparametric Combination 1644.2.4 Some Useful Combining Functions 1684.2.5 Why Combination Is Nonparametric 1774.2.6 On Admissible Combining Functions 1784.2.7 Problems and Exercises 1794.3 Consistency, Unbiasedness, and Power of Combined Tests 1824.3.1 Consistency 1824.3.2 Unbiasedness 1824.3.3 A Non-Consistent Combining Function 1844.3.4 Power of Combined Tests 1854.3.4.1 The Conditional Power Function 1854.3.4.2 The Unconditional Power Function 1864.3.5 Multivariate Conditional Confidence Region for δ 1874.3.6 Problems and Exercises 1884.4 Some Further Asymptotic Properties 1894.4.1 General Conditions 1894.4.2 Some Asymptotic Properties 1904.5 Finite-Sample Consistency 1944.5.1 Introduction and Motivations 1944.5.2 Finite-Sample Consistency 1954.5.3 Some Applications of Finite-Sample Consistency 2014.6 Some Examples of Nonparametric Combination 2074.6.1 Problems and Exercises 2294.7 Comments on the Nonparametric Combination 2314.7.1 General Comments 2315 Multiple Testing Problems and Multiplicity Adjustment 2335.1 Defining Raw and Adjusted p-Values 2335.2 Controlling for Multiplicity 2355.2.1 Multiple Comparison and Multiple Testing 2355.2.2 Some Definitions of the Global Type I Error 2365.3 Multiple Testing 2375.4 The Closed Testing Approach 2385.4.1 Closed Testing for Multiple Testing 2395.4.2 Closed Testing Using the MinP Bonferroni–Holm Procedure 2415.5 Weighted Methods for Controlling FWE and FDR 2455.6 Adjusting Stepwise p-Values 2475.6.1 Showing Biasedness of Standard p-Values for Stepwise Regression 2485.6.2 Algorithm Description 2485.6.3 Optimal Subset Procedures 2506 Analysis of Multivariate Categorical Variables 2516.1 Introduction 2516.2 The Multivariate McNemar Test 2526.2.1 An Extension of the Multivariate McNemar Test 2556.3 Multivariate Goodness-of-Fit Testing for Ordered Variables 2566.3.1 Multivariate Extension of Fisher’s Exact Probability Test 2586.4 MANOVA with Nominal Categorical Data 2586.4.1 A Formal Description 2586.5 Stochastic Ordering 2606.5.1 Formal Description 2606.5.2 Further Breaking Down of the Hypotheses 2616.5.3 Permutation Test 2616.6 Multifocus Analysis 2646.6.1 General Aspects 2646.6.2 The Multifocus Solution 2646.6.3 An Application 2676.7 Isotonic Inference 2686.7.1 Introduction 2686.7.2 Allelic Association Analysis in Genetics 2706.7.3 Parametric Solutions 2716.7.3.1 Chiano and Clayton’s Method 2716.7.3.2 Maximum-Likelihood Approach 2726.7.4 Permutation Approach 2736.8 Test on Moments for Ordered Variables 2746.8.1 General Aspects 2746.8.2 Score Transformations and Univariate Tests 2756.8.3 Multivariate Extension 2766.9 Heterogeneity Comparisons 2786.9.1 Introduction 2786.9.2 Tests for Comparing Heterogeneities 2796.9.3 A Case Study in Population Genetics 2816.10 Application to PhD Programme Evaluation 2826.10.1 Description of the Problem 2826.10.2 Global Satisfaction Index 2846.10.3 Multivariate Performance Comparisons 2857 Permutation Testing for Repeated Measurements 2877.1 Introduction 2877.2 Carry-Over Effects in Repeated-Measures Designs 2887.3 Modelling Repeated Measurements 2897.3.1 A General Additive Model 2897.3.2 The Hypotheses of Interest 2917.4 Testing Solutions 2927.4.1 Solutions Using the NPC Approach 2927.4.2 Analysis of Two-Sample Dominance Problems 2947.4.3 Analysis of the Cross-Over (AB–BA) Design 2947.4.4 Analysis of a Cross-Over Design with Paired Data 2957.5 Testing for Repeated Measurements with Missing Data 2967.5.1 General Aspects of Permutation Testing with Missing Data 2977.5.2 Bibliographic Notes 2977.6 On Missing Data Processes 2987.6.1 Data Missing Completely at Random 2997.6.2 Data Missing Not at Random 2997.7 The Permutation Approach 3007.7.1 Deletion, Imputation, and Intention to Treat Strategies 3017.7.2 Breaking Down the Hypotheses 3037.8 The Structure of Testing Problems 3037.8.1 Hypotheses for MNAR Models 3037.8.2 Hypotheses for MCAR Models 3057.8.3 Permutation Structure with Missing Values 3057.9 Permutation Analysis of Missing Values 3077.9.1 Partitioning the Permutation Sample Space 3077.9.2 Solution for Two-Sample MCAR Problems 3087.9.3 Extensions to Multivariate C-Sample Problems 3107.9.4 Extension to MNAR Models 3117.10 ‘GERMINA’ Data: An Example of a MNAR Model 3127.10.1 Problem Description 3137.10.2 The Permutation Solution 3157.11 Multivariate Paired Observations 3187.11.1 The Problem 3187.12 Repeated Measures and Missing Data 3197.12.1 An Example 3207.13 Botulinum Data: Example 3227.13.1 Data Description 3227.13.2 Research Questions and Methods 3237.14 Waterfalls Data: Example 3247.14.1 Data Description 3247.14.2 Research Questions and Methods 3258 Some Stochastic Ordering Problems 3278.1 Multivariate Ordered Alternatives 3278.2 Testing for Umbrella Alternatives 3298.2.1 Hypotheses and Tests in Simple Stochastic Ordering 3318.2.2 Permutation Tests for Umbrella Alternatives 3328.3 Tumour Growth Curves: Example 3358.4 PERC Data: Example 3368.4.1 Introduction 3368.4.2 A Permutation Solution 3399 NPC Tests for Survival Analysis 3419.1 Introduction and Main Notation 3419.1.1 Failure Time Distributions 3419.1.2 Data Structure 3429.2 Comparison of Survival Curves 3439.3 An Overview from the Literature 3459.3.1 Permutation Tests in Survival Analysis 3479.4 Two NPC Tests 3489.4.1 Breaking Down the Hypotheses 3499.4.2 The Test Structure 3509.4.3 NPC Test for Treatment-Independent Censoring 3509.4.4 NPC Test for Treatment-Dependent Censoring 3529.5 Case Study: In-Hospital Mortality 3559.5.1 Results 3559.6 Case Study: Esophageal Cancer 3569.6.1 A Permutation Solution 3589.6.2 Survival Analysis with Stratification by Propensity Score 3599.6.3 Integrating Propensity Score and NPC Testing 3619.7 Case Study: Comparison of Three Survival Curves 3669.7.1 Unstratified Survival Analysis 3679.7.2 Stratified Survival Analysis Using Propensity Score 36810 NPC Tests in Shape Analysis 37310.1 A Brief Overview of Statistical Shape Analysis 37510.1.1 How to Describe Shapes 37510.1.2 Multivariate Morphometrics 37710.2 Inference with Shape Data 38010.3 NPC Approach to Shape Analysis 38210.3.1 Comparative Simulation Study 38410.4 NPC Analysis with Correlated Landmarks 38710.5 An Application to Mediterranean Monk Seal Skulls 39010.5.1 Some Remarks 39311 Case Studies 39511.1 Autofluorescence Data 39511.1.1 A Permutation Solution 39711.1.2 Analysis Using R 39911.2 Confocal Data 40211.2.1 A Permutation Solution 40411.2.1.1 Evaluation of Treatment Effect Using R 40611.2.1.2 Evaluation of Time Effect Using R 40911.2.2 Two-Way (M)ANOVA 41111.2.2.1 Permutation Tests in Two-Way (M)ANOVA 41111.2.2.2 Analysis Using R 41311.3 SETIG Data 41411.3.1 Analysis Using R 41711.4 Logistic Regression and NPC Test for Multivariate Analysis 42011.4.1 Lymph Node Metastases Data 42011.4.2 Bladder Cancer Data 42311.4.2.1 Applying NPC: Some Results 42511.4.2.2 Analysis with Logistic Regression 42511.4.2.3 Final Comments 42712 Permutation Tests Over the Years 43112.1 Kind of Data 43212.1.1 Time-Series Data 43212.1.2 Thick Data 43212.1.3 Imaging Data 43512.1.4 Survival Data 43612.1.5 Censored Data 43612.2 Different Models 43712.2.1 Regression 43712.2.2 Classification 44012.2.3 Doe 44012.2.4 Ranking 44012.3 Different Tests 44112.3.1 Test for Centrality 44112.3.2 Test for Variability 44612.3.3 Test for Both Centrality and Variability 44612.3.4 ANOVA Test 44712.3.5 Testing Relationship 44712.3.6 NPC Tests 44812.4 Methodological and Computational Aspects 44912.4.1 Power 44912.4.2 Computational Problems 45212.4.3 Other Problems 452Appendix A Examples with Applications in R 453A. 1 Mult Data Example 453A. 2 Washing Test Data 456A. 3 Germina Data Example 460A. 4 Botulinum Data 464A. 5 Waterfalls Data 467A. 6 Tumour Growth Curves 469A. 7 PERC Data 471A. 8 Mediterranean Monk Seal Skulls 475Bibliography 479Index 517

FORTUNATO PESARIN is Emeritus Professor at the Department of Statistical Sciences, University of Padova, Italy. LUIGI SALMASO is Professor of Statistics at the Department of Management and Engineering, University of Padova, Italy.

• Preface xviiAcknowledgments xxiAcronyms xxiiiNotation xxvAbout the Companion Website xxix1 Introduction 11.1 On Permutation Analysis 11.2 Basic Notation 61.3 The Permutation – Conditional – Testing Principle 81.3.1 Nonparametric Family of Distributions 91.3.2 The Permutation – Conditional – Testing Principle 101.4 Permutation Approaches 131.5 When and Why Conditioning Is Appropriate 151.6 Randomization and Permutation 181.7 Computational Aspects 191.8 A Problem with Paired Observations 201.8.1 Modelling Responses 211.8.2 Symmetry Induced by Exchangeability 231.8.3 Further Aspects 241.8.4 Student’s t-Paired Solution 241.8.5 The Signed Rank Solution 261.8.6 The McNemar Solution 271.9 The Permutation Solution 281.9.1 General Aspects 281.9.2 The Permutation Sample Space 281.9.3 The Conditional Monte Carlo Method 291.9.3.1 An Algorithm for Inspecting Permutation Sample Spaces 291.9.3.2 Approximating the Permutation Distribution 311.9.4 Problems and Exercises 321.10 A Two-Sample Problem 331.10.1 Modelling Responses 341.10.2 The Student t Solution 361.10.3 The Permutation Solution 361.10.4 Rank Solutions 381.10.5 Fisher’s Exact Probability Solution 391.10.6 Problems and Exercises 391.11 One-Way ANOVA 401.11.1 Modelling Responses 411.11.2 Permutation Solutions 421.11.3 Problems and Exercises 432 Theory of One-Dimensional Permutation Tests 452.1 Introduction 452.1.1 Notation and Basic Assumptions 452.1.2 The Conditional Reference Space 492.1.3 Conditioning on a Set of Sufficient Statistics 552.2 Definition of Permutation Tests 592.2.1 General Aspects 592.2.2 Randomized Permutation Tests 602.2.3 Non-Randomized Permutation Tests 612.2.4 Unbiasedness of Permutation Tests 632.3 Some Useful Test Statistics 652.4 Equivalence of Permutation Statistics 692.4.1 Two Examples 702.4.2 Problems and Exercises 722.5 Further Relevant Properties 742.5.1 The p-Value-Like Statistic 742.5.2 Unbiasedness of p-Value-Like Test Statistics 752.5.3 Weak Consistency of Permutation Tests 772.5.4 A CMC Algorithm for Estimating the p-Value-Like Statistic 802.6 Some Arguments for Choosing Permutation Tests 822.7 Power Functions of Permutation Tests 872.7.1 Definition and Algorithm for the Conditional Power 872.7.2 The Conditional ROC Curve 902.7.3 Definition and Algorithm for the Unconditional Power: Fixed Effects 902.7.4 Unconditional Power: Random Effects 922.7.5 Comments on Power Functions 922.8 Permutation Confidence Interval for δ 932.8.1 Problems and Exercises 982.9 Generalizing Inference from Conditional to Unconditional 992.10 Optimal Properties 1032.10.1 Problems and Exercises 1052.11 Some Further Asymptotic Properties 1072.11.1 Introduction 1072.11.2 Two Basic Theorems 1072.12 Permutation Central Limit Theorems 1092.12.1 Basic Notions 1092.12.2 Permutation Central Limit Theorems 1102.12.3 Problems and Exercises 1133 Some Examples of Unidimensional Permutation Tests 1173.1 Examples of One-Sample Problems 1173.1.1 A Problem with Repeated Observations 1253.1.1.1 Friedman’s Rank Test 1263.1.1.2 A Permutation Solution 1263.1.1.3 An Example 1273.1.2 Problems and Exercises 1283.2 Some Examples of Multi-Sample Problems 1303.2.1 Synchronized Permutations 1393.3 Analysis of Ordered Categorical Variables 1453.3.1 General Aspects 1453.3.2 A Solution Based on Score Transformations 1473.3.3 Typical Goodness-of-Fit Solutions 1483.3.3.1 An Example 1493.3.4 Extension to Non-Dominance Alternatives and C Samples 1493.3.5 Problems and Exercises 1514 The Nonparametric Combination Methodology 1534.1 Introduction 1534.1.1 General Aspects 1534.1.2 Some Preliminary Notes 1554.1.3 Main Assumptions and Notation 1574.1.4 Some Comments 1594.2 The Nonparametric Combination Methodology 1604.2.1 Assumptions on Partial Tests 1604.2.2 Desirable Properties of Combining Functions 1624.2.3 A Two-Phase Algorithm for Nonparametric Combination 1644.2.4 Some Useful Combining Functions 1684.2.5 Why Combination Is Nonparametric 1774.2.6 On Admissible Combining Functions 1784.2.7 Problems and Exercises 1794.3 Consistency, Unbiasedness, and Power of Combined Tests 1824.3.1 Consistency 1824.3.2 Unbiasedness 1824.3.3 A Non-Consistent Combining Function 1844.3.4 Power of Combined Tests 1854.3.4.1 The Conditional Power Function 1854.3.4.2 The Unconditional Power Function 1864.3.5 Multivariate Conditional Confidence Region for δ 1874.3.6 Problems and Exercises 1884.4 Some Further Asymptotic Properties 1894.4.1 General Conditions 1894.4.2 Some Asymptotic Properties 1904.5 Finite-Sample Consistency 1944.5.1 Introduction and Motivations 1944.5.2 Finite-Sample Consistency 1954.5.3 Some Applications of Finite-Sample Consistency 2014.6 Some Examples of Nonparametric Combination 2074.6.1 Problems and Exercises 2294.7 Comments on the Nonparametric Combination 2314.7.1 General Comments 2315 Multiple Testing Problems and Multiplicity Adjustment 2335.1 Defining Raw and Adjusted p-Values 2335.2 Controlling for Multiplicity 2355.2.1 Multiple Comparison and Multiple Testing 2355.2.2 Some Definitions of the Global Type I Error 2365.3 Multiple Testing 2375.4 The Closed Testing Approach 2385.4.1 Closed Testing for Multiple Testing 2395.4.2 Closed Testing Using the MinP Bonferroni–Holm Procedure 2415.5 Weighted Methods for Controlling FWE and FDR 2455.6 Adjusting Stepwise p-Values 2475.6.1 Showing Biasedness of Standard p-Values for Stepwise Regression 2485.6.2 Algorithm Description 2485.6.3 Optimal Subset Procedures 2506 Analysis of Multivariate Categorical Variables 2516.1 Introduction 2516.2 The Multivariate McNemar Test 2526.2.1 An Extension of the Multivariate McNemar Test 2556.3 Multivariate Goodness-of-Fit Testing for Ordered Variables 2566.3.1 Multivariate Extension of Fisher’s Exact Probability Test 2586.4 MANOVA with Nominal Categorical Data 2586.4.1 A Formal Description 2586.5 Stochastic Ordering 2606.5.1 Formal Description 2606.5.2 Further Breaking Down of the Hypotheses 2616.5.3 Permutation Test 2616.6 Multifocus Analysis 2646.6.1 General Aspects 2646.6.2 The Multifocus Solution 2646.6.3 An Application 2676.7 Isotonic Inference 2686.7.1 Introduction 2686.7.2 Allelic Association Analysis in Genetics 2706.7.3 Parametric Solutions 2716.7.3.1 Chiano and Clayton’s Method 2716.7.3.2 Maximum-Likelihood Approach 2726.7.4 Permutation Approach 2736.8 Test on Moments for Ordered Variables 2746.8.1 General Aspects 2746.8.2 Score Transformations and Univariate Tests 2756.8.3 Multivariate Extension 2766.9 Heterogeneity Comparisons 2786.9.1 Introduction 2786.9.2 Tests for Comparing Heterogeneities 2796.9.3 A Case Study in Population Genetics 2816.10 Application to PhD Programme Evaluation 2826.10.1 Description of the Problem 2826.10.2 Global Satisfaction Index 2846.10.3 Multivariate Performance Comparisons 2857 Permutation Testing for Repeated Measurements 2877.1 Introduction 2877.2 Carry-Over Effects in Repeated-Measures Designs 2887.3 Modelling Repeated Measurements 2897.3.1 A General Additive Model 2897.3.2 The Hypotheses of Interest 2917.4 Testing Solutions 2927.4.1 Solutions Using the NPC Approach 2927.4.2 Analysis of Two-Sample Dominance Problems 2947.4.3 Analysis of the Cross-Over (AB–BA) Design 2947.4.4 Analysis of a Cross-Over Design with Paired Data 2957.5 Testing for Repeated Measurements with Missing Data 2967.5.1 General Aspects of Permutation Testing with Missing Data 2977.5.2 Bibliographic Notes 2977.6 On Missing Data Processes 2987.6.1 Data Missing Completely at Random 2997.6.2 Data Missing Not at Random 2997.7 The Permutation Approach 3007.7.1 Deletion, Imputation, and Intention to Treat Strategies 3017.7.2 Breaking Down the Hypotheses 3037.8 The Structure of Testing Problems 3037.8.1 Hypotheses for MNAR Models 3037.8.2 Hypotheses for MCAR Models 3057.8.3 Permutation Structure with Missing Values 3057.9 Permutation Analysis of Missing Values 3077.9.1 Partitioning the Permutation Sample Space 3077.9.2 Solution for Two-Sample MCAR Problems 3087.9.3 Extensions to Multivariate C-Sample Problems 3107.9.4 Extension to MNAR Models 3117.10 ‘GERMINA’ Data: An Example of a MNAR Model 3127.10.1 Problem Description 3137.10.2 The Permutation Solution 3157.11 Multivariate Paired Observations 3187.11.1 The Problem 3187.12 Repeated Measures and Missing Data 3197.12.1 An Example 3207.13 Botulinum Data: Example 3227.13.1 Data Description 3227.13.2 Research Questions and Methods 3237.14 Waterfalls Data: Example 3247.14.1 Data Description 3247.14.2 Research Questions and Methods 3258 Some Stochastic Ordering Problems 3278.1 Multivariate Ordered Alternatives 3278.2 Testing for Umbrella Alternatives 3298.2.1 Hypotheses and Tests in Simple Stochastic Ordering 3318.2.2 Permutation Tests for Umbrella Alternatives 3328.3 Tumour Growth Curves: Example 3358.4 PERC Data: Example 3368.4.1 Introduction 3368.4.2 A Permutation Solution 3399 NPC Tests for Survival Analysis 3419.1 Introduction and Main Notation 3419.1.1 Failure Time Distributions 3419.1.2 Data Structure 3429.2 Comparison of Survival Curves 3439.3 An Overview from the Literature 3459.3.1 Permutation Tests in Survival Analysis 3479.4 Two NPC Tests 3489.4.1 Breaking Down the Hypotheses 3499.4.2 The Test Structure 3509.4.3 NPC Test for Treatment-Independent Censoring 3509.4.4 NPC Test for Treatment-Dependent Censoring 3529.5 Case Study: In-Hospital Mortality 3559.5.1 Results 3559.6 Case Study: Esophageal Cancer 3569.6.1 A Permutation Solution 3589.6.2 Survival Analysis with Stratification by Propensity Score 3599.6.3 Integrating Propensity Score and NPC Testing 3619.7 Case Study: Comparison of Three Survival Curves 3669.7.1 Unstratified Survival Analysis 3679.7.2 Stratified Survival Analysis Using Propensity Score 36810 NPC Tests in Shape Analysis 37310.1 A Brief Overview of Statistical Shape Analysis 37510.1.1 How to Describe Shapes 37510.1.2 Multivariate Morphometrics 37710.2 Inference with Shape Data 38010.3 NPC Approach to Shape Analysis 38210.3.1 Comparative Simulation Study 38410.4 NPC Analysis with Correlated Landmarks 38710.5 An Application to Mediterranean Monk Seal Skulls 39010.5.1 Some Remarks 39311 Case Studies 39511.1 Autofluorescence Data 39511.1.1 A Permutation Solution 39711.1.2 Analysis Using R 39911.2 Confocal Data 40211.2.1 A Permutation Solution 40411.2.1.1 Evaluation of Treatment Effect Using R 40611.2.1.2 Evaluation of Time Effect Using R 40911.2.2 Two-Way (M)ANOVA 41111.2.2.1 Permutation Tests in Two-Way (M)ANOVA 41111.2.2.2 Analysis Using R 41311.3 SETIG Data 41411.3.1 Analysis Using R 41711.4 Logistic Regression and NPC Test for Multivariate Analysis 42011.4.1 Lymph Node Metastases Data 42011.4.2 Bladder Cancer Data 42311.4.2.1 Applying NPC: Some Results 42511.4.2.2 Analysis with Logistic Regression 42511.4.2.3 Final Comments 42712 Permutation Tests Over the Years 43112.1 Kind of Data 43212.1.1 Time-Series Data 43212.1.2 Thick Data 43212.1.3 Imaging Data 43512.1.4 Survival Data 43612.1.5 Censored Data 43612.2 Different Models 43712.2.1 Regression 43712.2.2 Classification 44012.2.3 Doe 44012.2.4 Ranking 44012.3 Different Tests 44112.3.1 Test for Centrality 44112.3.2 Test for Variability 44612.3.3 Test for Both Centrality and Variability 44612.3.4 ANOVA Test 44712.3.5 Testing Relationship 44712.3.6 NPC Tests 44812.4 Methodological and Computational Aspects 44912.4.1 Power 44912.4.2 Computational Problems 45212.4.3 Other Problems 452Appendix A Examples with Applications in R 453A. 1 Mult Data Example 453A. 2 Washing Test Data 456A. 3 Germina Data Example 460A. 4 Botulinum Data 464A. 5 Waterfalls Data 467A. 6 Tumour Growth Curves 469A. 7 PERC Data 471A. 8 Mediterranean Monk Seal Skulls 475Bibliography 479Index 517

FORTUNATO PESARIN is Emeritus Professor at the Department of Statistical Sciences, University of Padova, Italy. LUIGI SALMASO is Professor of Statistics at the Department of Management and Engineering, University of Padova, Italy.

• Preface xviiAcknowledgments xxiAcronyms xxiiiNotation xxvAbout the Companion Website xxix1 Introduction 11.1 On Permutation Analysis 11.2 Basic Notation 61.3 The Permutation – Conditional – Testing Principle 81.3.1 Nonparametric Family of Distributions 91.3.2 The Permutation – Conditional – Testing Principle 101.4 Permutation Approaches 131.5 When and Why Conditioning Is Appropriate 151.6 Randomization and Permutation 181.7 Computational Aspects 191.8 A Problem with Paired Observations 201.8.1 Modelling Responses 211.8.2 Symmetry Induced by Exchangeability 231.8.3 Further Aspects 241.8.4 Student’s t-Paired Solution 241.8.5 The Signed Rank Solution 261.8.6 The McNemar Solution 271.9 The Permutation Solution 281.9.1 General Aspects 281.9.2 The Permutation Sample Space 281.9.3 The Conditional Monte Carlo Method 291.9.3.1 An Algorithm for Inspecting Permutation Sample Spaces 291.9.3.2 Approximating the Permutation Distribution 311.9.4 Problems and Exercises 321.10 A Two-Sample Problem 331.10.1 Modelling Responses 341.10.2 The Student t Solution 361.10.3 The Permutation Solution 361.10.4 Rank Solutions 381.10.5 Fisher’s Exact Probability Solution 391.10.6 Problems and Exercises 391.11 One-Way ANOVA 401.11.1 Modelling Responses 411.11.2 Permutation Solutions 421.11.3 Problems and Exercises 432 Theory of One-Dimensional Permutation Tests 452.1 Introduction 452.1.1 Notation and Basic Assumptions 452.1.2 The Conditional Reference Space 492.1.3 Conditioning on a Set of Sufficient Statistics 552.2 Definition of Permutation Tests 592.2.1 General Aspects 592.2.2 Randomized Permutation Tests 602.2.3 Non-Randomized Permutation Tests 612.2.4 Unbiasedness of Permutation Tests 632.3 Some Useful Test Statistics 652.4 Equivalence of Permutation Statistics 692.4.1 Two Examples 702.4.2 Problems and Exercises 722.5 Further Relevant Properties 742.5.1 The p-Value-Like Statistic 742.5.2 Unbiasedness of p-Value-Like Test Statistics 752.5.3 Weak Consistency of Permutation Tests 772.5.4 A CMC Algorithm for Estimating the p-Value-Like Statistic 802.6 Some Arguments for Choosing Permutation Tests 822.7 Power Functions of Permutation Tests 872.7.1 Definition and Algorithm for the Conditional Power 872.7.2 The Conditional ROC Curve 902.7.3 Definition and Algorithm for the Unconditional Power: Fixed Effects 902.7.4 Unconditional Power: Random Effects 922.7.5 Comments on Power Functions 922.8 Permutation Confidence Interval for δ 932.8.1 Problems and Exercises 982.9 Generalizing Inference from Conditional to Unconditional 992.10 Optimal Properties 1032.10.1 Problems and Exercises 1052.11 Some Further Asymptotic Properties 1072.11.1 Introduction 1072.11.2 Two Basic Theorems 1072.12 Permutation Central Limit Theorems 1092.12.1 Basic Notions 1092.12.2 Permutation Central Limit Theorems 1102.12.3 Problems and Exercises 1133 Some Examples of Unidimensional Permutation Tests 1173.1 Examples of One-Sample Problems 1173.1.1 A Problem with Repeated Observations 1253.1.1.1 Friedman’s Rank Test 1263.1.1.2 A Permutation Solution 1263.1.1.3 An Example 1273.1.2 Problems and Exercises 1283.2 Some Examples of Multi-Sample Problems 1303.2.1 Synchronized Permutations 1393.3 Analysis of Ordered Categorical Variables 1453.3.1 General Aspects 1453.3.2 A Solution Based on Score Transformations 1473.3.3 Typical Goodness-of-Fit Solutions 1483.3.3.1 An Example 1493.3.4 Extension to Non-Dominance Alternatives and C Samples 1493.3.5 Problems and Exercises 1514 The Nonparametric Combination Methodology 1534.1 Introduction 1534.1.1 General Aspects 1534.1.2 Some Preliminary Notes 1554.1.3 Main Assumptions and Notation 1574.1.4 Some Comments 1594.2 The Nonparametric Combination Methodology 1604.2.1 Assumptions on Partial Tests 1604.2.2 Desirable Properties of Combining Functions 1624.2.3 A Two-Phase Algorithm for Nonparametric Combination 1644.2.4 Some Useful Combining Functions 1684.2.5 Why Combination Is Nonparametric 1774.2.6 On Admissible Combining Functions 1784.2.7 Problems and Exercises 1794.3 Consistency, Unbiasedness, and Power of Combined Tests 1824.3.1 Consistency 1824.3.2 Unbiasedness 1824.3.3 A Non-Consistent Combining Function 1844.3.4 Power of Combined Tests 1854.3.4.1 The Conditional Power Function 1854.3.4.2 The Unconditional Power Function 1864.3.5 Multivariate Conditional Confidence Region for δ 1874.3.6 Problems and Exercises 1884.4 Some Further Asymptotic Properties 1894.4.1 General Conditions 1894.4.2 Some Asymptotic Properties 1904.5 Finite-Sample Consistency 1944.5.1 Introduction and Motivations 1944.5.2 Finite-Sample Consistency 1954.5.3 Some Applications of Finite-Sample Consistency 2014.6 Some Examples of Nonparametric Combination 2074.6.1 Problems and Exercises 2294.7 Comments on the Nonparametric Combination 2314.7.1 General Comments 2315 Multiple Testing Problems and Multiplicity Adjustment 2335.1 Defining Raw and Adjusted p-Values 2335.2 Controlling for Multiplicity 2355.2.1 Multiple Comparison and Multiple Testing 2355.2.2 Some Definitions of the Global Type I Error 2365.3 Multiple Testing 2375.4 The Closed Testing Approach 2385.4.1 Closed Testing for Multiple Testing 2395.4.2 Closed Testing Using the MinP Bonferroni–Holm Procedure 2415.5 Weighted Methods for Controlling FWE and FDR 2455.6 Adjusting Stepwise p-Values 2475.6.1 Showing Biasedness of Standard p-Values for Stepwise Regression 2485.6.2 Algorithm Description 2485.6.3 Optimal Subset Procedures 2506 Analysis of Multivariate Categorical Variables 2516.1 Introduction 2516.2 The Multivariate McNemar Test 2526.2.1 An Extension of the Multivariate McNemar Test 2556.3 Multivariate Goodness-of-Fit Testing for Ordered Variables 2566.3.1 Multivariate Extension of Fisher’s Exact Probability Test 2586.4 MANOVA with Nominal Categorical Data 2586.4.1 A Formal Description 2586.5 Stochastic Ordering 2606.5.1 Formal Description 2606.5.2 Further Breaking Down of the Hypotheses 2616.5.3 Permutation Test 2616.6 Multifocus Analysis 2646.6.1 General Aspects 2646.6.2 The Multifocus Solution 2646.6.3 An Application 2676.7 Isotonic Inference 2686.7.1 Introduction 2686.7.2 Allelic Association Analysis in Genetics 2706.7.3 Parametric Solutions 2716.7.3.1 Chiano and Clayton’s Method 2716.7.3.2 Maximum-Likelihood Approach 2726.7.4 Permutation Approach 2736.8 Test on Moments for Ordered Variables 2746.8.1 General Aspects 2746.8.2 Score Transformations and Univariate Tests 2756.8.3 Multivariate Extension 2766.9 Heterogeneity Comparisons 2786.9.1 Introduction 2786.9.2 Tests for Comparing Heterogeneities 2796.9.3 A Case Study in Population Genetics 2816.10 Application to PhD Programme Evaluation 2826.10.1 Description of the Problem 2826.10.2 Global Satisfaction Index 2846.10.3 Multivariate Performance Comparisons 2857 Permutation Testing for Repeated Measurements 2877.1 Introduction 2877.2 Carry-Over Effects in Repeated-Measures Designs 2887.3 Modelling Repeated Measurements 2897.3.1 A General Additive Model 2897.3.2 The Hypotheses of Interest 2917.4 Testing Solutions 2927.4.1 Solutions Using the NPC Approach 2927.4.2 Analysis of Two-Sample Dominance Problems 2947.4.3 Analysis of the Cross-Over (AB–BA) Design 2947.4.4 Analysis of a Cross-Over Design with Paired Data 2957.5 Testing for Repeated Measurements with Missing Data 2967.5.1 General Aspects of Permutation Testing with Missing Data 2977.5.2 Bibliographic Notes 2977.6 On Missing Data Processes 2987.6.1 Data Missing Completely at Random 2997.6.2 Data Missing Not at Random 2997.7 The Permutation Approach 3007.7.1 Deletion, Imputation, and Intention to Treat Strategies 3017.7.2 Breaking Down the Hypotheses 3037.8 The Structure of Testing Problems 3037.8.1 Hypotheses for MNAR Models 3037.8.2 Hypotheses for MCAR Models 3057.8.3 Permutation Structure with Missing Values 3057.9 Permutation Analysis of Missing Values 3077.9.1 Partitioning the Permutation Sample Space 3077.9.2 Solution for Two-Sample MCAR Problems 3087.9.3 Extensions to Multivariate C-Sample Problems 3107.9.4 Extension to MNAR Models 3117.10 ‘GERMINA’ Data: An Example of a MNAR Model 3127.10.1 Problem Description 3137.10.2 The Permutation Solution 3157.11 Multivariate Paired Observations 3187.11.1 The Problem 3187.12 Repeated Measures and Missing Data 3197.12.1 An Example 3207.13 Botulinum Data: Example 3227.13.1 Data Description 3227.13.2 Research Questions and Methods 3237.14 Waterfalls Data: Example 3247.14.1 Data Description 3247.14.2 Research Questions and Methods 3258 Some Stochastic Ordering Problems 3278.1 Multivariate Ordered Alternatives 3278.2 Testing for Umbrella Alternatives 3298.2.1 Hypotheses and Tests in Simple Stochastic Ordering 3318.2.2 Permutation Tests for Umbrella Alternatives 3328.3 Tumour Growth Curves: Example 3358.4 PERC Data: Example 3368.4.1 Introduction 3368.4.2 A Permutation Solution 3399 NPC Tests for Survival Analysis 3419.1 Introduction and Main Notation 3419.1.1 Failure Time Distributions 3419.1.2 Data Structure 3429.2 Comparison of Survival Curves 3439.3 An Overview from the Literature 3459.3.1 Permutation Tests in Survival Analysis 3479.4 Two NPC Tests 3489.4.1 Breaking Down the Hypotheses 3499.4.2 The Test Structure 3509.4.3 NPC Test for Treatment-Independent Censoring 3509.4.4 NPC Test for Treatment-Dependent Censoring 3529.5 Case Study: In-Hospital Mortality 3559.5.1 Results 3559.6 Case Study: Esophageal Cancer 3569.6.1 A Permutation Solution 3589.6.2 Survival Analysis with Stratification by Propensity Score 3599.6.3 Integrating Propensity Score and NPC Testing 3619.7 Case Study: Comparison of Three Survival Curves 3669.7.1 Unstratified Survival Analysis 3679.7.2 Stratified Survival Analysis Using Propensity Score 36810 NPC Tests in Shape Analysis 37310.1 A Brief Overview of Statistical Shape Analysis 37510.1.1 How to Describe Shapes 37510.1.2 Multivariate Morphometrics 37710.2 Inference with Shape Data 38010.3 NPC Approach to Shape Analysis 38210.3.1 Comparative Simulation Study 38410.4 NPC Analysis with Correlated Landmarks 38710.5 An Application to Mediterranean Monk Seal Skulls 39010.5.1 Some Remarks 39311 Case Studies 39511.1 Autofluorescence Data 39511.1.1 A Permutation Solution 39711.1.2 Analysis Using R 39911.2 Confocal Data 40211.2.1 A Permutation Solution 40411.2.1.1 Evaluation of Treatment Effect Using R 40611.2.1.2 Evaluation of Time Effect Using R 40911.2.2 Two-Way (M)ANOVA 41111.2.2.1 Permutation Tests in Two-Way (M)ANOVA 41111.2.2.2 Analysis Using R 41311.3 SETIG Data 41411.3.1 Analysis Using R 41711.4 Logistic Regression and NPC Test for Multivariate Analysis 42011.4.1 Lymph Node Metastases Data 42011.4.2 Bladder Cancer Data 42311.4.2.1 Applying NPC: Some Results 42511.4.2.2 Analysis with Logistic Regression 42511.4.2.3 Final Comments 42712 Permutation Tests Over the Years 43112.1 Kind of Data 43212.1.1 Time-Series Data 43212.1.2 Thick Data 43212.1.3 Imaging Data 43512.1.4 Survival Data 43612.1.5 Censored Data 43612.2 Different Models 43712.2.1 Regression 43712.2.2 Classification 44012.2.3 Doe 44012.2.4 Ranking 44012.3 Different Tests 44112.3.1 Test for Centrality 44112.3.2 Test for Variability 44612.3.3 Test for Both Centrality and Variability 44612.3.4 ANOVA Test 44712.3.5 Testing Relationship 44712.3.6 NPC Tests 44812.4 Methodological and Computational Aspects 44912.4.1 Power 44912.4.2 Computational Problems 45212.4.3 Other Problems 452Appendix A Examples with Applications in R 453A. 1 Mult Data Example 453A. 2 Washing Test Data 456A. 3 Germina Data Example 460A. 4 Botulinum Data 464A. 5 Waterfalls Data 467A. 6 Tumour Growth Curves 469A. 7 PERC Data 471A. 8 Mediterranean Monk Seal Skulls 475Bibliography 479Index 517