Results: permutation.sas

Permutation and randomization tests

The Print Procedure

Data Set WORK.SENIC

Obs id region mdschl census nbeds nurses lngstay age xratio culratio infpercent agecat quality mschool ne nc s w
1 1 Northeast No 237 298 115 12.01 52.8 96.9 10.8 4.8 53 & under 790.49 0 Yes No No No
2 2 Northeast Yes 144 184 151 10.05 52.0 87.5 36.7 4.5 53 & under 1707.90 1 Yes No No No
3 3 Northeast No 127 165 158 9.36 54.1 90.6 18.3 4.8 Over 53 775.64 0 Yes No No No
4 4 Northeast Yes 240 270 198 9.78 52.3 95.9 17.6 5.0 53 & under 1781.44 1 Yes No No No
5 5 West No 51 76 79 6.70 48.6 80.8 13.0 4.5 53 & under 579.30 0 No No No Yes
6 6 South No 59 95 56 8.93 56.0 72.5 6.2 2.0 Over 53 388.07 0 No No Yes No
7 7 South No 468 600 497 9.84 62.2 82.3 12.0 4.8 Over 53 1158.66 0 No No Yes No
8 8 South No 349 477 188 7.91 52.8 79.5 11.9 2.9 53 & under 775.59 0 No No Yes No
9 9 South No 230 297 263 9.44 52.5 58.5 10.9 4.5 53 & under 649.06 0 No No Yes No
10 10 North Central Yes 595 752 446 10.23 53.2 77.9 9.9 . Over 53 2501.54 1 No Yes No No

Permutation and randomization tests

The SENIC data

Fisher's exact test is part of the default output

The FREQ Procedure

The Freq Procedure

Table mdschl * agecat

Cross-Tabular Freq Table

Frequency
Row Pct
Table of mdschl by agecat
mdschl(Medical school affiliation) agecat(Average patient age)
53 & under Over 53 Total
No
35
44.30
44
55.70
79
 
Yes
10
62.50
6
37.50
16
 
Total
45
50
95
Frequency Missing = 5

Statistics for Table of mdschl by agecat

Chi-Square Tests

Statistic DF Value Prob
Chi-Square 1 1.7671 0.1837
Likelihood Ratio Chi-Square 1 1.7750 0.1828
Continuity Adj. Chi-Square 1 1.1126 0.2915
Mantel-Haenszel Chi-Square 1 1.7485 0.1861
Phi Coefficient   -0.1364  
Contingency Coefficient   0.1351  
Cramer's V   -0.1364  

Fisher's Exact Test

Fisher's Exact Test
Cell (1,1) Frequency (F) 35
Left-sided Pr <= F 0.1458
Right-sided Pr >= F 0.9460
   
Table Probability (P) 0.0919
Two-sided Pr <= P 0.2723

Sample Size = 95
Frequency Missing = 5

Permutation and randomization tests

The SENIC data

An exact permutation test

The FREQ Procedure

The Freq Procedure

Table mdschl * region

Cross-Tabular Freq Table

Frequency
Col Pct
Table of mdschl by region
mdschl(Medical school affiliation) region(Region of U.S.A.)
Northeast North Central South West Total
No
19
79.17
24
80.00
24
88.89
13
86.67
80
 
Yes
5
20.83
6
20.00
3
11.11
2
13.33
16
 
Total
24
30
27
15
96
Frequency Missing = 4

Statistics for Table of mdschl by region

Chi-Square Tests

Statistic DF Value Prob
WARNING: 38% of the cells have expected counts less than 5.
(Asymptotic) Chi-Square may not be a valid test.
Chi-Square 3 1.2600 0.7387
Likelihood Ratio Chi-Square 3 1.3029 0.7284
Mantel-Haenszel Chi-Square 1 0.8758 0.3493
Phi Coefficient   0.1146  
Contingency Coefficient   0.1138  
Cramer's V   0.1146  

Pearson Chi-Square Test

Pearson Chi-Square Test
Chi-Square 1.2600
DF 3
Asymptotic Pr > ChiSq 0.7387
Exact Pr >= ChiSq 0.7849

Sample Size = 96
Frequency Missing = 4

Permutation and randomization tests

The scab disease data

The Print Procedure

Data Set WORK.SCAB

Obs Condition Infection
1 Control 35.26
2 Control 30.69
3 Control 15.56
4 Control 31.59
5 Control 15.91
6 Control 15.77
7 Control 19.07
8 Control 17.15
9 Spring300 19.96
10 Spring300 30.12
11 Spring300 11.78
12 Spring300 5.13
13 Spring600 20.63
14 Spring600 22.75
15 Spring600 18.22
16 Spring600 11.4
17 Spring1200 19.84
18 Spring1200 15.45
19 Spring1200 8.11
20 Spring1200 13.61
21 Fall300 5.52
22 Fall300 4.8
23 Fall300 5.08
24 Fall300 12.59
25 Fall600 16.14
26 Fall600 14.54
27 Fall600 11.1
28 Fall600 20.23
29 Fall1200 2.01
30 Fall1200 8.48
31 Fall1200 7.43
32 Fall1200 5.08

Permutation and randomization tests

The scab disease data

Overall F-test with proc glm for Comparison

The GLM Procedure

 

Dependent Variable: Infection Ave percent surface covered

The GLM Procedure

Analysis of Variance

Infection

Overall ANOVA

Source DF Sum of Squares Mean Square F Value Pr > F
Model 6 1117.463850 186.243975 4.26 0.0043
Error 25 1091.830900 43.673236    
Corrected Total 31 2209.294750      

Permutation and randomization tests

The scab disease data

Classical Kruskal-Wallis Test: Compare F = 4.26, p = 0.0043

The NPAR1WAY Procedure

The Npar1way Procedure

Variable Infection

Wilcoxon Analysis

Scores

Wilcoxon Scores (Rank Sums) for Variable Infection
Classified by Variable Condition
Condition N Sum of
Scores		 Expected
Under H0		 Std Dev
Under H0		 Mean
Score
Average scores were used for ties.
Control 8 191.00 132.0 22.976145 23.8750
Spring300 4 71.00 66.0 17.548320 17.7500
Spring600 4 88.00 66.0 17.548320 22.0000
Spring1200 4 62.00 66.0 17.548320 15.5000
Fall300 4 24.50 66.0 17.548320 6.1250
Fall600 4 71.00 66.0 17.548320 17.7500
Fall1200 4 20.50 66.0 17.548320 5.1250

Kruskal-Wallis Test

Kruskal-Wallis Test
Chi-Square DF Pr > ChiSq
17.2844 6 0.0083

Box Plot

Box Plot of Wilcoxon Scores for Infection Classified by Condition

Permutation and randomization tests

The scab disease data

Randomization Test

The NPAR1WAY Procedure

The Npar1way Procedure

Variable Infection

Analysis of Variance

Class Means

Analysis of Variance for Variable Infection
Classified by Variable Condition
Condition N Mean
Control 8 22.62500
Spring300 4 16.74750
Spring600 4 18.25000
Spring1200 4 14.25250
Fall300 4 6.99750
Fall600 4 15.50250
Fall1200 4 5.75000

ANOVA

Source DF Sum of Squares Mean Square F Value Pr > F
Average scores were used for ties.
Among 6 1117.463850 186.243975 4.2645 0.0043
Within 25 1091.830900 43.673236    

Box Plot

Box Plot of Infection Classified by Condition

Permutation and randomization tests

The scab disease data

Randomization Test

The NPAR1WAY Procedure

Data Scores Analysis

Scores

Data Scores for Variable Infection
Classified by Variable Condition
Condition N Sum of
Scores		 Expected
Under H0		 Std Dev
Under H0		 Mean
Score
Control 8 181.000 122.7500 20.678623 22.62500
Spring300 4 66.990 61.3750 15.793559 16.74750
Spring600 4 73.000 61.3750 15.793559 18.25000
Spring1200 4 57.010 61.3750 15.793559 14.25250
Fall300 4 27.990 61.3750 15.793559 6.99750
Fall600 4 62.010 61.3750 15.793559 15.50250
Fall1200 4 23.000 61.3750 15.793559 5.75000

One-Way Analysis

Data Scores One-Way Analysis
Chi-Square DF Pr > ChiSq
15.6798 6 0.0156

Monte Carlo Estimates for the Exact Test

Monte Carlo Estimates for the Exact Test
Probability Estimate 99% Confidence Limits Samples Seed
Pr >= ChiSq 0.0041 0.0025 0.0057 10000 88888

Box Plot

Box Plot of Data Scores for Infection Classified by Condition

Permutation and randomization tests

Randomization multiple comparisons with proc multtest

The Multtest Procedure

The Multtest Procedure

Model Information

Model Information
Test for continuous variables Mean t-test
Degrees of Freedom Method Pooled
Tails for continuous tests Two-tailed
Strata weights None
P-value adjustment Permutation
Center continuous variables No
Number of resamples 20000
Seed 99999

Contrast Coefficients

Contrast Coefficients
Contrast   Condition
Control Spring300 Spring600 Spring1200 Fall300 Fall600 Fall1200
Control vs. Spring300 Centered 1 -1 0 0 0 0 0
Control vs. Spring600 Centered 1 0 -1 0 0 0 0
Control vs. Spring120 Centered 1 0 0 -1 0 0 0
Control vs. Fall300 Centered 1 0 0 0 -1 0 0
Control vs. Fall600 Centered 1 0 0 0 0 -1 0
Control vs. Fall1200 Centered 1 0 0 0 0 0 -1
Spring300 vs. Spring6 Centered 0 1 -1 0 0 0 0
Spring300 vs. Spring1 Centered 0 1 0 -1 0 0 0
Spring300 vs. Fall300 Centered 0 1 0 0 -1 0 0
Spring300 vs. Fall600 Centered 0 1 0 0 0 -1 0
Spring300 vs. Fall120 Centered 0 1 0 0 0 0 -1
Spring600 vs. Spring1 Centered 0 0 1 -1 0 0 0
Spring600 vs. Fall300 Centered 0 0 1 0 -1 0 0
Spring600 vs. Fall600 Centered 0 0 1 0 0 -1 0
Spring600 vs. Fall120 Centered 0 0 1 0 0 0 -1
Spring1200 vs. Fall30 Centered 0 0 0 1 -1 0 0
Spring1200 vs. Fall60 Centered 0 0 0 1 0 -1 0
Spring1200 vs. Fall12 Centered 0 0 0 1 0 0 -1
Fall300 vs. Fall600 Centered 0 0 0 0 1 -1 0
Fall300 vs. Fall1200 Centered 0 0 0 0 1 0 -1
Fall600 vs. Fall1200 Centered 0 0 0 0 0 1 -1
Control vs. Spring Centered 3 -1 -1 -1 0 0 0
Control vs. Fall Centered 3 0 0 0 -1 -1 -1
Spring vs. Fall Centered 0 1 1 1 -1 -1 -1

Continuous Variable Tabulations

Continuous Variable Tabulations
Variable Condition NumObs Mean Standard Deviation
Infection Control 8 22.6250 8.3638
Infection Spring300 4 16.7475 10.7825
Infection Spring600 4 18.2500 4.9274
Infection Spring1200 4 14.2525 4.8579
Infection Fall300 4 6.9975 3.7401
Infection Fall600 4 15.5025 3.7888
Infection Fall1200 4 5.7500 2.8701

P-Values

p-Values
Variable Contrast Raw Permutation
Infection Control vs. Spring300 0.1588 0.7717
Infection Control vs. Spring600 0.2900 0.9303
Infection Control vs. Spring120 0.0490 0.4017
Infection Control vs. Fall300 0.0007 0.0119
Infection Control vs. Fall600 0.0906 0.5909
Infection Control vs. Fall1200 0.0003 0.0057
Infection Spring300 vs. Spring6 0.7505 1.0000
Infection Spring300 vs. Spring1 0.5981 0.9981
Infection Spring300 vs. Fall300 0.0473 0.3914
Infection Spring300 vs. Fall600 0.7921 1.0000
Infection Spring300 vs. Fall120 0.0268 0.2579
Infection Spring600 vs. Spring1 0.4004 0.9777
Infection Spring600 vs. Fall300 0.0237 0.2348
Infection Spring600 vs. Fall600 0.5618 0.9970
Infection Spring600 vs. Fall120 0.0130 0.1473
Infection Spring1200 vs. Fall30 0.1331 0.7162
Infection Spring1200 vs. Fall60 0.7913 1.0000
Infection Spring1200 vs. Fall12 0.0808 0.5507
Infection Fall300 vs. Fall600 0.0807 0.5502
Infection Fall300 vs. Fall1200 0.7917 1.0000
Infection Fall600 vs. Fall1200 0.0472 0.3911
Infection Control vs. Spring 0.0501 0.4076
Infection Control vs. Fall 0.0002 0.0035
Infection Spring vs. Fall 0.0156 0.1702

Permutation and randomization tests

Randomization multiple comparisons with proc multtest

Output data set with permutation p-values

The Print Procedure

Data Set WORK.PVALS

Obs _test_ _var_ _contrast_ _value_ _se_ _nval_ raw_p perm_p sim_se
1 MEAN Infection Control vs. Spring300 188.08 129.501 25 0.15884 0.77165 .002968217
2 MEAN Infection Control vs. Spring600 140.00 129.501 25 0.28998 0.93030 .001800582
3 MEAN Infection Control vs. Spring120 267.92 129.501 25 0.04905 0.40170 .003466534
4 MEAN Infection Control vs. Fall300 500.08 129.501 25 0.00071 0.01190 .000766759
5 MEAN Infection Control vs. Fall600 227.92 129.501 25 0.09064 0.59090 .003476616
6 MEAN Infection Control vs. Fall1200 540.00 129.501 25 0.00032 0.00570 .000532330
7 MEAN Infection Spring300 vs. Spring6 -48.08 149.535 25 0.75048 1.00000 0
8 MEAN Infection Spring300 vs. Spring1 79.84 149.535 25 0.59811 0.99810 .000307928
9 MEAN Infection Spring300 vs. Fall300 312.00 149.535 25 0.04729 0.39135 .003451052
10 MEAN Infection Spring300 vs. Fall600 39.84 149.535 25 0.79210 1.00000 0
11 MEAN Infection Spring300 vs. Fall120 351.92 149.535 25 0.02677 0.25785 .003093246
12 MEAN Infection Spring600 vs. Spring1 127.92 149.535 25 0.40043 0.97765 .001045239
13 MEAN Infection Spring600 vs. Fall300 360.08 149.535 25 0.02374 0.23475 .002997019
14 MEAN Infection Spring600 vs. Fall600 87.92 149.535 25 0.56184 0.99700 .000386717
15 MEAN Infection Spring600 vs. Fall120 400.00 149.535 25 0.01299 0.14730 .002506020
16 MEAN Infection Spring1200 vs. Fall30 232.16 149.535 25 0.13310 0.71615 .003188096
17 MEAN Infection Spring1200 vs. Fall60 -40.00 149.535 25 0.79128 1.00000 0
18 MEAN Infection Spring1200 vs. Fall12 272.08 149.535 25 0.08083 0.55070 .003517311
19 MEAN Infection Fall300 vs. Fall600 -272.16 149.535 25 0.08075 0.55015 .003517705
20 MEAN Infection Fall300 vs. Fall1200 39.92 149.535 25 0.79169 1.00000 0
21 MEAN Infection Fall600 vs. Fall1200 312.08 149.535 25 0.04724 0.39110 .003450658
22 MEAN Infection Control vs. Spring 596.00 289.573 25 0.05014 0.40760 .003474638
23 MEAN Infection Control vs. Fall 1268.00 289.573 25 0.00019 0.00345 .000414614
24 MEAN Infection Spring vs. Fall 672.00 259.002 25 0.01562 0.17015 .002657056

Permutation and randomization tests

Randomization multiple comparisons with proc multtest

What is that critical value?

The IML Procedure

CritVal

CritVal
2.5758293

Permutation and randomization tests

Randomization multiple comparisons with proc multtest

99% confidence intervals for the permutation p-values

The Print Procedure

Data Set WORK.CI

Obs _contrast_ perm_p Lower99 Upper99
1 Control vs. Spring300 0.77165 0.76400 0.77930
2 Control vs. Spring600 0.93030 0.92566 0.93494
3 Control vs. Spring120 0.40170 0.39277 0.41063
4 Control vs. Fall300 0.01190 0.00992 0.01388
5 Control vs. Fall600 0.59090 0.58194 0.59986
6 Control vs. Fall1200 0.00570 0.00433 0.00707
7 Spring300 vs. Spring6 1.00000 1.00000 1.00000
8 Spring300 vs. Spring1 0.99810 0.99731 0.99889
9 Spring300 vs. Fall300 0.39135 0.38246 0.40024
10 Spring300 vs. Fall600 1.00000 1.00000 1.00000
11 Spring300 vs. Fall120 0.25785 0.24988 0.26582
12 Spring600 vs. Spring1 0.97765 0.97496 0.98034
13 Spring600 vs. Fall300 0.23475 0.22703 0.24247
14 Spring600 vs. Fall600 0.99700 0.99600 0.99800
15 Spring600 vs. Fall120 0.14730 0.14084 0.15376
16 Spring1200 vs. Fall30 0.71615 0.70794 0.72436
17 Spring1200 vs. Fall60 1.00000 1.00000 1.00000
18 Spring1200 vs. Fall12 0.55070 0.54164 0.55976
19 Fall300 vs. Fall600 0.55015 0.54109 0.55921
20 Fall300 vs. Fall1200 1.00000 1.00000 1.00000
21 Fall600 vs. Fall1200 0.39110 0.38221 0.39999
22 Control vs. Spring 0.40760 0.39865 0.41655
23 Control vs. Fall 0.00345 0.00238 0.00452
24 Spring vs. Fall 0.17015 0.16331 0.17699

Permutation and randomization tests

The SENIC Data

Pearson Product-moment Correlations

The CORR Procedure

The Corr Procedure

Variables Information

4 Variables: nurses lngstay age infpercent

Pearson Correlations

Pearson Correlation Coefficients
Prob > |r| under H0: Rho=0
Number of Observations
  nurses lngstay age infpercent
nurses
Aver # nurses during study period
1.00000
 
97
0.34813
0.0005
97
-0.01094
0.9157
96
0.42388
<.0001
93
lngstay
Av length of hospital stay, in days
0.34813
0.0005
97
1.00000
 
100
0.19828
0.0491
99
0.53491
<.0001
96
age
Average patient age
-0.01094
0.9157
96
0.19828
0.0491
99
1.00000
 
99
-0.00149
0.9886
95
infpercent
Percent acquiring infection in hospital
0.42388
<.0001
93
0.53491
<.0001
96
-0.00149
0.9886
95
1.00000
 
96

Permutation and randomization tests

The SENIC Data

Spearman Rank Correlations

The CORR Procedure

The Corr Procedure

Variables Information

4 Variables: nurses lngstay age infpercent

Spearman Correlations

Spearman Correlation Coefficients
Prob > |r| under H0: Rho=0
Number of Observations
  nurses lngstay age infpercent
nurses
Aver # nurses during study period
1.00000
 
97
0.49178
<.0001
97
-0.05074
0.6234
96
0.53123
<.0001
93
lngstay
Av length of hospital stay, in days
0.49178
<.0001
97
1.00000
 
100
0.12404
0.2212
99
0.53448
<.0001
96
age
Average patient age
-0.05074
0.6234
96
0.12404
0.2212
99
1.00000
 
99
-0.04065
0.6957
95
infpercent
Percent acquiring infection in hospital
0.53123
<.0001
93
0.53448
<.0001
96
-0.04065
0.6957
95
1.00000
 
96

The Freq Procedure

Table lngstay * age

Permutation and randomization tests

The SENIC Data

Randomization test on ranks: Compare p = 0.2212

The FREQ Procedure

Statistics for Table of lngstay by age

(Rows and Columns with Zero Totals Excluded)

Measures of Association

Statistic Value ASE
Gamma 0.0787 0.0770
Kendall's Tau-b 0.0783 0.0766
Stuart's Tau-c 0.0782 0.0765
Somers' D C|R 0.0781 0.0764
Somers' D R|C 0.0786 0.0768
Pearson Correlation 0.1983 0.1033
Spearman Correlation 0.1240 0.1078
Lambda Asymmetric C|R 0.9053 0.0300
Lambda Asymmetric R|C 0.7188 0.0465
Lambda Symmetric 0.8115 0.0307
Uncertainty Coefficient C|R 0.9684 0.0071
Uncertainty Coefficient R|C 0.9023 0.0117
Uncertainty Coefficient Symmetric 0.9342 0.0072

Spearman Correlation Coefficient

Spearman Correlation Coefficient
Correlation (r) 0.1240
ASE 0.1078
95% Lower Conf Limit -0.0873
95% Upper Conf Limit 0.3354

Spearman Correlation Test

Test of H0: Correlation = 0
ASE under H0 0.1078
Z 1.1504
One-sided Pr > Z 0.1250
Two-sided Pr > |Z| 0.2500

Monte Carlo Estimates for the Exact Test

Monte Carlo Estimates for the Exact Test
One-sided Pr >= r  
Estimate 0.1113
99% Lower Conf Limit 0.1032
99% Upper Conf Limit 0.1194
   
Two-sided Pr >= |r|  
Estimate 0.2299
99% Lower Conf Limit 0.2191
99% Upper Conf Limit 0.2407
   
Number of Samples 10000
Initial Seed 7777

Sample Size = 99
Frequency Missing = 1

The Freq Procedure

Table lngstay * age

Permutation and randomization tests

The SENIC Data

Randomization test on raw numbers: Compare p = 0.0491

The FREQ Procedure

Statistics for Table of lngstay by age

(Rows and Columns with Zero Totals Excluded)

Measures of Association

Statistic Value ASE
Gamma 0.0787 0.0770
Kendall's Tau-b 0.0783 0.0766
Stuart's Tau-c 0.0782 0.0765
Somers' D C|R 0.0781 0.0764
Somers' D R|C 0.0786 0.0768
Pearson Correlation 0.1983 0.1033
Spearman Correlation 0.1240 0.1078
Lambda Asymmetric C|R 0.9053 0.0300
Lambda Asymmetric R|C 0.7188 0.0465
Lambda Symmetric 0.8115 0.0307
Uncertainty Coefficient C|R 0.9684 0.0071
Uncertainty Coefficient R|C 0.9023 0.0117
Uncertainty Coefficient Symmetric 0.9342 0.0072

Pearson Correlation Coefficient

Pearson Correlation Coefficient
Correlation (r) 0.1983
ASE 0.1033
95% Lower Conf Limit -0.0042
95% Upper Conf Limit 0.4008

Pearson Correlation Test

Test of H0: Correlation = 0
ASE under H0 0.1262
Z 1.5717
One-sided Pr > Z 0.0580
Two-sided Pr > |Z| 0.1160

Monte Carlo Estimates for the Exact Test

Monte Carlo Estimates for the Exact Test
One-sided Pr >= r  
Estimate 0.0248
99% Lower Conf Limit 0.0208
99% Upper Conf Limit 0.0288
   
Two-sided Pr >= |r|  
Estimate 0.0505
99% Lower Conf Limit 0.0449
99% Upper Conf Limit 0.0561
   
Number of Samples 10000
Initial Seed 7777

Sample Size = 99
Frequency Missing = 1

Permutation and randomization tests

The SENIC Data

Randomization test on raw numbers: Big Monte Carlo sample size

The FREQ Procedure

 

Statistics for Table of lngstay by age

The Freq Procedure

Table lngstay * age

Monte Carlo Estimates for the Exact Test

Monte Carlo Estimates for the Exact Test
One-sided Pr >= r  
Estimate 0.0244
99% Lower Conf Limit 0.0241
99% Upper Conf Limit 0.0247
   
Two-sided Pr >= |r|  
Estimate 0.0507
99% Lower Conf Limit 0.0503
99% Upper Conf Limit 0.0511
   
Number of Samples 2000000
Initial Seed 7777