Sample size estimations for unpaired differences in measurements are provided in the Sample Size for Unpaired Differences Program Page , which provides two algorithms.

In the common situation where there are only two groups, sample size estimation is based on the z and t distributions, and four programs are provided.

• When there is no information available regarding a proposed research project, a pilot study is sometimes useful. The idea is to carry out a small and preliminary study, to test the feasibility of conducting the project, and to obtain some basic parameters to plan the main project. The sample size required here can be much smaller than that required for statistical inference, but nevertheless be informative. The emphasis is cost efficiency. The combination of a nominated Standard Deviation and sample size allows the calculation of the 95% confidence interval of the results. How this confidence interval decreases with increasing sample size is examined, and a decision on the most cost effective sample size made. Regardless of the values used however, most pilot study reach peak cost effectiveness between 6 and 30 subjects per group.
• Sample size calculation is used in the planning phase of the main research project, estimating sample size (per group) required, based on the following parameters.
  • The probability of Type I Error (α), to be used to decide whether the null hypothesis is to be rejected. The most common value is α=0.05
  • The power of the research model (1-β), the probability of detecting the difference if it is present. The most common value used is 0.8
  • The effect size, the ratio between the difference to be detected, and the known population or within group Standard Deviation. Two approaches are often used
    • If an accurate estimation of the Standard Deviation is known, and a clinically meaningful difference is determined, these values can be used in the calculation
    • If these values cannot be accurately determined, a ratio of 0.2 can be used to detect a small effect size, 0.8 for a large effect size, and 0.5 in most clinical scenarios for a moderate effect size. The values to be entered are then the effect size as the difference to be detected, with the Standard Deviation = 1
• Power estimation is useful at the data analysis phase of a research project, particularly if the Type I Error (α) is too large (>0.05) to reject the null hypothesis
  • If a high α value is because the difference between the means is smaller than the critical value defined at the time of planning, then the correct decision is to reject the null hypothesis and accept the alternative hypothesis.
  • Often however the difference between the means exceed the critical value defined during planning, but a high α value is related to Standard Deviations larger than that envisaged during planning. When this happens, power estimation provides an estimate of the extent of discrepancy between planning and the reality reflected by the data, allowing researchers to interpret the results with greater nuance or to provide remedies (such as increasing the sample size) to validate the results.
• Estimating the confidence interval is used to provide an estimation of 95% confidence interval of the difference found, as an alternative to the probability of Type I Error (α)

If the hypothesis to be tested is not whether a significant difference exists, but whether a significant no-difference exists (group 1 not less or not more than group 2, or equivalence), the model requires greater power and can relax robustness. A Type I Error of 0.2 instead of 0.05, and the power of 0.95 instead of 0.8, are commonly used.

When more than two groups are being compared, calculations for the sample size is more complicated. Two approaches can be used.

• Machin et.al., in their book on sample size (see references), suggested that sample size estimation should be the same as that for two groups, without the need to use the Bonferroni's correction for multiple comparisons.
• Cohen however provided algorithms for calculating sample size, based on the F distribution, suitable for most analysis of variance, including the One Way Analysis of Variance used to compare multiple group means. StatTools provides this calculation, but echo Cohen's caution on how this is used.
  • The use of a different probability distribution, particularly when calculations involve iterative approximations, will produce different results because of rounding errors. An example is in estimating sample size for two groups, α=0.05, power=0.8, and effect size = difference / Standard Deviation = 0.5. Algorithms based on the z distribution described in Machin's book results in a sample size of 64 subjects per group, but 63 subjects per group when algorithms based on the F distribution described in Cohen's book are used.
  • The effect size used in calculations is based on a ratio of the difference between groups and the background Standard Deviation. When there are more than two groups, the differences between each pair of groups tend to be different, so the effect size needs to be adjusted. Cohen suggested that the largest distance should be used, but provides 3 models for adjustment
    • The first model (f1) is when there is minimal variability in the other differences, that, other than the maximum difference used for calculation, all other differences are smaller and similar to each other
    • The second model (f2) is when all the differences are different and their sizes evenly distributed
    • The third model (f3) is that despite variabilities, the other differences between group means are similar to those used for calculations
    • In the first two models (f1 and f2), the averaged difference between group means is smaller than the maximum difference used for calculation, so sample size required per group need to be adjusted upwards to detect the smaller difference as the number of groups increases
    • In the third model (f3), all the differences are roughly the same as the maximum used in calculation, so that the sample size required per group decreases as the number of groups increases. This model (f3) therefore requires the smallest sample size per group. Cohen suggested that this model be used as a default unless there are reasons to use the other models.

The numbers in these examples are genersated by computers to demonstrate the statistics, they are not real.

When measurements are not continuous and normally distributed, they are not parametric. Without the assumptions of normal distribution, the powerful methods using partition of variance cannot be applied.

In many cases, the data has a distribution that is mathematically related to normal distribution, such as a squared or exponential, and after some form of mathematical transformation, the parametric statistical tests can be applied.

In other cases, the nature of the data does not allow for such transformations, and the data has to be analysed as ordinal or ordered arrays. Examples of these can be as finely granular as a personality or depression score with a wide range, or more commonly a 10 point semantic differential scale, a 5 point Likert scale, or as coarse as a 3 point pain score (none (0), little (1), lots (2)).

StatTools offers one nonparametric test comparing two or more groups, two comparing two groups, and one for multiple (>2) groups.

Nonparametric comparisons of two or more groups of measurements : The Median Test

This test evaluates whether the number of cases < and >= the median level in all the groups are similar, so testing the null hypothesis that all groups have similar median values. The values of all groups are ranked collectively, and the median value obtained. The number of cases < and >= the median value in each group are then calculated, and compared.
• When there are only two groups, the Fisher's Exact Probability Test is used if the total sample size is less than 20, otherwise the Chi Square Test with Yates Correction is used.
• Where there are more than 2 groups, the standard Chi Square Test for goodness of fit is used

Nonparametric comparisons of two groups of measurements :

The Wilcoxon Mann-Whitney Test is a test of the null hypothesis for two sets of ordinal data, assuming that the distributions in the two groups are similar.

The Mann-Whitney U Test, described in the 1962 edition of Siegal's Nonparametric Stratistics for behavioral Sciences, has been renamed the The Robust Rank Ordered Test , as the term Mann-Whitney U Test now refers to another test, described in Wikipedia. This test is not provided in StatTools, and the term Mann-Whitney U Test is only used to maintain backwards compatibility of this web site. The correct name of the test should be The Robust Rank Ordered Test .

The Robust Rank Ordered Test is considered more robust than the Wilcoxon Mann Whitney Test, because if makes no assumptions that the two groups are from the dsame population, and it is nearly as powerful as the parametric t test. It tests the null hypothesis that the two medians values are not different.

Nonparametric comparisons of three or more groups of measurements : The Kruskall Wallis One Way Analysis of Variance

This test tests the null hypothesis that all the groups are from a homogeneous population.
• As well as providing a significance test (α) for the null hypothesis, the program also produces the mean rank values for each group, which can be used in post hoc analysis comparing individual pairs of groups.
• The Dunn's Test is one of the post hoc analysis between groups, and is carried out with the main analysis as it requires the original data for computation
• The more flexible and commonly used post hoc test is the Least Significant Difference between Mean Ranks. This allows the comparison of the mean ranks between any two groups, assuming that every group is to be compared with every other group. This test is more flexible as it requires only the total sample size, and the sample sizes and mean rank values of the two groups being compared.

Sample size determination is difficult for nonparametric data, as a precise estimate depends on knowing what the distribution patterns in the groups to be tested are.

Such an approach is barely possible using specialised programs when there are only two groups and when the range of measurements are not too great. In most cases, only an approximation of the sample size requirements can be estimated.

In Siegel's book (see references), the term power efficiency is used to represent the difference in sample size required. The Wilcoxon-Mann-Whitney Test has similar power as the t test, but only when the sample size is large (>30 per group). The Robust Rank Ordered Test and the Kruskall-Wallis test each has 95.5% the power efficiency of the F or t Test in parametric Analysis of Variance, so the sample size must be calculated with a higher power requirement. The algorithm for calculating an approximate sample size requirement for nonparametric tests is described in the Sample Size Introduction and Explanation Page so will not be repeated here.

The data for the following examples are not real, but made up to demonstrate the statistics.

Example 1 : Comparing two groups

We want to study the attitudes towards educational affirmative action from different ethic groups, using a 5 point Likert Scale.

The statement is "Having a quota of university places reserved for each ethnic group is a good thing". The responses are 1=Strongly disagree, 2=disagree, 3=neutral, 4=agree, and 5=Strongly agree.

Sample size

If we take the range (1-5) as mean ± 1.96SD, then SD = range/3.92 = 5 / 3.92 = 1.28.

We would like to have a sample size capable of detecting a difference of 1 between the two groups. The effect size = difference / SD = 1 / 1.28 = 0.78

We want a power equivalent to 0.8 in a parametric test. As the non-parametric test has a power efficiency of 95.5%, we define the power required as 1-0.2*0.955 = 0.81.

Using α = 0.05, power = 0.81, and effect size = 0.78 in the program from the Sample Size for Unpaired Differences Program Page ,the sample size requirement per group is 28 respondents per group for a two tail study.

The study

	Af	Ca
SD	1	3
D	3	3
N	7	12
A	10	6
SA	9	6

We received the responses from 30 African Americans, and 30 Caucasian American, and found the results as shown on the left. Af = African-Americans, and Ca = Caucasian-Americans

Median Test Chi Sq=2.4027 df=1 p=0.1211 Wilcoxon-Mann-Whitney Test z = 1.3524   p = 0.0583 Af : n = 30   W = 1017   Mean rank = 33.9 Ca : n = 30   W = 813   Mean rank = 27.1 Robust Rank Ordered Test U = 1.663   p=0.0482

The results of analysis (to the right) show no significant difference using the Median and Wilcoxon Mann Whitney Test, but a significant difference using the Robust Rank Ordered Test. The marginal conclusion is therefore that African and Caucasian Americans do not differ significantly in their attitudes toward affirmative Actions in education

Example 2 : Comparing 3 groups

What was not shown in example 1 is the data from the third group, the Asian Americans, which will be shown in this example.

	Af	Ca	As
SD	1	3	6
D	3	3	7
N	7	12	12
A	10	6	3
SA	9	6	2

We received the responses from 30 African-Americans, 30 Caucasian-American, and 30 Asian-Americans, and found the results as shown on the left. Af = African-Americans, Ca = Caucasian-Americans, As = Asian-Americans

Median Test : Chi Sq=27.4667 df=2 p=<0.0001 Kruskall Wallis One Way Analysis of Variance Grp n mean rank Af 30 56.9 Ca 30 47.1 As 30 32.5 Kruskall-Wallis H = 14.09  df = 2  p = 0.0009 Minimum significant diff in rank (Siegel and Castellan) grp grp Diff Dif(0.05) Dif(0.01) Dif(0.005) Af Ca 9.9 16.1 19.8 21.2 Af As 24.4 16.1 19.8 21.2 Ca As 14.5 16.1 19.8 21.2 Minimum Significant diff in rank (minimum Q by Dunn) grp grp Q Q(0.05) Q(0.01) Q(0.001) Af Ca 1.5 2.4 2.9 3.6 Af As 3.7 2.4 2.9 3.6 Ca As 2.2 2.4 2.9 3.6

The results of analysis are as shown to the right.

The 3 groups have significantly different attitudes towards affirmative actions in university education, both in the Median Test and the Kruskall Wallis One Way Analysis of Variance.

Post hoc analysis using least significant difference shows that the least significant difference in mean ranks is 16.1 at the p (α) = 0.05 level.

Caucasian-Americans (grp 2) has a mean rank of 47.1, and its difference to African-American (grp 1, mean rank = 56.9) is 9.9, less than the least significant difference at the p<0.05 level. Its difference to Asian Americans (Grp 3, mean rank = 32.5) is 14.5, also less than the least significant difference at the p<0.05 level. Caucasian-Americans are therefore not significantly different to the other two groups.

African-Americans (grp 1, mean rank = 56.9) however are different to Asian-Americans (Grp 3, mean rank = 32.5), the difference is 24.4 which is larger than the least significant difference, even at the p (α)<0.005 level.

The Dunn Test shows a similar pattern and supports similar conclusions.

The conclusions to be drawn are therefore African-Americans are significantly more positive towards affirmative actions in education than Asian-Americans, with the Caucasian-Americans in between, not significantly different to the other two groups.

The Permutation Tests are the most basic of statistical tests, from which other models have developed. StatTools presents two models, the significance test for paired differences presented in Paired Difference Programs Page , and the significance test comparing two groups presented in Unpaired Difference Programs Page .

The general principles are that, in a randomly allocated study, the data obtained could have been in either of two groups being compared. The test consists of calculating every possible permutation of the data, and examine the results. If the results from the original data is near the extremes (e.g. less than 5 percentile or more than 95 percentile in a one tail model), then a decision can be made that it is unlikely to be null and therefore statistically significant.

The advantages of using the Permutation tests are :

• Exhaustive permutation allows the calculation of the precise probability that the data presented is null, so the tests calculate the Type I Error (α), with a power (1-β) of 100%.
• The tests are not dependent on any assumption of data distribution, so they can be used in any regular interval data (where 10-9 is the same as 4-3). The tests can therefore be used on parametric measurements, ratios, variances, and time.
• Because of the above two characteristics, the tests can be used with a very small sample size

The disadvantages of using the tests are related to the computation intensity required, both in the large memory use, and the time required for computation. In the unpaired situation, where the sample size of group 1 is n1 and that of group 2 is n2, and the total nt=n1+n2, the total number of permutation is the Binomial coefficient nt and n1,n2 (Number of permutation = nt!/(n1!n2!). Computation time therefore increases exponentially with increasing sample size, and large dataset may either crash the program when available RAM is exhausted, or the computation becomes unacceptably too long.

The Permutation Test is therefore ideal for comparing two groups using small sets of interval data with uncertain distributions. With larger sample size, the more common non-parametric (the Median Test, the Wilcoxon Mann Whitney Test or the Robust Rank Ordered Test), parametric (Unpaired t test) tests should be preferred.

In theory, the Permutation Test can cope with any sample size. However, a probability of <0.05 is not possible with less than 3 subjects in each group, and computation will take an unacceptably long time if the total sample size (n1 + n2) exceed 26 subjects.

The mathematical argument of the Permutation Test is as follows

• In two groups of measurements, the null hypothesis is that there is no difference between the means. In other words, that the values observed can be in either of the group.

  The Permutation Test therefore consists of examining the difference between the sums of values in the two groups, preserving the original sample sizes of the groups but have the data distributed in all possible permutations. The total number of permutation is nt!/(n1!n2!).

  The difference between the sums in the original data is then compared with all possible sums, so that its probability can be estimated.

We use the default data for the program in the Unpaired Difference Programs Page as an example.

grp	val
1	50
1	57
1	70
1	60
1	55
2	58
2	65
2	70
2	70
2	72
2	70
2	72
2	60
2	77
2	75

We have two groups, group 1 with 5 cases, and group 2 with 10 cases, a total of 15 cases. The data is in two columns, as shown to the right. Col 1 is group designation, and col 2 the value.

Step 1. The difference between the two groups. The sum of measurements in group 1 is 292 and that in group 2 is 689. The difference is -397.

Step 2. The mathematics of permutation. With sample sizes of 5 and 10, the Binomial Coefficient = 15!/(5!10!) =3005. If we are to use a two tail model at α<0.05, then there are 0.025 on either side. At the two tail α=0.05 level therefore, one would expect that 0.025 x 3005 = 75 permutations from the most extreme values that can be considered as statistically significant.

Step 3. The differences of sums from two groups for all 3005 permutations are calculated and compared with -397 from the data. There are 18 permutations where the difference between the groups is less than -297, 2974 permutations where the difference is more than -297, and 11 where the difference is -297.

Step 4. Drawing conclusions.

• As 18 is less than the 75 which define the decision border for α=0.05, (2 tail model), we can conclude that the difference obtained from the original data is unlikely, and therefore statistically significant at the p<0.05, level (two tails).
• Looking it another way, there are 18 permutations with values less than the -297 from the data. The data is therefore the 19th value from the minimum, or at 19/3005 x 100 = 0.63^th percentile. This is less than the 2.5 percentile required for statistical significance at the p=0.05 (two tails), so it is statistically significant. Another way of putting it is that the observed difference is statistically significant at the level of p=0.0063 (one tail) or doubling to 0.0126 for two tail model.