The precision of sample size calculation depends very much on a known and accurate population or within group Standard Deviation. The difficulty is that this Standard Deviation is mostly unknown. Most users use published Standard Deviations, but these are obtained by sampling, so they contain errors.

One way to get around the difficulty is to realise that sample size calculation is not so much based on the within group Standard Deviation, but on the Effect Size, which is a ratio of the difference to be detected and the within group Standard Deviation.

Cohen, in his 1992 paper, suggested that for most research situations, an approximate estimation of the required Effect Size can be nominated, from which an approximate sample size can be determined. Using sample size per group in a two group analysis of variance as a demonstration model, the arguments are as follows.

• If the researcher is interested in only the obvious, an Effect Size of 0.7 or more can be considered large. This is a model suitable to detect a difference between means that is 70% or more of the background Standard Deviation. For a two group comparison to detect an Effect Size of 0.7 (α=0.05 and power=0.8), a sample size of 25 subjects per group is required.
• If the researcher is interested in a difficult to detect small difference, an Effect Size of 0.3 or less can be considered small. This is a model suitable to detect a difference between means that is 30% or less of the background Standard Deviation. For a two group comparison to detect an Effect Size of 0.3 (α=0.05 and power=0.8), a sample size of 173 subjects per group is required.
• If the researcher is interested in a difference that is not immediately obvious but nevertheless clinically meaningful, an Effect Size of 0.5 can be considered moderate. This is a model suitable to detect a difference between means that is half (50%) of a Standard Deviation. For a two group comparison to detect an Effect Size of 0.5, a sample size of 63 subjects per group can be used.

In most clinical research situations, depending on the power required, and an assessment of how obvious the Effect is likely to be, A sample size between 30 and 100 per group can be used for parametric comparisons. Although such an approach appears to be somewhat crude and arbitrary, in reality it is not much worse than guesstimating a background Standard Deviation that in most cases are not reflective of the population or the data collected.

A number pf procedures to estimate sample size requirements for nonparametric comparisons have been proposed, but most of them are suitable only for specific circumstances, and difficult to use generally. For example, sample size for nonparametric paired and unpaired scales are available, but they require users to define the number of ranks in the data, and model the proportion of the groups in each rank. These procedures are therefore not presented in StatTools.

Instead, an approximation of the sample size requirement can be computed, using an assumption of normal distribution. Users are cautioned however that this is only an approximation, and many statisticians do not approved of this approach, as by the very nature of nonparametric measurements, any assumption of normal distribution incorporates an unknown error.

For those who nevertheless need to calculate a sample size, the algorithm and its reasoning are as follows

• According to the text book by Siegel, the nonparametric statistical tests have power efficiencies varying between 92% to 99% that of their parametric equivalents. This discussion will use the parametric and nonparametric (Kruskall Wallis) Analysis of Variance as a demonstration, the nonparametric model having a power efficiency of 95.5% that of the parametric.
  • The power can be adjusted by adjusting β
    • β_parametric = 1-power_parametric
    • β_{nonparametric} = β_parametric * 0.955
    • power_{nonparametric} = 1 - β_{nonparametric} = 1 - (1-power_parametric) * 0.955
  • The power to be used can therefore be tabulated as follows

    	Power to be used with percent power efficiency
    Power required	90%	95%	99%
    0.700(70.0%)	0.730(73.0%)	0.715(71.5%)	0.703(70.3%)
    0.800(80.0%)	0.820(82.0%)	0.810(81.0%)	0.802(80.2%)
    0.900(90.0%)	0.910(91.0%)	0.905(90.5%)	0.901(90.1%)
    0.950(95.0%)	0.955(95.5%)	0.952(95.2%)	0.950(95.0%)
    0.990(99.0%)	0.991(99.1%)	0.990(99.0%)	0.990(99.0%)

• The range of observed measurements, can be interpreted as the 95% confidence interval
  • Half the range can be interpreted as 1.96(Standard Deviation)
  • Therefore Standard Deviation SD = range / (2 * 1.96) = range / 3.92
• The calculated power and Standard Deviation values can then be used in the calculations for the parametric Paired t Test
• An Example :
  • We wish to compare patient satisfaction under two regime of nursing care, using a scale of 0 for completely unsatisfactory to 10 completely satisfactory. We decided a difference of score value of 1, either way, as clinically meaningful, and set up our model to detect this difference.
  • As the range of values is 11, the approximate estimation of the standard deviation is 11 / 3.92 = 2.8
  • the approximate power to be used in the calculation is 1-(0.2 * 0.955) = 0.81
  • The difference to be detected in our model is 1, the effect size therefore is ef = 1 / 2.8 = 0.36
  • We can therefore use the Sample Size for One Way Analysis of Variance Program Page to calculate sample size required, using the parameters α=0.05, power=0.81, number of groups=2, Difference=0.36, Standard Deviation=1. The sample size required is 122 patient responses per group, compared with 121 for the parametric model, a difference of just under 1%.

Taking the approximation approach to its logical limits, we can model the One Way Analysis of Variance comparing two groups, using the common parameters of α=0.05 and power=0.8, a comparative power efficiency of 95.5%, the sample size required (per group are as in the following table)

ES (Diff / SD)	Parametric One Way Analysis of Variance	Nonparametric One Way Analysis of Variance	Difference
0.1	1533	1573	40(2.6%)
0.2	385	394	9(2.3%)
0.3	173	176	3(1.7%)
0.4	96	99	3(3.1%)
0.5	63	64	1(1.6%)
0.6	44	44	0(0%)
0.7	32	34	2(6.3%)
0.8	25	26	1(4%)
0.9	20	22	2(10%)

Sample size calculations are carried out at the planning stage of a research project, and is based on 3 parameters

• The probability of Type I Error (α). 0.05 is the most commonly used value
• The power of the model (1 - β). 0.8 is the most commonly used value
• The effect size the research model is expected to detect. Calculations therefore differ according to the definitions of the effect size, and the assumption of distribution of this effect size. Conceptually, the effect size is an expression of the ratio of the difference found and the background variation of the measurements. Using the simplest model of comparing two means, the effect size is the ratio of the difference and the background or within group Standard Deviation (es = (mean1 - mean2) / within group SD)

As explained in the Probability Introduction and Explanation Page , the use of statistical significance has been increasingly criticised, because the estimation of background Standard Deviation are so often not correct. Increasingly, the data obtained is examined, to evaluate its power. In this context power is the probability that to reject the null hypothesis is correct, or the ability to detect a difference if it really exists.

Research results with an insufficiently large effect size will show a probability of Type I Error that is too large to reject the null hypothesis (α>0.05). However the effect size is a ratio of the effect (e.g. difference between two groups) and the background variation (e.g. within group Standard Deviation).

If the effect (e.g. difference between two groups) is less than the critical value proposed during research planning, then the correct conclusion that it is too small to be statistically significant can be made.

However, if the effect is adequate, but the background variation (e.g. within group Standard Deviation) is larger than that envisaged during planning, then the research design, execution, or both are flawed, and no statistical conclusion can be drawn.

Power estimate allows the researcher to determine whether the power from the data collected is comparable to that during planning (e.g. power=0.8). If the power is inadequate, the researcher can make the decision whether to abandon the results, or to collect more data until the power reaches the level planned.

The confidence interval is another method of evaluating data collected without making any assumptions other than normal distribution. Usually the 95% confidence interval estimated from the effect and it's Standard Error, and the null hypothesis rejected if this interval does not overlaps the null value.

Pilot studies are conducted for two main reasons. The first is to test the feasibility and organisational structure of a research project, and the second to establish some ball park figures for population parameters and what the results are likely to be, which can be used in the planning of the main research project.

The reference section lists two excellent papers on pilot studies, and those interested should read these papers to get greater detail and insight into pilot studies than can be provided in this introduction.

The remainder of this section addresses sample size for pilot studies, adapted from the descriptions provided by Johanson and Brooks (2010).

In hypothesis testing, sample size is estimated according to the tolerance of Type I and Type II Errors, so that the results can be interpreted with confidence.

For a pilot study however, the aim is to obtain some ball park figures of what the main research project is likely to find, so the sample size required depends on a balance between the precision that can be achieved and the cost incurred in data collection.

In other words, there is a diminishing return, with smaller and smaller improvements in precision as sample size increases.

In planning a pilot study, the researcher has to make a value judgement as to when a very small further improvement in precision is no longer worth the much greater costs of a further increase in sample size.

This decision is based on when the sample size/confidence interval curve flattens out, and in most cases this occurs when the sample size is between 10 and 40 cases.

An Example

SSiz	CI	Diff	Dec/case	%Dec/case
3	1.1316
4	0.98	0.1516	0.1516	13
5	0.8765	0.1035	0.1035	11
6	0.8001	0.0764	0.0764	9
7	0.7408	0.0594	0.0594	7
8	0.6929	0.0478	0.0478	6
9	0.6533	0.0396	0.0396	6
10	0.6198	0.0335	0.0335	5
11	0.5909	0.0288	0.0288	5
12	0.5658	0.0252	0.0252	4
13	0.5436	0.0222	0.0222	4
14	0.5238	0.0198	0.0198	4
15	0.5061	0.0178	0.0178	3
16	0.49	0.0161	0.0161	3
17	0.4754	0.0146	0.0146	3
18	0.462	0.0134	0.0134	3
19	0.4496	0.0123	0.0123	3
20	0.4383	0.0114	0.0114	3
21	0.4277	0.0106	0.0106	2
22	0.4179	0.0098	0.0098	2
23	0.4087	0.0092	0.0092	2
24	0.4001	0.0086	0.0086	2
25	0.392	0.0081	0.0081	2
26	0.3844	0.0076	0.0076	2
27	0.3772	0.0072	0.0072	2
28	0.3704	0.0068	0.0068	2
29	0.364	0.0064	0.0064	2
30	0.3578	0.0061	0.0061	2
31	0.352	0.0058	0.0058	2
32	0.3465	0.0055	0.0055	2
33	0.3412	0.0053	0.0053	2
34	0.3361	0.0051	0.0051	1
35	0.3313	0.0048	0.0048	1

This example shows the use of sample size estimation in a pilot study, based on the confidence interval of a difference between two means, calculated using the algorithm in the Sample Size for Comparing Two Means Program Page . The example here is to examine the 95% confidence interval of the effect size in terms of difference between two means / Standard Deviation units (which is the same as difference between two means if the background SD is 1). The table shows the relationship between increasing sample size (col 1) and the efficiency (col 5). This relationship can also be shown as a plot between the first 2 columns of the table, as in the figure below.

It can be seen from the table that the 95% CI decreases from 0.88 SD units with a sample size of 5 to 0.80 with sample size of 6, a decrease of 0.08 SDs (9% of 0.88).

However, the 95% CI decreases from 0.39 SD units with a sample size of 25 to 0.38 with a sample size of 26, a decrease of 0.01 (2% of 0.39) which is very much smaller than that occurring at the sample size of 5-6 level.

In other words, there is a diminishing return, with smaller and smaller improvements in precision as sample size increases. The researcher can therefore reasonably make a decision in this case that a sample size no larger than between 5 and 10 per group would be appropriate for the pilot study, as further increases only improve the precision of the study trivially.