Cohen's Kappa is a measurement of concordance or agreement between two raters or methods of measurement. The method can be applied to data that are not normally distributed, even binary (no/yes), but is best suited to a close ended ordinal scale, such as the 5 point Likert Scale.

The algorithm is well described by the original papers, text books, and on the Internet (see references). The book by Fleiss is particularly useful as it combines all the developments and enhancements, including the algorithm for estimating variance.

There are two ways of calculating Cohen's Kappa, and these produce different results. The first is by Cohen's original 1960 algorithm, now generally known as the unweighted Kappa. The second is by the weighted method, also described by Cohen but later in 1968, which includes a weighting for each cell, where weight for the cell i,j ( w_ij = 1 - |i-j|/(g-1) ), g being the number of categories of scores. Cohen argued that the weighted Kappa should be used particularly if the variables have more categories than binary (more than yes and no), because the distance from agreement should be taken into consideration. The results of both calculations are presented, and the recommendation is to use the weighted value.

Fless's Kappa

Fleiss's Kappa is an extension of Cohen's kappa to evaluate concordance or agreements between multiple raters, but no weighting is applied. Therefore, Fleiss's Kappa is similar to Cohen's unweighted Kappa (except for rounding errors) if the same data from two raters are submitted to the Fleiss algorithm.

Nomenclature

Ordinal data These are data sets where the numbers are in order, but the distances between numbers are unstated. In other words 3 is bigger than 2 and 2 is bigger than 1, but 3-2 is not necessarily the same as 2-1.

A common ordinal data is the Likert scale, where 1=strongly disagree, 2=disagree, 3=neutral, 4=agree, and 5=strongly agree. Although these numbers are in order, the difference between strongly agree and agree (5-4) is not necessarily the same as between disagree and strongly disagree (2-1).

Instrument is any method of measurement. For example, a ruler, a Likert Scale (5 point scale from strongly disagree to strongly agree), or a machine (e.g. ultrasound measurement of bone length).

raters are one or more of the instruments. This is usually a person, hopefully trained, that determines what score should be given to a subject. A carer evaluating a pain score is a rater, a judge in a beauty contest is a rater.

Subjects are the subjects of the measurements. They are patients, school children, members of the public, monkeys, rats, and so on.

Scores or measurements are the quantities produced by the instruments/raters. Measurements usually means something that are measured physically or chemically. Scores usually is a results produced by human decision.

Concordance This usually means how much scores or measurements produced by different instruments agree. Commonly, concordance is expressed as a number between 0 and 1, where 0 represents no agreements at all, and 1 represent complete replications. In some concordance measurements a negative value may be produced, which signifies opposite results.

Examples 1. Cohen's Kappa : Data entry option 1 The data consists of two obstetricians (raters), palpating the abdomen of 30 pregnant women (subjects), and rated each baby as growth retarded (0), normal (1) or macrosomic (2). The data is a table, where each row represent each pregnant abdomen palpated, the two columns representing the two obstetricians, and the value in each cell the classification given to that baby by that obstetrician.

The result consists firstly the display of the count matrix, with rows representing obstetrician 1's scoring, column obstetrician 2's scoring, and the cell the number of cases so scored by the two obstetrician. The diagonal cells representing where the two agree, the other cells where the two do not agree. Weighted Cohen's Kappa = 0.28, 95%CI = -0.01 to 0.57. As the 95% confidence interval overlaps the null value, the conclusion is that there is no agreement between the two obstetricians.

Example 2. Cohen's Kappa : Data Entry option 2. In this case, the data is a symmetrical matrix of counts. where two midwives reviewed 85 women at the beginning of labour as to how likely the delivery will require a Caesarean Section no risk at all (1), minimal risk (2), high risk (3) and almost certain (4). The data is a symmetrical table, where rows represents the evaluation of midwife 1, and columns the evaluation of midwife 2. The diagonals are where the two agreed (no risk (25), minimal risk (9) high risk (12) certain (21). The cells below the diagonals are the counts where midwife 1 evaluated a higher risk than midwife 2, and those above the diagonal the other way around.

Weighted Cohen's Kappa = 0.82 95% confidence interval = 0.74 to 0.90. As this interval does not overlap the null value (0), the conclusion that the risk assessment of these two midwives significantly agree can be made.

Example 3. Fleiss's Kappa : Data Entry option 1 and 2. The data table is similar to that for option 1 in Cohen's Kappa, except that more than two raters are involved. In this example, we have 5 midwives examining 10 pregnant abdomen, and classify each baby as growth retarded (0), normal (1) or macrosomic (2). The data is therefore a 5 column table, each row representing a baby being assessed each column one of the midwives, and the cell contains the scores.

The program first creates a count array, where the rows represent babies, each column represent the score (in this case 3 scores of 0, 1, and 2), and the cells the number of times that baby received that score. The sum of each row must therefore be 5 for the 5 rating midwives.

In data entry option 2, the counting table can be entered directly.

For both data entry options, Fleiss Kappa for this example is 0.42, 95% confidence interval = 0.28 to 0.56. As this interval does not cross the null (0) value, the conclusion that midwives agree significantly with each other can be made

The idea is that n subjects are ranked (0 to n-1) by each of the rankers, and the statistics evaluates how much the rankers agree with each other.

The program from the Kendall's W for Ranks Program Page modifies the input, so that the values entered by each ranker are ranked before calculation. This means the program can be used when the input data are scores, measurements, or ranks, and even if the scale of measurements used by different rankers are different (providing that they are ranking the same issues and in the same direction). For example, in cases of thyroid dysfunction, how much levels of T3, T4, and TSH agree with each other can be evaluated, as each measurement is converted to ranks before comparison.

Kendall's W is therefore useful in that it provides a calculation of concordance for many measurements without any assumption of distribution pattern.

Data entry and interpretation are best demonstrated using the example data from the Kendall's W for Ranks Program Page

Example

In a beauty contest with 10 finalists, 3 judges are to evaluate their relative beauty, with the least beautiful scoring 0 and the most beautiful scoring 9. The data is therefore a table, each row representing a finalist, each column one of the judges, and the cell contains the rank in beauty that judge gives to that contestant.

The results are Kendall W = 0.43, Chi Square = 11.65 degrees of freedom = 9 p = 0.23 Please note : Here the statistical significant test is for significant agreement and not significant difference. This means that this set of results indicates that the 3 judges have no significant agreement on their ranking of beauty.

Most users however requires methods that evaluates agreements in a more nuanced and comprehensive manner. StatTools presents 3 such methods, and they are are

• Intraclass Correlation Coefficient
• Bland and Altman Plot
• Lin's Concordance Coefficient

The rest of this panel will take users through the calculations presented in Agreement Program Page

Data Entry

The data is a matrix with two columns. Each row are measurements from a case, and the two columns, separated by white space, are the measurements from the two methods being evaluated (v1 and v2)

Initial Evaluation

The first step is to estimate the paired mean = (v1+v2)/2) and paired difference (v1-v2) of each pair. The original data and the estimates are presented as in the table to the right. The rest of the evaluations used the pair mean and pair difference

Statistical Evaluations

The first row is the mean and Standard Deviation of the paired difference (bias and distribution of bias), and the t test for significant departure from 0. The paired difference is named bias in the context of this evaluation. If significant bias exists (p_bias<0.05) then the two measurements produced different results which has to be corrected when used.

The second row is the results of linear regression analysis where paired difference = a + b(paired mean). The regression coefficient b representws changes to paired differences related to changing paired means, and is names proportional bias . If significant proportional bias exists (p_b<0.05), then the paired differences change significantly with the values of measurement, so the bias cannot be considered as stable.

The remainder of the table estimate the confidence intervals of bias. Please Note that these results differed from the common references of Bland & Altman, and Hanneman (see references) as follows

• Bland & Altman offered a single confidence interval of bias ± 2 Standard Deviations.
• Hanneman offered an interval of bias ± 1.96 Standard Devistions.
• This table offers 3 choices, 90%, 95%, and 99% confidence intervals of bias ±t(Standard Deviations), and t are calculated as two tail p=0.1 for 90% CI, 0.05 for 95% CI, and p=0.01 for 99% CI, based on the sample size of the data. In the example from Agreement Program Page , the 95% confidence interval for the bias uses t=2.05 because the sample size of the example is 30. Bland & Altman used t=2 regardless of sample size (t=2 if p=0.05 and sample size=60), and Hanneman used 1.96 regardless of sample size (t=2 if p=0.05 and sample size=population or is infinitely large)
• This table also counts the number of cases included within the 3 levels of confidence intervals

95% CI Precision Estimates by Bland & Altman Approximations

The results of analysis using the algorithm described in the paper by Bland & Altman are shown in the table to the right. These are presented as the Bland and Altman plot is now commonly used to evaluate agreements between measurements

Step 1, the 95% confidence, for all samples sizes, is calculated uses t=2, so that 95% CI = bias ±2SD. From the example used in Agreement Program Page

• The mean is -0.7667
• The Standard Deviation is 4.7828
• The borders are -0.7667±2(4.7828) The bias is therefore -0.7667, and the 95% confidence interval borders -10.3322 and 8.7988

Step 2 is to calculate t, based on p=0.05 for 95% confidence interval, and the sample size of the data. In the example from Agreement Program Page , the sample size is 30, so t=2.0452. This t value is then used to calculate the precision of the mean and the 95% confidence interval borders

Step 3 is to calculate the precision of the mean bias

• The Standard Error of the mean bias is sqrt(SD²/sample size) = sqrt(4.7828²/30) = 0.8732
• The 95% confidence interval for precision of bias is therefore -0.7667±2.0452(0.8732), -2.5526 to 1.0192

Step 4 is to calculate the precision of the confidence interval borders

• The Standard Error of both borders are the same, being sqrt(3SD²/sample size) = sqrt(3*4.7828²/30) = 1.5124
• The 95% confidence interval for precision of the borders are therefore
  • For the upper border, 8.7988±2.0452(1.5124) = 5.7056 to 11.8920
  • For the lower border, -10.3322±2.0452(1.5124) = -13.4254 to -7.2390

The Bland Altman Plot

After numerical analysis, Agreement Program Page produces the Bland Altman plot, to assist the user to interpret the agreement parameters. The plot follows that described in the reference, specifically

• The 3 horizontal lines are bias (mean paired difference), ± 2 x Standard Deviation of the paired difference
• The black round dots are paired mean and paird difference for each case

Bland and Altman (see references) suggested the duplicate or multiple measurements be taken from each case, and the variations between measurements compared with that within measurements. As these analysis represent an additional level of complexity and precision, they are not presented.

There are plenty of references easily available (see references), so the rationale and details of calculations will not be repeated here. The wikipedia provides an excellent introduction, and the algorithm and symbols used on the Agreement Program Page are based on Chapter 812 of PASS Sample Size Software, both readily accessible on the www.

Data Entry

The data is a matrix of 2 columns. Each row represents a case being measured. The two columns, separated by white space, are the paired measurements. If a gold standard is being compared, the gold standard is in the first (left) column.

The default example data from Agreement Program Page are generated from the computer and not real. There are only 30 pairs, too few for proper evaluation, but easier to demonstrate in this example. The data are shown in the table to the right. It represents 30 paired measurements of blood pressure. The first column (v1) is the gold standard, intra-arterial catheter, and the right (v2) the measurement being compared, external electronic cuffed manometer.

Calculations and Results : Please note that the program produces number with 4 decimal places, but 2 decimal places are presented in this discussion for easier reading

Step 1 evaluates the basic parameters, in this example sample size n=30 pairs, mean μ_v1=120 and μ_v2=120.77, Standard Deviations σ_v1=9.27 and σ_v2=9.34

Step 2 evaluates precisions and accuracies in detail

• Precision, the variability of the relationship, is represented by the Pearson's correlation coefficient ρ=0.87
• Scale Shift, the ratio between the two Standard Deviations, ω = σ_v2 / σ_v1 = 1.01
• Effect Size, ratio of the difference between the two means and their combined variations, υ=abs(μ_v2 - μ_v1) / sqrt(σ_v1σ_v2) = 0.08
• Accuracy, a combination of scale shift and effect size, χ_a=υ²+ω+1/ω=0.997

Step 3 calculates Lin's Coefficient of Concordance and its 95% confidence interval

• The Coefficient of Concordance is a combination of precision and accuracy, so that
  Lin's Coefficient of Concordance ρ_c = ρυ = 0.86
• The 95% confidence intervals are calculated according to the complex algorithm described in the Pass Manual, so that
  95% Confidence Interval (2 tail) = 0.74 to 0.93, and
  95% Confidence Interval (1 tail) = >0.76 or <0.93

Interpretation

According to McBride (see references)

The one tail lower border is usually used. In this example, the one tail lower border is 0.76, so the immediate interpretation is that the manometer measurement has a poor agreement with the gold standard intra-arterial measurement.

A more detailed examination shows that the accuracy is high (χ_a=0.997), but the precision is low (ρ=0.87). It is the low precision of the manometer measurements that contributed to the poor agreement with the gold standard.

ICC has advantages over correlation coefficient, in that it is adjusted for the effects of the scale of measurements, and that it will represent agreements from more than two raters or measuring methods.

The calculation involves an initial Two Way Analysis of Variance, so the program can also be used to conduct a parametric Two Way Analysis of Variance.

Data input and interpretation of results are best demonstrated using the default example data in the Intraclass Correlation Program Page

Example

We are testing different methods of measuring blood pressure, and wishes to know if the readings from the mercury and electronic manometer agree with each other. The data is therefore a two column table, each row represents a patient, and the two rows the two methods of measurement.

Please note This is a tiny made up data set to demonstrate the method. In reality there may be many more methods to compare ( more than 2 columns), and the data set should contain many more cases for the results to be stable.

The initial Two Way Analysis of Variance produces the table to the right. It shows that variations between patients (rows) are significantly greater than random measurement error (p=0.0006), but the different between method of measurement (columns) are not statistically significant (p=0.78). We can therefore draw the conclusions that, although blood pressure varies from patient to patient, there is no significant different between the methods of measurement in any patient. Please note significant test in this case is a test of significant difference and not agreement, and a no significant difference between columns indicate that there is no significant disagreement.

The program now proceed to provide a coefficient of agreement. It produces 6 in fact, described as follows.

There are three models for Intraclass Correlation

• Model 1 assumes that each method of measurement or rater is different, being subsets of a larger set of methods or raters, randomly chosen. This would be the case of having different research assistants measuring the height of children, with different sets of people doing the measurements at different sites, then combining the data together.
• Model 2 assumes the same methods or raters performs the evaluations in all cases, although these methods or raters may be a subset of a larger set of methods or raters. This is the model most frequently encountered in clinical research, when the same researchers carried out measurements on all the subjects.
• Model 3 makes no assumptions about the methods or raters.

In most cases, unless the methodology involves special arrangements, Model 2, individual, is usually used. From our example data therefore the Intraclass Correlation Coefficient is 0.98

Portney & Watkins suggested (see reference) that Intraclass Correlation Coefficient can be interpreted as follows: 0-0.2 indicates poor agreement: 0.3-0.4 indicates fair agreement; 0.5-0.6 indicates moderate agreement; 0.7-0.8 indicates strong agreement; and >0.8 indicates almost perfect agreement.