Cohen Kappa Using Data : Cohen Kappa Using Table of Counts : Fleiss Kappa Using Data : Cohen Kappa Using Table of Counts :

Cohen's Kappa for 2 raters Using Data : Data is a table of 2 columns

• Each row contains data from a case
• Col 1 = the ordinal score from rater or test 1
• Col 2 = the ordinal score from rater or test 2
• Each cell contains the ordinal score from that rater (column) for that case (row)

Cohen's Kappa for 2 Raters Using a Table of Counts is a symmetrical table

• There are same numbers of rows and columns, depending on the scale of the scores
• The rows are in rank order of scoring from rater 1 or test 1
• The columns are in order of scoring from rater 2 or test 2
• The number in each cell is the number of subjects receiving those scores from the raters

Fleiss's Kappa for 3 or More Raters or Tests Using Data as Collected : is a table of variable columns

• Each row contains data from a case
• The number of columns = number of raters or tests, each column represents a rater or test
• Each cell contains the ordinal score from that rater (column) for that case (row)

Fleiss's Kappa for 3 or More Raters or Tests Using a Table of Counts is a table of variable columns

• Each row represents a case
• The number of columns = the number of ccores, each column represents a score, in rank order
• The number in each cell is the number of raters of tests that provide that score (column) to that case (row)

Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 20:37-46, 1960.

Cohen J. Weighted kappa: nominal scale agreement with provision for scale and disagreement or partial credit. Psychol. Bull. 70:213-20. 1968.

Fleiss, Joseph L.; Cohen, Jacob; Everitt, B. S. (1969) Large sample standard errors of kappa and weighted kappa. Psychological Bulletin, Vol 72(5): p 323-327

Fleiss JL Statistical methods for rates and proportions second edition. Wiley Series in probability and mathematical statistics. Chapter 13 p. 212-236

Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33: 159-74.

wikipedia.org/wiki/Cohen's_kappa

University of York Department of Health Sciences Measurement in Health and Disease Cohen's Kappa a teaching paper with easy to understand and full formulation for Cohen's Kappa, weighted and unweighted, and Standard Error calculations.