This page supports the program from the Curve Fitting for Receiver Operator Characteristics (ROCs) Program Page . The algorithm is derived from publications listed in the references panel

The Need for Curve Fitting the Receiver Operator Characteristic (ROC) Curve

The underlying assumptions of the ROC curve is that the Outcome is binary, but the Test is a continuous variable. In establishing the ROC from a set of reference data, it is assumed that the sample size provided is large enough, and the granularity of the values in the data set is fine enough, to produce a reproducible curve.

The reference data set unfortunately often failed these assumptions. A major problem is that test measurements are often rounded to clinically meaningful intervals, so that the granularity of the data is coarse, and changes occurred in a series of steps rather than in a smooth curve.

Furthermore, with rounding of values, often there are multiple cases with the same test value, and to include or exclude values at the cut off makes a large difference to the subsequent use. Also the changes to the False Positive Rate (FPR) is not always accompanied by any change to the True Positive Rate (TPR)

An example of this is shown in the plot to the left. If the cut off for decision making is 155 (the black line), data points can be clearly separated into above or below that value. However, if the cut off value is set at 154, there are 3 data points in the Outcome Positive group and 2 in the Outcome Negative group that have this exact value. Whether these are considered above the cut off, below the cut off, or excluded from consideration, will have a large impact on subsequent results produced.

The resulting ROC plot would be as the black line in the plot to the right.

Curve fitting would result in a ROC curve that has the closest fit with the data, yet allows a continuous interpretation along the whole range of the Test values, as shown in the red line in the plot to the right.

Test Value	FPR	TPR
162.5	0.96	1
162	0.92	1
161.5	0.84	1
160	0.84	0.96
159.5	0.76	0.96
159	0.68	0.96
158.5	0.48	0.92
157.5	0.4	0.88
157	0.36	0.88
156.5	0.36	0.8
156	0.24	0.68
155.5	0.2	0.6
154	0.12	0.48
153.5	0.04	0.4
153	0.04	0.36
152.5	0	0.28
151.5	0	0.24
151	0	0.2
150.5	0	0.16
150	0	0.12
149	0	0.08

This example uses the default input data from the program in the Curve Fitting for Receiver Operator Characteristics (ROCs) Program Page . This set of the data, in turn is obtained from producing a Receiver Operator Characteristics (ROC) curve from the first 2 columns of data in the Receiver Operator Characteristics (ROC) Program Page . The original data are computer generated and do not represent any real information.

We wished to establish maternal height (cms) as the Test to predict the need for Caesarean Section as the Outcome. Using the first 2 columns of default example data from Receiver Operator Characteristics (ROC) Program Page , We produced a table of False and True Positive Rates. A subset of columns from that table is used as the default example in the Curve Fitting for Receiver Operator Characteristics (ROCs) Program Page for curve fitting. The data is as shown in the table to the right. Test value is maternal height in cms, FPR and TPR are False and True Positive Rates for that value.

Curve Fitting

Two curve fitting procedures are carried out by the program, both use the False Positive Rate as the independent variable.

• Curve fitting for False Positive Rate.
  • The independent variable (x) is the maternal height
  • The y variable is the logit of the FPR, so that y = log(FPR(1-FPR))
  • The standard Regression analysis as in the Correlation and Regression Program Page is carried out, so that the formula y=a+bx is produced. In this example, a = -101.6295 and b = 0.6439
  • The result is therefore Logit(FPR) = -101.6295 + (0.6439)(maternal height in cm), or
    FPR = 1 / (1 - exp(-(-101.6295 + (0.6439)(maternal height in cm))))
• Curve fitting for True Positive Rate.
  • The logit of FPR and TPR are firstly calculated, so that u = log(FPR(1-FPR)) and v = Log(TPR(1-TPR)
  • The y (dependent) variable is then calculated as the Log Odds Ratio y = u-v
  • The x (independent variable is calculated as the Standard Error of the Log Odds Ratio x = u+v
  • The standard Regression analysis as in the Correlation and Regression Program Page is carried out, so that the formula y=a+bx is produced. In this example, a = 2.2319 and b = -0.0445
  • The formula to obtain TPR from FPR is TPR = 1 / (1 + exp(-a / ( 1 - b)) exp(log((1- FPR) / FPR) * ((1 + b) / (1 - b))))
  • The term (1 + exp(-a / ( 1 - b)) is a constant, and in this example = 0.118
  • The term ((1 + b) / (1 - b)) is also a constant, and in this example = 0.9148. Please note that b is changes in Standard error, and in this curve fitting it contains mostly rounding errors, and is not significantly different from 0. As a consequence, the second term ((1 + b) / (1 - b)) is not significantly different from 1, and can be left out without materially affecting the end results. However the program in the Curve Fitting for Receiver Operator Characteristics (ROCs) Program Page has included this term.
  • The result is therefore TPR = 1 / (1 + 0.118(exp(log((1-FPR)/FPR)(0.9148))))

Data used in curve fitting

The program provides 3 options for the user to choose, each has its advantages and disadvantages

• A: All data included : These include all data presented, including where the True Positive Rate (TPR) or the False Positive Rate (FPR) have values of 0 or 1.

  The advantage is that the fitted curve is a true reflection of all data. The disadvantages are that this includes numerous non-overlapping values where the False Positive Rate is 0 (FPR=0) or where the True Positive Rate is 1 (TPR=1). Although the overall fitting is more accurate, the part that is of interest to the researcher, where values from the two groups overlapped, is distorted by extreme non-overlapping values that provide no prediction insight.

• B: Only data with values >0 and <1 are included : This includes only those data points where values from the two groups overlapped.

  The advantage is that the resulting curve is more representative of those data that are of interest to the researcher. The disadvantage is that not all the data available have been included.

  As this option most closely reflect the researcher's needs, B: Only data with values >0 and <1 are included is the recommended option

• C: Only data with True Positive Rate >=0.5 and False Positive Rate <=0.5 are included : This option is provided so that the calculations are comparible with the original reference from Moses et.al.

  The idea is that all data points that would lead to erroneous prediction should not be excluded. This makes sense when curve fitting is used in meta-analysis, but is not recommended for general use, as many researchers requires the prediction properties of a test through out where the values from the two outcomes overlapped.

Results and their Interpretations

The program in the Curve Fitting for Receiver Operator Characteristics (ROCs) Program Page produces one plot and two tables. Using the default example input data, the plots produced from different options are shown to the left.

• The red curve uses algorithm A, inclusive of all data presented, providing 0.00001 to replace 0, and 0.99999 to replace 1 to avoid division by zero. This produces the most accurate fit, but the curve is dominated by the numerous data points that hugged the borders of the ROC space, so that the fit is poor in the range where the two outcomes overlapped.
• The green curve uses algorithm C, including only data where TPR>=0.5 and FPR<=0.5. This was the option suggested by Moses et.al. (see references) for meta-analysis of prediction studies. Although this produced a very good fit for data in the top left quadrant, it fitted poorly for the rest of the scale.
• The black curve uses algorithm B, including only data that have values >0 and <1. In other words, it uses those data points where the values from both outcome groups overlapped. This curve produced the best fit for that part of the data that are of interest to the researcher, and therefore recommended.

The plot to the right and the two tables on this page show the relationship between original data (black circular dots) and the curve fitted ROC curve using algorithm B, including data that have values >0 and <1. The curve is the fitted line, and the short straight line joins the coordinate of the original values to the values occupied on the fitted curve for each data point.

Data			Curve Fitted
Test Value	FPR	TPR	FPR	TPR
162.5	0.96	1	0.95	0.99
162	0.92	1	0.94	0.99
161.5	0.84	1	0.91	0.99
160	0.84	0.96	0.8	0.97
159.5	0.76	0.96	0.75	0.96
159	0.68	0.96	0.68	0.94
158.5	0.48	0.92	0.61	0.93
157.5	0.4	0.88	0.45	0.87
157	0.36	0.88	0.37	0.84
156.5	0.36	0.8	0.3	0.8
156	0.24	0.68	0.24	0.74
155.5	0.2	0.6	0.18	0.68
154	0.12	0.48	0.08	0.47
153.5	0.04	0.4	0.06	0.4
153	0.04	0.36	0.04	0.33
152.5	0	0.28	0.03	0.27
151.5	0	0.24	0.02	0.17
151	0	0.2	0.01	0.13
150.5	0	0.16	0.01	0.1
150	0	0.12	0.01	0.08
149	0	0.08	0	0.04

Please note that the program produces results to 4 decimal point precision, and these are rounded to 2 for most columns on this page.

Table of Transformed Data is the table on the left. The first 3 columns displays the original data as they were entered, and the last two column the curve fitted FPR and TPR. This table is for the user to review, to consider whether the algorithm chosen is appropriate, whether the results reflect the inputted data, and whether the parameters of the curve fitted ROC can be generalized for future clinical use.

It can be seen that, above a height of 160cms, all mothers delivered vaginally, and below 153cms all mothers required a Caesarean Section The data overlapped between these two values were used for curve fitting (in this example).

It can also be seen that, where the data overlapped, the curve fitted TPR and maternal height are closed enough to the original for practical use.

Test Value	FPR	TPR	TNR	YI	TPR/TNR	TNR/TPR	LR+	LR-
150.69	0.01	0.11	0.99	0.10	0.11	8.81	11.24	0.90
151.78	0.02	0.19	0.98	0.17	0.20	5.05	9.71	0.82
152.43	0.03	0.26	0.97	0.23	0.27	3.72	8.68	0.76
152.89	0.04	0.32	0.96	0.28	0.33	3.03	7.91	0.71
153.25	0.05	0.36	0.95	0.31	0.38	2.61	7.29	0.67
153.55	0.06	0.41	0.94	0.35	0.43	2.31	6.77	0.63
153.81	0.07	0.44	0.93	0.37	0.48	2.10	6.33	0.60
154.03	0.08	0.48	0.92	0.40	0.52	1.93	5.95	0.57
154.23	0.09	0.51	0.91	0.42	0.56	1.80	5.61	0.54
154.41	0.10	0.53	0.90	0.43	0.59	1.69	5.32	0.52
154.58	0.11	0.56	0.89	0.45	0.62	1.60	5.05	0.50
154.73	0.12	0.58	0.88	0.46	0.66	1.52	4.82	0.48
154.87	0.13	0.60	0.87	0.47	0.69	1.45	4.60	0.46
155.01	0.14	0.62	0.86	0.48	0.72	1.39	4.41	0.45
155.13	0.15	0.63	0.85	0.48	0.75	1.34	4.23	0.43
155.25	0.16	0.65	0.84	0.49	0.77	1.29	4.06	0.42
155.36	0.17	0.67	0.83	0.50	0.80	1.25	3.91	0.40
155.47	0.18	0.68	0.82	0.50	0.83	1.21	3.77	0.39
155.57	0.19	0.69	0.81	0.50	0.85	1.17	3.64	0.38
155.67	0.20	0.70	0.80	0.50	0.88	1.14	3.52	0.37
155.77	0.21	0.72	0.79	0.51	0.91	1.10	3.41	0.36
155.86	0.22	0.73	0.78	0.51	0.93	1.07	3.30	0.35
155.95	0.23	0.74	0.77	0.51	0.96	1.04	3.21	0.34
156.04	0.24	0.75	0.76	0.51	0.98	1.02	3.11	0.33
156.12	0.25	0.76	0.75	0.51	1.01	0.99	3.02	0.33
156.20	0.26	0.76	0.74	0.50	1.03	0.97	2.94	0.32
156.28	0.27	0.77	0.73	0.50	1.06	0.94	2.86	0.31
156.36	0.28	0.78	0.72	0.50	1.09	0.92	2.79	0.30
156.44	0.29	0.79	0.71	0.50	1.11	0.90	2.72	0.30
156.51	0.30	0.80	0.70	0.50	1.14	0.88	2.65	0.29
156.58	0.31	0.80	0.69	0.49	1.16	0.86	2.59	0.29
156.66	0.32	0.81	0.68	0.49	1.19	0.84	2.53	0.28
156.73	0.33	0.82	0.67	0.49	1.22	0.82	2.47	0.27
156.80	0.34	0.82	0.66	0.48	1.25	0.80	2.42	0.27
156.86	0.35	0.83	0.65	0.48	1.27	0.79	2.37	0.26
156.93	0.36	0.83	0.64	0.47	1.30	0.77	2.32	0.26
157.00	0.37	0.84	0.63	0.47	1.33	0.75	2.27	0.26
157.07	0.38	0.84	0.62	0.46	1.36	0.73	2.22	0.25
157.13	0.39	0.85	0.61	0.46	1.39	0.72	2.18	0.25
157.20	0.40	0.85	0.60	0.45	1.42	0.70	2.13	0.24
157.26	0.41	0.86	0.59	0.45	1.46	0.69	2.09	0.24
157.32	0.42	0.86	0.58	0.44	1.49	0.67	2.06	0.24
157.39	0.43	0.87	0.57	0.44	1.52	0.66	2.02	0.23
157.45	0.44	0.87	0.56	0.43	1.56	0.64	1.98	0.23
157.51	0.45	0.88	0.55	0.43	1.59	0.63	1.95	0.23
157.58	0.46	0.88	0.54	0.42	1.63	0.61	1.91	0.22
157.64	0.47	0.88	0.53	0.41	1.67	0.60	1.88	0.22
157.70	0.48	0.89	0.52	0.41	1.71	0.59	1.85	0.22
157.76	0.49	0.89	0.51	0.40	1.75	0.57	1.82	0.21
157.83	0.50	0.89	0.50	0.39	1.79	0.56	1.79	0.21
157.89	0.51	0.90	0.49	0.39	1.83	0.55	1.76	0.21
157.95	0.52	0.90	0.48	0.38	1.88	0.53	1.73	0.21
158.01	0.53	0.90	0.47	0.37	1.92	0.52	1.71	0.20
158.08	0.54	0.91	0.46	0.37	1.97	0.51	1.68	0.20
158.14	0.55	0.91	0.45	0.36	2.02	0.49	1.66	0.20
158.20	0.56	0.91	0.44	0.35	2.08	0.48	1.63	0.20
158.26	0.57	0.92	0.43	0.35	2.13	0.47	1.61	0.19
158.33	0.58	0.92	0.42	0.34	2.19	0.46	1.58	0.19
158.39	0.59	0.92	0.41	0.33	2.25	0.44	1.56	0.19
158.46	0.60	0.92	0.40	0.32	2.31	0.43	1.54	0.19
158.52	0.61	0.93	0.39	0.32	2.38	0.42	1.52	0.19
158.59	0.62	0.93	0.38	0.31	2.45	0.41	1.50	0.18
158.65	0.63	0.93	0.37	0.30	2.52	0.40	1.48	0.18
158.72	0.64	0.93	0.36	0.29	2.60	0.39	1.46	0.18
158.79	0.65	0.94	0.35	0.29	2.68	0.37	1.44	0.18
158.86	0.66	0.94	0.34	0.28	2.76	0.36	1.42	0.18
158.93	0.67	0.94	0.33	0.27	2.85	0.35	1.41	0.18
159.00	0.68	0.94	0.32	0.26	2.95	0.34	1.39	0.17
159.07	0.69	0.95	0.31	0.26	3.05	0.33	1.37	0.17
159.14	0.70	0.95	0.30	0.25	3.16	0.32	1.35	0.17
159.22	0.71	0.95	0.29	0.24	3.28	0.31	1.34	0.17
159.29	0.72	0.95	0.28	0.23	3.40	0.29	1.32	0.17
159.37	0.73	0.95	0.27	0.22	3.54	0.28	1.31	0.17
159.45	0.74	0.96	0.26	0.22	3.68	0.27	1.29	0.17
159.53	0.75	0.96	0.25	0.21	3.83	0.26	1.28	0.17
159.62	0.76	0.96	0.24	0.20	4.00	0.25	1.26	0.16
159.70	0.77	0.96	0.23	0.19	4.18	0.24	1.25	0.16
159.79	0.78	0.96	0.22	0.18	4.38	0.23	1.24	0.16
159.88	0.79	0.97	0.21	0.18	4.60	0.22	1.22	0.16
159.98	0.80	0.97	0.20	0.17	4.84	0.21	1.21	0.16
160.08	0.81	0.97	0.19	0.16	5.10	0.20	1.20	0.16
160.18	0.82	0.97	0.18	0.15	5.40	0.19	1.18	0.16
160.29	0.83	0.97	0.17	0.14	5.72	0.17	1.17	0.16
160.40	0.84	0.97	0.16	0.13	6.09	0.16	1.16	0.16
160.52	0.85	0.98	0.15	0.13	6.51	0.15	1.15	0.16
160.65	0.86	0.98	0.14	0.12	6.99	0.14	1.14	0.16
160.78	0.87	0.98	0.13	0.11	7.54	0.13	1.13	0.16
160.92	0.88	0.98	0.12	0.10	8.18	0.12	1.12	0.16
161.07	0.89	0.98	0.11	0.09	8.94	0.11	1.10	0.16
161.24	0.90	0.98	0.10	0.08	9.84	0.10	1.09	0.16
161.42	0.91	0.99	0.09	0.08	10.96	0.09	1.08	0.16
161.62	0.92	0.99	0.08	0.07	12.34	0.08	1.07	0.16
161.84	0.93	0.99	0.07	0.06	14.13	0.07	1.06	0.16
162.10	0.94	0.99	0.06	0.05	16.51	0.06	1.05	0.16
162.40	0.95	0.99	0.05	0.04	19.84	0.05	1.04	0.16
162.76	0.96	0.99	0.04	0.03	24.84	0.04	1.04	0.16
163.22	0.97	1.00	0.03	0.03	33.17	0.03	1.03	0.16
163.87	0.98	1.00	0.02	0.02	49.84	0.02	1.02	0.17
164.96	0.99	1.00	0.01	0.01	99.82	0.01	1.01	0.18

Table of Curve Fitted Values

The table to the right is intended to be downloaded and used as reference for clinical decision making, should the user decides to accept the curve fitted results.

• Column 1 is the reversed calculated Test value for the FPR, and in this example it is the maternal height (cms).
• Column 2 is the False Positive Rate (FPR), from 0.01 to 0.99 at 0.01 intervals. This is the independent variable (x) used to calculate the values in the remaining columns
• Column 3 is the curve fitted True Positive Rate, and Columns 2 and 3 define the curve fitted ROC.
• Column 4 is the True Negative Rate TNR = 1-FPR. This is used in some of the subsequent calculations
• Column 5 is the Youden Index YI = TPR+TNR-1, which represents the overall accuracy of the values. The Test value associated with the maximum YI is sometimes used as the cut off value for clinical decision making, and in this example, the closest is a maternal height of 156.04cms, with FPR=0.24, TPR=0.75, TNR=0.76, and Youden Index = 0.51
• Column 6 is the ratio of TPR/TNR. This is commonly used for two purposes
  • An alternative to the Youden Index for selecting the best cut off value is Q*, suggested by Moses et.al (see references). This is a point where TPR=TNR, and represents the intersection between the ROC and a line joining the top left and bottom right corner of the plot. In this example, the closest is a maternal height of 156.12cms, with FPR=0.25, FNR=0.76, TNR=0.75, Youden Index=0.51, and TPR/TNR=1.01
  • If the test is being used for screening purposes, to create an early warning system, then a High TPR is preferred over a high TNR. A TPR/TNR ratio of between 2 and 3 is therefore used for this purposes. In our example, the closest is a maternal height of 158.65cms, where TPR/TNR=2.52. This can be set as an alert trigger, possibly making a rule that all mothers admitted to the labour ward shorter than 158.65cms should be reviewed by an experienced obstetrician in early labour.
• Column 7 is a ratio of TNR/TPR, the inverse of column 5. This is commonly used for intervention decisions, especially if the intervention is costly or risky. A high TNR is preferred over high TPR, and a TNR/TPR ratio of 2 to 3 can be used for this purpose. In our example, the closet is a maternal height of 153.55cms, where TNR/TPR=2.31 . This can be used for clinical decision, such as a rule that all mothers admitted to the labour ward shorter than 153.55cms be considered for elective Caesarean Section in early labour.
• Column 8 and 9 are Likelihood Ratios for Test Positive and Test negative. Discussions on Likelihood Ratio are provided in the Prediction Statistics Explanation Page and will not be repeated here.

Hanley JA, McNeil BJ (1982) The meaning and use of the Area Under a Receiver Operating Characteristic (ROC) curve. Radiology 143:29-36

Simel D.L., Samsa G.P., Matchar D.B. (1991) Likelihood ratios with confidence: sample size estimation for diagnostic test studies. J. Clin. Epidemiology vol 44 No. 8 pp 763-770

Moses LE, Shapiro D, and Littenberg B(1993) Combining independent studies of a diagnostic test into a Summary ROC Curve: Data analysis approaches and some additional considerations. Statistics in Medicine 12:p1293-1316.