Reproducing analyses from the Cochrane DTA handbook in MetaBayesDTA
v2.0
Reproducing analyses from the Cochrane DTA handbook in MetaBayesDTA
v2.0
Tom Morris (tm428@leicester.ac.uk)
Tom Morris (tm428@leicester.ac.uk)
April 2024
April 2024
For comments or question on MetaBayesDTA please e-mail apps@crsu.org.uk. For comments or question on this document please e-mail tm428@leicester.ac.uk.
1 Introduction
This guide contains step-by-step instructions on how to reproduce the example meta-analyses in the Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy Version 2 (https://training.cochrane.org/handbook-diagnostic-test-accuracy/current), using the MetaBayesDTA app (https://crsu.shinyapps.io/MetaBayesDTA/). Six of the examples contained in chapters 9 and 10 of the handbook can be carried out in MetaBayesDTA. Before starting, the reader should familiarise themselves with the example(s) they wish to reproduce and download the accompanying datasets.
Sections 2 to 7 each correspond to one of the models in the handbook. Since there are many similarities between them, the guide does not duplicate instructions. For example, the required format of the dataset is the same in each case, and therefore instructions on formatting the datasets are only given for the first model, in section 2. For this reason the reader should not jump straight to the section corresponding to the model they wish to reproduce. They should instead read the sections in order, according to the prerequisite table below.
| Section | Model | Prerequisite sections |
|---|---|---|
| 2 | 9.4.2 Example 1 continued: anti‐CCP for the diagnosis of rheumatoid arthritis | None |
| 3 | Example 2, 9.4.4, Rheumatoid factor as a marker for rheumatoid arthritis | 2 |
| 4 | 9.4.6.3 Example 1 continued: Investigation of heterogeneity in diagnostic performance of anti‐CCP | 2 |
| 5 | 9.4.6.5 Example 2 continued: Investigating heterogeneity in diagnostic accuracy of rheumatoid factor (RF) | 2, 3 and 4 |
| 6 | 9.4.7.3 Example 3: CT versus MRI for the diagnosis of coronary artery disease | 2 and 4 |
| 7 | 10.8 Meta‐analysis with imperfect reference standard: latent class meta‐analysis | 2 |
2 Bivariate Model, 9.4.2 Example 1 continued: anti‐CCP for the diagnosis of rheumatoid arthritis
2.1 Format the dataset
The dataset that accompanies the Cochrane handbook is anti-cpp.csv. It must be reformatted before it will be accepted by MetaBayesDTA, by renaming the first six columns to author, year, TP, FN, FP, and TN. These column names must be exactly as they appear here, and are case-sensitive.
2.2 Load the data
2.3 Analyse the data
On the Priors tab there are two drop-down menus that contain options for sampling in Stan and prior distributions. They can both be left at their default values in this example. Navigate to the Run model tab.
Run the model.
A text box will pop up. Click OK to close it.
It will take some time for the model to run, and may appear that nothing is happening. Be patient! When the analysis has finished running, the SROC plot will appear on the right of the screen.
2.4 Compare the parameters
Navigate to the Parameter Estimates tab.
The table below is from chapter 9, page 16, of the handbook, and contains the parameter estimates and their standard errors from the frequentist Bivariate model.
Figure 2.1: Parameter estimates from the frequentist bivariate model
Next are the estimated means and standard deviations of the parameters, together with quantiles, from the Bayesian model, found in chapter 10, page 17 of the handbook.
Figure 2.2: Parameter estimates from the Bayesian bivariate model
The three sources use different notation for the parameters. The table below shows which parameters are which.
| Parameters | |||||
|---|---|---|---|---|---|
| Cochrane frequentist | \(\mu_A\) | \(\mu_B\) | \(\sigma_A^2\) | \(\sigma_B^2\) | \(\sigma_{AB} / \sigma_A \sigma_B\) |
| Cochrane Bayesian | mu[1] | mu[2] | tau.sq[1] | tau.sq[2] | rho |
| MetaBayesDTA | \(\mu_1\) | \(\mu_0\) | \(\sigma_1^2\) | \(\sigma_0^2\) | \(\rho\) |
MetaBayesDTA does not currently produce estimates of all of the parameters in the Cochrane handbook. However, all the output in the handbook can be estimated from MetaBayesDTA output. This is done for each parameter in the following subsections, with justification for the calculations provided in the appendix.
Each subsection has a summary table in which the parameter estimates and standard deviations are displayed. For the frequentist models these are parameter estimates and their standard errors. For the Bayesian models they are the median and standard deviation of the posterior distributions. In addition, there are 95% posterior intervals for the Bayesian models.
Since the estimation methods used for the Bayesian analysis are based on simulations, values can vary slightly with each analysis. The variability can be reduced by increasing the number of iterations.
2.4.1 Mean logit sensitivity (\(\mu_A\))
| Estimate | Standard deviation | Lower 95% limit | Upper 95% limit | |
|---|---|---|---|---|
| Cochrane frequentist | 0.6534 | 0.1275 | NA | NA |
| Cochrane Bayesian | 0.6536 | 0.1288 | 0.4037 | 0.910 |
| MetaBayesDTA | 0.6510 | 0.1420 | 0.3860 | 0.931 |
2.4.2 Mean logit specificity (\(\mu_B\))
| Estimate | Standard deviation | Lower 95% limit | Upper 95% limit | |
|---|---|---|---|---|
| Cochrane frequentist | 3.1090 | 0.1459 | NA | NA |
| Cochrane Bayesian | 3.1028 | 0.1454 | 2.8282 | 3.4031 |
| MetaBayesDTA | 3.0820 | 0.1600 | 2.7790 | 3.4120 |
2.4.3 Variance of random effects for logit sensitivity (\(\sigma_A^2\))
| Estimate | Standard deviation | Lower 95% limit | Upper 95% limit | |
|---|---|---|---|---|
| Cochrane frequentist | 0.5426 | 0.1463 | NA | NA |
| Cochrane Bayesian | 0.5318 | 0.1511 | 0.3276 | 0.9125 |
| MetaBayesDTA | 0.5806 | 0.1369 | 0.3422 | 0.994 |
In MetaBayesDTA, between study SD for logit(sensitivity) = 0.762. This is the square root of the variance, so the variance estimate is \(0.762^2 = 0.5806\). The posterior interval can be similarly transformed into an interval for the variance:
\[(0.585 ^2, 0.997 ^2) = (0.342, 0.994).\]
The standard deviation of \(\sigma_A^2\) can be estimated by
\[SD(\sigma_A^2) = \sqrt{2} \times 0.5806 / \sqrt{36} = 0.1369\]
where 36 is the number of trials minus 1.
2.4.4 Variance of random effects for logit specificity (\(\sigma_B^2\))
| Estimate | Standard deviation | Lower 95% limit | Upper 95% limit | |
|---|---|---|---|---|
| Cochrane frequentist | 0.5717 | 0.1873 | NA | NA |
| Cochrane Bayesian | 0.54 | 0.1867 | 0.2907 | 1.0138 |
| MetaBayesDTA | 0.6178 | 0.1456 | 0.3215 | 1.212 |
In MetaBayesDTA, between study SD for logit(specificity) = 0.786. This is the square root of the variance, so the variance estimate is \(0.786^2 = 0.6178\). The posterior interval can be similarly transformed into an interval for the variance:
\[(0.567 ^2, 1.101 ^2) = (0.321, 1.212).\]
The standard deviation of \(\sigma_B^2\) can be estimated by
\[SD(\sigma_B^2) = \sqrt{2} \times 0.6178 / \sqrt{36} = 0.1456\]
where 36 is the number of trials minus 1.
2.4.5 Correlation of logit sensitivity and logit specificity (\(\rho\))
| Estimate | Standard deviation | Lower 95% limit | Upper 95% limit | |
|---|---|---|---|---|
| Cochrane frequentist | -0.4855 | 0.1274 | NA | NA |
| Cochrane Bayesian | -0.4362 | 0.1567 | -0.6993 | -0.0911 |
| MetaBayesDTA | -0.408 | 0.16 | -0.6810 | -0.0590 |
The frequentist model provides an estimate of the covariance \(\sigma_{AB}\) rather than the correlation \(\rho\). The covariance estimate is \(-0.2704\). This can be converted into a correlation by dividing by the product of the standard deviations of logit sensitivity and logit specificity:
\[-0.2704 / (\sqrt{0.5426} \times \sqrt{0.5717}) = -0.4855.\]
Its standard error can be estimated as
\[(1 - (-0.4855)^2) / \sqrt{36} = 0.127\]
where 36 is the number of trials minus 1.
3 Rutter and Gatsonis HSROC model, 9.4.4 Example 2: Rheumatoid factor as a marker for rheumatoid arthritis
MetaBayesDTA does not directly fit the HSROC model. However, when there are no covariates the HSROC model is equivalent to the bivariate model, and the parameter estimates from each model can be derived from the other. MetaBayesDTA provides parameter estimates for the HSROC model in this case by deriving them from the parameters of the bivariate model.
Since there is no practical difference for the user in fitting the HSROC model compared to the bivariate model, the instructions from the previous example can be followed. The exception is that the SROC plot does not include a curve by default, so the following are instructions on how the curve can be added.
The dataset in this example is RF.csv.
3.1 Add the SROC curve
By default the SROC plot should look like this.
On the sROC Plot tab, select the settings drop-down menu.
Under Options, tick the sROC curve box.
The SROC plot should now look like this.
3.2 Compare the parameters
These are the parameter estimates from MetaBayesDTA.
The table below is from chapter 9, page 20, of the handbook, and contains the parameter estimates and their standard errors from the frequentist HSROC model. (The parameter \(\sigma_\theta\) is incorrectly written as \(\sigma_\beta\)).
Next are the estimated means and standard deviations of the parameters, together with quantiles, from the Bayesian model, found in chapter 10, page 24 of the handbook.
The three sources use different notation for the parameters. The table below shows which parameters are which.
| Parameters | |||||
|---|---|---|---|---|---|
| Cochrane frequentist | \(\Lambda\) | \(\Theta\) | \(\beta\) | \(\sigma_\alpha^2\) | \(\sigma_\theta^2\) |
| Cochrane Bayesian | LAMBDA | THETA | beta | tau.sq[1] | tau.sq[2] |
| MetaBayesDTA | \(\Lambda\) | \(\Theta\) | \(\beta\) | \(\sigma_\alpha^2\) | \(\sigma_\theta^2\) |
4 Meta-regression in the bivariate model, 9.4.6.3 Example 1 continued: Investigation of heterogeneity in diagnostic performance of anti‐CCP
This is an extension of the model from section 2, using the same data set, anti-cpp.csv. Only the steps that are different from those in that section are shown.
4.1 Format the dataset
The name of a covariate should end with .cat for categorical covariates and .cts for continuous covariates. The covariate in this example is Generation, which should be renamed to Generation.cat.
4.2 Analyse the data
There is one step different from section 2, to select Meta-regression instead of Meta-analysis, and one additional step, to select the covariate in the Model set up and priors tab, which is Generation.cat.
4.3 Compare the parameters
There are now three tables of parameter estimates: Shared parameters, Group-specific parameters, and Pairwise accuracy differences and ratios.
In the handbook, the parameter estimates from the frequentist model are on page 25 of chapter 9:
And from the Bayesian model on page 45 of chapter 10:
The three sources use different notation for the parameters. The table below shows which parameters are which.
| Parameters | |||||||
|---|---|---|---|---|---|---|---|
| Cochrane frequentist | \(\mu_A\) | \(\mu_B\) | \(\sigma_A^2\) | \(\sigma_B^2\) | \(\sigma_{AB} / \sigma_A \sigma_B\) | \(\nu_A\) | \(\nu_B\) |
| Cochrane Bayesian | mu[1] | mu[2] | tau.sq[1] | tau.sq[2] | rho | nu[1] | nu[2] |
| MetaBayesDTA | \((\mu_1L=CCP1)\) | \((\mu_0L=CCP1)\) | \(\sigma_1^2\) | \(\sigma_0^2\) | \(\rho\) | - | - |
4.3.1 Logit sensitivity parameters (\(\mu_A\) and \(\nu_A\))
| Parameter | Source | Estimate | Standard deviation | Lower 95% limit | Upper 95% limit |
|---|---|---|---|---|---|
| \(\mu_A\) | Cochrane frequentist | -0.0965 | 0.2203 | NA | NA |
| Cochrane Bayesian | -0.1001 | 0.2287 | -0.5547 | 0.3478 | |
| MetaBayesDTA | -0.0670 | 0.247 | -0.52 | 0.46 | |
| \(\nu_A\) | Cochrane frequentist | 0.9626 | 0.2513 | NA | NA |
| Cochrane Bayesian | 0.9685 | 0.2605 | 0.4634 | 1.4906 | |
| MetaBayesDTA | 0.9370 | 0.2544 | 0.4384 | 1.4356 |
MetaBayesDTA provides the mean logit sensitivity for the two subgroups, which are \((\mu_1L=CCP1) = \mu_A\) in the CCP1 group and \((\mu_1L=CCP2) = \mu_A + \nu_A\) in the CCP2 group. The statistics for \(\mu_A\) can be read from the output for \((\mu_1L=CCP1)\).
The parameter estimate for \(\nu_A\) is \(0.87 - -0.067 = 0.937\) where \(0.87\) is \((\mu_1L=CCP2)\) and \(-0.067\) is \((\mu_1L=CCP1)\).
The standard deviation of \(\nu_A\) is
\[SD(\nu_A) = 0.637 \sqrt{37} / \sqrt{8 \times 29} = 0.2544\]
where 0.637 is \(\sigma_A\) (\(\sigma_1\) in MetaBayesDTA), 8 and 29 are the number of trials in the CCP1 and CCP2 groups respectively, and 37 is the total number of trials.
The 95% posterior interval for \(\nu_A\) is then estimated by
\[0.937 \pm 1.96 \times 0.2544 = (0.4384, 1.4356)\]
5 Meta-regression in the HSROC model, 9.4.6.5 Example 2 continued: Investigating heterogeneity in diagnostic accuracy of rheumatoid factor (RF)
This is an extension of the model from section 3, using the same data set, RF.csv. As stated in that section, MetaBayesDTA obtains the parameters from the HSROC model by fitting the bivariate model and then converting the parameters. The procedure is valid because the two models are equivalent when there are no covariates. In this example, covariates are added to the cutpoint (\(\theta\)) and accuracy (\(\alpha\)) parameters, but not to the shape (\(\beta\)) parameter. This model is equivalent to the bivariate model with the addition of the same covariates to both the mean logit sensitivity and mean logit specificity. Therefore the procedure of fitting the bivariate model and deriving the parameter estimates for the HSROC model is also valid in this situation.
5.1 Format the dataset
Remember to rename the covariate Method to Method.cat.
Three of the trials are excluded from the example, so they must be deleted from the dataset. They are the ones that have Method.cat equal to “Not reported” or “RA hemagglutination”. Amongst the remaining forty seven, some have Method.cat equal to “Nephelometry” and some to “Nephelometry ”, with or without a space at the end. MetaBayesDTA will interpret these as different levels of the variable, so the spaces must be removed where they appear.
5.3 Add the SROC curve
In addition to ticking the sROC curve box, as in section 3, this time the extrapolate HSROC curve beyond observed Se/Sp data points should also be ticked.
The SROC plot should look like this:
6 Comparing index tests in the bivariate model, 9.4.7.3 Example 3: CT versus MRI for the diagnosis of coronary artery disease
In practical terms this example is no different from the previous two, so the procedure in section 4 can be followed. The dataset is schuetz.csv.
6.1 Format the dataset
In this example it is especially important to rename the index test variable from Test to Test.cat. If this is not done, the HSROC curve will not be produced.
7 Imperfect reference standard, 10.8 Latent class meta-analysis
7.1 Format the dataset
The data in this example are provided only in the R script rjags - LC meta-analysis model (11.8).R, shown below, where n11 is TP, n10 is FP, n01 is FN, and n00 is TN. They can be typed by hand into a CSV file, though great care must be taken to avoid a transcription error when using this method. There are various alternatives in which the data can be copied without resorting to painstaking typing, one of which is now described.
Open the script in RStudio and run these lines.
Run the command view(cell), then right click on the data and Select All. When the dataset is highlighted, copy it to the clipboard via Ctrl+C.
Open a blank spreadsheet and paste the dataset in. It should have maintained its structure, i.e. the numbers should each be in their own cell.
Manually edit the spreadsheet until it is in the format required by MetaBayesDTA. Dummy values can be entered for author and year. The reference standard must be specified in a column named reference.cat and must have the same value in each trial, which is “Culture” in the screenshot.
7.2 Analyse the data
Select Latent class meta-analysis under Imperfect gold standard, and then enter the name of the index test if desired.
In contrast to the other models, this one will not produce correct results with the settings in MetaBayesDTA left at their default values. This is due to the problems described in the handbook (section 10.8.2 Monitoring convergence). The handbook recommends choosing initial values closer to the solution and/or using more informative prior knowledge. A combination of these allows the results in the handbook to be reproduced. First, change the initial values of log sensitivity and log specificity to 1 and 3 respectively.
Then narrow the central 95% range of the prior distributions of sensitivity to the interval [0.5, 0.8] by selecting the drop-down menu
and editing the range for the reference test
and the index test.
When the model has run, the HSROC curve will be squashed because the specificity values are clustered close to 1. To improve the display, select set x and y axis limits to (0,1) from the drop-down menu.
7.3 Compare the parameters
The parameter table can be found in section 10.8, page 76 of the handbook chapter 10.
8 Appendix
Not all of the output given in the Cochrane handbook is provided by MetaBayesDTA. In most cases, however, the missing parameter estimates can be estimated from MetaBayesDTA output. The derivation of these estimates is described below.
8.1 Variances
This sections refers to \(\sigma_A^2\) and \(\sigma_B^2\) in the bivariate model, and \(\sigma_\alpha^2\) and \(\sigma_\theta^2\) in the HSROC model.
Assuming the parameter \(\sigma^2\) has distribution satisfying
\[\sigma^2 (n-1) / \sigma_m^2 \sim \chi^2_{n-1}\]
where \(E(\sigma^2) = \sigma_m^2\) and \(n\) is the number of trials, its standard deviation can be estimated by
\[SD(\sigma^2) = \sqrt{2} \sigma_m^2 / \sqrt{n-1}.\]
This follows immediately from the distributional assumption, because \(Var(\chi^2_{n-1}) = 2(n-1)\).
8.2 Correlations
This sections refers to \(\rho\) in the bivariate model.
The standard deviation of the correlation \(\rho\) between two parameters can be estimated as
\[(1 - r^2) / \sqrt{n-1}\]
where \(r\) is an estimate of their correlation and \(n\) is the number of trials. Note that a confidence interval for \(\rho\) is not required, only a point estimate \(r\).
8.3 Difference of two means
This sections refers to \(\nu_A\) and \(\nu_B\) in the bivariate model with a covariate.
In 9.4.6.3 example 1 the handbook gives parameter estimates for \(\mu\) and \(\nu\) (omitting the subscript \(A\) or \(B\)), whereas MetaBayesDTA gives estimates for the mean logit sensitivity (and specificity) within the two subgroups, namely \(\mu_1 = \mu\) for group 1 (CCP1) and \(\mu_2 = \mu + \nu\) for group 2 (CCP2). This subsection describes how to obtain a point estimate, standard deviation and 95% posterior interval for \(\nu\).
Let \(n_i\) be the number of trials in group \(i\) with total number of trials \(n = n_1 + n_2\).
A point estimate of \(\nu\) is \(\hat{\nu} = \hat{\mu}_2 - \hat{\mu}_1\), and a 95% posterior interval is
\[\hat{\nu} \pm 1.96 SD(\nu).\]
The standard deviation of \(\nu\) can be estimated by
\[SD(\nu) = \hat{\sigma} \sqrt{n} / \sqrt{n_1 n_2},\]
with the details as follows. Focusing on the marginal distribution of sensitivity or specificity, the model is described by \(Y_i \sim N(\mu + \nu Z_i, \sigma^2)\), where \(Z_i = 0\) in group 1 and \(Z_i = 1\) in group 2. Writing the model in its usual matrix form
\[E(Y) = X \beta \hspace{10pt} \text{and} \hspace{10pt} Var(Y) = \sigma^2 I_n,\]
where \(\beta = (\mu, \nu)^T\), the parameter estimates (posterior medians) are approximately \(\hat{\beta} = (X^T X)^{-1} X^T y\) with covariance matrix
\[Var(\beta) = \sigma^2 (X^T X)^{-1}.\]
The design matrix \(X\) is given by
\[X^T = \begin{pmatrix} 1 & ... & 1 & 1 & ... & 1 \\ 0 & ... & 0 & 1 & ... & 1 \end{pmatrix}\]
where there are \(n\) \(1\)s in the first row, and \(n_1\) \(0\)s and \(n_2\) \(1\)s in the second. Therefore
\[X^T X = \begin{pmatrix} n & n_2 \\ n_2 & n_2 \end{pmatrix}\]
and
\[(X^T X)^{-1} = \frac{1}{n_1 n_2} \begin{pmatrix} n_2 & -n_2 \\ -n_2 & n \end{pmatrix}.\]
Thus
\[Var(\beta) = \frac{\sigma^2}{n_1 n_2} \begin{pmatrix} n_2 & -n_2 \\ -n_2 & n \end{pmatrix}\]
and it follows that
\[SD(\nu) = \sigma \sqrt{n} / \sqrt{n_1 n_2}.\]