Grading of Recommendations Assessment, Development, and Evaluation and Confidence in Network Meta-Analysis showed moderate to substantial concordance in the evaluation of certainty of the evidence

Objectives: Grading of Recommendations Assessment, Development, and Evaluation (GRADE) and Confidence in Network Meta-Analysis (CINeMA) are available to assess the confidence in network meta-analysis (NMA) results. They share common aspects, but their operationalization differs. We evaluated interrater reliability (IRR) among assessors, the approaches' concordance, and application time. Methods: Two dichotomous ("seizure response", "ischemic stroke") and two continuous ("change in attention deficit hyperactivity disorder (ADHD)", "weight loss") outcomes with networks of different sizes, structures and complexities were chosen from four NMAs. Thirteen assessors were randomly assigned to four groups to apply both GRADE and CINeMA on one continuous and one dichotomous outcome. We measured IRR and concordance using Gwet's AC on the overall evaluation and each tool's domain. We calculated time spent evaluating each network and tool, including time to consult the guidance papers. Results: IRR ranged from 0.49 ("seizure response") to 0.70 ("ischemic stroke") with GRADE, from 0.02 ("ischemic stroke") to 0.73 ("change in ADHD symptoms") with CINeMA. Overall concordance was 1, 0.90, 0.68, and 0.42 for "seizure response", "ADHD symptoms", "weight loss", and "ischemic stroke", respectively. The median time spent to assess each network ranged from 160 to 481 minutes with GRADE, from 150 to 330 minutes with CINeMA.