Skip to main content

Table 3 Intra-rater-, Inter-rater reliability and accuracy measured by two questionnaire rounds of 20 case vignettes each

From: Reliability and validity of multicentre surveillance of surgical site infections after colorectal surgery

  

Years of working experience in infectious disease surveillance

Intra-rater reliability (κ, 95%-CI)

Inter-rater reliability per hospital (κ, 95%-CI)#

Accuracy (%, First round/Second round)

Accuracy in determination of presence of anastomotic leakage, n = 4. (%, First round/Second round)

Accuracy in determination of absence of anastomotic leakage, n = 16 (%, First round/Second round)

Hospital A

Rater 1

4–5

0.78 (0.46–1.00)

0.61 (0.23–0.83)

95/85

75/100

93/87

Hospital A

Rater 2

2–3

0.95 (0.74–1.00)

 

85/80

100/75

93/93

Hospital B

Rater 1

11–15

0.83 (0.49–0.99)

0.72 (0.42–1.00)

80/85

75/100

93/93

Hospital B

Rater 2

6–10

0.73 (0.44–1.00)

 

95/90

100/100

93/93

Hospital C

Rater 1

11–15

1.00 (1.00–1.00)

0.94 (0.75–1.00)

90/90

75/75

93/93

Hospital C

Rater 2

11–15

0.94 (0.76–1.00)

 

90/95

75/75

93/93

Hospital D

Rater 1

0–1

0.75 (0.47–1.00)

0.69 (0.36–0.92)

90/85

100/100

93/87

Hospital D

Rater 2

4–5

0.89 (0.72–1.00)

 

90/95

100/100

93/87

Hospital E*

Rater 1

2–3

0.89 (0.59–1.00)

0.65 (0.38–0.92)

80/80

100/100

93/93

Hospital E*

Rater 2

4–5

0.73 (0.46–1.00)

 

85/70

100/100

93/81

Hospital F*

Rater 1

2–3

0.79 (0.57–1.00)

0.69 (0.34–0.92)

90/90

100/100

87/81

Hospital F*

Rater 2

11–15

0.89 (0.59–1.00)

 

90/90

100/100

87/87

Hospital G*

Rater 1

0–1

0.79 (0.55–1.00)

0.84 (0.61–1.00)

90/90

100/100

87/93

Hospital G*

Rater 2

 > 25

0.94 (0.75–1.00)

 

95/90

100/100

93/93

  1. κ, Cohen’s Kappa coefficient; 95% CI, 95% confidence interval, n, number
  2. *Hospitals that started surveillance for the purpose of this study
  3. #Inter-rater reliability was calculated from the second round questionnaire case vignettes