This module illustrates the issue of range-restricted reliability and the difficulties with maximum likelihood estimation, described in more detail in the context of inter-rater reliability in grant proposal review in Erosheva, Martinkova, & Lee (2021). We use their AIBS grant proposal peer-review dataset for presentation.

Below, you may select the ratio and type of range restriction given by the proportion of rated proposals. In contexts other than grant review, the subject/object of interest may be different: a student in educational assessment, a job application in hiring, a patient in a medical study, etc. Further, you may select the direction of restriction (top or bottom). The left plot illustrates the variability in ratings for the whole dataset outshading the data which would be lost due to range-restriction. The right plot provides the estimates of the calculated inter-rater reliability estimates, defined by intraclass corelation in the simplest model including the ratee effect only. The estimates are accompanied by a bootstrapped 95% confidence interval based on 25 bootstrap samples.

Interpretation

Selected R code

library(ggplot2)
library(purrr)
library(ShinyItemAnalysis)

# loading data
data(AIBS, package = "ShinyItemAnalysis")

# estimate reliability with ICC for complete AIBS dataset
ICCrestricted(Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj")

# estimate range-restricted ICC
ICCrestricted(Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj",
              sel = 0.90, dir = "top")

# caterpillar plot
AIBS %>%
  ggplot(aes(x = ScoreRankAdj, y = Score, group = ID)) +
  geom_line(col = "gray") +
  geom_point(shape = 1, size = 1.5) +
  stat_summary(fun = mean, fun.args = list(na.rm = TRUE), geom = "point",
               col = "red", shape = 5, size = 2.5, stroke = .35) +
  labs(x = "Ratee rank", y = "Rating (score)") +
  coord_cartesian(ylim = c(1, 5)) +
  theme_app()

# estimate all possible top-restricted subsets
all_top_restricted <- map_dfr(2:72,
                              ~ ICCrestricted(Data = AIBS, case = "ID", var = "Score",
                                              rank = "ScoreRankAdj", sel = .x, nsim = 10))
all_top_restricted

# or alternatively, in base R:
base_way <- lapply(2:72, function(x) {
  ICCrestricted(Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj",
                sel = x, nsim = 10)})
do.call(rbind.data.frame, base_way)

# plot
all_top_restricted %>%
  ggplot(aes(prop_sel, ICC1, ymin = ICC1_LCI, ymax = ICC1_UCI)) +
  geom_pointrange() + scale_x_continuous(labels = scales::percent) +
  labs(x = ("Proportion of top ratees"), y = "Reliability") +
  coord_cartesian(ylim = c(0, 1), xlim = c(0, 1)) +
  theme_app()