This module illustrates the issue of range-restricted reliability and the difficulties with maximum likelihood estimation, described in more detail in the context of inter-rater reliability in grant proposal review in Erosheva, Martinkova, & Lee (2021). We use their AIBS grant proposal peer-review dataset for presentation.

Below, you may select the ratio and type of range restriction given by the proportion of rated proposals. In contexts other than grant review, the subject/object of interest may be different: a student in educational assessment, a job application in hiring, a patient in a medical study, etc. Further, you may select the direction of restriction (top or bottom). The left plot illustrates the variability in ratings for the whole dataset outshading the data which would be lost due to range-restriction. The right plot provides the estimates of the calculated inter-rater reliability estimates, defined by intraclass corelation in the simplest model including the ratee effect only. The estimates are accompanied by a bootstrapped 95% confidence interval based on 25 bootstrap samples.


Selected R code


# loading data
data(AIBS, package = "ShinyItemAnalysis")

# estimate reliability with ICC for complete AIBS dataset
ICCrestricted(Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj")

# estimate range-restricted ICC
ICCrestricted(Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj",
              sel = 0.90, dir = "top")

# caterpillar plot
AIBS %>%
  ggplot(aes(x = ScoreRankAdj, y = Score, group = ID)) +
  geom_line(col = "gray") +
  geom_point(shape = 1, size = 1.5) +
  stat_summary(fun = mean, fun.args = list(na.rm = TRUE), geom = "point",
               col = "red", shape = 5, size = 2.5, stroke = .35) +
  labs(x = "Ratee rank", y = "Rating (score)") +
  coord_cartesian(ylim = c(1, 5)) +

# estimate all possible top-restricted subsets
all_top_restricted <- map_dfr(2:72,
                              ~ ICCrestricted(Data = AIBS, case = "ID", var = "Score",
                                              rank = "ScoreRankAdj", sel = .x, nsim = 10))

# or alternatively, in base R:
base_way <- lapply(2:72, function(x) {
  ICCrestricted(Data = AIBS, case = "ID", var = "Score", rank = "ScoreRankAdj",
                sel = x, nsim = 10)}), base_way)

# plot
all_top_restricted %>%
  ggplot(aes(prop_sel, ICC1, ymin = ICC1_LCI, ymax = ICC1_UCI)) +
  geom_pointrange() + scale_x_continuous(labels = scales::percent) +
  labs(x = ("Proportion of top ratees"), y = "Reliability") +
  coord_cartesian(ylim = c(0, 1), xlim = c(0, 1)) +



ShinyItemAnalysis Modules are developed by the Computational Psychometrics Group supported by the Czech Science Foundation under Grant Number 21-03658S awarded to Patricia Martinkova. Elena Erosheva and Carole Lee's contributions to the development of this module and the data collection were further supported by the National Science Foundation under Grant Number 1759825.
Disclaimer: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.