Comparing the methods of measuring multi-rater agreement on an ordinal rating scale: a simulation study with an application to real data
Küçük Resim Yok
Tarih
2013
Dergi Başlığı
Dergi ISSN
Cilt Başlığı
Yayıncı
Taylor and Francis Online
Erişim Hakkı
info:eu-repo/semantics/closedAccess
Özet
Agreement among raters is an important issue in medicine, as well as in education and psychology. The agreement among two raters on a nominal or ordinal rating scale has been investigated in many articles. The multi-rater case with normally distributed ratings has also been explored at length. However, there is a lack of research on multiple raters using an ordinal rating scale. In this simulation study, several methods were compared with analyze rater agreement. The special case that was focused on was the multi-rater case using a bounded ordinal rating scale. The proposed methods for agreement were compared within different settings. Three main ordinal data simulation settings were used (normal, skewed and shifted data). In addition, the proposed methods were applied to a real data set from dermatology. The simulation results showed that the Kendall’s W and mean gamma highly overestimated the agreement in data sets with shifts in data. ICC4 for bounded data should be avoided in agreement studies with rating scales <5, where this method highly overestimated the simulated agreement. The difference in bias for all methods under study, except the mean gamma and Kendall’s W, decreased as the rating scale increased. The bias of ICC3 was consistent and small for nearly all simulation settings except the low agreement setting in the shifted data set. Researchers should be careful in selecting agreement methods, especially if shifts in ratings between raters exist and may apply more than one method before any conclusions are made.
Açıklama
Anahtar Kelimeler
Agreement, Multi-rater, Bounded ordinal scale, Normal distribution, Skewed distribution
Kaynak
Journal of Applied Statistics
WoS Q Değeri
Scopus Q Değeri
Q2
Cilt
41
Sayı
5
Künye
Sertdemir, Y., Burgut, H. R., Alparslan, Z. N., Ünal, I. ve Günaştı, S. (2013). Comparing the methods of measuring multi-rater agreement on an ordinal rating scale: a simulation study with an application to real data. Journal of Applied Statistics. 41(5), s. 1506-1519.
