Comparing the methods of measuring multi-rater agreement on an ordinal rating scale: a simulation study with an application to real data

Basit Öğe Kaydı
dc.authorid0000-0002-8335-1927en_US
dc.contributor.authorSertdemir, Yaşar
dc.contributor.authorBurgut, Hüseyin Refik
dc.contributor.authorAlparslan, Zeliha Nazan
dc.contributor.authorÜnal, İlker
dc.contributor.authorGünaştı, Suhan
dc.date.accessioned2024-07-12T21:04:38Z
dc.date.available2024-07-12T21:04:38Z
dc.date.issued2013en_US
dc.departmentFakülteler, Tıp Fakültesien_US
dc.description.abstractAgreement among raters is an important issue in medicine, as well as in education and psychology. The agreement among two raters on a nominal or ordinal rating scale has been investigated in many articles. The multi-rater case with normally distributed ratings has also been explored at length. However, there is a lack of research on multiple raters using an ordinal rating scale. In this simulation study, several methods were compared with analyze rater agreement. The special case that was focused on was the multi-rater case using a bounded ordinal rating scale. The proposed methods for agreement were compared within different settings. Three main ordinal data simulation settings were used (normal, skewed and shifted data). In addition, the proposed methods were applied to a real data set from dermatology. The simulation results showed that the Kendall’s W and mean gamma highly overestimated the agreement in data sets with shifts in data. ICC4 for bounded data should be avoided in agreement studies with rating scales <5, where this method highly overestimated the simulated agreement. The difference in bias for all methods under study, except the mean gamma and Kendall’s W, decreased as the rating scale increased. The bias of ICC3 was consistent and small for nearly all simulation settings except the low agreement setting in the shifted data set. Researchers should be careful in selecting agreement methods, especially if shifts in ratings between raters exist and may apply more than one method before any conclusions are made.en_US
dc.identifier.citationSertdemir, Y., Burgut, H. R., Alparslan, Z. N., Ünal, I. ve Günaştı, S. (2013). Comparing the methods of measuring multi-rater agreement on an ordinal rating scale: a simulation study with an application to real data. Journal of Applied Statistics. 41(5), s. 1506-1519.en_US
dc.identifier.endpage1519en_US
dc.identifier.issn1360-0532
dc.identifier.issue5en_US
dc.identifier.scopusqualityQ2en_US
dc.identifier.startpage1506en_US
dc.identifier.urihttps://www.tandfonline.com/doi/full/10.1080/02664763.2013.788617
dc.identifier.urihttps://hdl.handle.net/20.500.12415/3796
dc.identifier.volume41en_US
dc.institutionauthorBurgut, Hüseyin Refik
dc.language.isoenen_US
dc.publisherTaylor and Francis Onlineen_US
dc.relation.ispartofJournal of Applied Statisticsen_US
dc.relation.publicationcategoryUluslararası Hakemli Dergide Makale - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/closedAccessen_US
dc.snmzKY01794
dc.subjectAgreementen_US
dc.subjectMulti-rateren_US
dc.subjectBounded ordinal scaleen_US
dc.subjectNormal distributionen_US
dc.subjectSkewed distributionen_US
dc.titleComparing the methods of measuring multi-rater agreement on an ordinal rating scale: a simulation study with an application to real dataen_US
dc.typeArticle
dspace.entity.typePublication

Dosyalar

Koleksiyon

Tıp Fakültesi Koleksiyonu