Published January 1, 2022 | Version v1
Journal article Open

Using generalizability theory to investigate the variability and reliability of EFL composition scores by human raters and e-rater

  • 1. Karadeniz Tech Univ, Trabzon, Turkey
  • 2. Ordu Univ, Ordu, Turkey

Description

Using the generalizability theory (G-theory) as a theoretical framework, this study aimed at investigating the variability and reliability of holistic scores assigned by human raters and e-rater to the same EFL essays. Eighty argumentative essays written on two different topics by tertiary level Turkish EFL students were scored holistically by e-rater and eight human raters who received a detailed rater training. The results showed that e-rater and human raters assigned significantly different holistic scores to the same EFL essays. G-theory analyses revealed that human raters assigned considerably inconsistent scores to the same EFL essays although they were given a detailed rater training and more reliable ratings were attained when e-rater was integrated in the scoring procedure. Some implications are given for EFL writing assessment practices.

Files

bib-6fdfc051-f1eb-45b6-92a7-413a7a70e06e.txt

Files (192 Bytes)

Name Size Download all
md5:ae365263631d4a751bca1b8d15123baa
192 Bytes Preview Download