Email this article 
Analyzing the Reliability of Thesis Assessment Instruments: Generalizability Theory
Abstract
This study aims to analyze the reliability of the thesis assessment instrument using Generalizability Theory (G-Theory). This research is a quantitative study that applies G-Theory. G-Theory is unique and useful in evaluating the sources of actual variance and error in any measurement design. Thus, G-Theory contributes to improving reliability and validity in assessment. In this study, data were collected using a thesis assessment instrument that had been tested on thesis assessments conducted at the Faculty of Teacher Training and Education, Universitas Jambi. G-Theory analysis in this study includes several main points, namely G-Study, Optimization (D-Study), and G-facets analysis. The results of the analysis show that the assessment instrument has a fairly high level of reliability with a G coefficient of 0.78. This indicates that the instrument can be used effectively in the thesis assessment. In addition, based on the results of the analysis carried out, recommendations were obtained to reassess item number two, increase the number of raters to six people to increase the reliability of the instrument to 0.80 to meet the satisfactory category. The resulting recommendations can be used by developers to improve the reliability of instruments in future assessment activities. This research is also expected to be a reference for future researchers in applying G-Theory in analyzing the reliability of measurement instruments.
Keywords
Full Text:
PDFReferences
Brennan, R. L. (1992). Generalizability Theory. Educational Measurement: Issues and Practice, 11(4), 27–34.
Brennan, R. L. (2010). Generalizability Theory and Classical Test Theory. Applied Measurement in Education, 1(1), 1–21. https://doi.org/10.1080/08957347.2011.532417
Briesch, A., Swaminathan, H., Welsh, M. E., & Chafouleas, S. M. (2014). Generalizability Theory: A Practical Guide to Study Design, Implementation, and Interpretation. Journal of School Psychology, 52(1), 13–35. https://doi.org/10.1016/j.jsp.2013.11.008
Cardinet, J., Johnson, S., & Pini, G. (2010). Applying Generalizability Theory Using EduG. In Routledge. Routledge of the Taylor& Francis Group. https://revistas.ufrj.br/index.php/rce/article/download/1659/1508%0Ahttp://hipatiapress.com/hpjournals/index.php/qre/article/view/1348%5Cnhttp://www.tandfonline.com/doi/abs/10.1080/09500799708666915%5Cnhttps://mckinseyonsociety.com/downloads/reports/Educa
Cardinet, J., Johnson, S., & Pini, G. (2011). Applying Generalizability Theory using EduG. In Routledge. Routledge of the Taylor& Francis Group. https://revistas.ufrj.br/index.php/rce/article/download/1659/1508%0Ahttp://hipatiapress.com/hpjournals/index.php/qre/article/view/1348%5Cnhttp://www.tandfonline.com/doi/abs/10.1080/09500799708666915%5Cnhttps://mckinseyonsociety.com/downloads/reports/Educa
Chiu, C. W.-T. (2001a). Scoring Performance Assessments Based on Judgements: Generalizability Theory. In Springer Science & Business Media (Vol. 50). Springer Science & Business Media.
Chiu, C. W.-T. (2001b). Scoring Performance Assessments Based on Judgements. In In Scoring Performance Assessments Based on Judgements. https://doi.org/10.1007/978-94-010-0650-7
Clauser, B. (2008). A Review of the EDUG Software for Generalizability Analysis. International Journal of Testing, 8(3), 296–301.
Crocker, L. M., & Algina, J. (1986). Introduction to Classical and Modern Test Theory. ERIC.
Falani, I., & Kumala, S. A. (2017). Kestabilan Estimasi Parameter Kemampuan pada Model Logistik Item Response Theory Ditinjau Dari Panjang Tes. SAP (Susunan Artikel Pendidikan), 2(2). https://doi.org/10.30998/sap.v2i2.2028
Fiangga, S., & Sari, Y. M. (2017). Analisis Generalisabilitas Multi Faset pada Instrumen Penalaran Matematika SMP. Jurnal Elemen, 3(2), 118–129.
Hapsan, A., & Rosnawati, R. (2023). Koefisien Realibilitas dengan Teori Generalizabilitas (G-Theory) two-Facet i: h: p design.
Hove, D. Ten, Jorgensen, T. D., & van der Ark, A. (2022). Interrater Reliability for Multilevel Data: A Generalizability Theory Approach. Psychological Methods, 4(4), 650.
Jackson, C. P., & Tinkler, P. (2001). Back to Basics: a Consideration of the Purposes of the PhD viva. Assessment & Evaluation in Higher Education, 26(4), 355–366.
Joughin, G. (1998). Dimensions of Oral Assessment. Assessment & Evaluation in Higher Education1, 23(4), 367–378.
Khosim, F. (2022). Pengujian Teori Generalizabiliti serta Pengaruh Motivasi, Pembelajaran Regulasi Kendiri dan Gaya Pembelajaran terhadap Pencapaian Akademik Murid Orang Asli di Perak. Universiti Utara Malaysia.
Lumaurridlo. (2019). Estimasi Keandalan Penilaian Munaqosah. Jurnal Tawadhu, 3(1), 665–673.
Nurmala, M. D., & Retnowati, T. H. (2013). Pengembangan Instrumen Penilaian Skripsi Mahasiswa. Jurnal Evaluasi Pendidikan, 1(1), 25–33.
Paul, V. K. (1994). Assessment of Clinical Competence of Undergraduate Medical Students. The Indian Journal of Pediatrics, 61, 145–151.
Ponder, N., Beatty, S., & Foxx, W. (2004). Doctoral Comprehensive Exams in Marketing: Current Practices and Emerging Perspectives. Journal of Marketing Education, 26(3), 226–235.
Ramadhan, W. F. R. (2019). Sistem Informasi Penilaian Tugas Akhir. Universitas Islam Indonesia.
Retnowati, T. H. (2009). Pengembangan Instrumen Penilaian Karya Seni Lukis Anak di Sekolah Dasar. Jurnal Penelitian Dan Evaluasi Pendidikan, 13(1).
Safitri, I., Rosnawati, R., Ansyari, R., & Abada, R. (2024). Estimasi Kesalahan Pengukuran dalam Penilaian Sidang Skripsi: Generalizability Theory Analysis. Afeksi: Jurnal Penelitian Dan Evaluasi Pendidikan, 5(1), 162–168.
Setyonugroho, W. (2017). Gentle Introduction of Generalizability Theory Analysis in OSCE Using EduG for Medical Educators. Advanced Science Letters, 23(12), 12656–12659.
Shavelson, R. J., & Webb, N. M. (2012). Generalizability theory. In In Handbook of Complementary Methods in Education Research (pp. 309–322). Routledge.
Society, S., & Group, E. W. (2010). Edug User Guide. Edumetrics.
Soesana, A., Subakti, H., Karwanto, Kuswandi, A. F. S., Sastri, L., Falani, I., Aswan, N., Hasibuan, F. A., & Lestari, H. (2023). Metodologi Penelitian Kuantitatif. Yayasan Kita Menulis.
Susilaningsih, E. (2014). Instrumen Penilaian Praktikum Kimia dan Estimasi Reliabilitasnya dengan Koefesien Generalisabilitas. Prosiding, Seminar Nasional Kimia Dan Pendidikan Kimia VI UNS.
Susongko, P. (2010). Studi Generalizabilitas Tes Tipe Dua Facet dengan Menggunakan Analisis Varian Tiga Jalur. OSEATEK, 07.
Suwita, F. S. (2020). Pengembangan Sistem Informasi Tugas Akhir dan Skripsi (SIMITA) di Universitas Komputer Indonesia (UNIKOM). Jurnal Teknologi Dan Informasi, 10(1), 71–82.
Swift, J., & Douglas, A. (1997). The viva voce: A Research Guide. Research Training Initiative, Birmingham Institute of Art & Design, University of Central England.
Wass, V., Vleuten, V. der, Shatzer, J., & Jones, R. (2001). Assessment of Clinical Competence. The Lancet, 357(9260), 945–949.
Webb, N. M., Shavelson, R. J., & Steedle, J. T. (2012). Generalizability Theory in Assessment Contexts. In In Handbook on measurement, assessment, and evaluation in higher education (pp. 152–169). Routledge.
Webb, N., Shavelson, R. J., & Haertel, E. H. (2006). 4 Reliability Coefficients and Generalizability Theory. Handbook of Statistics, 26, 81–124.
DOI: http://dx.doi.org/10.26737/jetl.v9i1.5715
Refbacks
- There are currently no refbacks.
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Published by:
Institute of Managing and Publishing of Scientific Journals STKIP Singkawang
Sekolah Tinggi Keguruan dan Ilmu Pendidikan (STKIP) Singkawang
Address : STKIP Singkawang, Jalan STKIP - Kelurahan Naram Singkawang, Kalimantan Barat, INDONESIA, 79251
No. Telp. : +62562 420 0344
No. Fax. : +62562 420 0584
JETL (Journal of Education, Teaching, and Learning)
e-ISSN : 2477-8478
p-ISSN : 2477-5924
Editor in Chief Contact: [email protected] / Wa: +6282142072788
Publisher Contact: [email protected] / Wa: +6282142072788
Management Tools
JETL Indexed by:
