Assessment of Teachers’ Ability to Develop Hots-Based Test Items: A GRM Analysis
DOI:
https://doi.org/10.26811/peuradeun.v13i3.1555Keywords:
HOTS, Assessment Design, Teacher Competence, Graded Response Model, Islamic EducationAbstract
Developing Higher-Order Thinking Skills (HOTS)-based test items is a critical competency for teachers in 21st-century education. However, its implementation remains limited, particularly among Islamic education teachers. This study explores teachers’ ability to participate in the Professional Teacher Education Program under the Indonesian Ministry of Religious Affairs at the State Institute for Islamic Studies (IAIN) Sultan Amai Gorontalo to construct HOTS-based assessments. Utilizing an exploratory, cross-sectional design, the study involved 495 madrasah and Islamic education teachers across Indonesia. Data were collected through document analysis of teacher-developed test items and classroom observations, then analyzed using the Graded Response Model (GRM) in the R program. The findings reveal that most participants demonstrated moderate ability levels in developing HOTS-based test items, with ability scores ranging between –1.00 and 1.00. The study highlights significant challenges in item construction, particularly in aligning cognitive levels with stimulus and question indicators. These results emphasize the need for improved pedagogical training and targeted support to strengthen teachers’ competence in designing cognitively demanding assessments. The study contributes to the global discourse on HOTS-based assessment in religious education and offers practical implications for curriculum design, teacher training, and assessment policy in culturally rooted educational contexts.
References
Abd Ghani, A. R., Huda, M., Sholihah, M., Tabrani, Z. A., & Hussin, H. (2025). Tahfiz (Memorization Sciences) Curriculum Practice: An Empirical Study from Private School in Johor, Malaysia. Nazhruna: Jurnal Pendidikan Islam, 8(1), 233–245. https://doi.org/10.31538/nzh.v8i1.265
Abdullah, A. H., Soh, H. M., Mokhtar, M., Hamzah, M. H., Ashari, Z. M., Ali, D. F., Samah, N. A., Jumaat, N. F., Ibrahim, N. H., Surif, J., & Rahman, S. N. S. A. (2021). Does the Use of Smart Board Increase Students’ Higher Order Thinking Skills (HOTS)? IEEE Access, 9, 1833-1854. https://doi.org/10.1109/ACCESS.2020.3042832
Abella-Peña, L. E., & García-Martínez, Á. (2023). Conceptual and Methodological Bases of a Training Proposal Supported by ICT for Preservice Chemistry Teachers. Investigacoes Em Ensino de Ciencias, 28(1), 23-38. https://doi.org/10.22600/1518-8795.ienci2023v28n1p23
Ackerman, T., Ma, Y., Ma, M., Pacico, J. C., Wang, Y., Xu, G., Ye, T., Zhang, J., & Zheng, M. (2022). Item Response Theory. In International Encyclopedia of Education: Fourth Edition. https://doi.org/10.1016/B978-0-12-818630-5.10010-7
Ahmad, S., Prahmana, R. C. I., Kenedi, A. K., Helsa, Y., Arianil, Y., & Zainil, M. (2018). The instruments of Higher Order Thinking Skills. Journal of Physics: Conference Series, 943(1), 012053. https://doi.org/10.1088/1742-6596/943/1/012053
Albab, U., Budiyono, B., & Indriati, D. (2020). Metacognition Skills and Higher Order Thinking Skills (HOTS) in Mathematics. Journal of Physics: Conference Series, 1613(1), p.012017. https://doi.org/10.1088/1742-6596/1613/1/012017
Aleo, G., Pagnucci, N., Walsh, N., Watson, R., Lang, D., Kearns, T., White, M., & Fitzgerald, C. (2024). The Effectiveness of Continuing Professional Development for the Residential Long-Term Care Workforce: A Systematic Review. In Nurse Education Today, 137, 106161. https://doi.org/10.1016/j.nedt.2024.106161
An, X., & Yung, Y. F. (2014). Item Response Theory: What It Is and How You Can Use the IRT Procedure to Apply It. SAS Institute Inc, 10(4), 364-377. https://support.sas.com/resources/papers/proceedings14/SAS364-2014.pdf
Anita, A., Hasan, M., Warisno, A., Anshori, M. A., & Andari, A. A. (2023). Pesantren, Kepemimpinan Kiai, dan Ajaran Tarekat sebagai Potret Dinamika Lembaga Pendidikan Islam di Indonesia. Scaffolding: Jurnal Pendidikan Islam Dan Multikulturalisme, 4(3), 509-524. https://doi.org/10.37680/scaffolding.v4i3.1955
Basterfield, L., Machaira, T., Jones, D., Rapley, T., Araujo-Soares, V., Cameron, N., & Azevedo, L. B. (2023). Early Years Physical Activity and Motor Skills Intervention—A Feasibility Study to Evaluate an Existing Training Programme for Early Years Educators. Children, 10(1), 145-. https://doi.org/10.3390/children10010145
Botetano, C., & Abrahamson, D. (2022). The Botetano Arithmetic Method: Introduction and Early Evidence. In International Journal of Mathematical Education in Science and Technology, 53(2), 516-534. https://doi.org/10.1080/0020739X.2020.1867916
Brovina, N., & Arifi, D. (2023). Institutional And Non-Institutional Actors In Policy-Making Processes: A Case Study. Journal of Governance and Regulation, 12(2), 147-155. https://doi.org/10.22495/jgrv12i2art13
Destiniar, D., Mulbasari, A. S., Fuadiah, N. F., Octaria, D., Ningsih, Y. L., Retta, A. M., & Isroqmi, A. (2020). Pelatihan Penyusunan Soal HOTS untuk Mengembangkan Kemampuan Pedagogik Guru. J-ABDIPAMAS (Jurnal Pengabdian Kepada Masyarakat), 4(1), 163–169. https://doi.org/10.30734/j-abdipamas.v4i1.585
Fayyaz, S., Moeinaddini, M., Pourebrahim, S., Khoshnevisan, B., Kazemi, A., Toufighi, S. P., Schjønberg, M. S., & Birkved, M. (2024). Assessing Environmental Enhancement Scenarios in a Petrochemical Port: A Comprehensive Comparison using a Hybrid LCA-GRM Model. Journal of Cleaner Production, 445, 141079. https://doi.org/10.1016/j.jclepro.2024.141079
Gu, J., Tang, L., Liu, X., & Xu, J. (2022). Promoting Pre-service Teacher Students’ Learning Engagement: Design-Based Research in a Flipped Classroom. Frontiers in Psychology, 13, 810275. https://doi.org/10.3389/fpsyg.2022.810275
Hadjeris, F. (2021). Revisiting Sustainable Development Goal 4 in the Context of COVID-19 Pandemic: A Case Study of Online Teaching in Algerian Higher Education Institutions. Human Behavior and Emerging Technologies, 3(1), 160-168. https://doi.org/10.1002/hbe2.245
Hakim, B. R., Ardiyanto, A., & Wijayanti, A. (2021). Analisis Hots pada Instrumen Penilaian Siswa Kelas IV Sekolah Dasar. Wawasan Pendidikan, 1(2), 246–254. https://doi.org/10.26877/wp.v1i2.9028
Harjanti, F. D., Anam, F., & Kirana, A. (2022). Pengembangan Kompetensi Pedagogi Guru Pamong Sekolah Mitra Fakultas Bahasa dan Sains Universitas Wijaya Kusuma Surabaya melalui Pembelajaran HOTS. Jurnal Pengabdian Kepada Masyarakat, 2(1), 3741–3748. https://doi.org/10.53625/jabdi.v2i1.2271
Ishartono, N., Desstya, A., Prayitno, H. J., & Sidiq, Y. (2021). The Quality of HOTS-Based Science Questions Developed by Indonesian Elementary School Teachers. Journal of Education Technology, 5(2), 236-245. https://doi.org/10.23887/jet.v5i2.33813
Jerebine, A., Heering, T., & Barnett, L. M. (2024). Educator-Perceived Barriers and Facilitators to Structured-Physical Activity in Early Childhood Centres: A Systematic Review. Research Quarterly for Exercise and Sport, 95(1), 243-262. https://doi.org/10.1080/02701367.2023.2193243
Köhler, T., Rumyantseva, M., & Welch, C. (2023). Qualitative Restudies: Research Designs for Retheorizing. Organizational Research Methods, 28(1), 32-57. https://doi.org/10.1177/10944281231216323
Krathwohl, D. R., & Krathwohl, D. R. (2017). A Revision of Bloom’s Taxonomy: Routledge.
Lawrence, L. E. M., Echeverria, V., Yang, K., Aleven, V., & Rummel, N. (2024). How Teachers Conceptualise Shared Control with an AI Co-Orchestration Tool: A Multiyear Teacher-Centred Design Process. British Journal of Educational Technology, 55(3), 823-844. https://doi.org/10.1111/bjet.13372
Mahmudati, R. (2023). The Characteristics and Developing an Assessment Instrument of High Order Thinking Skill (HOTS) Items With 4-D Models. Jurnal Pendidikan Matematika Dan IPA, 14(1), 107-117. https://doi.org/10.26418/jpmipa.v14i1.55497
Merino-Soto, C. (2023). Aiken’s V Coefficient: Differences in Content Validity Judgments. MHSalud, 20(1), 23-32. https://doi.org/10.15359/mhs.20-1.3
Miharja, F. J., Fauzi, A., & Hudha, A. M. (2022). Pelatihan Pengembangan Soal Berbasis HOTS bagi Guru Sekolah Dasar. Sasambo: Jurnal Abdimas (Journal of Community Service). https://doi.org/10.36312/sasambo.v4i2.677
Muchson, M., Cobern, W. W., & Saefi, M. (2024). The Science Education Research trends (SERT) in Indonesian secondary schools: A Systematic Review and Bibliometrics Study. Cogent Education, 11(1), 2308407. https://doi.org/10.1080/2331186X.2024.2308407
Na, S. J., Ji, Y. G., & Lee, D. H. (2021). Application of Bloom’s Taxonomy to Formative Assessment in real-Time Online Classes in Korea. Korean Journal of Medical Education, 33(3), 191-201. https://doi.org/10.3946/KJME.2021.199
Niu, S. J., Niemi, H., Harju, V., & Pehkonen, L. (2021). Finnish Student Teachers’ Perceptions of Their Development of 21t-Century Competencies. Journal of Education for Teaching, 47(5), 638-653. https://doi.org/10.1080/02607476.2021.1951602
Nugraha, E. (2020). Implementasi Program Tahfizh Qur’an di PAUD Inklusif dengan Model HOTS. As-Sibyan: Jurnal Pendidikan Anak Usia Dini, 5(2), 95-106. https://ftk.uinbanten.ac.id/journals/index.php/assibyan/article/view/10551
Nurtawab, E. (2019). The Decline of Traditional Learning Methods in Changing Indonesia: Trends of Bandongan-Kitāb Readings Pesantrens. In Studia Islamika, 26(3), 511-541. https://doi.org/10.36712/sdi.v26i3.11026
Osborne, J. F., Henderson, J. B., MacPherson, A., Szu, E., Wild, A., & Yao, S. Y. (2016). The Development and Validation of a Learning Progression for Argumentation in Science. Journal of Research in Science Teaching, 53(6), 821–846. https://doi.org/10.1002/tea.21316
Paul, P. V., & Alqraini, F. (2019). Conclusion: Perspectives on Language, Literacy, and Deafness. Education Sciences, 9(4), 286. https://doi.org/10.3390/educsci9040286
Poerwanti, J. I. S., & Tribudhiarto. (2020). Pelatihan Merancang Instrumen Asesmen High Order Thinking Skills pada Guru-Guru SD di Kecamatan Laweyan Surakarta. Jurnal Widya Laksana, 9(1), 66–73. https://doi.org/10.23887/jwl.v9i1.16660
Ranta, M., Kruskopf, M., Kortesalmi, M., Kalmi, P., & Lonka, K. (2022). Entrepreneurship as a Neglected Pitfall in Future Finnish Teachers’ Readiness to Teach 21st Century Competencies and Financial Literacy: Expectancies, Values, and Capability. Education Sciences, 12(7), 463. https://doi.org/10.3390/educsci12070463
Rekan, A. A., Howell, H. R., Salleh, M. N. M., Rosin, N., Tabrani, Z. A., Ma`arif, M. A., & Adnan, M. A. M. (2025). Arabic Language Curriculum as a Foundation for Strengthening Religious Education in Public Higher Education. Jurnal Pendidikan Agama Islam, 22(1), 97–121. https://doi.org/10.14421/jpai.v22i1.11340
Sagala, P. N., & Andriani, A. (2019). Development of Higher-Order Thinking Skills (HOTS) Questions of Probability Theory Subject Based on Bloom’s Taxonomy. Journal of Physics: Conference Series, 1188(1), p.012025. https://doi.org/10.1088/1742-6596/1188/1/012025
Saido, G. M., Siraj, S., Nordin, B. A. B., & Al_Amedy, O. S. (2018). Higher Order Thinking Skills among Secondary School Students in Science Learning. IEEE Transactions on Antennas and Propagation, 66(9), 4936–4941. https://doi.org/10.1109/TAP.2018.2845550
Saraswati, N., Inderawati, R., & Vianty, M. (2023). Applying Higher Order Thinking Skills (Hots) Pedagogy: A Case Study of an EFL Core Teacher. English Review: Journal of English Education, 11(3), 767-776. https://doi.org/10.25134/erjee.v11i3.7818
Seman, S. C., Yusoff, W. M. W., & Embong, R. (2017). Teachers challenges in teaching and learning for Higher Order Thinking Skills (HOTS) in Primary School. International Journal of Asian Social Science, 7(7), 534–545. https://doi.org/10.18488/journal.1.2017.77.534.545
Setyarini, S., Salim, H., & Purnawarman, P. (2023). Higher-Order Thinking Skills (HOTS)-Based Literacy Media: An Innovative Learning Strategy to Promote the Secondary Students’ Social Awareness. Forum for Linguistic Studies, 5(2), 1706. https://doi.org/10.59400/FLS.V5I2.1706
Sidiq, Y., Ishartono, N., Desstya, A., Prayitno, H. J., Anif, S., & Hidayat, M. L. (2021). Improving Elementary School Students’ Critical Thinking Skill in Science Through Hots-Based Science Questions: A Quasi-Experimental Study. Jurnal Pendidikan IPA Indonesia, 10(3), 378-386. https://doi.org/10.15294/JPII.V10I3.30891
Solissa, J., Hasbullah, B., Rumahlewang, E., & Latar, I. M. (2023). Pendampingan dalam Mengenali dan Menyelesaikan Soal Uji Pengetahuan (UP) PPG Berbasis HOTS pada guru. German Für Gesellschaft (J-Gefüge), 2(2), 79–85. https://doi.org/10.30598/jgefuege.2.2.79-85
Suanto, E., Maat, S. M., & Zakaria, E. (2023). The Effectiveness of the Implementation of Three Dimensions Geometry KARA Module on Higher Order Thinking Skills(HOTS) and Motivation. International Journal of Instruction, 16(3), 95-116. https://doi.org/10.29333/iji.2023.1636a
Suardipa, I. P. (2018). Guru sebagai Agen Inovator Berbasis Higher Order Thinking Skills. Purwadita, 2(2), 73–83.
Suhady, W., Roza, Y., & Maimunah, M. (2020). Pengembangan Soal untuk Mengukur Higher Order Thinking Skill (HOTS) Siswa. Jurnal Gantang, 5(2), 143–150. https://doi.org/10.31629/jg.v5i2.2518
Suryani, H. (2024). Higher Order Thinking Skills in EFL Classrooms: Evidence in Five Secondary Schools. Kasetsart Journal of Social Sciences, 45(1), 103-112. https://doi.org/10.34044/j.kjss.2024.45.1.11
Syafitri, A., & Anggraini, D. P. (2023). Workshop Pembuatan Soal HOTS bagi Guru di SMA Negeri 1 Secanggang. Jurnal Pengabdian Kepada Masyarakat, 4(1), 34–39. https://jurnal.stkipalmaksum.ac.id/index.php/jpkm/article/view/377
Tanujaya, B., Mumu, J., & Margono, G. (2017). The Relationship between Higher Order Thinking Skills and Academic Performance of Student in Mathematics Instruction. International Education Studies, 10(11), 78–85. https://doi.org/10.5539/ies.v10n11p78
Tibrani, M., Madang, K., & Anggraini, N. (2023). Pengembangan Asesment Higher Order Thinking Skills (HOTS) Berbasis Computer Based Test (CBT) pada Mata Kuliah Prinsip Pengajaran dan Asesmen yang Efektif. Bioscientist : Jurnal Ilmiah Biologi, 11(2), 1509–1519. https://doi.org/10.33394/bioscientist.v11i2.9639
Tyas, M. A., Nurkamto, J., Marmanto, S., & Laksani, H. (2019). Developing Higher Order Thinking Skills (HOTS) – Based Questions: Indonesian EFL Teachers’ Challenges. In Proceedings of the International Conference on Future of Education, 2(1), 52-63. https://doi.org/10.17501/26307413.2019.2106
Ulvik, M., Eide, H. M. K., Eide, L., Helleve, I., Jensen, V. S., Ludvigsen, K., Roness, D., & Torjussen, L. P. S. (2022). Teacher Educators Reflecting on Case-Based Teaching–A Collective Self-Study. Professional Development in Education, 48(4), 657-671. https://doi.org/10.1080/19415257.2020.1712615
Winarti, W., Hairida, H., & Lestari, I. (2021). Deskripsi Kemampuan Guru Membuat Soal Berdasarkan pada Kurikulum 2013 di Sekolah Menengah Atas Kabupaten Landak. Jurnal Ilmiah Wahana Pendidikan, 7(2), 108–115. https://doi.org/10.5281/zenodo.4659018
Yani, A., & Mulyadi, A. (2022). Higher Order Thinking Skills (HOTS) - Oriented Learning in Geography Learning. Jurnal Pendidikan Ilmu Sosial, 31(1), 1-5-118. https://doi.org/10.17509/jpis.v31i1.47407
Yudha, R. P. (2023). Higher Order Thinking Skills (HOTS) Test Instrument: Validity and Reliability Analysis with the Rasch Model. EduMa: Mathematics education learning and teaching, 12(1), 21–38. https://doi.org/10.24235/eduma.v12i1.9468.g5505
Published
Issue
Section
License
Copyright (c) 2025 Lian Gafar Otaya, Kasim Yahiji, Rasimin Rasimin, Rahmawati Rahmawati, Nurul Imtihan, Norhisham Bin Muhamad (Author)
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with this journal agree to the following terms: (1) Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-ShareAlike 4.0 International License (CC-BY-SA) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal; (2) Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal; (3) Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website), as it can lead to productive exchanges, as well greater citation of published work (See The Effect of Open Access).
