Using Rasch Model to Analyze Reliability and Validity of Concept Mastery Test on Electricity and Magnetism Topic

Yusri Handayani; Rahmawati Rahmawati; Widiasih Widiasih

doi:10.26737/jipf.v8i2.3877

Using Rasch Model to Analyze Reliability and Validity of Concept Mastery Test on Electricity and Magnetism Topic

Yusri Handayani, Rahmawati Rahmawati, Widiasih Widiasih

Abstract

This study aims to analyze the reliability and validity of the electricity and magnetismconcept mastery test (EM-CMT) using Rasch model aided by Winsteps version 3.68.2. The multiple choice concept mastery test consists of 40 items about material electric current, resistance, resistivity, Ohm's law, electric voltage, energy and electrical conductivity, resistors in series, parallel, and mixed circuits, Kirchhoff's law, RC charging and discharging capacitor circuits, magnetic fields, magnetic forces, and sources magnetic field. The sample of this study were prospective physics teacher students who had passed basic physics courses. The results showed that the EM-CMThad a value of the Alpha Cronbach reliability with a good and acceptable category (0.70). The value of person reliabilityis in the quite good category (0.69). Meanwhile, the personnel separation coefficient of 1.50 which indicates the response of respondents is quite good and consistent. Analysis of the observational aspects of item fit shows that there are no items that need to be changed or removed. For the aspect of person fit, the results of the analysis showed that there were seventeen out of sixty-three respondents who experienced unusual response patterns. A review of the observations of the map variable shows the distribution of respondents' abilities and the items are proportional. Analysis of unidimensionalaspects shows the value of "raw variance explained by measure" is in the acceptable category. Based on the results of the analysis on a number of aspects, it can be concluded that theEM-CMT test is reliable and valid so that it can be used to measure the students’ concepts.

Keywords

Concept Mastery Test; Rasch Model; Validity; Reliability; Winsteps 3.68.2

Full Text:

PDF (English)

References

Shipstone, D. M. (1984). A study of children's understanding of electricity in simple DC circuits. European journal of science education, 6(2): 185-198.

Shipstone, D. (1988). Pupils' understanding of simple electrical circuits. Some implications for instruction. Physics education, 23(2): 92.

Arnold, M., & Millar, R. (1987). Being constructive: An alternative approach to the teaching of introductory ideas in electricity. International Journal of Science Education, 9(5): 553-563.

Osborne, R. J., & Cosgrove, M. M. (1983). Children's conceptions of the changes of state of water. Journal of research in Science Teaching, 20(9): 825-838.

Osborne, R. (1983). Towards modifying children's ideas about electric current. Research in Science & Technological Education, 1(1): 73-82.

Hekkenberg, A., Lemmer, M., & Dekkers, P. (2015). An analysis of teachers' concept confusion concerning electric and magnetic fields. African Journal of Research in Mathematics, Science and Technology Education, 19(1): 34-44.

Tarciso Borges, A., & Gilbert, J. K. (1999). Mental models of electricity. International journal of science education, 21(1): 95-117.

Cosgrove, M. (1995). A study of science‐in‐the‐making as students generate an analogy for electricity. International journal of science education, 17(3): 295-310.

Cohen, R., Eylon, B., & Ganiel, U. (1983). Potential difference and current in simple electric circuits: A study of students’ concepts. American journal of Physics, 51(5): 407-412.

Paatz, R., Ryder, J., Schwedes, H., & Scott, P. (2004). A case study analysing the process of analogy‐based learning in a teaching unit about simple electric circuits. International Journal of Science Education, 26(9): 1065-1081.

Psillos, D., Koumaras, P., & Valassiades, O. (1987). Pupils’ representations of electric current before, during and after instruction on DC circuits. Research in Science & Technological Education, 5(2): 185-199.

Engelhardt, P. V., & Beichner, R. J. (2004). Students’ understanding of direct current resistive electrical circuits. American journal of physics, 72(1): 98-115.

Finkelstein, N. (2005). Learning physics in context: A study of student learning about electricity and magnetism. International Journal of Science Education, 27(10): 1187-1209.

Zacharia, Z. C., & De Jong, T. (2014). The effects on students’ conceptual understanding of electric circuits of introducing virtual manipulatives within a physical manipulatives-oriented curriculum. Cognition and instruction, 32(2): 101-158.

Stocklmayer, S. M., & Treagust, D. F. (1996). Images of electricity: How do novices and experts model electric current?. International Journal of Science Education, 18(2): 163-178.

Heller, P. M., & Finley, F. N. (1992). Variable uses of alternative conceptions: A case study in current electricity. Journal of Research in Science Teaching, 29(3): 259-275.

Heywood, D., & Parker, J. (1997). Confronting the analogy: primary teachers exploring the usefulness of analogies in the teaching and learning of electricity. International Journal of Science Education, 19(8): 869-885.

Dega, B. G., Kriek, J., & Mogese, T. F. (2013). Students' conceptual change in electricity and magnetism using simulations: A comparison of cognitive perturbation and cognitive conflict. Journal of Research in Science Teaching, 50(6): 677-698.

Tiruneh, D. T., De Cock, M., Weldeslassie, A. G., Elen, J., & Janssen, R. (2017). Measuring critical thinking in physics: Development and validation of a critical thinking test in electricity and magnetism. International Journal of Science and Mathematics Education, 15(4): 663-682.

Groundlund, R. L., & Linn, N.E. (1990). Measurement and Evaluation in Teaching. Prentice Hall College Div.

Thorndike, R. L. (1971). Educational Measurement, Second Ed. American Council on Education.

Aydın, U., & Ubuz, B. (2015). The thinking-about-derivative test for undergraduate students: development and validation. International Journal of Science and Mathematics Education, 13(6): 1279-1303.

Haladyna, T. M. (2004). Developing and Validating Multiple-Choice Test Items. Routledge.

Haladyna, T. M., Downing, S. M., & Rodriguez, M. C. (2002). A review of multiple-choice item-writing guidelines for classroom assessment. Applied measurement in education, 15(3): 309-333.

Haladyna, T. M., & Rodriguez, M. C. (2013). Developing and Validating Test. Routledge Taylor & Francis Group.

Groundlund, N. E. (1982). Constructing Achievement Test, Third Ed. Prentice-Hall, Inc.

Groundlund, N. E. (2003). Assessment of Student Achievement, Seventh Ed. Pearson Education, Inc.

Gall, M. D., Borg, W. R., & Gall, J. P. (1996). Educational research: An introduction. Longman Publishing.

Russell, M. K., Airasian, P. W. (1967). Classroom Assessment: Concepts and Applications, 7th ed. McGraw-Hill Companies.

Mehrens, W. A. & Lehmann, I. J. (1991). Measurement and Evaluation in Education and Psychology, Fourth Ed. Wadsworth/Thomson Learning.

Skorupiński, P. M. (2015). American Educational Research Association, American Psychological Association, National Council on Measurement in Education, Standards for educational and psychological testing. Kwartalnik Pedagogiczny, 238(4): 201-203.

Susongko, P. (2016). Validation of science achievement test with the rasch model. Jurnal Pendidikan IPA Indonesia, 5(2): 268-277.

Retnawati, H. (2016). Validitas, Reliabilitas, & Karakteristik Butir (Panduan untuk Peneliti, Mahasiswa, dan Psikometrian). Parama Publishing.

Hambleton, R. K. & Swaminatan, H. (1985). Item Response Theory: Principles and Applications. Springer Science+Business Media, LLC.

Hambleton, R. K., Swaminatan, H., & Rogers, H. J. (1991). Fundamentals of Item Respons Theory. Sage Publication, Inc..

Demars, C. (2010). Item Respons Theory: Understanding Statistics Measurement. Oxford University Press.

Bond, T. G., & Fox, C. M. (2001). Applying the Rasch Model: Fundamental Measurement in the Human Sciences. Lawrence Erlbaum Associates.

Bond, T. G., & Fox, C. M. (2015). Applying the Rasch Model: Fundamental Measurement in the Human Science, Third Edit. Routledge Taylor & Francis Group.

Bond, T. G., & Fox, C. M. (2007). Applying the Rasch Model: Fundamental Measurement in the Human Sciences, Second Edi. Lawrence Erlbaum Associates.

Boone, W. J. (2016). Rasch analysis for instrument development: Why, when, and how?. CBE—Life Sciences Education, 15(4): 1-7.

Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch Analysis in the Human Sciences. Springer.

Abdaziz, A., Jusoh, M. S., Amlus, H., & Salleh, T. S. (2014). Construct validity: A Rasch measurement model approaches. Journal of Applied Science and Agriculture, 9(12): 7-12.

Cheng, M. M., & Oon, P. T. (2016). Understanding metallic bonding: Structure, process and interaction by Rasch analysis. International Journal of Science Education, 38(12): 1923-1944.

Gänswein, W. (2011). Effectiveness of Information Use for Strategic Decision Making. Gabler.

Luo, T., Wang, J., Liu, X., & Zhou, J. (2019). Development and application of a scale to measure students’ STEM continuing motivation. International Journal of Science Education, 41(14): 1885-1904.

Tabatabaee-Yazdi, M., Motallebzadeh, K., Ashraf, H., & Baghaei, P. (2018). Development and Validation of a Teacher Success Questionnaire Using the Rasch Model. International Journal of Instruction, 11(2): 129-144.

Yang, W., & Liu, E. (2016). Development and validation of an instrument for evaluating inquiry-based tasks in science textbooks. International Journal of Science Education, 38(18): 2688-2711.

Commons, M. L., Goodheart, E. A., Pekker, A., Dawson, T. L., Draney, K., & Adams, K. M. (2008). Using Rasch scaled stage scores to validate orders of hierarchical complexity of balance beam task sequences. Journal of Applied Measurement, 9(2): 182-199.

Fulmer, G. W. (2015). Validating proposed learning progressions on force and motion using the force concept inventory: Findings from Singapore secondary schools. International Journal of Science and Mathematics Education, 13(6): 1235-1254.

bin Khairani, A. Z., & bin Abd Razak, N. (2012). Advance in educational measurement: A Rasch model analysis of Mathematics proficiency test. International Journal of Social Science and Humanity, 2(3): 248.

Kuo, C. Y., Wu, H. K., Jen, T. H., & Hsu, Y. S. (2015). Development and validation of a multimedia-based assessment of scientific inquiry abilities. International Journal of Science Education, 37(14): 2326-2357.

Van Zile-Tamsen, C. (2017). Using Rasch analysis to inform rating scale development. Research in Higher Education, 58(8): 922-933.

Briggs, A. L., Jackson, T. R., Bruce, S., & Shapiro, N. L. (2005). The development and performance validation of a tool to assess patient anticoagulation knowledge. Research in social and administrative Pharmacy, 1(1): 40-59.

Nielsen, J. B., Kyvsgaard, J. N., Sildorf, S. M., Kreiner, S., & Svensson, J. (2017). Item analysis using Rasch models confirms that the Danish versions of the DISABKIDS® chronic-generic and diabetes-specific modules are valid and reliable. Health and Quality of Life Outcomes, 15(1): 1-10.

Tran, V. D., Dorofeeva, V. V., & Loskutova, E. E. (2018). Development and validation of a scale to measure the quality of patient medication counseling using rasch model. Pharmacy Practice (Granada), 16(4).

Runnels, J. (2012). Using the Rasch model to validate a multiple choice English achievement test. International Journal of Language Studies, 6(4): 141-155.

Saidi, S. S., & Siew, N. M. (2019). Reliability and Validity Analysis of Statistical Reasoning Test Survey Instrument Using the Rasch Measurement Model. International Electronic Journal of Mathematics Education, 14(3): 535-546.

Linacre, J. M. (2002). Optimizing rating scale category effectiveness. Journal of applied measurement, 3(1): 85-106.

Sumintono, B., & Widhiarso, W. (2015). Aplikasi pemodelan rasch pada assessment pendidikan. Trim komunikata.

Sumintono, B., & Widhiarso, W. (2014). Aplikasi model Rasch untuk penelitian ilmu-ilmu sosial (edisi revisi). Trim Komunikata Publishing House.

Boone, W. J., & Scantlebury, K. (2006). The role of Rasch analysis when conducting science education research utilizing multiple‐choice tests. Science Education, 90(2): 253-269.

Davidowitz, B., & Potgieter, M. (2016). Use of the Rasch measurement model to explore the relationship between content knowledge and topic-specific pedagogical content knowledge for organic chemistry. International Journal of Science Education, 38(9): 1483-1503.

Nevin, E., Behan, A., Duffy, G., Farrell, S., Harding, R., Howard, R., ... & Bowe, B. (2015, July). Assessing the validity and reliability of dichotomous test results using Item Response Theory on a group of first year engineering students. In The 6th Research in Engineering Education Symposium (REES 2015), Dublin, Ireland, July (pp. 13-15).

Anderson, L. W., & Krathwohl, D. R. (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom’s Taxonomy of Educational Objectives. David McKay Company, Inc.

Giancoli, D. C. (2016). Physics Principles With Applications, Global Ed. Pearson.

Serway, R. A., & Jewett, J. W. (2014). Physics for Scientists and Engineers with Modern Physics, Ninth Ed. Brooks/Cole Cengage Learning.

Walker, J., Halliday, D., & Resnick, R. (2014). Fundamental Physics, Tenth Ed. John Wiley & Sons, Inc.

Rahmawati, R., Rustaman, N. Y., Hamidah, I., & Rusdiana, D. (2018). The development and validation of conceptual knowledge test to evaluate conceptual knowledge of physics prospective teachers on electricity and magnetism topic. Jurnal Pendidikan IPA Indonesia, 7(4): 283-490.

Sumintono, B., & Widhiarso, W. (2015). Aplikasi Pemodelan Rasch Pada Asesmen Pendidikan, I. Penerbit Trim Komunikasi.

Krishnan, S., & Idris, N. (2014). Investigating reliability and validity for the construct of inferential statistics. International Journal of Learning, Teaching and Educational Research, 4(1): 51-60.

DOI: http://dx.doi.org/10.26737/jipf.v8i2.3877