Enhancing a Computer-Based Testing Environment with Optimum Item Response Time

Erhan Delen

doi:10.12973/eurasia.2015.1404a

Full Text (PDF)

Erhan Delen ¹ ^*

More Detail

¹ Giresun University^* Corresponding Author

Abstract

Background:
As technology has become more advanced and accessible in teaching and learning settings, there has been an upward trend in computer-based testing in the last decades.

Materials and methods:
The present experimental study examines students’ behaviors during computer-based testing in two different conditions and explores to what extent the test results were affected by those conditions.

Results:
Study results indicate that some of the psychometric features of a test (reliability and validity) could be enriched on computer-based testing when students are provided optimum item response time. In addition, it was found that providing optimum response time for each item influenced the students in the experimental group to not engage in rapid guess behaviors. Thereby, students spent a reasonable amount of time answering the questions, which resulted in more reliable and valid scores, aforementioned. Lastly, there was no statistically significant difference in two groups in terms of student performance.

Conclusions:
Psychometric features of a test could be enriched when students are provided additional features on a computer-based testing environment.

Keywords

reliability
validity
computer-based testing
psychometric features

License

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Article Type: Research Article

EURASIA J Math Sci Tech Ed, Volume 11, Issue 6, 2015, 1457-1472

https://doi.org/10.12973/eurasia.2015.1404a

Publication date: 29 Sep 2015

Article Views: 2098

Article Downloads: 954

Open Access References How to cite this article

References

Abedi, J. (2014). The use of computer technology in designing appropriate test accommodations for English language learners. Applied Measurement in Education, 27(4), 261-272. doi: 10.1080/08957347.2014.944310
Adesina, A., Stone, R., Batmaz, F., & Jones, I. (2014). Touch Arithmetic: A process-based Computer-Aided Assessment approach for capture of problem solving steps in the context of elementary mathematics. Computers & Education, 78, 333-343. doi:10.1016/j.compedu.2014.06.015
Allen, M. J., & Yen, W. M. (1979). Introduction to measurement theory. Monterey, CA: Brooks/Cole.
Brown, H. D., & Abeywickrama, P. (2010). Language assessment: Principles and classroom practices. White Plains, NY: Pearson Education.
Bulut, O., & Kan, A. (2012). Application of computerized adaptive testing to entrance examination for graduate studies in Turkey. Egitim Arastirmalari-Eurasian Journal of Educational Research, 49, 61-80.
Charman, D. & Elmes, A. (1998) Computer Based Assessment (Vol 1): a Guide to Good Practice SEED Publications. University of Plymouth.
Chou, C., Moslehpour, M., & Le Huyen, N. T. (2014). Concurrent and Predictive Validity of Computer-adaptive Freshman English Test for College Freshman English in Taiwan. International Journal of English Language Education, 2(1), pp-143. doi:10.5296/ijele.v2i1.4919
Chua, Y. P. (2012). Effects of computer-based testing on test performance and testing motivation. Computers in Human Behavior, 28, 1580–1586. doi:10.1016/j.chb.2012.03.020
Chua, Y. P., & Don, Z. M. (2013). Effects of computer-based educational achievement test on test performance and test takers’ motivation. Computers in Human Behavior, 29(5), 1889-1895. doi:10.1016/j.chb.2013.03.008
Clariana, R. and Wallace, P. (2002) . Paper-based versus computer-based assessment: key factors associated with the test mode effect. British Journal of Educational Technology, 33(5), 593–602. doi:10.1111/1467-8535.00294
DeAngelis, S. (2000). Equivalency of computer-based and paper-and-pencil testing. Journal of Allied Health. 29(3) 161–164.
DeBoer, G. E., Quellmalz, E. S., Davenport, J. L., Timms, M. J., Herrmann‐Abell, C. F., Buckley, B. C., ... & Flanagan, J. C. (2014). Comparing three online testing modalities: Using static, active, and interactive online testing modalities to assess middle school students' understanding of fundamental ideas and use of inquiry skills related to ecosystems. Journal of Research in Science Teaching, 51(4), 523-554. doi:10.1002/tea.21145
Debuse, J. C., & Lawley, M. (2015). Benefits and drawbacks of computer‐based assessment and feedback systems: Student and educator perspectives. British Journal of Educational Technology. doi:10.1111/bjet.12232
Delen, E., Liew, J., & Willson, V. (2014). Effects of interactivity and instructional scaffolding on learning: Self-regulation in online video-based environments. Computers & Education. 78, 312-320. doi: 10.1016/j.compedu.2014.06.018
Dindar, M., Yurdakul, I. K., & Dönmez, F. I. (2013). Multimedia in Test Items: Animated Questions vs. Static Graphics Questions. Procedia-Social and Behavioral Sciences, 106, 1876-1882. doi:10.1016/j.sbspro.2013.12.213
García Peñ alvo, F. J. (2008). Advances in E-Learning: Experiences and Methodologies. Hershey, PA, USA: Information Science Reference (formerly Idea Group Reference).
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis (7th ed.). Englewood Cliffs: Prentice Hall.
Hosseini, M., Abidin, M. J. Z., & Baghdarnia, M. (2014). Comparability of Test Results of Computer based Tests (CBT) and Paper and Pencil Tests (PPT) among English Language Learners in Iran. Procedia-Social and Behavioral Sciences, 98, 659-667. doi:10.1016/j.sbspro.2014.03.465
Jawaid, M., Moosa, F. A., Jaleel, F., & Ashraf, J. (2014). Computer Based Assessment (CBA): Perception of residents at Dow University of Health Sciences. Pak J Med Sci, 30(4), 688- 691. doi:10.12669/pjms.304.5444
Jeong, H. (2014). A comparative study of scores on computer-based tests and paper-based tests. Behaviour & Information Technology, 33(4), 410-422. doi:10.1080/0144929X.2012.710647
Kaya, F. & Delen, E. (2014). A computer-based peer nomination form to identify gifted and talented students. The Australasian Journal of Gifted Education. 23(2), 29-36.
Kong, X. J., Wise, S. L., Harmes, J. C., & Yang, S. (2006, April). Motivational effects of praise in response-time based feedback: A follow-up study of the effort-monitoring CBT. In annual meeting of the National Council on Measurement in Education, San Francisco.
Kong, X. J., Wise, S. L., & Bhola, D. S. (2007). Setting the response time threshold parameter to differentiate solution behavior from rapid-guessing behavior. Educational and Psychological Measurement, 67(4), 606-619.
Lee, M. (2009). CBAs in Korea: Experiences, results and challenges. In F. Scheurermann & J. Björnsson (Eds.), The transition to computer-based assessment: New approaches to skills assessment and implications for large-scale testing (pp. 187–193). Luxembourg, LU: Office for Official Publications of the European Communities.
Lee, Y. H., & Jia, Y. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-scale Assessments in Education, 2(1), 1-24. doi:10.1186/s40536-014-0008-1
Mason, B. J., Patry, M., & Berstein, D. J. (2001). An examination of the equivalence between non- adaptive computer-based and traditional testing. Journal of Educational Computing Research. 24(1) 29–39. doi:10.2190/9EPM-B14R-XQWT-WVNL
Ogletree, A., Ogletree, S., & Allen, B. (2014). Transition to Online Assessments: A Personal Perspective of Meeting Common Core State Standards in an Elementary School in Georgia. Georgia Educational Researcher, 11(1), 170. Available at: http://digitalcommons.georgiasouthern.edu/gerjournal/vol11/iss1/7
Olea, J., Abad, F. J., Ponsoda, V., Barrada, J. R., & Aguado, D. (2011). eCAT-listening: Design and psychometric properties of a computer-adaptive test on English listening. Psicothema, 23(4), 802-807.
Peat, M., & Franklin, S. (2002). Supporting student learning: the use of computer–based formative assessment modules. British Journal of Educational Technology, 33(5), 515- 523. doi:10.1111/1467-8535.00288
Pintrich, P. R. (1999). Understanding interference and inhibition processes from a motivational and self-regulated learning perspective: Comments on Dempster and Corkill. Educational Psychology Review, 11(2), 105-115. doi:10.1023/A:1022020308002
Prensky, M. (2001). Digital natives, digital immigrants Part 1. On the Horizon, 9(5), 1-6. doi:10.1108/10748120110424816
Ricketts, C., & Wilks, S. J. (2002). Improving student performance through computer-based assessment: Insights from recent research. Assessment & evaluation in higher education, 27(5), 475-479. doi:10.1080/0260293022000009348
Rosner, B. (1983). Percentage points for a generalized ESD many-outlier procedure. Technometrics, 25(2), 165-172. doi:10.1080/00401706.1983.10487848
Russell, M., Goldberg, A., & O'connor, K. (2003). Computer-based testing and validity: a look back into the future. Assessment in Education: Principles, Policy & Practice, 10(3), 279- 293. doi:10.1080/0969594032000148145
Schatz, P., & Browndyke, J. (2002). Applications of computer-based neuropsychological assessment. Journal of Head Trauma Rehabilitation, 17(5), 395-410.
Schatz, P., & Putz, B. O. (2006). Cross-validation of measures used for computer-based assessment of concussion. Applied Neuropsychology, 13(3), 151-159. doi:10.1207/s15324826an1303_2
Schatz, P., & Zillmer, E. A. (2003). Computer-based assessment of sports-related concussion. Applied Neuropsychology, 10(1), 42-47. doi:10.1207/S15324826AN1001_6
Sly, L & Rennie L J. (1999) Computer managed learning as an aid to formative assessment in higher education in Brown, S., Race, P. and Bull, J. (eds.) Computer Assisted Assessment in Higher Education. London: Kogan Page.
The Joint Information Systems Committee. (2007). Effective Practice with e-Assessment: An overview of technologies, policies and practice in further and higher education. Bristol: The Joint Information Systems
Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press.
Weinerth, K., Koenig, V., Brunner, M., & Martin, R. (2014). Concept maps: A useful and usable tool for computer-based knowledge assessment? A literature review with a focus on usability. Computers & Education, 78, 201-209. doi:10.1016/j.compedu.2014.06.002
Wirth, J. (2008). Computer-based tests: alternatives for test and item design. In J. Hartig, E. Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 235–252). Göttingen: Hogrefe.
Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163-183. doi:10.1207/s15324818ame1802_2
Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effortmoderated IRT model. Journal of Educational Measurement, 43, 19-38. doi:10.1111/j.1745-3984.2006.00002.x
Wise, S. L., Bhola, DS, & Yang, S. (2006). Taking the time to improve the validity of low-stakes tests: the effort-monitoring CBT. Educational Measurement: Issues and Practice, 25(2), 21–30. doi:10.1111/j.1745-3992.2006.00054.x
Zimmerman, B. J. (1989). A social cognitive view of self-regulated academic learning. Journal of Educational Psychology, 81(3), 329-339. doi:10.1037/0022-0663.81.3.329

How to cite this article

APA

Delen, E. (2015). Enhancing a Computer-Based Testing Environment with Optimum Item Response Time. Eurasia Journal of Mathematics, Science and Technology Education, 11(6), 1457-1472. https://doi.org/10.12973/eurasia.2015.1404a

Vancouver

Delen E. Enhancing a Computer-Based Testing Environment with Optimum Item Response Time. EURASIA J Math Sci Tech Ed. 2015;11(6):1457-72. https://doi.org/10.12973/eurasia.2015.1404a

AMA

Delen E. Enhancing a Computer-Based Testing Environment with Optimum Item Response Time. EURASIA J Math Sci Tech Ed. 2015;11(6), 1457-1472. https://doi.org/10.12973/eurasia.2015.1404a

Chicago

Delen, Erhan. "Enhancing a Computer-Based Testing Environment with Optimum Item Response Time". Eurasia Journal of Mathematics, Science and Technology Education 2015 11 no. 6 (2015): 1457-1472. https://doi.org/10.12973/eurasia.2015.1404a

Harvard

Delen, E. (2015). Enhancing a Computer-Based Testing Environment with Optimum Item Response Time. Eurasia Journal of Mathematics, Science and Technology Education, 11(6), pp. 1457-1472. https://doi.org/10.12973/eurasia.2015.1404a

MLA

Delen, Erhan "Enhancing a Computer-Based Testing Environment with Optimum Item Response Time". Eurasia Journal of Mathematics, Science and Technology Education, vol. 11, no. 6, 2015, pp. 1457-1472. https://doi.org/10.12973/eurasia.2015.1404a