ReCo – Automatically Processing Text Responses from Large-Scale Assessments

Project Description

The computer programme ReCo was originally developed at the Technical University of Munich and the Centre for International Student Assessment (ZIB). In cooperation with both institutions, the Centre for Technology Based Assessment (TBA Centre) at DIPF is now developing the software package further. The project comprises the general software development as well as scientific studies employing ReCo. For this, TBA is responsible for the conceptual and technical development on the one hand and acts as the project manager for projects such as ReCo-Multi on the other hand. Also, TBA supports external research groups using the program.

Enhancing Educational Measurement

When constructing tests, research groups are often faced with the question which response format should be implemented. If they offer response options, such as in the multiple choice format, responses can be easily processed. The downside is that this closed format entails guessing and other undesirable effects. But if the respondents can write their response into an open text field, the processing is often significantly more complicated. At the same time, open response formats are better suited to assessing deeper understanding than closed ones are for most domains, because the respondents are required to produce a response themselves rather than just recognise the correct one.

Therefore, the OECD already decided for PISA 2000  to include some tasks with open response format. The resulting effort for processing the text responses is particularly challenging for large-scale studies like PISA. Each participating country needs to train a team of coders diligently in order to evaluate the large amount of text responses from more than now 500,000 students. The coding team has guidelines for this task so that the coding takes place as objectively and comparable across countries as possible.

Because human beings have subjective perspectives and cultural backgrounds, the risk of inconsistent coding needs to be considered. Owing to a lack of stamina, humans are also prone to errors when being exposed to such a large volume of data. Computer programmes such as ReCo can improve consistency and, in turn, objectivity within as well as across countries. Also, the large volume of the data is not an obstacle for the computer, guaranteeing further consistency at a large scale. First studies with German data have shown that ReCo is capable of such automatic coding to a promising extent.

ReCo combines technologies from Natural Language Processing and Machine Learning. Among others, the program conducts a Latent Semantic Analysis on a text corpus specifically designed for this purpose. The meanings of words can thus be compared to each other. This way, it can identify different (semantic) response types in the data and connect them through Machine Learning with, for example, response correctness. In addition, ReCo can extract further response features, such as whether a student included information beyond the original text in their response or whether the student simply repeated the text.

ReCo-Multi: ReCo Learns Multiple Languages

For the purpose of extending ReCo’s scope to more languages, several PISA National Centers have joined the cooperation project ReCo-Multi. They share their text data comprising multiple test languages, further develop software components, and evaluate whether the satisfying findings of the German study can be replicated with other languages. Together with the German PISA National Center at the ZIB, TBA is in charge of managing and carrying out this international cooperation project.

Free Program

ReCo can be used upon request. Currently, the usage is constrained to German text data only. Over the course of the project, TBA will develop a graphical user interface for the programme and both will be freely available for download.

Funding

This project is funded from the institutional budget.

Cooperations

This project is being carried out in cooperation with...

• Technical University of Munich
• Centre for International Student Assessment (ZIB) e.V.

Selected Publications

• Zehner, F., Goldhammer, F., Lubaway, E., & Sälzer, C. (2018). Unattended consequences: How text responses alter alongside PISA's mode change from 2012 to 2015. Education Inquiry, 10(1), 34–55. doi: 10.1080/20004508.2018.1518080
• Zehner, F., Goldhammer, F., & Sälzer, C. (2018). Automatically analyzing text responses for exploring gender-specific cognitions in PISA reading. Large-scale Assessments in Education, 6:7. doi: 10.1186/s40536-018-0060-3
• Zehner, F. (2016). Automatic processing of text responses in large-scale assessments (Dissertation). Technische Universität München, München. doi: 10.13140/RG.2.2.26846.84800
• Zehner, F., Sälzer, C., & Goldhammer, F. (2016). Automatic coding of short text responses via clustering in educational assessment. Educational and Psychological Measurement, 76(2), 280–303. doi: 10.1177/0013164415590022
• Zehner, F., Goldhammer, F., & Sälzer, C. (2015). Using and improving coding guides for and by automatic coding of PISA short text responses. In Proceedings of the IEEE ICDM Workshop on Data Mining for Educational Assessment and Feedback (ASSESS 2015), Atlantic City. doi: 10.1109/icdmw.2015.189

Project Management

• Prof. Dr. Frank Goldhammer
• Dr. Fabian Zehner

Project Team

Project Details