Logo: Deutsches Institut für Internationale Pädagogische Forschung


National PISA 2012 CBA Add-On Study

PISA 2012 CBA NaBe investigated the practicability of computer-based assessment (CBA) in the context of PISA studies.

Project Description

Today, computers leave a mark upon both everyday and working life. Consequently, knowledge and skills are increasingly assessed by using computers, as was the case in the assessment of 15-year-old students in PISA 2012. Computer-based testing is not only cost-effective and effort-saving, but also, compared to conventional pen and paper-based formats, computer-based assessment (CBA) allows new test designs and response types. Hence, the national add-on study PISA 2012 in Germany aimed to find solutions for the possible computer applicability particularly for assessing reading and mathematics comprehension skills.


The add-on study focused on conducting computer-based assessment (CBA) in the context of PISA. In particular, research-oriented answers were to be given to the following areas of investigation:

  1. Computer-based measurement of reading hypertexts (Digital Reading) 
    On the contrary to conventional linear texts, digital hypertext formats own certain characteristics that request special demands of the reader. Gathering information from a website does not only mean reading text passages. Moreover, the user has to decide, which additional websites to access via hyperlinks and in what order. Other criteria need to be taken into account when choosing a website, e.g. regarding the relevance and trustworthiness of the pages. In the case of capturing digital texts, the computer is thus not only an output medium, but it furthermore enables the simulation of hypertexts. In the national add-on study to PISA, the survey investigated empirically whether linear text reading competency can be distinguished from hypertext reading competency, and to what extent differences can be explained concerning individual skills, such as computer literacy. 

    Examples from NaBe

    This is how the implementation of digital reading tasks looks like:

    The tasks are constructed similarly to homepages. Links can be clicked and refer to further pages, of course.


  2. Computer-based testing of mathematics (CBA of Mathematics)
    Compared to the questions regarding digital text reading, similar ones were addressed by the introduction of computer-based assessment of mathematics in PISA 2012. But unlike in the case of reading, it was not assumed that the assessment of mathematics is fundamentally changed by the computer as a medium of delivery. Rather, computers were considered merely as a distribution system assessing the same – regarding convergent validity – mathematical competencies as the pen-and paper formats.

  3. Investigation of mode effects
    Further aspects to be addressed by the PISA add-on study related to effects that might occur by transferring linear paper test formats to computers. To maintain comparability of linear reading between the two different modes, paper vs. computer, particular attention was paid to a preferable high congruence of paper-based with computer-based format. Besides technical transferability per se, it was of particular interest, if the change of mode affects the psychometric characteristics of tasks, e.g. difficulty of solution, and, if so, how occurring changes might be explained. The investigation focused on the question of how mode effects depend on particular response types, e.g. free text writing or multiple choice, of the reading tasks.

    Examples from NaBe

    This is how the tasks for the investigation of mode effects look like:

    The tasks should be as similar as possible compared to the paper-based version. Therefore the tasks do not include hyperlinks in the text, for example.

  4. Mode effects on motivation
    The given performance in the competence fields in PISA not only depends on cognitive skills of students, but also on their present mood and motivation. Hence, the PISA add-on study questioned to what extent the modes, paper- vs. computer-based presentation, affect the actual motivation – and therefore maybe the performance likewise.

Methodologic Approach

Data assessment of the CBA add-on study took place as the second day of the PISA 2012 main assessment. For taking part in the add-on study 77 of 212 PISA schools were acquired, i.e., data of nearly 900 students could be analysed. Mainly, the students answered tasks delivered by a computer. Beside a test battery verifying their computer skills, working memory capacity and basic reading skills, they were working on reading comprehension tasks and questions collecting their respective stances. Some students answered tasks in a test book in paper format. Furthermore, research questions were addressed by integrating findings from reading, mathematics and problem solving tasks that were administered computer-based on the first day of the PISA study.


Center for International Student Assessment (ZIB)




  • Hahnel, C., Goldhammer, F., Naumann, J., & Kröhne, U. (2014, September). Validierung von Digital Reading: Befunde zum Erklärungsbeitrag ICT-bezogener Komponenten [Validating Digital Reading: Findings about the contribution in explanation of ICT-related components]. Talk presented at the 49. Kongress der Deutschen Gesellschaft für Psychologie (DGPs), Bochum, Germany, September 21-25, 2014.
  • Hahnel, C., Kröhne, U., Naumann, J., & Goldhammer, F. (2013, August). Explaining hyperlink selection by PISA reading and cognitive component skills. Talk presented at the 15th Biennal Conference of the European Association for Research on Learning and Instruction (EARLI), Munich, Germany, August 27-31, 2013.
  • Kröhne, U., Hahnel, C., Schiepe-Tiska, A., & Goldhammer, F. (2013, August). Analyzing Mode Effects of PISA Print Reading Including a Comparison of Time-related Information. Talk presented at the 15th Biennal Conference of the European Association for Research on Learning and Instruction (EARLI), Munich, Germany, August 27-31, 2013.

Project management

Prof. Dr. Frank Goldhammer

Project Details

Completed projects
12/2011 – 12/2016
External funding
last modified Feb 22, 2018