Search results in the DIPF database of publications

Your query:

(Schlagwörter: "Programm")

Practical significance of item misfit and its manifestations in constructs assessed in large‑scale […] Fährmann, Katharina; Köhler, Carmen; Hartig, Johannes; Heine, Jörg‑Henrik Journal Article | In: Large-scale Assessments in Education | 2022 42893 Endnote: Author(s): Fährmann, Katharina; Köhler, Carmen; Hartig, Johannes; Heine, Jörg‑Henrik
Title: Practical significance of item misfit and its manifestations in constructs assessed in large‑scale studies
In: Large-scale Assessments in Education, 10 (2022) , S. 7
DOI: 10.1186/s40536‑022‑00124‑w
URL: https://largescaleassessmentsineducation.springeropen.com/articles/10.1186/s40536-022-00124-w
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Abstract (english): When scaling psychological tests with methods of item response theory it is necessary to investigate to what extent the responses correspond to the model predictions. In addition to the statistical evaluation of item misfit, the question arises as to its practical significance. Although item removal is undesirable for several reasons, its practical consequences are rarely investigated and focus mostly on main survey data with pre-selected items. In this paper, we identify criteria to evaluate practical significance and discuss them with respect to various types of assessments and their particular purposes. We then demonstrate the practical consequences of item misfit using two data examples from the German PISA 2018 field trial study: one with cognitive data and one with non-cognitive/metacognitive data. For the former, we scale the data under the GPCM with and without the inclusion of misfitting items, and investigate how this influences the trait distribution and the allocation to reading competency levels. For non-cognitive/metacognitive data, we explore the effect of excluding misfitting items on estimated gender differences. Our results indicate minor practical consequences for person allocation and no changes in the estimated gender-difference effects. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen

Effective use of learner-generated data in teacher training activities Drachsler, Hendrik Working Papers | 2022 43208 Endnote: Author(s): Drachsler, Hendrik
Title: Effective use of learner-generated data in teacher training activities
Published: Brussels: European Schoolnet Academy, 2022
URL: http://www.eun.org/documents/411753/817341/Effective+Use+of+Learner-Generated+Data+in+Teacher+Training+Activities+v3.pdf/6e1bfeb2-b981-4170-a251-01e496173d08
Publication Type: 5. Arbeits- und Diskussionspapiere; Handreichung für die Praxis
Language: Englisch
Keywords: Analyseverfahren; Computerprogramm; Datenanalyse; Datenschutz; Datenschutzgesetz; Ethik; Learning Analytics; Lehrerausbildung; Technologie; Trainingsprogramm
Abstract: This report contains an analysis of the effective use of learner-generated data with a specific focus on teacher training. The use of learner-generated data in research is coined Learning Analytics. So far very little is published about learner-generated data in the context of teacher training. The following report aims to introduce teacher trainers, policymakers and practitioners to learning analytics. It defines learning analytics and describes the core dimensions of it, such as stakeholders, objectives, data, technologies, ethics & privacy, and required competencies to exploit the benefits of learning analytics. After a comprehensive introduction to learning analytics, the report shows practical examples of learning analytics applications. It describes how learner-generated data can be identified, captured, and visualised on learner analytics dashboards. It also provides examples of public-funded and commercial applications of learning that take advantage of learning analytics in schools these days. Finally, the report outlines a potential teacher training program inspired by university master programs on learning analytics. It also suggests two training scenarios (a low-tech and a high-tech scenario) that could be used for the training of teachers on learner-generated data. (DIPF/Orig.)
DIPF-Departments: Informationszentrum Bildung

shinyReCoR: A shiny application for automatically coding text responses using R Andersen, Nico; Zehner, Fabian Journal Article | In: Psych | 2021 41458 Endnote: Author(s): Andersen, Nico; Zehner, Fabian
Title: shinyReCoR: A shiny application for automatically coding text responses using R
In: Psych, 3 (2021) 3, S. 422-446
DOI: 10.3390/psych3030030
URL: https://www.mdpi.com/2624-8611/3/3/30
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Beitrag in Sonderheft
Language: Englisch
Keywords: Natürliche Sprache; Sprachverarbeitung; Text; Codierung; Computerprogramm; Methodologie
Abstract (english): In this paper, we introduce shinyReCoR: a new app that utilizes a cluster-based method for automatically coding open-ended text responses. Reliable coding of text responses from educational or psychological assessments requires substantial organizational and human effort. The coding of natural language in responses to tests depends on the texts' complexity, corresponding coding guides, and the guides' quality. Manual coding is thus not only expensive but also error-prone. With shinyReCoR, we provide a more efficient alternative. The use of natural language processing makes texts utilizable for statistical methods. shinyReCoR is a Shiny app deployed as an R-package that allows users with varying technical affinity to create automatic response classifiers through a graphical user interface based on annotated data. The present paper describes the underlying methodology, including machine learning, as well as peculiarities of the processing of language in the assessment context. The app guides users through the workflow with steps like text corpus compilation, semantic space building, preprocessing of the text data, and clustering. Users can adjust each step according to their needs. Finally, users are provided with an automatic response classifier, which can be evaluated and tested within the process. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen

Separating PIAAC competencies from general cognitive skills. A dimensionality and explanatory […] Engelhardt, Lena; Goldhammer, Frank; Lüdtke, Oliver; Köller, Olaf; Baumert, Jürgen; […] Journal Article | In: Studies in Educational Evaluation | 2021 41542 Endnote: Author(s): Engelhardt, Lena; Goldhammer, Frank; Lüdtke, Oliver; Köller, Olaf; Baumert, Jürgen; Carstensen, Claus H.
Title: Separating PIAAC competencies from general cognitive skills. A dimensionality and explanatory analysis
In: Studies in Educational Evaluation, 71 (2021) , S. 101069
DOI: 10.1016/j.stueduc.2021.101069
URL: https://www.sciencedirect.com/science/article/pii/S0191491X2100095X
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: PIAAC <Programme for the International Assessment of Adult Competencies>; Schreib- und Lesefähigkeit; Rechenkompetenz; Kognitive Kompetenz; Validität; Leistungsmessung; Test
Abstract: This study aims to investigate how test scores from PIAAC (Programme for the International Assessment of Adult Competencies) can be interpreted, by comparing the PIAAC competencies literacy and numeracy to reasoning and perceptual speed. Dimensionality analyses supported, that the PIAAC competencies can be separated into a common factor overlapping with reasoning and perceptual speed, and domain-specific factors. For the common and specific factors, relations to other variables were analyzed. The nested factor for PIAAC literacy was as expected unrelated to age, positively related to learning opportunities during one's lifetime, and positively related to literacy skill use. The nested factor for PIAAC numeracy was also as expected unrelated to age, against expectation unrelated to learning opportunities during one's lifetime, and as expected positively related to numeracy skill use. Results support the validity of the intended test score interpretation for PIAAC literacy, while results for PIAAC numeracy were less clear. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen

From byproduct to design factor. On validating the interpretation of process indicators based on […] Goldhammer, Frank; Hahnel, Carolin; Kroehne, Ulf; Zehner, Fabian Journal Article | In: Large-scale Assessments in Education | 2021 41612 Endnote: Author(s): Goldhammer, Frank; Hahnel, Carolin; Kroehne, Ulf; Zehner, Fabian
Title: From byproduct to design factor. On validating the interpretation of process indicators based on log data
In: Large-scale Assessments in Education, 9 (2021) , S. 20
DOI: 10.1186/s40536-021-00113-5
URN: urn:nbn:de:0111-pedocs-250050
URL: https://nbn-resolving.org/urn:nbn:de:0111-pedocs-250050
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Leistungstest; Logdatei; PISA <Programme for International Student Assessment>; PIAAC <Programme for the International Assessment of Adult Competencies>; Datenanalyse; Interpretation; Leistungsmessung; Messverfahren; Indikator; Typologie; Testkonstruktion; Testtheorie
Abstract (english): International large-scale assessments such as PISA or PIAAC have started to provide public or scientific use files for log data; that is, events, event-related attributes and timestamps of test-takers' interactions with the assessment system. Log data and the process indicators derived from it can be used for many purposes. However, the intended uses and interpretations of process indicators require validation, which here means a theoretical and/or empirical justification that inferences about (latent) attributes of the test-taker's work process are valid. This article reviews and synthesizes measurement concepts from various areas, including the standard assessment paradigm, the continuous assessment approach, the evidence-centered design (ECD) framework, and test validation. Based on this synthesis, we address the questions of how to ensure the valid interpretation of process indicators by means of an evidence-centered design of the task situation, and how to empirically challenge the intended interpretation of process indicators by developing and implementing correlational and/or experimental validation strategies. For this purpose, we explicate the process of reasoning from log data to low-level features and process indicators as the outcome of evidence identification. In this process, contextualizing information from log data is essential in order to reduce interpretative ambiguities regarding the derived process indicators. Finally, we show that empirical validation strategies can be adapted from classical approaches investigating the nomothetic span and construct representation. Two worked examples illustrate possible validation strategies for the design phase of measurements and their empirical evaluation. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen

Controlling speed in component skills of reading improves the explanation of reading comprehension Goldhammer, Frank; Kroehne, Ulf; Hahnel, Carolin; De Boeck, Paul Journal Article | In: Journal of Educational Psychology | 2021 41462 Endnote: Author(s): Goldhammer, Frank; Kroehne, Ulf; Hahnel, Carolin; De Boeck, Paul
Title: Controlling speed in component skills of reading improves the explanation of reading comprehension
In: Journal of Educational Psychology, 113 (2021) 5, S. 861-878
DOI: 10.1037/edu0000655
URN: urn:nbn:de:0111-pedocs-237977
URL: https://nbn-resolving.org/urn:nbn:de:0111-pedocs-237977
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Lesekompetenz; Fertigkeit; Kognitive Prozesse; Leistung; Antwort; Zeit; Wort; Semantik; Text; Leseverstehen; PISA <Programme for International Student Assessment>; Schüler; Messverfahren; Test; Experimentelle Untersuchung; Empirische Untersuchung; Deutschland
Abstract: Efficiency in reading component skills is crucial for reading comprehension, as efficient subprocesses do not extensively consume limited cognitive resources, making them available for comprehension processes. Cognitive efficiency is typically measured with speeded tests of relatively easy items. Observed responses and response times indicate the latent variables of ability and speed. Interpreting only ability or speed as efficiency may be misleading because there is a within-person dependency between both variables (speed-ability tradeoff [SAT]). Therefore, the present study measures efficiency as ability conditional on speed by controlling speed experimentally with item-level time limits. The proposed timed ability measures of reading component skills are expected to have a clearer interpretation in terms of efficiency and to be better predictors for reading comprehension. To support this claim, this study investigates two component skills, visual word recognition and sentence-level semantic integration (sentence reading), to understand how differences in ability in a timed condition are related to differences in ability and speed in a traditional untimed condition. Moreover, untimed and timed reading component skill measures were used to explain reading comprehension. A German subsample from Programme for International Student Assessment (PISA) 2012 completed the reading component skills tasks with and without item-level time limits and PISA reading tasks. The results showed that timed ability is only moderately related to untimed ability. Furthermore, timed ability measures proved to be stronger predictors of sentence-level and text-level reading comprehension than the corresponding untimed ability and speed measures, although using untimed ability and speed jointly as predictors increased the amount of explained variance.
Abstract (english): Efficiency in reading component skills is crucial for reading comprehension, as efficient subprocesses do not extensively consume limited cognitive resources, making them available for comprehension processes. Cognitive efficiency is typically measured with speeded tests of relatively easy items. Observed responses and response times indicate the latent variables of ability and speed. Interpreting only ability or speed as efficiency may be misleading because there is a within-person dependency between both variables (speed-ability tradeoff [SAT]). Therefore, the present study measures efficiency as ability conditional on speed by controlling speed experimentally with item-level time limits. The proposed timed ability measures of reading component skills are expected to have a clearer interpretation in terms of efficiency and to be better predictors for reading comprehension. To support this claim, this study investigates two component skills, visual word recognition and sentence-level semantic integration (sentence reading), to understand how differences in ability in a timed condition are related to differences in ability and speed in a traditional untimed condition. Moreover, untimed and timed reading component skill measures were used to explain reading comprehension. A German subsample from Programme for International Student Assessment (PISA) 2012 completed the reading component skills tasks with and without item-level time limits and PISA reading tasks. The results showed that timed ability is only moderately related to untimed ability. Furthermore, timed ability measures proved to be stronger predictors of sentence-level and text-level reading comprehension than the corresponding untimed ability and speed measures, although using untimed ability and speed jointly as predictors increased the amount of explained variance.
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen

Akzeptanz der Nutzung von automatisiertem Assessment im Rahmen einer virtuellen Vorlesung Horn, Florian; Schiffner, Daniel; Krömker, Detlef Book Chapter | Aus: Greubel, André; Strickroth, Sven; Striewe, Michael (Hrsg.): Proceedings of the Fifth Workshop "Automatische Bewertung von Programmieraufgaben" (ABP 2021), virtual event, October 28-29, 2021 | Bonn: Gesellschaft für Informatik | 2021 41563 Endnote: Author(s): Horn, Florian; Schiffner, Daniel; Krömker, Detlef
Title: Akzeptanz der Nutzung von automatisiertem Assessment im Rahmen einer virtuellen Vorlesung
In: Greubel, André; Strickroth, Sven; Striewe, Michael (Hrsg.): Proceedings of the Fifth Workshop "Automatische Bewertung von Programmieraufgaben" (ABP 2021), virtual event, October 28-29, 2021, Bonn: Gesellschaft für Informatik, 2021 , S. 28-29
DOI: 10.18420/abp2021-4
URL: https://dl.gi.de/handle/20.500.12116/37542
Publication Type: 4. Beiträge in Sammelbänden; Tagungsband/Konferenzbeitrag/Proceedings
Language: Englisch; Deutsch
Keywords: Computerunterstützter Unterricht; Digitalisierung; Computerprogramm; Leistungsbeurteilung; Studierender; Erfahrung; Fragebogen; Programmierung; Aufgabe
Abstract: Durch die Umstellung auf virtuelle Lehre findet auch die Verwendung von automatischen Tools zur Bewertung von Programmieraufgaben immer mehr den Einzug in die Lehre. Im Rahmen einer solchen virtuellen Vorlesung wurde eine Bewertung durch die Studierenden vorgenommen, um daraus Erkenntnisse für die zukünftige Einbettung in der Lehre zu ziehen. Die Vorlesung zielt dabei auf höhere Semester des Bachelorstudiengangs ab und nutzt dabei Vorerfahrungen der Studierenden. Insgesamt wurde Feedback von 47 Studierende durch einen Fragebogen erhoben, und daraus Rückschlüsse auf die Qualität und Einsetzbarkeit von Unit-Tests gezogen. (DIPF/Orig.)
DIPF-Departments: Informationszentrum Bildung

Global competence in PISA 2018. Einstellungen von Fünfzehnjährigen in Deutschland zu globalen und […] Weis, Mirjam; Reiss, Kristina; Mang, Julia; Schiepe-Tiska, Anja; Diedrich, Jenniger; Roczen, Nina; […] Monograph | Münster: Waxmann | 2020 40667 Endnote: Author(s): Weis, Mirjam; Reiss, Kristina; Mang, Julia; Schiepe-Tiska, Anja; Diedrich, Jenniger; Roczen, Nina; Jude, Nina
Title: Global competence in PISA 2018. Einstellungen von Fünfzehnjährigen in Deutschland zu globalen und interkulturellen Themen
Published: Münster: Waxmann, 2020 (Wissenschaft macht Schule, 2)
DOI: 10.31244/978383099300
URN: urn:nbn:de:0111-pedocs-210696
URL: https://www.pedocs.de/frontdoor.php?source_opus=21069
Publication Type: 1. Monographien (Autorenschaft); Monographie
Language: Deutsch
Keywords: Deutschland; Einstellung <Psy>; Eltern; Fragebogenerhebung; Globales Denken; Globales Lernen; Globalisierung; Interkulturalität; Interkulturelle Kompetenz; Internationaler Vergleich; Jugendlicher; Lehrer; PISA <Programme for International Student Assessment>; Schüler; Schülerperspektive; Schulform; Schulleiter; Selbsteinschätzung
Abstract: In der PISA-Studie 2018 wurde als innovative Domäne erstmals Global Competence bei fünfzehnjährigen Schülerinnen und Schülern erfasst. In dieser Zusatzerhebung werden das selbsteingeschätzte Wissen von Schülerinnen und Schülern zu Themen mit lokaler und globaler Bedeutung (z. B. Klimawandel, Armut, Pandemien) sowie ihre Einstellungen zu globalen und interkulturellen Themen in den Blick genommen. Dabei geht es beispielsweise um den respektvollen Umgang mit Menschen unterschiedlicher nationaler Herkunft und entsprechendem ethnischen, religiösen, sozialen oder kulturellen Hintergrund. Diese Broschüre stellt die Ergebnisse der Schülerinnen und Schüler in Deutschland aus der Zusatzauswertung Global Competence bei der PISA-Studie 2018 vor und betrachtet diese im internationalen Vergleich. Zusätzlich werden die Sicht der Schulleitungen und Lehrkräfte in den verschiedenen Schularten sowie die Sicht der Eltern einbezogen. (DIPF/Orig.)
DIPF-Departments: Bildungsqualität und Evaluation

Do student ratings provide reliable and valid information about teaching quality at the school […] Aditomo, Anindito; Köhler, Carmen Journal Article | In: Educational Assessment, Evaluation and Accountability | 2020 40512 Endnote: Author(s): Aditomo, Anindito; Köhler, Carmen
Title: Do student ratings provide reliable and valid information about teaching quality at the school level? Evaluating measures of science teaching in PISA 2015
In: Educational Assessment, Evaluation and Accountability, 32 (2020) 3, S. 275-310
DOI: 10.1007/s11092-020-09328-6
URL: https://link.springer.com/article/10.1007/s11092-020-09328-6
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Naturwissenschaftlicher Unterricht; Qualität; Messung; Evaluation; Schüler; Bewertung; Schülerurteil; Validität; Reliabilität; Schulklima; Einflussfaktor; Wirkung; PISA <Programme for International Student Assessment>; Modell; Faktorenanalyse; Empirische Untersuchung; OECD-Länder
Abstract: Large-scale educational surveys, including PISA, often collect student ratings to assess teaching quality. Because of the sampling design in PISA, student ratings must be aggregated at the school level instead of the classroom level. To what extent does school-level aggregation of student ratings yield reliable and valid measures of teaching quality? We investigate this question for six scales measuring classroom management, emotional support, inquiry-based instruction, teacher-directed instruction, adaptive instruction, and feedback provided by PISA 2015. The sample consisted of 503,146 students from 17,678 schools in 69 countries/regions. Multilevel CFA and SEM were conducted for each scale in each country/region to evaluate school-level reliability (intraclass correlations 1 and 2), factorial validity, and predictive validity. In most countries/regions, school-level reliability was found to be adequate for the classroom management scale, but only low to moderate for the other scales. Examination of factorial and predictive validity indicated that the classroom management, emotional support, adaptive instruction, and teacher-directed instruction scales capture meaningful differences in teaching quality between schools. Meanwhile, the inquiry scale exhibited poor validity in almost all countries/regions. These findings suggest the possibility of using student ratings in PISA to investigate some aspects of school-level teaching quality in most countries/regions. (DIPF/Orig.)
DIPF-Departments: Bildungsqualität und Evaluation

Measurement invariance testing in questionnaires. A comparison of three Multigroup-CFA and […] Buchholz, Janine; Hartig, Johannes Journal Article | In: Psychological Test and Assessment Modelling | 2020 39818 Endnote: Author(s): Buchholz, Janine; Hartig, Johannes
Title: Measurement invariance testing in questionnaires. A comparison of three Multigroup-CFA and IRT-based approaches
In: Psychological Test and Assessment Modelling, 62 (2020) 1, S. 29-54
URL: https://www.psychologie-aktuell.com/fileadmin/Redaktion/Journale/ptam-2020-1/03_Buchholz.pdf
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Beitrag in Sonderheft
Language: Englisch
Keywords: PISA <Programme for International Student Assessment>; Item-Response-Theorie; Faktorenanalyse; Schülerleistung; Leistungsmessung; Messung; Invarianz; Validität; Statistische Methode
Abstract (english): International Large-Scale Assessments aim at comparisons of countries with respect to latent constructs such as attitudes, values and beliefs. Measurement invariance (MI) needs to hold in order for such comparisons to be valid. Several statistical approaches to test for MI have been proposed: While Multigroup Confirmatory Factor Analysis (MGCFA) is particularly popular, a newer, IRT-based approach was introduced for non-cognitive constructs in PISA 2015, thus raising the question of consistency between these approaches. A total of three approaches (MGCFA for ordinal and continuous data, multi-group IRT) were applied to simulated data containing different types and extents of MI violations, and to the empirical non-cognitive PISA 2015 data. Analyses are based on indices of the magnitude (i.e., parameter-specific modification indices resulting from MGCFA and group-specific item fit statistics resulting from the IRT approach) and direction of local misfit (i.e., standardized parameter change and mean deviation, respectively). Results indicate that all measures were sensitive to (some) MI violations and more consistent in identifying group differences in item difficulty parameters.
DIPF-Departments: Bildungsqualität und Evaluation