Search results in the DIPF database of publications

Your query:

(Personen: "Eichmann," und "Beate")

Detecting the disengaged reader. Using scrolling data to predict disengagement during reading Biedermann, Daniel; Schneider, Jan; Ciordas-Hertel, George-Petru; Eichmann, Beate; Hahnel, Carolin; […] Book Chapter | Aus: Hilliger, Isabel, Khosravi, Hassan; Rienties, Bart; Dawson, Shane (Hrsg.): LAK23 Conference Proceedings: The Thirteenth International Conference on Learning Analytics & Knowledge, March 13-17, 2023, Hybrid, Arlington, Texas, USA | New York; NY: Association for Computing Machinery | 2023 43666 Endnote: Author(s): Biedermann, Daniel; Schneider, Jan; Ciordas-Hertel, George-Petru; Eichmann, Beate; Hahnel, Carolin; Goldhammer, Frank; Drachsler, Hendrik
Title: Detecting the disengaged reader. Using scrolling data to predict disengagement during reading
In: Hilliger, Isabel, Khosravi, Hassan; Rienties, Bart; Dawson, Shane (Hrsg.): LAK23 Conference Proceedings: The Thirteenth International Conference on Learning Analytics & Knowledge, March 13-17, 2023, Hybrid, Arlington, Texas, USA, New York; NY: Association for Computing Machinery, 2023 , S. 585-591
DOI: 10.1145/3576050.3576078
URL: https://dl.acm.org/doi/10.1145/3576050.3576078
Publication Type: 4. Beiträge in Sammelbänden; Beiträge in Proceedings mit Peer-Review-System
Language: Englisch
Abstract: When reading long and complex texts, students may disengage and miss out on relevant content. In order to prevent disengaged behavior or to counteract it by means of an intervention, it is ideally detected an early stage. In this paper, we present a method for early disengagement detection that relies only on the classification of scrolling data. The presented method transforms scrolling data into a time series representation, where each point of the series represents the vertical position of the viewport in the text document. This time series representation is then classified using time series classification algorithms. We evaluated the method on a dataset of 565 university students reading eight different texts. We compared the algorithm performance with different time series lengths, data sampling strategies, the texts that make up the training data, and classification algorithms. The method can classify disengagement early with up to 70% accuracy. However, we also observe differences in the performance depending on which of the texts are included in the training dataset. We discuss our results and propose several possible improvements to enhance the method. (DIPF/Orig.)
DIPF-Departments: Informationszentrum Bildung; Lehr und Lernqualität in Bildungseinrichtungen

Test zur Evaluation von Online-Informationen (EVON) Hahnel, Carolin; Eichmann, Beate; Goldhammer, Frank Research Data, Diagnostic Instruments | DIPF | Leibniz-Institut für Bildungsforschung und Bildungsinformation / Forschungsdatenzentrum Bildung | 2022 42547 Endnote: Author(s): Hahnel, Carolin; Eichmann, Beate; Goldhammer, Frank
Title: Test zur Evaluation von Online-Informationen (EVON)
Published: Frankfurt am Main: DIPF | Leibniz-Institut für Bildungsforschung und Bildungsinformation / Forschungsdatenzentrum Bildung, 2022
DOI: 10.7477/680:337:58
URL: https://doi.org/10.7477/680:337:58
Publication Type: 6. Forschungsdaten; Instrumente; Diagnostische Instrumente (inkl. Testverfahren)
Language: Sonstiges
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen

Applying psychometric modeling to aid feature engineering in predictive log-data analytics. The […] Zehner, Fabian; Eichmann, Beate; Deribo, Tobias; Harrison, Scott; Bengs, Daniel; Andersen, Nico; […] Journal Article | In: Journal of Educational Data Mining | 2021 41457 Endnote: Author(s): Zehner, Fabian; Eichmann, Beate; Deribo, Tobias; Harrison, Scott; Bengs, Daniel; Andersen, Nico; Hahnel, Carolin
Title: Applying psychometric modeling to aid feature engineering in predictive log-data analytics. The NAEP EDM Competition
In: Journal of Educational Data Mining, 13 (2021) 2, S. 80-107
DOI: 10.5281/zenodo.5275316
URN: urn:nbn:de:0111-dipfdocs-250034
URL: https://nbn-resolving.org/urn:nbn:de:0111-dipfdocs-250034
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Beitrag in Sonderheft
Language: Englisch
Keywords: Psychometrie; Modellierung; Protokoll; Datenanalyse; Testverhalten; Cluster
Abstract (english): The NAEP EDM Competition required participants to predict efficient test-taking behavior based on log data. This paper describes our top-down approach for engineering features by means of psychometric modeling, aiming at machine learning for the predictive classification task. For feature engineering, we employed, among others, the Log-Normal Response Time Model for estimating latent person speed, and the Generalized Partial Credit Model for estimating latent person ability. Additionally, we adopted an n-gram feature approach for event sequences. Furthermore, instead of using the provided binary target label, we distinguished inefficient test takers who were going too fast and those who were going too slow for training a multi-label classifier. Our best-performing ensemble classifier comprised three sets of low-dimensional classifiers, dominated by test-taker speed. While our classifier reached moderate performance, relative to the competition leaderboard, our approach makes two important contributions. First, we show how classifiers that contain features engineered through literature-derived domain knowledge can provide meaningful predictions if results can be contextualized to test administrators who wish to intervene or take action. Second, our re-engineering of test scores enabled us to incorporate person ability into the models. However, ability was hardly predictive of efficient behavior, leading to the conclusion that the target label's validity needs to be questioned. Beyond competition-related findings, we furthermore report a state sequence analysis for demonstrating the viability of the employed tools. The latter yielded four different test-taking types that described distinctive differences between test takers, providing relevant implications for assessment practice. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen

Using process data to explain group differences in complex problem solving Eichmann, Beate; Goldhammer, Frank; Greiff, Samuel; Brandhuber, Liene; Naumann, Johannes Journal Article | In: Journal of Educational Psychology | 2020 39868 Endnote: Author(s): Eichmann, Beate; Goldhammer, Frank; Greiff, Samuel; Brandhuber, Liene; Naumann, Johannes
Title: Using process data to explain group differences in complex problem solving
In: Journal of Educational Psychology, 112 (2020) 8, S. 1546-1562
DOI: 10.1037/edu0000446
URN: urn:nbn:de:0111-pedocs-232721
URL: https://nbn-resolving.org/urn:nbn:de:0111-pedocs-232721
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: PISA <Programme for International Student Assessment>; Problemlösen; Schülerleistung; Leistungsmessung; Geschlechtsspezifischer Unterschied; Migrationshintergrund; Computerunterstütztes Verfahren; Logdatei; Interaktion; Exploration; Verhalten; Vorwissen; Wirkung; Indikator; Leistung; Unterschied; Messverfahren; OECD-Länder
Abstract: In large-scale assessments, performance differences across different groups are regularly found. These group differences (e.g., gender differences) are often relevant for educational policy decisions and measures. However, the formation of these group differences usually remains unclear. We propose an approach for investigating this formation by considering behavioral process measures as mediating variables between group membership and performance on the 2012 Programme for International Student Assessment complex problem solving (CPS) items. We found that across all investigated countries interactive behavior can fully explain gender differences in CPS, but cannot explain differences between students with and without a migration background. However, in some countries these results differ from the cross-country results. Our results indicate that process measures derived from log data are useful for further investigating and explaining performance differences between girls and boys and students with and without migration background. (DIPF/Orig.)
DIPF-Departments: Bildungsqualität und Evaluation

Exploring behavioural patterns during complex problem‐solving Eichmann, Beate; Greiff, Samuel; Naumann, Johannes; Brandhuber, Liene; Goldhammer, Frank Journal Article | In: Journal of Computer Assisted Learning | 2020 40129 Endnote: Author(s): Eichmann, Beate; Greiff, Samuel; Naumann, Johannes; Brandhuber, Liene; Goldhammer, Frank
Title: Exploring behavioural patterns during complex problem‐solving
In: Journal of Computer Assisted Learning, 36 (2020) 6, S. 933-956
DOI: 10.1111/jcal.12451
URN: urn:nbn:de:0111-pedocs-232225
URL: https://nbn-resolving.org/urn:nbn:de:0111-pedocs-232225
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Problemlösen; Exploration; Datenanalyse; PISA <Programme for International Student Assessment>; Verhaltensmuster; Sequenz; Analyse
Abstract: In this explorative study, we investigate how sequences of behaviour are related to success or failure in complex problem‐solving (CPS). To this end, we analysed log data from two different tasks of the problem‐solving assessment of the Programme for International Student Assessment 2012 study (n = 30,098 students). We first coded every interaction of students as (initial or repeated) exploration, (initial or repeated) goal‐directed behaviour, or resetting the task. We then split the data according to task successes and failures. We used full‐path sequence analysis to identify groups of students with similar behavioural patterns in the respective tasks. Double‐checking and minimalistic behaviour was associated with success in CPS, while guessing and exploring task‐irrelevant content was associated with failure. Our findings held for both tasks investigated, from two different CPS measurement frameworks. We thus gained detailed insight into the behavioural processes that are related to success and failure in CPS. (DIPF/Orig.)
DIPF-Departments: Bildungsqualität und Evaluation

Evaluation of online information in university students. Development and scaling of the screening […] Hahnel, Carolin; Eichmann, Beate; Goldhammer, Frank Journal Article | In: Frontiers in Psychology | 2020 40881 Endnote: Author(s): Hahnel, Carolin; Eichmann, Beate; Goldhammer, Frank
Title: Evaluation of online information in university students. Development and scaling of the screening instrument EVON
In: Frontiers in Psychology, (2020) , S. 11:562128
DOI: 10.3389/fpsyg.2020.562128
URN: urn:nbn:de:0111-pedocs-232241
URL: https://nbn-resolving.org/urn:nbn:de:0111-pedocs-232241
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Deutschland; Internet; Informationskompetenz; Ressource; Glaubwürdigkeit; Relevanz; Bewertung; Test; Testentwicklung; Itemanalyse; Suchmaschine; Simulation; Technologiebasiertes Testen; Interview; Erhebungsinstrument; Evaluation; Student; Rasch-Modell; Empirische Untersuchung;
Abstract: As Internet sources provide information of varying quality, it is an indispensable prerequisite skill to evaluate the relevance and credibility of online information. Based on the assumption that competent individuals can use different properties of information to assess its relevance and credibility, we developed the EVON (evaluation of online information), an interactive computer-based test for university students. The developed instrument consists of eight items that assess the skill to evaluate online information in six languages. Within a simulated search engine environment, students are requested to select the most relevant and credible link for a respective task. To evaluate the developed instrument, we conducted two studies: (1) a pre-study for quality assurance and observing the response process (cognitive interviews of n = 8 students) and (2) a main study aimed at investigating the psychometric properties of the EVON and its relation to other variables (n = 152 students). The results of the pre-study provided first evidence for a theoretically sound test construction with regard to students' item processing behavior. The results of the main study showed acceptable psychometric outcomes for a standardized screening instrument with a small number of items. The item design criteria affected the item difficulty as intended, and students' choice to visit a website had an impact on their task success. Furthermore, the probability of task success was positively predicted by general cognitive performance and reading skill. Although the results uncovered a few weaknesses (e.g., a lack of difficult items), and the efforts of validating the interpretation of EVON outcomes still need to be continued, the overall results speak in favor of a successful test construction and provide first indication that the EVON assesses students' skill in evaluating online information in search engine environments. (DIPF/Orig.)
DIPF-Departments: Bildungsqualität und Evaluation

The NAEP EDM competition. On the value of theory-driven psychometrics and machine learning for […] Zehner, Fabian; Harrison, Scott; Eichmann, Beate; Deribo, Tobias; Bengs, Daniel; Andersen, Nico; […] Book Chapter | Aus: Rafferty, Anna N.; Whitehill, Jacob; Romero, Cristobal; Cavalli-Sforza, Violetta (Hrsg.): Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020) | Worcester; MA: International Educational Data Mining Society | 2020 40325 Endnote: Author(s): Zehner, Fabian; Harrison, Scott; Eichmann, Beate; Deribo, Tobias; Bengs, Daniel; Andersen, Nico; Hahnel, Carolin
Title: The NAEP EDM competition. On the value of theory-driven psychometrics and machine learning for predictions based on log data
In: Rafferty, Anna N.; Whitehill, Jacob; Romero, Cristobal; Cavalli-Sforza, Violetta (Hrsg.): Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020), Worcester; MA: International Educational Data Mining Society, 2020 , S. 302-312
URL: https://educationaldatamining.org/files/conferences/EDM2020/papers/paper_118.pdf
Publication Type: 4. Beiträge in Sammelbänden; Tagungsband/Konferenzbeitrag/Proceedings
Language: Englisch
Keywords: Logdatei; Psychometrie; Theorie; Technologische Entwicklung; Wettbewerb; Prognose; Test; Erfolg
Abstract: The 2nd Annual WPI-UMASS-UPENN EDM Data Minng Challenge required contestants to predict efficient test-taking based on log data. In this paper, we describe our theory-driven and psychometric modeling approach. For feature engineering, we employed the Log-Normal Response Time Model for estimating latent person speed, and the Generalized Partial Credit Model for estimating latent person ability. Additionally, we adopted an n-gram feature approach for event sequences. For training a multi-label classifier, we distinguished inefficient test takers who were going too fast and those who were going too slow, instead of using the provided binary target label. Our best-performing ensemble classifier comprised three sets of low-dimensional classifiers, dominated by test-taker speed. While our classifier reached moderate performance, relative to competition leaderboard, our approach makes two important contributions. First, we show how explainable classifiers could provide meaningful predictions if results can be contextualized to test administrators who wish to intervene or take action. Second, our re-engineering of test scores enabled us to incorporate person ability into the estimation. However, ability was hardly predictive of efficient behavior, leading to the conclusion that the target label's validity needs to be questioned. The paper concludes with tools that are helpful for substantively meaningful log data mining. (DIPF/Orig.)
DIPF-Departments: Bildungsqualität und Evaluation

The role of planning in complex problem solving Eichmann, Beate; Goldhammer, Frank; Greiff, Samuel; Pucite, Liene; Naumann, Johannes Journal Article | In: Computers & Education | 2019 38657 Endnote: Author(s): Eichmann, Beate; Goldhammer, Frank; Greiff, Samuel; Pucite, Liene; Naumann, Johannes
Title: The role of planning in complex problem solving
In: Computers & Education, 128 (2019) , S. 1-12
DOI: 10.1016/j.compedu.2018.08.004
URN: urn:nbn:de:0111-dipfdocs-174087
URL: https://nbn-resolving.org/urn:nbn:de:0111-dipfdocs-174087
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Problemlösen; Planung; Technologiebasiertes Testen; Logdatei; PISA <Programme for International Student Assessment>
Abstract: Complex problem solving (CPS) is a highly transversal competence needed in educational and vocational settings as well as everyday life. The assessment of CPS is often computer-based, and therefore provides data regarding not only the outcome but also the process of CPS. However, research addressing this issue is scarce. In this article we investigated planning activities in the process of complex problem solving. We operationalized planning through three behavioral measures indicating the duration of the longest planning interval, the delay of the longest planning interval and the variance of intervals between each two successive interactions. We found a significant negative average effect for our delay indicator, indicating that early planning in CPS is more beneficial. However, we also found effects depending on task and interaction effects for all three indicators, suggesting that the effects of different planning behaviors on CPS are highly intertwined. (DIPF/Orig.)
DIPF-Departments: Bildungsqualität und Evaluation