Search results in the DIPF database of publications

Your query:

(Personen: "Bengs," und "Daniel")

Simultaneous constrained adaptive item selection for group-based testing Bengs, Daniel; Kröhne, Ulf; Brefeld, Ulf Journal Article | In: Journal of Educational Measurement | 2021 40702 Endnote: Author(s): Bengs, Daniel; Kröhne, Ulf; Brefeld, Ulf
Title: Simultaneous constrained adaptive item selection for group-based testing
In: Journal of Educational Measurement, 58 (2021) 2, S. 236-261
DOI: 10.1111/jedm.12285
URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/jedm.12285
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Adaptives Testen; Aufgabe; Auswahl; Computerunterstütztes Verfahren; Empirische Untersuchung; Gruppe; Leistungsmessung; Modell; Simulation; Technologiebasiertes Testen; Test
Abstract (english): By tailoring test forms to the test‐taker's proficiency, Computerized Adaptive Testing (CAT) enables substantial increases in testing efficiency over fixed forms testing. When used for formative assessment, the alignment of task difficulty with proficiency increases the chance that teachers can derive useful feedback from assessment data. The application of CAT to formative assessment in the classroom, however, is hindered by the large number of different items used for the whole class; the required familiarization with a large number of test items puts a significant burden on teachers. An improved CAT procedure for group‐based testing is presented, which uses simultaneous automated test assembly to impose a limit on the number of items used per group. The proposed linear model for simultaneous adaptive item selection allows for full adaptivity and the accommodation of constraints on test content. The effectiveness of the group‐based CAT is demonstrated with real‐world items in a simulated adaptive test of 3,000 groups of test‐takers, under different assumptions on group composition. Results show that the group‐based CAT maintained the efficiency of CAT, while a reduction in the number of used items by one half to two‐thirds was achieved, depending on the within‐group variance of proficiencies.
DIPF-Departments: Bildungsqualität und Evaluation

Applying psychometric modeling to aid feature engineering in predictive log-data analytics. The […] Zehner, Fabian; Eichmann, Beate; Deribo, Tobias; Harrison, Scott; Bengs, Daniel; Andersen, Nico; […] Journal Article | In: Journal of Educational Data Mining | 2021 41457 Endnote: Author(s): Zehner, Fabian; Eichmann, Beate; Deribo, Tobias; Harrison, Scott; Bengs, Daniel; Andersen, Nico; Hahnel, Carolin
Title: Applying psychometric modeling to aid feature engineering in predictive log-data analytics. The NAEP EDM Competition
In: Journal of Educational Data Mining, 13 (2021) 2, S. 80-107
DOI: 10.5281/zenodo.5275316
URN: urn:nbn:de:0111-dipfdocs-250034
URL: https://nbn-resolving.org/urn:nbn:de:0111-dipfdocs-250034
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Beitrag in Sonderheft
Language: Englisch
Keywords: Psychometrie; Modellierung; Protokoll; Datenanalyse; Testverhalten; Cluster
Abstract (english): The NAEP EDM Competition required participants to predict efficient test-taking behavior based on log data. This paper describes our top-down approach for engineering features by means of psychometric modeling, aiming at machine learning for the predictive classification task. For feature engineering, we employed, among others, the Log-Normal Response Time Model for estimating latent person speed, and the Generalized Partial Credit Model for estimating latent person ability. Additionally, we adopted an n-gram feature approach for event sequences. Furthermore, instead of using the provided binary target label, we distinguished inefficient test takers who were going too fast and those who were going too slow for training a multi-label classifier. Our best-performing ensemble classifier comprised three sets of low-dimensional classifiers, dominated by test-taker speed. While our classifier reached moderate performance, relative to the competition leaderboard, our approach makes two important contributions. First, we show how classifiers that contain features engineered through literature-derived domain knowledge can provide meaningful predictions if results can be contextualized to test administrators who wish to intervene or take action. Second, our re-engineering of test scores enabled us to incorporate person ability into the models. However, ability was hardly predictive of efficient behavior, leading to the conclusion that the target label's validity needs to be questioned. Beyond competition-related findings, we furthermore report a state sequence analysis for demonstrating the viability of the employed tools. The latter yielded four different test-taking types that described distinctive differences between test takers, providing relevant implications for assessment practice. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen

The NAEP EDM competition. On the value of theory-driven psychometrics and machine learning for […] Zehner, Fabian; Harrison, Scott; Eichmann, Beate; Deribo, Tobias; Bengs, Daniel; Andersen, Nico; […] Book Chapter | Aus: Rafferty, Anna N.; Whitehill, Jacob; Romero, Cristobal; Cavalli-Sforza, Violetta (Hrsg.): Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020) | Worcester; MA: International Educational Data Mining Society | 2020 40325 Endnote: Author(s): Zehner, Fabian; Harrison, Scott; Eichmann, Beate; Deribo, Tobias; Bengs, Daniel; Andersen, Nico; Hahnel, Carolin
Title: The NAEP EDM competition. On the value of theory-driven psychometrics and machine learning for predictions based on log data
In: Rafferty, Anna N.; Whitehill, Jacob; Romero, Cristobal; Cavalli-Sforza, Violetta (Hrsg.): Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020), Worcester; MA: International Educational Data Mining Society, 2020 , S. 302-312
URL: https://educationaldatamining.org/files/conferences/EDM2020/papers/paper_118.pdf
Publication Type: 4. Beiträge in Sammelbänden; Tagungsband/Konferenzbeitrag/Proceedings
Language: Englisch
Keywords: Logdatei; Psychometrie; Theorie; Technologische Entwicklung; Wettbewerb; Prognose; Test; Erfolg
Abstract: The 2nd Annual WPI-UMASS-UPENN EDM Data Minng Challenge required contestants to predict efficient test-taking based on log data. In this paper, we describe our theory-driven and psychometric modeling approach. For feature engineering, we employed the Log-Normal Response Time Model for estimating latent person speed, and the Generalized Partial Credit Model for estimating latent person ability. Additionally, we adopted an n-gram feature approach for event sequences. For training a multi-label classifier, we distinguished inefficient test takers who were going too fast and those who were going too slow, instead of using the provided binary target label. Our best-performing ensemble classifier comprised three sets of low-dimensional classifiers, dominated by test-taker speed. While our classifier reached moderate performance, relative to competition leaderboard, our approach makes two important contributions. First, we show how explainable classifiers could provide meaningful predictions if results can be contextualized to test administrators who wish to intervene or take action. Second, our re-engineering of test scores enabled us to incorporate person ability into the estimation. However, ability was hardly predictive of efficient behavior, leading to the conclusion that the target label's validity needs to be questioned. The paper concludes with tools that are helpful for substantively meaningful log data mining. (DIPF/Orig.)
DIPF-Departments: Bildungsqualität und Evaluation

Adaptive item selection under matroid constraints Bengs, Daniel; Brefeld, Ulf; Kröhne, Ulf Journal Article | In: Journal of Computerized Adaptive Testing | 2018 38642 Endnote: Author(s): Bengs, Daniel; Brefeld, Ulf; Kröhne, Ulf
Title: Adaptive item selection under matroid constraints
In: Journal of Computerized Adaptive Testing, 6 (2018) 2, S. 15-36
DOI: 10.7333/1808-0602015
URN: urn:nbn:de:0111-dipfdocs-166953
URL: http://www.dipfdocs.de/volltexte/2020/16695/pdf/JCAT_2018_2_Bengs_Brefeld_Kroehne_Adaptive_item_selection_under_matroid_constraints_A.pdf
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Adaptives Testen; Algorithmus; Computerunterstütztes Verfahren; Itembank; Messverfahren; Technologiebasiertes Testen; Testkonstruktion
Abstract (english): The shadow testing approach (STA; van der Linden & Reese, 1998) is considered the state of the art in constrained item selection for computerized adaptive tests. The present paper shows that certain types of constraints (e.g., bounds on categorical item attributes) induce a matroid on the item bank. This observation is used to devise item selection algorithms that are based on matroid optimization and lead to optimal tests, as the STA does. In particular, a single matroid constraint can be treated optimally by an efficient greedy algorithm that selects the most informative item preserving the integrity of the constraints. A simulation study shows that for applicable constraints, the optimal algorithms realize a decrease in standard error (SE) corresponding to a reduction in test length of up to 10% compared to the maximum priority index (Cheng & Chang, 2009) and up to 30% compared to Kingsbury and Zara's (1991) constrained computerized adaptive testing.
DIPF-Departments: Bildungsqualität und Evaluation

Visualisation of complex question pools Horn, Florian; Schiffner, Daniel; Krömker, Detlef; Bengs, Daniel; Fabriz, Sabine; […] Book Chapter | Aus: Schiffner, Daniel (Hrsg.): Proceedings of DeLFI Workshops 2018 - co-located with 16th e-Learning Conference of the German Computer Society (DeLFI 2018) | Aachen: RWTH | 2018 38893 Endnote: Author(s): Horn, Florian; Schiffner, Daniel; Krömker, Detlef; Bengs, Daniel; Fabriz, Sabine; Goldhammer, Frank; Horz, Holger; Kröhne, Ulf; Libbrecht, Paul; Niemeyer, Jana; Tillmann, Alexander; Wenzel, Franziska
Title: Visualisation of complex question pools
In: Schiffner, Daniel (Hrsg.): Proceedings of DeLFI Workshops 2018 - co-located with 16th e-Learning Conference of the German Computer Society (DeLFI 2018), Aachen: RWTH, 2018 (CEUR Workshop Proceedings), S. 1-8
URL: http://ceur-ws.org/Vol-2250/WS_Pro_paper6.pdf
Publication Type: 4. Beiträge in Sammelwerken; Tagungsband/Konferenzbeitrag/Proceedings
Language: Englisch
Abstract (english): In this paper, we discuss the conceptualisation and implementation of an interactive visualisation for complex question pools. In our case we require a way to organize and interact with a pool, including composition and selection of questions, e.g. for creating a test. We therefore use an ontology, which is a primary dimension of the questions, as a default view. Starting from a user-driven design process, we expand it with filter, search and data display functionality. After completion of the first implementation cycle, we evaluated the visualisation by conducting expert interviews and a formal requirement review. These showed that the visualisation solves some of the issues. To address the remainder, we propose a new version of the visualisation and ways to interact with the question pool. (DIPF/Orig.)
DIPF-Departments: Bildungsqualität und Evaluation; Informationszentrum Bildung

Mining implications from data Boubekki, Ahcène; Bengs, Daniel Book Chapter | Aus: Seidl, Thomas;Hassani, Marwan;Beecks, Christian (Hrsg.): Proceedings of the 16th LWA Workshops: KDML, IR and FGWM | Aachen: RWTH | 2014 35176 Endnote: Author(s): Boubekki, Ahcène; Bengs, Daniel
Title: Mining implications from data
In: Seidl, Thomas;Hassani, Marwan;Beecks, Christian (Hrsg.): Proceedings of the 16th LWA Workshops: KDML, IR and FGWM, Aachen: RWTH, 2014 (CEUR Workshop Proceedings, 1226), S. 205-216
URL: http://ceur-ws.org/Vol-1226/paper32.pdf
Publication Type: 4. Beiträge in Sammelbänden; Tagungsband/Konferenzbeitrag/Proceedings
Language: Englisch
Keywords: Datenanalyse; Itemanalyse; Struktur; Test; Fragebogen; Algorithmus; Experiment; Methode
Abstract: Item Tree Analysis (ITA) can be used to mine deterministic relationships from noisy data. In the educational domain, it has been used to infer descriptions of student knowledge from test responses in order to discover the implications between test items, allowing researchers to gain insight into the structure of the respective knowledge space. Existing approaches to ITA are computationally intense and yield results of limited accuracy, constraining the use of ITA to small datasets. We present work in progress towards an improved method that allows for effcient approximate ITA, enabling the use of ITA on larger data sets. Experimental results show that our method performs comparably to or better than existing approaches. (DIPF/Orig.)
DIPF-Departments: Informationszentrum Bildung

Computer-based adaptive speed tests Bengs, Daniel; Brefeld, Ulf Book Chapter | Aus: Stamper, J.;Pardos, Z.;Mavrikis, M.;McLaren, B. M. (Hrsg.): Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014) | Worcester; MA: International Educational Data Mining Society | 2014 35205 Endnote: Author(s): Bengs, Daniel; Brefeld, Ulf
Title: Computer-based adaptive speed tests
In: Stamper, J.;Pardos, Z.;Mavrikis, M.;McLaren, B. M. (Hrsg.): Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014), Worcester; MA: International Educational Data Mining Society, 2014 , S. 221-224
URL: http://www.educationaldatamining.org/EDM2014/index.php?page=proceedings
Publication Type: 4. Beiträge in Sammelwerken; Tagungsband/Konferenzbeitrag/Proceedings
Language: Englisch
Keywords: Adaptives Testen; Algorithmus; Analyse; Antwort; Computerunterstütztes Verfahren; Empirische Forschung; Konzentration; Leistungstest; Lesefertigkeit; Lesetest; Psychometrie; Test; Theorie; Zeit
Abstract: The assessment of a person's traits is a fundamental problem in human sciences. Compared to traditional paper & pencil tests, computer based assessments not only facilitate data acquisition and processing but also allow for adaptive and personalized tests so that competency levels are assessed with fewer items. We focus on speeded tests and propose a mathematically sound framework in which latent competency skills are represented by belief distributions on compact intervals. Our algorithm updates belief based on directional feedback; adaptation rate and difficulty of the task at hand can be controlled by user-defined parameters. We provide a rigorous theoretical analysis of our approach and report on empirical results on simulated and real world data, including concentration tests and the assessment of reading skills. (DIPF/Orig.)
DIPF-Departments: Informationszentrum Bildung

A learning agent for parameter adaptation in speeded tests Bengs, Daniel; Brefeld, Ulf Book Chapter | Aus: European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD) (Hrsg.): Proceedings of the ECML/PKDD Workshop on Reinforcement Learning from Generalized Feedback: Beyond Numeric Reward (ECML/PKDD 2013) | Prag: EMCL/PKDD | 2013 34022 Endnote: Author(s): Bengs, Daniel; Brefeld, Ulf
Title: A learning agent for parameter adaptation in speeded tests
In: European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML/PKDD) (Hrsg.): Proceedings of the ECML/PKDD Workshop on Reinforcement Learning from Generalized Feedback: Beyond Numeric Reward (ECML/PKDD 2013), Prag: EMCL/PKDD, 2013 , S. 1-13
URL: http://www.kma.informatik.tu-darmstadt.de/fileadmin/user_upload/Group_KMA/kma_publications/posm.pdf
Publication Type: 4. Beiträge in Sammelwerken; Tagungsband/Konferenzbeitrag/Proceedings
Language: Englisch
Keywords: Adaptives Testen; Empirische Untersuchung; Fähigkeit; Kompetenz; Lernen
Abstract: The assessment of a person's traits such as ability is a fundamental problem in human sciences. Compared to traditional paper and pencil tests, computer based assessment not only facilitates data acquisition and processing, but also allows for real-time adaptivity and personalization. By adaptively selecting tasks for each test subject, competency levels can be assessed with fewer items. We focus on assessments of traits that can be measured by determining the shortest time limit allowing a testee to solve simple repetitive tasks (speed tests). Existing approaches for adjusting the time limit are either intrinsically non-adaptive or lack theoretical foundation. By contrast, we propose a mathematically sound framework in which latent competency skills are represented by belief distributions on compact intervals. The algorithm iteratively computes a new difficulty setting, such that the amount of belief that can be updated after feedback has been received is maximized. We rigorously prove a bound on the algorithms' step size paving the way for convergence analysis. Empirical simulations show that our method performs equally well or better than state of the art baselines in a near-realistic scenario simulating testee behaviour under different assumptions.
DIPF-Departments: Informationszentrum Bildung

Adaptive speed tests Bengs, Daniel; Brefeld, Ulf Book Chapter | Aus: KDML (Hrsg.): Proceedings of the German Workshop on Knowledge Discovery and Machine Learning (KDML 2013) | Bamberg: KDML | 2013 34025 Endnote: Author(s): Bengs, Daniel; Brefeld, Ulf
Title: Adaptive speed tests
In: KDML (Hrsg.): Proceedings of the German Workshop on Knowledge Discovery and Machine Learning (KDML 2013), Bamberg: KDML, 2013 , S. 1-4
URL: http://www.kma.informatik.tu-darmstadt.de/fileadmin/user_upload/Group_KMA/kma_publications/speedtest.pdf
Publication Type: 4. Beiträge in Sammelwerken; Tagungsband/Konferenzbeitrag/Proceedings
Language: Englisch
Keywords: Adaptives Testen; Empirische Untersuchung; Fähigkeit; Kompetenz; Lernen
Abstract: The assessment of a person's traits such as ability is a fundamental problem in human sciences. We focus on assessments of traits that can be mea- sured by determining the shortest time limit al- lowing a testee to solve simple repetitive tasks, so-called speed tests. Existing approaches for ad- justing the time limit are either intrinsically non- adaptive or lack theoretical foundation. By con- trast, we propose a mathematically sound frame- work in which latent competency skills are rep- resented by belief distributions on compact inter- vals. The algorithm iteratively computes a new difficulty setting, such that the amount of be- lief that can be updated after feedback has been received is maximized. We provide theoretical analyses and show empirically that our method performs equally well or better than state of the art baselines in a near-realistic scenario.
DIPF-Departments: Informationszentrum Bildung