-
-
Author(s): Bengs, Daniel; Kröhne, Ulf; Brefeld, Ulf
Title: Simultaneous constrained adaptive item selection for group-based testing
In: Journal of Educational Measurement, 58 (2021) 2, S. 236-261
DOI: 10.1111/jedm.12285
URL: https://onlinelibrary.wiley.com/doi/abs/10.1111/jedm.12285
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Adaptives Testen; Aufgabe; Auswahl; Computerunterstütztes Verfahren; Empirische Untersuchung; Gruppe; Leistungsmessung; Modell; Simulation; Technologiebasiertes Testen; Test
Abstract (english): By tailoring test forms to the test‐taker's proficiency, Computerized Adaptive Testing (CAT) enables substantial increases in testing efficiency over fixed forms testing. When used for formative assessment, the alignment of task difficulty with proficiency increases the chance that teachers can derive useful feedback from assessment data. The application of CAT to formative assessment in the classroom, however, is hindered by the large number of different items used for the whole class; the required familiarization with a large number of test items puts a significant burden on teachers. An improved CAT procedure for group‐based testing is presented, which uses simultaneous automated test assembly to impose a limit on the number of items used per group. The proposed linear model for simultaneous adaptive item selection allows for full adaptivity and the accommodation of constraints on test content. The effectiveness of the group‐based CAT is demonstrated with real‐world items in a simulated adaptive test of 3,000 groups of test‐takers, under different assumptions on group composition. Results show that the group‐based CAT maintained the efficiency of CAT, while a reduction in the number of used items by one half to two‐thirds was achieved, depending on the within‐group variance of proficiencies.
DIPF-Departments: Bildungsqualität und Evaluation
-
-
Author(s): Brandt, Naemi D.; Isreal, Anne; Becker, Michael; Wagner, Jenny
Title: The joint power of personality and motivation dynamics for occupational success. Bridging two largely separated fields
In: European Journal of Personality, 35 (2021) 4, S. 480-509
DOI: 10.1177/0890207021996965
URL: https://journals.sagepub.com/doi/10.1177/0890207021996965
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Arbeitsbedingungen; Berufserfolg; Deutschland; Einflussfaktor; Erwachsener; Erwartung; Faktorenanalyse; Junger Erwachsener; Längsschnittuntersuchung; Latente Wachstumskurvenmodelle; Messverfahren; Motivation; Persönlichkeit; Persönlichkeitsmerkmal; Schulabschluss; Selbstkonzept; Selbstwirksamkeit; Strukturgleichungsmodell; Überzeugung; Veränderung; Zufriedenheit
Abstract (english): When establishing a career in adulthood, two major socioemotional ingredients are expected to affect people's success:
how people act (personality) and what motivates them to act this way (motivation). However, little is known about
whether and how personality and motivation change together and how their possible dynamic interplay predicts success.
We investigated the roles that changes in personality and expectancy beliefs played in explaining occupational success in
4121 participants assessed after high school (Mage ¼ 22.80, SDage ¼ 0.70; 63% female) and about 20 years later. We used
latent change models and moderated structural equation modeling to investigate correlated change and latent change
interactions of personality and expectancy beliefs in predicting success. Results illustrated that besides being related in a
nomological net, personality and expectancy beliefs also illustrated a strong interrelatedness in change across time. We
found the clearest joint change dynamics between emotional stability, conscientiousness, self-concept, and self-efficacy.
Changes in personality and expectancy beliefs were furthermore associated with objective and subjective occupational
success. The results call for a more integrative view on personality-motivation dynamics across time for understanding
the long-term adaptive ingredients of occupational success stories in adulthood.
DIPF-Departments: Struktur und Steuerung des Bildungswesens
-
-
Author(s): Deribo, Tobias; Kröhne, Ulf; Goldhammer, Frank
Title: Model‐based treatment of rapid guessing
In: Journal of Educational Measurement, 58 (2021) 2, S. 281-303
DOI: 10.1111/jedm.12290
URL: https://onlinelibrary.wiley.com/doi/10.1111/jedm.12290?af=R
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Leistungstest; Testkonstruktion; Messverfahren; Computerunterstütztes Verfahren; Frage; Antwort; Verhalten; Dauer; Problemlösen; Modell; Student; Medienkompetenz; Item-Response-Theory; Multiple-Choice-Verfahren; Validität; Panel; Längsschnittuntersuchung
Abstract (english): The increased availability of time-related information as a result of computer-based assessment has enabled new ways to measure test-taking engagement. One of these ways is to distinguish between solution and rapid guessing behavior. Prior research has recommended response-level filtering to deal with rapid guessing. Response-level filtering can lead to parameter bias if rapid guessing depends on the measured trait or (un-)observed covariates. Therefore, a model based on Mislevy and Wu (1996) was applied to investigate the assumption of ignorable missing data underlying response-level filtering. The model allowed us to investigate different approaches to treating response-level filtered responses in a single framework through model parameterization. The study found that lower-ability test-takers tend to rapidly guess more frequently and are more likely to be unable to solve an item they guessed on, indicating a violation of the assumption of ignorable missing data underlying response-level filtering. Further ability estimation seemed sensitive to different approaches to treating response-level filtered responses. Moreover, model-based approaches exhibited better model fit and higher convergent validity evidence compared to more naïve treatments of rapid guessing. The results illustrate the need to thoroughly investigate the assumptions underlying specific treatments of rapid guessing as well as the need for robust methods. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen
-
-
Author(s): Engelhardt, Lena; Naumann, Johannes; Goldhammer, Frank; Frey, Andreas; Horz, Holger; Hartig, Katja; Wenzel, S. Franziska C.
Title: Development and evaluation of a framework for the performance-based testing of ICT skills
In: Frontiers in Education, 6 (2021) , S. 668860
DOI: 10.3389/feduc.2021.668860
URL: https://www.frontiersin.org/articles/10.3389/feduc.2021.668860/full
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Informations- und Kommunikationstechnologie; Praktische Fertigkeit; Wissen; Problemlösen; Textverständnis; Bildverstehen; Bewertung; Modell; Item; Entwicklung; Testvalidität; Itemanalyse; Rasch-Modell; Implementation; Evaluation; Test; Testverhalten; Schüler; Sekundarstufe I; Baden-Württemberg; Rheinland-Pfalz; Deutschland
Abstract (english): This paper addresses the development of performance-based assessment items for ICT skills, skills in dealing with information and communication technologies, a construct which is rather broadly and only operationally defined. Item development followed a construct-driven approach to ensure that test scores could be interpreted as intended. Specifically, ICT-specific knowledge as well as problem-solving and the comprehension of text and graphics were defined as components of ICT skills and cognitive ICT tasks (i.e., accessing, managing, integrating, evaluating, creating). In order to capture the construct in a valid way, design principles for constructing the simulation environment and response format were formulated. To empirically evaluate the very heterogeneous items and detect malfunctioning items, item difficulties were analyzed and behavior-related indicators with item-specific thresholds were developed and applied. The 69 item's difficulty scores from the Rasch model fell within a comparable range for each cognitive task. Process indicators addressing time use and test-taker interactions were used to analyze whether most test-takers executed the intended processes, exhibited disengagement, or got lost among the items. Most items were capable of eliciting the intended behavior; for the few exceptions, conclusions for item revisions were drawn. The results affirm the utility of the proposed framework for developing and implementing performance-based items to assess ICT skills. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen
-
-
Author(s): Heininger, Susanne Katharina; Baumgartner, Maria; Zehner, Fabian; Burgkart, Rainer; Söllner, Nina; Berberat, Pascal O.; Gartmeier, Martin
Title: Measuring hygiene competence. The picture-based situational judgement test HygiKo
In: BMC Medical Education, 21 (2021) , S. 410
DOI: 10.1186/s12909-021-02829-y
URL: https://bmcmededuc.biomedcentral.com/articles/10.1186/s12909-021-02829-y
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Hygiene; Kompetenz; Testverfahren; Gesundheitswesen; Medizin; Student; Arzt; Medizinisches Personal; Situation; Bewertung; Vignette; Item-Response-Theory; Rasch-Modell
Abstract: Background: With the onset of the COVID-19 pandemic at the beginning of 2020, the crucial role of hygiene in healthcare settings has once again become very clear. For diagnostic and for didactic purposes, standardized and reliable tests suitable to assess the competencies involved in "working hygienically" are required. However, existing tests usually use self-report questionnaires, which are suboptimal for this purpose. In the present study, we introduce the newly developed, competence-oriented HygiKo test instrument focusing health-care professionals' hygiene competence and report empirical evidence regarding its psychometric properties.
Methods: HygiKo is a Situational Judgement Test (SJT) to assess hygiene competence. The HygiKo-test consists of twenty pictures (items), each item presents only one unambiguous hygiene lapse. For each item, test respondents are asked (1) whether they recognize a problem in the picture with respect to hygiene guidelines and, (2) if yes, to describe the problem in a short verbal response. Our sample comprised n = 149 health care professionals (79.1 % female; age: M = 26.7 years, SD = 7.3 years) working as clinicians or nurses. The written responses were rated by two independent raters with high agreement (α > 0.80), indicating high reliability of the measurement. We used Item Response Theory (IRT) for further data analysis.
Results: We report IRT analyses that show that the HygiKo-test is suitable to assess hygiene competence and that it allows to distinguish between persons demonstrating different levels of ability for seventeen of the twenty items), especially for the range of low to medium person abilities. Hence, the HygiKo-SJT is suitable to get a reliable and competence-oriented measure for hygiene-competence.
Conclusions: In its present form, the HygiKo-test can be used to assess the hygiene competence of medical students, medical doctors, nurses and trainee nurses in cross-sectional measurements. In order to broaden the difficulty spectrum of the current test, additional test items with higher difficulty should be developed. The Situational Judgement Test designed to assess hygiene competence can be helpful in testing and teaching the ability of working hygienically. Further research for validity is needed. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen
-
-
Author(s): Keyserlingk, Luise von; Dicke, Anna-Lena; Becker, Michael; Eccles, Jacquelynne J.
Title: What matters when? Social and dimensional comparisons in the context of university major choice
In: AERA Open, 7 (2021)
DOI: 10.1177/23328584211020711
URN: urn:nbn:de:0111-pedocs-251726
URL: https://nbn-resolving.org/urn:nbn:de:0111-pedocs-251726
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Schüler; Student; Studienfach; Fächerwahl; Leistung; Vergleich; Standard; Fähigkeit; Selbstkonzept; Mathematik; Deutsch; Sekundarbereich; Schuljahr 12; Universität; Längsschnittuntersuchung; Leistungstest; Messung; Befragung; Datenanalyse; Strukturgleichungsmodell; Mehrebenenanalyse; Empirische Untersuchung; Deutschland
Abstract: Students compare their achievement to different standards in order to evaluate their ability. We built on the theoretical frameworks of situated expectancy-value theory, dimensional comparison theory, and the big-fish-little-pond effect literature to examine the role of social and dimensional comparisons for ability self-concept and subjective task value (STV) in secondary school and university major choice. We used two German longitudinal data sets from different cohorts with data collection in 12th grade and 2 years after high school graduation (Study 1: N = 2,207, Study 2: N = 1,710). Dimensional and social comparisons predicted students' self-concept and domain-specific STV in school: Individual achievement was positively related to ability self-concept and STV in the corresponding domain and negatively related in the noncorresponding domain. School-level mean achievement was negatively related to ability self-concept and STV in the corresponding domain. Dimensional comparisons were directly related to university major choice, social comparisons were only indirectly related. (DIPF/Orig.)
DIPF-Departments: Struktur und Steuerung des Bildungswesens
-
-
Author(s): Köhler, Carmen; Hartig, Johannes; Naumann, Alexander
Title: Detecting instruction effects. Deciding between covariance analytical and change-score approach
In: Educational Psychology Review, 33 (2021) 3, S. 1191-1211
DOI: 10.1007/s10648-020-09590-6
URN: urn:nbn:de:0111-pedocs-252368
URL: https://nbn-resolving.org/urn:nbn:de:0111-pedocs-252368
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Empirische Forschung; Unterricht; Wirkung; Unterrichtsforschung; Methode; Messverfahren; Lernerfolg; Variable; Prognose; Schüler; Unterstützung; Selbstwirksamkeit; Lehrer; Forschungsdesign; Datenanalyse; Modell
Abstract: The article focuses on estimating effects in nonrandomized studies with two outcome measurement occasions and one predictor variable. Given such a design, the analysis approach can be to include the measurement at the previous time point as a predictor in the regression model (ANCOVA), or to predict the change-score of the outcome variable (CHANGE). Researchers demonstrated that both approaches can result in different conclusions regarding the reported effect. Current recommendations on when to apply which approach are, in part, contradictory. In addition, they lack direct reference to the educational and instructional research contexts, since they do not consider latent variable models in which variables are measured without measurement error. This contribution assists researchers in making decisions regarding their analysis model. Using an underlying hypothetical data-generating model, we identify for which kind of data-generating scenario (i.e., under which assumptions) the defined true effect equals the estimated regression coefficients of the ANCOVA and the CHANGE approach. We give empirical examples from instructional research and discuss which approach is more appropriate, respectively. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen
-
-
Author(s): Kolleck, Nina; Schuster, Johannes; Hartmann, Ulrike; Gräsel, Cornelia
Title: Teachers' professional collaboration and trust relationships. An inferential social network analysis of teacher teams
In: Research in Education, 111 (2021) 1, S. 89-107
DOI: 10.1177/00345237211031585
URL: https://journals.sagepub.com/doi/10.1177/00345237211031585
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Beitrag in Sonderheft
Language: Englisch
Keywords: Lehrer; Kooperation; Teamarbeit; Unterricht; Zwischenmenschliche Beziehung; Vertrauen; Professionalisierung; Soziales Netzwerk; Analyse; Zufall; Modell
Abstract (english): In recent years, teachers around the world have been increasingly confronted with various expectations concerning the improvement of their classroom practices and school activities. One factor widely acknowledged to facilitate school and classroom improvement is a strong collaborative culture among teachers. As such, teachers are expected to work in teacher teams, to collaborate closely with colleagues, to co-construct classroom practices, and thus to strengthen trust relationships within the team. A growing number of researchers has analyzed how teachers address these expectations. They suggest that there is a link between teachers' embeddedness in collaboration networks and teachers' trust relationships. The present study seeks to contribute to the research literature by presenting results of Social Network Analyses (SNA) and exponential random graph models (ERGMs) on teacher collaboration in nine secondary schools in Germany (N = 366 teachers). We investigate how the involvement of teachers in co-constructive collaboration in schools, measured by the amount of team teaching (TT), relates to teachers' trust levels. Results of our analyses suggest that a high amount of TT is not necessarily related to a higher degree of trust among teachers at the school level. However, a high involvement of teachers in TT is related positively to their being perceived as trustworthy. Furthermore, the emergence of trust relations in teacher networks depends on general network characteristics, such as homophily, reciprocity and transitivity. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen
-
-
Author(s): Kolovou, Dimitra; Naumann, Alexander; Hochweber, Jan; Praetorius, Anna-Katharina
Title: Content-specificity of teachers' judgment accuracy regarding students' academic achievement
In: Teaching and Teacher Education, 100 (2021) , S. 103298
DOI: 10.1016/j.tate.2021.103298
URN: urn:nbn:de:0111-pedocs-238937
URL: https://nbn-resolving.org/urn:nbn:de:0111-pedocs-238937
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Englisch
Keywords: Schülerleistung; Lehrer; Bewertung; Genauigkeit; Konsistenz <Psy>; Unterrichtsfach; Bildungsinhalt; Mathematik; Deutsch; Test; Mehrebenenanalyse; Multivariate Analyse; Modell; Sekundarstufe I; Schuljahr 07; Empirische Untersuchung; Zürich; Kanton; Schweiz
Abstract: Teachers' accuracy in judging students' achievement is often assumed to be a general ability of teachers. Based on this assumption, teachers should be at least consistent in their accuracy across different content domains within a school subject. Yet, this assumption has rarely been investigated empirically so far. Data from 54 mathematics teachers (N = 1170 students) and 55 language teachers (N = 1255 students) were analysed using a Bayesian multivariate multilevel modelling approach. Results indicate that latent accuracy measures across content domains indeed are substantially correlated within both investigated subjects, but may still be considered to represent different dimensions. (DIPF/Orig.)
DIPF-Departments: Lehr und Lernqualität in Bildungseinrichtungen
-
-
Author(s): Kraus, Thorben; Weishaupt, Horst; Hosenfeld, Ingmar
Title: Segregierte Schulmilieus, variierende Unterrichtsbedingungen und Lernleistungen der Schülerinnen und Schüler. Eine Analyse mit Daten der Grundschulen in Rheinland-Pfalz 2015/16
In: Zeitschrift für Grundschulforschung, 14 (2021) 1, S. 129-148
DOI: 10.1007/s42278-020-00102-7
URL: https://link.springer.com/article/10.1007%2Fs42278-020-00102-7
Publication Type: 3a. Beiträge in begutachteten Zeitschriften; Aufsatz (keine besondere Kategorie)
Language: Deutsch
Keywords: Bedingung; Deutschland; Grundschule; Korrelation; Lernen; Lernstandserhebung; Leseverstehen; Mathematische Kompetenz; Migrationshintergrund; Milieu; Modellbildung; Papier-Bleistift-Test; Personal; Rheinland-Pfalz; Schule; Schüler; Schülerleistung; Schulorganisation; Schulstatistik; Segregation; Sonderpädagogischer Förderbedarf; Soziale Benachteiligung; Sprache; Unterricht
Abstract: Auf der Grundlage der VERA3-Daten der Grundschulen in Rheinland-Pfalz wird der Zusammenhang zwischen der sozialen Zusammensetzung der Schülerinnen und Schüler der einzelnen Schulen und den durchschnittlichen Leistungen im Leseverstehen und in Mathematik analysiert. Besonders interessiert der vermittelnde Einfluss der schulstatistisch erfassten schul- und unterrichtsorganisatorischen Bedingungen, weil sie - im Gegensatz zu der familiären Situation der Schülerinnen und Schüler - schulpolitisch beeinflusst werden können. Neben deskriptiven Befunden wird ein Pfadmodell berechnet, in das die schulorganisatorischen Variablen als vermittelnde Prozessvariablen des Zusammenhangs zwischen Merkmalen der Schülerschaft und den gemessenen durchschnittlichen Leistungen enthalten sind. (DIPF/Orig.)
Abstract (english): Drawing on data from the VERA3 study of primary schools in Rhineland-Palatinate, the authors analyze the correlation between social compositions of students in individual schools and average reading and mathematics achievement. Particular attention is paid to the mediating influence of school statistically relevant conditions of school and instructional organization, which - other than the family background - can be influenced by educational policy-making. Alongside presenting descriptive findings, the authors calculate a path model which contains the school organizational variables as mediating process variables of the correlation between student population variables and measured average achievement. (DIPF/Orig.)
DIPF-Departments: Struktur und Steuerung des Bildungswesens