13:00h, online talk.

Language: English.

Using diagnostic classification models for examining PISA's Collaborative Problem Solving

Lale Khorramdel

IEA's TIMSS & PIRLS International Study Center, Boston College

Collaborative problem solving (CPS) as the innovative domain in PISA 2015 is defined as a critical skill in education and the workforce where individuals solve problems together by combining their understanding, effort and work (OECD, 2017a; Assessment Framework). The CPS test is based on simulated conversations with computer-based agents. Students are instructed to choose an optimal response from a multiple-choice list as part of the conversation, or they need to perform actions programmed in the unit. The items within units are not discrete. As a result, students have a relatively high reading load when tracking the conversation as it evolves throughout the course of the unit. 

The CPS items assess seven subscales, which address (i) individual processes (exploring and understanding, representing and formulating, planning and executing, monitoring and reflecting) and (ii) collaborative competencies (establishing and maintaining shared understanding, taking appropriate action to solve the problem, establishing and maintaining team organization). These are crossed, forming a matrix of twelve CPS skills. However, as intended, the results based on the item response theory (IRT) scaling in PISA 2015 showed that a unidimensional scale could be achieved (OECD, 2017b; Technical Report).

One aim of the current study is to examine whether unidimensionality still holds when the 117 CPS items are scaled jointly with the 103 items from the PISA domain of Reading Literacy, i.e. that CPS measures more than Reading. Moreover, the study examines whether the different CPS subscales provide diagnostic value in addition to an overall test score. We applied uni- and multidimensional IRT models and Diagnostic Classification Models (von Davier, 2005, 2008; von Davier & Rost, 2016) to PISA 2015 data from six English speaking countries (n = 53,989). Results are discussed with regard to the dimensionality of CPS and with regard to the diagnostic value of the CPS subscales. The talk concludes with an outlook on further plans to include process data for innovative scoring of the CPS items with the goal to increase the information of test scores.

Bio

Lale Khorramdel is an Associate Research Director for Psychometrics and Data Science at the TIMSS & PIRLS International Study Center at Boston College (USA). She earned her PhD in Psychology in 2010 from the University of Vienna, specializing in Psychological Assessment and Applied Psychometrics. She also performed a postgraduate training in work and organizational psychology, clinical and health psychology, as well as work and organizational psychology. 

After working as professor and lecturer for Psychological Assessment and Applied Psychometrics at the University in Vienna (Austria), she joined the Educational Testing Service (ETS, USA) for several years where she was responsible for research and operational work in the area of international large-scale assessments, and leading the PISA and PIAAC psychometrics team. Her research was concerned with the psychometric modeling of cognitive and non-cognitive data, the improvement of test designs, measurement invariance and score comparability, and the implementation of computer-based assessments. For a couple of years, she worked at the National Board of Medical Examiners (NBME, USA) where she was responsible for leading and conducting research related to the measurement of non-cognitive constructs and problem-solving competencies. 

In 2020, she joined the TIMSS & PIRLS International Study Center at Boston College where she returned to her work in the area of international large-scale assessments. She is responsible for leading research that focusses on the use of log and process data from computer-based assessments, as well as on the improvement and extension of psychometric modeling approaches in international large-scale assessment (especially TIMSS and PIRLS). She is also responsible for different projects that develop literacy and numeracy readiness assessments for developing and low-income countries with the goal to establish internationally comparable benchmarks for literacy and numeracy.