P318: Using evidence based on the response process to support the validity of inferences from educational measures

Author: Jack Barbera, University of Northern Colorado, USA

Co-Author: Brent Ferrell and Paul Schwartz, University of Northern Colorado, USA; David Wren, Wake Forest University, USA

Date: 8/4/14

Time: 4:00 PM4:20 PM

Room: LOH 164

Related Symposium: S25

Instructors and education researchers often rely on assessment instruments when evaluating students. These instruments are typically multiple-choice content assessments (e.g., concept inventories) or evaluations of affective dimensions (e.g., motivation) using a Likert-type self-report survey. Results from these types of assessments are often used to make inferences about the state of students or the impact of teaching practices. However, how do we know what student responses on these measures really mean? Data collected from all assessment instruments should be supported with evidence based on the response process. Response process validity focuses on evidence that supports the meaning of student responses. This talk will focus on the importance of establishing the response process validity of assessment data using examples from a variety of projects within our research group. Data will be presented from interviews collected with students during completion of items from the Chemistry Concepts Inventory and during the evaluation of items to assess aspects of student motivation. On a larger scale, Rasch analysis data will be used to evaluate the response process validity of over 1000 student responses to the Thermochemistry Concept Inventory.

P62: Design, development and evaluation of the Thermochemistry Concept Inventory

Author: David Wren, Wake Forest University, USA

Co-Author: Jack Barbera, University of Northern Colorado, USA

Date: 8/3/14

Time: 2:05 PM2:25 PM

Room: LOH 164

Related Symposium: S10

Students enter physical chemistry classrooms with varying degrees of conceptual understanding with regard to foundational principles taught in the general chemistry series. Many of these concepts are revisited and built upon during physical chemistry courses. Knowledge of student conceptual understanding before additional instruction can provide educators with valuable information to guide reviews and remediate conceptual misunderstandings. Data collected using appropriate concept inventories have the potential to provide instructors with this valuable information, which can be used as formative assessment. The Thermochemistry Concept Inventory (TCI) is a newly-developed instrument focusing on thermochemical concepts taught in first-semester general chemistry classrooms. This talk will focus on the methodology used in the development of TCI items and evidence for the validity of potential uses and interpretations of TCI data by chemical educators and researchers. Results from both qualitative and quantitative studies will be discussed, including those from the psychometric analysis of a large data set (N = 1331) using the Rasch model. Specifically, evidence for response process validity, structural validity, concurrent validity, and reliability of items will be presented.