P864: Analysis of students’ self-efficacy, interest, and effort beliefs in general chemistry

Author: Brent Ferrell, University of Northern Colorado, USA

Co-Author: Jack Barbera, University of Northern Colorado, USA

Date: 8/6/14

Time: 2:45 PM3:05 PM

Room: MAK B1138

Related Symposium: S59

Research in academic motivation has highlighted a number of salient constructs, which are predictive of positive learning strategies and academic success. Most of this research has centered on college-level social sciences, or secondary school student populations. The purpose of this study was to adapt existing measures of personal interest, effort beliefs, and self-efficacy for a college chemistry context and put them together into one instrument. The instrument was administered at the beginning and end of the fall semester to 294 students enrolled in a first-semester general chemistry course. Confirmatory factor analysis (CFA) was conducted on each subscale and at each time point to assess the degree to which the proposed model fit the data. The parameter estimates and fit indices from the CFAs together with qualitative data from student interviews were used to investigate modifications to the original subscale items. These results led to a decrease in the total number of items on the instrument from 24 to 19. The attenuated sub-scales showed adequate to good model fit, with all fit indices within acceptable ranges. Furthermore, as evidence of concurrent validity, chemistry majors reported higher self-efficacy and interest than other majors. Cronbach’s alpha estimates ranged from 0.76 to 0.91 for the individual sub-scales. With continued use and further validation, this instrument could be a useful tool for assessing general chemistry students’ motivation and the motivational impacts of various teaching practices.

P723: Evaluation of a diagnostic assessment for incoming organic chemistry students

Author: Victor Kiryak-Klein, University of Northern Colorado, USA

Co-Author: Jack Barbera, University of Northern Colorado, USA

Date: 8/6/14

Time: 9:35 AM9:55 AM

Room: MAK BLL 126

Related Symposium: S53

Prerequisites play an essential role in many curricula including the organic chemistry sequence. General chemistry, the primary prerequisite for most courses in organic chemistry, is deemed to help students build a solid foundation of the chemistry concepts needed for their further chemistry classes. However, as performance in general chemistry varies and since many students fulfill their general chemistry prerequisite within various time frames and often at different institutions before enrolling in organic chemistry, the question of students’ residual general chemistry knowledge remains open. Based on previous research, an assessment instrument has been developed to address students’ preparedness for organic chemistry class and to assess their residual general chemistry knowledge when they start organic chemistry. This presentation discusses the initial phases of evaluation for the developed instrument. A brief outline of the test development will be presented in addition to more in-depth discussion of item and test functioning through the prism of Classical Test Theory.

P638: Using eye-tracking studies to evaluate student responses to multiple-choice items

Author: Travis Knowles, University of Northern Colorado, USA

Co-Author: Jack Barbera, University of Northern Colorado, USA

Date: 8/5/14

Time: 5:15 PM6:30 PM

When using multiple-choice items in assessing students on their chemistry content knowledge, there are a variety of reasons for why students may select a particular response option. While the intention of an assessment item is that students’ responses are a valid measure of their content knowledge, there are several test taking issues that can reduce the validity of this inference. Several validity issues may arise based on the format of the item alone. Prior studies have investigated issues of option order effects and how the order of the response options can alter the difficulty of the items. Other studies have probed how the use of particulate-level images, data tables, and graphs or figures can influence student responses. Prior studies of these aspects have focused only on how student performance changes as an item’s structure is altered. This study plans to use a combination of eye-tracking and student interviews to obtain further evidence of the effects of various item features. A variety of research questions and sample items will be presented and discussed. The goal of the studies will be to obtain novel data about how students’ complete multiple-choice items and how various item characteristics might impact the inferences derived from item results.

P474: X-ray of the GOB chemistry course

Author: Corina E. Brown and Richard Hyslop, University of Northern Colorado, USA

Co-Author: Jack Barbera, University of Northern Colorado, USA

Date: 8/5/14

Time: 10:15 AM10:35 AM

Room: LOH 164

Related Symposium: S41

As the General, Organic, and Biological (GOB) chemistry course is designed to serve allied-health majors, the course content must incorporate expectations across disciplines. During the past decade, the School of Nursing at the University of Northern Colorado has requested significant changes to the credit load of the GOB course. Over this time it has been reduced from a two-semester (8-credit hour lecture/2-credit hour lab) course to a one-semester (3-credit hour/1-credit hour lab) course. With this vast reduction in the number of hours for the course, it became vital to carefully consider which topics would be retained and which topics would be removed. The new one-semester course has been designed utilizing the topics considered important or foundational by GOB instructors and nurse educators. This information was gathered through a qualitative study (N =14 ) and evaluated for generalizability by a national survey (N = 280). The results of the national survey and the course outline, with appropriate topics and experiments, will be presented.

P318: Using evidence based on the response process to support the validity of inferences from educational measures

Author: Jack Barbera, University of Northern Colorado, USA

Co-Author: Brent Ferrell and Paul Schwartz, University of Northern Colorado, USA; David Wren, Wake Forest University, USA

Date: 8/4/14

Time: 4:00 PM4:20 PM

Room: LOH 164

Related Symposium: S25

Instructors and education researchers often rely on assessment instruments when evaluating students. These instruments are typically multiple-choice content assessments (e.g., concept inventories) or evaluations of affective dimensions (e.g., motivation) using a Likert-type self-report survey. Results from these types of assessments are often used to make inferences about the state of students or the impact of teaching practices. However, how do we know what student responses on these measures really mean? Data collected from all assessment instruments should be supported with evidence based on the response process. Response process validity focuses on evidence that supports the meaning of student responses. This talk will focus on the importance of establishing the response process validity of assessment data using examples from a variety of projects within our research group. Data will be presented from interviews collected with students during completion of items from the Chemistry Concepts Inventory and during the evaluation of items to assess aspects of student motivation. On a larger scale, Rasch analysis data will be used to evaluate the response process validity of over 1000 student responses to the Thermochemistry Concept Inventory.

P62: Design, development and evaluation of the Thermochemistry Concept Inventory

Author: David Wren, Wake Forest University, USA

Co-Author: Jack Barbera, University of Northern Colorado, USA

Date: 8/3/14

Time: 2:05 PM2:25 PM

Room: LOH 164

Related Symposium: S10

Students enter physical chemistry classrooms with varying degrees of conceptual understanding with regard to foundational principles taught in the general chemistry series. Many of these concepts are revisited and built upon during physical chemistry courses. Knowledge of student conceptual understanding before additional instruction can provide educators with valuable information to guide reviews and remediate conceptual misunderstandings. Data collected using appropriate concept inventories have the potential to provide instructors with this valuable information, which can be used as formative assessment. The Thermochemistry Concept Inventory (TCI) is a newly-developed instrument focusing on thermochemical concepts taught in first-semester general chemistry classrooms. This talk will focus on the methodology used in the development of TCI items and evidence for the validity of potential uses and interpretations of TCI data by chemical educators and researchers. Results from both qualitative and quantitative studies will be discussed, including those from the psychometric analysis of a large data set (N = 1331) using the Rasch model. Specifically, evidence for response process validity, structural validity, concurrent validity, and reliability of items will be presented.