Development of a machine learning-based tool to evaluate correct Lewis acid–base model use in written responses to open-ended formative assessment items

Abstract

Acid–base chemistry is a key reaction motif taught in postsecondary organic chemistry courses. More specifically, concepts from the Lewis acid–base model are broadly applicable to understanding mechanistic ideas such as electron density, nucleophilicity, and electrophilicity; thus, the Lewis model is fundamental to explaining an array of reaction mechanisms taught in organic chemistry. Herein, we report the development of a generalized predictive model using machine learning techniques to assess students’ written responses for the correct use of the Lewis acid–base model for a variety (N = 26) of open-ended formative assessment items. These items follow a general framework of prompts that ask: why a compound can act as (i) an acid, (ii) a base, or (iii) both an acid and a base (i.e., amphoteric)? Or, what is happening and why for aqueous proton-transfer reactions and reactions that can only be explained using the Lewis model. Our predictive scoring model was constructed from a large collection of responses (N = 8520) using a machine learning technique, i.e., support vector machine, and subsequently evaluated using a variety of validation procedures resulting in overall 84.5–88.9% accuracies. The predictive model underwent further scrutiny with a set of responses (N = 2162) from different prompts not used in model construction along with a new prompt type: non-aqueous proton-transfer reactions. Model validation with these data achieved 92.7% accuracy. Our results suggest that machine learning techniques can be used to construct generalized predictive models for the evaluation of acid–base reaction mechanisms and their properties. Links to open-access files are provided that allow instructors to conduct their own analyses on written, open-ended formative assessment items to evaluate correct Lewis model use.