Evaluating NLP Embedding Models for Handling Science-Specific Symbolic Expressions in Student Texts

In recent years, natural language processing (NLP) has become integral to automated assessment practices, particularly in the analysis of student-generated language products. For research and assessment purposes, so-called embedding models are typically employed to generate numeric representations of text that capture its semantic content for use in subsequent quantitative analyses. Yet when it comes to sciencerelated language, symbolic expressions such as equations and formulas introduce challenges that current embedding models struggle to address. Research studies and practical applications often either overlook these challenges or remove symbolic expressions altogether, potentially leading to biased research findings and diminished performance of practical applications. This study therefore explores how contemporary embedding models differ in their capability to process and interpret science-related symbolic expressions. To this end, various embedding models are evaluated using physics-specific symbolic expressions drawn from authentic student responses, with performance assessed via two approaches: (1) similarity-based analyses and (2) integration into a machine learning classification. Our findings reveal significant differences in model performance, with OpenAI’s GPT-text-embedding-3-large outperforming all other examined models, though its advantage over other models was moderate rather than decisive. Overall, this study highlights the importance of carefully selecting NLP embedding models for educational assessment and research involving science-related language that includes symbolic expressions.