Evaluating the Impact of AI-Generated Prompts on Programming Proficiency in Physics Study Education Program: A Case Study on Computational Physics Tasks

Abstract

This study investigates the use of artificial intelligence (AI)-assisted code generation in a computational physics course for physics education students. The study examines students' ability to generate effective prompts for Pascal code, the quality of the generated code, and the resulting computational outputs. A cohort of 28 students was tasked with solving three critical tasks: numerical differentiation, numerical integration, and root-finding. The students' performance was assessed based on three criteria: prompt generation, Pascal code quality, and output quality. Descriptive statistics show that the mean prompt scores for all topics are close to 1.0, with Integration slightly outperforming other topics. Program scores for Integration were higher (mean = 1.25) compared to Differentiation and Root-Finding, suggesting students performed relatively better in Integration tasks. Output scores were closely aligned with program scores, indicating strong student learning transfer. Correlation analysis revealed high relationships between program and output scores, especially for Integration and Root-Finding, highlighting the students’ ability to translate learning into practical applications. Statistical analysis indicates significant variation in student performance across the three tasks, with notable differences in AI-assisted code generation quality. These findings emphasize the varied impact of AI tools on student proficiency in computational tasks.