Jeng-Her Alex Chen
Assistant Professor, Discipline of Business Management, Yuan Ze University
E-mail：alex@saturn.yzu.edu.tw
Abstract
This study investigates decision-making behavior among undergraduate business students engaged in GenAI-assisted data analysis tasks. Using a matched dataset of 161 students (from three class sections) derived from Quiz01 knowledge assessments and W2 survey responses, the study addresses three research questions: (1) Can prior statistical knowledge predict task blunder behavior? (2) Can self-reported Automation Bias Index (ABI) and Critical Thinking Index (CI) predict actual task errors? (3) Does multi-tool cross-verification reduce or increase error rates?
Results address the three research questions as follows: (1) Prior statistical knowledge did not predict task blunders (OR = 1.215, p = .556), with ceiling effects rendering the knowledge test nearly non-discriminatory. (2) Neither ABI (p = .858) nor CI (p = .323) predicted task errors, confirming the co-existence of knowledge–behavior and attitude–behavior gaps. (3) Students using multiple AI tools for cross-verification exhibited a higher error rate (18.9%) than single-tool users (6.5%; OR = 2.839, p = .072), demonstrating a “confirmation bias amplification effect.” Additionally, Gemini users were 3.5 times more likely to commit errors than non-Gemini users (OR = 3.515, p = .045), revealing a tool-specific accuracy confound. These findings challenge core assumptions of current AI literacy education. The study recommends that AI literacy curricula shift from “comparing answers” to “comparing procedures, evidence, and reproducibility”; students should be trained to verify AI outputs using independent computational methods rather than querying multiple AI tools. The proposed TRV (Traceability–Reproducibility–Verifiability) framework offers a directly actionable evaluation rubric for AI-assisted assignments in higher education.
Keywords：automation bias, generative AI, knowledge–behavior gap, multi-tool verification, AI literacy education