Unlocking Tech Skills Potential: AI assistance boosts completion rates by 2x and grades by 15%
New research to be presented at the 2025 American Society of Engineering Education (ASEE) Annual Conference in Montreal reveals that an Artificial Intelligence-powered learning assistant can have a dramatic impact on student success in tech skills education. In one of the largest studies of its kind, courses that integrated an AI assistant saw a 15% increase in median grade and a x2 uplift to the number of completed courses compared to those without.
The study, which analyzed engagement and performance data from over 1,800 learners in online computer science courses, points to the growing role of AI in helping students overcome barriers to successful learning outcomes and build confidence in complex subjects like programming.
Among the assistant types, “Explain this error” was by far the most widely used, accounting for 56% of all interactions. This module provides students with plain-language explanations of compiler or runtime errors – a common pain point in early-stage programming education. It was followed by the hint generation module (27%), which offers Socratic-style prompts to help guide problem-solving, and the assignment summarization tool (17%), which simplifies complex task instructions.
“These findings are an important proof point that AI, when thoughtfully designed and embedded in the learning experience, can drive real improvements in outcomes,” said Philip Snalune, CEO and Co-Founder of Codio. “What’s exciting is not just the statistical uplift, but what it tells us about the potential for AI to support more impactful and scalable tech skills education.”
The research focused on Codio Coach, a specialized AI assistant built on Anthropic’s Claude large language model (LLM) and tailored for computer science learners. Rather than giving answers, Coach is designed to guide students with Socratic-style hints, plain-language explanations of error messages, and task summaries that help demystify assignment briefs. Usage was highest in beginner and intermediate courses, with error explanation accounting for over half of all interactions.
The findings also highlight an evolution in how generative AI is being applied in education beyond content generation, bringing context-aware support to improve learner comprehension and reduce frustration without compromising academic integrity.
Key findings include:
- 15% Median Grade Improvement: Learners in courses using Codio Coach achieved a 15% higher median grade compared to those without AI assistance, with average grades also rising by 12%.
- Doubled Completion Rates: 2x course completion rates as part of a longer form review of research data (insight from an extended data set post the ASEE submission window).
- Error Support Most Valued: The most-used feature (56% of all interactions) was the “Explain this error” assistant, confirming that AI support is most impactful when helping students interpret and overcome compiler and runtime errors.
- High Effectiveness in Intro and Intermediate Courses: Usage was highest in beginner and intermediate programming courses, suggesting AI assistants may be especially valuable in reducing early attrition and supporting foundational learning.
- Immediate and Sustained Uptake: Within three months of deployment, Coach logged over 9,000 usage events across 39 courses, used by more than 1,800 unique learners, indicating strong organic engagement.
Codio Coach was first launched in April 2024 and has since been adopted by institutions and training providers across both higher education and the workforce upskilling sector. The study data was drawn from Codio-hosted online courses between January 2023 and December 2024 and analyzed performance differences across 60 courses before and after Coach’s introduction.
The full research paper, Investigating the Impact of Codio Coach: A Specialized AI Learning Assistant on Computing Student Engagement and Performance, will be presented during the ASEE conference’s Computing and Information Technology Division sessions.
