The AI Tutor That Outperformed the Classroom
What Harvard’s Physics RCT and Dartmouth’s NeuroBot Mean for Teaching at Scale
Issue #2
Here’s the tension that defines this moment in higher education: 90% of faculty say AI is weakening critical thinking. Ninety percent. That number keeps me up at night, because it reflects real concerns about how students learn and what education is for. And yet, in the same landscape, a peer-reviewed randomized control trial published in Nature Scientific Reports shows that AI tutors achieved double the learning gains compared to in-class active learning, with higher engagement and motivation.
Both things are true. How do we reconcile this? .
The Evidence: What the Research Actually Shows
The paradox starts to resolve when you look at the details. It’s not AI itself that drives learning gains or learning loss. It’s how AI is deployed.
Notice what’s different about these implementations: they’re not unstructured AI chatbots answering any question in any way. Harvard’s AI tutors were designed with pedagogical intent — deployed to supplement, not replace, active learning, with clear learning outcome alignment. Dartmouth’s NeuroBot uses retrieval-augmented generation (RAG) to ground responses in course materials that faculty have vetted. In both cases, the AI is structured by human expertise. And that structure is where the learning gains come from.
The studies the research community has published show a consistent pattern: well-designed, pedagogically informed AI tutoring produces measurable learning gains. But “well-designed” is the operative phrase. Generic AI assistance doesn’t cut it.
Three More Data Points from the Field
UniDistance Suisse (Switzerland): A semester-long study with 51 psychology students using AI-generated microlearning questions derived from course materials showed an average of 15 percentile point improvement. The AI wasn’t generating original content—it was generating learning activities from course materials. That distinction matters.
Coursera AI Coach: Their AI-powered personalized coaching system supports 1M+ learners globally and shows concrete performance gains: a 9.5% higher quiz pass rate on first attempt and 11.6% more lessons completed per hour. At scale, with 1 million learners, this is evidence of what AI tutoring can do.
Carnegie Mellon University: Researchers developed a methodology that enables teachers to create intelligent tutoring systems in approximately 30 minutes by simply demonstrating problem-solving approaches. This is crucial: CMU’s work shows that faculty don’t need computer science backgrounds to build effective AI tutoring systems. They need structured design thinking and a clear pedagogical goal.
The Numbers: Faculty Concerns vs. Research Evidence
Here’s where the paradox becomes visible:
Faculty are concerned, yet committed to expanding AI use. They sense both the risk and the opportunity.
Who Needs to Be in the Room: A Stakeholder Engagement Map
Deploying AI tutoring at scale isn’t a technology project. It’s a governance and change management project. These conversations need to happen:
The institutions that are getting this right don’t approach AI tutoring as a pilot. They approach it as an implementation that requires governance. That means representation across all seven stakeholder groups, with explicit decision rights and communication protocols.
The Governance Layer That Determines Success or Failure
Before you launch an AI tutoring pilot, your governance framework needs to address five critical areas:
These questions aren’t obstacles. They’re the structure that makes sustainable implementation possible. Institutions that skip them run into faculty resistance, union grievances, and sustainability problems.
The EdGenerative AI Governance Toolkit
Every AI deployment — whether it’s a tutoring system, a chatbot, or a grading platform — eventually surfaces the same question: who governs this? Through my work advising institutions, I developed a policy-neutral, audit-ready governance framework mapped to NIST AI RMF, DOJ Title II, FERPA, HECVAT, and ISO/IEC 42001. It’s built around Ten Core Components that give institutions a clear path from pilot to practice.
For institutions deploying AI tutoring systems like the ones profiled in this issue, three components are especially critical: Component 3 (Risk & Impact Assessment) ensures you evaluate the tutoring tool’s pedagogical claims, data handling, and bias risks before scaling. Component 7 (Human Oversight & Recourse) defines when students can override AI recommendations and how faculty maintain pedagogical authority. And Component 9 (Training & Culture) ensures faculty aren’t just handed a tool — they’re supported in integrating it into their teaching practice.
Your Week: Build the Business Case
Your Longer-Term Strategy: Design a Rigorous Pilot
Your 90-Day Implementation Plan
At the 90-day mark, you’re not looking for a final verdict. You’re looking for evidence of learning and a clear sense of the institutional change required to scale. That’s when the real work begins.
Why This Matters Right Now
The timeline matters. According to a 2026 study by the Brookings Global Task Force, involving over 500 stakeholders across 50 countries: AI risks to learning currently outweigh demonstrated benefits. That doesn’t mean AI in tutoring is bad. It means the unstructured, ungoverned use of AI in tutoring is risky. Structured, pedagogically informed AI tutoring—the kind Harvard and Dartmouth are running—is exactly the evidence institutions need to demonstrate responsible innovation.
But here’s the catch: you can’t build that evidence without running the pilot. And you can’t build sustainable pilots without the governance structure in place first. That’s why the next 90 days are critical. Institutions that start now will have semester-long outcome data by fall. That data becomes your institutional narrative—the difference between being a fast-follower and a leader in responsible AI adoption.
Coming Next: Issue #3
“The Chatbot That Saved 357 Staff Hours” — how the University of Hawaii and Georgia State built AI student support systems that freed human staff to do higher-value work. We’ll break down the economics, the implementation path, and what happens when you get the governance right.
With gratitude,
Dr. Aviva Legatt
Founder, EdGenerative · Affiliated Faculty, University of Pennsylvania · Forbes Contributor
What an interesting read! It just goes to show that AI is a great tool but is going to be a game changer for education. We are going to need to start redefining what it means to learn.