What Purdue Built — and What Every Higher Ed Leader Can Learn From It
Use Case Lab Recap: Charlie, AI Writing Feedback, and the Art of Knowing When Your Tool Actually Works
Use Case Lab, Episode #1
Welcome to the first recap of The Use Case Lab — our monthly live conversation series where we open the hood on how universities are operationalizing AI.
For our inaugural episode, I was joined by Ian Pytlarz, Principal Data Scientist at Purdue University’s Institutional Data Analytics + Assessment (IDA+A) team. Ian’s team sits under the Office of the Provost and has an unusually wide mandate: from AI models that saved Purdue $1.27 million annually on campus utility bills, to student-facing learning tools. He’s also co-chair of Purdue’s Data Ethics Committee, which, as you’ll see, makes him exactly the kind of balanced voice this conversation needed.
We spent nearly an hour on one specific use case: Charlie — Purdue’s AI-powered writing feedback tool currently in use in 40 classes, serving approximately 1,400 students. The conversation went well beyond Charlie. It became a masterclass in how to evaluate, build, govern, and scale AI tools in higher ed without making expensive mistakes.
Here’s what you need to know.
What Is Charlie?
Charlie started as a faculty research project by Dr. Lindsey Hamm, who saw a simple but stubborn problem: there’s never enough time to give students feedback on writing drafts before their final grade. Students submit work. They wait. They get a grade. They don’t learn as much as they could.
Her original idea? Use AI to score essays by rubric. That approach mostly fell flat, not because the idea was wrong, but because the execution needed to be refined. Giving a student a score without explanation isn’t helpful. And TAs don’t grade consistently, which confused the underlying models.
When Ian’s team got involved around 2023, they tore the idea down and rebuilt it. The new version shifted entirely away from scoring and toward written feedback — analyzing the student’s draft against the rubric and generating guidance to help them improve, not just judge where they landed.
The next challenge: getting the AI to stop trying to do the assignment for the student.
Getting it to coach rather than complete took significant prompt engineering, iteration, and what Ian describes as a deliberate feedback loop.
The Key Insight: Building a Feedback Loop to Know If It Works
This is where Ian said something I want every higher ed leader to hear:
Most people deploying AI have no idea if their tool is actually working.
Ian’s team built a rigorous evaluation process.
