Only 24% of Higher Ed’s AI Use Cases Can Define Success. Here Are 6 Lessons From One That Can.
UCF’s Knightbot has handled 585,000 student messages, resolves 85–88% of them without a human, and saves six full-time staff a year. What makes it work isn’t the technology — it’s six disciplines most
When I analyzed 139 AI use cases in higher education, only 24 percent clearly defined a metric for success. Three out of four could not say what “working” was supposed to look like — no outcome, no measurement, no baseline. I have written about that gap. What I had not done until last week was sit down with an institution squarely inside the 24 percent and ask, in detail, how they got there.
Tyler Walsh, Director of the Center for Higher Education Innovation at the University of Central Florida, joined the Use Case Lab alongside Jason Fife of Mainstay to open the box on Knightbot — UCF’s student-support AI. What I expected was a story about technology. What I got was a story about discipline, and one sentence that is the single cleanest statement of the 24 percent principle I have heard from any practitioner.
Walsh’s governing rule: before UCF sends any message, the team asks whether they can track the action they are asking the student to take. If they cannot track it, they question whether they should be sending it at all — because otherwise they will never know whether it worked.
That is the 24 percent, stated as an operating constraint rather than a theory. And it is why UCF can produce numbers most institutions cannot.
What Being in the 24% Actually Looks Like
Most institutions deploying student-support AI can tell you it exists. Walsh can tell you what it does. From Knightbot’s launch in January 2024, UCF tracks:
• Unique users — over 340,000, across both text and web
• Incoming messages — over 585,000 — roughly 650 a day
• Resolved without any human intervention — 85–88%, depending on the year, and improving as the system matures
• Staff time recovered — the equivalent of about six full-time staff a year
• Opt-out rate — 1.6% — roughly 5,000 students out of every user since launch
• Speed — basic questions answered in about seven seconds
• Campaign engagement and lift — in one registration campaign to just over 2,000 students, engaged students with registration holds registered at rates 32% higher than students with holds who didn’t engage — and roughly a week faster (about 6.5 days versus about 15)
• Across repeated campaigns — engaged students have registered 14% to 40% higher than non-engaged students
• Course-level outcomes — DFW rates, grade distribution, and performance in the next course, now three years into a four-year study.
That is not a dashboard. That is an institution that decided, before launch, what it was trying to change and how it would know. Meanwhile, three-quarters of the sector cannot answer the first question.
The Origin: A Gap, Not a Mandate
Knightbot did not begin as an AI initiative. UCF had been using Mainstay since 2019 for a narrow job — financial-aid outreach and drop-for-nonpayment notices. Then, around 2022, student government and student-support staff arrived at the same conclusion independently: there was nowhere for students to get basic questions answered. No one-stop shop. That was the gap.
UCF considered the full range of solutions, and this is the part worth pausing on. A brick-and-mortar one-stop costs staffing and a building, and it is only open when it is open. A website requires maintenance. A chat layer on a tool the institution already had could reach all 70,000-plus students, at any hour, at a fraction of the cost. AI was not the answer they went looking for. It was the option that best fit a problem they had already defined. Design and development ran from mid-2023 to January 2024, when Knightbot launched in earnest.
Today it operates on two layers. The first is the one-stop shop — web and SMS, answering the questions that used to consume three to five minutes of a staff member’s time each. The second, and the one Walsh calls the most powerful, is proactive: identifying students approaching a barrier — a registration hold, a nonpayment drop — and texting them before it becomes a crisis, with a way to raise their hand for help. A third layer, course-level bots, is now running in 13 faculty members’ courses, aimed at gateway math and English, where UCF is studying whether better organization translates into better grades and stronger performance in the next course.
The full Use Case Lab recording — Walsh and Fife, unedited — is below, along with the six lessons. Paid subscribers get the complete session and the full lab archive.
Below, for paid subscribers: the six disciplines that put UCF in the 24 percent — the ones that are transferable to your campus regardless of your budget or your vendor — plus the full recording of the session and the governance templates that operationalize them.