I Analyzed 139 AI Use Cases in Higher Ed. Only 24% Could Define Success.

The most creative AI work in the sector and the most accountable AI work in the sector are mostly not the same projects — and that gap is the most important number I have produced all year

For the past several months I have kept a public spreadsheet of AI use cases in higher education. It began as a way to answer a question I kept getting from provosts and CIOs: what does real implementation look like, and who is doing it well? The tracker has grown into one of the more comprehensive open catalogs of its kind, and earlier this month the Chronicle of Higher Education’s Jeff Young went through it and pulled out several use cases that surprised him.

I will share those, because they are genuinely inventive. But the entry that matters most is not any single project. It is a number I produced by analyzing the catalog as a whole — and it is the most clarifying thing I have found all year about why so much AI spending in higher education cannot be defended.

The Number

I analyzed 139 of the use cases in the tracker for one property: does the institution clearly define a metric for success? Not whether the project is good. Not whether it works. Simply whether anyone stated, in advance, what “working” would look like.

Twenty-four percent did. Three out of four AI deployments in higher education cannot tell you what success was supposed to be. No baseline. No target. No defined outcome. Which means that a year after launch, no one can say whether the tool earned its budget line or simply consumed it — because there was never a standard to measure it against.

This is not an AI problem. The tools largely work. It is an evidence problem, and it is the quiet line dividing two kinds of institutions: the ones building durable capacity and the ones accumulating expensive theater. The difference between them is rarely budget or sophistication. It is whether someone asked, before launch, a single question. What is the metric?

The Inventive Ones — and What Separates the Best of Them

First, the use cases worth admiring. These show how wide the aperture of serious AI work in higher education has become.

A “Grammar Laboratory” for deaf students at RIT. Developed by lecturer Erin Finton at the Rochester Institute of Technology’s National Technical Institute for the Deaf, with support from Google.org, the tool takes a bilingual ASL-and-English approach to a real pedagogical problem: ASL has its own grammar, and the gap between it and written English creates genuine learning barriers. The design choice I find most instructive is small and telling — the AI-generated questions are framed as an assignment, not an assessment, and every question includes an “I’m not sure” option that triggers further explanation. Someone thought about what the tool was for, which is exactly the discipline most deployments lack.

An institutional data dashboard at Florida State that unifies 47 databases. FSU’s Snowflake implementation pulls from 47 separate source systems — and growing — so a leader can ask a question like “how many first-year students are at high risk of dropping out based on attendance, mid-term grades, and payment status?” and get an answer in minutes instead of a three-week data request. Note what makes this one a model: FSU reports responding 95 percent faster to stakeholder data needs. That is a defined, measured outcome. It belongs to the 24 percent.

An extension chatbot for farmers and agricultural workers. Trained on 400,000 articles and fact sheets from 30 state extension networks and the U.S. Department of Agriculture, it puts a century of land-grant extension expertise within reach of farmers and gardeners at any hour. It is a reminder that the higher-education mission AI can serve extends well past the classroom — into the public-service obligations that land-grant institutions were built to carry.

Creative, specific, mission-grounded. But notice: the FSU example came with a number, and the other two largely did not surface one in public reporting. That is the pattern across the whole catalog in miniature. Inventiveness is everywhere. Defined success is rare. And the projects that pair the two are the only ones an institution can actually defend at renewal.

Why the Evidence Gap Persists

If defining a success metric were easy and obviously necessary, the number would not be 24 percent. So it is worth being honest about why three out of four projects skip it.

Defining success is harder than launching. A demo is exciting; a baseline is tedious. Writing down what you expect to happen creates the possibility of being wrong, and institutions are not in the habit of inviting that. It is safer, bureaucratically, to deploy something impressive and never specify what it was supposed to accomplish — because then it can never be said to have failed. The absence of a metric is not an oversight. It is, often, a quiet form of risk avoidance. And it is exactly backward, because the project with no defined outcome is the one most exposed when the budget tightens.

This is the Evidence and Accountability pillar of the AI-Ready Institution Framework, and it is consistently the lowest-scoring pillar I see. Institutions can show you activity — pilots, licenses, committees. Far fewer can show you outcomes. The 24 percent figure is what that pillar looks like measured across a large, real sample. It is not a flattering picture, and it is the most useful one a cabinet can start from, because it locates the problem precisely: not too little AI, but too little evidence.

The next Use Case Lab is Tuesday, June 30, noon ET (rescheduled from 6/25) with Tyler Walsh of UCF on Knightbot — a student-support system with exactly the kind of operating evidence most deployments lack. Paid subscribers get the live room and the recording. We open the box on one real deployment together, every month.

Below, for paid subscribers: the three-question test that moves a use case from the 76 percent to the 24 percent, the way to apply it before you sign anything, and the governance templates archive built to make the test enforceable on your campus.