Student Retention and Early Alert Software Problems: The 5 That Cost Real Money, and How to Avoid Them

Why does routing and ownership get underscoped so often?

Vendors sell the model and buyers buy the model, so requirements documents in this category are thick on prediction and thin on what happens after a student is flagged. That imbalance is the single most reliable predictor of a retention project that produces nothing.

The model is not the hard part. A reasonable risk score can be built from a dozen well chosen features and it will be roughly as good as anything sold to you. The hard part is that your advising structure is institution specific in ways a generic product cannot absorb. You may have professional advisers by college, faculty advisers by major after the sophomore year, a separate student support services unit with its own participant list, athletics academic services carrying eligibility obligations, an honours college, and a nursing cohort with its own progression rules. A single student can sit in three of those at once and only one of them should own the outreach.

When routing is underscoped, the build defaults to what is easy, which is assigning alerts to a queue per unit. A queue has no owner, no due date and no disposition, so nothing closes and nothing is measurable. The second failure follows immediately: the same student gets contacted by four units in one week, which is how a well intentioned retention programme annoys students into leaving.

Fix it in scoping. Ask for the routing rule to be written as a rule before any modelling work is quoted, including the case where a financial hold alert on an athlete in week three should reach athletics academic services with the bursar contact attached rather than the general advising queue. Insist that every case carries an owner, a due date, a required outcome disposition and an escalation if untouched, and that a contact suppression rule exists so nobody is the fourth caller that week.

What goes wrong when historic student data is migrated for the model?

A risk model needs several years of outcomes to train on, and this is where institutions discover the true state of their student information system history.

Three problems recur. First, definitional drift. What counts as enrolled, as a major, as full time and as a withdrawal has changed at some point, usually through a policy change or a system upgrade, and a model trained across that boundary learns the boundary. Second, programme reorganisation, where departments merge, majors are renamed and codes are reused, so a naive join produces cohorts that never existed. Third, the pandemic years, where grading policy, attendance and delivery mode all moved at once, so any model treating those terms as ordinary training data will be confidently wrong about current students.

The other trap is case history. If you already run a vendor platform, moving several years of advising notes is a governance question as much as a technical one, because those notes contain judgements written when a narrower audience was assumed.

Treat the history assessment as a discrete, funded piece of work that happens before the model is scoped, with someone holding institutional memory in the room, because the person who knows a code changed meaning is a registrar rather than a data engineer. Exclude periods you cannot reconcile rather than patching them, and record which periods were excluded and why, since that document is the first thing a sceptical faculty member will ask for.

Why do the student information and learning management integrations break after launch?

The signal that actually predicts attrition early is behavioural, which means the learning management system (LMS) feed is load bearing, and load bearing feeds decay.

Canvas offers both a data warehouse export and Caliper Analytics event streams, and both change. A course design change alters what an assignment submission event looks like. A term rolls over and the previous term's courses stop reporting, which is correct behaviour that looks identical to an outage if nobody modelled it. Blackboard and Moodle estates often support only scheduled reports, so a change to the report definition breaks the parse silently. On the student information side, Banner and Colleague extracts break when a field is added or a code list is extended, and card access systems, which frequently carry your best local signal, are usually the least maintained integration on campus.

The symptom is a risk score that quietly goes stale. Nothing errors. Advisers keep working alerts that were computed on last month's engagement, and the drift is invisible until somebody notices a flagged student who withdrew in week five.

Design for decay. Every source gets an expected arrival interval with an alert on silence rather than only on error. Every feed validates a schema contract on each load so a new field or a renamed column raises a specific exception. Add a freshness indicator visible to advisers on every risk score, showing when the underlying engagement data was last updated, because an adviser who can see that the data is four days old makes a better decision than one who assumes it is live. And name an owner on campus for each integration, since a feed nobody owns is a feed nobody notices failing.

What happens when access design and model scrutiny are not covered?

Two governance gaps get deferred and both come back at the worst time.

The first is access design. The Family Educational Rights and Privacy Act permits sharing education records internally with school officials who have a legitimate educational interest, and your entire programme runs on that clause. It has real edges. Card swipe data, counselling contact and health centre visits are not the same category as grades, and combining everything into one screen visible to every adviser is how an institution ends up explaining itself publicly. The design answer is to separate the score from the evidence: the score can be computed from sensitive signals while the evidence shown to a given user is filtered by role, with every view of a student record logged. Counselling records should stay out entirely unless the student consents, and that should be a written decision rather than an assumption.

The second is model scrutiny. A dean will ask why this student is flagged and that one is not, and eventually faculty senate or general counsel will ask what features the score uses. If protected characteristics are doing work in the model, that needs to have been a deliberate decision you can explain. Our position is firm: exclude them from the features and use them only in a fairness audit that checks whether the model performs differently across groups, run every term. That audit is the document that ends the conversation, and building it after the first challenge is far more expensive than building it into release one.

Should you build custom or configure what you already own?

There is a real case for not building. Below roughly 2,000 students, where advisers can name the at risk students without a system, Aviso Retention is reasonable and your learning management system already ships analytics that will surface the obvious cases. Buying is also the right call if you have no data engineering capacity and no plan to acquire any, because custom pipelines with nobody to maintain them become broken pipelines within about eighteen months.

If you already own EAB Navigate360, look hard before replacing it. It handles caseload management competently at scale, and if your gap is one routing rule and two missing signals, extending it beats a new platform on both total cost and political cost. Civitas Learning is built around learning management data and does that well. Watermark Student Success and Engagement is a serious product too.

The build case appears when your advising structure has more than three units with genuinely different ownership rules, when your best signal lives in a system the vendors will not connect to such as card access or a homegrown degree audit, when you have been asked to explain the risk model and could not, or when your team has quietly built a shadow spreadsheet beside the platform you already pay for. That last signal is the clearest in the category. Before deciding, take one real routing rule to your incumbent vendor and ask them to configure it. Their answer decides it.

How do hidden costs get into the quote?

Five items sit outside most retention quotes. The number of advising units with different rules, because five units with five rulebooks is five times the routing logic and quotes are usually written against one. Learning management data access, since Canvas data warehouse access is straightforward and some Blackboard and Moodle estates practically require scheduled reports, which is a different integration entirely.

Third, single sign on and role design, which sounds like plumbing and is actually the governance conversation, and which no supplier can compress. Fourth, historic data cleanup, discussed above, which is often priced as discovery when it is closer to a project. Fifth, outcome measurement, which gets deferred to phase three and then cannot be retrofitted without losing a year of unevaluable interventions.

The defence is a fixed first release with a written exclusion list, plus a decision to include outcome measurement in it regardless of what else gets cut. Your board will eventually ask what the retention programme bought. Decide now whether you will have an answer.

What separates a build that works from one that fails here?

Scope narrow and launch on a term boundary. The builds that work start with first year students only, two signal sources and one advising unit, and go live at the start of a term so week two signals are available from the beginning. The builds that fail launch mid term across every population, which means advisers meet the system during their busiest weeks with alerts computed from partial data.

Run one parallel term. Advisers work the new alerts alongside their existing process for a full term, and that is when the routing rules nobody wrote down finally surface. Every institution has them, and no requirements workshop finds them.

Build outcome measurement first, not last. Record every intervention as an event with a timestamp, an owner and a type, then compare persistence among students with similar risk profiles who did and did not receive it. That is not a randomised trial, but propensity matched comparison on your own population is defensible enough to move budget, and it beats each unit claiming the same successful student.

Finally, own it. The repository, the trained model, the feature pipelines and the cloud accounts should be yours, with the right to hire another firm, written down before kickoff. A model that makes judgements about your students and that you cannot inspect, retrain or move is a liability regardless of how accurate it is this year.