LM Fine-Tuning & Deployment — "Will You Get In?"
Fine-tuning a language model to predict graduate admissions — and the checks that stopped a respectable-looking accuracy from hiding a broken model.
The problem
Modules 2 through 12 turned roughly 100,000 self-reported Grad Café results into a clean, analyzed dataset. The open question was whether a pretrained language model could read an applicant's details as text and predict the outcome better than the obvious shortcuts — and whether I could deploy it somewhere a person could actually use it.
Approach
- Serialize each applicant into a single text “card” — program, university, degree, term, citizenship, GPA, three GRE scores, free-text comments — using one template that the model reads the same way every time.
- Fine-tune DistilBERT for binary classification (Accepted / Rejected) at sequence length 256 for three epochs, checkpointing on best validation F1 rather than the last epoch.
- Rebuild the modeling dataset from the raw scrape with a written ledger, recovering degree types that earlier cleaning had discarded (77,443 → 79,798 rows).
- Serve the saved model behind a Flask page so anyone can enter their own numbers and get a prediction.
Challenges
Training and serving could silently disagree. If the web form ever formatted an applicant even slightly differently than training did, the model would score inputs it had never really seen — and nothing would error.
How I solved itOne module, serialize.py, owns the applicant-to-text template and is imported by both the training pipeline and the web app. There is no second copy to drift, so a change to the format changes both sides at once.
A headline accuracy can hide a collapse. A model that learned nothing but “PhD applicants get rejected” could still post a respectable-looking number.
How I solved itEvaluation breaks the held-out test set down by degree and reports acceptance rate, accuracy and recall for every group of at least 20 — so a collapse in either direction is visible instead of averaged away.
Duplicate applicant cards could inflate the score by letting the model recognize rows it had already memorized.
How I solved itScored the test set split into cards seen during training and cards never seen. Unseen accuracy came in at 0.770 against 0.796 overall — close enough to show the model generalizes rather than recalls.
Comparing against the previous module's neural network was not apples to apples: it kept only two degree types and a different row count.
How I solved itReported both numbers with the majority-class baseline beside them and stated plainly why they are not directly comparable, rather than claiming a clean win.
serialize.py — the module both sides import
Training and serving disagreeing about input format is a silent failure: the model scores cards it never saw in training and nothing raises. This file is the fix. One template, one formatter, imported by the training pipeline and by the web app, so a change to the format changes both sides at once and drift is not expressible.
# Structured fields precede free text so that if the tokenizer's max
# length is ever exceeded, truncation trims comment tails, not facts.
TEMPLATE = (
"Program: {program}\n"
"University: {university}\n"
"Degree: {degree}\n"
"Term: {term}\n"
"Citizenship: {us_or_international}\n"
"GPA: {gpa}\n"
"GRE Quant: {gre}\n"
"GRE Verbal: {gre_v}\n"
"GRE AW: {gre_aw}\n"
"Comments: {comments}"
)
# Every spelling of "missing" found in the scrape or in a blank form.
_MISSING = {"", "none", "nan", "<na>", "nat", "unknown", "null"}
def to_card(record: dict) -> str:
"""Render one applicant as the text the model reads.
Imported by train_model.py and by inference.py. The outcome label
never appears: the model sees only what an applicant could
truthfully supply before knowing their result.
"""
fields = {f: _fmt(record.get(f), numeric=f in NUMERIC_FIELDS)
for f in ALL_FIELDS}
return TEMPLATE.format(**fields)The repository is private — publishing full solutions to a course still being taught would not be fair to the students taking it. This excerpt is the load-bearing part.
What it looks like
What I took from it
The modeling was the short part. Most of the work was building the checks that could prove the result was real — the per-degree breakdown, the seen/unseen split, the duplicate ceiling. I would rather report 0.796 that I can defend than a higher number I cannot explain.