Hire Machine Learning Engineers: Rates, Vetting and Engagement Models

What a machine learning engineer does, and where hiring goes wrong

A machine learning engineer is not a data scientist who codes, and the difference is where most of the money gets wasted.

We see this pattern constantly. A logistics company hires a strong ML person off a portfolio full of Kaggle medals and a clean demand forecasting notebook. Three months in, the model exists. It is even accurate. It also lives in a Jupyter notebook on a laptop, reads a CSV that someone exports from the warehouse system by hand every Monday, and has no way to reach production. The founder asks how to use it. The answer involves rewriting most of it.

The bulk of the job is the 80 percent around the model. Getting features computed the same way at training time and at serving time, so the model does not silently degrade because the training pipeline used a 30 day rolling average and the API used 7. Deciding whether this needs a real time endpoint on FastAPI, TorchServe or vLLM, or whether a nightly batch job writing predictions into Postgres is enough, which it usually is, and the good ones tell you that. Wiring up MLflow or Weights and Biases so you can tell which model version produced which prediction six weeks later when a customer complains. Putting drift detection in place with something like Evidently so you find out the model broke before your customers do. Building the retraining loop and the shadow deployment so a new model gets compared against the old one on live traffic before it takes over.

Here is where hiring goes wrong: companies interview for modeling skill and then hand the person a systems job. You end up paying senior money for someone who can explain gradient boosting beautifully and has never been paged at 2am because a feature pipeline emitted nulls and the model started predicting the mean for every customer. The other direction fails too. A backend engineer who has read the scikit-learn docs will ship you a model with target leakage in the training set that scores 0.97 offline and 0.61 in production, and will not understand why.

Engagement models: what fits

In-house hire. Right when ML is core product, not a feature: recommendation, pricing, fraud, ranking, anything where the model quality is the thing customers pay for. The person accumulates knowledge about your data that is hard to transfer, such as which columns are lies, which timestamps are in the wrong timezone, why 2023 data is unusable. That knowledge is the asset. Wrong when you need one model, once. You hire someone senior, they finish in four months, they get bored and leave, and you own a production system nobody understands.

Freelancer. Good for scoped, well-defined work with a hard edge: build this recommender, tune this pipeline, port this model to ONNX and cut inference cost. Cheapest per hour. The failure mode is specific to ML. These systems do not fail loudly like a broken API. They fail quietly, months later, when the data shifts. A freelancer who is gone by then leaves you with a model nobody can retrain and a feature pipeline only they understood. If you go freelance, make the handover artifact contractual: a retraining runbook someone else has executed, not a README.

Agency. Right when you need the whole thing and you have no ML person to supervise. A real ML delivery is not one skill. It is a data engineer to make the pipelines reliable, an ML engineer for the modeling and serving, and someone who knows Terraform and cloud cost, because a GPU endpoint left running is a four figure surprise. You get all three without hiring three people, and you pay for that. The downside is that an agency can leave you dependent if the engagement ends without a real handover, so treat knowledge transfer as a deliverable with a date on it.

Staff augmentation. The most common right answer, and what Digital Heroes ships most often for this role. You have engineering leadership. You lack the ML specialist. You embed one or two ML engineers into your team, your standups, your repo, your review process. They follow your conventions. When the engagement ends, the code is already in your codebase in your style, not a delivered artifact you now have to absorb. Best when you can technically judge the work but cannot hire fast enough or cannot justify permanent headcount yet.

What it costs

These ranges come from Digital Heroes delivery and from the market we hire in and quote against. They are not survey data, and they move a lot by region, seniority and how specialized the problem is.

Hourly, contract or staff augmentation. Offshore ML engineers in South Asia and Eastern Europe typically land $45 to $90 per hour, with the top of that band being people who have run models in production at scale, not just trained them. Onshore US or Western Europe generally runs $110 to $190 per hour, and $200 plus is normal for people with deep specialization: large scale recommender systems, LLM inference optimization, or anything touching regulated model risk.

In-house, US. Base commonly lands $150,000 to $230,000 for mid to senior in the offers we watch candidates weigh against ours, and higher in the Bay Area or for people coming out of a serious ML org. Base is not the cost. Employer taxes, health coverage, equipment, software seats and general overhead push the real cost well above base, and we model clients at roughly 1.25x to 1.4x base when we help them compare options. Then add recruiting, which is real money whether you pay an agency percentage or burn engineering hours on interviews. Then add ramp, which for ML is longer than for regular backend work because the person has to learn your data before they can model it. Two to four months to genuinely productive is realistic.

The line item people forget. Compute: GPU training, inference hosting, vector database, experiment tracking. A single production ML feature can carry a monthly infra bill that rivals a fraction of the engineer's salary. Ask any candidate what they did to control it. If the answer is nothing, they have never owned a system that mattered.

How to vet a machine learning engineer

Skip the algorithm trivia. Nobody derives backpropagation on the job.

Ask about a model that failed in production. This is the highest signal question available. A real answer sounds like: "our churn model degraded over two months because an upstream team renamed a field and it started coming through as null, our imputation filled it with the median, and accuracy dropped slowly enough that nobody noticed until sales complained." A weak answer talks about overfitting on a validation set. The difference is production scars versus coursework.

Probe the train and serve skew. Ask how they make sure a feature is computed identically in training and at inference. Listen for a real mechanism: a shared transform module imported by both paths, a feature store such as Feast, or an explicit contract test. Listen for whether they mention point-in-time correctness, meaning they know that joining a feature computed after the label event is leakage. This one concept separates people who have shipped from people who have not.

Ask how they chose the baseline. Good ML engineers start with logistic regression, LightGBM or XGBoost and a heuristic, and only reach for deep learning when the simple thing is beaten. Someone who reaches for a transformer on a 40,000 row tabular problem will cost you six months.

Ask what metric they shipped against and why. Not accuracy. Ask about class imbalance handling, the precision versus recall trade-off, and whether the offline metric was connected to a business number. "We optimized AUC" is incomplete. "We optimized recall at a fixed 5 percent false positive rate because ops could only review 200 flags a day" is a person who has talked to a stakeholder.

The take-home. Do not ask them to maximize accuracy on a clean dataset, which tests Kaggle rather than the job. Give them a small, deliberately messy dataset with a time component, and ask for a working prediction service, not a notebook. Judge it on: did they split by time with something like TimeSeriesSplit rather than a random KFold, since a random split on temporal data is leakage and a surprising number of candidates do it; is the preprocessing inside an sklearn Pipeline with a ColumnTransformer fitted only on the training fold, or scattered across cells that leak test statistics; is there a data validation step; is the model artifact versioned; does the serving code load the model once at import rather than per request; is the request body validated with a Pydantic schema; is there a test that catches a schema change. Cap it at four hours and pay for it. If they hand you a notebook when you asked for a service, that is your answer.

Portfolio review. Look for repos with a Dockerfile, a training script separate from the serving script, and a config file. Look for MLflow or DVC. Kaggle rank is not a negative signal, but it is not the signal you need. Ask them to walk you through the one system they built that is still running.

Red flags

Everything is a neural network. If every project in the portfolio is deep learning regardless of data size, you have someone optimizing for their own resume. Ask instead: "What is the simplest model that would have worked here, and why did you not use it?"

No answer on data versioning or model versioning. If they cannot tell you how to reproduce a model from six months ago, they cannot debug one. Ask instead: "A prediction from March looks wrong. Walk me through how you find which model and which data produced it."

Talks about accuracy without mentioning the base rate. A 96 percent accurate fraud model where fraud is 4 percent of transactions is a model that predicts "no fraud" every time. Ask instead: "What was the class balance, and by how much did your model beat the majority-class baseline?"

Random train and test split on time series data. This is the most common competence tell in the discipline, and it inflates offline scores badly. If it shows up in the take-home, that is disqualifying. Ask instead: "How did you split this, and what would change if the data were temporal?"

No opinion on monitoring or retraining. A model without drift detection is a liability with a countdown on it. Ask instead: "How would you know this model got worse, and what triggers a retrain?"

When to hire this role at all, and how Digital Heroes staffs it

Often you should not. If your data lives in five systems and nobody trusts any of them, an ML engineer will spend six months doing data engineering badly. Hire a data engineer first. If you want to add a chat interface or document summarization to your product, that is an application engineer who knows how to call an API and build good retrieval, not an ML engineer, and hiring the latter for the former is expensive and mutually frustrating. If you need dashboards and to know what happened last quarter, that is an analyst. The trigger for a real ML engineer is this: you have data you trust, you have a prediction that would change a decision you make repeatedly and at volume, and the value of getting it right is larger than the cost of maintaining a model forever.

Digital Heroes staffs this as a pod, not a single seat, because a lone ML engineer bottlenecks on pipeline work every time. A typical engagement is one ML engineer on modeling and serving, a data engineer for roughly half the time to make ingestion reliable, and cloud support for the deployment and cost side. We start with a scoped diagnostic on the data before anyone touches a model, because that is where the timeline risk sits. The first milestone is a working prediction path in production, deliberately with a simple model, then quality work once the plumbing is proven. If you eventually want this in-house, the pod runs in your repo with your conventions from day one, so the handover is a conversation and not a migration.