# Machine Learning Engineer, Evals

Source: https://realanalystjobs.com/job/machine-learning-engineer-evals-nous-research-b511f6c8
Machine Learning Engineer, Evals at Nous Research, an on-site role based in Americas (US time zones), USA. Experience level: 3+ Years. Posted on 13 August 2026, read from Nous Research's own careers page (Ashby), which RealAnalystJobs re-checks daily and removes the role when it closes.

Updated 22 November 2026.

## Key facts

- Company: [Nous Research](https://realanalystjobs.com/companies/nous-research)
- Location: Americas (US time zones), USA
- Work mode: On-site
- Posted: Aug 13, 2026
- Employment type: Full time
- Experience level: 3+ Years
- Source: Nous Research's careers page (Ashby), re-checked daily

Skills: RAG, Agents, Fine-tuning, Evals, Prompt engineering, LLM, Docker, [Python](https://realanalystjobs.com/skills/python)

## About this role

The Role

You'll work across the lab on agent capability evals, benchmark design, LLM-as-judge systems, failure analysis, and the infrastructure that ties it together. This is a high-growth, high-ownership role on a small team, and you'll ship evaluation infrastructure that researchers depend on from day one.

Responsibilities

• Run the full eval pipeline end to end and reproduce known results during onboarding, pairing with a senior engineer on your first task

• Build a judge calibration protocol: sample human-labeled decisions, measure agreement (κ, per-class P/R), identify drift zones, and document it so anyone can re-run it

• Extend an existing benchmark (GAIA, τ-Bench, SWE-bench slice, etc.) with new tasks targeting known capability gaps, including the prompt, environment, rubric, automated grader, and QA

• Run failure analysis on model outputs: categorize failure modes, quantify prevalence, and write up findings with recommendations for training data, judge prompts, or benchmark changes

• Own a recurring eval workflow (weekly regression suite, judge drift dashboard, red-team evaluation for a new capability) and ship tooling researchers actually use

Qualifications

• 3+ years in software engineering, ML engineering, data science, or a research-adjacent role, with concrete evaluation experience from coursework, an internship, a side project, open source work, or a job

• Experience with at least one LLM evaluation framework (Harbor, Nemo Evaluator, etc.), with real opinions on what it does well and where it falls short

• Hands-on experience with LLMs: prompting, few-shot design, and ideally fine-tuning or RAG; regular use of coding agents

• Solid Python. You write clean, tested, version-controlled code that a colleague could run without you babysitting it

• Comfort with Git, CI/CD basics, Docker, and the Linux command line (SSH, tmux, debugging a remote job)

• Understanding of basic eval statistics: why accuracy misleads on imbalanced judges, what Cohen's κ measures, how to think about confidence intervals on a metric

• At least 3 of the following: you can explain why LLM-as-judge needs calibration; you've done failure analysis and can tell model bugs apart from prompt, grader, or retrieval issues; you know at least two agent benchmarks (GAIA, AgentBench, τ-Bench, MINT, SWE-bench, WebShop, ALFWorld) and a limitation of each; you've designed or extended an eval dataset with happy paths, edge cases, and adversarial examples; you've thought about non-determinism in eval, how you sample, how many runs, how you report variance

• You communicate clearly to both researchers and engineers, in the right language for each

• You're comfortable with ambiguity, can turn a half-formed request into a plan, and know when to ask for help

Preferred

• RLVR / RLHF pipeline experience

• Training data curation experience

• Distributed eval orchestration experience

• Benchmark design from scratch

• Red teaming and adversarial eval experience

• Familiarity with psychometrics or measurement theory

Description from Nous Research's public careers feed, reproduced so you can read the role here. Apply on the company's own site; Real Analyst Jobs never submits anything for you.

## How to apply

Open [this role on Real Analyst Jobs](https://realanalystjobs.com/job/machine-learning-engineer-evals-nous-research-b511f6c8) and use Apply now there to reach the employer's application. The listing is re-checked against Nous Research's careers page daily and removed when it closes.

## Nous Research

Analyst roles on Real Analyst Jobs. [More roles at Nous Research](https://realanalystjobs.com/companies/nous-research)

## More like this

- [Data analyst jobs in the USA](https://realanalystjobs.com/jobs/data-analyst-jobs-in-usa)
- [Analyst job market report](https://realanalystjobs.com/analyst-job-market)
- [What analyst roles pay](https://realanalystjobs.com/blog/data-analyst-salaries)

## Sources and links

- [This role on Real Analyst Jobs](https://realanalystjobs.com/job/machine-learning-engineer-evals-nous-research-b511f6c8)
- [Nous Research on Real Analyst Jobs](https://realanalystjobs.com/companies/nous-research)
- [All analyst jobs](https://realanalystjobs.com/jobs)
