# Principal Applied Scientist Source: https://realanalystjobs.com/job/principal-applied-scientist-jobgether-7d75cb06 Principal Applied Scientist at Jobgether, an on-site role based in USA. Posted on 13 November 2026, read from Jobgether's own careers page (Lever), which RealAnalystJobs re-checks daily and removes the role when it closes. Updated 15 November 2026. ## Key facts - Company: [Jobgether](https://realanalystjobs.com/companies/jobgether) - Location: USA - Work mode: On-site - Posted: Nov 13, 2026 - Employment type: Full time - Source: Jobgether's careers page (Lever), re-checked daily Skills: [Python](https://realanalystjobs.com/skills/python) ## About this role This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Principal Applied Scientist based in the United States. We are seeking an experienced Principal Applied Scientist to lead the development of a rigorous evaluation framework for enterprise AI agents. In this research-intensive, customer-facing role, you will work closely with engineering teams to assess planner models, execution models, and inference optimization strategies through controlled, statistically sound experiments. You will own both the experimental methodology and the production-quality tools that make evaluation repeatable, from realistic user simulation and isolated experiment execution to statistical analysis and automated insights. Your work will help distinguish meaningful improvements from inconclusive results, quantify the relationship between quality and cost, and translate technical findings into measurable business value. You will also provide technical leadership, establish evaluation standards, and deliver actionable recommendations that support enterprise deployment decisions. Accountabilities • Design and lead an end-to-end evaluation framework for planner-executor agents, covering scenario definition, user simulation, configuration, execution, grading, comparison, and insight generation. • Define scenario schemas, grading specifications, and validation processes in collaboration with engineering teams, ensuring scenarios capture goals, personas, starting states, and expected outcomes. • Develop an LLM-powered user simulator capable of realistic multi-turn conversations, using hidden goals and stop conditions, and validate its behavior against production interaction logs. • Build isolated, parallel experiment execution systems with configurable runs and adapters that route model traffic through existing LLM gateways. • Develop and implement an initial selection of optimization plug-ins, such as failure-based model cascading, cache-aware model switching, and prompt caching. • Instrument model calls and integrate tracing systems to attribute token consumption, cache utilization, latency, and costs to individual requests. • Establish grading methods based primarily on verified end states, with optional LLM-based evaluation of answer and planning quality. • Calculate and report key performance indicators, including verified success rates, cost per successful task, p95 latency, and failures by pipeline stage. • Apply rigorous statistical methods, including paired comparisons, confidence intervals, win/loss/tie analysis, and non-inferiority testing, while clearly identifying inconclusive findings. • Translate experimental results into projected financial impact at production scale and develop cost-versus-quality analyses to guide optimization decisions. • Build an automated insight engine that uses statistical computation for all reported metrics and LLM-generated explanations to communicate evidence-backed findings. • Develop a locally hosted, API-first dashboard with launch, live monitoring, results, comparison, insights, and tracing capabilities, supported by straightforward deployment and integration documentation. • Lead staged experimentation campaigns covering production baselines, model screening, optimization testing, and validation of the strongest combinations. • Assess how well agent performance generalizes across scenario categories, potentially incorporating data analytics benchmark scenarios. • Deliver final recommendations with confidence intervals and projected dollar impact, alongside technical reports, runbooks, and integration guides. • Collaborate with engineers and business leaders to establish evaluation criteria, success thresholds, and acceptable quality margins, communicating findings to both technical and non-technical stakeholders. • Set experimental standards, provide technical leadership, and mentor engineers in rigorous agent evaluation and AI-assisted development practices. Requirements: • Demonstrated experience with LLM and AI agent evaluation methodologies, including state-based verification, LLM-as-judge or agent-as-judge approaches, trajectory-level scoring, and user simulation. • Hands-on experience building or evaluating production-grade planner-executor systems or other multi-step, tool-using AI agent architectures. • Strong expertise in experimental design and statistical inference, including paired testing, non-inferiority analysis, power analysis, confidence intervals, and the ability to recognize when evidence is inconclusive. • Experience with multi-objective optimization or cost-aware machine learning, including Pareto frontier analysis and translating technical results into projected financial savings. • Familiarity with LLM routing, model cascades, model selection, and inference optimization techniques such as prompt and plan caching, context compaction, reasoning budgets, and cache-aware model switching. • Understanding of LLM inference internals and the ability to evaluate the impact of optimization strategies on quality, latency, and cost. • Experience implementing observability and tracing for LLM systems, including request-level attribution of tokens, cache usage, latency, and cost. • Ability to develop production-quality research tooling using Python services, containerized and sandboxed execution environments, and data pipelines. • Previous customer-facing or consulting experience, including scoping and delivering research-oriented technical engagements. • Excellent communication skills, with the ability to explain complex technical findings, respond confidently to detailed questions, and connect results to business outcomes. • Additional advantages include: • Publications or open-source contributions in LLM evaluation, AI agents, or efficient inference. • Experience developing internal dashboards or web applications that can be adopted and maintained by Description from Jobgether's public careers feed, reproduced so you can read the role here. Apply on the company's own site; Real Analyst Jobs never submits anything for you. ## How to apply Open [this role on Real Analyst Jobs](https://realanalystjobs.com/job/principal-applied-scientist-jobgether-7d75cb06) and use Apply now there to reach the employer's application. The listing is re-checked against Jobgether's careers page daily and removed when it closes. ## Jobgether 198 open analyst roles on Real Analyst Jobs. [More roles at Jobgether](https://realanalystjobs.com/companies/jobgether) ## More like this ### Other open roles at Jobgether - [Blockchain Intelligence Analyst, Transnational Crime Organizations](https://realanalystjobs.com/job/blockchain-intelligence-analyst-transnational-crime-organizations-jobgether-a69032ed), France (On-site) - [Blockchain Intelligence Analyst, Transnational Crime Organizations](https://realanalystjobs.com/job/blockchain-intelligence-analyst-transnational-crime-organizations-jobgether-874a2e71), Portugal (On-site) - [Senior Reimbursement Analyst- Regulatory and Reporting](https://realanalystjobs.com/job/senior-reimbursement-analyst-regulatory-and-reporting-jobgether-50b894e7), USA (On-site) - [Blockchain Intelligence Analyst, Transnational Crime Organizations](https://realanalystjobs.com/job/blockchain-intelligence-analyst-transnational-crime-organizations-jobgether-bbb64ea5), UK (On-site) - [Senior Data Business Analyst / Product Owner (Contractor)](https://realanalystjobs.com/job/senior-data-business-analyst-product-owner-contractor-jobgether-228735bb), Germany (On-site) - [Data analyst jobs in the USA](https://realanalystjobs.com/jobs/data-analyst-jobs-in-usa) - [Analyst job market report](https://realanalystjobs.com/analyst-job-market) - [What analyst roles pay](https://realanalystjobs.com/blog/data-analyst-salaries) ## Sources and links - [This role on Real Analyst Jobs](https://realanalystjobs.com/job/principal-applied-scientist-jobgether-7d75cb06) - [Jobgether on Real Analyst Jobs](https://realanalystjobs.com/companies/jobgether) - [All analyst jobs](https://realanalystjobs.com/jobs)