Principal Data Scientist

About the team At Zillow Group, our mission is to help more people get home. The CoreTech Org delivers on this mission by building the foundational platforms, data, and developer capabilities that power every consumer, partner, and employee experience at Zillow. You will work most closely with the Connections Platform (CP) — the engine that turns consumer interest into a connection with the right partner, owning the conversational experience, the lead funnel and follow-up workflows, and the shared agent services that sit underneath both. Increasingly, that experience is powered by LLM-based and agentic systems: agents that qualify leads, summarize and extract intelligence from conversations for partner teams, and drive proactive outreach. Beyond CP, you will be a measurement partner for the broader CoreTech Org — the foundational platforms every consumer, partner, and employee experience at Zillow is built on — wherever a hard data science problem needs an owner. These are platforms with hundreds of touchpoints and dozens of dependent teams — and they have historically been among the hardest places at Zillow to measure well. CP itself is mid-transformation, shifting from NLP to LLM and agent-based systems, with agents increasingly built and relied on in production but no rigorous way to evaluate them yet. That combination — high dependency, hard-to-measure, and changing fast — is exactly where rigorous measurement creates the most leverage. About the role We are seeking a Principal Data Scientist to be a measurement and impact partner in CoreTech. This is a senior individual-contributor role embedded with the Data & Infrastructure Platform team and who will partner tightly with Product leadership. You will own four outcomes: (1) make the impact of platform work legible — quantifying what shipped, what moved, and what didn’t, in terms leaders can act on; (2) define the metrics that matter for CP and the other platforms you take on — what belongs on a progress dashboard, what is a leading vs. lagging indicator, and what is vanity; (3) build the evaluation layer CP is missing — standing up rigorous, repeatable evals for its LLM and agentic systems (including LLM-as-a-judge) so the team can tell whether an agent actually works, not just that it shipped; and (4) operationalize and instrument those metrics and evals in partnership with engineering — getting the events emitted, the pipelines stable, the dashboards live, and the data trusted. The defining challenge of this role is twofold. The first is measuring causal impact : CP has live, expensive, executive-facing questions that hinge on making a credible impact claim when a clean A/B test isn’t available — conversational and agent-driven flows that span a lead’s journey across sessions and channels, and changes that roll out to everyone at once. The second is establishing an evaluation feedback loop : much of CP now runs on LLM-based and agentic systems that were built by engineers with no rigorous way to measure whether they actually work. You will own how we evaluate these systems — defining the signals that matter, standing up eval and experimentation frameworks (including LLM-as-a-judge), and separating activity and adoption from genuine quality and outcomes. The same question recurs across CoreTech in many different areas, and you will need to help prove whether new features or infrastructure actually pay off, rather than trusting activity or adoption as proof. Owning these questions is the heart of the job. This is a high-leverage, cross-Zillow role: your work will set the measurement bar for platforms every product team at Zillow depends on, and your point of view will directly shape how Data & Infrastructure Platform OKRs are written and reviewed at the executive level. You will: • Define and operationalize the measurement strategy for the Connections Platform and the broader Data & Infrastructure Platform team — owning the throughline from raw events to executive dashboards and causal inference attribution. • Own the evaluation of CP’s LLM-based and agentic systems: design the evals, define the quality signals, and build the frameworks (including LLM-as-a-judge and using tooling such as Braintrust) that tell us whether an agent is actually working. • Own the causal inference and experimentation program for these teams: design and analyze A/B tests where you can, and where you can’t, build the quasi-experimental causal inference design (switchbacks, difference-in-difference, synthetic control, interrupted time-series) that produces a defensible estimate of impact. Know which method the situation calls for, what assumptions it rests on, and make the call for the best method to answer each hard question. • Be the credible authority on “what to measure” and “what to experiment on” for these teams — pushing back on vanity metrics, identifying the leading indicators we’ve been missing, and knowing which signals actually reflect the customer experience. • Own the impact dimension that platform measurement usually lacks — connecting adoption of CP’s agents and other platform investments to business and revenue outcomes, not just activity, speed, and quality signals. • Partner with engineering ICs to instrument the metrics: events, schemas, and pipelines needed to get quality measurements. Engineers will wire up the engine; you make the data infrastructure for these surfaces something teams can trust, and you shape what gets emitted in the first place. • Translate complex measurement realities into clear narratives for VP and C-level partners, including the trade-offs they need to consider. • Set the bar for how data science is practiced inside platform teams at Zillow , treating the platform itself as a product whose adoption, quality, and impact get measured as rigorously as any consumer product (Platform-as-a-Product measurement). • Help decipher the impact of product changes that are multi-dimensional at once – touching consum