Data Scientist, EngThrive Insights - CoreAI

Overview Imagine making AI coding tools more effective for tens of thousands of Microsoft engineers— that’s what you’ll do on the Engineering Thrive Insights team. You’ll join the team behind the data platform, metrics, and insights that measure and improve engineering at Microsoft, spanning developer satisfaction, change velocity, build quality, and AI tool impact. Your primary focus will be the AI Data Flywheel, our platform for collecting signals from engineers using AI coding tools and transforming them into actionable insights and experiments that drive continuous product improvement. In this role, you’ll define success metrics, design and analyze experiments, develop statistical and machine-learning models, and uncover opportunities to improve AI-assisted software development. You’ll build rigorous measurement frameworks, evaluate product changes, and translate complex behavioral and product data into recommendations for engineering and product teams. You’ll deepen your expertise in experimentation, causal inference, developer productivity frameworks such as DORA and SPACE, AI product evaluation, and privacy-conscious analytics—while using the latest AI tools to help shape the next generation of developer experiences at Microsoft scale. Responsibilities • Leverages understanding of data science and business to examine a project and consider factors that can influence final outcomes within a technical area. Evaluates project plan for resources, risks, contingencies, requirements, assumptions, and constraints. Documents key business objectives. Effectively communicates business goals in analytical and technical terms. Consistently shares insights with stakeholders. • Collaborates with immediate team members on defining metrics and frameworks for evaluating somewhat complex AI/large language model (LLM) systems and agentic product performance. Experiments with model before selection (e.g., LLM, prompting approach) for tasks within a product or solution Diagnose basic issues and identifies opportunities to drive improvements within AI/LLM systems within domain area. Prototypes and develops AI models using containerized environments (e.g., AI Azure Foundry, LangChain) to operationalize, build, and validate scalable tools/applications/solutions for use by the immediate team. Develops and deploys systems for basic retrieval-augmented generation workflows, assisting in the selection of data sources, retrieval systems, and generative models to ensure quality output. Assists senior members in assessing and refining LLM systems and models. • Understands where to acquire data necessary for successful completion of the project plan. Utilizes querying, visualization, and reporting techniques to describe acquired data, including format, quantity, identities, and other surface properties. Explores data for key attributes and contributes to the development of data quality report describing results of the task, initial findings, and impact on the project by leveraging appropriate AI tools (e.g., Codex, GitHub Copilot). Collaborates with others to perform data-science experiments using established methodologies, statistics, optimization, and probability theory for general purpose software and statistical packages. Assesses different tools and techniques and selects the appropriate one. Serves as an effective partner in data preparation efforts to Solution Architects, Consultants, and Data Engineers. Adheres to Microsoft's privacy policy related to collecting data, preparing data, and using AI tools (e.g., Codex, GitHub Copilot). Identifies data integrity problems. • Leverages knowledge of modeling and machine learning solutions (e.g., state-of-the-art statistical analysis, causal modeling, reinforcement learning) to identify the best approach to complete objectives. Understands modeling techniques (e.g., validation, hypothesis testing, optimal use of data) and machine learning in order to select the correct approach to prepare data, train and optimize the model, and evaluate the output for statistical and business significance. Understands the risks of data leakage, the bias/variance tradeoff, methodological limitations, etc. Writes all necessary scripts in the appropriate language: T-SQL, U-SQL, KQL, Python, R, etc. Constructs hypotheses, designs controlled experiments, analyzes results using statistical tests, and communicates findings to business stakeholders. Effectively communicates with diverse audiences on data-quality issues and initiatives. Understands operational considerations of model deployment, such as performance, scalability, monitoring, maintenance, integration into engineering production system, stability. Develops operational models that run at scale through partnership with data engineering teams. Understands and keeps up to date on embedding models and vector databases. • Learns and understands the current state of the industry, including knowledge of tools, techniques, strategies, and processes that can be utilized to improve process efficiency and performance. Maintains knowledge of current trends within the discipline. Attends internal research conferences and participates in on-hands training, when appropriate. Actively contributes to the body of thought leadership and intellectual property (IP) best practices. Qualifications Required Qualifications: • Doctorate in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field • OR Master's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 1+ year(s) data-science experience (e.g., managing structured and unstructured data, applying statistical techniques and reporting results) or consulting experience • OR Bachelor's Degree in Data Science, Mathematics, Statistics, Econometrics, Economics, Operations Research, Computer Science, or related field AND 2+ years data-science experie