About this role
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a GenAI Platform Engineer – LLM Gateway based in India.
We are seeking an experienced GenAI Platform Engineer to help build, operate, and continuously improve an enterprise-scale generative AI platform. In this role, you will contribute to the development of an LLM Gateway that provides secure, scalable, and governed access to multiple large language model providers. You will design and maintain backend services, model integrations, routing capabilities, and usage management features while supporting advanced agentic AI solutions. Working within a globally distributed engineering team, you will focus on platform reliability, security, performance, and automation. You will also contribute to open-source technologies, developer enablement, and innovative AI tooling. This is an exciting individual contributor opportunity to shape the future of enterprise GenAI infrastructure in a fast-evolving technological environment.
Accountabilities:
• Design, develop, and operate enterprise LLM Gateway features, primarily using Python, enabling secure and scalable access to multiple large language model providers.
• Build and maintain multi-provider model integrations, intelligent routing, failover mechanisms, API key management, and access control capabilities.
• Develop platform features for usage metering, budget management, cost transparency, and chargeback reporting.
• Integrate agentic AI capabilities into the platform while ensuring appropriate security and governance controls.
• Develop and maintain Model Context Protocol (MCP) servers, cross-provider tool calling, and agent-to-agent connectivity.
• Maintain and enhance the platform's open-source foundations by assessing upstream releases, managing custom extensions, and delivering upgrades without introducing regressions.
• Contribute to administrative interfaces, self-service portals, developer experiences, and technical documentation.
• Build and maintain automated quality gates within CI/CD pipelines, including unit, integration, contract, and performance testing.
• Implement observability solutions using metrics, distributed tracing, and logging tools such as OpenTelemetry and Langfuse.
• Monitor platform performance and proactively identify and resolve latency, reliability, error-rate, and infrastructure cost issues.
• Participate in production support and on-call rotations, leading root-cause investigations and resolving complex incidents.
• Conduct blameless postmortems and translate incident findings into sustainable engineering improvements.
• Apply secure-by-default engineering practices, including Single Sign-On (SSO), OAuth 2.0/OIDC, Role-Based Access Control (RBAC), and secure secrets management.
• Maintain dependency security, version pinning, and software supply-chain hygiene.
• Evaluate emerging generative AI technologies, including model routing solutions, guardrails, agent registries, and agentic frameworks, and provide recommendations for adoption.
• Support internal engineering teams in onboarding and effectively using the platform through technical guides, runbooks, code samples, and reusable AI agent skills.
• Contribute to InnerSource initiatives, participate in code reviews, and collaborate with open-source communities and technology vendors.
• Identify opportunities to improve platform capabilities through upstream contributions, technical experimentation, and continuous innovation.
Requirements:
• Bachelor's or Master's degree in Computer Science, Computer Applications, Engineering, or a related discipline, or equivalent practical experience.
• At least 9 years of professional software engineering experience, including a minimum of 3 years developing and delivering production-grade GenAI or LLM-based applications and platforms.
• Strong proficiency in Python backend development, including asynchronous programming, FastAPI or similar frameworks, REST APIs, and streaming responses.
• Hands-on experience integrating LLM provider APIs, such as OpenAI, Azure OpenAI, and AWS Bedrock.
• Experience implementing tool calling, function calling, and integrations across multiple LLM providers.
• Proven experience deploying and operating services on AWS using containerization technologies and CI/CD pipelines.
• Practical knowledge of Docker, GitHub Actions, and automated testing integrated into software delivery workflows.
• Strong understanding of backend architecture, scalable services, API integrations, and production software engineering practices.
• Demonstrated ability to troubleshoot complex production issues, conduct root-cause analysis, and implement long-term solutions.
• Excellent written and verbal communication skills in English, with the ability to collaborate effectively across globally distributed teams.
• Ability to work independently, take ownership of technical deliverables, and contribute to cross-functional engineering initiatives.
Preferred qualifications and experience:
• Experience working with AI/LLM gateways or API gateways, particularly model routing, API key management, access controls, and cost tracking.
• Experience operating and extending open-source software in production environments, including custom patches, upgrades, regression testing, and upstream contributions.
• Knowledge of agentic AI architectures and tools, particularly Model Context Protocol (MCP).
• Experience with LangGraph, LangChain, or similar agent orchestration frameworks.
• Understanding of multi-agent orchestration, Retrieval-Augmented Generation (RAG), and vector search technologies.
• Experience with performance and load testing, as well as observability platforms such as OpenTelemetry, Langfuse, Prometheus, and Grafana.
• Familiarity with Kubernetes and Infrastructure as Code (IaC) tools such as Terraform.
• Frontend development experience with React and TypeSc
Description from Jobgether's public careers feed, reproduced so you can read the role here. Apply on the company's own site; RealAnalystJobs never submits anything for you.
