Lead Principal Site Reliability Engineer

Mandatory

• 8+ years of hands-on experience across software development, site reliability engineering, monitoring, and automation, with demonstrated ownership of production platforms.
• Strong programming skills in Python and experience developing backend services, REST APIs, integrations, and event-driven workflows.
• Proven experience building monitoring and observability solutions that collect and analyze alerts, metrics, logs, and service context.
• Hands-on experience integrating AI/LLM capabilities into applications for incident analysis, contextual recommendations, and operational decision support.
• Ability to develop reusable agents that combine live monitoring data, historical incidents, and previous remediation actions to recommend next steps.
• Experience implementing alert correlation, deduplication, enrichment, and prioritization to reduce noise and improve incident detection.
• Strong experience developing automated remediation workflows with approval controls, execution validation, audit trails, and recovery safeguards.
• Experience integrating monitoring platforms with incident management systems, knowledge repositories, and notification or escalation services.
• Strong understanding of distributed systems, Linux, networking, databases, and application performance, with the ability to translate failure patterns into platform capabilities.
• Experience applying software engineering practices, including automated testing, code reviews, CI/CD, version control, and secure development.
• Ability to evaluate platform reliability and AI output quality, measure operational outcomes, and improve capabilities using engineer feedback.
• Excellent communication and collaboration skills, with the ability to work with monitoring teams, service owners, and engineers to onboard services and define requirements.

Good-to-Have

• Experience with agentic AI frameworks, tool calling, retrieval-augmented generation, and vector databases.
• Knowledge of anomaly detection, time-series analysis, and predictive techniques for identifying recurring alarm patterns.
• Experience building centralized notification services, automated escalation calls, and incident communication workflows.
• Familiarity with OCI, Kubernetes, and Infrastructure-as-Code tools such as Terraform or Ansible.
• Experience with observability platforms such as OCI Observability, Prometheus, Grafana or equivalent tools.
• Exposure to Oracle Database and Exadata monitoring, performance diagnostics, and remediation automation.
• Experience using AI coding assistants to improve operational scripts and validate proposed changes.

• Knowledge of SLOs, error budgets, and incident metrics such as detection time, recovery time, and recurrence rate.

   

  Key Responsibilities

Capacity Ingestion and Management:

-       Serves as a consultant across teams and leads the design and architecture of infrastructure and/or service, ensuring alignment with reliability and functionality standards.

-       Takes full ownership of forecasting infrastructure demands and strategically responds to capacity needs, ensuring systems have sufficient resources to handle current and future workloads, anticipating resource gaps.

-       Owns collaborations and serves as a decision maker with the software development team in designing and developing infrastructures that are highly reliable and scalable, removing barriers to meet deployment requirements.

-       Seeks opportunities for prototyping and encourages the implementation of prototyping initiatives (e.g., testing new applications or infrastructures, assisting in onboarding), driving new approaches.

-       Identifies and projects resource gaps using cost calculators, identifying alternative solutions to lower costs, when necessary.

Incident and Service Lifecycle Management:

-       Recommends methods for data collection, triage, technical analysis, and redirection to maintain and optimize operations and infrastructure reliability.

-       Provides guidance to others when monitoring services, maintains up-to-date knowledge of their performance, and meticulously documents their condition.

-       Oversees root cause analyses for incidents and/or maintenance on assigned services (e.g., software installs, version upgrades, security updates, backup and recovery), ensuring efficient execution and preventing incident reoccurrence.

-       Provides strategic, future-oriented health and performance reporting, and anticipates actions needed based on trends in data.

-       May guide provisioning to support infrastructure, applications, and services.

-       May leverage non-conventional approaches and encourages experimentation in decommissioning approaches (e.g., shutting down servers, removing data from databases) to remove objects that are no longer needed.

Automation:

-       Contributes to strategies for automation, identifying opportunities and quantifying potential benefits for their line of business.

-       Reviews the development and implementation of design, automation tools, or scripts to provide solutions, gather metrics, monitor, analyze, mitigate, or remediate issues/defects within infrastructures.

-       Guides others to provide solutions, designs, and executes testing methodologies to ensure automation effectiveness.

-       Provides thought leadership and encourages others testing automations.

Technical Communication and Guidance:

-       Provides expert-level release notes, communication, and/or guidance on the scale, capacity, security, performance attributes, and requirements of services and technology to customers, cross-functional teams, leadership, and external stakeholders.

-       Anticipates, analyzes, and effectively explains the nuanced impacts of infrastructure, feature, and tool changes, considering the organization's strategic goals, industry trends, and emerging technologies.

-       Provides guidance to team members on what information to communicate and how to communicate.

Troubleshooting and Resolution:

-       Serves as a role model in providing operational support for technology, guiding the resolution of critical incidents and a wide range of complex issues across Oracle services.

-       Provides leadership in the on-call shifts.

-       Leads the resolution of multifaceted technical issues spanning multiple services, functions, and customers, collaborating with cross-functional teams and leveraging advanced investigation and debugging techniques to ensure the achievement of SLOs (service level objectives).

-       Reviews documentation for accuracy, guides the performance of root cause analyses, and contributes to the development of incident documentation standards, recommending best practices, insights, and improvements.

-       Owns post-mortem procedures to prevent incident reoccurrence, making recommendations when necessary.

Innovation and Improvement:

-       Leads the research, experimentation, and implementation of tools and technologies to revolutionize infrastructure performance and reliability, investigating implications of adherence to security standards on other integrations.

-       Provides input on strategic initiatives to address opportunities to improve performance bottlenecks and deployments, maximizing resource utilization, cost efficiency, speed, and scalability across their line of business.

-       Provides expertise in site reliability trends, guiding the creation and sharing of insights and best practices to shape the future of building, testing, deploying, and running services.

-       Leads the performance of a wide range of complex analyses and provides clear data on production to guide and heavily influence business development decisions (e.g., design changes).

Core Responsibilities

Planning & Execution:

-       Manages and provides direction on timelines, deliverables, and budgets when applicable for critical high-impact projects or initiatives that impact the line of business, ensuring timely completion and adherence to requirements. Anticipates and plans for shifts in resources or timelines based on changing business priorities, ensuring optimal outcomes.

Collaboration & Partnership:

-       Influences cross-functional leaders and external stakeholders to gain alignment on strategic objectives. Fosters partnerships with key business leaders, stakeholders, and/or customers, identifying opportunities for expanding partnerships and promoting long-term organizational success. Champions transparency and inclusivity by actively seeking, listening to, and incorporating diverse perspectives.

Problem Solving:

-       Leads specialized, advanced problem-solving efforts, serving as an escalation point for complex issues. Guides others to leverage innovative data-driven techniques to address ambiguous or novel issues, identify root causes, and drives the implementation of solutions that prevent future issues.

Continuous Learning:

-       Leverages deep industry knowledge and expertise to serve as a thought leader within the organization. Contributes to the advancement of the field or industry through thought leadership (e.g., conference presentations, white papers, research contributions). Maintains and evolves expertise in relevant areas by proactively monitoring emerging trends, technologies, and industry standards, ensuring the organization remains current with best practices. Champions continuous learning and knowledge sharing, promoting professional development across teams. Applies new knowledge to drive advancement and mentors others to do the same.

Continuous Improvement:

-       Develops innovative solutions and drives the implementation of ideas that increase the efficiency and effectiveness of processes, protocols, and workflows across the organization. Evaluates effectiveness of updated approaches and methods for continued improvement to enhance efficiencies and ensure changes align with organizational goals. Designs and develops metrics to measure success of improvement initiatives.

Performance and Development:

-       Serves as a subject matter expert regarding talent needs and organizational talent strategy. Imparts leadership and expert knowledge throughout the talent development pipeline including candidate interviews, candidate assessment, and hiring decisions, ensuring alignment with organizational talent strategy.