Principal Site Reliability Engineer

Mandatory

• 5+ years of hands-on experience across software development, site reliability engineering, monitoring, and automation, with demonstrated ownership of production platforms.
• Strong programming skills in Python and experience developing backend services, REST APIs, integrations, and event-driven workflows.
• Proven experience building monitoring and observability solutions that collect and analyze alerts, metrics, logs, and service context.
• Hands-on experience integrating AI/LLM capabilities into applications for incident analysis, contextual recommendations, and operational decision support.
• Ability to develop reusable agents that combine live monitoring data, historical incidents, and previous remediation actions to recommend next steps.
• Experience implementing alert correlation, deduplication, enrichment, and prioritization to reduce noise and improve incident detection.
• Strong experience developing automated remediation workflows with approval controls, execution validation, audit trails, and recovery safeguards.
• Experience integrating monitoring platforms with incident management systems, knowledge repositories, and notification or escalation services.
• Strong understanding of distributed systems, Linux, networking, databases, and application performance, with the ability to translate failure patterns into platform capabilities.
• Experience applying software engineering practices, including automated testing, code reviews, CI/CD, version control, and secure development.
• Ability to evaluate platform reliability and AI output quality, measure operational outcomes, and improve capabilities using engineer feedback.
• Excellent communication and collaboration skills, with the ability to work with monitoring teams, service owners, and engineers to onboard services and define requirements.

Good-to-Have

• Experience with agentic AI frameworks, tool calling, retrieval-augmented generation, and vector databases.
• Knowledge of anomaly detection, time-series analysis, and predictive techniques for identifying recurring alarm patterns.
• Experience building centralized notification services, automated escalation calls, and incident communication workflows.
• Familiarity with OCI, Kubernetes, and Infrastructure-as-Code tools such as Terraform or Ansible.
• Experience with observability platforms such as OCI Observability, Prometheus, Grafana or equivalent tools.
• Exposure to Oracle Database and Exadata monitoring, performance diagnostics, and remediation automation.
• Experience using AI coding assistants to improve operational scripts and validate proposed changes.
• Knowledge of SLOs, error budgets, and incident metrics such as detection time, recovery time, and recurrence rate.

Key Responsibilities

Capacity Ingestion and Management:

-       Designs and architects infrastructure and/or service according to terms for reliability and functionality.

-       Forecasts demands for infrastructure and responds to capacity needs, ensuring systems have sufficient resources to handle current and future workloads and identifying resource gaps.

-       Collaborates with the software development team to develop infrastructures, ensuring features are reliable and scalable according to deployment requirements.

-       Proactively identifies opportunities for prototyping and drives prototyping initiatives (e.g., testing new applications or infrastructures, assisting in onboarding) to explore novel approaches.

Incident and Service Lifecycle Management:

-       Exercises judgment when performing data collection, triage, technical analysis, and redirection to maintain and optimize operations and infrastructure reliability.

-       Takes proactive steps to monitor services, maintain up-to-date knowledge of their performance, and document their condition.

-       Leverages advanced knowledge to perform incident response, root cause analyses, and/or maintenance on assigned services (e.g., software installs, version upgrades, security updates, backup and recovery).

-       Provides comprehensive health and performance reporting and takes appropriate actions based on trends in data.

-       May perform provisioning to support infrastructure, applications, and services.

-       May experiment with new approaches for and performs decommissioning (e.g., shutting down servers, removing data from databases) to remove objects that are no longer needed.

Automation:

-       Identifies and recommends opportunities for automation and assesses potential benefits to enhance operational efficiency.

-       Develops and implements design, automation tools, or scripts to provide solutions, gather metrics, monitor, analyze, mitigate, or remediate issues/defects within infrastructures.

-       Conducts testing on moderately complex automations to ensure they perform tasks correctly and produce expected results.

Technical Communication and Guidance:

-       Writes release notes and/or communicates comprehensive information about the scale, capacity, security, performance attributes, and requirements of services and technology with customers and immediate and related teams.

-       Proactively anticipates and articulates the potential impact of infrastructure, feature, and tool changes, considering their impact across team operations.

-       Serves as a resource to team members on what information to communicate and how to communicate.

Troubleshooting and Resolution:

-       Provides comprehensive operational support for technology, serving as a key escalation point for incidents and moderately complex issues arising within Oracle services.

-       Drives and actively participates in on-call shifts to address issues.

-       Executes the resolution of technical issues spanning multiple services, applying advanced investigation and debugging techniques to achieve SLOs (service level objectives).

-       Documents incidents according to reporting methods and performs root cause analyses, capturing essential information for analysis and future reference.

-       Performs post-mortem procedures to prevent incident reoccurrence.

Innovation and Improvement:

-       Conducts advanced experiments and evaluations of cutting-edge tools and technologies to optimize infrastructure performance and reliability, taking proactive steps to adhere to security standards.

-       Identifies and seeks opportunities to execute improvements for performance bottlenecks and deployments, ensuring efficient resource usage, speed, and scalability.

-       Develops and maintains advanced knowledge of site reliability trends, sharing valuable insights and information with senior team members, management, and beyond to promote innovative building, testing, deploying, and running services.

-       Performs moderately complex analyses and provides clear data on production to drive business development decisions (e.g., design changes).

Core Responsibilities

Planning & Execution:

-       Manages and coordinates moderately complex tasks, monitoring timelines and deliverables to ensure timely completion and adherence to requirements for a moderately sized project or initiative. Efficiently delegates, monitors, and prioritizes work across multiple projects, providing technical oversight and adjusting plans to address shifts in resources or timelines.

Collaboration & Partnership:

-       Collaborates across the organization to align on expectations and achieve shared objectives. Leverages understanding of business leaders, stakeholders, and/or customers to ensure proposed solutions meet their needs. Supports inclusivity by actively seeking and listening to diverse perspectives, ensuring others feel heard and respected.

Problem Solving:

-       Identifies and addresses moderately complex issues by analyzing a wide range of data and/or information to identify solutions in accordance with standard practices. Proactively escalates unresolved or critical issues with a thorough assessment and suggests potential solutions. Reviews, contributes to, and documents problem solving strategies.

Continuous Learning:

-       Pursues learning opportunities to expand knowledge and skills and/or tools in new areas and stays abreast of the latest industry trends and best practices. Proactively seeks and leverages ongoing feedback and training to improve skills. Coaches and mentors junior team members, fostering continuous learning and knowledge sharing within and across teams.

Continuous Improvement:

-       Develops ideas, recommends updates, and/or collaborates on the implementation of process improvements to increase the efficiency and effectiveness of processes, protocols, and workflows across teams, and evaluates the impact on key stakeholders. Solicits feedback from others on ideas for alternative approaches and methods for continued improvement.

Performance and Development:

-       Contributes to the talent development pipeline by participating in candidate interviews, assessing candidates, and providing hiring recommendations.

-