Principal Site Reliability Engineer

Mandatory

• 5+ years of hands-on experience across site reliability engineering, monitoring, and automation, with demonstrated ownership of production platforms.
• Strong application skills in EBS, Database, weblogic, middleware, ....
• Expertise in Oracle Database and Exadata monitoring, performance diagnostics, and remediation automation.
• Good to have programming skills in Python, shell scripting, integrations, and event-driven workflows.
• Proven experience building monitoring and observability solutions that collect and analyze alerts, metrics, logs, and service context.
• Good to have experience integrating AI/LLM capabilities into applications for incident analysis, contextual recommendations, and operational decision support.
• Experience implementing alert correlation, deduplication, enrichment, and prioritization to reduce noise and improve incident detection.
• Strong experience developing automated remediation workflows with approval controls, execution validation, audit trails, and recovery safeguards.
• Experience integrating monitoring platforms with incident management systems, knowledge repositories, and notification or escalation services.
• Strong understanding of distributed systems, Linux, networking, databases, and application performance, with the ability to translate failure patterns into platform capabilities.
• Ability to evaluate platform reliability and AI output quality, measure operational outcomes, and improve capabilities using engineer feedback.
• Excellent communication and collaboration skills, with the ability to work with monitoring teams, service owners, and engineers to onboard services and define requirements.

Good-to-Have

• Experience with agentic AI frameworks, tool calling, retrieval-augmented generation, and vector databases.
• Knowledge of anomaly detection, time-series analysis, and predictive techniques for identifying recurring alarm patterns.
• Experience building centralized notification services, automated escalation calls, and incident communication workflows.
• Familiarity with OCI, Kubernetes, and Infrastructure-as-Code tools such as Terraform or Ansible.
• Experience with observability platforms such as OCI Observability, Prometheus, Grafana or equivalent tools.
• Experience using AI coding assistants to improve operational scripts and validate proposed changes.
• Knowledge of SLOs, error budgets, and incident metrics such as detection time, recovery time, and recurrence rate.

Key Responsibilities

Production Reliability and Application Operations

• Own the reliability, availability, and performance of production platforms supporting Oracle E-Business Suite (EBS), Oracle Database, Exadata, WebLogic, and middleware services.
• Diagnose and resolve issues across applications, databases, Linux systems, networking, and distributed services.
• Analyze performance and capacity trends, identify bottlenecks, and implement improvements to support workload growth and service reliability.

Monitoring and Observability

• Build and enhance monitoring and observability solutions that collect and analyze alerts, metrics, logs, and service context.
• Implement alert correlation, deduplication, enrichment, and prioritization to reduce noise and improve incident detection.
• Develop health dashboards and actionable alerts for Oracle Database, Exadata, EBS, WebLogic, and supporting infrastructure.
• Integrate monitoring platforms with incident management systems, knowledge repositories, and notification or escalation services.

Automation and Remediation

• Identify recurring operational issues and develop automated workflows for diagnostics, mitigation, remediation, and recovery.
• Implement approval controls, execution validation, audit trails, and recovery safeguards to ensure safe and traceable automation.
• Test remediation workflows against expected outcomes and failure scenarios before production adoption.
• Develop and maintain operational scripts, integrations, and event-driven workflows using appropriate tools and languages.

Incident Management and Troubleshooting

• Participate in on-call support and lead incident triage, technical investigation, and service restoration across supported platforms.
• Use monitoring data and service dependencies to assess incident impact, prioritize response, and coordinate with service owners.
• Conduct root cause analyses and post-incident reviews, document findings, and implement corrective actions to prevent recurrence.
• Maintain runbooks and knowledge articles that support consistent incident response and operational readiness.

AI-Assisted Operations and Innovation

• Evaluate and, where appropriate, integrate AI/LLM capabilities for incident analysis, contextual recommendations, and operational decision support.
• Explore retrieval-augmented generation, tool calling, and agentic workflows to connect operational knowledge with diagnostic and remediation capabilities.
• Assess AI output quality, relevance, and reliability using defined evaluation criteria and engineer feedback.
• Evaluate anomaly detection and time-series techniques to identify recurring alert patterns and emerging service issues.

Collaboration and Continuous Improvement

• Partner with monitoring teams, service owners, and engineers to define requirements, onboard services, and establish actionable monitoring and remediation coverage.
• Measure operational outcomes through service-level objectives, alert noise, detection time, recovery time, and incident recurrence.
• Use incident trends, platform performance data, and engineer feedback to prioritize improvements and validate their impact.
• Communicate platform changes, operational risks, and incident findings clearly, and share knowledge through documentation and mentoring.

Core Responsibilities

Application Support and Service Ownership

• Own day-to-day production support for Oracle EBS, Oracle Database, Exadata, WebLogic, and middleware services, ensuring availability, stability, and performance.
• Manage support requests and incidents according to business impact, urgency, and agreed service levels.
• Coordinate application maintenance, patching, upgrades, and production changes, including readiness checks, validation, and recovery planning.

Planning and Execution

• Plan and prioritize application support activities, incident follow-ups, automation initiatives, and service improvements.
• Coordinate dependencies across application, database, infrastructure, and development teams to deliver timely resolutions.
• Maintain clear ownership, timelines, and status updates for support activities and outstanding issues.

Collaboration and Communication

• Work with business users and technical teams to understand application issues, assess business impact, and restore service.
• Provide timely incident updates, communicate workarounds and resolution progress, and coordinate escalations.
• Partner with development teams to translate recurring production issues into permanent fixes and support requirements.

Problem Solving and Escalation

• Investigate application failures, transaction errors, integration issues, and performance degradation using logs, metrics, and diagnostic tools.
• Escalate unresolved issues with clear evidence, business impact, troubleshooting history, and recommended next steps.
• Conduct root cause analysis and track corrective actions through completion to reduce recurring incidents.

Operational Readiness and Knowledge Management

• Maintain application support runbooks, troubleshooting guides, dependency documentation, and known-error records.
• Validate monitoring, alert routing, support procedures, and recovery readiness before accepting new applications or changes into production.
• Ensure effective shift handovers and knowledge transfer to maintain continuity of support.

Continuous Improvement and Development

• Identify opportunities to automate repetitive support tasks and improve monitoring, diagnostics, and remediation.
• Track incident trends, service-level performance, recovery time, and user feedback to prioritize support improvements.
• Build application and platform expertise, mentor junior engineers, and share effective troubleshooting practices.
• Contribute to hiring and onboarding by assessing relevant application support skills and helping new team members become operationally ready.