Director, Site Reliability Engineer

Job Description:

AssetMark is a leading strategic provider of innovative investment and consulting solutions serving independent financial advisors. We provide investment, relationship, and practice management solutions that advisors use in helping clients achieve wealth, independence, and purpose.

The Opportunity

We are seeking a strategic and technically strong Director of Service/Site Reliability Engineering and Observability to lead the reliability, monitoring, performance, and operational excellence of an enterprise environment hosted exclusively on Microsoft Azure.

This leader will own the strategy and operating model for SRE and observability across both Azure Infrastructure-as-a-Service (IaaS) and native Azure managed services. The scope includes real-time applications, APIs, virtual machines, containers, serverless workloads, integrations, databases, messaging platforms, data pipelines, and batch-processing systems.

The role requires a combination of enterprise leadership, Azure architecture knowledge, application-development experience, production troubleshooting expertise, and the ability to build scalable practices that improve availability, performance, resilience, security, and customer experience.

The Director will partner closely with application development, cloud infrastructure, security, architecture, data, service management, and business teams to ensure Azure-hosted systems are observable, supportable, secure, resilient, and ready for production.

At AssetMark, reliability is a first-order expression of client obsession. Our SRE team plays a critical role in delivering the consistent, trusted technology experience that advisors depend on to run their businesses.

We can only consider candidates for this position who are able to accommodate a hybrid work schedule and are close to our Charlotte, NC office.

Key Responsibilities

SRE and Observability Strategy

• Define and execute an enterprise-wide SRE and observability strategy aligned with business priorities, technology objectives, and risk-management requirements.
• Establish standards for metrics, logs, traces, events, dashboards, alerting, instrumentation, service ownership, telemetry quality, and data retention.
• Develop a multi-year roadmap for Azure observability platforms, reliability engineering, automation, resilience, and operational maturity.
• Define service-level indicators, service-level objectives, service-level agreements, error budgets, availability targets, latency objectives, and performance standards.
• Establish operational-readiness requirements for new applications, major releases, infrastructure changes, Azure service implementations, and third-party integrations.
• Evaluate and govern observability platforms, tools, vendors, integrations, licensing, access, security, and operating costs.
• Promote consistent reliability and observability practices across Azure subscriptions, resource groups, environments, and service teams.

Azure Platform Reliability and Observability

• Lead reliability and observability practices across Azure IaaS and native Azure services, including virtual machines, scale sets, containers, serverless workloads, managed databases, storage, messaging, integration services, and data platforms.
• Establish Azure-native monitoring and telemetry standards using capabilities such as Azure Monitor, Log Analytics, Application Insights, Azure Managed Grafana, Activity Logs, Resource Health, and Azure Service Health.
• Ensure end-to-end visibility across application, infrastructure, platform, network, identity, security, data, and third-party dependencies.
• Define standards for Azure diagnostic settings, centralized logging, metric collection, distributed tracing, correlation IDs, alert routing, dashboard design, and telemetry retention.
• Partner with cloud platform and architecture teams to improve the reliability of Azure subscriptions, management groups, networking, identity, access controls, policies, and shared services.
• Ensure Azure-hosted services are designed with appropriate availability, scalability, recoverability, monitoring, and operational support requirements.

Real-Time Monitoring and Operational Excellence

• Lead the design and operation of real-time monitoring for business-critical applications, customer-facing services, APIs, Azure infrastructure, databases, networks, messaging systems, and integrations.
• Improve the quality, relevance, and actionability of alerts while reducing alert fatigue and unnecessary escalations.
• Create executive, operational, and engineering dashboards that clearly communicate system health, business impact, trends, and emerging risks.
• Establish effective on-call, escalation, incident response, major incident management, and service-restoration practices.
• Participate in root-cause analysis, problem management, post-incident reviews, and corrective-action tracking.
• Drive proactive detection of reliability, capacity, performance, security, and availability risks.
• Develop runbooks, playbooks, automated remediation, self-healing capabilities, and other mechanisms that reduce manual intervention and accelerate recovery.
• Use Azure telemetry and operational data to identify trends, dependencies, recurring failures, and opportunities for continuous improvement.

Application Development and Troubleshooting

• Partner with application-development teams to improve instrumentation, diagnosability, performance, resilience, and production supportability.
• Provide technical leadership during complex production issues involving application code, APIs, microservices, middleware, databases, queues, files, integrations, Azure services, and infrastructure.
• Apply a strong understanding of software-development practices to review application behavior, identify failure patterns, and support code-level troubleshooting.
• Promote observability as part of the software-development lifecycle, including design, development, testing, deployment, and post-release validation.
• Support performance engineering, profiling, capacity planning, release readiness, and production validation.
• Troubleshoot issues across Azure compute, networking, identity, storage, messaging, databases, application runtimes, and service dependencies.
• Use telemetry and operational data to distinguish application defects, infrastructure issues, data problems, configuration errors, and dependency failures.

Batch Systems and Data Processing

• Ensure reliable operation and monitoring of scheduled jobs, batch processes, data pipelines, file transfers, reporting processes, and other time-sensitive workloads hosted on Azure.
• Establish visibility into batch dependencies, job duration, success and failure rates, late starts, missed schedules, data-quality issues, retries, restarts, and downstream impacts.
• Define recovery, restart, reconciliation, and escalation procedures for failed or delayed batch processes.
• Monitor and improve Azure-based batch and data services, including scheduled workloads, serverless processing, messaging, integration services, and data-platform components.
• Partner with application, data, and business teams to protect batch-processing service levels and ensure timely delivery of critical data and outputs.

Reliability, Resilience, and Risk Management

• Improve system availability, scalability, recoverability, and fault tolerance across Azure-hosted enterprise platforms.
• Lead capacity planning, performance monitoring, resilience testing, disaster-recovery readiness, and business-continuity improvements.
• Evaluate resiliency across Azure regions, availability zones, virtual-machine workloads, managed services, and application dependencies.
• Identify single points of failure, operational dependencies, unsupported services, configuration risks, and other sources of reliability risk.
• Partner with security and compliance teams to ensure monitoring, logging, access, retention, and incident practices meet enterprise requirements.
• Establish governance for technology debt, unsupported components, operational risk, Azure service dependencies, and recurring production issues.

Leadership and Organizational Development

• Lead, mentor, and develop SRE, observability, production engineering, and monitoring professionals.
• Build an effective operating model, including team structure, ownership boundaries, escalation paths, career development, and engineering standards.
• Recruit and retain high-performing technical talent.
• Establish a culture of automation, accountability, continuous improvement, learning, and blameless problem solving.
• Communicate reliability posture, operational risks, major incidents, investment needs, and improvement plans to senior technology and business leaders.
• Manage strategic vendor relationships, budgets, contracts, and technology investments as appropriate.

Required Qualifications

• At least 7 years of experience in an enterprise technology environment, including experience supporting production systems at scale.
• Progressive experience leading SRE, observability, production engineering, monitoring, platform operations, or related functions.
• Demonstrated success developing and implementing an enterprise observability strategy.
• Demonstrated experience operating and supporting workloads hosted on both Azure IaaS and native Azure services.
• Strong experience with real-time monitoring, alerting, logging, dashboards, distributed tracing, event management, and service-health reporting.
• Experience with Azure observability capabilities such as Azure Monitor, Log Analytics, Application Insights, Azure Managed Grafana, Activity Logs, Resource Health, or comparable technologies.
• Experience supporting highly available, customer-facing, transaction-processing, or business-critical systems.
• Strong understanding of application architecture, software-development practices, APIs, databases, integrations, messaging, and distributed systems.
• Demonstrated ability to troubleshoot complex production issues across application, Azure platform, infrastructure, data, network, identity, and third-party dependency layers.
• Direct experience with batch systems, scheduled processing, data pipelines, job dependencies, recovery procedures, and time-sensitive enterprise workloads.
• Experience defining and managing SLIs, SLOs, SLAs, error budgets, availability targets, performance objectives, and operational metrics.
• Strong background in incident command, major incident management, root-cause analysis, problem management, and continuous improvement.
• Experience driving automation, self-service, runbook development, infrastructure as code, CI/CD improvements, or automated remediation.
• Demonstrated ability to lead cross-functional initiatives and influence teams without direct authority.
• Excellent written, verbal, analytical, and executive-communication skills.

Preferred Qualifications

• Experience leading SRE or observability functions in a large-scale Microsoft Azure enterprise environment.
• Experience with Azure landing zones, management groups, subscriptions, resource groups, Azure Policy, role-based access control, managed identities, networking, private endpoints, and shared platform services.
• Familiarity with Azure services such as Azure Kubernetes Service, App Service, Functions, Service Bus, Event Grid, Logic Apps, Azure Storage, Azure SQL, Cosmos DB, Data Factory, Synapse, and related managed services.
• Experience with Azure DevOps, GitHub Actions, Bicep, ARM templates, Terraform, containers, Kubernetes, CI/CD, and cloud-native deployment practices.
• Familiarity with observability tools such as Splunk, Datadog, Dynatrace, New Relic, Grafana, Prometheus, OpenTelemetry, Elastic, or comparable platforms.
• Knowledge of performance engineering, capacity management, high-availability architecture, disaster recovery, and resilience engineering.
• Experience modernizing legacy monitoring environments or consolidating multiple observability platforms within Azure.
• Experience managing technology budgets, vendor relationships, licensing, and platform strategy.
• Familiarity with ITIL, SRE, DevOps, Agile, or other service-management and engineering frameworks.
• Bachelor’s degree in computer science, information systems, engineering, or a related field, or equivalent professional experience.

Measures of Success

Success in this role will include:

• Improved availability, reliability, performance, and resilience of critical Azure-hosted enterprise services.
• Increased observability coverage across Azure IaaS, native Azure services, real-time applications, integrations, and batch systems.
• Faster detection, diagnosis, communication, and resolution of incidents.
• Reduced alert noise, repeat incidents, manual effort, and operational risk.
• Consistent adoption of SLOs, instrumentation, Azure monitoring standards, operational-readiness requirements, and effective runbooks.
• Higher batch-processing success rates and improved on-time completion of critical workloads.
• Stronger collaboration between development, operations, cloud infrastructure, security, data, and business teams.
• A mature, scalable, and high-performing SRE and observability organization.

Compensation: The Base Salary range for this position is between $180,000-$219,000.

This information reflects a base salary range that AssetMark reasonably expects to pay for the position based on a number of factors which may include job-related knowledge, skills, education, experience, and actual work location. This position will also be eligible for additional variable incentive compensation and competitive benefits.

Candidates must be legally authorized to work in the US to be considered. We are unable to provide visa sponsorship for this position.

#LI-hybrid

#LI-TN1

Who We Are & What We Offer:

We are AssetMark, a company on the move, shaping the future of financial services. Growth is in our DNA. Every day, we combine technology, insight, and collaboration to create new possibilities for advisors, for our people and for our investors. At AssetMark your ideas matter; they’re heard, valued, and drive meaningful change. Join a team that sets new standards and creates space for you to thrive and do your best work. 

Our Mission 

Our mission is simple: to help our 10,500+ financial advisors make a meaningful difference in their clients’ lives. We do this by combining powerful technology, holistic support, and expert consulting to help advisors run stronger, more efficient businesses. Backed by a comprehensive suite of investment solutions and a trust company that boasts of $150B+ AUM, our platform empowers advisors to deliver exceptional service and an outstanding client experience.

Our Values 

Heart. Client Success. Integrity. Respect. Excellence. Our values are how we show up every day.  

We believe in: 

• Leading with Heart, in truly making a difference in the lives of others: teammates, clients, investors and communities. 

• Obsessing over Client Success, bringing a relentless focus on what matters to clients that sets us apart and creates loyal, lasting relationships. 

• Unyielding Integrity, doing what’s right, always. Even when it’s hard. 

• Collective Respect, in being authentic, inclusive and valuing all voices while winning together. 

• Operating with Excellence, in learning fast, continuously improving, innovating and collaborating to find new and better solutions.  

These values shape our culture, guide our decisions, and define what it means to be part of the AssetMark family. 

Our Culture & Benefits 

Our culture brings our mission and values to life. Here, we do what’s right, embrace diverse ideas, and innovate together. We also offer a wide range of benefits to support you and your family—because thriving at work starts with thriving in life. 

• Flex Time or Paid Time Off and Sick Time Off 

• 401K – 6% Employer Match 

• Medical, Dental, Vision – HDHP or PPO 

• HSA – Employer contribution (HDHP only) 

• Volunteer Time Off 

• Career Development / Recognition 

• Fitness Reimbursement 

• Hybrid Work Schedule 

   

As an Equal Opportunity Employer, AssetMark is committed to building a diverse and inclusive workplace where everyone feels valued.