SRE Splunk Enterprise Administrator – (w/ active Secret)

SRE Splunk Enterprise Administrator – (w/ active Secret)

Location: Norfolk, VA
Clearance: Secret
Type: Full-time, Onsite
Salary Range: $68,000 - $87,000

JOB DESCRIPTION

Critical Solutions is seeking a SRE Splunk Enterprise Administrator to support our federal customer.

The Splunk Enterprise Administrator will be part of the Site Reliability Engineers (SRE) supporting mission-critical cybersecurity operations by administering and maintaining distributed Splunk Enterprise platforms across hybrid cloud and on-premises environments. The role is responsible for daily platform operations, performance and ingestion monitoring, data onboarding support, incident troubleshooting, security compliance, documentation, and modernization activities in an Agile/DevOps operating culture.

PRIMARY ROLES AND RESPONSIBILITIES:

• Administer, maintain, and perform daily Operations and Maintenance (O&M) for distributed Splunk Enterprise environments, including Search Heads, Indexers, Heavy Forwarders, Intermediate Forwarders, Universal Forwarders, and Deployment Servers across hybrid cloud and on-premises infrastructure.
• Monitor platform health, data ingestion, indexing throughput, search performance, and retention utilization; troubleshoot data onboarding, parsing, field extraction, forwarding, indexing, and ingestion issues.
• Support Splunk Cloud integrations and associated hybrid operational activities.
• Maintain Splunk applications, dashboards, alerts, saved searches, and knowledge objects.
• Support patching, vulnerability remediation, Security Technical Implementation Guide (STIG) compliance, and system hardening activities.
• Support platform upgrades, migrations, infrastructure modernization, technology refresh, and automation initiatives using scripting and configuration-management tools.
• Participate in incident response, outage troubleshooting, problem resolution, and root cause analysis.
• Maintain operational documentation, architecture diagrams, runbooks, and standard operating procedures (SOPs), and coordinate with cybersecurity, network, server, engineering, and customer teams during operational and modernization activities.
• Participate in after-hours support and an on-call rotation as required.

BASIC QUALIFICATIONS:

• Must be a U.S. citizen and possess an active DoD Secret Security Clearance or higher.
• Requires BS degree and prior relevant experience or Master's prior relevant experience.
• Minimum of DoD 8570.01 IAT Level II Certification required.
• Must have a Splunk Enterprise Certified Admin or Splunk Cloud Certified Admin Certification.
• Minimum three years of experience administering Splunk Enterprise v9 and supporting distributed Splunk architectures in hybrid cloud and on-premises production environments.
• Experience administering and troubleshooting Red Hat Enterprise Linux (RHEL) 8 and/or RHEL 9 servers.
• Experience with STIGs, the Risk Management Framework (RMF), vulnerability management, and compliance frameworks.
• Experience developing and maintaining operational dashboards, visualizations, and executive reporting in Splunk Enterprise and Splunk Cloud.
• Strong working knowledge of Splunk data ingestion pipelines, indexing, parsing, forwarding, search optimization, and retention management.
• Experience using at least one scripting or automation language: Python, Bash, or PowerShell.
• Strong troubleshooting, analytical, communication, and documentation skills, with the ability to work independently, manage competing priorities, collaborate across technical teams, and maintain a customer-focused approach in a high-tempo production environment.
• Strong communication, analytical, and problem-solving skills, ability to work in a team environment.
• Ability to work onsite at Norfolk Station Monday through Friday day shift.
• Must be able to participate in after-hours support and an on-call rotation as required.

CERTIFICATION REQUIREMENT:

• Minimum of DoD 8570.01 IAT Level II Certification AND must have a Splunk Enterprise Certified Admin or Splunk Cloud Certified Admin Certification.

PREFERRED QUALIFICATIONS:

• Experience with automated script design, coding, debugging, and maintenance skills (using bash, python, etc.) preferred.
• •Experience working in classified government or Department of Defense environments.
• •Familiarity with RHEL 10

LOCATION:

• On-site 100%. Norfolk, VA
• Ability to work onsite at Norfolk Monday through Friday day shift.
• Participate in after-hours support and an on-call rotation as required.

Key Metrics of Success for the Team:

• Improved system reliability, as measured by adherence to Service Level Objectives (SLOs) and reduced Mean Time to Recovery (MTTR).
• Comprehensive and regularly updated automated test coverage for all critical systems and infrastructure components.
• Timely identification and resolution of performance bottlenecks and failure points.
• Integration of automated testing into the CI/CD pipeline, ensuring continuous reliability validation.
• Increased scalability and performance of systems under high load due to effective performance testing.

ADDITIONAL INFORMATION:

CLEARANCE REQUIREMENT: Must possess an active DoD Secret Clearance. In addition, selected candidate must undergo background investigation (BI) and finger printing by the federal agency and successfully pass the preceding to qualify for the position. US CITIZENSHIP IS REQUIRED.

CRITICAL SOLUTIONS PAY AND BENEFITS:

Salary range $68,00 - $87,000. The salary range for this position represent the typical salary range for this job level and this does not guarantee a specific salary. Compensation is based upon multiple factors such as responsibilities of the job, education, experience, knowledge, skills, certifications, and other requirements.

BENEFIT SNAPSHOT: 100% premium coverage for Medical, Dental, Vision, and Life Insurance, Supplemental Insurance, 401K matching, Flexible Time Off (PTO/Holidays), Higher Education/Training Reimbursement, and more.