Research Computing Systems Engineer

Stanford Research Computing is looking for a talented systems engineer to join our team of collaborative and innovative professionals helping Stanford's faculty and students use advanced computing and data tools to explore new frontiers in knowledge and solve some of humanity's most urgent problems. Our staff work directly with some of the world's top researchers in a broad range of disciplines, across all of Stanford's seven schools — while also supporting and learning from each other in cross-project endeavors. We maintain and steadily improve an advanced research computing facility, and we support a variety of environments for Stanford research. In Stanford Research Computing, you'll have a rare opportunity to contribute to discoveries and inventions that have global reach and positive impact, and to share in the curiosity and commitment of the scholars and scientists who lead these projects.

About the RoleStanford Research Computing operates a portfolio of large-scale HPC clusters and petabyte-scale storage systems serving thousands of researchers. Every one of those platforms depends on a layer of shared infrastructure that researchers never see: the bastion hosts our administrators pass through to reach management networks, the network license servers that let research software start, the container hosts carrying our internal services, and the observability stack that tells us something is wrong before a researcher has to.

In this role you will be the primary steward of that layer, serving as lead for the "backbone" services the rest of our environment depends on. You will also contribute to a co-location initiative, standing up management infrastructure and geographically distinct replication in a remote data center. The work spans a wide range of technologies, with substantial engineering ahead in hardware lifecycle, redundancy, and configuration management.

Why Stanford? You won't be maintaining back-office IT. You will be building the connective tissue that a multi-petabyte, multi-platform research computing environment runs on — the systems that decide whether a downtime is a scheduled inconvenience or a lost quarter of someone's research.

Responsibilities

Bastion hosts: Lead the administration, hardening, and modernization of bastion and jump hosts across multiple data centers, and design the next generation with redundancy and high availability. Support partner unit administrators moving from VDI to bastion-based access.

Network license services: Operate and improve the primary and secondary FLEXlm license servers behind commercial research and statistical software. Right-size firewall policy, retire orphaned license hosts, and bring runbooks and user documentation current.

Central observability service: Lead the central observability service and the servers behind it — Prometheus, Grafana, Splunk, XDMoD, and facility telemetry. Consolidate legacy monitoring and build alerting that reaches the right person with enough context to act.

Co-location project: Support a funded co-location initiative from design through production hand-off: network architecture, management infrastructure and a bastion host in the remote facility, and a geographically distinct replication target for independent backups.

Container hosts and backbone services: Lead the container hosts and the internal services they carry, along with cluster head and service nodes, bare-metal provisioning, and the VMs behind other Research Computing backbone services. Bring these under Ansible and Git-based configuration management.

Security and compliance: Establish and maintain secure build standards across the systems you own, in line with university, regulatory, and contractual requirements. Deploy endpoint and logging agents, remediate scan findings, and manage credential and access lifecycle.

Resilience and documentation: Reduce single points of failure, and document core service requirements and details so that any member of the team can operate, restore, or rebuild a service.

Planning and hardware lifecycle: Meet regularly with platform leads and service owners to understand upcoming needs, own hardware lifecycle and capital planning for the infrastructure you lead, and help develop business cases for replacement and expansion.

Vendor engagement: Liaise with hardware and software vendors and support partners to triage and resolve issues, manage RMAs, and keep systems under appropriate warranty and support coverage. 

Responsibilities

Core Duties:

• Lead the design, development, installation and maintenance ofoperating systems, utilities, and applications software on computingsystems.

• Anticipate risks, de-escalate issues, and prevent emergencies tolimit disruptions to system operations and protect the integrity of userdata and systems.

• Safeguard the university’s data and system assets – formulatesystem security strategies and develop viable policies and proceduresthat will enable the design and implementation of system securitymeasures at the university.

• Establish and enforce systems policies and procedures andvalidate that university software/hardware standards are aligned withexternal best practice.

• Partner with other information technology specialty areas toconfirm information technology strategies, devise and deploy plans toensure information technology objectives are met, and advise ontechnical feasibility of information technology initiatives,particularly regarding system compatibility within the university’scurrent, or proposed technical or structural framework(s).

• Review and conduct capacity planning for system configuration,software services, network services, load distribution, and serviceinterrelationships among computer systems.

• Act as technical expert or lead for university-wide computersystem administration. May manage system administration staff.

• Provide project management for large and complex university-widecomputing projects.

• Manage vendor relationships and negotiate cost effective hardwareand software maintenance agreements with vendors.

Minimum Education and Experience:

Bachelor's degree and ten years of relevant experience, or acombination of education and relevant experience.

Knowledge, Skills and Abilities:

• Extensive experience with complex, multi-system platforms andvendors.

• Notable experience coordinating multi-system and computingenvironments in independent computing facilities.

• Extensive experience developing/implementing a businesscontinuation and disaster recovery plan.

• Exceptional ability to develop appropriate plans to meetcomputing needs.

• Expert ability to program in multiple programming languages inmultiple operating systems.

• Superior ability to lead and work on large/complex systemdeployment projects in a team environment.

• Expert knowledge of security trends and best practice.