Site Reliability Engineer, Reliability/Database

Site Reliability Engineers (SREs) are responsible for keeping all user-facing services and other GitLab production systems running smoothly. SREs are a blend of pragmatic operators and software craftspeople that apply sound engineering principles, operational discipline, and mature automation to our environments and the GitLab codebase. We specialize in systems, whether it be networking, the Linux kernel, or some more specific interest in scaling, algorithms, or distributed systems.

The Database Reliability Team's mission is to build, run and own the entire lifecycle of the PostgreSQL database engine for GitLab.com. The team is focused on owning the reliability, scalability, performance & security of the database engine and its supporting services. The team should be seeking to build their services on top of Reliability::Foundations services and cloud vendor managed products, where appropriate, to reduce complexity, improve efficiency and deliver new capabilities quicker. 

GitLab.com is a unique site and it brings unique challenges–it’s the biggest GitLab instance in existence. In fact, it’s one of the largest single-tenancy open-source SaaS sites on the internet. The experience of our team feeds back into other engineering groups within the company, as well as to GitLab customers running self-managed installations

• Be on an on-call (PagerDuty) rotation to respond to incidents that impact GitLab.com availability, and provide support for service engineers with customer incidents.
• Use your on-call shift to prevent incidents from ever happening.
• Run our infrastructure with Chef, Ansible, Terraform, GitLab CI/CD, and Kubernetes.
• Build monitoring that alerts on symptoms rather than on outages.
• Document every action so your findings turn into repeatable actions and then into automation.
• Use the GitLab product to run GitLab.com as a first resort and improve the product as much as possible
• Improve operational processes (such as deployments and upgrades) to make them as boring as possible.
• Design, build and maintain core infrastructure that enables GitLab scaling to support hundreds of thousands of concurrent users.
• Debug production issues across services and levels of the stack.
• Plan the growth of GitLab’s infrastructure.

• Experience supporting PostgreSQL in large production environments.
• Experience with infrastructure automation and configuration management, using tools such as Chef, Ansible, Terraform, etc
• Think about systems: edge cases, failure modes, behaviors, specific implementations.
• Know your way around Linux and the Unix Shell.
• Have an urge to collaborate and communicate asynchronously.
• Have an urge to document all the things so you don’t need to learn the same thing twice.
• Have an enthusiastic, go-for-it attitude. When you see something broken, you can’t help but fix it.
• Have an urge for delivering quickly and effectively, and iterating fast.
• Share our values, and work in accordance with those values.
• Ability to use GitLab
• Bonus: Strong programming skills as a (former) backend engineer - Preferably with Ruby and/or Go

• Coding infrastructure automation with Chef, Ansible, Terraform, and GitLab CI/CD
• Improving our Prometheus Monitoring or building new metrics
• Helping release managers deploy and fix new versions of GitLab-EE.
• Plan, prepare for, and execute the migration of GitLab.com from virtual machines running on Google Cloud to cloud-native container-based deployments with Kubernetes using Google Kubernetes Engine.
• Develop a relationship with a product group, define their SLAs, share GitLab.com data on those SLAs and improve their reliability

Technical:

1. Deep knowledge in 2 areas of expertise and general knowledge of all areas of expertise. Capable of mentoring Junior in all areas and other SRE in their area of deep knowledge.
2. Contributes small improvements to the GitLab codebase to resolve issues

Execution:

1. Identifies significant projects that result in substantial cost savings or revenue
2. Identifies changes for the product architecture from the reliability, performance and availability perspective with a data driven approach.
3. Proactively work on the efficiency and capacity planning to set clear requirements and reduce the system resources usage to make GitLab cheaper to run for all our customers.
4. Identify parts of the system that do not scale, provides immediate palliative measures and drives long term resolution of these incidents.
5. Identify Service Level Indicators (SLIs) that will align the team to meet the availability and latency objectives.

Collaboration and Communication:

1. Know a domain really well and radiate that knowledge
2. Perform and run blameless RCAs on incidents and outages aggressively looking for answers that will prevent the incident from ever happening again.

Influence and Maturity:

1. Lead Production SREs and Junior Production SREs by setting the example.
2. Show ownership of a major part of the infrastructure.
3. Trusted to de-escalate conflicts inside the team

Site Reliability Engineers have the following job-family performance indicators:

• GitLab.com Availability
• GitLab.com Performance
• Apdex and Error SLO per Service
• Mean Time to Detection
• Mean Time to Resolution
• Mean Time Between Failure
• Mean Time to Production
• Disaster Recovery Time to Recovery