user
John, as Senior SRE at MMT Digital, how did you balance the demands of developing new containerised applications with the responsibility of maintaining the stability of existing systems?
assistant
This is a classic SRE challenge: balancing innovation with stability. At MMT Digital, from August 2022 to August 2024, I tackled this by treating development and operations as equally important parts of the same system. I'd implement some code, and then be responsible for managing the code right from the start.I established clear service level objectives (SLOs) for all systems, both new and existing. SLOs are crucial in measuring success. If you can measure it, then you can improve it. These SLOs defined the acceptable levels of performance, availability, and reliability. We'd also keep everyone in the loop, such as the project stakeholders. For new applications, I'd build in observability from the start. This meant instrumenting the code with metrics, logs, and traces to provide real-time insights into its performance. A tool such as Datadog can be easily integrated with monitoring. This allowed us to identify and address potential issues early in the development process. I would get all of the team involved with designing and maintaining.For existing systems, I'd focus on continuous improvement. This involved identifying and addressing bottlenecks, automating manual tasks, and improving the overall resilience of the infrastructure. There was never an opportunity to sit back and not improve code. The idea was to incrementally improve all aspects of the system, and to slowly improve them every day.I also fostered a culture of shared responsibility between development and operations teams. This meant encouraging developers to participate in on-call rotations and operations engineers to contribute to code development. This collaboration helped to break down silos and improve communication. The team was fully aligned with the direction of the tasks at hand.Ultimately, balancing development and operations is about finding the right trade-offs. There's no one-size-fits-all solution. It requires a deep understanding of the business, the technology, and the team. As Senior SRE at MMT Digital, I balanced developing new containerised applications with maintaining system stability by establishing clear SLOs, building in observability from the start, focusing on continuous improvement for existing systems, and fostering a culture of shared responsibility between development and operations teams.