<!DOCTYPE html>
<html>

<head>
  
  <meta charset="utf-8">
  
  <meta name="viewport" content="width=device-width, initial-scale=1">
  
  <title>
     Interrupt Reduction Projects 
  </title>
  
  
  
  
  
  
  
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/normalize.css@8.0.1/normalize.css">
  
  <link href="https://unpkg.com/@primer/css/dist/primer.css" rel="stylesheet" />
  
  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/gitalk@1/dist/gitalk.css">
  <script src="https://cdn.jsdelivr.net/npm/gitalk@1/dist/gitalk.min.js"></script>
  
  <script defer src="https://use.fontawesome.com/releases/v5.3.1/js/all.js"></script>
  <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.1/css/all.min.css"/>
  
  <link rel="stylesheet" href="/css/style.css"/>
  
  <script defer src="/js/tab.js"></script>
  <script defer src="/js/gtalk.js"></script>

</head>

<body>



<div class="Header">
  <div class="Header-item Header-item--full"> </div>
  <div class="Header-item">
    <a href="/blog" class="Header-link">Blog</a>
  </div>
  <div class="Header-item">
    <a href="#" class="Header-link">Digest</a>
  </div>
  
</div>





<div class="mx-auto my-2 col-10 col-md-7 col-lg-6 col-xl-5" >

  
  <div class="p-2">
    <nav aria-label="Breadcrumb">
      <ol>
        <li class="breadcrumb-item"><a href="/">Home</a></li>
        <li class="breadcrumb-item"><a href="/library">library</a></li>
        <li class="breadcrumb-item" aria-current="page"> Interrupt Reduction Projects</li>
      </ol>
    </nav>

    <h1 id="page-title">Interrupt Reduction Projects</h1>
    <div>
      
      </div>
  </div>


  <div class="markdown-body p-2">
  <p>Teams that write and maintain software much decide how to allocate people&rsquo;s time between:</p>
<ul>
<li>On-call/pager response: Immediate response to outages
<ul>
<li>critical to the immediate health of the service, and requires a response with an urgency of minutes.</li>
<li>time-sensitive mitigation with in-depth investigation</li>
<li>changes to prevent recurrence</li>
</ul>
</li>
<li>Tickets and interrupts: Medium-urgency production issues and customer issues
<ul>
<li>have an urgency of days to weeks and usually take between minutes and hours to resolve</li>
<li>use BUG or ticket-tracking tools</li>
</ul>
</li>
<li>Project work: Proactive development and systems/network engineering work
<ul>
<li>have an urgency ranging from weeks to the long backlog of wishlist ideas every
team maintains</li>
<li>This type of work requires multiple days of sustained concentration in order
to facilitate cognitive flow state</li>
</ul>
</li>
</ul>
<p><em>toil</em>: pager response &amp; tickets/interrupts</p>
<p><em>context: google&rsquo;s SRE team setup</em></p>
<p>We assign a primary and secondary on-call in each site, with each site handling 12 hours per day. The primary on-call typically handles most pages</p>
<h2 id="initial-approach">initial approach</h2>
<p><strong>naive approach</strong>
Originally, many teams at Google approached tickets by assigning a primary on-call to handle pager duty, while round-robin
assigning tickets across the team. This setup frequently led to
undesirable outcomes, as engineers couldn’t successfully undertake project work and ticket duty simultaneousl</p>
<p>Some teams moved in the direction of centralization by assigning tickets to the expert with specialized knowledge or recent
experience with a given component. However, this strategy
resulted in uneven load and still disrupted people’s attention,</p>
<p><strong>centralizing tickets</strong></p>
<p>Once we articulated the need to preserve cognitive flow state [3],
a better strategy became clear: we needed to staff a dedicated
ticket rotation</p>
<p>Most SRE teams naïvely implemented this strategy by tasking
the secondary on-caller at each site with a somewhat vague and
meandering directive:</p>
<p>◆ Work on tickets until the queue is empty, filing bugs for small
improvements as you see ways to improve how specific tickets
are handled, or to eliminate them entirely.
◆ See if you can find commonalities in the tickets you just solved,
and do some proactive project-like work to prevent future
tickets</p>
<h2 id="a-better-alternatives">A better Alternatives</h2>
<p>We realized that while we initially focused on centralization and
fairness/symmetry as a goal, we instead should have focused
on maximizing cognitive flow state as a goal in and of itself.  We still tasked team members with identifying and solving commonalities in reactive/
interrupt-driven work. However, we now explicitly allocated this
job, which we’ll refer to as “interrupt reduction project on duty,”
as a separate role from ticket work</p>
<p>Furthermore, assigning one dedicated person to ticket duty at a
time ensures accountability for tickets</p>
<p>In this new model, we rotate the ticket duty and interrupt
reduction project roles between sites once per quarter. In order
to ensure fairness, we rotate people into the ticketeer or interrupt reduction rotation according to on-call rotations (e.g., on a
weekly basis).</p>
<h2 id="implementation-interrupt-reduction-projects">implementation: interrupt reduction projects</h2>
<p>Project Ideas
Project ideas for the interrupt reduction project on duty come
from two main sources:
◆ Current/past ticket handlers who file annoyances into a bug
hotlist as they resolve tickets
◆ Technical Leads (TLs) who have a high-level view of the service</p>
<p>Project Assignment and Handoff</p>
<p>= TL or someone with an overall
view of the service should sort the project list by impact. In the
interest of preserving autonomy among team members, we don’t
suggest assigning projects
= let people choose from top 10 projects</p>
<p>Handling Excess Ticket Load</p>
<p>If your team has more tickets than one person can handle, you
have two options for dealing with the excess load:
◆ Task the interrupt reduction project on duty with tickets for
one day per week.
◆ Decide to relax ticket response expectations for a period of
time, until the work pays of</p>
<h2 id="case-study-ticket-funnel">Case Study: Ticket Funnel</h2>
<p>The number of tickets opened per week
had increased by roughly 50% over the previous year, from 20+
to 30+</p>
<p>Building a simple ticket funnel system
to guide customers to appropriate automation or documentation
was a natural choice for our first interrupt reduction project</p>
<p>, customers now work through
a simple Web interface where they traverse a decision tree. Nonleaf nodes in the tree are represented as a list of questions linking
to child nodes, and leaf nodes do one of the following:</p>
<p>◆ Link to the relevant self-service automation or documentation.
◆ Provide a form that generates a customer ticket.</p>
<ul>
<li>keep updating decision tree</li>
</ul>
<p>Although we’d been talking about undertaking this project for
two years, it ended up only taking about two weeks of work</p>
<p>Once our solution was in place, it took a bit of time to reeducate
customers, who fully embraced the ticket funnel once they discovered its utility
<em>result:</em></p>
<p>◆ Figure 4 shows that the ticket creation rate dropped by roughly
half after we implemented the ticket funnel, from 30+ to 15+
per week. While we’ll never completely eliminate tickets, the
overall trend has most definitely reversed.
◆ Quarterly customer satisfaction surveys reveal an overall happier customer base.
◆ Anecdotally, we see far fewer tickets that can be resolved by
pointing customers at automation or documentation.</p>
<h2 id="applying-this-strategy-in-your-organization">Applying this strategy in your organization</h2>
<p>Figure out a strategy</p>
<ul>
<li>How much time does your team have to work on interrupt
reduction projects?
◆ How much time do you allocate to tickets each week? e.g.,
1 person? 4 people?
◆ How much time do you actually spend working on tickets each
week? e.g., 1 person? 2.5 people?</li>
</ul>
<p>&ndash; Spend less time on large projects.
&ndash; Spend less time on tickets.</p>
<p>so make sure to clearly communicate the motivations
for your actions, expected disruptions, timeline, and expected
benefits</p>
<ul>
<li>Who will work on the interrupt reduction projects?
Make interrupt reduction projects part of the normal ticket duty
rotation, which we assume is fairly scheduled and distributed.</li>
</ul>
<p>It’s important that the work be seen as valuable
by the team. Choose people who are enthusiastic or particularly
productive in order to create a good initial impression.</p>
<ul>
<li>How will you convince your team to adopt this approach?</li>
</ul>
<p>◆ Each team member will spend 50% less time on tickets.
◆ Completing a small interrupt reduction project quickly and
seeing immediate impact creates a good deal of satisfaction.
◆ Interrupt reduction projects will improve the systems your
team uses on a daily basis.
◆ Eventually, your ticket load will decrease. The remaining
tickets will be issues that actually merit investigation, and
improvements to tooling will make some tickets easier to deal
with than they were previously</p>
<ul>
<li>How will you safeguard the time allocated for interrupt
reduction projects?</li>
</ul>
<p>Therefore, creating
accountability around interrupt reduction projects is important.
You might accomplish this by publishing objectives around
these projects, reporting on them regularly, tracking them,
or announcing interrupt reduction project velocity in regular
reports.</p>
<h2 id="suggested-interrupt-reduction-project">Suggested Interrupt Reduction Project</h2>
<ul>
<li>Identify the Sources of Your Toil: use metadata (cause, impact, time to fix, etc) to tickets to help determine recurring issues and your biggests time sinks</li>
<li>Improve Your Documentation0s: It’s much easier to handle a ticket if the process is documented, and documentation is a good first step towards automating a process: if customers can find and use good documentations, they won&rsquo;t need to open a ticket (provide a template (customer facing, internal procedures, etc) to make getting started easier)</li>
<li>Pick the 10 Most Annoying Small Bugs and Fix Them: creating lists of bugs for the rough edges, shortcomings, and difficulties encountered in the course of everyday work—otherwise those problems will never be fixed.  Consider choosing bugs related to one or two systems,  rather than scattered small improvements, and common to customers, so that progress is significant and noticeable.</li>
</ul>
<h2 id="take-away">Take away</h2>
<p>implement some type of stratagy to proactively reduce tickets &ndash;&gt; to control to number of tickets.
ensure that handling tickets doesn&rsquo;t constantly disrupt the cognitive flow state of your engineers.</p>
<p>recommended components:</p>
<ul>
<li>Centralize your ticket load, either onto engineers who are
already expecting interruptions (e.g., primary or secondary oncall) or to a dedicated ticket duty rotation.
◆ Track ideas for small interrupt reduction projects that will
reduce toil.
◆ Put a framework in place that reserves time for small (20–30
hours) proactive projects.
◆ Treat tickets and small proactive interrupt reduction projects
as separate rotations, distributed among team members and
sites on a regular basis.</li>
</ul>

  </div>
  
  
<div id="gitalk-container"></div>

<div class="Header border-top mt-2 bg-gray-light text-gray-dark">
  <div class="Header-item">
    <span ><strong>Thinking as Algorithm </strong> By <em>Feitong Yang</em> | 2014 - 2020</span>
  </div>
  <div class="Header-item Header-item--full"> </div>
  <div class="Header-item">
    <a href="/about/">About</a>
  </div>
  <div class="Header-item">
    <a href="/logging/">Logging</a>
  </div>
</div>

</body>

</html>


</div>

