From Ground Zero to Production: Go's Journey at Google
From Ground Zero to Production: Go's Journey at Google
Recently, Jeremy
Mason
and Sameer
Ajmani
wrote about the saga to make Go one of Google’s internal languages. Go
is currently the 8th most popular programming language in the
world
and it is still growing, so people are interested to learn about the
early days and how we got here.I thought I would write about the perspective of an SRE, framework
developer and early adopter. All information I’m sharing is related to
systems that Google already documented publicly, so I don’t think I’m
revealing any secrets. There are some important parts of this story
(e.g: envelope) that I haven’t seen mentioned elsewhere so I won’t
discuss them.Breaking the Ice: My Introduction to Go Programming at GoogleI started looking at Go before it was released publicly, and when it
launched, I became an instant fan and an early user inside Google. I
loved its simplicity.I worked a bit on core libraries and was active in the community,
helping users in the go-nuts mailing list early on and writing
open-source libraries. Later on, I helped lead the Go Meetup in
Seattle and co-organized a beloved
conference, Go
Northwest.To the best of my knowledge, I wrote the first production-critical tool
and, later on, the first user-facing service in Go at Google.The first one was a service used to monitor the health of
Google+’s
Bigtable servers. That was one
of my jobs as a SRE. Bigtable had detailed internal stats about the
performance of each tablet, but sometimes we needed to understand why
a certain tablet was so overloaded and what was happening elsewhere in
the system, so we could understand the root causes. We needed to collect
that data over time and analyse it. So I built a crawler that would
inspect many thousands of servers and present detailed stats in a global
dashboard.In 2011, Andrew Gerrand gave an interview to The
Register
where he mentioned this work. He confirmed to me at the time that this
referred to my project. I was thrilled! He said this in the interview:“Google has people who administer apps and services, and they need
to, say, write tools that say scrape a few thousand machines statuses
and aggregate the data,” he says. “Previously, these operations people
would write these in Python, but they’re finding that Go is much
faster in terms of performance and time to actually write the code.”Go was indeed faster to run and faster to write. Most importantly, it
felt fun to use. It made me more productive, so I got hooked!Low-level libraries: node authentication and RPCsWhen Go started, it couldn’t talk to Google’s internal infrastructure.First, the team had to build a protocol-buffers-based stubby
RPC
system. This required implementing
LOAS
to encrypt and authenticate communication with remote nodes, and
Chubby
for name resolution (similar to etcd,
used in kubernetes).Stubby and Chubby are notoriously complex. Stubby requires a complicated
state machine for managing connections but most of the heavy lifting is
done by Chubby, which needs to provide a consistent view of the world
even when
Borg
nodes run out of CPU, or are temporarily disconnected from the network
because someone is running a MapReduce and eating all your rack’s switch
bandwidth. It’s easy to end-up with deadlocks or reliability issues.Due to Hyrum’s law, which states that all
observable behaviors of your system will be depended on by somebody, the
team had to make sure to match the exact behaviour expected by the
existing production network and watch out for corner cases. For example,
healthchecking is notoriously easy to get wrong, and should not be too
strict otherwise they leave the door open for cascading failures when a
part of the network is temporarily overloaded or disconnected from the
other part. Other tricky distributed system features that had to be
implemented, such as backend subsetting and load-balancing. We needed to
diagnose when things went wrong, so logging and metrics libraries were
added early on.In order to find which host:port to talk to, services used Chubby for
name resolution. It worked as a fully-consistent storage system for
small amounts of data, and its most used feature was to resolve
BNS addresses -
similar to what you would see today with etcd in kubernetes.Systems sent and received data to and from other services using the
Stubby protocol. In Stubby (like in gRPC), messages
were encoded using the protocol-buffers wire format. Creating
protocol-buffers payload at runtime using reflection would be too slow
and resource intensive. Engineers would also miss the lack of feedback
from a strong-typed system. For those reasons, Google used generated
code libraries for all languages. Luckily, protocol buffers are
language-agnostic. The team just had to write
Blaze extensions to
the existing build system logic and, voila, we had high-quality client
library code for all internal RPC services.Curiously, there is a small incremental build time cost to generate code
for another language, and Google had many, many thousands of RPC
services. Therefore, it was decided that owners of each RPC service had
to opt-in to allow the build system to generate Go code for their
particular service. Although a little bit bureacratic, over time we saw
thousands of CLs (google’s equivalent Pull Requests) flying around to
add Go to the set of generated code for each service. This served as a
silly but fun measure of progress for our community since we could count
the number of instances of the “enable Go” flags in the code base.Influencing the Global Master Selection and Executing Bigtable DrainageAs an early adopter of those early libraries and an engineer focused in
production systems, I was able to learn how internal systems worked. I
helped debug and fix many weird problems. Over time, I acquired the
confidence to build systems to automate operational SRE work. Noticing
that most of the user-facing outages on our services happened in the
storage layer (Bigtable or Colossus), I had the idea to create a control
system that would monitor the health of Bigtable partitions and
carefully drain them in
GSLB when it detected
problems. At the time, when an outage occurred, an SRE would get paged
and, after confirming it was a storage issue, they would simply drain
the cluster and go back to sleep.I wanted to replace this manual whackamole with a proper control system.
Draining traffic could lead to cascading failures so it was a dangerous
operation. At the time, most SREs did not want to take this kind of risk
with an automated system. Luckily, I had a good team. They carefully
reviewed my proposal, provided lots of feedback about potential failure
modes, and we eventually came up with a design that we were confident
enough with. We needed to carefully aggregate information from different
monitor systems (which could fail or provide incorrect data), use the
global load balancer to safely traffic away from a cluster, then finally
open a ticket in
Buganizer
for the oncall SRE to clean-up during work hours.The system needed multiple replicas to be always on to react to an
outage, but it was critical that a single replica remained live at a
time. To support that, I wrote a global “master election” library for Go
that would ensure a single replica of the system would be active at a
time. It used the global chubby lock service to provide a high-level
library to tell the application to start operating or shut itself down
if it can’t prove that we hold the “global lock”.To support this work, I also wrote minor utilities here and there, and
worked with the Go team to fix bugs. I reported issues I’d find, and
they fixed them.At the time, the Go’s team focus was on external users. All their
attention was on releasing Go 1.0. It was a small team with few
resources, but their “secret sauce” is that they were exceptional
engineers and the team was very productive. Somehow, they supported
internal users very well even though their time was so limited. The
internal mailing list was very active with googlers mostly playing
around with Go for side-projects, but the Go team adopted very robust
internal processes to make things run smoothly. They reviewed everyone’s
code carefully and helped build a strong internal code quality culture.
Whenever they released a new candidate version of Go, they would rebuild
all internal projects with the new version and re-run our tests to make
sure things were OK. They always did things the right way.Initial Insights from JID Proxy Deployment in ProductionA few months later I wrote the first user-facing service in Go at
Google. By user-facing I mean that if it stopped working many
user-facing products would stop working. It was a simple RPC service,
but it was used by all Google messaging services.This service converted data to and from JID
format based on internal
user ids obtained from another RPC service. The service was simple but
it was massive, doing hundreds of thousands of requests per second at
the time. It was critical to the core of Google’s messaging services
that powered Android, Hangouts and other products.This migration was a very important test bed for Go at Google.
Critically, it gave us an incredible base to compare the performance of
Go vis-a-vis our other production languages - specifically Java. This
service was replacing a Java-based one that was difficult to maintain
(not because of Java but for other reasons) so we ran both of them at
the same time with real production traffic and compared their
performance closely.We learned important lessons from that first large-scale experiment: Go
used more CPU cores than Java to serve the same traffic, but the garbage
collection (GC) pauses were extremely short. As an SRE that worked hard
to reduce GC pauses to improve tail latency in user-facing services,
that was very promising to see. The Go team was happy with that result
but they weren’t surprised: Go was just doing what it was designed to
do!In fact, years later when the SRE leadership officially reviewed Go’s
readiness for
production
and asked the Go team to ensure Go had good GC performance, I think it
was largely pro-forma. Go had proved early on that Go had exceptional GC
performance, and it kept getting better over the years.Encountering Absent In-house LibrariesIn those early days, before
flywheel,
before the dl.google.com service, before
Vitess, Go was ignored by most of the engineers at
Google. If someone wanted to ship a product to users, they would first
have to write the basic building blocks that let them connect to other
services at Google. That was a non-starter for most.The lower-level libraries for the lock service  (chubby) and the RPC
system (stubby) popped up relatively quickly (again, the Go team was
extremely good), the most important libraries at Google were the
interfaces with our storage systems: Bigtable, Megastore, Spanner,
Colossus. If you wanted to read or write data, you basically couldn’t
use Go yet. But, slowly, the Go team, sometimes in partnership with core
infrastructure teams, started to tackle this challenge.One by one, they eventually created libraries for Bigtable, Colossus and
even Spanner (not Megastore, since it was largely a library that was
replaced by Spanner). That was a major achievement.Usage at Google was still limited but our community was growing. I gave
the first official Introduction to Go Programming class at
Google and helped people in Zurich find interesting projects to work in
Go. Around this time I finally got “readability” in Go, and later joined
the Go readability team.The need for Site Reliability Engineers to guide application functionalityThe other thing missing in Go were production-related features that we
learned over the years were necessary for production teams. That is, if
you want to run large-scale systems without constantly being in ops
mode, fighting fires.Whenever an outage occurs and we diagnose the root causes, over time we
learn the weaknesses in the system that should be improved. The goal is
to reduce outages and operational overhead. Often times, to make the
system more reliable, we have to make changes to the application
runtime. It’s hard appreciate the depth of details we need to observe
and control a system to truly make it reliable.For example, we need to make sure that, in addition to logging incoming
requests, applications should also log details about outgoing requests
that were involved in that operation. This way, we can pinpoint with
certainty that, say, our “CallBob” service became slow at 11:34am
because of increased latency to the “FindAddress” calls. When we operate
large-scale systems, we can’t be satisfied with guess work and weak
correlations. There are too many red herrings and “optimistic” root
cause assignment. We need to higher certainty about the causes: we want
to see that the specific requests that failed did experience
high-latency, and exclude other explanations (i.e: incoming requests
that did not trigger a slow FindAddress call shouldn’t be failing).Similarly, over the years we noticed that a large chunk of SRE time was
spent coordinating between teams to determine the exact number of
connections and requests per second that one service should send to
another, and how those connections should be established exactly. For
example, if several services want to connect to a backend, we want to be
smart about which exact nodes are connecting to which other nodes. This
is called backend subsetting. It needs to be carefully adjusted
considering the overall system health and not just of one node or a node
pair, but of the entire network. Excessively large subsets cause too
much resource usage, and excessively small subsets can lead to load
imbalances. For this reason, over time, SRE teams started to help
maintain the client libraries used to talk to their service, so they
could instrument what was happening and retain some control over how
other nodes talked to their systems.The model where SRE co-owned the client libraries worked very well in
pratice, and over time we learned that it was a great idea to also add
traffic and load management to these libraries.What do you do to incoming RPCs when your system is starting to
overload?Should you hold those requests in a queue, or immediately reject
them?What metrics should you use to determine that your system is
overloaded?How can you avoid entering a cascading failure when too many parts
of the system think they are overloaded?Alejo Forero Cuervo wrote about the lessons learned in the SRE book
chapter Handling
Overload, it’s worth
reading. One by one, we added careful logic to the libraries to
automatically set these parameters based on experience and internal
sensors.In The Evolving SRE Engagement
Model, my
former colleague Ashish Bhambhani and my former boss Acacio Cruz
explained that we eventually evolved the SRE engagement model to include
work and adoption of Server Frameworks. This model allowed SREs to
directly influence the behaviour of systems in nuanced areas that
benefitted from our extensive field experience.My SRE team and I wanted to bring these features to Go, but they were
too exotic and specialized for the Go team to handle. I set up a 20%
project (which later became a full-time project) and recruited a bunch
of experienced engineers that wanted to contribute. I flew to New York
and met with a very awesome Go team member and we worked together to
build a roadmap for a “server framework” in Go.The Go team was reluctant at first with our approach. The whole
“framework” concept was a little bit of a red flag for them. This could
have become a religous war of sorts, but the Go team took the time to
explain in detail why they were concerned. Sameer specifically has an
uncanny ability to reflect and explain why he thinks something works
better one way vs another, in technical terms.Sameer felt strongly that Go should not have inconsistent developer
experiences, internal vs external, with or without a “framework”. It
would be a disservice to the internal Go community if there were
different ways to build Go applications at Google. In agreement with his
concern, our ragtag team of 20%-ers went through great lengths to ensure
our “framework” felt more like just another library, not a framework,
and that it would not introduce a different programming model for Go.
The goal was to introduce our reliability features with a simple library
import. If you wrapped your Go HTTP or Stubby server with our libraries,
everything would look the same in code, but you magically got logging,
instrumentation, load-shedding, traffic management and even per-request
experimentation support, out of the box.In order to create this magic library that made services better, we had to
make significant changes to Google’s internal RPC libraries and even to the
build system - to enable our framework team to create arbitrary “extensions”
to the RPC systems that would operate seamlessly without significant performance
overhead when receiving and sending requests.The result was worth it. It worked really well. Our project made services significantly
easier to manage, without imposing a programming style that was different from what the
Go team wanted. We called it a server “toolkit” to avoid confusion, and it the 
became the Right Way to build production-ready systems at Google. People often
cite our internal server framework in their LinkedIn profile :). It was
called Goa, not to be confused with the unrelated external Goa
framework. Here’s an example from someone’s LinkedIn
profile:With its production-readiness features, our Go toolkit removed a major
roadblock for Go’s internal growth. Engineers could now be confident that their
Go projects would perform as well as, and be as debuggable as, their older Java
and C++ siblings. That said, growth didn’t quite happen yet. Go needed a killer
use-case to become popular at Google.Go’s Adoption Across Several SRE TeamsAt the time, the SRE team I was part of was a special place at Google,
the Social SRE team. We had great engineers and exceptional management
in both SWE and SRE. So we were able to do things the right way. Some
SRE teams were chasing their tails fighting fires, but we had the luxury
of engineering things properly. That created a virtuous cycle where we
continuously fixed problems before they became big, which means we had
time to optimize operations further, and so on.As a result, our SRE team wrote a lot of useful code. Like my fellow
senior engineers, I helped people find things to do, so I helped
kickstart many early production-related tools in Go. One of those tools
would automatically, and safely, remove traffic away from an entire
Bigtable cluster if it noticed something wrong.There were also other projects, in Java and C++, related to traffic and
load management, led by other senior engineers. This innovative
environment attracted talent and we continued to deliver good results,
so our SRE team grew.Our engineering director Acacio
Cruz (responsible for many of the
positive things happening with our team, along with his peers in
Mountain View) was very attuned to engineering efficiency: are we using
our engineering time for the most impactful things?  He understood that
there is efficiency in standardization, and he saw that our engineers
were happy and productive. He had the idea of pushing for Go to become
the tool of choice for any automation within our team. The proposal was
to avoid Python and use Go to write production tooling. To my surprise,
none of my teammates objected. That accelerated the usage of Go within
our social SRE team, and soon folks outside of our area took notice.The core libraries, server framework, the successful production tools
and the social SRE standardization around Go - they all contributed to a
changing perception that Go was becoming a serious language at Google.At the same time, SREs had seen a couple of generations of tools written
in Python that worked extremely well but became very difficult to
maintain over time. Google SREs enjoyed Python, we wrote a ton of Python
code. Unfortunately, at the time, the lack of types and compile-time
syntax error checks caused many hard-to-fix issues:When you work on a project that someone else started, that project
may or may not have good test coverage. It’s difficult to add tests
for code you didn’t write. You don’t really know what is being used
and how. So you end up testing too many things or testing too
little. In production-critical tools we can’t take risks when making
changes.At the time, people generally wrote code in one moment and ran the
tests in another moment. If you only realize you have syntax errors
when you run tests, maybe you already context switched to doing
something else, so now you have to go back and fix it. That wastes
time and adds uncertainty.As more and more SREs started to write automation in Go, it became clear
that those teams were happy and productive and were less likely to get
stuck with hard-to-maintain code. It started to dawn on people that Go
projects are easier to evolve and maintain, and that it was not just an
effect of those projects being newer, cleaner or just better engineered.SRE leadership noticed this effect and decided to take action and
communicate very broadly within the organization: SRE teams should
preferably use Go for production-related
projects, and avoid
Python. I don’t know if this is now seen at Google as something
dictatorial, but at the time I think it just felt like good org-wide
communication and decision-making.Go Production Platform and explosive growthThings accelerated quickly after that. We created a production platform
that had strong support for Go since the early days and replaced a lot
of boilerplate configuration and repetitive procedures with high-level
abstractions. This platform saw strong growth and eventually other
platforms surfaced. Go and our server framework became ubiquitous. I
eventually left Google but I remember those days with joy.While I was just a user of the language, the experience of watching a
project go  from zero to being a top-10 programming language has taught
me a lot. I could see with my own eyes that a strong team, surrounded by
a strong community, can really make big things.Observing Go’s Ascend to ProminenceMy time working with Go programming at Google has been a game-changer,
giving me a great understanding of the project’s technical side and how
a world-famous team operates. As the project went on, I could clearly
see how Go can make project and team scaling easier.Go’s emphasis on minimalistic design facilitated uniform coding, making
it easy to integrate new programmers into the project, a feature
particularly useful in projects on a tight schedule. As the project
grew, new libraries and toolkits also emerged, increasing its popularity
and facilitating its adoption by several big tech companies including
Apple, Facebook, and Docker.Despite Rust having an extensive range of features, the widespread
acceptance of Go across various industries shows that powerful software
doesn’t necessarily need to be complicated.Looking back, it’s clear that while our journey was filled with
challenges, each twist and turn, each adjustment and advancement, was
key to shaping today’s Go. As the community moves forward, I am excited to see
where we Go next.Go gopher was designed by Renee French and licensed under the Creative Commons 3.0 Attributions license.