20221123-1230 - HangOps conversation discussing some of the origins of Google SRE

https://www.usenix.org/publications/loginonline/oncall-equal-opportunity-waste-time

204 replies

this is a weird article?

It sort of meanders around the fact that a lot of places just retitled their Ops team to SRE and asked them to do more “devops”

nhruby 5 days ago

It also misses a lot in the “The Fetishization of Incident Response” – books could be written on this and yet, it gets a quick pass with no point other than a clickbait headline

pwnguin 5 days ago

is it poor form to say i havent read it yet and just saw it elsewhere and cross-posted it here as a reminder to read it tonight

nhruby 5 days ago

haha, no, it happens

nhruby 5 days ago

but uhh, maybe skip it and just read some funny tweets that troll Elon instead

pwnguin 5 days ago

lot of places just retitled their Ops team to SRE

im starting to think that everyone would understand the role, and the gap between them and google, if it had been titled “Revenue Reliability Engineer”

nhruby 5 days ago

Given the convo I just had this morning, yes, I agree with that

rnelson0 5 days ago

The weird thing is this person worked at Google and references a lot of the SRE practices there, but I agree that it seems a “relabeled Ops team” article. Was Google SRE at the time actually much different than the book described or something?

rnelson0 5 days ago

There’s a lot of good stuff there tho for misnamed Ops teams!

pwnguin 5 days ago

like, https://stripe.com/guides/atlas/business-of-saas#the-fundamental-equation-of-saas exists and SRE teams should be able to match their work to improving many of those numbers

pwnguin 5 days ago

Screenshot 2022-11-18 at 3.20.50 PM.png

[

Screenshot 2022-11-18 at 3.20.50 PM.png

](https://files.slack.com/files-pri/T047L58T5-F04BL2MDYE9/screenshot_2022-11-18_at_3.20.50_pm.png)

rnelson0 5 days ago

I struggle with that in healthcare and infrastructure SRE. Yeah we sign up providers, but we get paid based on billing events. All I can do is work on the costs of the overall infra. The rest aren’t something we can really influence, and trying to figure out which billing events might be worth working on is a guess (spent a lot of money decreasing costs in Covid stuff just in time for states to end testing!)

pwnguin 4 days ago

sure, its hard to attribute churn to specific causes like “that 30 minute outage last week”

kurin 4 days ago

what are the units on that

pwnguin 4 days ago

dollars?

pwnguin 4 days ago

ARPU does the magic for unit analysis

pwnguin 4 days ago

“say what you will about capitalists, at least its an ethos”

rnelson0 4 days ago

Lol yeah we do our best but units is just the first of many issues.

rnelson0 4 days ago

We can do churn for things like missing features leading to clients leaving, just not THEIR clients.

pwnguin 4 days ago

what are the units on that

oh, if the question is churn, it would basically be # of cancelled paying accounts (in the time window) (edited)

pwnguin 4 days ago

and sure, healthcare is like, full of “other people’s money” problems

rnelson0 4 days ago

I guess we can probably quantify all the numbers on that equation, it’s just the relation to the work we do that’s hard.

pwnguin 4 days ago

acquisition: is marketing working, do they have budget to ramp inbound customers up. is google indexing our site successfully? conversion: are our landing pages working? does the signup flow work? ARPU: do our features work well enough to pay for? should we add a new pricing tier with dedicated support and SLAs? churn: are we missing SLOs? is that leading to cancelled accounts? can we make the site faster? (edited)

pwnguin 4 days ago

arguably cost savings projects wont show up on this analysis unless you present it as a cost vs latency tradeoff

nhruby 4 days ago

cost savings influence burn rate and cash on hand, they are generally on the other set of equations from revenue (whihc you also need to be aware of)

Also sent to the channel

knyar 2 days ago

It’s a good article, but it could definitely use a more detailed intro to provide a bit more context about the problem space and the intended audience.Dave has basically built Google’s storage SRE org in Dublin, which (as I understand) was initially an outlier in terms of how much operational load they had: a bunch of products with internal customers (what people call “platform” nowadays) that was very toil-heavy. He has a lot of expertise on how to run teams like that (see Bad Machinery paper that predates the SRE book) and this article mostly focuses on how these sort of teams are perceived by the “Dev partners” (at Google most SRE teams are “funded” by the development org, and for each additional headcount leadership for each product basically makes a decision on whether to invest into Dev or SRE).I don’t think Google had an “Ops team” to rename into SRE, but SRE at Google was created as an ops team, just a one that is staffed by software engineers (SRE is what happens when you ask a software engineer to design an operations team). I think it’s still true today: majority of services are supported by their respective dev teams (DevOps model), but the most critical services also have dedicated SRE teams that do oncall + reliability work.In the last ~5 years I believe many senior SRE folks have been re-evaluating whether the ops aspect of SRE is critical/important, and what the value prop of SRE without oncall is (as alluded in this article, Dev leaders often see SRE as a way to isolate oncall/toil from the main dev team). Some folks (like Yaniv) suggest that SRE could focus primarily on reliability analysis, but this basically requires changing the core skill set for SREs from “software engineering + systems + networking” to “software engineering + data science”.So that article would probably make little sense if you don’t have a large SRE organization with oncall/ops being core to the identity and funding model of the org. Most companies don’t have these problems, and that’s a good thing. (edited)

Alex 1 day ago

@knyar There was a sysadmin org at Google that existed alongside SRE for a long time. They were indeed at one point told they were now also SRE in a wholesale renaming process. (edited)

Alex 1 day ago

Many of them based in Dublin even.

Alex 1 day ago

This was before my time so I can’t speak to all of the details, but I do know some people were scared about the change because they didn’t think they could code well enough to be “SRE” and thought they’d end up performing badly in terms of their renamed role.

pugmajere 1 day ago

SRE @ Google began as a group of software engineers who realized that nobody was actually making sure that production ran. That things stayed running. This group began as “production-oncall” and, I think, production software (as one group).Later, those two split, the organization was renamed to SRE, Ads SRE was created in a panic to be able to IPO (Google was, I think, the first company to IPO post-SOX and therefore, one of the very first companies to have to meet all the requirements).Ads SRE was built up by stealing a bunch of people from “sysops”. Sysops was a traditional ops organization. When I started in 2006, people on that team were doing short (~8 hour) oncall shifts, infrequently, in a rotation of something like 100+ people. The odds of being paged for a system you worked on were abysmal, so it was all reading docs and such.Dave was the second SRE hired in Dublin, IIRC, as part of ramping up a group to be SRE for Gmail (the 3rd SRE team built.)The group as “production-oncall” eventually picked up a bunch of transfers from sysops, split into Search SRE and Traffic SRE as the scope got bigger.In parallel with some of this, Sysops was entirely focused on the corp side of things, and production was being neglected, so another ops team was built up, called “clusterops”. This was a team of very junior people, asked to do all the crap things of keeping GFS running, which was … not a fun job at that point. (Think: thousands of machines, ~zero automation for taking bad machines out of the cluster.)Eventually, clusterops was reformed into 3 SRE teams, with better support and staffing and formed the first real “storage” SRE team.So, SRE at Google had at least 3 separate origins, 2 of which were heavily ops-practice based, though the renaming into SRE was a strong signal that nobody was liking that mode.

pugmajere 1 day ago

The way headcount is done in SRE @ Google is a major part of SRE’s disfunction there. It’s very transactional: “You need more headcount? What service are you going to run so my developers don’t have to?“This ends up with SRE teams owning a dozen loosely related systems and only really understanding how a few of them work. Usually, this ends up meaning that people understand the noisiest systems, and not the most important ones, which is a fun situation.

pugmajere 1 day ago

As to the more modern “Storage” SRE teams … when GCP was really growing and trying to become relevant in the Cloud market, the Storage SRE teams somehow ended up as second tier support (maybe 3rd, I’m a little fuzzy on that). This was incredibly toil filled for them, AIUI, since everyone who did something like, “I made a bad Spanner schema and I don’t know why my performance sucks” would get to them. Eventually, this was improved by realizing that SRE isn’t that kind of organization and much of that support was shifted to the dev team. (This sets up appropriate incentives much better.)

Alex 1 day ago

Corp Sysops was also at one point pretty much redefined as Corp SRE.

Alex 1 day ago

Corp here being the demarcation point between google’s “production/customer facing” services and internal services relied upon by Google employees more directly.

pugmajere 1 day ago

Oh, I suppose I should clarify that most of what I wrote there is about the production side of SRE.

Alex 1 day ago

Think financial databases, legal databases, the actual desktop and laptop fleet, etc.

pugmajere 1 day ago

Though the line got pretty blurry as the Corp side was empowered to use production for stuff.

Alex 1 day ago

Yeah, just adding a bit more flavor since I spent time in Corp.

Alex 1 day ago

For anyone following along a home.

pugmajere 1 day ago

I think that Corp Sysops had been acting more like SRE, out of self-preservation if nothing else, before that rename happened…. but that might just be my impression from a distance.

pugmajere 1 day ago

(btw, <== rda@)

Alex 1 day ago

(ahidalgo@ here – Ads SRE, Managed Systems, Prodmon, CRE) (edited)

knyar 1 day ago

Oh nice, thanks for sharing the actual history behind this, folks. I only joined in 2014 at which point SRE was already very large and reached the level of maturity described in the first book.So I guess we could say that the comment about this article coming from a “relabeled Ops team” perspective is actually accurate since that’s exactly what Google did?

pugmajere 1 day ago

It’s… well, it’s really just more complicated than that, but yeah, it’s not wrong, just not where the first group of people who were SREs came from.