1 of 45

Move Fast & Don't Break Things

Ankit Mehta, Google

2 of 45

Ankit Mehta, Google

Move Fast & Don’t Break Things

GTAC 2014

Welcome to GTAC 2014. This time we wanted to make this a bright and sunny GTAC. so Seattle was the obvious choice

Last year it was in NY and hugely successful. It was thematic and this year we elected to keep it open as some felt constrained by a theme. Last year there were 500 submissions for talks and this year over 1500 so this continues to be something the community is highly engaged with.

1524 applicants (238 Googler, 1286 external) - about 3x as many as last year

250 attendees selected (including speakers)

Over the next 2 days; prepare to be amazed by how much people can talk and even more amazed by how much you can listen to.

What excites me about GTAC: Booze. though well at Google; we get lots of it

On a more serious note; I continue to be amazed every year with all the stuff folks are doing in the industry and the huge variance if how we approach testing. Google especially has a different take on Testing and it immensely helps us share with all of you what we do and learn from you all wrt how you guys approach testing.

3 of 45

Why am I talking @ GTAC

Sharing my experience of a decade @ Google in Test Engineering
Passionate about balance between velocity and quality
Taking an opportunity to share Google’s take at balancing velocity and quality

Interesting side projects @ Google:

MAD (Millions of Automated Documents)

Scale Google’s Bug database

Surveytool

Microprocessor controlled pick n place manipulator (school)

Move Fast & Don’t Break Things

MAD (Millions of Automated Documents): had to create millions of docs to stress test Google Search Appliance and Desktop. automated word excel AIM etc.

Scale Google’s bug DB: Whoever was maintaining it; one fine day left it under my desk as my team/product accounted for 1/5th of the bugs @ Google. and then I sped it up to 10x (12x to be exact) via drum roll.. moving it to a datacenter from under my desk :)

Thankfully; Google replaced that with a better state of the art bug database. so googlers weren’t left to deal with bugs I encountered while backing up db, updating packages in the bugs DB. Yes. we must do deeper..

Surveytool. My learning wrt how not to scale products. 20% project. App engine, Google Forms and Google+ Polls didn’t exist back then. Much to my initial joy and later dismay; it took on too much traction; that one year Google HR deduced to use it for Google’s annual survey. Within mins of the email going out; it melted.

I spent the next 2 days recovering all data happily retired the tool a month after that :)

Before Google worked on a Robot with a buddy; Learned that software is much easier than hardware to mold.

4 of 45

Test Engineering

“what is our purpose?”

Move Fast & Don’t Break Things

Build world class infrastructure to launch high quality innovative products fast that delight our users

5 of 45

Google from the outside..

Move Fast & Don’t Break Things

6 of 45

Google from the inside…

Move Fast & Don’t Break Things

7 of 45

More code

30K check-ins per day
A check-in every 3s!

Move Fast & Don’t Break Things

8 of 45

More releases

2x more releases

Move Fast & Don’t Break Things

9 of 45

State of releases

Releases have long cycles; hence everyone wants in
Lack of discipline/time pressure leads to regressions and further delays
No way to isolate issue and hence further delay and work around it

Move Fast & Don’t Break Things

10 of 45

Moving Fast is Good!

Innovate
Address flaws quickly
Better productivity
Better Code Health

Move Fast & Don’t Break Things

Innovate: to build products quickly and iterate on the feedback and make them better. We all know what happens to companies that don’t innovate and do it quickly; especially in the tech industry. Innovation is now not only the sexy thing to do; it is necessary for survival . It is also a great tool to attract the best talent in the industry and retain it. Go back a decade and remember yourself in a similar conference. you were sitting in your chair without a smartphone in your end and at best checking your emails over and over again. now you can play Candy crush while you are listening. not that I am recommending you do that over GTAC :)

Address flaws quickly: Address critical bugs quickly and painlessly. This is a chicken and egg kinda thing. you can’t move fast without this ability and you can’t do this without moving fast. at some point; you just have to take a leap of faith; believe in yourself and your team and take the plunge. This is highly addictive; once you know you can react this quickly; you have the discipline to ensure not too many things are slipped under the rug and shipped to prod.

Better productivity: Automation is a key ingredient to this and that in turn helps rapid development. if you write code that doesn’t compile your IDE will complain as it continuously runs tests in the background. if you write code that breaks an existing test; it will complain; if you write code without a test; the presubmit will complain and so will the human reviewer. We now have tools @ Google which show incremental code coverage numbers for every cl.

Better Code Health: Folks don’t try to slide things in to catch a release as there is one going every day! if you got a key integration wrong; you find out about it immediately in local development or worse case in fishfood(more abt that later). and this in the long term really helps a product survive longer. Typically IMO every software has a 15 year cycle: 0-2 year build time(depending on the complexity); a 7 year shelf live. and then it’s in maintenance mode for 3 more years and then in derpreciation/migration mode for the next 3. Focusing on Code Health from the start will help made this ride smoother; especially the later part of the ride :)

11 of 45

… But Breaking things isn’t

User trust/satisfaction
Uphold the brand
Launch products
Set a high bar

Move Fast & Don’t Break Things

12 of 45

My Testing Philosophy

What many teams do

How it should be done

Move Fast & Don’t Break Things

Maintenance

Slower tests

Flakiness

Alister is here in the crowd.

Very often; when folks talk about automation; they think about UI automation. Very often; that’s a hard thing to do as it has lots of moving pieces and lots of infrastructures in the test stack.

I know a team is doing too much UI automation when they are finding NPEs via their UI automated tests.

Manual Session based testing is a fancy way of saying exploratory testing. I strongly believe that if testers are executing a set of test cases over and over again; we should automate them; unless in rare circumstances; the ROI to automate them isn’t there.

Exploratory testing is extremely valuable and I keep wishing my teams do more of this. As your automation can only hit so many code paths and it keeps hitting them over and over; whereas; exploratory testing helps expose bugs in code paths which could have never been hit before. For e.g. in Google+ Photos; there are various ways to upload Photos. from a post; from the Photos landing page; from within an album landing page, upload them to a drive folder etc. we could write tests for all these scenarios; but we will never cover all the different permutations and what a user might possibly do. That is an area where having automation is great; but we need to back it up with exploratory testing

Automated API tests are the best bang for the buck wrt tests that are easy to write and stable and a reliable signal.

Integration Tests are always valuable; but we need to ration the number of integration tests one writes as they can be more difficult to debug and have a high noise factor due to all the moving pieces

Automated component testing allows one to test a particular component (server) in isolation. you mock everything else out and ensure this component behaves as intended

As evident; the cone needs to be inverted in a lot of teams and companies. There is too much focus on Automated UI testing. At Google; we realized that half a decade back and have been putting in the required effort in the Integration testing and unit testing layer.

13 of 45

Move Fast & Don’t Break Things

14 of 45

Push on Amber

Move Fast & Don’t Break Things

15 of 45

Push on Green..

“As soon as test suites go green, deployment to production is automatically started”

This has evolved at Google. We have tens of thousands of tests for some projects, some of which could be failing and/or flaky

Move Fast & Don’t Break Things

First a detour to “Push on Green”

Kent Beck in 2003 came up with the idea.

This is a great step in the right direction. if you are not doing this already; please do so.

@ Google we started reaching the limits of Push on Green and it enabling or limiting us in what we wanted to do.

In an ideal world; we should push on green; but we live in a practical world.

I want baby carriers with beer holders; but they don’t make them. so I improvise and use my denim back pocket as my beer holder.

Coming back to Google:

Some teams have non-hermetic tests or tests that are hermetic; but at some point depend on some staging/dev server.

A test (manual or automated) might be failing due to a backend bug; should the frontend be blocked on that bug?

The point I am trying to make is; folks need to assess and make educated judgement calls. and we do that and that’s what I refer to as “Push on Amber”

16 of 45

Push on Amber

Daily pushes to prod
Stable top of tree
Smarter regression testing
Critical tests cannot be bypassed

Move Fast & Don’t Break Things

Push on Green was the past..

The expectation we set and try to meet almost every day is that you check in your code at 4PM today; it will be launched by 4PM tomorrow.

We have hourly release cuts and at 6PM we graduate the latest cut to a release headed to prod.

We have a team of testers ready in a different timezone; we have spent years getting them calibrated on the product and they regression test and log regressions. We have folks ready to cull through change list delta and they they revert buggy code and make new patches and by the the time we come in to work next day; the release is ready to go to prod. We do some final verification in early AM; push to canary; look at numbers and then push to prod. At every step all automated tests are re-run to ensure nothing has regressed further or no new regressions have been introduced.

Some things we do wrt smarter regression testing:

Just regression test affected part of the code base for patches

We instrumented servers to help us evaluate code coverage of manual testing.

Have 7 parallel test stacks to help narrow down regressions

17 of 45

Prevent Bugs

Push on Amber

Move Fast & Don’t Break Things

18 of 45

Prevent Bugs

Prevent bugs and not catch them
Deterministic hermetic tests
Prevent bad code from getting in
High presubmit coverage and usage

Move Fast & Don’t Break Things

While catching bugs is great; my mantra is: “Prevent bugs and not catch them”. That helped the team think beyond just manual and automated testing and also about hermetic automation. more about that later.

To stay ahead of the curve; you need less bugs in the product to start with and one way to achieve that is via preventing crappy code from been checked in.

On the products I have worked on; we made our automation framework a feature of the product; it compiled along with the product; code for that feature didn’t ship to prod and hence users didn’t have to download it.

It also allowed us to just test the SUT and fake everything else out.

Over the last couple of years; we have taken it to the next level and we don’t even have the fake servers. we push back recorded responses to the SUT and verify it’s behavior and the responses are auto updated as changes are made to the server. This has helped us speed up our tests further.

For e.g.

200 total bugs/5 bugs fixed per day = 40 days

10 total bugs/2 bugs fixed per day = 5 days

Which one of these 2 scenarios is better. the choice is clear.

19 of 45

What is a Hermetic Test?

The short definition would be a “test in a box”.

My version: run a test while on a airplane *without network

Move Fast & Don’t Break Things

20 of 45

Non-Hermetic Servers

Before

Move Fast & Don’t Break Things

Consider a complex and rich web app. Under the hood, it is probably a maze of servers, each performing a different task and most talking to each other. Any user action navigates this server maze on its round-trip from the user to the datastores and back. A lot of Google’s web apps are like this. So how do we write end-to-end tests for them?

An end-to-end test in the Google testing world is a test that exercises the entire server stack from a user request to response. Here is a simplified view of the System Under Test (SUT) that an end-to-end test would assert. Note that the frontend server in the SUT connects to a third backend which this particular user request does not need.

One of the challenges to writing a fast and reliable end-to-end test for such a system is avoiding network access. Tests involving network access are slower than their counterparts that only access local resources, and accessing external servers might lead to flakiness due to lack of determinism or unavailability of the external servers.

21 of 45

Hermetic Servers

After

Move Fast & Don’t Break Things

One of the tricks we use at Google to design end-to-end tests is Hermetic Servers.

What is a Hermetic Server? The short definition would be a “server in a box”. If you can start up the entire server on a single machine that has no network connection AND the server works as expected, you have a hermetic server! This is a special case of the more general “hermetic” concept which applies to an isolated system not necessarily on a single machine.

Why is it useful to have a hermetic server? Because if your entire SUT is composed of hermetic servers, it could all be started on a single machine for testing; no network connection necessary! The single machine could be a physical or virtual machine.

The end-to-end test does the following steps:

starts the entire SUT as shown in the diagram on a single machine
makes requests to the server via the test client
validates responses from the server

This end-to-end test is more reliable because it uses no network connection. It is faster because everything it needs is available in-memory or in the local hard disk. We run such tests on our continuous builds, so they run at each changelist affecting any of the servers in the SUT. If the test fails, the logging module helps track where the failure occurred in the SUT.

22 of 45

Prevent Bugs

Push on Amber

Push Testing upstream

Move Fast & Don’t Break Things

23 of 45

Pushing Testing Upstream

Move Fast & Don’t Break Things

This was probably the single most important prioritization work I did for my team @ Google. and as with all good things; this was by accident :)

I drew this up mostly to help me understand how we would scale our testing needs; but it immediately helped the team and other disciplines understand our SLA

we require integration tests for only large features. ideally we should have tests for all features; but with lots of features launching all the time; we throttle the number of tests.

Highly encourage you all to do something similar if you are dealing with trying to test too much…

One of the mismatches I realized was the feature owner and the automation owner were 2 different folks. that led to the tests getting out of sync with the product and some teams were in forever test maintenance mode. When I asked the SWEs why they were not writing tests; they said the framework was too complicated; they didn’t want to learn and/or write their tests in java; when they were building complex AJAX apps in js. for the FE. so we built a framework which made writing UI tests as simple as unit tests and rest is history. We reduced the barrier for writing tests. Let me repeat that: “We reduced the barrier for writing tests”

On the Test engineer aspect; Google+ for example launches 100 features in it’s first 100 days. It was impossible to test all the features. so we helped the SWEs and PMs involved in testing. We mentored them to write test plans and organize their own bug bashes and we would triage the bugs together. We made bug bashes self service. Again repeating myself: we made the bug bashes self service.

24 of 45

Prevent Bugs

Push on Amber

Push Testing upstream

Delineate product releases and features

Move Fast & Don’t Break Things

25 of 45

Delineate product releases and features

Releases always keep rolling
Dark launch features
Revert features and not revert releases
Launch releases without leaking features

Move Fast & Don’t Break Things

26 of 45

Prevent Bugs

Push on Amber

Push Testing upstream

Productivity First

Delineate product releases and features

Move Fast & Don’t Break Things

27 of 45

Productivity First..

Invisible Tests
Tests an asset and not a liability
Fast UI Automation
Zero tolerance on flakiness

Move Fast & Don’t Break Things

28 of 45

Attack Bad Tests

Slow Tests
Flaky Tests

Move Fast & Don’t Break Things

29 of 45

Flaky tests are worse than no tests

Move Fast & Don’t Break Things

30 of 45

Robosheriff

Move Fast & Don’t Break Things

31 of 45

Social @ Google	2012	2013	2014	Assessment
Average presubmit time	21 min	28 min	22 min	Productivity
Code Coverage	???	72.80%	75.60%	Automation
% Green Cycles	71.40%	82.40%	89.60%	Test Hygiene
Avg Submit to Prod Time	17 hrs	11 hrs	13 hrs	Velocity
Total presubmit run time	196 d	11392 d	8033 d	Better Tests
Total automation time	22697 d	52785 d	114040 d	Better Tests
P1 bugs avg resolution time	69 d	28 d	13 d	Bug Hygiene

32 of 45

Prevent Bugs

Push on Amber

Push Testing upstream

Productivity First

Fishfood

Delineate product releases and features

Move Fast & Don’t Break Things

33 of 45

Fishfood

Live on the bleeding edge
Bugs get found/fixed
No SLA for fishfood from test
Rapid iterations. 4 hours from design to bug bash for a feature.

Move Fast & Don’t Break Things

34 of 45

Prevent Bugs

Push on Amber

Push Testing upstream

Productivity First

Fishfood

Prioritize Releases

Delineate product releases and features

Move Fast & Don’t Break Things

35 of 45

Prioritize Releases

Must be an ongoing commitment
All must want to fix root problems (post-mortem!)
Have a dedicated release team
Make sure everyone understands it

Move Fast & Don’t Break Things

36 of 45

Prevent Bugs

Push on Amber

Push Testing upstream

Productivity First

Fishfood

Prioritize Releases

Treat Regressions as build blockers

Delineate product releases and features

Move Fast & Don’t Break Things

37 of 45

Treat regressions as build breaks

Cultural shift

Rollback == guaranteed fix

Verifications are simpler

Devs not under gun for fix

Move Fast & Don’t Break Things

38 of 45

Prevent Bugs

Push on Amber

Push Testing upstream

Productivity First

Fishfood

Prioritize Releases

Treat Regressions as build blockers

Remember Murphy’s Law

Delineate product releases and features

Move Fast & Don’t Break Things

39 of 45

Remember Murphy’s Law

Kill switches for features
Big refactorings behind flags
All user visible changes behind experiments

Move Fast & Don’t Break Things

Again: Anything that can go wrong will go wrong.

Be a pessimist. Hope for the best; but expect the worst

Put on your tester aka how do I break things hat here.

Some of the things we have done here: Missing backend testing. to help eek out hard dependencies and so that your product grcefully degrades.

An outage google services had earlier in the year:

most Google users who use logged-in services like Gmail, Google+, Calendar and Documents found they were unable to access those services for approximately 25 minutes

an internal system that generates configurations—essentially, information that tells other systems how to behave—encountered a software bug and generated an incorrect configuration. The incorrect configuration was sent to live services over the next 15 minutes, caused users’ requests for their data to be ignored, and those services, in turn, generated errors

40 of 45

Prevent Bugs

Push on Amber

Push Testing upstream

Productivity First

Fishfood

Prioritize Releases

Treat Regressions as build blockers

Remember Murphy’s Law

Delineate product releases and features

Balance Velocity vs Quality

Move Fast & Don’t Break Things

41 of 45

Balance Velocity vs Quality

Gate keepers
Provide risk assessment
Trusted Testers
Things that can’t be risked with: ACLs, data, security, upgrades, migrations

Move Fast & Don’t Break Things

42 of 45

Prevent Bugs

Push on Amber

Push Testing upstream

Productivity First

Fishfood

Prioritize Releases

Treat Regressions as build blockers

Remember Murphy’s Law

Delineate product releases and features

Balance Velocity vs Quality

Move Fast & Don’t Break Things

43 of 45

It’s no secret mobile is taking over.

PCS

SMARTPHONES & TABLETS

Move Fast & Don’t Break Things

44 of 45

“Mobile First” Challenges

Balance release velocity - can’t push daily to users.
Mobile app updates use battery and cellular data.
Cannot roll back a bad mobile app easily; higher quality bar needed.

Come join the discussion @ “Move Fast & Don’t Break Things” G+ Community

Move Fast & Don’t Break Things

45 of 45

Prevent Bugs

Push on Amber

Push Testing Upstream

Productivity First

Fishfood

Prioritize Releases

Treat Regressions as build blockers

Remember Murphy’s Law

Delineate product releases and features

Balance Velocity vs Quality

Move Fast & Don’t Break Things

Cultural

Cultural + Tooling

Tooling