1 of 115

Large Scale Development

Culture Change:

Google and the US Government

by Mike Bland

https://mike-bland.com/

2014-09-25

Slide deck: http://goo.gl/TU2pii

2 of 115

October 2013

3 of 115

November 2013

4 of 115

April 2014

5 of 115

So now what?

6 of 115

But first…

...let’s talk about image and expectations.

7 of 115

Can Culture Change?

Google, Governments & Geeks Rapidly

Redefine Test Culture

8 of 115

HA! HA!

9 of 115

What happened next?

How did I get involved?

10 of 115

2001-2005: Northrop Grumman

2005-2011: Google

Testing Grouplet

Test Mercenaries

Websearch Infrastructure

11 of 115

2013: Return to Berklee College of Music

(not Berkeley!)

12 of 115

February 2014

if ((err = ReadyHash(&SSLHashSHA1, &hashCtx)) != 0)

goto fail;

if ((err = SSLHashSHA1.update(&hashCtx, &clientRandom)) != 0)

goto fail;

if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0)

goto fail;

if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0)

goto fail;

if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0)

goto fail;

13 of 115

April 2014

14 of 115

May 2014

“Finding More than One Worm in the Apple”

“Goto Fail, Heartbleed, and Unit Testing Culture”

I published two articles that included working code proving that both bugs were indeed easily reproducible--hence, testable--using the smallest of unit test programs.

I had been out of the industry by two-and-a-half years at this point; the “goto fail” test took only a couple hours to write, and the “Heartbleed” test took only about half a day.

But besides promoting automated testing as a socially responsible development practice, I did it for me. I liked it. I was good at it. And, I was really...I was alive.

“GFHBUTC”: May 12

“FMTOWA”: May 13 in ACM Queue, July 2014 in Communications of the ACM

On May 14, I received “the call”...

----------------------------------------------------------------

Image: http://cacm.acm.org/system/assets/0001/6274/062014_CACMpg58_Finding-More.large.jpg?1403287045&1403287045

“Finding More than One Worm in the Apple” fulltext in ACM Queue: http://queue.acm.org/detail.cfm?id=2620662

15 of 115

How I Learned To Stop Worrying and Love the Bomb...Again

My former colleague Jason Huggins, aka “Hugs”, had just found me on LinkedIn. He wrote:

“As you may or may not be aware, I helped late last year with the healthcare.gov web site rescue….

“It's going to take awhile before the government gets good at software development (and testing) -- and they're going to need a culture change. (You can start to guess why I'm writing you. :-) Luckily, the White House ‘gets’ it now, that fundamental change is needed in how they create and test software, and --more importantly-- that change imposed ‘top down’ is likely to fail. It needs to be grass-roots, with the appropriate amount of air cover from officials when necessary….

“There's lot's to chat about, and I know you're semi-retired, but I'm wondering if you'd ever be interested in talking about your Google experience to the teams now embedded on the various hc.gov projects. I'm specifically interested in bootstrapping something of a Tech Talk program for people with great experiences like yours to present to peers working on these gov projects….

“Again, no pressure. You're quite clear in your LinkedIn bio about jobs and networking. I respect your desire to focus on other things. (I've got my own art projects I desperately want to spend more time on...) But, there's this whole ‘your country needs you’ thing. :-) So I figured I'd just put that out there for you to think about.”

(- Jason Huggins, private email, 2014-05-14)

Consequently, I couldn’t focus on music any longer; it was back to the Dark Side.

----------------------------------------------------------------

Image from the film Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb: http://www.imdb.com/title/tt0057012/?ref_=nv_sr_1

Image: http://www.filmcaptures.com/wp-content/uploads/2013/08/Dr_Strangelove_8.jpg

From: http://www.filmcaptures.com/dr-strangelove-or-how-i-learned-to-stop-worrying-and-love-the-bomb/

16 of 115

June 2014

Left Berklee College of Music

July 2014

Talks at USCIS and CFPB

17 of 115

November 2014

18 of 115

How does culture change?

19 of 115

Not like this...

20 of 115

The Chasm

This model is a direct adaptation of that described by Geoffrey A. Moore in Crossing the Chasm.

Different people react differently to different forces and stimuli, and to different methods of persuasion.

Innovators and Early Adopters are like-minded seekers.
The Early Majority is sympathetic to change, but need accessible tools and procedures.
The Late Majority will go along with whatever seems to work for the Early Majority.
Laggards are hopeless; disregard them.

Notice there are discrete gaps between each segment, with the “chasm” between the Early Adopters and Early Majority being the largest.

On the left side, the Innovators and Early Adopters are already convinced of the value of change; on the other side, the rest of the population must be delicately persuaded.

The Innovators and Early Adopters are highlighted here because...

----------------------------------------------------------------

Crossing the Chasm is a book that explores the differences between successful and unsuccessful tech marketing.

Laggards are a waste of human potential and resources, and produce nothing but a negative investment in energy. They are bad actors that can’t see beyond the preservation of their jobs or empires, and care not for the expense to users, citizens, and taxpayers. Identify them, maneuver around them, and disregard them. (These are mostly my conclusions, of course, not necessarily Moore’s.)

Image by Catherine Laplace based on other illustrations of the “Crossing the Chasm” model.

21 of 115

Instigators

Small, ad-hoc coalitions of Innovators and Early Adopters

22 of 115

“Never doubt that a small group…”

“Never doubt that a small group of thoughtful, committed citizens can change the world;

indeed, it’s the only thing that ever has.”

Margaret Mead

23 of 115

Challenges

Inertia

Insecurity

Ignorance

Friction

Communication Breakdown

Priority Structure

Priority Structure: existing values

This can be both a cause and a produce of the other forces.
Often internalized as "corporate religion".

Inertia: lack of motivation
Insecurity: negative motivation; lack of confidence; arrogance

This is the only active force holding back progress, which motivates people to willfully resist change.

Ignorance: lack of awareness or knowledge
Friction: physical constraints that cost time and energy

This is the greatest impediment, as it is based on physical and temporal constraints. It is also the most straightforward to remedy, as it is an external factor remedied by improved tools and processes.

Communication Breakdown: misunderstanding (and yes, this is a Led Zeppelin reference)

What these all amount to, as Steve Vance pointed out during the review…

----------------------------------------------------------------

Though an external factor from the point of the individual, shared priorities often produce an internal understanding that is seldom questioned--a relatively unspoken "corporate religion".

24 of 115

We Fear Change

25 of 115

Crossing the Chasm

26 of 115

Albert Wong’s “Framework for Helping”

27 of 115

Rainbow Bridge

The Jimi Hendrix disciple in me wants to call this the “Rainbow Bridge”, but the name that I really like is...

----------------------------------------------------------------

Image by Catherine Laplace based on other illustrations of the “Crossing the Chasm” model and Albert’s original model image.

Photo cropped/adapted from :

http://www.thirteen.org/13pressroom/files/2013/09/Hendrix-at-Miami-Pop_creditc-Ken-Davidoff_Authentic-Hendrix-LLC.jpeg

http://www.thirteen.org/13pressroom/press-release/american-masters-2013-season-jimi-hendrix-hear-my-train-a-comin/

Jimi Hendrix Experience performs at the 1968 Miami Pop Festival. Performance footage from the festival is among the previously unseen treasures featured in American Masters: Jimi Hendrix - Hear My Train A Comin’. Photo Credit: © Ken Davidoff/Authentic Hendrix LLC

28 of 115

Albert’s Rainbow of Death, USCIS

Intervene

Validate

Inform

Inspire

Empower

Mentor

Dependent

Independent

Write a report

Give a techtalk

Explain why their work is important to you.

Change something with them. Be their scaffold.

Give positive feedback

Describe how things work elsewhere. "It can get better"

Have 1:1s

Offer continued support. �FOLLOW UP.

Publicly challenge assumptions

29 of 115

“If people believe they lack the power…”

If people believe they lack the power to solve a problem,

Saul Alinsky, paraphrased from Rules for Radicals

they won’t even think of trying to solve it.

30 of 115

Goals and Principles

Embrace, rectify, and learn from mistakes

Understand that lasting change will take time

Improve productivity to better serve users/citizens

31 of 115

Goals and Principles

Cultivation

Avoid cult behavior

Adoption, not imposition

32 of 115

Cultivating a Mythology

Beware of heroes, echo chambers

The best mythology is a useful model

33 of 115

How did Google’s culture change?

34 of 115

Priority Structure

Which leads to…

Google values...

features

resources

throughput

impressions

revenue

performance reviews

promotions

I’m not talking about the external priorities Google advertises, such as “Organize all the world’s information and make it universally accessible and useful,” the “Ten things we know to be true,” or even “Don’t be evil.”

Please remember: I’m an ex-Googler recounting my own personal impressions of my Google experience; I do not speak for Google in any official capacity whatsoever; and I am speaking in generalities. That said, I’m not the only one who’s said any of this before.

So, when you (as a Googler) sit down at your desk every day, what are the building blocks of the “focus on the user” that you’ll be evaluated upon? What do you care about?

features: user-visible artifacts
resources: saving CPU cycles, RAM, disk space, and network bandwidth through performance improvements, or better utilizing available resources, which should lead to increased...
throughput: volume of traffic served, i.e. queries per second (qps), which contributes to...
impressions: number of (relevant) ads served to users, which is of course critical to...
revenue: the ultimate measure of one’s contribution, talent, and value as a human being

All of these measurable results are critical to one’s...

perf: the yearly (all-consuming) peer performance review cycle, the process by which one ultimately receives
promotions: movin’ on up the Engineering Ladder

Admittedly, data-driven management has worked very, very well for Google in general. The problem is when a practical methodology...

35 of 115

If it can’t be measured,

(e.g. more clicks)

it doesn’t matter.

(i.e. won’t get me promoted)

Internalization:

36 of 115

Numbers tell an important part of the story...

...but not the only important part of the story.

37 of 115

Inertia

Extremely bright developers

Overconfidence in status quo

Enormous early success

38 of 115

Insecurity

Overconfidence/arrogance from early-stage success

39 of 115

Insecurity

Impostor Syndrome

Pressure to deliver features, not write tests

Extremely rapid onboarding of new developers

40 of 115

Ignorance

Overconfidence in ability/Dunning-Krueger

“My code is too hard to test”

Lack of testing/testable design experience

...most developers at the time had very little testing experience, any.

Since Google hired only the “best of the best”, that also fed into…

[click]

The D-K effect is the phenomenon where people with relatively little experience or skill in a particular domain overestimate their abilities. Googlers of the day, old and new, certainly lacked both testing experience and, generally speaking, the experience of a failed project, i.e. a death march.

This overconfidence was often expressed as the dismissive assertion...

[click]

This exact phrase was an official response to a yearly employee survey question designed to provide insight into why automated testing wasn’t more widely practiced.

No one would say “my code is defect-free”, but they would say it’s “too hard to test” to handwave away the suggestion that it’s worth even trying to test it.

Then again, they also believed it was true.

[click]

Most people just didn’t know any better.

However, the biggest reason most developers weren’t writing automated tests far more concrete.

41 of 115

Friction

Build & test system pushed beyond its limits

“I don’t have time to test”

42 of 115

xkcd 303

43 of 115

Ignorance/Communication Breakdown

(After-the-fact: goto fail; and Heartbleed)

“What can’t be measured doesn’t matter”

Impact of testing is impossible to measure a priori

Of course it is possible to observe the impact of the lack of a testing culture...

[click]

...after-the fact. Still, this brings us back to the internalized principle of…

[click]

I see the expression of this principle as a function of both...

[click]

Ignorance, because of a lack of testing (or death march) experience; and a Communication Breakdown, because the presumed lack of value closes minds.

So how did automated testing adoption overcome all of these challenges?

----------------------------------------------------------------

Yes, I claim that it is “impossible”, not just “difficult”; I have a section of a blog post explaining the futility of collecting “productivity” metrics. Of course, I’d be delighted to be proven wrong somehow, someday, so call this the “Bland Conjecture” if you want.

44 of 115

Testing Grouplet

Volunteers pooling 20% time

to drive adoption of automated testing

The Testing Grouplet were the Instigators within Google.

We had very little budget and zero authoritah, but we had the freedom to explore creative solutions to the problems facing automated testing adoption, from many different angles.

One of our first initiatives was the unit test lecture and lab that was part of every developer’s two-week onboarding program.

----------------------------------------------------------------

20% time was a tradition whereby any developer could spend roughly one day a week working on a Google-related project other than his/her primary project. Grouplets (aka Intergrouplets) were teams volunteering their 20% time to solve company-wide problems together.

We operated under our own direction; there were no marching orders from the top, but there were no explicit constraints placed upon us, either.

Image Source: http://www.flickr.com/photos/dullhunk/3712840085/

Image License: CC BY 2.0 https://creativecommons.org/licenses/by/2.0/

Logo and slogan by Johannes Henkel; small contribution from yours truly in the form of tragedy/comedy mask suggestion.

45 of 115

“Noogler” Lecture and Codelab

Cultivated training materials facilitating early exposure

Remember, “Noogler” == “new Googler”.

The lecture was followed by a lab proctored by a Testing Grouplet member, using internal wiki-based training materials known as “Codelabs”.

[click]

This is a reproduction of a diagram from that lecture that introduced the concept of Small, Medium, and Large test “sizes”.

----------------------------------------------------------------

We produced these materials in collaboration with EngEDU, Google’s in-house training department. EngEDU was responsible for much of the two-week onboarding material. It hosted Tech Talks by both internal and external speakers, and contracted external consultants to provide training seminars.

Codelabs walked developers through each step of a sample problem and solution using commands they could run on their own workstations.

Image by Catherine Laplace, based on my hand-drawn sketch of a Noogler unit test lecture slide. Nick Lesiecki drew the original testing pyramid in 2005. No idea if there was prior art, but he didn't consult it. The green and yellow arrows weren't added by Nick, but the "Confidence" concept was hammered out by Nick, Patrick Doyle, and Dominic Cooney c. 2009.

The pyramid was later popularized by Mike Cohn:

http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid (2009) and http://www.amazon.com/gp/product/0321579364 (2009).

Not sure if Mike had seen the Noogler lecture slide or had independently conceived of the idea, but he definitely was a visitor at Google at the time I was there.

46 of 115

Small/Medium/Large test sizes

Replaced misunderstood terminology

Raised awareness of concepts

Inside Google, the existing terms had lost their meaning. At the time, a “unit” test was generally understood to be a test that took less than five minutes or so.

The new terminology came with no baggage, and we could rigorously define what each term meant.

They helped raise awareness of the different functions each “level” of testing aimed to serve, and how specific techniques and tools applied to each.

----------------------------------------------------------------

Image by Catherine Laplace, based on my hand-drawn sketch of a Noogler unit test lecture slide. Not sure of the origin of the pyramid, whether it was Grouplet-invented or known beforehand, but it was in use by 2005 or 2006, some years before http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid (2009) and http://www.amazon.com/gp/product/0321579364 (2009).

47 of 115

Documentation

Extensive, well-curated wikis

Tech Talks

Sometimes people don’t want to RTFM

We continued to provide information after that one lecture and lab. But our biggest breakthrough was...

----------------------------------------------------------------

The documentation and tech talks produced by Grouplet members benefited from the consistent Small/Medium/Large concept and terminology. It gave developers a clear and tangible perspective on testing that they could use to absorb all the other detailed information we provided.

Image by Catherine Laplace, based on my hand-drawn sketch of a Noogler unit test lecture slide. Not sure of the origin of the pyramid, whether it was Grouplet-invented or known beforehand, but it was in use by 2005 or 2006, some years before http://www.mountaingoatsoftware.com/blog/the-forgotten-layer-of-the-test-automation-pyramid (2009) and http://www.amazon.com/gp/product/0321579364 (2009).

48 of 115

Testing on the Toilet

Broke down geographic boundaries of discussion

Standardized terms, avoided echo chamber

Testing front-and-center for everybody from day 1

[click]

Once we solved the production and distribution challenges--outlined in my TotT blog post--we were able to post a new episode each week and publish it in nearly every bathroom in nearly every office worldwide. It’s doubtful an online-only publication would’ve involved people to the same degree.

[click]

TotT helped standardize the use of Small/Medium/Large and other concepts throughout the company, and the feedback we received from readers was critical to ensuring our message was relevant and useful to the company as a whole.

[click]

As Antoine Picard points out, TotT injected automated testing into everyone’s awareness from day one, even before they had a chance to attend the unit testing lecture and lab.

I chose this episode for this slide not just because I wrote it (ahem), but because it happened to encapsulate two other significant Testing Grouplet initiatives. The first of which being...

----------------------------------------------------------------

TotT was the most powerful and successful communication channel, and continues to be so to this day. In fact, thanks to friends on the inside, I recently became the first “outsider” to publish not one, but two TotT episodes: one for “goto fail”; and one for Heartbleed.

It helped eliminate the “echo chamber” effect. (Ironic, considering the reflective nature of bathroom surfaces.)

Image taken from http://www.flickr.com/photos/dullhunk/1019220380/ of an episode by yours truly.

49 of 115

Test Certified (TC)

Three-level roadmap, tailored to quarterly goals

Overcame psychological obstacle of “How to?”

...the Test Certified program.

Nick Lesiecki: “TC levels ‘hack’ the internal priorities of the culture. Testing wasn't measurable, so we created [snip “5”] measurable milestones that people could engage with and brag about at perf time. Sure the measure wasn't as smooth as ‘clicks’ but what wasn't measurable became measurable.”

[click]

Level one: rapidly establishing baseline metrics
Level two: setting a policy and reaching for early coverage goals
Level three: striving towards suitable long-term coverage goals

This roadmap format helped...

[click]

...to overcome that first, scary obstacle of not knowing where or how to start.

We also provided volunteers as “mentors” to each team who would provide advice and ultimately “validate” their progress in climbing the “TC Ladder”.

The “validation” function eventually took the greatest precedence, as Googlers are suckers for measurable indications of progress. In fact, TC was eventually expanded to five levels.

It’ll always be three to me, though; if you ask me, levels four and five are for...

----------------------------------------------------------------

That’s the TC “shield” logo in the upper-left of the episode, designed by Mamie Rheingold.

Image taken from http://www.flickr.com/photos/dullhunk/1019220380/ of an episode by yours truly.

50 of 115

Achievers

51 of 115

Test Mercenaries

Hands-on help for teams that needed it

Direct feedback to Testing Tech and Build Tools

Tool, practice advocacy

...the Test Mercenaries.

[click]

...the Mercs were “internal consultants” that would help teams improve their testing practices, using Test Certified as both a guide and a goal.

Nick Lesiecki: “The Grouplet advocated and got funding for a staffed team of internal consultants. This [was] an important step. Management was getting involved, but not with edicts. They were demonstrating the importance of testing by funding a consultancy to train people.”

We were the “boots on the ground” practicing what the Grouplet preached, and...

[click]

...we were also able to influence internal tool development. Once the Testing Tech and Build Tools teams produced promising new tools…

[click]

...the Test Mercenaries could then drive adoption of these improved tools and the practices they enabled.

----------------------------------------------------------------

That’s the Mercs’ “knight” logo in the center/lower right, also designed by Mamie Rhiengold. You’ll notice that the TC “shield” logo is incorporated into it.

Image taken from http://www.flickr.com/photos/dullhunk/1019220380/ of an episode by yours truly.

52 of 115

Fixits

Company-wide events, usually one day long

Address “important but not urgent” backlog

New tool rollouts address “I don’t have time to test”

53 of 115

Fixits

Focus everyone on concrete goals

Motivation: schwag; recognition; good karma

54 of 115

Fixits

Punctuate story arc

Raise morale

Advance state-of-the-art

Fixits provided missions at points in time that generated excitement and energy, which helped…

[click]

Trying to effect culture change is sometimes a very bleak and disparaging affair.

Organizing a big event and getting people moving in the same direction generates positive energy, and ends up…

[click]

...advancing the state-of-the-art. The long-term culture change mission reaches a new plateau with every big, visible effort.

The topic of Fixits is another worthy of its own talk, but for now, let’s take the pieces of the Testing Grouplet puzzle and see how they fit into...

----------------------------------------------------------------

My scan of a physical copy of an episode by Antoine Picard announcing the second company-wide Fixit I organized.

Fixits originated as company-wide events, but in practice, Fixits can be whatever size you want them to be. When I was leading the Fixit Grouplet (after leading the Testing Grouplet for a while), we championed the idea of "Fixlets", which could be team-sized affairs organized with much less overhead. In fact, I envision the ideal company-wide Fixit as being a framework enabling individual teams to run their own “Fixlet” that happens to align with everybody else’s Fixlets that same day.

55 of 115

Albert’s RoD, Testing Grouplet Ver.

Intervene

Validate

Inform

Inspire

Empower

Mentor

Dependent

Independent

Fixits

Revolution

Test Certified

Have 1:1s

Offer continued support. �FOLLOW UP.

TAP

Build Orbs

Lectures

TotT

Codelabs

EngEDU Collaboration

Tool development

(w/ Testing Tech, Build Tools)

Test Mercenaries

Tech Talks

Testing Grouplet

Schwag (T-shirts)

Bear in mind that in reality, many of our activities spread across different functions, not just the ones I’ve listed them under here.

[click 2x]

TC also served the “inform”, “inspire”, and “mentor” functions, but for sanity’s sake I’ll only list it here under “validate”.

[click 2x]

Build Orbs were “information radiators” that would provide a conspicuous visual indication of the state of a team’s continuous build.

Testing Grouplet members would often hold “orb-building parties” to construct these devices, which we’d eventually award to teams achieving Test Certified Level One status.

T-shirts are the fru-its of the dev-eel.

[click]

The Testing Grouplet was invested in providing knowledge, skills, and encouragement to developers interested in testing, as well as to one another.

A lot of times that interaction would be in the form of documentation, meetings, or tech talks, but it also frequently involved a lot of one-on-one personal relationships and interaction.

[click]

Applying a suite of build tools to set up a continuous build system for my first Test Mercenaries engagement inspired the “Revolution Fixit”, the event that spread adoption of these tools throughout Google and provided the foundation for the development of the Test Automation Platform (TAP), Google’s company-wide continuous integration system.

The core Testing Grouplet members built relationships that lasted well beyond any fixit or Test Certified mentorship or Test Mercenary engagement.

56 of 115

Testing Grouplet Strategy

2005-mid 2007: Brainstorm, experiment, Disagree and Commit

Mid 2007: Strategy emerges

Get all teams to Test Certified Level 3 by end of 2009

“I don’t have time to test” =>

Build system, Revolution Fixit, Test Automation Platform

Our strategy was not fully-realized from the start. It emerged over time, following a pattern from “brainstorming” to a high-level “mission” to “eliminating the biggest obstacle to the mission”.

[click]

The first couple years was all about experimenting with ideas to discover those that held promise.

[click]

By mid-2007 we had most of our big ideas in-place, to the point where the different efforts risked fragmentation. At that point...

[click]

...the overriding strategy that aligned all of the Testing Grouplet-spawned efforts emerged.

It was lofty, yet concrete and measurable. Every different initiative could meaningfully contribute to it.

The biggest, most obvious obstacle to this goal was, by far...

[click]

Shortly after the formulation and adoption of the “TC everywhere” strategy was when we also stumbled upon…

[click]

...the right tools at the right time, working together the right way.

----------------------------------------------------------------

Antoine Picard pointed out during the review of this presentation the similarity to the “Disagree and Commit” model, apparently originally codified at Intel. We certainly followed that model in practice, though no one ever called it out explicitly (and I never heard of the term before Antoine’s comment).

57 of 115

Testing Grouplet Strategy

Measurable goals via TC, Fixits

Real objective: Experience

Power and knowledge to do the right thing

[click]

We adapted our methods to fit as comfortably within the existing priority structure as possible.

The ultimate goal of these strategies was to produce...

[click]

Experience is the most effective form of persuasion, especially when data are hard to come by.

----------------------------------------------------------------

A corollary to the earlier quote from Saul Alinsky is that if people feel they do have the power to effect change, and the knowledge of how to apply that power to achieve their goals, they’ll be self-motivated to improve their circumstances.

The measurability aspect speaks to another Alinsky tenet, which is to work within the experience of your community.

Where data is unavailable and logical arguments are dismissed, a well-cultivated positive experience can ultimately win hearts and minds.

58 of 115

How long did all this take?

59 of 115

2005

Spring: Testing Grouplet founded

60 of 115

2006

Early/Mid: Testing on the Toilet

August 3: First Testing Fixit

Test Certified

October: Test Mercenaries

61 of 115

2007

March 8: Second Testing Fixit

Summer: Test Certified strategy

Summer: Revolution Fixit, TAP conceived

62 of 115

2008

January 31: Revolution Fixit

Summer: Test Certified Challenge

September: Phynancial crisis hits

December: Vendors/contractors fired

At the beginning of 2008, we were on fire. The Revolution really “Revolutionized” the company-wide development environment.

The TC Challenge was a months-long Fixit aimed at maximizing Test Certified adoption.

And then, just as the TC Challenge reached its conclusion...

[click 2x]

...For the first time in its history, Google had to fire practically all vendors and contract employees, most of whom were manual testers.

But the majority of the Test Mercenaries team were from contract vendors as well…

----------------------------------------------------------------

The TC Challenge was organized by two Software Engineers in Test from Cambridge, MA: Matt Vail and Tayeb Karim. Aside from the length, it was unique in that it used automation to determine who did what and publish the results, and people were "pleasantly surprised" that their testing efforts were acknowledged. (Most were, anyway.) This had the intended effect of creating a company-wide buzz around Test Certified, and the TC Ladder exploded with new teams.

I spell it “phynancial” because I love how that spelling, to me, speaks to the sleazy nature of the subject. I picked it up from Alfred Jarry’s Ubu Roi.

63 of 115

2009

Spring: Test Mercenaries disband

October: Forgeability Fixit

Despite the loss of so many manual testers and the Test Mercenaries, there was no quality crisis.

Our efforts had taken hold and nudged the Google development community as a whole past the Tipping Point just in time.

Thanks to the widespread adoption of the new tool suite, Rachel Potvin of the Build Tools team organized what was called...

[click]

...so called because of the goal of getting as much code as possible to build in parallel in the cloud using the Forge infrastructure (originally developed by Ambrose Feinstein).

The Forgeability Fixit was perfectly timed, creating momentum for the…

----------------------------------------------------------------

Teams were largely able to take responsibility for their own quality assurance, thanks in no small part to increased adoption of automated developer testing. What limited manual testing resources were available were well-utilized, because most potential quality issues were halted upstream by developer testing.

64 of 115

2010

March 6: TAP Fixit

65 of 115

Testing Grouplet Outcome

O(5 years), O(100s of volunteers)

TAP Fixit effectively met TC Level 3 goal

(Only one quarter behind schedule)

66 of 115

Testing Grouplet Outcome

Thorough automated testing now the norm

Most breakages fixed before clients notice

67 of 115

David and Golaith

68 of 115

What are the issues with US Government IT development?

69 of 115

US Government Priority Structure

limiting perceived risk

meeting regulatory requirements

job security

70 of 115

Don’t rock the boat.

Internalization:

71 of 115

Inertia/Insecurity

No incentives for quality

Pendleton Civil Service Reform Act

(jwz: People who want to work for a successful company vs. work to make a company successful)

Maintenance of status quo

On both sides of the contractor/purchasing agent equation, a job-well-done often isn’t a contractual requirement or rewarded outcome.

Salary caps and a lack of stock options act to limit motivation. A lack of internal recognition, intended to eliminate peer resentment, also limits the perceived benefit of taking risks.

[click]

The Pendleton Civil Service Reform Act replaced the “spoils” system for civil service job appointments (i.e. political appointments) with a “merit” system.

It also makes it hard to fire or replace someone with someone more motivated and capable.

This reminds me of one of my favorite quotes from Jamie Zawinski, one of the original Netscape developers, in which he differentiates between…

[click]

...when Netscape started becoming successful, it began to hire more of the former and fewer of the latter.

A desire for job security at a successful company translates into a desire…

[click]

...to not rock the boat and risk losing your job. This is, in effect, a combination of…

[click]

...inertia, since folks lack the incentive to change their environment;

and insecurity, since change and the risk required to produce it is likely perceived to be at odds with job security.

----------------------------------------------------------------

The incentive for purchasing agents is to trim the price of the initial bid, rather than the final cost or the best choice (Steve Vance).

The incentive for contractors is to maximize billable time and materials (Bill Shelton).

Pendleton: In other words, it supplies job security for civil servants across administrations, but dulls the nimbleness of an organization. (Rob Read)

Reinforcing this perception is a direct quote from one of my reviewers, Isaac Truett: “As a former Federal IT contractor, I can confirm that, from my experience and perception, by far the leading objective was to generate more billable hours. Employees at contracting companies were rewarded for helping land new contracts or adding tasks to existing work orders. Reducing the necessary work by adding automation or reducing defects by adopting quality-improving practices, was rarely recognized positively.”

72 of 115

Inertia/Ignorance

Waterfall thinking persists

Testing is someone else’s problem

It doesn’t help that most Federal IT development continues to follow the Waterfall method.

Overcoming this obstacle is as much as battling inertia--why change when this is the way it’s been done for years?--as it is battling ignorance, making people aware of the value of more modern development practices.

This perception that Government IT is stuck in the 70’s that contributes to the perception that...

----------------------------------------------------------------

One of the unfortunate outcomes of this method is the common belief that testing should come near the end of the process, and should only be performed by those with no insight into the code and an adversarial perspective.

While black-box testing (i.e. client-side testing with no knowledge of the internals of the code under test) can prove valuable, the concept that testing should only be a black-box process at the very end of development is a wasteful perspective. Why wait for a manual tester to catch a defect days, weeks, or months after it was introduced, when a unit test could have caught it before the original developer submitted the code?

73 of 115

Insecurity

Perception: Gov’t can’t attract/wastes talent

Lack of career development (e.g. training, conferences)

Hopelessness

...the Government can’t get the best programmers, since the best programmers won’t want to work there.

Private industry offers exciting, innovative, modern work environments, and internal recognition, as well as stock options and no mandated salary caps. Why would any good programmers settle for less?

Hence, as Rob Read (PIF, 18F) points out, there’s been a “brain drain” that makes it difficult to maintain a culture of technological competence.

While there may be some truth to that, such low expectations risk eroding the self-esteem of many programmers doing government work. Compounding this problem is...

[click]

...the apparent lack of value placed on technical career development. The budgets for it are notoriously thin. This lack of opportunity produces a sense of…

[click]

...which ultimately is an expression of…

[click]

i.e. Many developers lack the confidence to push for changes, because they feel like they’re on the wrong side of a closed door.

Part of this problem can be attributed to the fact that...

74 of 115

Ignorance/Communication Breakdown

Policy often mandated by nontechnical people

Development teams disconnected from end users

They don’t know what they don’t know

75 of 115

Friction

Outdated tools and deployment procedures

Vendor lock-in via ownership of code, deployment procedures, even data

Old tools and procedures either required by obsolete requirements or obsolete thinking are a problem wherever you go; the Government’s no different.

The problem does appear to be compounded by the increasingly stringent security procedures imposed upon Government IT development, and...

[click]

The people who need access to the code, deployment systems, or data can’t get to it.

If the underlying human insecurity issues are solved, however, much of the technical friction goes away.

----------------------------------------------------------------

Bill Shelton, CFPB Dev Lead: “Vendor lock-in via products and services is an unwritten goal and I assume it's rewarded by vendors and the metrics would be contract renewals. The government frequently does not have the technical expertise to guide development or keep vendors honest, so, once inside, it's very expensive to change.”

76 of 115

Accountability

In the current atmosphere, it’s often punitive.

“Holding someone accountable” is tantamount to getting them fired.

77 of 115

Meeting with the Bobs

78 of 115

Challenges

Inertia

Insecurity

Ignorance

Friction

Communication Breakdown

Priority Structure

79 of 115

What Federal agencies are involved in the reform effort?

80 of 115

United States Digital Service (USDS)

Founded July 2014

Under Exec Office of the President aka “White House”

“Air cover”: rapid, precise technical work and consultation where needed

USDS Playbook (GitHub), TechFAR

81 of 115

18F

Under the General Services Administration

Located at 18th and F Streets

Founded March 2014 by Presidential Innovation Fellows

82 of 115

18F

Simplify contract procurement via Inter-Agency Agreements

Products developed as open-source, using Agile methodologies

Software as a “General Service”

83 of 115

18F

Visible example of Agile development

Outreach still necessary

Examples, docs, lectures, consulting, and hands-on contract work

Wide exposure == opportunity for cross-agency, ad-hoc collaboration

84 of 115

18F

85 of 115

Consumer Financial Protection Bureau (CFPB)

July 2010: Dodd-Frank phynancial regulation

July 2011: CFPB founded

Applied Agile practices from the start

Several 18F/USDS recruits come from CFPB

86 of 115

Consumer Financial Protection Bureau (CFPB)

Homepage

Blog

Source Code Policy

Regulations Parser: Travis CI; Coveralls

qu Data Platform: GitHub; Travis CI

87 of 115

USDS/18F Client Agencies

Veterans Affairs

Social Security

US Citizenship and Immigration Services

CIO Mark Schwartz’s Blogs:

Government IT; Thoughtful IT

88 of 115

How do Google’s lessons apply to US Government IT development?

89 of 115

Mapping Grouplet experiences to US Gov’t

Details boil down to same fundamental issues

“Crossing the Chasm” model applies

Functional “Rainbow of Death” needs the same

90 of 115

Mapping Grouplet experiences to US Gov’t

Emotions, psychology > Technical issues

Creativity, fun => Long-term sustainability

91 of 115

Rebel Alliance

Coalition of individuals from the “White House”, USDS, 18F, CFPB & other agencies

Aimed at instigating government IT reform

92 of 115

Rebel Alliance

Connecting with and empowering great talent

Bust echo chambers, not create a new one

93 of 115

Straightforward Tactics

Documentation/Wikis

In-house training/EngEDU

Mailing list/“help desk”

Tech Talks

94 of 115

Tactics Requiring Adaptation

Test Certified-like program => CFPB “mentors”

Fixit-like events

Testing on the Toilet?

USDS & 18F => Test Mercenaries

95 of 115

The details aren’t important.

The community is.

96 of 115

Albert’s RoD, Rebel Alliance Version

Intervene

Validate

Inform

Inspire

Empower

Mentor

Dependent

Independent

USDS

Success stories in “albatross” projects

Tech Talks

TotT?

Agile Assisted Acquisitions (18F)

Useful Mythology

Develop/share internal expertise

External consultants

FOIA

Direct 18F contracts

Fixits?

CFPB example

Test Certified-like program

Rebel Alliance (18F, CFPB, USDS)

Schwag

(not T-shirts)

1-on-1s

Continued support

(18F, Rebel Alliance)

Encourage risk-taking

Docs/Wikis

“Help Desk”

Career development

Learn from mistakes (not punish)

97 of 115

Concrete Goals

Elimination of Friction

Cultivation of knowledge and expertise

Adoption of automated testing

Adoption of Agile methodologies

98 of 115

New Methods and Habits

Risk-taking (vs. thought-slavery)

Cultivation of soft power

Ad-hoc collaboration

99 of 115

What accountability looks like

Responsibility for quality up-front

e.g. Test your own code!

Responsibility for mistakes/defects as they arise

No excuses, no blame

100 of 115

What accountability looks like

Willingness to learn and improve

Accountable behavior encouraged and rewarded

Genuine effort to avoid preventable issues

----------------------------------------------------------------

Bill Shelton, CFPB Dev Lead: “Being truly accountable demonstrates a genuine willingness to learn and improve for the sake of the effort (project, organization, etc.). Testing is similar. We shouldn't blame individuals for bugs (we all inject bugs), though we are accountable. Testing is meant to uncover or prevent software faults. A positive side effect if testing is done right and part of the culture is that as we practice it more more we learn and write better software. => Accountability == Learning”

Steve Vance: “So think about the example of catching mistakes of finding a bug someone else wrote. Do you clobber them with it? Do you ‘triage’ it to them so they can fix their mistake? Do you fix it and share the learning with them? Do you fix it and stay quiet?

“What does accountability look like on the prevention side? Writing code that someone else tests? Writing unit tests knowing someone else will manually test it? Writing unit tests assuming someone else will write automated larger tests? Writing it all yourself and manually exercising it fully? How about engaging domain experts to anticipate reasonable edge cases that you wouldn't have thought of otherwise?”

101 of 115

Meta Goals

Security/Procurement/Hiring Reform

Contract reform through incentive reform

Replace self-preservation with shared goals

[click 2x]

I want to share an exchange I had with Erie Meyer, a member of the “White House” (the Office of Science and Technology Policy to be precise), while we were at dinner with several Rebel Alliance members at the end of July.

Remarking on the enormity of the challenges and the fact that most of the current Alliance members were sitting at the table, she said…

----------------------------------------------------------------

An example of shared goals would be Google’s “Objectives and Key Results” system, whereby individuals and teams set their own quarterly goals and self-grade progress towards each on a scale of 0.0 to 1.0 at the conclusion of each quarter. The ideal score is 0.7, because it proved the goal was a “stretch goal”, whereby it was pushing your current capabilities.

102 of 115

“You should be scared that it’s just us.”

Erie Meyer

(on the Rebel Alliance):

“You should be scared that it’s just us.”

Me: “I’d be scared if it wasn’t.”

103 of 115

Can it succeed?

104 of 115

Yes.

105 of 115

Will it succeed?

106 of 115

I don’t know.

[click]

But it’s 100% certain it won’t if we don’t take our chances when we get ’em.

Charles Worthington, Senior Advisor to the US CTO: “If you think there is a problem with how government does tech, and that you could help government do tech better, then the question is what are you doing to improve it? It's not going to get fixed on its own, and the people in charge have never been more open to new ideas.

“If we don't try, who else are we expecting to?”

Back to me: Besides, who cares whether the odds are against us?

----------------------------------------------------------------

Charles W.’s entire comment: “The Government is not this big monolithic other thing that just exists in a big building closed off to everyone. It is made up of Americans who have chosen to work in public service... i.e. to help make other American's lives better.

“If you think there is a problem with how government does tech, and that you could help government do tech better, then the question is what are you doing to improve it? It's not going to get fixed on its own, and the people in charge have never been more open to new ideas.

“If we don't try, who else are we expecting to?

“Some percentage of America's top lawyers choose to work at the Dept. of Justice prosecuting bad actors instead of corporate law firms.

“Some percentage of America's top scientists choose to work at the NIH/CDC fighting infectious diseases instead of at a big pharma company.

“Where are the top technologists working to build awesome digital services for the American people?”

Michael Byrne, CFPB: “i think here might be a good place to add challenges still a head. one of those key things is technology for technologies sake doesn't do anything in government. in the private sector success is determined by sale of units. not so in the government. there aren't the same clear metrics in government. success is not a pretty web page. if the ACA had a working web site at day 1, but the issue was about say, how far off the atomic clock is from an iPhone clock, is it the same success? success in government is 100 pct about getting the policy right. in the ACA case, the right policy is a combination of policy about healthcare AND getting people to sign up for it. not every high goaled policy effort in government has the same implementation and we need to recognize that workable web sites don't solve every problem. challenges a head help keep our efforts in check. what does the next administration look like? what does the next congress look like? what if there is a new great recession? what if the virus (e.g. ebola, or cpog or something else) causes a new world scare? what if there is a new 911? what if there is a student revolt to FARM, or common core? these policy issues do not have the same approaches to tech development as ACA, so we need to make sure we are agile enough to know what methods support the policy. in many many cases the policy answer is so unclear that the debate needs to be fostered (e.g. education about a plan), rather than here is the implementation. in all cases, there isn't the same metric (7 million people signed up only works for ACA). i might say that a core issue here is gov metrics is WAY hard.”

107 of 115

“It is better to die on your feet…”

“It is better to die on your feet

than to live on your knees.”

Emiliano Zapata

108 of 115

What’s at stake?

109 of 115

What’s at stake?

Lives and livelihoods

Efficient allocation of resources

The UK already has one up on us

The social responsibility of software devs

[click 2x]

UK Digital Services is renowned for its high standards. Whether or not every aspect of its model will apply to the US Government, it certainly sets a shining example.

[click]

I’d like to see the US Government set a high, visible bar for the social responsibility of developers in general.

On top of all this, I look forward to deriving a great deal of personal pleasure and validation from being able to say…

----------------------------------------------------------------

Steve Vance: IEEE and ACM have a code of ethical conduct for software that parallels the one for engineering, but I rarely find people aware of it, much less knowing and caring what it says.

http://www.acm.org/about/se-code

Quoth myself, from “Goto Fail, Heartbleed, and Unit Testing Culture”:

“Hand washing alone does not a doctor make, nor a patient save, but we would not trust a doctor who didn’t wash his or her hands. As software developers, we should assume a similar duty of care on behalf of our users. No one should trust software developed without unit tests.”

110 of 115

If the US Federal Government can do it...

If the United States Federal Government can reform its development practices...

111 of 115

...then what’s your team’s excuse?

112 of 115

None More Black

113 of 115

Partners-in-Crime

Brought to you with valuable feedback and contributions from:

114 of 115

How to get in touch

http://goo.gl/TU2pii

mbland@acm.org

https://mike-bland.com/

115 of 115

Google vis-a-vis US Government IT

	Google	US Government
Internal Priorities	Measurement, promotions	Extended contracts, deferred accountability, job security
Challenges	Overconfidence, delivery pressure, ignorance, build friction	Underconfidence, contract pressure, legal restrictions, waterfall thinking, accountability
Instigators	Testing Grouplet, Test Mercenaries, Testing Tech, Build Tools, Eng Prod	Rebel Alliance, White House/USDS, GSA/18F, key folks in USCIS, VA, SS
Methods	Soft power, ad-hoc collaboration, experimentation, programs geared to culture of measurement, tools, docs, talks	Agile teams, Inter-Agency Agreements, internal consulting, Agile procurements, training…???