Execution and Learning

Every tech project needs to progress rapidly on two different fronts at the same time - Execution, building the product and learning - systematically finding what product addresses user needs best. Focusing on execution with little or no learning exposes us to the all so common risk of launching a great product no one needs. Over focusing on learning may get us stuck in the analysis paralysis quagmire.  How to balance?  Here are some common approaches I experienced.


Waterfall, the best project management practice when I got started tech, splits the project into neat stages, each handled by different team assembly-line-like. From a learning perspective there are three main phases: some learning happens at the Product Definition stage, where preliminary research is carried out; Almost no learning occurs in the Development and QA phase, and most of the learning happens after the product is launched, when the team is expected to iterate quickly. That’s the theory anyway.    

Putting aside all other issues with waterfall (primarily that it doesn’t work), from a learning perspective it is very suboptimal:  

Waterfall attempts to optimize for execution, but in so doing all but kills any opportunity to learn early. 

Sidenote: Agile Development a much better process for learning, however most companies have only partially adopted Agile in the development phase, using the so called Water-Scrum-Fall model, which isn’t much better for learning:

Dogfood, Alpha/Beta, Previews and % Launches

Recognizing the risks of pushing learning to the very end, tech companies started introducing project milestones aimed at getting customer/user feedback earlier - through preview versions, alpha and beta testing and %-launches. Similarly many companies now have their employees “eat their own dogfood” and use not-yet-launched products internally where possible. Done right these test milestones can have very positive effects on the product.  

To see why let’s zoom into the Development & QA part of the waterfall. Schematically the development project starts when the specifications are handed down to the engineering team and ends when the team launches the specified product, let’s call it ‘X’ (this is obviously a huge oversimplification, but bare with me).  

Adding a single beta test milestone injects a dose of real user input into the process and, if handled properly, can send the project into a new trajectory, resulting in X’ -  an improved version of X that better matches user needs. Effectively, the beta milestone moved learning to an earlier point in the project, reducing wasted effort on building and launching the wrong product.  

However this approach is far from a complete fix for waterfall. The version we tests are all too often the finished product minus bug fixes or a few features. Testing starts at a late point in the project life cycle where the launch deadline is looming and sheer inertia is pushing against major changes or pivots. It takes a very disciplined team to stop at the beta phase and go back to the drawing board.  And so X’ is typically quite close to X and the end result is the same - a product that misses the mark.

Done right Alpha, Beta, Dogfood and Preview releases can put your product on a better trajectory, but they cannot turn a fundamentally bad product idea into a good one.  

Lean Startup

Lean startup flips the balance on its head - learning is the key focus and product execution is happens the process of learning.  This introduces a number of important innovations:

Validated learning is the antidote to the lethal problem of achieving failure: successfully executing a plan that leads nowhere.

Eric Ries

Lean Startup is heavily inspired by the process of scientific discovery starting from an idea that is based on what we know now, and gradually arriving at a much more profound product that may be radically different.

I think Lean Startup is great, and a major leap forward, but it’s not without its limitations;  

Learning Milestones

This is a method that we used a lot in Gmail, generally to positive results. It doesn’t break away completely from the sequential progression of waterfall, but does facilitate  early learning, course-correction and pivots.  

The thing to notice is that the projects tend to be planned around a series of intermediate milestones.  Traditionally these are about internal design and dev gate steps and bare little or no meaning to customers.

The trick is to replace these somewhat opaque eng/UX milestone with meaningful learning milestones, each producing a somewhat more evolved version of the product that is immediately tested with an ever larger group of users for a longer period of time. What’s learned in each milestone is then shared back with the team and is used to decide what to build next - an opportunity to update the spec or to pivot. The team is instructed to always focus on the next milestone, rather than optimize for final launch.

The goal of the team is to learn by putting an ever more more complete version of product in front of an increasingly larger group of users for longer durations of time.  



If you’ve never tried them I highly recommend trying Lean Startup and Learning Milestones.  The former is probably more appropriate when there’s more uncertainty in your project, for example launching a radically new product, while the latter better for more incremental changes.