Notes: Running a startup on haskell

this presentation was given at Strange Loop 2011 conference. here’s the video: http://www.infoq.com/presentations/Running-a-Startup-on-Haskell

Brial O’Sullivan is “CTO” of MailRank (“email productivity company”, 3 people) which was recently acquired by Facebook, for an undisclosed amount, in what appears to be a talent acquisition. O’Sullivan wrote Real World Haskell.

TL:DR (dustin opinion)

PG wrote (emphasis mine):

Most hackers I know have been disappointed by the ML family. Languages with static typing would be more suitable if programs were something you thought of in advance, and then merely translated into code. But that's not how programs get written.

The inability to have lists of mixed types is a particularly crippling restriction. It gets in the way of exploratory programming (it's convenient early on to represent everything as lists), and it means you can't have real macros.

http://www.paulgraham.com/lispfaq1.html

So, startups do lots of exploratory coding -- is a language like haskell going to slow them down? haskell forces you to have perfect code, and simply doesn’t let you use “quick hacks” -- it forces you to refactor and rethink your program.

Sometimes, you just want to hack in a quick fix, i speculate this happens all the time in a startup, because business concerns outweigh any long-term code maintenance concerns. sometimes you wanna say “man, if only i could sneak a database query in here in the guts of my security layer”, which in haskell could require rethinking your entire architecture so you don’t need type-checked access to IO-monad from a place 8 levels deep in the call stack, but in python you just do the one liner.

In my current experience writing Java enterprise apps on a strict schedule where we aren’t sure what the final solution is going to look like (exploratory coding), this happens all the time. maybe less so with haskell and a more expert team, but surprises happen.  

OSullivan didn’t directly address exploratory coding, but he did attribute the speed at which he produced code to both haskell and that he already had strong opinions about what the solutions should look like. I speculate O’Sullivan understood the architecture required by his problem domain well enough that he didn’t do any exploratory coding.

notes on the video

first 10 minutes: bio (nobody cares bud!). I’m browsing his github, they appear to have written a ridiculous amount of functionality in haskell for MailRank. lots of haskell wrappers for stuff -- they have written a lot of libraries from scratch that are “batteries included” in java and python. its interesting how much raw output they appear to have done.

his app has desktop client C#, cloud environment haskell

why haskell --

• can i be productive enough?
• can I hire?
• do i know the terrain well enough to predict and navigate problems?

hiring:signal

• “coding chops of haskell experts are mind-blowing”
• interest in haskell (& clojure, scala, etc) is predictive of good hires
• plenty of enthusiastic almost-haskellers

hiring:noise -- just don’t be dumb. its a signal.

haskell -- daily rhythms

• write a few lines of code
• load it into the repl
• is there a type error? fix it
• did the code load? try it out by hand (??)
• did it work? build and try
• dozens of times per hour, but only try the native code a few times an hour. very fast loodback. repl means fast turnaround which shouldn’t be underestimated.

differences from other environments

• “edit, then mess around in repl” feels like python or lisp, but you can’t livecode (attach to running program and redefine things on the fly, which is “trippy”)
• the typechecker tells you how “loopy” your reasoning is
• compile a native app feels like C++

• lack of IDE “feels like 1994” but everyone uses emacs and vim anyway

architecture

• load balancer, SSL http-based backend, “vaguely restful”
• “fairly rare that backend service makes sense to use REST” -- [ed -- interesting! compare middle tier to service]

datastore

• riak (distributed key-value store. no central server. no master/slave with pain while you switch to a slave). wrote his own bindings.
• ditched it due to no secondary indices for data -- can only index on one piece of information at a time. if you’re doing anything remotely relational, you have to go through immense pain to model the data.

• vector clocks (?) -- tell you your data is inconsistent, but not why. huge pain to write a real app with vector clocks.

ditched riak for MySQL.

• everyone hates their datastore, no matter what it is, so hatred of mysql isn’t a good signal
• stick with something that works until you’ve proven the rest of your business model
• existing haskell sucked/were awkward, so he wrote his more

• interested in software that is easy to use, and is fast. so he invested time in writing his own bindings. quite small amount of work, especially because he abstracted riak from most of his application.

network IO performance in GHC

• used to suck, its great now. its plenty fast, according to him. I don’t understand meaning behind numbers like “90k requests per second” but he says its fast.
• two good frameworks: Yesod, which looks something like django and rails -- less featureful, but much faster. Snap (which o’sullivan used)

json

• wrote their own bindings, standard impl wasn’t good enough (had features, but wasn’t efficient enough, and wasn’t fast enough)
• theme of interested in ease of use, high performance for all his code components.
• “awful lot of inventing of wheels”? -- he says, amount of work invested was a couple of days. all this stuff combined was a week extra.
• Q: is haskell why he is so fast a coder? A: yes. but also, osullivan has strong opinions, and he already knew the desired design up front (experience) makes it faster.

other things he wrote

• threadsafe resource pooling
• app configuration
• and more, see github.com/mailrank

other tooling

• used cabal (standard build too)
• really, he used cabal-dev, something about “sandboxed” which is more reproducible builds?
• jenkins for continuous builds
• deployment: tarballs of native code (not jars)
• deployment is just redirecting a symlink, then bouncing the server -- downtime is fraction of a second

Testing

• haskell has ridiculously good testing support
• type system eliminates whole classes of bugs.
• quickcheck -- he loves it -- scala has it too -- Figure out how to use it or write your own -- remarkably awesome!
• property based testing: in unit tests, you can only test the scenarios that you remember to deal with, then you run out of steam after not writing enough tests.
• quicktest fuzzes paramaters -- auto-generated garbage to flush out badness in your code. “no way to overstate how much more effective this is than unit tests”
• after this, all haskell code has no bugs, right? “yes. wait, no” -- compiler bug where a bad type at compile time would delete your source. his point is that pretty much between the type system and quickcheck, yes, you don’t have bugs. wow.

how did it work out?

• private beta -- app has been rock solid
• wrote their own http load tester

how about C# (their desktop client)? and how is that different than haskell?

• C# quite a nice language, even for an emacs veteran the IDE and reshaper is pretty nice
• LINQ is “pretty nice -- but complicated”

C# types compared to haskell

• limited local type inferences -- so achingly almost useful that its frustrating
• some languages “look like a great idea but make you write shit code”, he talks properties (smart getters and setters) he feels encourages shitty api design, and microsoft does this everywhere -- because what looks like a property access is actually “doing something ridiculously clever which will screw you without you noticing”
• com objects suck, code will look like its working but in practice in production is crap due to garbage collector interacting with shit com api design.

concurrency in C#

• haskell has STM of course, concurrent programming in C# makes him feel “sad, old, stupid -- dark ages”
• “by far” their hardest to find bugs are concurrency bugs in huge complicated app that they could only partly debug due to third party code
• is STM slower than explicit locks? => an application taht works is better than an app that doesn’t. implying its too hard to get concurrency correct without STM

shoudl you be using haskell -- it depends. here’s the criteria

• good libraries and tools almost all with liberal BSD licensing
• teaching materials are good
• community is helpful
• able to find people to hire

haskell --

• give yourself time to stuble. osullivan already knew how to think about distributed systems, but still mis-step with riak -- risk wasn’t worth it.
• tight deadline, unproven business, or don’t know exactly what you’re building -- don’t learn something new. haskell doesn’t matter.

Q: why did they need C#? because integrating with existing dotnet app.

Q: why not F#? because its less established than C#, dotnet is new to him as a unix guy, he wanted to limit his risk. much more info on getting things working in C# than F#.

Q: reference materials -- his book, also Learn You A Haskell, also Programming in Haskell (more of an intro to programming)

Q: quickcheck -- does it rely on the type system? -- not necessarily -- its ported to erlang, which has no type system, and still useful.

Q: using anything from Cloud Haskell paper? -- “inter process communication facilities -- language transparent cross address space messaging” -- not useful to this problem, not even sure if its a good idea.