Hello from the 90s.
I’m Howard Butler.
and I think I'm here because I'm good
at tweeting pithy in-jokes for
geospatial programmers who came up in the late 90s.
We all know how what can happen
when people are chosen for
their tweeting abilities.
I hope to provide some nuggets about
platforms for geospatial software,
overlay my software development history with GISes
describe how platforms are mixing
together faster than the 90s and 2000s
describe how good open source
software **build with** rather than **instead of**
show a little LiDAR in a browser,
and talk about how this all relates to our now favorite computing platform
** colosseum slide **
The foundation of one software civilization is the bones of another.
Like many, I got to geospatial software development in an unplanned and happenstance way.
** apple computer slide **
While I had a 80s nerd starter kit of an Apple IIgs
with BASIC and a 40 mb hard drive,
the reality that I would become someone
who makes their living giving away software
is still astonishing to me.
** tommy slide **
I grew up on a corn and soybean farm in Southern Minnesota.
I spent many summer days and fall nights in the fields,
Helping to plant, maintain, or harvest crops.
Throughout high school, I was given more responsibility and information about how to successfully do that.
Scratching out a living in the dirt wasn’t just a metaphor,
and I went to university looking to become better at it.
** agronomy slide **
I bounced around a few majors in college,
including some computer engineering,
before I settled on agricultural technology development.
I even took a course series
at Iowa State University conveniently nicknamed
and Super Duper Dirt.
Getting closer to graduation,
I started looking into geospatial technologies
to make a difference
for crop farming in the upper Midwest.
I soon came across a software program that would make that all crystal clear for me
** arcview slide **
ArcView gave me the ability to overlay whatever I wanted.
** population map slide **
In the hands of a newbie,
it's like an automatic spurious relationship finder, and
** population map slide 2 **
I was fantastic at creating horrific cartography
and computing indescribable statistics
of mostly just population maps.
** population map slide 3**
Eventually, button and menu functionality weren't enough for
me or my major professor,
and I needed to script to graduate.
** avenue cover slide**
Avenue my first real workaday computing platform
Avenue was a SmallTalk-related language
that powered the GUI and data
processing objects in ArcView 3.
Before I started with ArcView, I had the basics of software development down --
but in the course of three or four years,
I became a max-level Genji with Avenue.
** genji slide **
With it, I could DPS my way through anything in ArcView.
I could sling vectors and rasters around with impunity,
but despite my skills, I was pretty easy to one-shot.
Avenue was a closed system with a weak software ecosystem around it.
There was some user-to-user script sharing,
and it did allow scripts to call
shared DLL libraries.
At the 1999 ESRI users conference, I saw
power utilities mapping electrical grid
energy state in real time button clicks.
God, I know this sounds lame now,
but in the time of cargo pants, it really wasn't.
** rootbound slide **
ESRI became rootbound by the success of ArcView, however.
ArcView also integrated poorly with COM for Windows connectivity. If you think cargo shorts are ugly, write some COM software
** netscape slide **
ESRI pulled a Netscape and rewrote
themselves from the ground up to be
Windows First and
I’m sure they had their reasons.
** bubba slide **
Along the way they jettisoned
my GIS security blanket -- Avenue.
** Dodson slide **
During the transition, an ESRI Canada programmer
named Bruce Dodson
posted a DLL that hooked Avenue
to the Python programming language
and allowed you to eval whatever
you wanted through either interpreter.
Python 2.0 had just come out, and geospatial software in the Python space was dominated by GDAL and PROJ Python APIs
Ok, maybe dominated is too strong of a word.
There were GDAL and PROJ Python APIs,
but that was about it for geospatial Python at the time.
Yes I know those Python APIs still look
like they did 18 years ago.
** AVPython article slide **
Python access in ArcView gave me a transition path
that allowed me to build upon my past work
plus it gave me a massive productivity boost
in a batteries-included language with an open source ethos.
** avTerra slide **
I jumped into Python and wrote an
ArcView extension called avTerra that
downloaded TMS map tiles into ArcView map views --
except instead of TMS
** TerraServer slide **
it was called Microsoft TerraServer
and there was only one server on the whole internet.
Instead of any sane URLs and JSON,
it used a cargo shorts kinda thing called SOAP to do its communication with XML.
Python gave me super powers, and
I was able to use it to fill map views
with ten-year-old 3 meter black and white imagery.
I still take solace in knowing how many grad students
I freed from the drudgery of digitizing from paper maps
to enable heads' up digitizing in ArcView.
** Java logo/shorts slide **
Actually, let's chat about Java for a minute.
The cargo shorts developer crowd
was quite excited about the grand
promise of Java in the 90s
-- write once and run everywhere --
but the devils' bargain to be struck was rewriting
your software to live on the Java Virtual Machine.
Java succeeded poorly for many, especially in the 90s.
Early on, there was a speed disparity
between native code and Java.
That’s been gone for a long time,
but the impression stuck.
** GUI programming slide *
GUI programming in Java was the normal amount of GUI-programming-is-hell
plus dealing with Java's Swing API.
JNI, or Java Native Interface,
would seem to be the integration path,
but 20 years of history shows it to be
for Himalayan mountains goats
rather than a 6 -lane California highway of integration bliss.
The dominant problem of the time was
native windows applications,
and 90s Java was poorly suited for it.
Java didn't exactly fail, it's still everywhere, even today.
The market transition to Java didn't go as intended, however.
It didn't end up replacing everything and running everywhere.
The reason why was the cost of the bargain was too high --
porting working code to new environments
creates new bugs in addition
to carefully copying over your old ones.
The bargain meant tossing out battle tested designs.
Java poorly allowed systems to build upon their past,
and the entire global software library had to be rebuilt from scratch to live in Java.
Decades long investment still hasn't produced
a feature-parity pure Java GDAL or PROJ library.
Instead of one-for-one replacing of existing systems,
Java attracted new development that desired the JVM,
and older systems stayed where they were --
Ultimately tied by some C or Fortran compiler on native platforms.
In contrast to Java, Python
focused on integration with native code.
That investment has paid off in many of the
numerical computing, machine learning, and geospatial codebases now using it as their primary language.
Just a second, let's talk some more about platforms.
In ten years, in 2027, what platforms will still be alive?
** windows slide *
Windows should still be here.
Microsoft proved that getting their heads bashed
in on mobile can do wonders for their ability
to think about software development platforms.
They now make tools that play well with others.
They build on and with things they dont actually make or control.
They actually now care that you're using their tools.
The Microsoft in the time of peak cargo shorts only sweatily claimed they did all this.
** android slide *
Android will still be there.
Android is our friend Java and a bunch of JNI.
It's even GUI programming in Java too.
Android is a platform with two billion users, with the most entrenchment on the most successful hardware ecosystem in the history of computing.
** iOS slide *
iOS of course won't be dead by then.
iOS has the snobbiest installed base,
with its jail-like pastiche, emphasis on security, and stockholm syndrome, users can be counted on to ask for another.
** OSX slide *
OSX will be dead by then.
OSX is the Lannisters of platforms.
It needs to pay off its technical debt at the Iron Bank of Braavos, and It looks pretty but kills everything it touches.
** Linux slide *
Linux of course.
As a containerized platform, or a platform of containers.
Either way, Linux is the “78 percent of air is nitrogen” of platforms -- it is so assumed we don't even bother to think about it.
** Browser logos slide *
just about every computer has a browser.
every one of the native platforms
I just talked about.
This platform is as strong as ever as evidenced by
this conference and its evolution from talking about
novelty browser mapping capabilities
to full scale replacements for the entire spectrum of GIS.
Ok, maybe that's rather cliche.
the language is awful and arbitrary.
I think significant whitespace is a
perfectly reasonable way to block code.
once forced me into Java's devils' bargain
of rewriting everything I had already written.
A thirty year old, battle tested cartographic projection library?
A geometry algebra engine?
A geospatial format translator?
A lidar compression library?
the same way we have them in many
other languages, but ports are worse than forks.
All of the platforms I described earlier aren't equal,
but they all have something that can target them
** LLVM slide *
LLVM is just a name now,
although low-level virtual machine might once
have been a fitting acronym.
It has quietly infiltrated most
developer tools in recent years.
Native mobile developer tooling and continual investment
in LLVM by Apple and others pushed it over the threshold of resistance futility.
LLVM is compiler technology.
It's different than others in that it breaks the
compilation problem up into pieces.
The typical compiler scenario we might think of
is having a C or C++ frontend and
a machine backend that is x86 assembly.
Or how about a Python frontend and a JVM backend?
This exists today.
You can use the LLVM toolchain to compile C or C++
The tools to google are asm.js, WebAssembly, and Emscripten.
These tools are in use today.
It only performs a few multiples slower in a browser than it does on bare metal.
Every release tick of the browser and every release tick of the compiler stack has made that multiple decrease.
** plasio slide *
Many of the browser-based point cloud visualization libraries use it, including Potree and Plasio.
It's not a panacea.
It was probably equivalent to porting it to a new
** denmark movie slide *
This meant a simple C++ dereference of a polymorphic variable was three or four function calls.
It took a month to refactor it, and along the way,
we added open source niceties like --- tests.
we'd have burned a month forking
the library with yet another implementation.
A port is worse than a fork.
More chances for bugs.
More software to silently fall out of date.
More compatibility to lose.
** rotten apple slide *
-- and all its other backends --
enables what I call the "Rotten Core" development model .
Programmers are very good at detecting rotten code.
It's quite easy, really.
Anything not written by they themselves is rotten.
More seriously, Rotten Core plus the LLVM stuff
means we don't have to make the platform
devils' bargain again.
We can abide with the embedded lessons
decades' old software libraries
provide our modern software
without abiding outdated APIs.
** Lynn tweet slide *
Lynn doesn't like GDAL's API,
but usable here means
**usable to her**
in her context.
What Lynn really wants is an abstraction layer,
-- an abstraction layer to the geodata abstraction library --
within the context of language and
programming idioms she already knows.
** rasterio slide *
rasterio is the most idiomatic Python
geospatial raster processing library.
It's all GDAL all the way down, but
it protects users from the most jagged bits
by hiding the necessarily complex GDAL API
and its use of C language idioms.
rasterio is Rotten Core and wraps the rotten-to-me stuff with
a convenient and idiomatic API
that has meaning within my own Python programming context.
we can start doing the same
for that environment too.
We did the same thing with the LiDAR compression
library. 5x less data transmission easily overcomes the 2x slower decompression performance. The tools are good enough to start doing this.
** GDAL.js slide *
Here’s Derek Dohler trying it on GDAL and PROJ. It’s super rough, but it appears gdalinfo works
I took over maintenance of the PROJ
library after Frank Warmerdam moved on.
It's everywhere, and it's going to continue to be
everywhere for an unpredictably long time.
The reason why is because it solves such a fundamental
problem of our domain
that once solved, doesn't
need to be solved over and over again.
None of its ports has feature parity.
PROJ is just now adding the ability
to do time dominant datum shifts.
Every feature, change, and bug fix radiates
to PROJ’s ports at time scales bested by the tectonic plates.
Software libraries that condense knowledge
of data formats and math
-- like GDAL and PROJ --
are going to be around for a very long time.
Longer than any of our careers.
Reflecting on Gerry Evenden for my PROJ talk
on Thursday made this fact very clear to me.
** diadem slide *
These libraries are heirlooms that will be patched up,
with a kaleidoscope of abstraction layers
pasted on top of them to suit
It is folly to reimplement them
just to reach a new platform,
especially now that LLVM-enabled
Rotten Core programming can
it might in any native language.
** over and over slide *
and the tools are maturing enough that
we can treat it like any other one we've been building
boring geospatial software on for decades.
I hope you'll think of
before you dive into
some brownfield reimplementation
of an open source library
that's been around for 20 years.
** cargo shorts slide *
Don't think all those pockets
aren't on there for a reason.