1 of 22

The WorldMap Story, Chapter II:

From Homegrown Code to Software-as-a-Service

11th Forum on Spatially Integrated Humanities and Social Science

Zhejiang University, July 2021

Benjamin Lewis, Harvard Center for Geographic Analysis

​

2 of 22

Outline

  • A brief history of WorldMap
  • Description of the current system and pros and cons
  • Description of the AGOL system and pros and cons
  • How we made our decision
  • Steps in the migration work
  • Challenges encountered
  • Conclusions
  • Partners involved

​

​

​

3 of 22

A Brief History of WorldMap

2008 - AfricaMap created to enable researchers to more easily visualize, create, and share spatial data.

    • We were inspired by Google Maps and Google Earth which were revolutionizing the way people could explore geographic data online.
    • The goal: to put this new “slippy map” fluidity into the hands of any researcher, and enable them to publish their own content this way.
    • But there were no off-the-shelf platforms
    • So we built our own out of open source components.
    • AfricaMap, then ChinaMap and other collections

2011 - Moved content to GeoNode to create WorldMap a GIS-oriented content management system to let people create their own map collections

(continued)

​

4 of 22

5 of 22

6 of 22

A Brief History of WorldMap (continued)

​

After its release WorldMap grew quickly

​

​

​

​

​

​

​

​

Eventually commercial systems started catching up. Our uniqueness became less evident.

The burden of maintaining a growing system kept increasing - we became concerned about maintaining our data online over the long term

We reached out to Esri and they agreed to partner with us to host WorldMap.

​

7 of 22

The original WorldMap system

The WorldMap version of GeoNode is a Django (python) based content management system which wraps several open source geospatial components: PostGIS, GeoServer, PyCSW, ExtJS and OpenLayers.

WorldMap GeoNode is also integrated with with another system, HHypermap which combines Solr, Celery, RabbitMQ, MapProxy.

This is a complex system which requires significant maintenance.

​

8 of 22

How well does original WorldMap support the needs of CGA and the scholars we serve?

Pros

Decent functionality, good performance (when it is running well), code sharing, flexibility of development environment

Cons

No analytic capabilities, complexity of software stack, software maintenance, difficulty with upgrades and improvements, hardware hosting requirements, need for data and system backups, need for 24/7 monitoring

9 of 22

What is Esri ArcGIS Online (AGOL)?

AGOL was first released in 2012. It is an online mapping environment made available as a “Software as a Service” or SAAS platform.

Both the administrator and user access all AGOL functionality via a web browser. Administrator can create users and groups. Users can create, upload, edit, visualize, analyze, and share GIS data and stories. Harvard has been using AGOL for many years and has a license to it, along with other Esri products.

AGOL integrates with the ArcGIS family of desktop and web-based technologies, but is also independent of them.

​

​

10 of 22

How well does AGOL support needs of CGA and the scholars we serve?

Pros

No hardware or software to maintain, no data or system backups needed, strong analytical capabilities, functionality expected to improve as AGOL improves. Because of reduced overhead for CGA, there is a better chance to keep data online over the longer term.

Cons

We must trust Esri for everything from system uptime to backups. We don’t have access source code. Customization of system is limited by what AGOL supports.

11 of 22

Based on these factors we chose to migrate to AGOL

  • Better functionality for users
  • A more dependable system for users
  • A better chance of keeping user data online over the longer term

​

12 of 22

Over the past year...steps in the migration process

  1. A copy was made of the WorldMap PostGIS database and stored on Amazon EC2
  2. Scripts were written to convert data and metadata to form ready to load to AGOL and capture audit
  3. Map styles were retrieved from GeoServer and stored on EC2
  4. Converter written to convert map styles from SLD to AGOL format
  5. Defined approach for reformatting metadata and tags in new AGOL WorldMap
  6. Testing of data migration
  7. Actual data migration
  8. QAQC of data migration
  9. User accounts released to users - sent credentials to each data owner with new password
  10. User review period to prepare content to be made public (currently happening)
  11. Make data public July 12th
  12. Start redirecting traffic from worldmap.harvard.edu starting July 15th

13 of 22

Flow diagram of migration

14 of 22

Statistics on migration

Number of layers migrated to AGOL ~30,000

Number of vector layers ~ 25,000

Number of raster layers ~ 5,000

Number of users ~ 5000

​

15 of 22

Challenges

  • Sheer number of layers (we needed to use a premium instance temporarily to reduce upload time)
  • Messy data (WorldMap data was added by thousands of users over a period of 10 years with many variations in projection and file format)
  • Map styles (SLD) to AGOL map style conversion (we wrote a converter)
  • Bulk ingest of rasters to AGOL (we used a new beta Image capability to enable us to programmatically write rasters to AGOL)
  • User documentation (we created special documentation for WorldMap users because most functions are handled differently in the new system)

​

​

​

16 of 22

17 of 22

18 of 22

China content

More than 500 items on modern and historic China in the new system...

19 of 22

ChinaMap

20 of 22

Links

​

21 of 22

Special thanks to

  • Esri (the software company)
  • Harvard Institute for Quantitative Social Science
  • Harvard Hutchins Center for African and African-American Research
  • Office of the Geographer and Global Issues, Department of State, USA
  • Ritsumeikan University, Kyoto Japan
  • GeoMarvel (consulting firm)
  • Piensa Labs (consulting firm)

​

22 of 22

Thank you

For more information please contact blewis@cga.harvard.edu