1 of 13

Presents

GIS and HTCondor Working Together

Presented by

Patrick Claflin

3 of 13

Esri’s ArcGIS suite is the most common GIS platform used across higher ed institutions�in the U.S.

Clemson University has a campus site license which allows faculty and students unlimited access to GIS products

4 of 13

Esri provides GIS desktop software called ArcGIS Pro This is a Windows only software solution

Included with the installation of ArcGIS Pro is a licensed Python library called “ArcPy”

This library contains many geospatial processing functions.

5 of 13

Yale University’s GIS department came to us with a problem

How to create an Origin Destination Cost Matrix between Census Block Groups and Bank Branches

6 of 13

Example:�A 177 x 177 OD Cost Matrix �which equals�31,329 total routes calculated

7 of 13

Census Block Group Origins

Bank Branch Destinations

Calculating the OD Cost Matrix for

215,754 CBG Origins X 51,000 Bank Branch Destinations

This comes out to approximately 11 billion total routes that need to be calculated for the least amount of driving travel time

The

Challenge

8 of 13

CCGT GIS Lab Machines

34 Windows 10 PCs

8 cores each

32 GB Memory

ArcGIS Pro Installed

Our solution:

9 of 13

Step 1: We prepared the data

Took two CSV files of the bank branch and Census Block Group locations and geolocated them into Esri’s geodatabase feature class format.

We split the bank branch locations into 51 1,000 entry tables

We split the CBG locations into 22 10,000 entry tables

10 of 13

Step 2: Scheduled the 1122 jobs through HTCondor

Approximately 180 jobs were running concurrently

Each Job took approximately 25-35 minutes to complete

Each job returned an ArcGIS geodatabase containing the specific table that was processed.

11 of 13

Step 3: Converted the geodatabase feature classes into CSV files.

The feature classes were copied from each individual geodatabase into a central geodatabase

Each feature class was exported to a CSV file

We didn’t merge the CSV files together because an 11 billion record CSV file is too unwieldy

12 of 13

Finished Stats

1122 successful jobs ran to create 1122 individual tables of OD Cost Matrix data
The total processing time for all 1122 jobs was approximately 5 hours
The total time it took from start to finish including uploading data to cloud storage was approximately 2 weeks*

* An Issue came up using Esri’s ArcPy Python library with their licensing scheme. This would cause job failures when trying to import the ArcPy library. This issue had to be corrected before the jobs could be run under HTCondor.

13 of 13

Thank You

Questions?

Please feel free to email me if you’d like to ask questions or learn more about the process

Patrick Claflin

System Administrator/Developer�Clemson Center for Geospatial Technologies (www.clemsongis.org)�Clemson University

pat@clemson.edu