Aggregating Utah’s Statewide Roads Dataset

• 05.11.2017 • UGIC 2017 •

Greg Bunce

AGRC �Automated Geographic Reference Center

State of Utah’s map technology coordination office (est. 1984)

gis.utah.gov

SGID�State Geographic Information Database

one-stop access point to hundreds of statewide data layers (developed, aggregated and acquired)

utah.maps.arcgis.com (web services)

gis.utah.gov/data (data downloads)

​

SGID >>> Statewide Roads >>>

Goal

Provide a seamless, standardized, and regularly updated statewide dataset

Data Sources

City agencies�County agencies�VECC (Valley Emergency Communications Center)�UDOT

Data Users

Blue Stakes of Utah (811) �911 Community and Dispatch Centers

​

Extract

Extract the roads data from the authoritative data sources

Transform

Transform the data into statewide roads schema

Change Detection

Detect Changes between existing statewide roads data and source data

Reconcile and Load

Load updates and changes into statewide roads database

Publish

Publish the data to SGID and FTP sites (web services)

Extract

Transform

Change Detection

Reconcile and Load

Publish

Extract the roads data from the authoritative data sources...

Current Forms of Data Extraction

• Email request
• AGRC-hosted and provided AGOL Feature Layer Service
• FTP/Website Download page
• Direct Connection to agencies SDE Geodatabase
• ArcGIS Server Distributed Geodatabase (Geodata Service)

AGOL

GeoService

FTP/Download

SDE

Email

Extract

Transform

Change Detection

Reconcile and Load

Publish

Transform the data into statewide roads schema...

Python Scripts

Schema Transformation

• Each agency has unique set of needs for roads data
• Variety of schemas throughout the State
• Transform the data into SGID Schema
• Python Scripts

​

Extract

Transform

Change Detection

Reconcile and Load

Publish

Detect Changes between existing statewide roads data and source data...

Detect Feature Changes

• Compare (transformed) source data to existing SGID data - feature by feature
• Python tool built around ESRI’s Detect Feature Change geoprocessing tool
• search distance, spatial change tolerance, compare fields
• Tool outputs a feature class indicating change type
• New, Deleted, Attribute, Spatial, Attribute and Spatial, None

​

Python Tool

ArcCatalog

Extract

Transform

Change Detection

Reconcile and Load

Publish

Load updates and changes into statewide roads database...

Create User Versions for Editing

• Each editor creates a version in the roads editing database
• All edits are done via user’s version
• User reconcile and post to parent version

SDE Editing Database

… user version

… user version

… user version

… user version

Extract

Transform

Change Detection

Reconcile and Load

Publish

Load updates and changes into statewide roads database...

ArcMap Editing Environment

• Custom editor tool
• ArcObjects SDK and C#
• Tool provides a UI for the change detection output files
• Iterate through changes
• Visual inspection (features and attributes)
• Update | Import | Ignore
• Feedback loop

Extract

Transform

Change Detection

Reconcile and Load

Publish

Load updates and changes into statewide roads database...

Change Type:

New, Attribute, Spatial, Attribute and Spatial

County

Segment

AGRC

Segment

User can double-click the field name to transfer values from County segment.

Save Button: checks for expected values, formats, spatially assigns attributes (zip codes, address grid, cities, etc.)

Extract

Transform

Change Detection

Reconcile and Load

Publish

Load updates and changes into statewide roads database...

https://youtu.be/jZqsOUCc6XU

https://youtu.be/hbYOEnn9vMk

​

​

Extract

Transform

Change Detection

Reconcile and Load

Publish

Load updates and changes into statewide roads database...

Notify Agency >>> Resolve Difference

Notify options

Specify a note/description

Provide agency

spreadsheet to inspect

Extract

Transform

Change Detection

Reconcile and Load

Publish

Load updates and changes into statewide roads database...

Ignore option

on editor tool

Additional custom tool to preserve and log the segments we have ignored

Ignore Process

• When ignoring a difference, log the transaction
• The idea is to not flag/inspect the segment again if we’ve ignored it in the past
• When we get the agency’s data again, and if their data and our data is exactly the same (spatial and attributes), we have the option to utilize the ignore log to remove segments from the process

Extract

Transform

Change Detection

Reconcile and Load

Publish

Publish the data to SGID and FTP sites...

Editing Database

SGID Database

… user version

… user version

… user version

… user version

Roll-up the editing database

• Push all user versions to the Edit version (reconcile and post)
• Roll all versions to sde. DEFAULT
• Delete all child versions

​

Extract

Transform

Change Detection

Reconcile and Load

Publish

Publish the data to SGID and FTP sites...

Perform final quality checks

• Null values and empty ranges
• Range parity and order
• Missing spatial attributes
• UniqueID

​

Nulls and Blanks

Check Ranges

Assign Spatial

Looking

Forward

Shared Editing Environment

Multi-User Editing

trusted editors

Direct user input and feedback

trusted contributors

Greater Efficiency

less processing

More frequent updates

semi-live data, decrease shelf-life

ArcGIS Pro and web editing

Roads as a web service

NexGen911 and GIS (new schema)�

Thank You!

Questions?

Statewide Roads Dataset

gis.utah.gov/data/sgid-transportation/roads-system

Github Repo to custom ArcMap tool (arcobjects, c#)

github.com/gregbunce/UtransEditorArcMap

Contact Info

gbunce@utah.gov