1 of 28

Data Acquisition and Management�Unit 6 – Data Collection and Database Optimization � � �

Empowering Colleges:

Growing the Workforce

Based upon work supported by the National Science Foundation under Grants DUE 1304591, DUE 164409, DUE 1700496, DUE 1937177, Due 1938717 DUE 1937237, 2030206 and 2015927. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Author: Wing Cheung

Title: Professor, Palomar College

Assistant Director, GeoTech Center

Email: wcheung@palomar.edu

​

Source: Unsplash

2 of 28

DATA COLLECTION

DIFFERENT STRATEGIES FOR OBTAINING GIS DATA

Source: Unsplash

3 of 28

Types of data capture strategies

  • Primary data capture (first-hand)
    • Data collected in GIS compatible formats
    • Digital drone and satellite images, Surveying and GPS data
  • Secondary data capture (second-hand)
    • Data need to be converted to GIS formats
    • Historical paper maps, Scanned aerial film photos

Source: Humboldt State University

Source: Esri

4 of 28

Primary data capture: raster data

  • Drone and satellite images
    • Remote sensing
    • Resolutions:
      • Spatial
      • Spectral
      • Temporal
      • Radiometric (e.g. 2^8 vs 2^2)

Source: gisgeography.com

Source: (Kim and Yeom, 2015)

Source: NASA

5 of 28

Primary data capture: raster data�(RGB - DJI Zenmuse X5)

​

​

400-700nm

Image credit: B&H Photo, NASA

6 of 28

Primary data capture: raster data�(Multispectral – MicaSense RedEdge)

​

​

​

​

​

​

400-850nm

​

Image credit: Micasense, Quadrocopter, NASA

7 of 28

Primary data capture: raster data�(Thermal - DJI Zenmuse XT)

Image credit: The Drone Pro Shop, DJI, NASA

750-1350nm

​

​

​

8 of 28

Primary data capture: raster data�(LiDAR - Quanergy M8)

Image credit: Quanergy, CanDrone, NASA

900nm

9 of 28

Primary data capture: vector data

  • Field surveys
    • Professional surveying equipment
    • Global navigation satellite systems (GNSS, Real time kinematic GPS)
    • Mobile devices (phone, tablet)

​

Robotic Total Station

Source: SCCS

Real time kinematic GPS

Source: (Nasrullah, 2016)

10 of 28

Secondary data capture: vector data

  • Coordinate geometry (COGO)
    • Direction and distance descriptions
    • Parcel boundaries, road centerlines

Source: Esri

11 of 28

Secondary data capture: vector data

  • Manual and heads-up digitization
  • Automated digitization

Manual digitizing with a puck

Source: LearnGIS

Heads-up digitizing with a mouse

Source: LearnGIS

12 of 28

Secondary data capture: raster data

  • Scanned images
    • Georeferencing
    • Vectorization through digitizing

Source: Colortrac

Source: David Rumsey Map Collection

13 of 28

Editing Workflow�

  • (Create new feature class layer)
  • Edit geometries
  • Edit attributes
  • Save edits
  • *Failure to save edits first may prevent other changes*

​

14 of 28

Common digitization errors

  • Overshoots and undershoots
  • Dangles
  • Sliver and overlap polygons

Source: gisgeography.com

15 of 28

Data collection consideration

  • Quality, speed, budget tradeoff
  • Pilot project
  • In-house vs. outsource

Source: Unsplash

16 of 28

DATABASE OPTIMIZATION

IMPROVING DATABASE PERFORMANCE THRU SYSTEMATIC DATA ORGANIZATION

Source: Unsplash

Source: Flickr

17 of 28

Indexing

  • Speed up query and data retrieval
  • Attribute indexing
  • Spatial indexing

Source: pdfindexgenereator

18 of 28

Attribute indexing

  • Grouping records in alphabetically
  • Faster attribute query
  • DO for frequently queried fields
  • NOT for:
    • Frequently edited fields
    • “Simple” fields (e.g. binary values)

​

19 of 28

Improving search performance by organizing record values based on its initial letter

Source: (Nasser, 2014)

20 of 28

Spatial indexing

  • Divide study area into small grids and track features in each grid
  • Faster spatial query and navigation

​

21 of 28

Improving spatial query performance by narrowing down areas/grids that have features in them

Source: (Nasser, 2014)

22 of 28

File Geodatabase Compression

  • Group duplicate data values
  • Read-only, cannot edit data or schema after compression

23 of 28

  • Best practices:
    • Backup geodatabase first!
    • Lossless (reversible) vs. lossy (irreversible) compression
    • Relationship and topology classes can be compressed
      • Compressed feature classes cannot be in a topology
      • Compressed table can be in a relationship
    • Potential significant data storage savings

OID

Serves_Beer

1

Yes

2

Yes

3

Yes

4

No

5

No

6

Yes

7

No

8

No

9

No

10

Yes

11

Yes

OID

Serves_Beer

1,2,3,6,10,11

Yes (6)

4,5,7,8,9

No (5)

Pre-Compress

Post-Compress

File Geodatabase Compression

24 of 28

Simple geometry (layers containing features with fewest to no vertices) benefits the most from file geodatabase compression

Source: (Nasser, 2014)

Source: Esri

25 of 28

Compaction

  • Monthly cleaning/vacuuming
  • Physically delete records (not just mark them as deleted)
  • Faster query performance
  • Reduce storage size
  • After intense editing

26 of 28

Pre-Compaction

Post-Compaction

Source: (Nasser, 2014)

27 of 28

Conclusion

  • Many ways to acquire GIS data
    • Primary (first-hand) data collection
    • Secondary data conversion
  • Manage data and optimize performance
    • Organizing data and eliminating redundancies

Source: Unsplash

28 of 28

See GeoTech Center website (https://geotechcenter.org) �for additional Model Courses and other curriculum resources. �This Model Course Is Licensed Under Creative Commons CC BY-SA ����� By: https://creativecommons.org/licenses/by-sa/4.0/ ��Note: some content is a derivative of other CC authors���

Author: Wing Cheung

Title: Professor, Palomar College

Assistant Director, GeoTech Center

Email: wcheung@palomar.edu

​