1 of 64

Kite Mapping

Pascaline Alexandre, Julien Barde , Sylvain Poulain, Sylvain Bonhommeau,

Emmanuel Chassot, Beenesh Motah

2 of 64

Summary: different steps

Expected products:

  • Phase 0: Main goals and scientific aspects
  • Phase 1 (Fieldwork): collect photos and GPS tracks
  • Phase 2 (Once back at work): pre-processing => infer location of photos
  • Phase 3: Data processing: photogrammetry

​

3 of 64

Scientific aspects

Goals, expected products & needs:

  • Marine biology => sampling protocol (~ fly / ride plan, ~ Waze app)
    • Biodiversity => occurrences: recognition of individuals, species, habitats
    • Coastal mapping,
    • Long term monitoring:
      • seasonality of species distribution (eg sea cucumbers),
      • coral bleaching
    • Image recognition => by using deep learning with spatial and temporal dimensions
  • Citizen science => facilitation of public participation (“Citizen Kite")
  • Infrastructure => Large amount of data (~ 500 000 photos / year / person)
    • Data Infrastructure: archival and storage (Zenodo, Ifremer)
    • Spatial Data Infrastructure => metadata for data discovery and servers for data access
    • 3D printing => print the box to equip boards

4 of 64

Citizen science ?

Different sensors:

  • Connected watch
  • Phone
  • Action camera
  • …

​

Different platforms:

  • Snorkelling,
  • Kitesurf,
  • Surf,
  • SUP / Paddle,
  • Kayak,
  • Buoy…

​

Tools and bravery:

  • Driller,
  • Epoxy,
  • (Plexy)Glass,
  • …

​

​

5 of 64

Material

Sensors

Paddle

Kite board

Kite (aerial survey)

6 of 64

Citizen science

300€ < Cost < 1000 €:

  • Watch or Phone (~ from 100€)
  • Action camera (~ from 100 €)
  • Kitesurf, surf, Paddle,Kayak,Boue…(~ from 500€)

Data :

  • GPS tracks,
  • Geolocated images (> 500 000 images / person / year),
  • Customized sensors (e.g. temperature, depth, …).

Products :

  • Area of practice,
  • Underwater georeferenced photos,
  • Fieldwork for ground truth (remote sensing image processing)
  • Photogrammetry

Partnerships:

  • Schools, Kite Clubs,
  • Research institutes, NGOs..

​

7 of 64

8 of 64

Citizen science => connected watch (or phone)

9 of 64

A few numbers and comments

50 sessions per year for one person (~ 1TB of photos)

  • Kite
    • 50 km per session (~ 2 hours), can be anywhere (almost)
    • One image every ½ second (14 000 images per session => ~ 30 GB) or a video (bigger)
      • poor quality when fast and shallow water
      • Kite sessions (and tracks direction) depend on the wind
  • Surf
    • 5 km per session
      • Great quality but same area and few spots
  • Paddle
    • 10 km per session
      • Great quality, anywhere
      • Anywhere if not too much wind
  • Snorkeling...cf workshop in Zanzibar

​

10 of 64

Computing

Using infrastructures:

  • R programming language:
    • RStudio in a VRE (Virtual Research Environment: example )
    • R packages:
      • Exifr to extract photos metadata
      • trackeR to handle GPS data or cycleRtools
      • R for OGC standards: Geometa / Geonapi / Geosapi / ows4R…
  • Servers:
    • SQL: Postgres / Postgis
    • Metadata (CSW via Geonetwork), Data (WFS via Geoserver)
  • Correlation of photos and GPS tracks: with Postgis (or Gpscorrelate or exiftool)
  • Photogrammetry: techniques from UAV applied to marine usage (OpenDroneMap / MicMac)

​

​

11 of 64

Material: good spots for good time data collection

12 of 64

Material: remote sensing waiting for ground truth

13 of 64

GPS tracks with a watch (or a phone..)

14 of 64

Material: kite GPS tracks (multiple sessions)

15 of 64

Material: kite GPS tracks (one session)

16 of 64

Material: surf GPS tracks

17 of 64

Material: paddle GPS tracks

18 of 64

Material: on the kite (aerial view)

19 of 64

Material: on the kite (aerial view)

20 of 64

Material: on the kite (aerial view)

21 of 64

Scientific aspects: image recognition

Paddle (corals)

Kite (corals)

22 of 64

Scientific aspects: image recognition

​

Kite (seagrass)

Kite (seagrass)

23 of 64

Scientific aspects: image recognition

​

Kite (algae)

Kite (sea cucumber)

24 of 64

Scientific aspects: image recognition

Kite

Snorkeling

25 of 64

Scientific aspects: image recognition

Kite

Paddle

26 of 64

Products: infer geolocation of photos (Postgis)

27 of 64

Products: image geolocation (google map example)

28 of 64

Deep Learning => need for scientific neural networks

29 of 64

Product: image recognition (Azure deep learning)

30 of 64

Product: image recognition (AWS deep learning)

31 of 64

Product: image recognition (AWS deep learning)

32 of 64

Product: photogrammetry (One Eye, Mauritius)

​

33 of 64

Summary: different steps

Expected products:

  • Phase 0: Main goals and scientific aspects
  • Phase 1 (Fieldwork):
    • Checklist
    • Collect data
  • Phase 2 (Once back at work): pre-processing
  • Phase 3 (Anytime): post-processing (examples of products)

34 of 64

Fieldwork

Expected products:

  • Checklist:
    • Don’t forget anything (GPS tracker, action camera, kite gears…)
    • Keep your device on time:
      • GPS tracker : watch, smartphone,
      • Camera: update it.
  • GPS doesn’t work underwater !
  • GPS and photos correlation => need to calculate an offset
    • Update the clock of your watch / phone / GPS before taking a picture
    • GPS time is in UTC ! (not the time of the watch or the smartphone),
    • Take a photo of the GPS tracker
    • Timestamps with time zone is a key issue

​

​

35 of 64

Fieldwork : set up action cameras

Important points:

  • Digital clock drift quickly => update it with GPS before
  • Take a picture of the GPS clock with the action camera
  • Exploitation of underwater and aerial photos would be better with an IMU
  • Maps:
    • Watch
    • GPS / Hero 5 and issues
    • => Correlation
    • => RTK for accuracy
    • => IMU for orthorectification (https://github.com/JuanIrache/gopro-utils)
    • Photogrammetry

36 of 64

Summary: different steps

Expected products:

  • Phase 0: Main goals and scientific aspects
  • Phase 1 (Fieldwork):
  • Phase 2 (Once back at work): pre-processing
    • Describe metadata for the session, structure datasets
    • Raw data processing
      • Extract raw metadata (to archive) and load them and GPS tracks in Postgis database
      • Correlation with gpscorrelate / exiftool / Specific Postgis method to locate photos
      • Extract pictures by area / date / …
      • Label images and train Neural Networks => image recognition
  • Phase 3 (Anytime): post-processing (examples of products)

​

37 of 64

Phase 2 => Back at work

Metadata of sessions / deployments:

  • Once back from fieldwork:
    • New file structure:
      • Naming conventions for repositories and files
        • DCIM for photos
        • GPS for spatial information
        • METADATA for metadata (exif, thumbnail, OGC)
        • LABEL for training datasets (deep learning)
    • description of data with Metadata in a google spreadsheet which can be replicated directly in a table of a (Postgis) database,
  • R codes to read and transformed sessions metadata in OGC metadata (R geometa package), pushed in Geonetwork (R geonapi package)

​

​

38 of 64

Phase 2 => Back at work: files structure & naming

39 of 64

Phase 2 => Back at work

All codes on a Github repository :

  • Download / update codes from Github repository: R + SQL
  • R codes (can be executed online with RStudio server ):
    • Packages:
      • Exifr => Photos metadata stored in a CSV are uploaded in a Postgis database (COPY command) ,
      • rgdal, trackeR (cf doc) => extract and transform the GPS tracks (TCX, GPX, RTK)
      • Geometa and Geonapi => create OGC metadata and push them in Geonetwork,
      • Others: dplyr, data.table, RPostgreSQL..
      • Execution of codes online with RStudio server
  • SQL: Postgres & Postgis database (online Postgres server),
  • Need to add the ODM part.
  • Need a Docker container (at some point).

​

​

40 of 64

Geolocated photos => metadata storage in Postgis

Photos metadata stored in a “materialized view” of the postgis database:

  • Table metadata_sessions : from google spreadsheet
  • Table metadata_photos : to store metadata of the photos
    • Exif metadata / except thumbnail ?
    • Specific metadata:
      • photo type (aerial, underwater, out of water…)
      • If used for time calibration or geolocation
      • speed
  • Table label_photos: spreadsheets
    • identifier_photo: identifier of the photo,
    • identifier_category: identifier of the category found in the photo
    • use: if this photo should be used for validation, training…
  • Category: inventory of categories to be found in the set of photos

41 of 64

Geolocated photos => metadata storage in Postgis

42 of 64

Phase 2 => Back at work

  • Set up working directories for codes, metadata and data (GPS tracks and photos)
  • R exifr package to extract exif metadata from aerial or underwater photos and load them in the Postgis database (command => exiftool -n -j -q -b 'G0031487.JPG' ack_Rocks/DCIM/131GOPRO/not_synchronized)
  • Correlation of GPS tracks and camera clocks to infer the location of photos:
    • GPS tracks processing: R packages (trackeR:readTCX for TCX, rdgal:readOGR for gpx, home made code for RTK) to read GPS trackers data and load them in the Postgis database:
    • Photos location inferred by SQL Query (using Postgis ST_LineInterpolatePoint function) and storing results in a materialized view.

43 of 64

GPS tracks and photos processing

Main goals:

  • Extract and transform the file:
    • Formats: TCX, GPX, RTK,
      • Location (X,Y,Z)
      • Pace...
    • Calculate speed...,
  • Generate a map:
    • Tracker
    • Leaflet,
    • Qgis..
  • Infer location of photos

​

​

44 of 64

Postgres / Postgis database exploitation

Once metadata of photos are stored in a Postgis database, it is possible:

  • To discover photos made available by multiple contributors and sessions from a single data source and persistent storage (instead of reading exif metadata loaded in RAM),
  • To query photos within a given location (eg know sea grass spot):
    • To facilitate data annotation
    • To get a set of photos for photogrammetry / a mosaic
  • To feed a photogrammetry process

45 of 64

Summary: different steps

Expected products:

  • Phase 0: Main goals and scientific aspects
  • Phase 1 (Fieldwork):
  • Phase 2 (Once back at work): pre-processing
  • Phase 3 (Anytime): post-processing (examples of products):
    • Example of photogrammetry with OpenDroneMap
    • Outlooks: mosaic

​

46 of 64

Post processing : example of products

  • Simple mosaic => False Orthophoto
    • Fast to generate
    • not accurate : false ortho
    • No georeference

​

  • Photogrammetry, Structure from motion : �3D => Bathymetry + True Ortho�

=> Processes derived from aerial drones

47 of 64

Photogrammetry

Reminder =>

48 of 64

Photogrammetry

REAL WORLD 3D

2D IMAGES

DIGITAL 3D

49 of 64

Photogrammetry - processes

Tie points calculation and Bundle Adjustment

50 of 64

Photogrammetry - processes

​

Images calibration

​

51 of 64

Photogrammetry - processes

​

Sparse point cloud

​

52 of 64

Photogrammetry - processes

​

Dense point cloud

​

53 of 64

Photogrammetry - processes

​

Mesh + Texture generation (optional)

​

+

54 of 64

Photogrammetry - processes

​

DEM computation

​

55 of 64

Photogrammetry - processes

​

Orthophoto computation (from dense point cloud or DEM)

​

56 of 64

Photogrammetry - How to process ?

Main steps:

  • Data preparation
    • Images selection
    • Metadata manipulation
    • Modify images :
      • Resizing for fast processing
      • Writing in Exif
  • Photogrammetry software used in UAV Mapping
    • ODM
    • MicMac

57 of 64

Quick Overview from user: open source softwares

​

​

�OpenDroneMap (ODM) WebODM

  • Web interface
  • Easy to understand and handle
  • Easy to deploy : Docker and script for Docker (do not need to know docker)
  • Need to have GPS tags in exif
  • No GPU computation
  • Sometimes not that good but it’s just the beginning !

​

​

​

​

� MicMac

  • Command line with ton of parameters, not so intuitive
  • GUI exists but need to be enhance (InterfaceCEREMA and Qgis 2 plugin from Lia Duarte)
  • More scientific approach in my opinion
  • Limited GPU computation

​

Full process from images to final product : Point Cloud + Mesh + DEM + Orthophoto

58 of 64

Accessing to WebODM:

59 of 64

Micmac interfaces

60 of 64

Micmac interfaces

From Lia Duarte

61 of 64

Product: photogrammetry

​

One Eye, Mauritius

62 of 64

Conclusion

  • Products: geolocated (underwater ) photos, mapping, photogrammetry
  • Codes are (almost ready) to be reused by anybody => Github repository
    • Pre-processing entirely done with R & Postgis
    • R packages to manage collected data in a Spatial Data Infrastructure: ows4R (OGC WxS), geometa (OGC 19139), geonapi (binding geonetwork API), geosapi (binding geoserver API), RPostgreSQL
    • Post-processing:
      • Deep learning => preparing training data (spatial and temporal query to extract photos from Postgis metadata database)
      • Photogrammetry => extract set of photos from Postgis
  • Collaborative project:
    • requires multiple skills and fundings => applying SDI and drones methods,
    • requires research infrastructures for data storage, data discovery and access, data processing
    • Keep everything open => data (“as open as possible, as closed as necessary”) / sources codes / hardware

63 of 64

Outlooks

  • Try other spots (not only coral reefs and lagoons),
  • Try multiple sensors to be plugged in a generic box:
      • IMU for orthorectification: to manages angles of photos (kite and boards)
      • RTK for centimetric accuracy of GPS location
      • Environmental parameters (depth, temperature, pH…)
  • Deep learning
    • Filter photos which can’t be open data => remove endangered species
    • Automated annotation of objects => layers attributes for coastal mapping
  • Partnerships: everybody’s welcome => collaborative project (Citizen Kite)
    • Kite centers
    • Education and research
    • Private stakeholders

​

64 of 64

Contacts

For more information, please contact us:

​

​

Github repository : https://github.com/juldebar/Deep_mapping

​

​

​