ODK Part 2: How to set up your ODK Project

Raleigh Seamster, Google Earth Outreach

Buy and manage devices

Design and create surveys

Administrators

Create place to store data

Create data visualizations (maps!)

Manage data and users

Administrators | workflow

What we’ll learn

Create a form

ODK Build

XLS Forms

Create Google App Engine instance

Administrators

Install ODK Aggregate on Google App Engine

Validate or curate submitted data in ODK Aggregate

Devices

Devices

Design and create forms

Design and create forms

XForms: XML format for collecting inputs from web forms

Tools for authoring forms:

  • ODK Build
  • XLS Forms
  • 3rd party tools
    • Vellum
    • Kobo
    • Enketo
    • PurcForms

OR, edit raw XML.

Create a place to store the data

A clearinghouse for:

  • Deployment of forms
  • Submissions from data collectors

Why do I need to create a place to store data?

ODK Aggregate:

  • Google App Engine
  • AWS (Amazon Web Services)
  • a virtual machine ($999 onetime fee)
  • a local server (Tomcat Install)

Alternatives to ODK Aggregate for hosting data from ODK Collect:

  • formhub.org
  • Ona.io

Other storage solutions

ODK Briefcase

ODK is open source and customizable with additional development.

What is ODK Aggregate?

ODK Aggregate provides a ready-to-deploy server and data repository to:

  • provide blank forms to ODK Collect,
  • accept finalized forms (submissions) from ODK Collect and manage collected data,
  • visualize the collected data using maps and simple graphs,
  • export data (e.g., as CSV files for spreadsheets, or as KML files for Google Earth), and
  • publish data to external systems (e.g., Google Spreadsheets or Google Fusion Tables).

What is App Engine?

Google App Engine is a Platform as a Service (PaaS) offering that lets you build and run applications on Google’s infrastructure.

  • Requires administrator to have internet access
  • Requires free Google account
  • Can store over 1M submissions*
  • Data stored in the cloud
  • Data privacy settings
  • You maintain ownership over your data.

Billing:

  • Free usage up to certain quota

*After 7K submissions for one survey form, can not export to CSV or KML w/o customizations.

ODK Aggregate: navigate to imwca15.appspot.com

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

ODK Aggregate

ODK Aggregate:

Displays your forms and the completed forms or submissions

Admin settings for setting user access controls and permissions

Manage data and users

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Data Management

  • View and manage submissions
  • Options for data visualization and export

ODK Aggregate:

Displays your forms and the completed forms or submissions

Admin settings for setting user access controls and permissions

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Form Management

  • Upload forms
  • View and delete forms

ODK Aggregate:

Displays your forms and the completed forms or submissions

Admin settings for setting user access controls and permissions

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Admin Settings

  • Add new users
  • Change user types and settings
  • Preferences

ODK Aggregate:

Displays your forms and the completed forms or submissions

Admin settings for setting user access controls and permissions

Create data visualizations

Export data as CSV, JSON, or KML

Stream data to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Data visualization options

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Hands-on: Set up ODK Aggregate

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Hands-on activity: overview

Requirements

  • Google account (not Apps)
  • Java installed on laptop

Basic Steps:

  • Create a new App Engine project
  • Download and install ODK Aggregate on your laptop
  • Upload ODK Aggregate to the App Engine project

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Hands-on: create a new App Engine project

  • Log into your personal Google account.

  • Click “Create Project”

  • Give the project a name

  • Click “Create”

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Hands-on: Download and install ODK Aggregate on your laptop

  • Open a new tab and go to http://opendatakit.org/downloads/

  • Select and click on appropriate version of ODK Aggregate to install.

  • One the details page for the installer, click “Download” to download the file

  • Once downloaded, install the file.

  • Follow the ODK Aggregate Setup Wizard

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Hands-on: Upload ODK Aggregate to App Engine

Before you begin: Your Google account security settings will likely prevent the following steps from ending in success. You may need to turn off blocking access to less secure apps before proceeding:

  • Open a browser tab and go to https://www.google.com/settings/security/lesssecureapps
  • In the white box titled “Access for less secure apps”, check the “Turn on” box.
  • After you complete the steps below successfully, feel free to return to this page and turn off access.

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Hands-on: Upload ODK Aggregate to App Engine

  • In the Application or Terminal window, enter your Gmail account and password
  • A command line should run for about 1 minute or less, ending with message: “Update completed successfully.”
  • If the upload script fails, you can fix any errors and start it again by clicking on the UploadAggregratetoAppEngine file in the ODK directory on your computer.
  • Double-check whether ODK Aggregate was successfully uploaded to your AppEngine instance. Go to <your project id>.appspot.com and log in.

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Test your success!

  • Go to your App Engine URL - do you see Aggregate?
  • Upload a form
  • Configure your Android settings to point to your Aggregate instance
  • Download a form
  • Submit data via the form
  • See your submissions in ODK Aggregate

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Manage users

  • Change your admin password.
  • Add more users (if desired).
  • Change the user settings for anonymous user (i.e. the public).

Caution: There is not an easy recovery option if the admin forgets their password. Make sure the admin user name / password is remembered.

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Billing options

  • You are primarily charged for “Datastore Reads”
    • ~$0.06 per 100,000 reads
    • Every view of 1 submission = 1 read
    • One submissions page can show up to 100 submissions = 100 reads
    • Submissions tab is refreshed every 6 seconds x 100 reads...can add up!
    • Each stored image is 10 reads
  • Tips for minimizing usage fees
    • Do not keep browser window open on submission tab
    • Site Admin > Preferences: Disable faster background actions
  • Typical cost: ~$10/mo for survey with no images
  • Disable or set billing quotas

https://opendatakit.org/use/aggregate/deployment-planning/

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Visualization options

  • If using Fusion Tables: Set up Maps API credentials.
  • If you want your images, audio and video data to be displayed in Google Earth:

  • Go to the 'Site Admin / Permissions' sub-tab
  • Check the checkbox for: "Allow anonymous retrieval of images, audio and video data (needed for Google Earth balloon displays)"
  • Click the 'Save Changes' button

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

Other ODK Tools

  • ODK Briefcase
  • ODK Validate

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

More resources: Open Data Kit

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

More resources: Choosing your storage option

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Google Earth Outreach | Mobile Data Collection Overview

Open Data Kit | ODK Aggregate

More resources: Best practices

ODK Aggregate:

Export your data as CSV, JSON or KML

Send your existing data and stream new submissions to Google Spreadsheets or Google Fusion Tables

Thank you!

IMWCA15: ODK Project - Google Slides