1 of 37

We will start soon! (5 mins)
All are welcome (pets, kids, families)
Introduce yourself in chat

Name & where you’re calling in from
Favourite plant 🌴🍎

🎵Djie Ding Sranangman by Sonora Paramarera

Welcome!

Insiders call #3

2 of 37

Recording will be shared publicly after
Use the chat if you have a question
Caption enabled (please speak slowly)
Camera on if possible

Logistics

Reminder: please reserve support questions for the forum

3 of 37

👋 Welcome

🚀 Product roadmap update

⚡ Lightning talk from Rudo Kemper

🍿 Deep dive: Prevent duplicates or show summary with Entities

Agenda

4 of 37

getodk.org/roadmap

5 of 37

Lightning talk 🍿

6 of 37

Deep dive

Entities to prevent duplicates or show summary

7 of 37

User requests

Link two forms so they access each others’ data

Flow submissions from one form into another

Let forms access all submission data on device

Share resource lists across forms (e.g. districts)

Link submissions about the same subject

Show a summary of data collected on devices

The challenge

8 of 37

Multi-step

workflows

Multi-step workflows have required manual steps or programming to automate

The challenge

CSV attachments and last-saved make these workflows possible, but with limitations

Entities move away from one-off solutions to a comprehensive and non-disruptive approach

Every 6 months

Register person

Deliver stove

Measure stove impact

9 of 37

Entities help

automate workflows

An Entity is a thing used in your workflow. It can be physical (tree) or abstract (suggestion)

Our solution

Entities track state (e.g., last follow-up date) while forms define behavior (e.g., follow-up)

A form can use multiple entity lists

Can also represent static or rarely changing data shared between forms

10 of 37

You can think of…

CSVs in the cloud

Shared database

11 of 37

Should I use Entities?

Do you have a process that works well today?

Yes: don’t change it!
It’s annoying/slow/error-prone: try Entities�

Does your workflow have clearly defined, separate phases?

Yes: manually attaching CSVs is great!
No: try Entities

Our solution

Do you have data that needs to be shared between forms and updated?

Yes: try Entities

12 of 37

Data shared between forms that can be updated at any time adds complexity!

Make sure it’s worth it for your workflow

⚠️

13 of 37

Accessing Entities

14 of 37

instance('people')/root/item[place=${place}]/visits

List lookup expressions

Filter

(same as a choice filter)

Your entity list

Desired property

pulldata("people", "visits", "place", ${place})

15 of 37

instance('people')/root/item[place=${place}]/visits

List lookup expressions

Filter

(same as a choice filter)

Your entity list

Desired property

pulldata("people", "visits", "place", ${place})

16 of 37

Understanding list lookups

instance: places

root

item

geometry: 5.09 7.35 12 0

population: 270900

enumerator: 6234

item

geometry: 5.05 10.46 11 0�

population: 311300

enumerator: 2742

One or more “instances” of choice lists

Each list row is an item connected to a “root”

When there’s a filter, it’s evaluated against every item

17 of 37

instance('places')/root/item[enumerator='6234']/label

instance: people

root

item

place: c2139aae-5…

visits: 1

last_visit: 2024-5-1

phone: 2341745

item

visits: 3

last_visit: 2024-4-12

phone: 9868545

instance: places

root

item

geometry: 5.09 7.35 12 0

population: 270900

enumerator: 6234

item

geometry: 5.05 10.46 11 0�

population: 311300

enumerator: 2742

18 of 37

count(instance('places')/root/item[population > 300000])

instance: people

root

item

place: c2139aae-5…

visits: 1

last_visit: 2024-5-1

phone: 2341745

item

visits: 3

last_visit: 2024-4-12

phone: 9868545

instance: places

root

item

geometry: 5.09 7.35 12 0

population: 270900

enumerator: 6234

item

geometry: 5.05 10.46 11 0�

population: 311300

enumerator: 2742

19 of 37

sum(instance('people')/root/item[visits > 1]/visits)

instance: people

root

item

place: c2139aae-5…

visits: 1

last_visit: 2024-5-1

phone: 2341745

item

visits: 3

last_visit: 2024-4-12

phone: 9868545

instance: places

root

item

geometry: 5.09 7.35 12 0

population: 270900

enumerator: 6234

item

geometry: 5.05 10.46 11 0�

population: 311300

enumerator: 2742

20 of 37

sum(instance('people')/root/item[place=${place}]/visits))

instance: people

root

item

place: c2139aae-5…

visits: 1

last_visit: 2024-5-1

phone: 2341745

item

visits: 3

last_visit: 2024-4-12

phone: 9868545

instance: places

root

item

geometry: 5.09 7.35 12 0

population: 270900

enumerator: 6234

item

geometry: 5.05 10.46 11 0�

population: 311300

enumerator: 2742

21 of 37

Can’t you make these look up expressions easier to write?!

Yes, eventually likely similar to ${last-saved#something}

Is this a relational database?

Yes, you can think of Entity Lists as joinable tables

Why not use SQL?

SQL is also hard and not familiar to everyone
XPath is a query language that’s already part of our forms
${something} gets converted to an XPath of /data/something

Questions

22 of 37

Synthesize feedback & share on the forum
Next call:

Next steps

@adailycloud

23 of 37

Prevent duplicates

24 of 37

Prevent duplicates

Attach same Entity List that will be written to
Compare entered registration data to existing Entities
Warn or error if there’s a match

The concept

25 of 37

With unique

ID to match on

count(instance('people')/root/item[id=${id}])

Filter based on that value, count the matches
More than one match means an existing registration

Prevent duplicates

26 of 37

Prevent user

from submitting

Constraint: the value can’t match any existing item

Use the current question’s value with current()/.

count(instance('people')/root/item[id=current()/.]) = 0

Prevent duplicates

27 of 37

Warn user but

allow to continue

Relevance: show note if value matches one or more existing item

Can make the note required to make it like a constraint

count(instance('people')/root/item[id=${id}]) > 0

Prevent duplicates

28 of 37

Without unique

ID to match on

Use contains, starts-with, etc function to filter

Show the list of existing entities with search appearance

Show a map

Prevent duplicates

29 of 37

Notes

Entity list needs to be updated periodically, duplicates can happen offline

Offline Entities will address duplicates by same person

If you need server-side submission edits, think about recomputation
Anything you can do to limit overlap between data collectors will help

Prevent duplicates

30 of 37

Show work summary

31 of 37

Show summary

Attach Entity List(s) that represent completed and/or target work

Filter based on relevant values (enumerator, location, etc)

Show counts, sums, or other computed values in notes

count(instance('people')/root/item)

The concept

32 of 37

Show summary

count(instance('people')/root/item[created_by=${enumerator}])

Count of entities created by an individual

Sum of values in a location

Qualitative message

sum(instance('people')/root/item[place=${place}]/hh_members)

if(${enrolled_percent} < 50, "Let's keep trying hard to get to our goal!", "We're getting close!")

Examples

33 of 37

Note:

recompute on edit

By default, expressions are recomputed on edit

If you view/edit a submission on server, it will show an updated summary

Use once function if only reading from entity list or trigger if using form data
Can ignore this if you won’t view or edit submissions on server!

Work summary

34 of 37

Synthesize feedback & share on the forum
Next call:

Next steps

@adailycloud

35 of 37

Add all of this to documentation

Please give feedback!

Offline Entities in Collect
Make queries more approachable

Coming up

36 of 37

🎵Djie Ding Sranangman by Sonora Paramarera

Survey

Insiders call #3

37 of 37

Quick feedback survey 💌
Next call June 5th, 3pm UTC
Two Entities office hour sessions

Next steps