1 of 54

Cláudia Pinhão

UX/UI Designer

@Gemeente Amsterdam

Jason Fiawo

Copywriter

@Gemeente Amsterdam

Labeling Guide

Project Sidewalk

2 of 54

The anatomy of this guide

In this guide we want to share with you how to label Amsterdam correctly in Project Sidewalk. The guide is split into three chapters: labeling, validating, and visualizing data—each chapter has visual examples and a video.

Visit Project Sidewalk here.

Labeling data

1

Validating data

2

Visualizing data

3

Video tutorial + guidelines

Video tutorial

Video tutorial

Questions

4

Contact

3 of 54

Labeling data

HOW TO

4 of 54

5 of 54

General Guidelines

How to label?

You should place labels at the center of curb ramps or surface problems, and at the base of obstacles, otherwise it will give the wrong latitude and longitude values.

Try to get as close as possible to the object you are labeling. Labeling from farther away will result in less accurate results.

These are the currently available labels:

Curb Ramp

Missing

Curb Ramp

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

6 of 54

Do

Add a curb ramp label in the centre

Do

Add a pedestrian signal label in the centre/base

Do

Add a obstacle label in the centre/base

Don’t

Do not add the curb ramp labels in the ends

Don’t

Do not add the pedestrian signal label on the button or top

Don’t

Do not add the obstacle label on the top or ends of the obstacle

7 of 54

Levels of severity

We use severity levels to get a better evaluation how difficult would be a wheelchair user to pass by in a certain location. The levels can be defined as:

Fully passable. (Curb ramps + Crosswalks) Wheelchair users do not have to reduce their speed or change route. The route is completely fluid.

(Other labels) There is a problem but it should be fully passable for every wheelchair users.

1

2

3

4

5

Not passable. Wheelchair users cannot pass there at all and there is no alternative route.

Almost not passable. Only a few wheelchair users can pass but with some risk or a lot of effort or with help. Alternative route might be present.

Passable. Wheelchair users clearly identify the problem or obstacle but are still capable of overcoming it alone by reducing their speed, stopping or changing route.

Almost fully passable. Wheelchair users have to slightly slow down or deviate a little bit from obstacles or other problems but without thinking too much about it.

8 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Curb Ramps

What are they?

A curb ramp is a short ramp that cuts through or builds up to a curb. An accessible curb ramp is one that provides an accessible route for people with mobility impairments to safely transition from a curbed sidewalk to a roadway, or vice versa.

Where can we find them?

We can find them often on the both ends of crosswalks, or when a sidewalk continue on the other side of the street, even without crosswalk.

curb ramp

curb

plan

section

9 of 54

At both ends of a crosswalk there should be curb ramps

At street corners or along big paths without crosswalks, there might be curb ramps to cross the street

At both ends of a crosswalk with an island in between there should be curb ramps on all ends of the crosswalk including in the island

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Where shall we label curb ramps?

Curb ramp

Curb ramp

Curb ramp

10 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Tags

Curb ramps have 8 associated tags further explained below:

The curb ramp has a small width for a wheelchair user

narrow

tactile warning

steep

not enough landing space

not level with street

surface problem

The curb ramp has a textured guide that indicates to blind or visually impaired people there is a crossing

The curb ramp has a sharp angle that leads to a more difficult rising and uncontrolled falling for a wheelchair user

The curb ramp leads to a narrow space in the sidewalk where turning a wheelchair is difficult or impossible

The curb is slightly lowered but not enough to be at the same level of the street, creating a small step

The curb ramp is broken, uneven or has grass, pooled water or debris

11 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Do

Curb ramp | severity: 1 | No tags

Do

Curb ramp | severity: 1 | No tags

Do

Curb ramp | severity: 2 | narrow

Do

Curb ramp | severity: 2

Do

Curb ramp | severity: 1 | No tags

Do

Curb ramp | severity: 5 | tag: narrow, not enough landing space, not level w/street, surface problem

12 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Do

Curb ramp | severity: 3 | tag: narrow

Do

Curb ramp | sev: 4 | tags: steep, surface problem

Do

Curb ramp | sev: 4 | tag: not enough landing space

Do

Curb ramp | severity: 3 | description: pooled water

Do

Curb ramp | severity: 1 | tags: tactile warning

Do

Curb ramp | severity: 3 | tag: not level with street

13 of 54

The curb ramp for garages should not be labelled as curb ramp because it is meant for vehicles

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

The curb ramp for driveways should not be labelled as curb ramp because it is meant for vehicles

Where shall we label curb ramps?

Not a curb ramp

Not a curb ramp

At both ends of a crosswalk that is leveled with the sidewalk. It should be labelled instead as crosswalk and tagged as ‘leveled with sidewalk’

Not a curb ramp

14 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Don’t

The crosswalk is levelled with the sidewalk

Don’t

This ramp is for vehicles not meant for people

Don’t

This ramp is for vehicles not meant for people

Don’t

It is a driveway

15 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Missing curb ramp at one of the ends of a crosswalk

Curb ramp present on one side of the of the street but missing on the other side

No curb ramp or crosswalk and no alternative route nearby

Where shall we label missing curb ramps?

Missing curb ramp

Missing curb ramp

Missing curb ramp

16 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Where shall we label missing curb ramps?

Abrupt end of sidewalks even if there is no sidewalk on the other side

Driveways running into sidewalks that cause an high difference

Missing curb ramp

Missing curb ramp

17 of 54

Tags

Missing curb ramps have 3 associated tags further explained below:

There is a visible alternative route nearby such as a driveway

Alternate route present

No alternate route present

Unclear

There is no visible alternative route or it is too far away

Unclear whether there should be a curb ramp here

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

18 of 54

Do

Missing curb ramp | sev: 4 | tag: no alternate route

Do

Missing curb ramp | sev: 3 | tag: no alternate route

Do

Missing curb ramp | sev: 2 | tag: alternate route

Do

Missing curb ramp | sev: 5 | tag: no alternate route

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Do

Missing curb ramp | sev: 4 | tag: alternate route

Do

Missing curb ramp | sev: 5 | tag: no alternate route

19 of 54

When it is not a sidewalk it doesn’t need a curb ramp

Speed bumps are not crossings so they don’t need curb ramps

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Where shall we label missing curb ramps?

Not a missing curb ramp

Not a missing curb ramp

When the curb is even slightly lowered it is considered a curb ramp and tagged ‘not level with the street‘

Not a missing curb ramp

20 of 54

Don’t

There is no crosswalk or sidewalk on both sides

Don’t

If it is not a sidewalk it doesn’t need a curb ramp

Don’t

It is a speed bump and not a crosswalk so there is no need for curb ramps

Don’t

This curb ramp creates only a slight height difference in the road

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

21 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Obstacles in Path

What is it?

Obstacles are objects that are directly on the path of a pedestrian route, thus blocking the path. In Amsterdam the minimum sidewalk width should be 90cm of clear space. Clear space means the total width of the sidewalk subtracted by the width of obstacles such as bins, traffic signs, poles, between others.

This allows pedestrians, especially those using walkers or wheelchairs, to remain safely on the sidewalk or crosswalk. Moving off the path to avoid an obstacle may be impossible or may cause imbalance, tripping, or other hazards.

22 of 54

Where shall we label obstacles?

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Any obstacle (temporary or permanent) that leaves a remaining width of less than approx. 1,2 m in the sidewalk. Take as reference that a door is normally 80cm wide.

Any construction that blocks the sidewalk leaving less than approx. 1,2 m, should be labelled in the beginning and end (if it is long)

Obstacles that leave more than approx. 1,2 m of sidewalk free or that are in motion

Obstacle in path

Obstacle in path

Not an obstacle in path

23 of 54

Tags

Obstacles in path have 14 associated tags that help identifying their typology. Below we present some examples:

fire hydrant

pole

tree

vegetation

trash/recycling can

parked car

parked bike

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

construction

sign

stairs

height difference

narrow

litter/garbage

parked scooter/ motorcycle

24 of 54

Do

Obstacle in path | severity: 5 | tag: pole

Do

Obstacle in path | severity: 5 | tag: construction

Do

Obstacle in path | severity: 5 | tag: parked car

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Do

Obstacle in path | severity: 5 | tag: stairs

Do

Obstacle in path | severity: 2 | tag: garbage

Do

Obstacle in p. | sev: 5 | tag: parked bike + garbage

25 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Don’t

It leaves more than 1,2m passage

Don’t

It leaves more than 1,2m passage

Don’t

It leaves more than 1,2m passage

Don’t

It leaves more than 1,2m passage

Don’t

It leaves more than 1,2m passage

Don’t

Car seems to be in movement

26 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Surface Problem

What is it?

A surface problem is a problem that would cause a bumpy or otherwise uncomfortable experience for someone using a wheelchair or other assistive device. If something on a surface would make it hard or impossible to cross, it should be labeled as a surface problem. For surface problems that cover a large area, you should place a surface problem label at the start of the problem, and then continue placing labels every few meters (about 1 per image) until the end of the problem.

27 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Where shall we label surface problems?

Cracks, uneven or broken pavement that would be uncomfortable for wheelchair or other assistive devices users

Narrow sidewalks, that have less than 90cm.

Gravel, sand, grass or debris that interfere with an easy use of a wheelchair or assistive device

Surface problem

Surface problem

Surface problem

28 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Where shall we label surface problems?

Rail and tram tracks that are present in sidewalks

Brick pavement, if it isn’t uneven and does not have cracks, it is not considered a surface problem in Amsterdam

Surface problem

Not a surface problem

29 of 54

Tags

Surface problems have 10 associated tags that help identifying their typology. Below we present some examples:

bumpy

cracks

grass

narrow

construction

uneven/slanted

very broken

height difference

rail/tram track

sand/gravel

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

30 of 54

Do

Surface problem | severity: 2 | tag: uneven

Do

Surface problem | sev: 3 | tag: uneven, grass

Do

Surface problem | severity: 3 | tag: tram tracks

Do

Surface problem | severity: 5 | tag: narrow

Do

Surface problem | severity: 2 | tag: grass

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Don’t

This brick work is not uneven or cracked

31 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

No Sidewalk

What is it?

A sidewalk is a paved path for walking, often alongside a road. As such, sidewalks give a separated path only for pedestrians making it safer to walk or jog.

In the absence of a sidewalk, and if there is no alternative route, pedestrians have to travel on the road risking their safety.

We consider that a no sidewalk label should be placed if there is a missing sidewalk.

32 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Where shall we label no sidewalk?

Street without any sidewalk. Place labels every other image until you reach a place where the sidewalk begins again

The same pavement is shared with cars and pedestrians

No sidewalk

Sidewalk ends abruptly

No sidewalk

No sidewalk

33 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Where shall we label no sidewalk?

Median that separates the two directions of traffic, unless the median has a sidewalk that ends abruptly

Not a no sidewalk

Add tag ‘street has a sidewalk’ if there is a sidewalk on the other side of the road

The pavement is for pedestrians only

Not a no sidewalk

No sidewalk

34 of 54

Tags

No sidewalk has 5 associated tags further explained below:

The sidewalk was expected to continue but ends abruptly

ends abruptly

street has a sidewalk

street has no sidewalks

gravel/dirt road

shared pedestrian/car space

The street has a sidewalk on the other side of the road

The street has no sidewalk at all

The street is a secondary road without any planned pavement. It is still in gravel or dirt

The entire street pavement is shared by both pedestrians, cars and other vehicles

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

35 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Do

No sidewalk | severity: 5 | tag: ends abruptly

Do

No sidewalk | sev: 5 | tag: street has a sidewalk

Do

No sidewalk | severity: 5 | tag: ends abruptly

Do

No sidewalk | severity: 2 | tag: street has a sidewalk

Do

No sidewalk | severity: 2 | tag: street has a sidewalk

Don’t

Median that separates the two directions of traffic

36 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Crosswalk

What is it?

A crosswalk is a legally defined space to cross a road, where the pedestrians have priority (providing they have taken due precautions before beginning to cross the road).

37 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Where shall we label a crosswalk?

Zebra crossing. If it has cracks or rail lines, increase severity and add the corresponding tag

Perpendicular dashed lines are an indication of crossing but pedestrians do not have priority. Add ‘no pedestrian priority’ tag

Speed bumps

Crosswalk

Crosswalk

Not a crosswalk

38 of 54

Tags

Crosswalks have 5 associated tags that help identifying their typology. Below we present some examples:

paint fading

broken surface

uneven surface

bumpy

rail/tram track

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

39 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Do

Crosswalk | sev: 1 (left), 2 (right)| tag: tram track

Do

Crosswalk | sev: 3 | tag: paint fading, broken surface

Do

Crosswalk | severity: 1 | tag: leveled with sidewalk

Do

Crosswalk | severity: 4 | tag: no priority

Don’t

It is a speed bump

Don’t

Not a crosswalk

40 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Pedestrian Signal

What is it?

A pedestrian signal allows a safer way for pedestrians to cross the street at signalized intersections. The pedestrian signal normally when green, provides enough time to the pedestrian to cross the street until it turns red again. Some of this signals are activated by a pedestrian detector push-button, that causes the controller to operate a pre-programmed time sequence.

41 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Waar moeten we een voetgangerssignaal neerzetten?

Pedestrian signal with push button or waist height push button. Don’t forget to add the label at the base and not at the button level

Pedestrian signal without push button

Push-buttons for bike riders

Pedestrian signal

Pedestrian signal

Not a pedestrian signal

42 of 54

Tags

Pedestrian signal have 2 associated tags further explained below:

The pedestrian signal has a push-button

has button

button waist height

The pedestrian signal has a button at a waist height so it is easier for a wheelchair user to reach it

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

43 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Do

Pedestrian signal | tag: button waist height

Do

Pedestrian signal | tag: has button

Do

Pedestrian signal | tag: has button

Do

Pedestrian signal | tag: no button

Do

Pedestrian signal | tag: no button

Don’t

Push-button for bike riders

44 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Can’t see the sidewalk

When does it happen?

Sometimes the view of the sidewalk is completely blocked. In these cases, you should place a "Can't See the Sidewalk" label so it can be manually inspected later on.

45 of 54

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

Where shall we label can’t see the sidewalk?

Car driving or parked in front of the sidewalk, disabling the view

Pixelated or blurry image that disables a proper classification

Stretch of sidewalk is totally blocked from view by a short wall or vegetation between the road and sidewalk

Can’t see the sidewalk

Can’t see the sidewalk

Can’t see the sidewalk

46 of 54

Do

Can’t see the Sidewalk | no severity or tags

Do

Can’t see the Sidewalk | no severity or tags

Do

Can’t see the Sidewalk | no severity or tags

Do

Can’t see the Sidewalk | no severity or tags

Do

Can’t see the Sidewalk | no severity or tags

Do

Can’t see the Sidewalk | no severity or tags

Curb Ramps

Missing Curb Ramps

Obstacle in Path

Surface Problem

No Sidewalk

Crosswalk

Pedestrian Signal

Can’t see the Sidewalk

47 of 54

Validating data

HOW TO DO

48 of 54

Validating Data

Why do we do it?

Data validation allows us to filter wrongly labelled data and thus, increase quality.

In this section you will be shown images labelled by other participants and you will have to decide if you agree, disagree or if you are not sure.

Also, each participant with an account can see their level of accuracy. This is given by the amount of labels that were validated as correct or incorrect by others.

49 of 54

50 of 54

Visualizing data

HOW TO CHECK RESULTS

51 of 54

Visualizing data

Why do we do it?

We can visualize data in the following different ways:

1. Results map - offers a overview map where it is possible to check how far complete are the neighbourhoods and the amount of labels for each category;

2. Label map - offers a overview map of the city, but instead of neighbourhoods it pinpoints all labels by category;

3. Gallery - offers an image gallery where it is possible to filter by city, category, severity level and tag;

4. Leaderboard - Shows the top users, the amount of missions, tags, and distance complete. It also shares the accuracy level (e.g. if you label 500 labels and 250 were validated ‘disagree’, you have 50% accuracy.

1.

2.

3.

4.

52 of 54

53 of 54

Let’s share our ideas!

More info at:

sidewalk-amsterdam.cs.washington.edu

amsterdamintelligence.com

54 of 54