Data extraction from piano-roll scans

Craig Stuart Sapp

craig@ccrma.stanford.edu

​

IAML 2018

Leipzig, Deutschland

24 Juli 2018

http://bit.ly/iaml2018-pianoroll

Data processing overview

Scan

Raw MIDI file

without

expression

Performance MIDI file

with

expression

Stanford Libraries

Digital Production

Group (DPG)

Me

Zhengshan “Kitty” Shi

Online test scans

http://pianoroll.sapp.org/full-rolls-set3b

Image analysis markup

Graphical hole markup syntax

• Yellow horizontal line: start of note

​

• Orange vertical line: bass side of note

​

• Red vertical line: treble side of note

​

• Red horizontal line: ending of a note hole

​

• Green vertical line: treble tracker-bar position

​

• Yellow horizontal dotted lines: note attacks

(or expression hole starts)

​

Tracker bar hole positions

Bass

expression

Treble

expression

Treble notes

Bass notes

1

2

3

4

5

6

7

8

9

10

1 Mezzo-forte off

2 Mezzo-forte on

3 Crescendo off

4 Crescendo on

5 Forzando off

6 Forzando on

7 Sustain off

8 Sustain on

9 Electric cutoff

10. Rewind
11. Highest note (G7)

11

Left-right drift of roll while scanning

~35 pixel spacing between hole rows

Scanning operator manually adjusted

Left/right position of roll while scanning

Operator shift marked in scan

Dynamic tracking for drift/shift

• Edges of paper are tracked to monitor shifts and drifting of the paper.

​

• Light blue color of holes mean they are in a region which is shifting left/right.

​

• Coordinate system is adjusted to correct for shift.

​

• Horizontal orange lines mark where the operator shifted the roll while scanning (more than 1/100^th of an inch over 2/3rds of an inch).

​

• Green/Blue vertical lines are the intended center of the tracker-bar holes (notice horizontal movment to match hole centers).

​

• Dashed lines mean that no holes in paper for that tracker-bar position.

Automatic tear analysis

https://ccrma.stanford.edu/~craig/piano-roll-project/acceptance_test-20171219/redwelte-0245-waltz-op42-chopin-scharwenka/druid-mj502fy3866-analysis/index.html#trebletear004

Automatic identification of bad holes

Bad holes (2)

• Bad holes not ignored when inside a note sustain.

• Bad holes not allowed to start a note.

Analysis summary

https://docs.google.com/spreadsheets/d/16s-SUqWki41trQW_0UvyRTyfPO3Yz5dyzRabzM-CsG0/edit?usp=sharing

Github

https://github.com/pianoroll

Software and extracted MIDI data for Stanford Piano rolls

https://github.com/pianoroll/midiroll/tree/master/data/stanford/scandata

Example “raw” MIDI Files:

Resources for each roll scan

Expressionless rendering

(audio file)

Starting image

Detailed analysis page

Click roll icon to

view information

http://pianoroll.sapp.org/full-rolls-set3b/#jg699wz4435

Automatic analysis pages

https://ccrma.stanford.edu/~craig/piano-roll-project/redwelte-20180301/jg699wz4435/jg699wz4435-analysis

Automatically extracted roll properties

https://ccrma.stanford.edu/~craig/piano-roll-project/redwelte-20180301/jg699wz4435/jg699wz4435-analysis