Pseudonymization of

learner corpora

Elena Volodina

Språkbanken Text, University of Gothenburg, Sweden

Why learner data?

methodological

issues

NLP/ICALL

ICALL

theory

building

Why personal data?

NLP/ICALL

ICALL

theory

building

methodological

issues

Minimizing risks

NLP/ICALL

ICALL

methodological

issues

theory

building

Outline

• Challenges of learner data sharing in the age of GDPR

• Pseudonymization of the data - status and prospects (+demo)

• Visions & agenda

• Introduction to SweLL project

Introduction to the SweLL project

SweLL

SweLL – �

research infrastructure for

Swedish as a Second Language

Swedish Learner Language

https://spraakbanken.gu.se/en/projects/swell

(2017-2020)

SweLL promises (main)

1. Deliver a well-annotated (gold standard) corpus of L2 essays
• 600 essays at different levels of proficiency
• Incl manual correction annotation (& manually checked linguistic annotation)
• Make available for research (and public?)

→

SweLL promises (main)

2. Set a platform (and workflow) for
• Continuous collection of new essays
• Manual correction annotation
• Automatic linguistic annotation

​

→

→

SweLL promises (main)

• Set a platform for browsing L2 essays
• in concordance fashion (+parallel view)
• In full text fashion

https://spraakbanken.gu.se/en/projects/swell/l2korp

BIG GOAL

• … to empower second language learners / immigrants / teachers

​

• … to support Second Language Acquisition (SLA) research

​

• … to establish the field of “Computational SLA” (in Sweden) and make it attractive (for researchers, funders, PhD students, etc)

​

• … to make a change!

and IT ALL STARTS WITH...

Challenges of learner data collection and availability

challenges

NLP/ICALL

ICALL

theory

building

methodological

issues

Risks of sharing personal data

• Personal data is the highest commodity of the 21st century

​

• We need to protect those who share their personal data for research (data subjects) -- out of ethics

​

• We are obliged to protect our subjects -- by law

General Data Protection Regulation (GDPR)

• Restricts use of digital data containing personal information

​

• Hands back ownership of personal data to data subjects

​

• Imposes data protection by design and by default (Art 25)

​

​

Personal data is...

...any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an on-line identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person.

GDPR, Article 4

​

​

GDPR

• Anonymous data is outside the scope of GDPR (Recital 26)

​

• No data is truly anonymous (e.g. Rocher et al. 2019)

​

• Pseudonymization is recognized as a way to reduce risks of re-identification of data subjects (Recital 28)

​

• What are we supposed to pseudonymize?

​

​

​

​

​

Anonymization vs pseudonymization

https://www.linkedin.com/pulse/anonymization-does-work-big-data-due-lack-protection-direct-lafever

Sweeney (2000)

http://ggs685.pbworks.com/w/file/fetch/94376315/Latanya.pdf

87% of US population could be identified uniquely

→

A lá Sweeney (2000) ?

http://ggs685.pbworks.com/w/file/fetch/94376315/Latanya.pdf

​

​

PII-1

PII-2

...

→

Can we

identify

a person?

Text

information

+

Personal

metadata

+

Task metadata

​

​

Public statistics/

records

+

Research data

+

Accidental

context

​

​

Learner corpus

Other sources

PII → Personally Identifiable

Information

→

Pseudonymization

pseudo-

nymization

Data protection (Danezis et al. 2014)

https://www.enisa.europa.eu/publications/privacy-and-data-protection-by-desig

• Encryption

• Data minimization

• Authenti-

fication

• Anonymization

• Pseudonymization

• Privacy Preserving Data Mining

• De-identification

• ...

Pseudonymization

​

• detection & neutralization of personally identifiable information (PII)

​

• predominantly manually handled

​

• automatic approaches are NER-based or rule-based

​

• most studies within medical domain

Pseudonymization of learner corpora

No proper guidelines on pseudonymization

Lack of systematic studies

Stemle et al (2019)

https://bia.unibz.it/bitstream/handle/10863/13346/stemle_etal-postprint.pdf?sequence=2&isAllowed=y

(Section 2.2)

Stemle et al (2019)

• CzeSL (Rosen 2017):

all names → Adam, Eva or Sin (+ keeping suffixes)

geo → village<priv>

• CroLTec (Preradovic et al. 2015):

hard-coded; risky passages deleted

• ASK (Tenfjord et al. 2006):

@name, @place, @something

• … etc.

https://bia.unibz.it/bitstream/handle/10863/13346/stemle_etal-postprint.pdf?sequence=2&isAllowed=y

(Section 2.2)

Megyesi et al. (2018)

SweLL approach (principles for manual pseudonymization)

http://www.ep.liu.se/ecp/152/006/ecp18152006.pdf

Principles (domain-specific, of course)

Clear mark-up of pseudonymized segments

15 head categories, 40 subcategories, morph. markers

• names, geo names, institutions, transportation, age, date, miscellaneous, mark/sensitive

​

​

​

Guidelines: https://spraakbanken.github.io/swell-project/Anonymization_guidelines

Categories (domain-specific, of course)

SVALA - SweLL annotation tool

• Parallel text
• Visualized diff
• Semi-automatic word alignment
• Annotation on source–target links

​

​

Dan Rosén,

research engineer

Arild Matsson,

research engineer

Samir Ali Mohammed,

systems developer

Volodina et al. (2020)

SweLL automatic pseudonymizer service in SVALA

Why automatize?

​

• To speed up annotation work
• To boost essay collection
• GDPR
• Ethical reasons
• Ultimate goal - online collection of essays

​

​

​

​

​

​

SVALA - pseudonymizer: facts

• Rule-based vs Rules+POS
• regular expressions
• minor spelling correction (Levenstein distance)
• Python
• IN: text; OUT: json / text

​

• Three steps:
• detection
• placeholder labelling
• pseudonymization

​

Git Repo: https://github.com/SamirYousuf/LR_project

​

Pseudonymizer demo

DEMO

SVALA pseudonymizer – hands-on demo

https://spraakbanken.gu.se/swell/dev/

​

​

Example essay (translation into English + mocking errors)

I live in Stockholm on apartement . Jag är 29 år gammal . I live with my boyfriend . His name is Cezary . Apartement mine has a pattio and tree room . I enjoy there in Stockhulm but a lot of time to goto shop , fortifive minut . I have the buss and the Stockholm train . I lived in Danmark bifore , in Odense . It was less than Stockholm . I enjoy their too becaus I had more friends . I think it is hard to have friends here . But I enjoy better job here . In Odense jobbe I only on one website . In Stockholm I work on many website . I am webdevelooper . But Stockholm is closser to Luxembourg than Odense . It is important how one lives because I am not in my country . I mess my mother and my father but I live her with my boyfriend .

https://tinyurl.com/y3m8uqjs (Slides 38 + 39)

A link to this presentation

https://tinyurl.com/y3m8uqjs

How it works?

KNOW-HOW

I live in Stockholm on apartement . I am 29 years old . I live with my boyfriend . His name is Cezary .

I live in Stockholm on apartement . Jag är 29 år gammal . I live with my boyfriend . His name is Cezary .

I live in @city on apartement . Jag är @age år gammal . I live with my boyfriend . His name is @name .

I live in Gothenburg on apartement . Jag är 31 år gammal . I live with my boyfriend . His name is Johan .

ORIGINAL

DETECTION

LABELING

PSEUDOMNYMIZATION

Data for evaluation

To

​

​

​

​

​

​

TOTAL: 285 essays, ≈55.000 tokens, several levels / genres / topics

Results

Accuracy

• 89% average, but
• low accuracy for surname, place, date_digits

​

​

Depends on

• lack of capitalization
• misspellings
• ambiguity (can belong to several domains)

Topic/genre specifics

Evaluative texts / Investigative texts:

• film/book reviews
• response to an article

​

• Switch off pseudonymization?
• “Listen” to personal pronouns, e.g. In Vietnam, we... ?

Tag statistics

• Most used → city, country, firstname, year

​

• Most confused → city-country, place-city/country

​

Detection - (non-)capitalizing

Detection - misspellings

Detection - heavily unstructured PIIs

Detection (sensitive)

Detection - heavily unstructured PIIs

Labeling - ambiguity

Pseudonymizing -

linguistic constraints

Pseudonymizing - grammar constraints

I had vacation on Cuba → I had vacation on Portugal

Pseudonymizing - projecting errors?

Stockhulm --> Gothinburg ?

Pseudonymizing - projecting grammar features?

I ate at Frank’s house → I ate at Harry house

(Think Swedish: at Lars house → At Sven house)

How to evaluate the pseudonymization step?

• Should it read as an original ?
• Should it preserve learner errors ?
• Should it avoid adding new errors ?

​

Visions and agenda

Visions

Agenda

Agenda

• Can we agree on the categories in the community?

​

• Can we agree on the procedure -- detection - labeling - pseudonymization -- and convertibility of formats?

​

• Shared tasks -- detection, labeling, pseudonymization, readability, sensitive markup, de-identification, risk assessment, etc.

​

• How to evaluate pseudonymization step?

​

Data for shared tasks

• How much do we need?
• size of the data sets

​

• How to ensure access?
• reduce personal metadata?
• only task metadata?

​

• How to “manipulate” the original data so that we (legally) can use it for shared tasks?
• keep original strings, but randomly insert them into different essays?

Datasets (e.g. Medlock 2006)

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.380.8037&rep=rep1&type=pdf

​

Visions: Pseudonymization on the fly

Imagine scenario:

Write essays online

Automatic pseudo

Correction of pseudo- suggestions

Automatic analysis of an essay

Corr-reports to improve

pseudo-algorithm

Online consent

Online metadata

Upload essay to a database

Informed consent

Personal metadata

Account + ID

Essay

Automatic pseudo

Manual correction

Approval of pseudo

Essay analysis Add to a corpus

Informed consent

Personal metadata

Account + ID

Essay

Essay analysis Add to a corpus

Automatic pseudo

Manual correction

Approval of pseudo

Reports with

• Orig string:
• Auto-label

(if any)

• Corrected label

(if any)

• We may want to look into the full context, too, though

Thank you!

Link to the presentation:

https://tinyurl.com/y3m8uqjs