PRIVACY AND THE CONTROL OF CITIZENS

Jean Tirole, Kyiv School of Economics, March 30, 2022

2

I. RISE OF THE SURVEILLANCE SOCIETY

​

• Technological evolution (smart phones, AI, facial recognition, car sensors, social networks, digital CB money, smart cities…) =>
• marginal cost of learning a person’s preferences and social graph has gone down to 0

🡪 expansion of public sphere (democracies), surveillance society (autocracies)

• Various societal developments, including
• Chinese Social Credit System (and milder forms elsewhere)
• digital subversion of democracies and the fostering of polarization.

​

​

3

Huxley vs Orwell on the control of citizens

​

• 1984: violence/coercion

​

• When 1984 was published (1949), Huxley wrote to Orwell:

“Whether in actual fact the policy of the boot-on-the-face can go on indefinitely seems doubtful. My own belief is that the ruling oligarchy will find less arduous and wasteful ways of governing and of satisfying its lust for power, and these ways will resemble those which I described in Brave New World.”

4

PRIVACY IS NOT AN INDIVIDUAL CHOICE (even in democratic society)

(a) Data externalities: others disclose information about us anyway. Social norm?

(b) Users’

• rationality: high transaction costs in granting informed consent or staying away from data collection and storage
• bounded rationality: compounded by impulsiveness

(c) Unraveling

• “Nothing to hide”, as when prospective employer states “you legally do not have to give me access to your Facebook private page”
• From Posner (1978)’s “At some point nondisclosure becomes fraud” to

Dystopia in Science fiction (The Circle, Black Mirror) and not really Sci-Fi (GDR’s gläserne Mensch)

5

II. CONSENSUAL ISSUES: THE CASE FOR TRANSPARENCY

Consensual issue: pollution, crime….

• Transparency creates accountability
• workers (career concerns)
• suppliers: through ratings (platforms; financial, ESG ratings agencies…), reviews & rankings of schools, restaurants…
• politicians and countries (Transparency International, war coverage)
• Large lab & field experimental literature showing that people behave more prosocially when observed by others: charitable contributions, public good provision, voting, health care behavior, blood donation…

​

6

Two caveats to the benefits of transparency for consensual behaviors

​

• Public and private spheres
• Decent behavior in public sphere offers moral licensing for worse behavior in private sphere
• Moral licensing implies that an expansion of the public sphere may reduce overall prosociality despite composition effect
• Theoretical prediction and empirical evidence [Hong et al]

​

• Right to oblivion
• Data must be erased at some point to allow for a second chance.

​

7

III. DIVISIVE ISSUES

• Sexual orientation
• Politics
• Ethnicity
• Religion, secularism
• Vegans vs. meat-eaters, abortion, vaccines, social roles

​

Ostracism/discrimination/hatred fueling/violence (deterioration of social fabric)…

…by employers/coworkers, anonymous hatemongers, blackmailers, indelicate governments

​

Neutral agents, who don’t take sides, are viewed suspiciously by both sides (amalgam effect), are asked to take sides.

​

​

8

(1) Changing one’s behavior altogether

• Chilling effect of public spaces: people don’t dare to speak their mind if don’t want to trigger antagonistic reactions [also: to check into drug rehab center or share info with physician if no assurance of privacy]
• Obedience to majoritarian views on religion, sexuality or soft drugs, conformity with stereotypes and social roles…

​

(2) Maintaining the same behavior, but hiding in safe spaces to generate less hostility

• Safe space: physical (home, private club, church, masonic lodge, bullfight ring, political party, friends, diary…) or virtual (Facebook group)

​

9

• Retreat in a safe space comes with private costs
• reduced use of public space (reproved sexual minorities cannot enjoy the public space together; drug users or aborting women resort to costly and untrustworthy black market providers)
• forgoing desirable relationships and diversity of social graph

​

• Collateral social costs of safe spaces: once in a safe space, one-upmanship/ holier than thou attitude (voluntary or enforced by threat of outing/exclusion)
• → one-sided narratives, hate speech, conspiracy theories
• Facebook groups

10

IV SOCIAL SCORES

​

CHINA’S SOCIAL CREDIT SYSTEM

​

Aggregates according to a variety of criteria (not cast in stone)

• Some reasonable: credit history, tax compliance, environmentally-friendly behavior, traffic violations
• Some manipulable: spreading of fake news (definition?)
• Some clearly unappealing ones: social graph, personal traits, political or religious opinions…

What for?

• Public stigmatization/modern pillory (trend in the US)?
• Restrictions of access to employment, transportation, visas abroad, access (of individual or children) to the best schools or universities?

11

Key mixture of consensual and divisive behavior

​

If focused on consensual behaviors, social score boosts image concerns and thereby pro-social behavior

• The agent’s reputation is extended to future new partners
• Even stable partners get a clearer assessment of the agent’s social attitude, if imperfect observation in their own interaction with her

​

State (but also platforms, religious organizations…) can exploit agents' interest in each other's score to promote political compliance, in two ways

• Bundling prosocial behavior with political compliance (dissent/accommodate) into a single score
• Using guilt by association to further hit dissenters

12

Lost cost approach: Huxley more than Orwell

​

Main insights

​

1) The state bundles the two dimensions (consensual and divisive) if and only if it is sufficiently autocratic

2) Its ability to affect behavior through bundling is much higher in a society of strangers (of transient relationships) than in a tight knit society of stable relationships

3) Transparency/ citizens’ awareness about the way the social score is computed is essential (but opaqueness about its inputs)

4) To be effective, the state must suppress competition from independent, privately-provided social ratings. Private platforms would expunge information about political views from their ratings, leading to de facto unbundling

​

13

ENLISTING THE CITIZEN’S SOCIAL GRAPH

​

​

Guilt by association (coloring of a person's perception by the company she keeps): relationship with someone on a blacklist taints the reputation of someone who a priori would not be on the list.

​

Insight: Embodying an individual’s social graph into her social score

• generates costs, most prominently the destruction of the social fabric as citizens cut beneficial ties with others (no-one will want to be seen or communicate with a low-rating person)
• appeals to autocratic regimes as it strengthens the state’s hold on society.

​

​

14

• Ancient strategy
• with very rudimentary means, the Stasi managed to break the social fabric: friends, family, even spouses and children were no longer part of circle of trust.
• religious orders (excommunication - “a low score”- combined with guilt by association has been a major disciplining device)
• But face recognition and artificial intelligence applied to surveilled communications and social network activities have substantially reduced the state's (or private platforms’) cost of drawing an accurate social graph of relationships among its citizens. Today some servers and artificial intelligence suffice to accomplish these tasks.

​

15

V Countering techno-authoritarian power

1. First bulwark: independent Supreme Court (fragile though, as shown by recent experience)
2. Do not leave legacy data around that an autocratic government will seize when comes to power.
3. International agreements? [in the spirit of international humanitarian law- Geneva convention- which maintains some humanity in armed conflicts. Hard to enforce]
4. Protest movements? Are Budapest (1956) or Tiananmen Square (1989) still conceivable in 2022? Need access to encrypted private networks [Protests work through social structures. Bursztyn et al 2021: strategic complementarities in protest within social networks.]
5. Last resort: revolt within party, ruling elite, military…

​

THANK YOU FOR YOUR ATTENTION

17

IA ACT

Quick overview: IAA’s three buckets

• Unacceptable risk: AI systems considered a clear threat to the safety, livelihoods and rights of people will be banned
• circumvent users' free will
• social scoring
• High risk: clear obligations (traceability, data quality, appropriate human oversight…)
• Access-to-“essential services” issues: scoring of exams, CV sorting software, justice, credit scoring for loans, verification of authenticity of travel documents, biometric identification, applications for social security benefits….
• Safety: robot-assisted systems, autonomous vehicles… New machinery regulation.
• Limited and minimal risk