Enforcing Demographic Coherence: �A Harms-Aware Framework for Reasoning about Private Data Release

*IBM **University of Maryland

Data’s role in our society

2

Data’s role in our society

3

Data’s role in our society

4

Data’s role in our society

5

Complementary approaches to privacy

6

ATTACKS

Complementary approaches to privacy

7

ATTACKS

“I could link those data sets and put [governor Weld’s] name to his health record uniquely. And that was a pretty eye-popping experience.”�Latanya Sweeney

Complementary approaches to privacy

8

ATTACKS

“With the data-anonymization approach the Census Bureau used in 2010, we were able to identify 605 trans kids.”�Abraham D. Flaxman, Os Keys

“I could link those data sets and put [governor Weld’s] name to his health record uniquely. And that was a pretty eye-popping experience.”�Latanya Sweeney

Complementary approaches to privacy

9

ATTACKS

“With the data-anonymization approach the Census Bureau used in 2010, we were able to identify 605 trans kids.”�Abraham D. Flaxman, Os Keys

“I could link those data sets and put [governor Weld’s] name to his health record uniquely. And that was a pretty eye-popping experience.”�Latanya Sweeney

Provide intuition and motivation..

​

Complementary approaches to privacy

10

ATTACKS

“With the data-anonymization approach the Census Bureau used in 2010, we were able to identify 605 trans kids.”�Abraham D. Flaxman, Os Keys

“I could link those data sets and put [governor Weld’s] name to his health record uniquely. And that was a pretty eye-popping experience.”�Latanya Sweeney

Provide intuition and motivation..

​

do not provide a direct path towards designing data protection mechanisms.

Complementary approaches to privacy

11

ATTACKS

“With the data-anonymization approach the Census Bureau used in 2010, we were able to identify 605 trans kids.”�Abraham D. Flaxman, Os Keys

k-anonymity: a model for protecting privacy

Latanya Sweeney�

“I could link those data sets and put [governor Weld’s] name to his health record uniquely. And that was a pretty eye-popping experience.”�Latanya Sweeney

Provide intuition and motivation..

​

do not provide a direct path towards designing data protection mechanisms.

Complementary approaches to privacy

12

ATTACKS

“With the data-anonymization approach the Census Bureau used in 2010, we were able to identify 605 trans kids.”�Abraham D. Flaxman, Os Keys

k-anonymity: a model for protecting privacy

Latanya Sweeney�

“I could link those data sets and put [governor Weld’s] name to his health record uniquely. And that was a pretty eye-popping experience.”�Latanya Sweeney

L-diversity: privacy beyond k-anonymity

A. Machanavajjhala et. al.

Provide intuition and motivation..

​

do not provide a direct path towards designing data protection mechanisms.

Complementary approaches to privacy

13

ATTACKS

“With the data-anonymization approach the Census Bureau used in 2010, we were able to identify 605 trans kids.”�Abraham D. Flaxman, Os Keys

k-anonymity: a model for protecting privacy

Latanya Sweeney�

“I could link those data sets and put [governor Weld’s] name to his health record uniquely. And that was a pretty eye-popping experience.”�Latanya Sweeney

L-diversity: privacy beyond k-anonymity

A. Machanavajjhala et. al.

Calibrating Noise to Sensitivity in Private Data Analysis

Cynthia Dwork et. al.

Provide intuition and motivation..

​

do not provide a direct path towards designing data protection mechanisms.

differential privacy

Complementary approaches to privacy

14

ATTACKS

“With the data-anonymization approach the Census Bureau used in 2010, we were able to identify 605 trans kids.”�Abraham D. Flaxman, Os Keys

k-anonymity: a model for protecting privacy

Latanya Sweeney�

“I could link those data sets and put [governor Weld’s] name to his health record uniquely. And that was a pretty eye-popping experience.”�Latanya Sweeney

L-diversity: privacy beyond k-anonymity

A. Machanavajjhala et. al.

t-Closeness: Privacy Beyond k-Anonymity and l-Diversity�Ninghui Li et. al.

Calibrating Noise to Sensitivity in Private Data Analysis

Cynthia Dwork et. al.

Provide intuition and motivation..

​

do not provide a direct path towards designing data protection mechanisms.

robust guarantees

composable

Complementary approaches to privacy

15

ATTACKS

“With the data-anonymization approach the Census Bureau used in 2010, we were able to identify 605 trans kids.”�Abraham D. Flaxman, Os Keys

k-anonymity: a model for protecting privacy

Latanya Sweeney�

“I could link those data sets and put [governor Weld’s] name to his health record uniquely. And that was a pretty eye-popping experience.”�Latanya Sweeney

L-diversity: privacy beyond k-anonymity

A. Machanavajjhala et. al.

t-Closeness: Privacy Beyond k-Anonymity and l-Diversity�Ninghui Li et. al.

Calibrating Noise to Sensitivity in Private Data Analysis

Cynthia Dwork et. al.

Provide intuition and motivation..

​

do not provide a direct path towards designing data protection mechanisms.

“𝞮 is hard to understand”�

“too abstract”

“not intuitive”

robust guarantees

composable

Complementary approaches to privacy

Demographic Coherence Enforcement: a necessary condition for privacy which, by being more concrete, can enable sociotechnical conversations that bridge protocols, attack demonstrations, and formal guarantees.

16

ATTACKS

“With the data-anonymization approach the Census Bureau used in 2010, we were able to identify 605 trans kids.”�Abraham D. Flaxman, Os Keys

k-anonymity: a model for protecting privacy

Latanya Sweeney�

“I could link those data sets and put [governor Weld’s] name to his health record uniquely. And that was a pretty eye-popping experience.”�Latanya Sweeney

L-diversity: privacy beyond k-anonymity

A. Machanavajjhala et. al.

t-Closeness: Privacy Beyond k-Anonymity and l-Diversity�Ninghui Li et. al.

Calibrating Noise to Sensitivity in Private Data Analysis

Cynthia Dwork et. al.

Provide intuition and motivation..

​

do not provide a direct path towards designing data protection mechanisms.

“𝞮 is hard to understand”�

“too abstract”

“not intuitive”

robust guarantees

composable

A harms-aware framework for reasoning about the privacy impact of data release

17

1. Intuitively captures privacy concerns�concretely models privacy loss via its effect on predictive harms
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

A harms-aware framework for reasoning about the privacy impact of data release

18

1. Intuitively captures privacy concerns�concretely models privacy loss via its effect on predictive harms
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

A harms-aware framework for reasoning about the privacy impact of data release

19

1. Intuitively captures privacy concerns�concretely models privacy loss via its effect on predictive harms
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

A harms-aware framework for reasoning about the privacy impact of data release

20

1. Intuitively captures privacy concerns�concretely models privacy loss via its effect on predictive harms
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

A harms-aware framework for reasoning about the privacy impact of data release

21

1. Intuitively captures privacy concerns�concretely models privacy loss via its effect on predictive harms
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

Intuition behind our framework

22

Intuition behind our framework

23

Intuition behind our framework

24

Intuition behind our framework

25

A harms aware approach is crucial for intuition

26

A harms aware approach is crucial for intuition

27

A harms aware approach is crucial for intuition

28

A harms aware approach is crucial for intuition

29

A harms aware approach is crucial for intuition

30

A harms aware approach is crucial for intuition

31

A harms aware approach is crucial for intuition

32

A harms aware approach is crucial for intuition

33

The value of privacy is tied intrinsically to the harms the data could cause!

A harms aware approach is crucial for intuition

34

The value of privacy is tied intrinsically to the harms the data could cause!

Adversarial Model

35

The value of privacy is tied intrinsically to the harms the data could cause!

Adversarial Model

36

The value of privacy is tied intrinsically to the harms the data could cause!

What is a ‘bad event’?

37

Predictive models capture harms that could occur when individual privacy is violated.

The value of privacy is tied intrinsically to the harms the data could cause!

What is a ‘bad event’?

38

Predictive models capture harms that could occur when individual privacy is violated.

The value of privacy is tied intrinsically to the harms the data could cause!

What is a ‘bad event’?

39

What is a ‘bad event’?

40

prediction on

Asahi

prediction on

Blair�(not in data)

What is a ‘bad event’?

41

not sure, maybe not?

prediction on

Asahi

prediction on

Blair�(not in data)

What is a ‘bad event’?

42

prediction on

Asahi

prediction on

Blair�(not in data)

What is a ‘bad event’?

43

prediction on

Asahi

prediction on

Blair�(not in data)

What is a ‘bad event’?

44

prediction on

Asahi

prediction on

Blair�(not in data)

What is a ‘bad event’?

45

prediction on

Asahi

prediction on

Blair�(not in data)

measuring confidence is critical!

What is a ‘bad event’?

46

prediction on

Asahi

prediction on

Blair�(not in data)

the harms in experiment 2 do not depend on the accuracy of the prediction

measuring confidence is critical!

A harms-aware framework for reasoning about the privacy impact of data release

47

1. Intuitively captures privacy concerns�concretely models privacy loss via its effect on predictive harms
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

A harms-aware framework for reasoning about the privacy impact of data release

48

1. Intuitively captures privacy concerns�- takes into account the entire data pipeline����
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

A harms-aware framework for reasoning about the privacy impact of data release

49

1. Intuitively captures privacy concerns�- takes into account the entire data pipeline�- ties to predictive harms���
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

A harms-aware framework for reasoning about the privacy impact of data release

50

1. Intuitively captures privacy concerns�- takes into account the entire data pipeline�- ties to predictive harms�- has a concrete adversarial model��
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

A harms-aware framework for reasoning about the privacy impact of data release

51

1. Intuitively captures privacy concerns�- takes into account the entire data pipeline�- ties to predictive harms�- has a concrete adversarial model�- evaluates risk without relying on ground truth�
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

A harms-aware framework for reasoning about the privacy impact of data release

52

1. Intuitively captures privacy concerns�- takes into account the entire data pipeline�- ties to predictive harms�- has a concrete adversarial model�- evaluates risk without relying on ground truth�-allows identification of effects local to specific vulnerable groups
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

Formal Definition

53

prediction on Asahi

prediction on Blair

Formal Definition

54

prediction on Asahi

prediction on Blair

Formal Definition

55

prediction on Asahi

prediction on Blair

Formal Definition

56

prediction on Asahi

prediction on Blair

Incoherent Predictions

57

prediction on Asahi

prediction on Blair

Incoherent Predictions

58

Definition:

prediction on Asahi

prediction on Blair

Incoherent Predictions

59

Definition:

not sure, maybe not?

prediction on Asahi

prediction on Blair

Incoherent Predictions

60

Definition:

not sure, maybe not?

prediction on Asahi

prediction on Blair

Incoherent Predictions

61

Definition:

not sure, maybe not?

not sure, maybe?

prediction on Asahi

prediction on Blair

Incoherent Predictions

62

Definition:

not sure, maybe not?

not sure, maybe?

almost definitely

prediction on Asahi

prediction on Blair

Incoherent Predictions

63

Definition:

prediction on Asahi

prediction on Blair

Incoherent Predictions

64

prediction on Asahi

prediction on Blair

Enforcing Demographic Coherence

65

prediction on Asahi

prediction on Blair

Enforcing Demographic Coherence

66

Enforcing Demographic Coherence

67

Enforcing Demographic Coherence

68

Enforcing Demographic Coherence

69

Enforcing Demographic Coherence

70

Enforcing Demographic Coherence

71

Enforcing Demographic Coherence

72

Enforcing Demographic Coherence

73

Enforcing Demographic Coherence

74

Enforcing Demographic Coherence

75

A harms-aware framework for reasoning about the privacy impact of data release

76

1. Intuitively captures privacy concerns�- takes into account the entire data pipeline�- ties to predictive harms�- has a concrete adversarial model�- evaluates risk without relying on ground truth�-allows identification of effects local to specific vulnerable groups
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

Enforcing Demographic Coherence

77

1) Intuitively captures privacy concerns:

Enforcing Demographic Coherence

78

1) Intuitively captures privacy concerns:

• takes into account the entire data pipeline

Enforcing Demographic Coherence

79

1) Intuitively captures privacy concerns:

• takes into account the entire data pipeline
• ties to predictive harms

​

Enforcing Demographic Coherence

80

1) Intuitively captures privacy concerns:

• takes into account the entire data pipeline
• ties to predictive harms
• has a concrete adversarial model

​

​

Enforcing Demographic Coherence

81

1) Intuitively captures privacy concerns:

• takes into account the entire data pipeline
• ties to predictive harms
• has a concrete adversarial model
• evaluates risk without relying on ground truth

​

​

Enforcing Demographic Coherence

82

1) Intuitively captures privacy concerns:

• takes into account the entire data pipeline
• ties to predictive harms
• has a concrete adversarial model
• evaluates risk without relying on ground truth
• allows identification of effects local to specific vulnerable subgroups

​

​

Enforcing Demographic Coherence

83

1) Intuitively captures privacy concerns:

• takes into account the entire data pipeline
• ties to predictive harms
• has a concrete adversarial model
• evaluates risk without relying on ground truth
• allows identification of effects local to specific vulnerable subgroups

2) Lends itself to experimental auditing:

• has a natural translation to an experimental setup for concretely comparing PETs

​

​

Related Work

84

Necessary Conditions

​

Designing definitions to protect against specific attacks:

• Reconstruction attacks �[Balle et al. 22, Cummings et al.‘24] �
• Reconstruction attacks, membership inference attacks, singling out attacks etc. �[Cohen et al. ‘25]

​

Privacy Auditing

​

Privacy Auditing: Measure privacy of a system through the efficacy of attacks on the system [Jagielski et al’ 20, Jayaraman et al. ’19, Steinke et al. ‘23 ]

​

Systematize Attacks: Works that classify attacks on practical systems and help determine their realistic threat �[Cohen ‘20, Giomi et al. ‘22, Salem et al. ‘23, �Rigaki and Garcia ’24, Cummings et al.‘24]

A harms-aware framework for reasoning about the privacy impact of data release

85

1. Intuitively captures privacy concerns�- takes into account the entire data pipeline�- ties to predictive harms�- has a concrete adversarial model�- evaluates risk without relying on ground truth�-allows identification of effects local to specific vulnerable groups
2. Lends itself to experimental auditing�natural translation to an experimental setup for comparing PETs
3. Supports rigorous analytical arguments�all differentially private algorithms enforce demographic coherence*
4. Is achievable**�**black box reduction from DP + toy algorithm imply existence of better algorithms

​

86

Theorem:

87

Theorem:

88

Theorem:

89

.

.

.

.

.

.

Theorem:

90

.

.

.

.

.

.

demographic coherence enforcement surfaces benefits of data minimization

Theorem:

Future Work

91

Future Work

92

1) Algorithms

Future Work

93