Types of Attack Surfaces�From Safety & Security Perspectives

Alex Leavitt (Roblox; UC Berkeley)

Kevin Fumai (Oracle)

Learning Objectives

Safety

• Identify system entry points for abuse
• Understand how and why systems are susceptible to abuse
• Learn how to limit abuse

Security

• Identify threat actors and cyber and AI attack types
• Understand the anatomy of cyber attacks
• Learn cyber best practices

​

Attack Surfaces�From a Safety Perspective

Definitions

• Surface: a technical “entry point” to interact with a technological system
• For each technology on the next page:
• What surfaces do they have in common? Which surfaces are unique?
• How do users benefit from the surfaces from functionality or content perspectives?
• How could these surfaces be manipulated, abused, or attacked?

Examples of Surfaces

Definitions (ctd.)

Features → Affordances → Outcomes

• Features: design elements that offer specific types of capabilities
• Affordances: possibilities for action available in a given environment
• Outcomes: actions or other behaviors connected with user goals

​

When we think about entry points for abuse in a system:

• What design elements allow users to create or consume content?
• What do these design elements let people do?
• What do people do with them, and what are the results?

Feature Analysis

Looking at the systems design of a platform, what interactive elements are present?

Entry Points for Abuse

Entry Points for Abuse (ctd.)

Each one of these entry points can have significant layers of detail for potential abuse. For example:

​

Video Editing

​

• Misinformation/disinformation
• Terrorist content/recruitment
• Deep fakes/AI-generated content

• Copyright infringement
• Bullying/harassment
• Scams
• Suicide/self-injury content

ABC: Actors, Behavior & Content

Actor-Behavior-Content framework: originally applied to disinformation, but also applies to understand attempts to abuse systems

​

• Actors: manipulative/bad actors

Who are they? What’s their intent? What networks are they situated in?

​

• Behavior: deceptive/bad acts

What do the actors do? What actions do repeat offenders take?

​

• Content: harmful content

What types of content are created or viewed that can be harmful?

ABC … DEF

• Design: What did the designers, engineers, and other stakeholders imagine would impact user behavior (i.e., that will “afford” technology-mediated activities within the platform)?
• Evaluations: How did they assess and measure if users perform those activities … or if bad actors take action to manipulate those activities?
• Features: What features were ultimately included? Which ones were successful? Did any fail due to bad actors?

Media Richness

• Researchers think about the “modality” of information/content that users encounter on platforms and the relative effects that different modalities have.

​

• Media Richness
• How the medium of communication (e.g., text vs. video) affects the reception and effectiveness of the communication.
• For example, video is “more rich” because it communicates social or non-verbal clues (e.g., facial expressions) better and faster than text.
• Therefore, video could also introduce more abuse in the system.

Text

• Where a lot of Trust & Safety started: text spam
• World Economic Forum
• Internet Society: History of Spam
• Legislation: CAN-SPAM Act of 2003
• Practice: The Art & Science of How Spam Filters Work

Accounts, Login & Profiles

• Account security
• Technical hacking → Passwords
• Social hacking → 2-factor authentication
• Account takeovers

​

• Impersonation
• Age verification & gating
• Repeat offender account replication

Feed, Ranking & Content Distribution

• Content quality (based on feed interaction signals)
• Social content needs (e.g., friends, family, networks, communities)
• Informational content needs (e.g., interests and social issue topics)
• Spam/clickbait
• Misinformation

​

• Engagement quality signals (e.g., time spent, “meaningful interactions”)
• Distribution amplification/suppression of content visibility through ranking
• Intersection of feed system vs. content flags (e.g., misinformation labels)
• Well-being/“scrolling addiction”/perceptions of control

Sidenote: Content Policy

• Meta Transparency Center
• Meta: Types of Content We Demote
• Meta: A Blueprint for Content Governance and Enforcement

Comments & Other Social Signals

• Comments
• Text content quality: hate speech, misinformation, etc.
• Bullying & harassment
• Hate speech
• Comment labels (e.g., Facebook COVID authoritative information labels)

​

• Social signals
• Likes, reactions, and “minutiae”
• Negative feedback
• Angry reactions
• Case study: Facebook & engagement on civic/health content

Private Messaging

• Threaded messaging as a surface/feature set has different social and informational trends than a feed-based system
• More “hidden” abuses
• Bullying/harassment
• Targeted messaging (e.g., child grooming)
• Private messaging may have more restrictive investigatory policies internally at companies
• Encryption (e.g., at Google)
• Masking content/behavior around terrorism, child safety, illicit goods, etc.

User-Generated Images & Video

• User-generated content quality issues/policies still apply as in text
• But images/video have “richer” media
• More technical difficulty to analyze content
• Augmentation of detection capability with AI tools (e.g., image classification, sub-image detection)
• Video exponentially harder than images
• Every frame to be analyzed (video scrubbing for moderation)
• Substantial additional moderation costs
• Pornography
• Unique systems case study: Stop NCII

Short-Form & Streaming Video

• Short-form video
• More production, faster consumption (“smaller” or “larger” effects?)
• Filters present additional trust/safety concerns (e.g., equity and biometric data collection/storage)

​

• Stream video
• Live-streaming detection presents additional difficulties
• Case study: Christchurch Attacks

Advertising

• Scams & low-quality advertisements
• Ad-related tracking, data-based targeting, and ad targeting
• Problematic targeting and discrimination (Facebook Job Ads)
• Political and social issue advertising
• Competing platform policies: Twitter vs. Meta

Ecommerce

• Fraud and financial scams (FBI)
• Consumer feedback systems and fake product reviews
• Illicit goods: drugs, medical supplies, sexual health products, etc.

Side Note: Reporting Flows

• Most platforms have reporting systems for users to report violating content
• But even reporting flows can be an entry point for abuse
• Harassment via coordinated reporting (Facebook)
• Brigading

Digital Self-Defense

• Activists and digital security practitioners – as well as members of protected communities – need appropriate security practices to protect themselves, their devices, and their data
• In fact, everyone should become familiar with strong digital hygiene!
• Activist or Protestor

Preventing Abuses – Policies

• Actors
• Only certain types of actors are allowed on the platform (e.g., Twitter)

​

• Behavior
• Only certain types of behaviors are allowed on the platform (e.g., Meta)

​

• Content
• Only certain types of content are allowed on the platform (e.g., TikTok)

Preventing Abuse – Regulatory

• Design
• Certain processes must be followed to ensure systems are designed appropriately to respect privacy (e.g., Privacy by Design)
• Evaluations
• Certain evaluations must be conducted to ensure systems are designed appropriately to measure harm, bias, etc. (e.g., Platform Transparency & Accountability Act)
• Features
• Certain types of designs/features can only be produced to ensure systems work appropriately after launch (e.g., no “dark patterns”; California)

Attack Surfaces�From a Security Perspective

Security as an Asset

Cybersecurity is a core dimension of responsible innovation with tangible benefits, including:

• Better products
• Trust
• Reputation
• Compliance

Security as a Liability

• Threat actors can quickly transform cybersecurity into a significant liability.
• Average breach = $9.5M in US, $4.5M globally (+2% YOY)
• With additional, long(er)-term costs:
• Regulatory investigations
• Litigation
• Attorneys fees
• Technology enhancements

Threat Actors: Who & Why

Attack Type – Malware

• Any malicious code run on a system or device that alters its state or function without the owner’s informed consent.
• It’s present in virtually every compromised data record.

Attack Type – Social Engineering

• Art of using social or psychological skills to obtain knowledge (passwords, etc.) to gain unauthorized access.
• Generally combined with malware and publicly-available information.
• Variations depend upon the target:
• Phishing (broad audience)
• Spear phishing (specific individual)
• Whaling (wealthy individual)

Spotlight on Ransomware

Malware launched through a social engineering attack that prevents access to a system until a ransom is paid (e.g., in crypto).

Spotlight on BEC

• Specific type of spear phishing attack that attempts to trick an employee into taking harmful action (e.g., sending money)
• BEC has doubled YOY, and now represents 50% of social engineering attacks.

Attack Type – Zero-Day

Any attack that exposes an unknown vulnerability.

Attack Type – Denial of Service

Brute-force method that attempts to overload the capacity of a system or network to render it inoperable.

Attack Type – Insider

Any person who, intentionally or unintentionally, uses their authorized access or knowledge to do harm to an organization’s mission or resources (e.g., privilege misuse, stolen assets, or mistake).

Cost, Frequency by Attack Type

Attack Surface

• The number of all possible points where an unauthorized user can access connected hardware, devices, networks, or applications.
• The smaller the surface, the easier to protect – e.g., with zero-trust principles.

Cyber Kill Chain

Comparison to Kinetic Attack

While there are obvious differences, there are also some similarities:

• Phased progression
• Reliance on intelligence gathering
• Use of:
• Delivery vehicle
• Navigation system
• Payload

​

Cyber Trends

• Cybercrime has evolved like big business (e.g., with as-a-Service offerings).
• 1,265% increase in phishing since ChatGPT launch.
• Double and triple extortion attacks are more prevalent.
• Cybercriminals have begun to weaponize breach disclosure rules (as “whistle blowers”).
• 74% of all breaches included the human element.
• On average, it took 204 days to identify a breach and 73 more to contain it.

Cybersecurity Best Practices

Choose a Framework (NIST, etc.)

Spotlight on the NIST Framework

Adopt Best Practices

• Penetration testing: exercise reflecting real-world conditions in which assessors use all available documentation (e.g., system design, source code, and manuals) to try to circumvent the security features of an information system.
• Red teaming: exercise reflecting real-world conditions that simulates an adversarial attempt to compromise an organization’s security.

Pen Testing vs. Red Teaming

Emerging AI Attacks

Prompt Injections

Manipulating an AI system using carefully crafted prompts that make the model ignore filters or previous instructions or perform unintended actions.

Data Poisoning

Deliberate contamination of AI training data to compromise model performance (e.g., by inducing biases, errors, or vulnerabilities that manifest post-deployment).

Training Data Extraction

Prompting an AI model to use its output to infer some of its training data.

AI Red Teaming

• Recently, the concept of cyber red teaming has been extended more broadly to AI governance, notably by:
• Microsoft
• Google
• NVIDIA
• It has been endorsed as a best practice (e.g., in President Biden’s October 2023 Executive Order) and will likely be required under all major AI laws.

​