1 of 42

Amass Workshop

Scripting OWASP Amass for a Customized Experience

Red Team Village

2 of 42

Would you like greater control over your recon process?

  • For gathering other types of information

​

  • For answering additional questions with the collected data

​

  • For easier management of the storage and retrieval of the discoveries.

Agenda

3 of 42

Speaker

Jeff Foley

Project Leader, OWASP

The Amass Flagship Project

Over 20 years of experience focused on applied research & development, security assessment, vulnerability management, and attack surface management:

​

  • Vice President of Research at ZeroFox
  • Global Head of Attack Surface Management at Citigroup
  • Global Mgr of Vulnerability Engineering at National Grid
  • Program Mgr of Offensive Cyber Warfare Research & Development at Northrop Grumman Corporation

​

GitHub: @caffix

Discord: @caffix

Twitter: @jeff_foley

Email: jeff.foley@owasp.org

Mastodon: @caffix@infosec.exchange

LinkedIn: https://www.linkedin.com/in/caffix

4 of 42

RTV - Volunteers

James Bettke

Security Researcher, ZeroFox

Adem Rosic

Security Researcher, ZeroFox

5 of 42

Introduction

6 of 42

What is OWASP Amass?

  • Amass is an OWASP Flagship Project that performs in-depth attack surface mapping and asset discovery.

  • The project actively enhances the framework used by security practitioners and programs across the globe.

​

  • The framework allows users to collect attack surface intelligence, and then analyze the findings, for better understanding of external exposures.

7 of 42

Where can you find OWASP Amass?

  • Socials:
    • GitHub: @owasp-amass
    • Twitter: @owaspamass
    • Mastodon: @amass@infosec.exchange

​

8 of 42

Why was the Amass Project Started?

  • Raising awareness around securing externally exposed assets.
    • The project was started nearly six years ago, when very little attention was being given to an organization’s external attack surface.
    • Amass provided a more turnkey solution, to help organizations better understand their public-facing assets on the Internet.

​

  • Amass was designed to provide an automated solution.
    • Amass is not an interactive OSINT investigative tool
    • Users can execute tasks in the background while the Amass engine performs the recursive discovery process.
    • The framework saves findings and metadata in a database that is updated and queried across sessions.

9 of 42

Following Along with the Exercises

  • Install Go on your system (https://go.dev/dl/)

​

  • Be sure that SQLite has been installed on your system.

​

  • Install the OWASP Amass command-line collection tool

​

  • Execute an enumeration against owasp.org
    • amass enum -dir testdir -d owasp.org

​

  • Change directories into the test output directory
    • cd testdir

10 of 42

The Open Asset Model

11 of 42

The Announcement at DEF CON 30!

​

​

Amass Project Leader announced that its definition of attack surface would be broadened to go beyond IT and internet infrastructure assets.

Jeff Foley, Jason Haddix and Ben Sadeghipour at the Recon Village Attack Surface Management panel discussion.

​

12 of 42

What exactly did this mean for Amass?

  • The ability to collect intelligence on other types of assets exposed on the Internet.

​

  • A new data model capable of supporting the agile and extendable taxonomy.

​

  • Implementing this on popular and trusted database management systems that are easily accessible by users.

13 of 42

What is the Open Asset Model?

  • The Open Asset Model (OAM) provides a new standard for asset definitions, representing a comprehensive framework for describing and categorizing diverse internet-facing assets.

​

  • OAM defines not just the assets themselves, but also the relationships across types of assets, allowing the model to express the real-world interconnectedness that exist between assets.

​

  • The Amass community can quickly adapt the model to include new types of assets exposed on the Internet, and their relationships to each other.

14 of 42

What is the Asset Database?

  • The Asset Database implements the OAM, offering the interaction layer to store discovered assets in the popular SQLite and PostgreSQL database management systems.

​

  • The Asset Database allows scanning and analysis tools to store and analyze assets from the OAM and their relationships.

​

  • The data is in a graph-like structure, yet different types of database management systems can be used for storage.

15 of 42

Under the Hood

  • The assets table includes basic details about the findings.
  • Rows in the relations table are the glue that tie the assets together.
  • All the data is time stamped when first created and last seen.

ID

Type

From

To

1

a_record

1

2

2

contains

3

2

3

announces

4

3

4

managed_by

4

5

ID

Type

JSON

1

FQDN

{name: ‘owasp.org’}

2

IPAddress

{address: ‘172.67.10.39’}

3

Netblock

{cidr: 172.67.10.0/25}

4

ASN

{number: 13335}

5

RIROrg

{name: ‘Cloudflare, Inc.’,�desc: ‘AS13335’}

Assets

Relations

16 of 42

The Advantages

​

  • There’s nothing but assets and their relationships.

​

​

​

  • Powerful queries can be crafted to answer interesting questions.

​

​

KEY BENEFITS

The relations tie everything together, making it possible to navigate the data in new ways

Ask new questions from your data while not having to rescan targets

The design and taxonomy are very extendable, allowing the community to create new asset types

17 of 42

Open Asset Model Workshop Exercises

18 of 42

Exploring the Collected Data

  • There’s no need to have your recon process limited by what tool developers assume you want.
  • The Amass framework, and future recon tools, can collect the intelligence and allow you to ask the right questions

​

  • The method of communication between your use cases and the recon tools is the Open Asset Model!

19 of 42

Extraction of Subdomains

sqlite3 amass.sqlite "SELECT content->>'name' FROM assets WHERE type = 'FQDN' AND content->>'name' LIKE '%owasp.org'"

SELECT content->>’name’ – Request the FQDN names�FROM assets �WHERE type = ‘FQDN’ – We are only interested in FQDN assets�AND content->>’name’ LIKE ‘%owasp.org’ – Filter by namespace

20 of 42

Names and IP Addresses

sqlite3 amass.sqlite "SELECT fqdns.content->>'name',ips.content->>'address' FROM ((assets AS fqdns INNER JOIN relations ON fqdns.id = relations.from_asset_id) INNER JOIN assets AS ips ON relations.to_asset_id = ips.id) WHERE fqdns.type = 'FQDN' AND ips.type = 'IPAddress' AND relations.type IN ('a_record', 'aaaa_record') AND fqdns.content->>’name’ LIKE ‘%owasp.org’'"

SELECT fqdns.content->>'name',ips.content->>'address' – Request the JSON data�FROM ((assets AS fqdns �INNER JOIN relations ON fqdns.id = relations.from_asset_id) �INNER JOIN assets AS ips ON relations.to_asset_id = ips.id) �WHERE fqdns.type = 'FQDN' AND ips.type = 'IPAddress' – Only specific assets types�AND relations.type IN ('a_record', 'aaaa_record') – Multiple types of interest�AND fqdns.content->>’name’ LIKE ‘%owasp.org’ – Filter by namespace

21 of 42

Extract Zones in the Namespace

sqlite3 amass.sqlite "SELECT content->>'name' FROM assets WHERE type = 'FQDN' AND content->>'name' LIKE '%owasp.org' AND id IN (SELECT from_asset_id FROM relations WHERE type = 'node')"

SELECT content->>’name’ – Request the JSON data�FROM assets�WHERE type = 'FQDN' – Only specific assets types

AND content->>’name’ LIKE ‘%owasp.org’ – Filter by namespace�AND id IN (SELECT from_asset_id FROM relations WHERE type = ‘node’) – Apex?�

22 of 42

Names Resolve to what CIDRs?

sqlite3 amass.sqlite "SELECT DISTINCT nets.content->>'cidr' FROM ((((assets AS fqdns INNER JOIN relations AS rels1 ON fqdns.id = rels1.from_asset_id) INNER JOIN assets AS ips ON rels1.to_asset_id = ips.id) INNER JOIN relations AS rels2 ON ips.id = rels2.to_asset_id) INNER JOIN assets AS nets ON rels2.from_asset_id = nets.id) WHERE rels1.type IN ('a_record', 'aaaa_record') AND rels2.type = 'contains' AND rels1.last_seen > '2023-08-01 00:00:00' AND rels2.last_seen > '2023-08-01 00:00:00' AND fqdns.type = 'FQDN' AND ips.type = 'IPAddress' AND nets.type = 'Netblock' AND fqdns.content->>'name' LIKE '%owasp.org'"

​

SELECT DISTINCT nets.content->>'cidr' -- Requesting unique CIDR blocks

FROM ((((assets AS fqdns

INNER JOIN relations AS rels1 ON fqdns.id = rels1.from_asset_id) -- Connect FQDNs and IP addresses

INNER JOIN assets AS ips ON rels1.to_asset_id = ips.id)

INNER JOIN relations AS rels2 ON ips.id = rels2.to_asset_id) -- Connect Netblocks to IP addresses

INNER JOIN assets AS nets ON rels2.from_asset_id = nets.id)

WHERE rels1.type IN ('a_record', 'aaaa_record') AND rels2.type = 'contains' -- Check for the correct types of relations

AND rels1.last_seen > '2023-08-01 00:00:00' AND rels2.last_seen > '2023-08-01 00:00:00' -- Filter by date/time

AND fqdns.type = 'FQDN' AND ips.type = 'IPAddress' AND nets.type = 'Netblock' -- Check for the correct asset types

AND fqdns.content->>'name' LIKE '%owasp.org' -- All the FQDNs must exist in the DNS namespace of interest

23 of 42

Extra Example: Name Servers

sqlite3 amass.sqlite "SELECT apex.content->>'name',servers.content->>'name' FROM ((assets AS apex INNER JOIN relations ON apex.id = relations.from_asset_id) INNER JOIN assets AS servers ON relations.to_asset_id = servers.id) WHERE apex.type = 'FQDN' AND servers.type = 'FQDN' AND relations.type = 'ns_record' AND apex.content->>'name' LIKE '%owasp.org'"

SELECT apex.content->>'name',servers.content->>'name' – Request the JSON data

FROM ((assets AS apex

INNER JOIN relations ON apex.id = relations.from_asset_id)

INNER JOIN assets AS servers ON relations.to_asset_id = servers.id)

WHERE apex.type = 'FQDN' AND servers.type = 'FQDN' – Only specific assets types

AND relations.type = 'ns_record'

AND apex.content->>'name' LIKE '%owasp.org' – Filter by namespace�

24 of 42

Extra Example: Dangling PTR Records

sqlite3 amass.sqlite "SELECT ptrs.content->>'name',fqdns.content->>'name' FROM ((assets AS ptrs INNER JOIN relations ON ptrs.id = relations.from_asset_id) INNER JOIN assets AS fqdns ON relations.to_asset_id = fqdns.id) WHERE ptrs.type = 'FQDN' AND fqdns.type = 'FQDN' AND relations.type = 'ptr_record' AND fqdns.content->>'name' LIKE '%owasp.org' AND fqdns.id NOT IN (SELECT from_asset_id FROM relations WHERE type IN ('a_record','aaaa_record','cname_record'))"

SELECT ptrs.content->>'name',fqdns.content->>’name’ – Request the JSON data�FROM ((assets AS ptrs �INNER JOIN relations ON ptrs.id = relations.from_asset_id) �INNER JOIN assets AS fqdns ON relations.to_asset_id = fqdns.id) �WHERE ptrs.type = 'FQDN' AND fqdns.type = ‘FQDN’ – Only specific assets types�AND relations.type = 'ptr_record'

AND fqdns.content->>’name’ LIKE ‘%owasp.org’ – Filter by namespace�AND fqdns.id NOT IN (SELECT from_asset_id FROM relations �WHERE type IN (‘a_record’,’aaaa_record’,’cname_record’)) – Dangling records?�

25 of 42

Extra Example: Orgs Managing the CIDRs

sqlite3 amass.sqlite "SELECT nets.content->>'cidr',asns.content->>’number’,orgs.content->>'name' FROM ((((assets AS nets INNER JOIN relations AS rels1 ON nets.id = rels1.to_asset_id) INNER JOIN assets AS asns ON rels1.from_asset_id = asns.id) INNER JOIN relations AS rels2 ON asns.id = rels2.from_asset_id) INNER JOIN assets AS orgs ON rels2.to_asset_id = orgs.id) WHERE rels1.type = 'announces' AND rels2.type = 'managed_by' AND orgs.type = 'RIROrg' AND asns.type = 'ASN' AND nets.type = 'Netblock' AND nets.content->>'cidr' IN ('172.67.0.0/16','2606:4700:10::/44','104.22.16.0/20','13.225.60.0/22')"

​

SELECT nets.content->>’cidr’,asns.content->>’number’,orgs.content->>’name’ -- Requesting org names

FROM ((((assets AS nets

INNER JOIN relations AS rels1 ON nets.id = rels1.to_asset_id) -- Connect Netblocks and CIDRs

INNER JOIN assets AS asns ON rels1.from_asset_id = asns.id)

INNER JOIN relations AS rels2 ON asns.id = rels2.from_asset_id) -- Connect Netblocks to IP addresses

INNER JOIN assets AS orgs ON rels2.to_asset_id = orgs.id)

WHERE rels1.type = ‘announces’ AND rels2.type = 'managed_by' -- Check for the correct types of relations

AND orgs.type = 'RIROrg' AND asns.type = 'ASN' AND nets.type = 'Netblock' -- Check for the correct asset types

AND nets.content->>'cidr' IN ('172.67.0.0/16',’2606:4700:10::/44’,’104.22.16.0/20’,’13.225.60.0/22’)

26 of 42

Amass Data Source Scripting

27 of 42

How Do They Work?

  • Amass Data Source (ads) scripts are event driven.
    • Responding to events that take place within the Amass Engine.
    • The events cause callback functions to be executed within the loaded scripts.
    • If a script is loaded, and it implements the callback routine, then it will be executed when the event takes place.
  • The scripts are written in the Lua programming language.
    • The same scripting language leveraged by Nmap and Wireshark.
    • The language is designed primarily for embedded use in applications.
      • This makes it easy to extend the language and have the scripts interact with functionality implemented in the Go executable.

28 of 42

How Do They Work Cont?

  • When the Amass Engine initializes, scripts are loaded from static and configurable locations.
    • The default scripts are built into the amass executable file.
    • Others can be discovered in a directory named ‘scripts’ in the output directory.

​

  • Users can determine which scripts are loaded by using flags on the command line.
    • The ‘-include’ flag only allows the specified scripts to be loaded.
    • The ‘-exclude’ flag will prevent the specified scripts from loading.

29 of 42

The Events That Are Supported

  • Some of the callback routines are attempted for all scripts during loading and unloading
  • Most are called in response to events within the Amass Engine
  • The callback routine names are listed here ->
  • start
  • Stop
  • check
  • vertical
  • horizontal
  • resolved
  • subdomain
  • address
  • asn

30 of 42

Let’s Take a Look at One

  • A very simple script in the GitHub repo:
  • Some takeaways from investigating the script:
    • All scripts must declare a unique name global variable.
    • All scripts must declare a type global variable.
      • While the type could be set to anything, some types have special meaning and are considered more trusted than others.
    • Scripts can implement a start callback routine that gets executed when the script is first loaded.
    • We saw the vertical callback routine, which is executed by the engine for vertical domain name correlation.

31 of 42

Writing a Simple Script

name = "Simple"

type = "ext"

​

function vertical(ctx, domain)

local path = output_dir(ctx) .. "/names.txt"

​

local f = io.open(path, "r")

send_names(ctx, f:read("*all"))

f:close()

end

​

32 of 42

Script that Executes a Tool

name = "Assetfinder"

type = "ext"

​

function vertical(ctx, domain)

local cmd = output_dir(ctx) .. "/assetfinder --subs-only " .. domain

local data = assert(io.popen(cmd))

​

for line in data:lines() do

new_name(ctx, line)

end

data:close()

end

33 of 42

Basic Testing Techniques

  • A Sometimes, your best friend during testing is print.
    • It can be easiest to understand what’s happening inside the script you’re writing by printing out the values being created and acquired.

​

  • When printing does not reveal the problem, you should check the amass.log file.
    • When you run amass, an amass.log file is created in your output directory.
    • The log file includes major errors experienced by the Amass Engine and output from the data source scripts.

34 of 42

Integration with a Data Source

  • Lua was designed to be a data processing language.
    • It’s usually easier to perform powerful network programming in the Go runtime environment, and then export those features into Lua.
  • The Amass Engine has some important functions exposed to data source scripts that include:
    • HTTP requests, DNS queries, and web crawling
    • TLS certificate grabbing and basic socket communications
    • Obtaining access to cached data and the engine config
    • Functions for sharing discovered open-source intelligence.
  • Integrating with a data source on the Internet often requires using these functions exposed by the Amass Engine.

35 of 42

Integration with a Data Source Cont

  • Let’s go back to our Wayback Machine script from earlier.
  • This script uses the ‘scrape’ function implemented by the engine.
    • The scrape function performs a HTTP GET request and provides all the DNS domain names in-scope to the engine.
  • Definitions for the other functions can be found here:

36 of 42

Wrapping Up

37 of 42

Improve Recon Processes

  • Keeping cleaned up text files from the various tools being leveraged.
  • Combining output files and providing as seed data to other tools.
  • This can require complex scripts and cause you to lose valuable data collected by your tools.

Hub and Spoke Approach

Managing Output from Tools

OAM

38 of 42

Continuous Reconnaissance

Extract the information in scope of the current recon task / tool for use as seed data

Retrieval

Newly discovered and/or modified assets are quickly shared with appropriate systems and practitioners

Notify

The tool collects the intelligence it’s designed to gather and prepares it for storage

Discovery

Contribute findings to the OAM database for easy access to other discovery and analysis tools

Store

OAM

01

02

03

04

39 of 42

Imagine the OAM with New Asset Types

IT Assets

Non-IT Assets

  • TLS Certificates
  • WHOIS Records
  • Domain Name Registrars
  • Exposed Applications
  • Ports & Tech Stacks
  • Email Addresses
  • Accounts & Credentials
  • API Endpoints
  • Hosted Brand Materials
  • Individuals
  • Organizations
  • Phone Numbers
  • Street Addresses
  • Affiliated Individuals
  • Third-party Vendors
  • Organization’s Employees
  • Organization’s Executives
  • Organization’s Facilities

40 of 42

The OAM is Community-Driven

  • The Open Asset Model is a community-driven effort to uniformly describe assets and the relationships connecting them.

​

  • The Open Asset Model is only as good as the community contributing to it.

​

  • Reach out to the project if you’d like to help shape the OAM to even more accurately represent external attack surfaces.

41 of 42

Thank you

42 of 42

Questions?