1 of 51

Talking about Pros and Cons

Jacob Torrey - Thinkst Labs

2 of 51

Outline

History

CC / ATAT
ThinkstScapes

Present

Citation
Current stats

Data formats
Time to analyze

Community

Pros & Cons

Topics

Spread
Solved problems / broken records

Conclusions and next steps (future?)

3 of 51

What’s in a name?

Talking about Pros and Cons

Pros - The people who give talks, and join “the circuit”
Cons - The conferences around the world

Our field is getting bigger, but does bigger mean better?

If we cannot measure our progress, how can we hope to improve?

4 of 51

Personal note

Felt like a fraud being asked to give a keynote

Keynotes are for people who have done lots of impressive things

Why was I asked?

Spent a lot of time reviewing other keynotes
Good ones:

Get discussion going (even if it’s disagreeing – if you hate my talk, maybe that’s good :P )
Reflect, and then look forward

I hope to do that, so let’s take a step back in time…

5 of 51

2013 (a decade ago)

6 of 51

A Talk about Talks

7 of 51

A Talk about Talks

Release of CC, the world’s (first?, best?, biggest?, ugliest?) collection of infosec talks
Looked at the trends of # of cons, # of talks, # of speakers

To the moon

Highlighted the entertainment value of cons, and asked tough questions

Does anyone remember the outcome of the most hyped talk from last year’s Black Hat?
Have we solved any problems, shared that solution, and not had to deal with it again?
Are there bug classes now laid to rest?
Are conferences the right venue for sharing information and collaborating?

8 of 51

ThinkstScapes

While CC was being built, Thinkst was hard at work on a related effort: ThinkstScapes
Haroon and Marco reviewing and extracting the signal from the noise from infosec talks for paying customers

Helped fund Thinkst’s start

Helps gain perspective

9 of 51

2021 (2 years ago)

10 of 51

ThinkstScapes was dead, long live ThinkstScapes

It’s back (celebrating two years of resurrection!)
It’s free!
It’s valuable (I think)
It’s available @ thinkst.com/ts

Go get it (even if only for the nature photos)

11 of 51

2023 (0 years ago)

12 of 51

CC is dead, long live Citation

Took the data from CC, made it prettier and better (citation.thinkst.com)

Had a collection gap ~2020-2021 (can you remember something happening then?)

13 of 51

Citation dataset

Probably the biggest collection of infosec talks

Impressive!

Can use PostGREST to query programmatically

Why?

Unless we know our history, we’re doomed to repeat it

14 of 51

Exported to Neo4j

Easier for me to reason about relationships

I think in graphs -> I can query in graphs

15 of 51

What questions do I have?

During a typical keynote, now I’d use my impressive background to offer insightful answers about the community, and then finish with an optimistic call to action!

Instead of my unimpressive background, I’ll ask the impressive dataset

So let’s ask about:

Trends
Siloes
Topics
What have we accomplished?

16 of 51

Community size (pros) and growth (cons)

As you can see, our industry generates a lot! A couple caveats, 2020/2021 we had a gap in collection, some of the conferences have archives we can recover, but some are lost. Also, in 2021 we reduced the number of conferences collected, many of the smaller BSides were too difficult to get data for, and I dropped the ball on a few but have been adding them slowly (e.g., the 2021 BsidesLV was added yesterday).

That said, this chart has to be log scale due to the immense growth. Around 2013, there’s a peak, and then it levels off. I would expect that it’s leveling off now if I collected a better set from 2021-now. COVID really changed things, old established conferences took a break, some didn’t come back, some didn’t survive, etc. There are some great new conferences too, it will be interesting to see how they establish themselves in a crowded, but dynamic market.

17 of 51

Number of talks

18 of 51

Number of recycled talks

19 of 51

Conferences are incestuous

A lot of speaker overlap between conferences
> 31k authors have presented at multiple cons
Conferences are tightest between editions

BSides - BlackHat - DEF CON are closest

Then OWASP - ShmooCon - DerbyCon - HITB

Small cryptography clique: (EU|Asia)Crypt and Crypto

20 of 51

Conferences are incestuous

21 of 51

Conferences that are unique

Some conferences form the nexus of a unique community and are less connected to other conferences by the Pros e.g.:

THOTCON
HOPE
TROOPERS
Virus Bulletin
ZaCon

These are either regional conferences, or more specialized without much “competition”

22 of 51

Hot topics

For each of the following we’ll see a chart showing the number of talks on that topic per year, [then grouped by conference]

Where is cutting-edge research promo’d?
Where does it gain momentum?

Topics:

AI/ML

LLMs

OT/ICS/SCADA
Fuzzing
Phishing/Social Engineering
Mobile

23 of 51

24 of 51

25 of 51

26 of 51

27 of 51

28 of 51

29 of 51

30 of 51

Solved problems?

Let’s see if there are any solved problems, reviewing the earliest topics:

Cryptography libraries
Secure telnet authentication
Viruses/malware
Cyber-crime
EMF side-channels
Hacking cellular phones
Session-layer encryption
Solving buffer overflows

Challenge: Can anyone guess the dates for these first talks?

31 of 51

Solved problems?

Let’s see if there are any solved problems, reviewing the earliest topics:

Cryptography libraries (1990)
Secure telnet authentication (1990)
Viruses/malware (1993)
Cyber-crime (1994)
EMF side-channels (1994)
Hacking cellular phones (1994)
Session-layer encryption (1995)
Solving buffer overflows (1998)

Let’s look at each of these

Nokia 2010 (1994)

32 of 51

Solved problems

Crypto libraries

33 of 51

Solved problems

Crypto libraries

34 of 51

Solved problems

Secure telnet

Viruses/malware

35 of 51

Solved problems

Secure telnet

Viruses/malware

36 of 51

Solved problems

Cyber crime

37 of 51

Solved problems

Cyber crime

38 of 51

Solved problems

Mobile phone security

EMF side-channels

39 of 51

Solved problems

Mobile phone security

EMF side-channels

40 of 51

Solved problems

Session-layer encryption

41 of 51

Solved problems

Session-layer encryption

42 of 51

Solved problems

Solving buffer overflows

43 of 51

Solved problems

Solving buffer overflows

44 of 51

2023+ (0+ years from now)

45 of 51

Conclusions 10 years ago

46 of 51

Aside: Chess vs. poker

Me (along with many others) have remarked that there are different “games” we play in security

Chess is focused on the game
Poker is more about playing the players

If we don’t know what game we’re playing, how can we possibly win?

Just to be meta, here is a screenshot from my boss’ keynote, with a screenshot of my blog

Boss

Me

47 of 51

Conclusions now

48 of 51

Pretty doom and gloom, huh?

49 of 51

Area of hope? Pre-registration and selection bias

Few talks on failing to hack something will be accepted
Talks on failure are actually interesting: Eugene’s analysis of no one pwning Synology at Tianfu cup

But I think, we are not seeing the whole picture in the data we have alone. I have an intuition that we’re making more progress than we think, but we don’t share the results where we can’t hack the target. I would wager it’s harder to get RCE from a browser payload now than even a few years ago, and the mitigations and improvements in place are making it safer. At HICSS I saw the keynote there, Jevin West talk about biases in results. For this specific case, it was about an antidepressant and how it was working. If you looked at the academic journal publications, it was looking pretty good, only 3 negative results! The FDA, which in the US regulates drugs, requires the results of all studies to be reported, those same tests were a bit less promising when you see what’s below the cut line.

Jevin called for more conferences to accept pre-registered experiments. You submit the idea for what you want to do, and based on that you are accepted/rejected, then regardless of results you can present. This would let us find more data on where we are doing well–for example Eugene Kim aka spaceraccoon gave a talk at shmoocon about how despite going into the Tianfu hacking cup in China with a team and bucket of 0-days, they, and all the other teams were unable to beat a consumer Synology device through good engineering and hardening. Perhaps if we see more of the unsuccessful hacks, it can balance out the parade of bugs and fails.

50 of 51

Conclusions now

We should recognize conferences for what they are:

Social / collaboration-focused

We should aim to collect better representation of the work we do
We should find other ways for the things they are not good at:

Dissemination of information

[I think] regional conferences are the ideal fit

They help make connections
They are less impactful on the environment
They can give a forum for new speakers, and for repeat speakers alike

Blogs, social media, email are good to disseminate information

We want searchable summaries with references and links to code
[I think] ThinkstScapes is important
[I think] knowledgehub.social is useful

51 of 51

Thank you!

Questions?

@jacob@mountaincommunity.co