1 of 36

Being a Better Reverse Engineer

With Pictures

by Kyle Martin

2 of 36

whoami

A Computer Engineer at NYU

A Reverse Engineer at Large

An Intern at Trail of Bits and Vector35

The Vice President and Head of Research at the OSIRIS Lab

Cyber Security Researcher

CTF Challenge Writer

Insane

And more

And more

And more….

3 of 36

Let’s Start

4 of 36

What is this?

5 of 36

Let’s Get Some Hints

6 of 36

Hint One

Does this help? (Usage?)

7 of 36

Hint Two

Let’s look inside….

(How it works?)

8 of 36

Hint Three

What does it do?

(What it’s called?)

9 of 36

Hint Four

Where does it belong?

10 of 36

What is this?

11 of 36

Here It Is

12 of 36

What Worked Best?

  • Hint One
    • Usage
  • Hint Two
    • Internals
  • Hint Three
    • What it’s called
  • Hint Four
    • Context
  • Video
    • Putting it all together

13 of 36

What’s The End Goal?

Understand:

  • What it does?
  • How it works?
  • How it is used?
  • How it was made?
  • Why it was made?
  • ...how it can be broken?

But to these ends, we need to know as much as possible about it.

14 of 36

How Do Learn About

It

Quickly?

15 of 36

Build Some Tooling For It.

Runbooks/Guides (Crime scenes, disaster recovery, forensics)

Policy (What to/not to do; Criminal investigation, penetration testing)

Analysis Tools (Crime - DNA, data matching, categorization, last week’s talk)

Something Else

​

16 of 36

Applying This To

The It

We Care About Here

17 of 36

x86

18 of 36

Reduced Instruction Set Computing

Turing Complete

A system that can be used to solve any computation problem

Simple Turing Completeness?

Break into fundamental operations:

Math : Add

Control-Flow : Jump if ____?

19 of 36

Ultimate Reduced Instruction Set Computing

One instruction:

Subtract and branch if less than or equal to zero

jmp c:

subleq Z, Z, c

20 of 36

Ultimate Reduced Instruction Set Computing

One instruction:

Subtract and branch if less than or equal to zero (src, dest)

mov a, b:

  • subleq b, b
  • subleq a, Z
  • subleq Z, b
  • subleq Z, Z

21 of 36

The Most Ultimate Reduced Instruction Set Computing

And because x86 is bad and Intel/AMD are monsters:

The ZERO-INSTRUCTION turing complete compiler.

  • x86’s fault handling is turing complete. Wonderful.

​

But is any of this clear?

22 of 36

No.

No It’s Not.

23 of 36

What’s the problem?

​

​

​

​

​

1Sn’7 iT 3Asy???

24 of 36

How do we fix this?

Less Ultimate-RISC

More instructions that perform simple tasks

25 of 36

New Language

14 Operations:

  • Add, sub, mul, div
  • Xor, And, Or, Not
  • Jump If Less Than, Jump If Greater Than, Jump If Equal, Jump
  • Mov
  • Interrupt - to interact with the rest of the system

26 of 36

Did this help?

14 instructions:

add, sub, mul, div, xor, or, and, not, jl, jg, je, jmp, mov, int; (src, dest)

jmp c:

jmp c

mov a, b:

mov a, b

27 of 36

Back to x86

Over 1700 different instructions

Over 7400 total different forms those instructions can take

  • mov reg, reg; mov mem, reg; mov reg, mem; etc

Much undocumented behavior

28 of 36

Intermediate Representations

A Turing Complete system can be used to solve any computation problem

  • We can represent anything in x86 in our Turing Complete System

Example: Jump if greater or equal

  • jl end
  • jmp dest
  • end:

29 of 36

Reinventing The Wheel

Tons of Intermediate Representations (IRs) already Exist:

  • BinaryNinja’s Low Level and Medium Level Intermediate Representations
  • LLVM
  • GNU RTL
  • CIL
  • SIL
  • PCODE
  • And tons more...

30 of 36

What have we done?

Become faster learners:

  • We don’t need to know what every instruction does
    • DO NOT READ EVERY INSTRUCTION (as a human)
  • Let the tool do the work, use the IR’s to build most your understanding

Obtained the facts

31 of 36

What Do We Do With The Facts?

32 of 36

Remember, What Is The End Goal?

Understand:

  • What it does?
  • How it works?
  • How it is used?
  • How it was made?
  • Why it was made?
  • ...how it can be broken?

And now we are equipped with the fundamental facts about it (the local ‘what’).

33 of 36

What Is Important?

Generally, in software:

  • How our data is manipulated
    • (How input is manipulated to generate output)
    • Input => Authentication
    • Input => Goods (Amazon?)
    • Input => Services (Amazon?)

​

34 of 36

Work To Be Done

You, the reverse engineer, need to

  • Stop wasting your time
    • Limit your scope
    • Don’t read everything
    • Don’t reinvent the wheel
    • Recognize the benefit and downfalls of different tools you’re using
    • Use the right tools for the right projects
  • Get to what’s most important quickly
    • Determine what’s important
    • Obtain that information
    • There’s still work to be done here (this is the hard part)

35 of 36

Additional Resources/Bibliography

https://en.wikipedia.org/wiki/Turing_completeness

https://softwareengineering.stackexchange.com/questions/132385/what-makes-a-language-turing-complete

http://cs.lmu.edu/~ray/notes/ir/

https://net.cs.uni-bonn.de/fileadmin/user_upload/plohmann/x86_opcode_structure_and_instruction_overview.pdf

​

36 of 36

Being a Better Reverse Engineer

by Kyle Martin

elyk@nyu.edu

GitHub.com/KyleMiles

KyleMiles.me

@ElykDeer

@Elyk