1 of 43

Introduction to antivirus evasion: Understanding and bypassing signature-based detection

Emmanuel HEMADOU aka W1z4rd�hem4emanuel@gmail.com 911

+229 9110375403754

https://www.linkedin.com/in/emmanuel-hemadou-902725190/

[01/03/2026]

2 of 43

whoami

$ initialize --profile

> Loading……

>Name : Emmanuel HEMADOU ( W1z4rd )

>Title : Cybersecurity Analyst at bjCSIRT

> - IR ( Incident Response )

> - Red Teaming

> - Monitoring

> - Formations (RSSI, étudiants, professionnels IT)

$ run – achievements

>Events : [Technical team which organizing ] the NextGeninCyber CTF 2024 ; Lesetho

>Projects : Technical Lead of Hackerlab, a CTF competition held annually in Benin

>Certifications : OSWE, OSEP, OSCP & eWPTX (Battle-hardened in offensive security )

3 of 43

Agenda

Overview of antivirus software
Simulation of the target environment
Process of bypassing antivirus signature detection
Limits of evasion by signature
Conclusion & Q&A

4 of 43

Why understand evasion ?

Organisation

Blue team

Red team

Antivirus software monitors and protects against malware and attacks.

Understanding escape to test and evaluate infrastructure security

Understanding escape strategies to anticipate and strengthen defenses

5 of 43

Overview of antivirus software

6 of 43

Overview of antivirus software

Yesterday / 20 years ago�Basic detection based solely on signatures�Ineffective against modern malware

Transition�Signature-based detection with the introduction of the first signature databases�Behavioral analyses

Today / Modern�Advanced multi-layered systems�Signature + heuristics + sandboxing�Cloud detection and AI

Evolution of antivirus software

7 of 43

Overview of antivirus software

How does an antivirus program work?

File downloaded / executed

Analysis (Signature / Heuristic)

Verdict

Removal / Quarantine

8 of 43

Overview of antivirus software

Detection methods

🧩 1. Signature-based detection

MD5/SHA-1 hashing

Unique byte sequences

Massive database

Fast and efficient… but fragile

🔎 2. Heuristic/Behavioral Detection

Execution Simulation

Sandbox

Analysis of Suspicious Behaviors

Slower but More Advanced

☁ 3. Cloud/AI Detection

Distributed Analytics

Machine Learning

More expensive

Less widespread

9 of 43

Simulation of the target environment

10 of 43

Simulation of the target environment

Red Team Engagement

Simulation ?

local reproduction of the target environment

Online services(Virustotal / Antiscan.Me)

11 of 43

Simulation of the target environment

Virustotal ⚠️

+50 antivirus engines

Quick and pratical analysis

⚠️ Share the samples with AV editors

Risk of disclosure of techniques�

AntiScan.Me

26 antivirus engines

3 free analysis

No sharing of samples with AV publishers

No risk of disclosure of techniques

Then a paid subscription�

12 of 43

Simulation of the target environment

For educational and controlled demonstration purposes, we will use:

🛡️ Virscan.org https://www.virscan.org

🛡️ ClamAV

🛡️ Microsoft Defender

📦 LitterBox (outil local multi-AV) https://github.com/BlackSnufkin itterBox

�

13 of 43

Process of bypassing antivirus signature detection

14 of 43

Process of bypassing antivirus signature detection

Location of signatures in files

🔍 Key idea

An antivirus signature can be based on:

📌 A complete hash of the file

📌 A specific sequence of bytes in binary

📌 A unique binary pattern identified par rétro-ingénierie

🧪 Theoretical approach to localizationRather than analyzing the antivirus engine itself (complex and proprietary),

�A simplified approach consists of:

1️⃣ Divide the file into segments

�2️⃣ Scan each segment individually

�3️⃣ Gradually reduce the size of the segments

�4️⃣ Identify the area that triggers detection

👉 Dichotomous reduction principle appliquée à l’analyse binaire

15 of 43

Process of bypassing antivirus signature detection

Location of signatures in files

🧪 Theoretical Approach to Location

This method was initially implemented in a popular tool called Dsplit. Since the original Dsplit tool is no longer available, we are using the following PowerShell script instead: Find-AVSignature: https://obscuresecurity.blogspot.com/2012/12/finding-simple-av-signatures-with

16 of 43

Process of bypassing antivirus signature detection

Location of signatures in files

🧪 Demo

During the live stream, we will generate an executable with Metasploit, analyze its detection with ClamAV, and then use Find-AVSignature to precisely identify the byte sequences in the binary flagged by the antivirus. We will then make targeted modifications to try to bypass the detection.

17 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

1️⃣ Encoder 🔄

Shellcode modification

Polymorphism

❌ Limitation :�Decoding stub detectable

18 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

1️⃣ Encodur 🔄

The x86/shikata_ga_nai encoder (highlighted above) is a commonly used polymorphic encoder that produces a different output each time it is run, making it effective for signature evasion.

19 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

1️⃣ Encoder 🔄

Generating a shelldode using the x86/shikata_ga_nai encoder

20 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

1️⃣ Encoder 🔄Since the assembler code has been obfuscated, we will copy the generated executable to our Windows 10 victim machine and scan it with ClamAV.

Based on the results above, ClamAV detected the shellcode encoded in the executable. This workaround failed because the encoded shellcode must be decoded to execute, which requires a decoding routine. However, this routine is not encoded; it is therefore static with each execution, making the decoder itself an ideal target for signature detection.

21 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

1️⃣ Encoder 🔄

Generating a shelldode using the x64/zutto_dekiru encoder, which incorporates many techniques from shikata_ga_nai.

However, the standard use of the encoder will also be reported, once again detecting the decoder or template signature. It's important to know that when msfvenom generates an executable, it inserts the shellcode into a valid executable.

This template executable is static and likely also has signatures. We can use the -x option to specify a different template. To do this, we will copy the Notepad application located in C:\Windows\System32\notepad.exe onto Kali and use it as a template as follows�

�

22 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

1️⃣ Encoder 🔄

Scan with ClamAV�

�

Based on the results above, ClamAV did not detect the shellcode.

23 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

1️⃣ Encodeur 🔄

Shellcode undetectable by ClamAV. But what about other AVs?

Out of 48 AVs, 6 detected our shellcode. This is a satisfactory score, especially since some AVs performed heuristic analysis.

24 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

2️⃣ Encryption 🔐

Encrypted payload (AES, RC4, etc.)

Custom key

Runtime decryption

Limitation:

Decryption routine

25 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

2️⃣ Encryption 🔐

Leveraging the power of aes256 encryption, we will generate an executable with aes256 encrypted shellcode and use a custom encryption key via the --encrypt-key option (List 24).

26 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

2️⃣ Encryption 🔐

Scan with ClamAV

Based on the results above, ClamAV did not detect the shellcode.

27 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with Metasploit

2️⃣ Encryption 🔐

Shellcode undetectable by ClamAV. But what about other AVS? �Out of 48 AVS, 11 detected our shellcode. This is quite high compared to the encoding. ��The encryption is effective, but the decryption routine itself remains detectable because it is static.

28 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Win32 APIs

Win32 APIs (or "Windows APIs") are the set of native interfaces provided by Microsoft for programming Windows at the system level: creating windows, managing files, threads, networking, security, GUIs, drawing, etc. They form the basis of "classic" Windows applications written in C/C++ (before .NET). They expose C functions exported by system DLLs (kernel32.dll, user32.dll, gdi32.dll, etc.).

29 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Typical Approach

The goal of this technique is to use custom code written in C# to bypass standard defenses and execute a payload (the shellcode) directly in RAM.

The process is broken down into three main steps, using three key functions of the Windows API (Win32 API) from Kernel32.dll:

VirtualAlloc Memory Allocation: Find and reserve memory space with special permissions.

RtlMoveMemory Copying the shellcode: Place the shellcode bytes into this memory space.

CreateThread Execution: Start a new thread at the shellcode address.

30 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Translation of the Win32 function into C#

Win32 APIs are written in C.

To call them in C#, you must translate their C prototype into a compatible C# signature. `P/invoke` (http://www.pinvoke.net)

You use the `[DllImport()]` attribute to tell the .NET runtime which DLL and function to call.

31 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Simple Shellcode in C#

32 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Simple Shellcode in C#

Scan with ClamAV

Based on the results above, ClamAV did not detect the shellcode.

33 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

2️⃣ Simple Shellcode in C# 🔐

Shellcode undetectable by ClamAV. But what about other AVS?Out of 48 AVS, 9 detected our shellcode.

This is slightly better compared to the Metasploit encryptor. The encryption is effective, but the decryption routine itself remains detectable because it is static.

34 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Shellcode : AES Encryption

Cyberchef

35 of 43

Limits of evasion by signature�

36 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Develop Custom Tools

Why is it more effective?

No existing signature
No known pattern in the framework
Reduced analysis area

⚠️ But:

Significant development time
Requires rigorous testing
Continuous maintenance is necessary

37 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Develop Custom Tools

Everyone knows that to perform a reverse shell, you can use the nc.exe tool followed by your IP address and port. But as we all know, this tool is flagged by several antivirus programs.

We will code a custom tool that behaves like nc.exe (a TCP reverse shell module), which will require no obfuscation or encryption since it doesn't use any known malicious components present in antivirus solution signature databases.

38 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Develop Custom Tools

39 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Develop Custom Tools

As you can see, no AV was able to detect it, so we obtain a score of 0/48

40 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Develop Custom Tools

As you can see, no AV was able to detect it, so we obtain a score of 0/48

41 of 43

Process of bypassing antivirus signature detection

Bypassing antivirus software with C#

Develop Custom Tools

When running the program, we get an interactive PowerShell session interface, and we weren't blocked by Microsoft Defender Antivirus because we didn't use anything known to be malicious. Even though it decides to perform a memory scan of our process when powershell.exe is running, since it's launched as a child process, nothing should be detected because we won't be comparing it to any known signatures.

1 of 43

2 of 43

3 of 43

4 of 43

5 of 43

6 of 43

7 of 43

8 of 43

9 of 43

10 of 43

11 of 43

12 of 43

13 of 43

14 of 43

15 of 43

16 of 43

17 of 43

18 of 43

19 of 43

20 of 43

21 of 43

22 of 43

23 of 43

24 of 43

25 of 43

26 of 43

27 of 43

28 of 43

29 of 43

30 of 43

31 of 43

32 of 43

33 of 43

34 of 43

35 of 43

36 of 43

37 of 43

38 of 43

39 of 43

40 of 43

41 of 43

42 of 43

43 of 43