Unit 3: Disassembly and Malware Debugging

I. Orientation — From Machine Code to Program Behaviour

Disassembly and debugging are complementary methods for understanding compiled malware without source code. Disassembly translates machine-code bytes into assembly instructions, while debugging executes those instructions under observation. Both methods should be performed in an isolated laboratory because a sample may alter files, contact remote systems, evade analysis, or damage the host.

• Governing principle: Static analysis predicts what a binary can do; dynamic debugging observes what it actually does during a particular execution.
• Core representation: On 32-bit and 64-bit Windows, analysts commonly inspect x86 or x86-64 instructions, registers, memory, stack frames, control flow, and calls into the Windows API.
• Executable format: A Portable Executable (PE) file contains structures such as DOS and PE headers, sections, an entry point, imports, exports, resources, and relocation data.
• Execution context: Behaviour depends on architecture, privileges, command-line arguments, environment variables, files, registry state, network availability, and process history.
• Laboratory convention: Use disposable virtual machines, snapshots, host-only or simulated networking, non-production credentials, and controlled transfer mechanisms.
• Evidence convention: Preserve the original sample, calculate hashes such as SHA-256, work from a copy, record tool versions, and document every modification made during analysis.
• Analytical limitation: Packing, encryption, obfuscation, anti-debugging, indirect calls, and self-modifying code can make displayed instructions differ from the code eventually executed.

II. Static Code Analysis — Reasoning Without Execution

A. Static code analysis

Static code analysis examines a binary’s structure, data, and instructions without intentionally running it, allowing broad behavioural hypotheses to be formed with comparatively low execution risk.

• Initial triage: Hashes, file type, architecture, PE timestamps, section names, section permissions, entropy, digital signatures, and embedded resources establish the sample’s basic identity.
  • A section that is writable and executable, or has unusually high entropy near 8.0 bits per byte, may contain packed or encrypted content.
• Strings: ASCII and UTF-16 strings can expose URLs, file paths, registry keys, commands, mutex names, error messages, and campaign identifiers.
  • Cross-references to "Software\\Microsoft\\Windows\\CurrentVersion\\Run" may indicate investigation of persistence logic.
• Imports: The Import Address Table identifies external functions expected by the program. CreateProcessW, RegSetValueExW, and WinHttpSendRequest suggest process creation, registry modification, and HTTP communication respectively.
• Instruction semantics: Analysts trace data movement, arithmetic, comparisons, calls, and branches. For example:

cmp eax, 0
jne success_path

• EAX is a general-purpose register often used for return values; JNE transfers control when the zero flag is clear, so the branch is taken when EAX != 0.
  • Control-flow graph: Basic blocks end at branches, calls, returns, or exceptions; directed edges show possible transfers between blocks. Loops and strongly connected regions often reveal decoding or repeated processing.
  • Data-flow reasoning: Follow values from sources to sinks, such as a filename passed from GetTempPathW to CreateFileW, rather than assigning meaning from an isolated API name.
  • Calling conventions: Parameter placement and stack cleanup depend on architecture and convention. Windows x64 normally passes the first four integer or pointer arguments in RCX, RDX, R8, and R9, with additional arguments on the stack.
  • Decompilation: Pseudocode improves readability but is an inference, not original source. Types, variable names, loop structures, and signedness may be reconstructed incorrectly.
  • Applications and limitations: Static analysis gives wide code coverage and supports detection signatures, but unreachable code, runtime-resolved APIs, packing, and environment-dependent paths require dynamic confirmation.

III. Windows API Disassembly — Reconstructing Operating-System Interactions

A. Disassembling Windows API

Disassembling Windows API usage means identifying imported or dynamically resolved functions, reconstructing their arguments, and interpreting each call in the context of surrounding control and data flow.

• Import mechanisms: A PE may call an address stored in its Import Address Table, use a thunk, or resolve a function at runtime through LoadLibraryW and GetProcAddress.
• Indirect calls: An instruction such as call qword ptr [rip+offset] on x64 commonly targets an imported function through an address calculated relative to the instruction pointer.
• API contracts: Correct interpretation requires the documented prototype, parameter types, return value, and error convention. Consider:

HANDLE CreateFileW(
  LPCWSTR name, DWORD access, DWORD share,
  LPSECURITY_ATTRIBUTES security,
  DWORD creation, DWORD flags, HANDLE templateFile
);

• name points to a UTF-16 path; access may include GENERIC_READ or GENERIC_WRITE; the return value is a handle or INVALID_HANDLE_VALUE on failure.
  • Argument recovery: On Windows x64, inspect RCX, RDX, R8, and R9 immediately before the call, then examine stack arguments. Earlier register assignments may be overwritten by intervening instructions.
  • Return-value checks: Code often compares RAX or EAX after a call. A subsequent GetLastError is meaningful only where the called API documents that extended error information is available.
  • API families: File APIs reveal collection or deployment; registry APIs reveal configuration or persistence; process and thread APIs reveal execution; Winsock or WinHTTP APIs reveal communication; cryptographic APIs reveal hashing, encryption, or key operations.
  • Unicode and ANSI forms: A suffix of W denotes UTF-16 parameters, while A denotes the Windows code-page form. Misreading a W string as bytes can hide paths and command lines.
  • Behavioural chains: A sequence is stronger evidence than a single call. OpenProcess followed by VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread is consistent with remote-process injection, subject to argument validation.
  • Wrapper functions: Malware may call a local function that transforms parameters before invoking an API. Rename wrappers by demonstrated behaviour only after tracing their inputs, outputs, and call sites.
  • Applications and limitations: API reconstruction converts low-level instructions into operating-system actions, but direct system calls, API hashing, forwarded exports, hooks, and undocumented interfaces can obscure the mapping.

IV. General Debugging Concepts — Controlled Observation of Execution

A. General concepts of debugging

Debugging controls a running program so that execution state can be paused, inspected, and altered at selected events, enabling hypotheses from static analysis to be tested.

• Debugger model: A debugger attaches to or launches a process, receives debug events, and exposes threads, registers, virtual memory, modules, exceptions, and instruction flow.
• Breakpoints: Software breakpoints typically replace an instruction byte with INT 3 (0xCC); hardware breakpoints use debug registers and can stop on execution, reads, or writes without patching code.
• Stepping: Step-into follows a call into the callee; step-over executes the call and pauses at the next instruction; step-out continues until the current function returns.
• Register state: RIP identifies the next x64 instruction, RSP points to the stack, RBP may anchor a stack frame, and RFLAGS stores condition flags used by branches.
• Memory model: Each process has virtual address regions with permissions such as read, write, and execute. The debugger maps addresses to loaded modules, heaps, stacks, and dynamically allocated pages.
• Call stack: Return addresses and unwind information reconstruct nested calls, although stack corruption, tail calls, omitted frame pointers, and hand-written assembly can reduce reliability.
• Exceptions: Access violations, illegal instructions, breakpoint events, and single-step events may belong to normal program logic or indicate failure. First-chance exceptions reach the debugger before application handlers.
• Address relocation: Address Space Layout Randomization changes module bases. Prefer module-relative locations such as sample.exe+0x1234 instead of assuming a fixed absolute address.
• Anti-debugging: Checks involving IsDebuggerPresent, timing differences, process information, exception behaviour, or breakpoint-byte inspection may alter the observed path.
• Analytical discipline: Change one condition at a time, log breakpoint addresses and observations, and distinguish observed facts from interpretations. Debugger modifications can invalidate later conclusions.
• Applications and limitations: Debugging provides precise state at a chosen moment, but one run covers only one path and may differ from execution outside the laboratory.

V. Debugging Binaries — A Repeatable Malware Workflow

A. Debugging binaries

Debugging binaries applies breakpoint, stepping, and memory-inspection techniques to compiled executables while containing risk and preserving a reproducible record of observed behaviour.

• Preparation: Verify the sample hash, identify architecture, snapshot the analysis VM, disable shared folders and clipboard where appropriate, and configure monitored or simulated network services.
• Entry strategy: Begin at the PE entry point for unpacked code, at a relevant imported API, or at a statically identified function. Stopping too early can bury the investigation in loader activity.
• Breakpoint selection: Break on calls tied to the hypothesis, such as file creation, process launch, registry updates, memory protection changes, or network connection attempts.
• Parameter inspection: At a breakpoint, capture the call site, thread, arguments, pointed-to buffers, and return address. After stepping over the call, record the return value and resulting memory or handle.
• Unpacking indicators: Packed code often allocates or modifies memory, writes decoded bytes, changes protection to executable, and transfers control into the new region.
  • A useful observation point is the transition from a small unpacking loop to a region containing coherent instructions, strings, and API calls.
• Conditional breakpoints: Conditions reduce noise by stopping only when an argument matches a target value, a specific thread executes, or an address falls within the sample’s module.
• Memory breakpoints: A write watchpoint on a decoded buffer reveals the producing instruction; an execution watchpoint on a newly written region can identify transfer to unpacked code.
• Control-flow validation: Compare executed branches with the static control-flow graph. A branch skipped because a file, locale, privilege, or network response is absent identifies an environmental dependency.
• Safe modification: Patching a conditional jump or return value can reveal a hidden path, but the resulting behaviour must be labelled as forced rather than naturally observed.
• Evidence capture: Record module-relative addresses, register values, argument data, memory dumps, created artefacts, network indicators, and the exact VM state. Revert to a clean snapshot before independent runs.
• Completion criteria: Stop when the investigation answers its behavioural question, such as locating decoded configuration, identifying persistence, or explaining a process launch; unrestricted execution adds risk without necessarily adding evidence.
• Applications and limitations: Binary debugging can expose runtime-only code and decrypted data, yet race conditions, kernel components, anti-VM logic, multi-process behaviour, and external dependencies may require additional monitoring or specialized debuggers.