Unit 2: Dynamic Analysis and Assembly Language

I. Orientation

Dynamic malware analysis examines a suspicious program while it executes in a controlled environment. Assembly language provides the low-level vocabulary needed to interpret that execution: instructions manipulate registers, memory, flags, and control flow, ultimately exposing malware behavior that may be hidden in source-free binaries.

• Governing principle: Observe effects rather than trusting appearance; filenames, icons, and embedded strings can be misleading, but actions such as creating C:\ProgramData\svc.exe or connecting to an IP address provide behavioral evidence.
• Controlled execution: Run samples only in an isolated virtual machine or sandbox with snapshots, restricted networking, and no access to production credentials or shared folders.
• Behavioral evidence: Record processes, files, registry entries, services, mutexes, API calls, network traffic, memory changes, and child-process relationships.
• Machine-level interpretation: Read disassembly using an architecture and syntax convention, commonly x86-64 with Intel syntax: destination, source.
• Execution state: At any instant, program behavior is determined by register values, memory contents, status flags, the instruction pointer, and operating-system state.
• Analytical limitation: A single execution reveals only the path taken for the supplied environment and inputs; dormant branches, delayed behavior, or anti-analysis checks may remain unseen.

II. Dynamic Malware Analysis — Observing Runtime Behavior

Dynamic analysis executes malware under instrumentation to identify its capabilities, persistence methods, communications, and effects on the host.

A. Dynamic analysis steps

Dynamic analysis follows a controlled sequence that preserves evidence and makes observations reproducible.

• 1. Prepare the laboratory: Create an isolated VM, disable host integration, take a clean snapshot, synchronize analysis-tool timestamps, and configure simulated services where appropriate.
• 2. Establish a baseline: Record running processes, listening ports, autorun locations, registry state, and selected filesystem hashes before execution.
• 3. Perform initial triage: Calculate hashes such as SHA-256, identify the file type, inspect imports and strings, and note required arguments without double-clicking the sample.
• 4. Start monitoring: Activate process, filesystem, registry, API, packet-capture, and debugging tools before launching the sample.
• 5. Execute deliberately: Supply expected input, document the exact command line, and interact with dialogs or decoy documents only inside the laboratory.
• 6. Stimulate behavior: Reboot the VM, alter the date, open created files, or emulate DNS and HTTP services to reveal persistence and network-dependent branches.
• 7. Capture artifacts: Preserve process trees, packet captures, dropped files, memory dumps, registry changes, screenshots, and tool logs with timestamps.
• 8. Compare and restore: Diff the final state against the baseline, summarize indicators and capabilities, then revert the VM snapshot.

B. Analysing malware

Malware analysis converts observed events into defensible conclusions about intent, capability, and impact.

• Process behavior: Identify parent-child relationships, command-line arguments, privilege changes, process injection, and unexpected launches such as a document reader spawning powershell.exe.
• Filesystem behavior: Track created, modified, deleted, and renamed files; hash dropped payloads and distinguish configuration data from executable content.
• Persistence: Examine services, scheduled tasks, startup folders, registry run keys, browser extensions, and WMI subscriptions.
• Registry activity: Interpret keys in context; a write to a Run key suggests autostart, whereas ordinary preference writes may be incidental.
• Network behavior: Record DNS requests, destination addresses, ports, TLS metadata, HTTP headers, beacon intervals, and transmitted content.
• Memory behavior: Search process memory for unpacked code, decrypted configuration, injected regions, network indicators, and credentials.
• Correlation: Treat repeated evidence as stronger than isolated events; an executable dropped to disk, registered as a service, and launched after reboot supports a persistence finding.
• Evasion awareness: Check for long sleeps, VM artifacts, debugger detection, user-activity checks, and environment-dependent execution.

C. DLL analysis

DLL analysis investigates library code that is loaded into another process and may lack an independent executable entry point.

• Loading model: Windows maps a Portable Executable DLL into a process and invokes its initialization routine with events such as process attach and detach.
• Exports: Inspect the export table for named or ordinal functions; a loader may invoke an export through rundll32.exe, a custom harness, or the program that normally imports it.
• Imports: APIs such as WinHttpOpen, CreateProcessW, or RegSetValueExW indicate possible capabilities, although imports alone do not prove behavior.
• Entry conditions: Supply the correct architecture, arguments, exported function, and host environment; otherwise the meaningful code path may not execute.
• Monitoring: Attribute activity to both the host process and loaded module because the process name alone may conceal which DLL generated an event.
• DLL search-order abuse: Compare the loaded module path with the expected trusted path; malware may exploit an application that searches a writable directory first.
• Safety constraint: Use a purpose-built isolated harness where possible, because system utilities may introduce unrelated behavior or expose the host if isolation fails.

D. Applications and limitations

Dynamic evidence is strongest when its scope and constraints are stated explicitly.

• Applications: Runtime analysis supports incident response, indicator extraction, capability assessment, detection engineering, configuration recovery, and unpacking.
• Path coverage: One run explores only one set of branches; vary arguments, files, privileges, locale, network responses, and elapsed time when justified.
• Environmental distortion: Sandboxes can differ from real endpoints in installed software, user history, domain membership, hardware, and network access.
• Attribution limit: A behavior may identify a malware family or technique, but it rarely proves who operated the sample.
• Combined method: Validate dynamic findings with static disassembly and memory analysis; each method exposes evidence the others can miss.

III. Assembly Language — Interpreting Machine-Level Execution

Assembly language is a symbolic representation of processor instructions. These notes use Intel-style x86/x64 notation, where mov rax, rbx copies the value in RBX into RAX.

A. Introduction to assembly language basics

Assembly connects encoded machine instructions to operations on registers, memory, and control flow.

• Instruction form: An instruction usually contains a mnemonic and operands, as in mov eax, 5; the mnemonic is mov, the destination is EAX, and the source is 5.
• Operand types: Operands may be immediate constants, registers, or memory locations such as [rbp-8].
• Data size: byte, word, dword, and qword denote 8, 16, 32, and 64 bits respectively.
• Endianness: x86 stores multi-byte integers in little-endian order; 0x12345678 appears in memory as 78 56 34 12.
• Disassembly: A disassembler translates bytes into probable instructions, but code and data can be confused, especially in packed or obfuscated binaries.

B. Registers

Registers are small, fast processor storage locations used for data, addresses, arguments, and execution control.

• General-purpose registers: x64 provides RAX, RBX, RCX, RDX, RSI, RDI, RBP, RSP, and R8–R15.
• Partial registers: RAX is 64-bit; EAX, AX, AH, and AL expose overlapping 32-, 16-, and 8-bit portions.
• Instruction pointer: RIP identifies the next instruction; relative addressing often uses RIP as a base.
• Stack pointer: RSP points to the top of the stack and changes during calls, returns, pushes, and local allocation.
• Flags: RFLAGS contains condition bits including zero (ZF), carry (CF), sign (SF), and overflow (OF).

C. Data transfer instructions

Data transfer instructions copy values without inherently changing the source operand.

• mov: mov rax, rbx copies RBX into RAX; most forms do not permit both operands to be memory locations.
• lea: lea rax, [rbx+rcx*4] calculates an address or arithmetic expression without reading that memory.
• Stack transfer: push rax decreases RSP and stores RAX; pop rax loads from the stack and increases RSP.
• Extension: movzx zero-extends a smaller value, while movsx sign-extends it, preserving a signed interpretation.

D. Arithmetic operations

Arithmetic instructions transform integer values and usually update status flags.

• Addition and subtraction: add eax, 3 increases EAX; sub eax, 3 decreases it.
• Increment and decrement: inc ecx and dec ecx change a value by one, although inc and dec do not update CF.
• Multiplication and division: imul supports signed multiplication; division may use a register pair such as RDX:RAX as the dividend.
• Comparison: cmp eax, ebx performs a conceptual subtraction, discards the result, and updates flags for a later conditional jump.

E. Bitwise operations

Bitwise instructions manipulate individual bits and are common in masks, flags, encoding, and obfuscation.

• Logical operations: and eax, 0xFF retains the low byte; or eax, 1 sets bit 0; xor eax, eax clears EAX; not eax inverts every bit.
• Testing: test eax, eax computes an AND only for flags, efficiently checking whether EAX is zero.
• Shifts: shl eax, 1 approximately multiplies an unsigned value by two; shr inserts zeros, while sar preserves the sign bit.
• Rotations: rol and ror move shifted-out bits to the opposite end and frequently appear in hash or decoding routines.

F. Branching and conditionals

Branches change RIP, allowing execution to select paths according to flags or computed targets.

• Unconditional branch: jmp target always transfers control to target.
• Equality branches: After cmp, je branches when ZF=1, while jne branches when ZF=0.
• Signed comparison: Instructions such as jg and jl interpret SF and OF for signed integers.
• Unsigned comparison: Instructions such as ja and jb use CF and ZF.
• Reconstruction:

cmp eax, 10
jle small

This represents the condition “branch to small when signed EAX <= 10.”

G. Loops and functions

Loops repeat control flow, while functions package reusable behavior behind calling conventions.

• Loop structure: A typical loop initializes a counter, checks a terminating condition, executes a body, updates the counter, and jumps backward.
• Calls and returns: call target stores a return address and transfers control; ret retrieves that address.
• Stack frame: A function may preserve RBP, allocate local storage by subtracting from RSP, and restore the stack before returning.
• Calling convention: The platform ABI defines argument registers, return registers, stack alignment, and which registers a callee must preserve.
• Analytical clue: Repeated calls inside a backward branch may indicate byte-by-byte decoding, hashing, copying, or string comparison.

H. Arrays and strings

Arrays and strings occupy contiguous memory, so element access combines a base address, index, and element size.

• Address calculation: For element size s, address = base + index × s; [rax+rcx*4] accesses a four-byte element.
• Strings: C-style strings end with byte 0x00; Windows wide strings commonly use two-byte UTF-16 code units.
• Traversal: A loop may load one byte, compare it with zero, process it, increment the pointer, and continue.
• String instructions: rep movsb copies RCX bytes using source and destination index registers under the applicable x64 convention.

I. Structures

Structures group fields at fixed offsets from a common base address.

• Field access: If RBX points to a structure, [rbx+8] accesses the field located eight bytes from its beginning.
• Layout: Field size, alignment, and padding determine offsets; a compiler may insert unused bytes before an aligned field.
• Reverse engineering: Repeated accesses such as [rcx], [rcx+8], and [rcx+10h] suggest fields of one object passed through RCX.
• Interpretation: Assign field meanings only after correlating offsets with API use, value patterns, and accesses across multiple functions.

J. x64 architecture

x64 extends x86 with 64-bit addresses, additional registers, and platform-specific calling conventions.

• Address width: Registers are 64 bits, although implementations use a canonical subset of the theoretical 64-bit address space.
• Register behavior: Writing to a 32-bit register zeroes its upper half; mov eax, 1 makes RAX equal to 1.
• Windows ABI: The first four integer or pointer arguments normally use RCX, RDX, R8, and R9; return values commonly use RAX.
• System V ABI: Many Unix-like systems pass the first integer arguments in RDI, RSI, RDX, RCX, R8, and R9.
• Stack discipline: Correct analysis requires recognizing alignment, saved registers, return addresses, local variables, and Windows shadow space.
• Security features: Address-space layout randomization, data-execution prevention, stack protection, and control-flow safeguards affect how malicious code locates data and transfers execution.