Unit 5: Obfuscation and Malware Forensics

I. Foundations of Obfuscation and Malware Forensics

Malware obfuscation conceals code, data, or behavior to delay detection and analysis, while malware forensics reconstructs that behavior from evidence such as executable files, volatile memory, registry data, processes, and network artifacts.

• Obfuscation objective: Malware may hide strings, configuration data, imported functions, control flow, or its complete executable image without changing its intended behavior.
• Forensic objective: Analysts identify what executed, how it persisted, which resources it accessed, and whether it communicated with other systems.
• Static evidence: Executables, DLLs, scripts, registry hive files, logs, and disk images can be examined without running the malware.
• Volatile evidence: RAM can contain decrypted payloads, active processes, injected code, network connections, encryption keys, command history, and unpacked executables.
• Evidence integrity: Acquired images should be hashed with algorithms such as SHA-256, preserved read-only, and analyzed on verified copies.
• Safe-analysis convention: Work is performed in an isolated laboratory using snapshots, restricted networking, controlled tools, and documented timestamps.
• Analytical principle: No single artifact proves maliciousness; conclusions should correlate process, memory, file, registry, service, and network evidence.

II. Data Obfuscation Mechanisms — Concealing Malware Content

A. Simple encoding

Simple encoding changes the representation of data but does not provide cryptographic security.

• Common forms: Base64, hexadecimal, URL encoding, ROT substitutions, and single-byte XOR are frequently used to conceal strings or scripts.
  • Base64 commonly uses the alphabet A–Z, a–z, 0–9, +, and /, with optional = padding.
  • Hexadecimal represents each byte as two characters; byte 4D represents ASCII M.
• Recognition clues: Repeated character sets, predictable padding, readable output after decoding, or constants embedded near a decoding loop suggest encoding.
• XOR operation: A byte is transformed with a key and recovered by applying the same key again.

C = P XOR K
P = C XOR K

Here, P is the plaintext byte, K is the key byte, and C is the encoded byte.

• Analytical method: Locate candidate strings, identify the decoding operation, recover the key or alphabet, and decode a copy of the data.
• Limitation: Encoding hides content from basic string searches but offers little protection once the transformation is identified.

B. Malware encryption

Malware encryption uses a key-based cipher to protect payloads, configurations, communications, or stolen information.

• Typical uses: Malware may encrypt an embedded second-stage payload, command-and-control configuration, ransomware targets, or exfiltrated archives.
• Cipher types:
  1. Symmetric encryption: AES, RC4, ChaCha, or custom stream ciphers use the same secret for encryption and decryption.
  2. Asymmetric encryption: RSA or elliptic-curve cryptography uses public and private keys, often to protect a symmetric session key.
• Investigation clues: High-entropy data, cryptographic constants, key-scheduling loops, or calls to APIs such as Windows CNG and CryptoAPI indicate encryption.
• Key recovery: Keys may be hard-coded, derived from host values, downloaded from a server, or temporarily present in process memory.
• Important distinction: High entropy can indicate encryption, compression, or packing; entropy alone does not establish which mechanism was used.

C. Custom encoding

Custom encoding is a malware-specific transformation designed to avoid signatures and standard decoding tools.

• Possible design: A routine may combine XOR, addition, rotation, byte swapping, substitution tables, and changing keys.
• Reversibility: Analysts reverse each operation in the opposite order; for example, subtraction reverses addition and right rotation reverses left rotation.
• Code indicators: Tight loops, repeated bitwise instructions, lookup tables, and a buffer that becomes readable after a function returns are useful anchors.
• Recovery approach:
  • Identify input and output buffers.
  • Trace key initialization and loop boundaries.
  • Reimplement the routine in a short script or capture output dynamically.
• Risk of misinterpretation: A custom algorithm may be lossy hashing rather than reversible encoding, so analysts must verify that decoded output reproduces meaningful bytes.

III. Packed Malware — Recovering Hidden Executable Code

A. Malware unpacking

Malware unpacking reconstructs the original executable image after a loader has decompressed, decrypted, or relocated it in memory.

• Packing indicators: A small import table, unusual section names, high-entropy sections, an entry point outside normal code, and writable-executable memory are common signs.
• Static unpacking: Known packers may be handled with dedicated unpackers or by reversing the packing algorithm without executing the sample.
• Dynamic unpacking: The sample runs in an isolated debugger until its unpacking stub transfers execution to the original entry point, or OEP.
• Typical workflow:
  1. Record the original hash, headers, sections, and imports.
  2. Trace memory allocation and permission changes such as VirtualAlloc and VirtualProtect.
  3. Stop after the payload has been written and control reaches unpacked code.
  4. Dump the reconstructed image and repair its import address table when necessary.
• Validation: The dumped file should have coherent PE headers, executable sections, meaningful imports, and disassemblable control flow.
• Limitation: Anti-debugging, process hollowing, multiple packing layers, and environment checks can prevent a straightforward OEP-based dump.

IV. Volatile-Memory Investigation — Acquisition and Framework

A. Memory forensics steps

Memory forensics follows a repeatable sequence from preservation to correlated interpretation.

• Preparation: Isolate the suspected system according to incident procedures while avoiding unnecessary shutdown, logon, or application activity.
• Acquisition: Capture RAM, record system time and tool details, then calculate a SHA-256 hash of the resulting image.
• Triage: Determine the operating-system family and inspect processes, command lines, DLLs, handles, services, registry artifacts, and connections.
• Deep analysis: Search for injected memory, hidden processes, unpacked payloads, credentials, encryption material, and suspicious kernel objects.
• Correlation: Connect a PID to its parent, executable path, loaded modules, open files, sockets, registry persistence, and timestamps.
• Reporting: Preserve commands, plugin versions, hashes, offsets, PID values, and extracted-file hashes so findings are reproducible.

B. Memory acquisition

Memory acquisition creates a bit-oriented representation of physical memory while minimizing changes to the live system.

• Windows tools: WinPmem, Magnet RAM Capture, and FTK Imager can acquire physical memory with appropriate administrative access.
• Linux tool: LiME, the Linux Memory Extractor, can capture memory to a file or transmit it over a controlled network connection.
• Order of volatility: RAM should generally be collected before less volatile disk artifacts because shutdown destroys active-memory evidence.
• Contamination control: Every acquisition utility changes memory slightly; analysts therefore document the tool, invocation, destination, start time, and completion time.
• Preservation: Store the original capture securely, compute a cryptographic hash, and perform examination on a verified working copy.

sha256sum memory.raw

C. Volatility overview

Volatility is an open-source framework that interprets operating-system structures contained in memory images.

• Plugin model: Specialized plugins enumerate processes, modules, network objects, registry keys, services, and other artifacts.
• Version distinction: Volatility 2 commonly requires a selected Windows profile, whereas Volatility 3 uses symbol tables and automated layer construction.
• Basic syntax:

python3 vol.py -f memory.raw windows.info
python3 vol.py -f memory.raw windows.pslist

• Interpretive caution: Plugin output depends on valid structures and suitable symbols; overwritten, paged-out, or deliberately unlinked objects may be incomplete.
• Good practice: Confirm suspicious results through multiple plugins, such as comparing active-list enumeration with memory-pool scanning.

V. Process and Module Examination — Mapping Runtime Activity

A. Enumerating processes

Process enumeration identifies active, terminated, hidden, or anomalously related processes.

• Core plugins: windows.pslist follows the active process list, windows.psscan scans memory for process objects, and windows.pstree displays parent-child relationships.
• Comparison value: A process found by psscan but absent from pslist may be terminated, unlinked, or concealed by a rootkit.
• Suspicion indicators: Unexpected parentage, impossible start times, misspelled system names, unusual paths, or duplicate system processes deserve examination.
• Concrete command:

python3 vol.py -f memory.raw windows.pstree

B. Listing process handles

Process handles reveal which kernel-managed resources a process has opened.

• Artifact types: Handles may reference files, registry keys, processes, threads, sections, events, mutexes, tokens, and named pipes.
• Analytical value: A process holding handles to another process may be injecting code, while a ransomware process may expose handles to numerous user documents.
• Command:

python3 vol.py -f memory.raw windows.handles --pid 2480

• Correlation: Handle values should be assessed with process identity, access rights, object name, and surrounding behavior rather than treated as standalone proof.

C. Dumping executable and DLL

Dumping recovers process images or mapped DLL data for hashing, string extraction, and reverse engineering.

• Executable recovery: Volatility 2 commonly uses procdump, while Volatility 3 plugins may support dumping through process or file-oriented options.
• DLL recovery: dlldump in Volatility 2 or dumping support associated with Volatility 3 module listings can recover mapped libraries.
• Example commands:

vol.py -f memory.raw --profile=<profile> procdump -p 2480 -D output/
vol.py -f memory.raw --profile=<profile> dlldump -p 2480 -D output/

• Caution: A memory dump may not match the on-disk PE layout because pages can be missing, modified, relocated, or unpacked only in memory.
• Preservation: Hash each recovered artifact and analyze it in an isolated environment.

D. Listing DLLs

DLL listing identifies modules mapped into each process and exposes unusual or injected libraries.

• Plugin: windows.dlllist reports module names, paths, base addresses, and sizes for a selected process.
• Command:

python3 vol.py -f memory.raw windows.dlllist --pid 2480

• Suspicious features: Temporary-directory paths, unsigned modules, name masquerading, unexpected network libraries, and modules absent from standard loader lists require validation.
• Limitation: Reflectively loaded or manually mapped DLLs may not appear in conventional lists, requiring memory-map or injection analysis.

VI. Network Artifacts — Identifying External Communication

A. Listing network connections and sockets

Network enumeration connects processes with local endpoints, remote systems, protocols, and connection states.

• Plugin: windows.netscan scans for TCP endpoints, TCP listeners, and UDP endpoints in supported Windows memory images.
• Command:

python3 vol.py -f memory.raw windows.netscan

• Fields: Local address, local port, remote address, remote port, state, PID, owner, and creation time help reconstruct communications.
• Indicators: Rare destination ports, repeated outbound connections, unexpected listeners, or a document process owning a socket can indicate compromise.
• Caution: Closed or reused structures may remain in memory, so timestamps and process lifetime must be correlated before attributing activity.

VII. Persistence Artifacts — Registry and Services

A. Inspecting registry

Registry inspection recovers configuration, execution, persistence, and user-activity evidence from memory-resident hive data.

• Plugins: windows.registry.hivelist identifies hives, while windows.registry.printkey displays selected keys and values.
• Persistence locations: Run keys, service configuration, Winlogon settings, and shell-related entries are frequent investigation targets.
• Command pattern:

python3 vol.py -f memory.raw windows.registry.hivelist
python3 vol.py -f memory.raw windows.registry.printkey --key "Software\Microsoft\Windows\CurrentVersion\Run"

• Interpretation: Examine value data, executable paths, quoting, user scope, timestamps where available, and links to observed processes.

B. Investigating service

Service investigation determines whether malware installed or modified a Windows service for persistence or privileged execution.

• Plugin: windows.svcscan scans memory for service records and associated configuration.
• Key fields: Service name, display name, state, start type, executable path, and service account provide context.
• Indicators: Random names, auto-start configuration, binaries in user-writable directories, or misleading descriptions are suspicious.
• Correlation: Confirm that the service path matches a recovered executable, registry service key, process, and relevant network activity.

VIII. Interactive Activity — Recovering Operator Commands

A. Extracting command history

Command-history extraction can reveal commands entered by an attacker, administrator, script, or malware-controlled shell.

• Volatility 2 plugins: cmdscan searches command-history structures, while consoles can recover console buffers, attached processes, and command output.
• Commands:

vol.py -f memory.raw --profile=<profile> cmdscan
vol.py -f memory.raw --profile=<profile> consoles

• Evidence value: Recovered entries may expose reconnaissance commands, account creation, file deletion, archive creation, or tool execution.
• Context requirement: A command string does not prove successful execution; correlate it with process creation, files, registry changes, logs, and network artifacts.
• Coverage limitation: PowerShell, remote shells, redirected consoles, overwritten buffers, and non-console execution may require process command-line and memory-string analysis.