Unit 6: Advanced Malware Detection Using Memory Forensics

I. Memory Forensics Foundations

Memory forensics examines a captured image of volatile physical memory to reconstruct processes, kernel objects, executable code, and transient attacker activity. Its governing principle is cross-view validation: evidence obtained through normal operating-system lists is compared with evidence recovered by scanning memory structures, page tables, and executable regions.

• Primary evidence: RAM contains process address spaces, loaded modules, kernel objects, network state, callbacks, timers, and remnants of terminated activity.
• Acquisition assumptions: The image should be collected with minimal system disturbance, hashed for integrity, and analysed using the correct operating-system version, architecture, and symbols.
• Analysis convention: A virtual address is interpreted through the owning process or kernel address space; permissions such as R, W, and X describe readable, writable, and executable pages.
• Detection principle: No single anomaly proves compromise. Findings gain strength when memory permissions, code bytes, object metadata, thread origins, and disk files independently disagree with expected state.
• Typical workflow: Establish the system profile, enumerate processes and modules, scan for hidden objects, inspect suspicious memory, correlate kernel structures, and preserve extracted artifacts for reverse engineering.
• Common tooling: Volatility 3 provides Windows memory-analysis plugins; commands vary by version, symbols, image format, and operating system.

II. User-Mode Injection — Foreign Code Inside Processes

User-mode injection places attacker-controlled code or modules in another process so execution appears to originate from a legitimate program. Analysis therefore focuses on private executable memory, inconsistent image mappings, suspicious threads, and discrepancies between memory and the executable on disk.

A. Detecting code injection

Code injection is detected by locating executable memory whose allocation, content, or execution context does not match normal process behavior.

• Memory characteristics: Shellcode commonly occupies a private Virtual Address Descriptor (VAD) with PAGE_EXECUTE_READWRITE protection rather than an image-backed mapping.
• VAD inspection: windows.malfind searches process VADs for suspicious protection and content; it may display the process, address, protection, hexadecimal bytes, and disassembly.

vol -f memory.raw windows.malfind
vol -f memory.raw windows.vadinfo --pid 2480

• Byte signatures: An MZ header in private executable memory can indicate a reflectively loaded PE file, while instructions such as a short decryption loop may indicate staged shellcode. Neither pattern is conclusive alone.
• Thread correlation: A thread start address inside an unbacked executable VAD is stronger evidence than an anomalous VAD by itself. Inspect thread objects, call stacks, and owning process identity.
• Injection mechanisms:
  1. Allocated injection: APIs such as VirtualAllocEx, WriteProcessMemory, and CreateRemoteThread produce private memory and a remotely created execution path.
  2. Mapped injection: Section mapping or reflective loading can avoid the classic API sequence and may resemble legitimate mapped content.
• False positives: JIT compilers, browsers, .NET runtimes, security software, and packed applications legitimately create executable private pages.
• Concrete validation: Dump the suspicious region, calculate its hash, identify embedded PE structures or strings, and compare the thread start address with the VAD range.

B. Investigating hollow process injection

Process hollowing replaces or unmaps the original executable image of a suspended process and transfers execution to a malicious replacement while retaining the legitimate process identity.

• Expected sequence: An attacker creates a process suspended, unmaps or overwrites its image, writes replacement code, changes the thread context, and resumes execution.
• Structural discrepancy: The Process Environment Block may name a legitimate image while the mapped executable region contains different headers, sections, entry point, or code bytes.
• Memory-to-disk comparison: Compare executable pages with the file identified by the process path. Normal relocation and import resolution alter some bytes, but wholesale section differences are suspicious.
• PE indicators: Check the in-memory MZ and PE signatures, image base, AddressOfEntryPoint, section permissions, and whether the entry point lies within the expected executable section.
• Process context: Parent process, command line, creation time, token, handle set, and network connections may expose an implausible execution chain, such as an office application spawning a hollowed system utility.
• Thread context: The primary thread instruction pointer should correspond to valid code in the declared image. Execution from a private VAD or altered image entry point supports hollowing.
• Investigation method: Use process listing and scanning together, inspect PEB and VAD mappings, dump the apparent image, then compare its imports and sections with the on-disk executable.
• Limitation: Header wiping and transacted or doppelgänging variants can remove familiar indicators, so conclusions must combine mapping, thread, and provenance evidence.

III. API Interposition — Changes to Function Resolution

API hooking redirects a function call to monitoring, compatibility, security, or malicious code. Memory analysis distinguishes legitimate instrumentation from covert hooks by identifying modified instructions or tables and resolving every target to its owning module.

A. Detecting API hooks

API-hook detection verifies whether function entry points and dispatch tables still direct execution to expected executable ranges.

• Inline hooks: The first bytes of a function may be replaced by a branch such as JMP target; compare those bytes with a trusted copy of the same module version.
• IAT hooks: An Import Address Table entry should resolve to the exporting DLL named by the import. An entry for CreateFileW that points into anonymous memory is anomalous.
• EAT hooks: An Export Address Table entry can be altered so callers resolve an exported name to malicious code; validate the target Relative Virtual Address against expected module sections.
• Target ownership: Resolve hook destinations to loaded modules and VADs. Targets in unsigned modules, private executable memory, or writable data sections deserve priority.
• Instruction validation: Disassemble enough bytes to follow trampolines and chained jumps; a hook may begin with an indirect branch rather than a simple five-byte jump.
• Paired interpretation:
  1. Legitimate hooks: Endpoint security, accessibility tools, profilers, and application shims often redirect APIs to signed, consistently loaded modules.
  2. Malicious hooks: Rootkits and credential stealers may redirect enumeration, file, registry, or authentication functions to conceal objects or capture data.
• Evidence standard: Record the original function, altered bytes or table entry, final destination, destination module, page permissions, and signer or file provenance.

IV. Kernel Object Analysis — Drivers, Devices, and I/O

Kernel malware operates with high privilege and may manipulate drivers, device objects, dispatch routines, and internal lists. Analysis relies on kernel symbols and cross-checks linked-list enumeration against pool scanning and address ownership.

A. Kernel mode rootkits

A kernel-mode rootkit modifies or abuses privileged components to conceal activity, intercept operations, or maintain control beneath user-mode monitoring.

• Capabilities: A malicious driver can access arbitrary kernel memory, filter I/O, hide processes, tamper with security controls, and communicate through device control requests.
• Loading routes: Rootkits may use a malicious signed driver, exploit a vulnerable legitimate driver, or manually map code without normal loader registration.
• Concealment: Direct Kernel Object Manipulation can unlink process or driver objects from lists while leaving pool allocations, handles, threads, or device references intact.
• Detection strategy: Compare list-based and scan-based views, map executable kernel addresses to known modules, and inspect unsigned or unbacked regions.
• Modern constraints: Driver-signing enforcement, PatchGuard, virtualization-based security, and HVCI raise the cost of kernel modification but do not eliminate vulnerable-driver abuse.
• Attribution caution: A hidden object or crashed invariant can also result from acquisition inconsistency or corruption; corroborate it with executable code and relationships.

B. Listing kernel modules

Listing kernel modules establishes the expected owners of kernel code and exposes hidden, unknown, or anomalously placed drivers.

• Loader view: windows.modules walks the normal loaded-module list and reports properties such as base address, size, path, and module name.
• Scan view: windows.modscan searches memory for module-related pool structures, potentially recovering unlinked or previously unloaded entries.

vol -f memory.raw windows.modules
vol -f memory.raw windows.modscan

• Cross-view anomaly: A module found by scanning but absent from the loader list may be hidden, stale, or partially unloaded and requires validation.
• Range validation: For each driver, calculate its interval as [base, base + size) and determine whether callbacks, dispatch pointers, and thread starts fall within it.
• Provenance checks: Examine path, timestamp, PE sections, imports, digital-signature evidence, and memory-to-disk hash differences; timestamps alone are easily forged.
• Manual mapping: Unregistered kernel code may lack a loader entry, so search executable pools and trace pointers from devices, callbacks, or system threads.

C. I/O processing

Windows I/O processing passes I/O Request Packets (IRPs) through driver stacks, creating interception points that rootkits can monitor or redirect.

• Core objects: A DRIVER_OBJECT contains a MajorFunction dispatch table; a DEVICE_OBJECT represents a device and links into an attached stack.
• IRP flow: Requests such as IRP_MJ_CREATE, IRP_MJ_READ, and IRP_MJ_DEVICE_CONTROL are delivered to the relevant dispatch routine.
• Dispatch anomaly: Each MajorFunction pointer should normally resolve inside the owning driver or an explainable framework module. A pointer into unknown executable memory suggests tampering.
• Filter behavior: Legitimate file-system, network, and security filters attach device objects to observe or modify requests; position in a stack is therefore context, not proof.
• Control channel: Malware may expose a named device and process custom IOCTL values through IRP_MJ_DEVICE_CONTROL, enabling commands from a user-mode controller.
• Concrete analysis: Correlate a suspicious device name, its driver object, dispatch addresses, owning module ranges, and any user-process handles to that device.

D. Displaying device trees

Displaying device trees reveals how driver and device objects are connected, helping identify hidden devices and unauthorized filters.

• Tree structure: Driver objects own one or more device objects; device objects may attach above or below others to form a processing stack.
• Enumeration: windows.driverscan can recover driver objects through pool scanning, while device-oriented plugins or debugger extensions expose associated devices and attachments.
• Named evidence: Unexpected names under namespaces such as \Device\ or symbolic links under \DosDevices\ can reveal a rootkit control interface.
• Stack interpretation: A previously unknown device attached to a keyboard, storage, or network stack may support keylogging, data interception, or concealment.
• Pointer checks: Validate DriverObject, AttachedDevice, and dispatch pointers against canonical kernel addresses and known module intervals.
• Limitation: Plug-and-play state changes and partially destroyed objects can leave stale structures, so active handles and valid code targets provide useful corroboration.

V. Kernel Execution Redirection — Hooks and Deferred Activity

Kernel execution can be redirected through modified tables and function pointers or scheduled through registered callbacks and timers. The central test is whether each executable pointer has a legitimate owner and expected registration context.

A. Detecting kernel space hooking

Kernel-space hook detection examines privileged dispatch paths for pointers or instructions redirected outside their expected implementation.

• Hook surfaces: Targets include driver dispatch tables, interrupt handlers, model-specific-register-controlled entry paths, and selected kernel function bodies.
• SSDT analysis: System Service Descriptor Table entries should resolve within the expected kernel code on applicable Windows versions; unexpected destinations can indicate system-call interception.
• Inline verification: Compare kernel function bytes with the matching trusted binary while accounting for relocations and legitimate runtime patching.
• Address resolution: Classify every suspect destination as belonging to a loaded module, an unlisted executable allocation, or invalid memory.
• Cross-architecture detail: Techniques differ between 32-bit and 64-bit Windows; PatchGuard makes classic SSDT modification less common on modern x64 systems.
• Strong finding: A dispatch pointer redirected from a signed driver into executable nonpaged pool, combined with an unlisted PE image there, is materially stronger than either anomaly alone.

B. Kernel callbacks and timers

Kernel callbacks and timers provide legitimate asynchronous execution but also offer malware persistent execution and monitoring opportunities.

• Callback classes: Process, thread, and image-load notifications can observe creation events; registry callbacks and object callbacks can monitor or restrict sensitive operations.
• Analysis command: windows.callbacks enumerates supported registered callback structures and attempts to resolve callback addresses to modules.

vol -f memory.raw windows.callbacks
vol -f memory.raw windows.timers

• Timer mechanics: Kernel timers expire at a due time and can queue Deferred Procedure Calls, causing a routine to execute later, often at elevated interrupt request level.
• Ownership test: Callback and DPC routine addresses should lie within known executable driver sections. Targets in unbacked pool memory are high-priority findings.
• Temporal role: A periodic timer can reapply hooks, restore terminated components, or signal hidden code without creating an obvious user-mode process.
• Corroboration: Link the routine address to module listings, executable memory scans, driver objects, device trees, and extracted code before classifying it as malicious.
• Operational caution: Security products legitimately register numerous callbacks and timers; module signature, path, function purpose, and surrounding objects determine significance.