Unit 4: Malware Persistence, Injection and Hooking

INT251 — Malware Analysis And Cyber Defence 10 min read

I. Foundations — How Malware Maintains and Extends Control

Malware is software intentionally designed to compromise confidentiality, integrity, availability, or system control. Persistence, injection, and hooking are related mechanisms: persistence restores access after interruption, injection places malicious code inside another process, and hooking redirects legitimate execution to attacker-controlled logic.

Governing principle: Malware attempts to execute while reducing its visibility and preserving access to the target.
Execution context: Every malicious action occurs through a process, thread, service, driver, script interpreter, or firmware component.
Privilege boundary: User-mode malware operates with process-level restrictions; kernel-mode malware can influence the operating system itself.
Stealth principle: Executing within trusted processes, modifying control flow, and using legitimate administration tools can weaken simple signature-based detection.
Defensive principle: Effective analysis correlates static evidence, memory state, process behaviour, persistence locations, and operating-system telemetry.
Ethical convention: Malware must be examined only in authorised, isolated laboratories using snapshots, controlled networking, and non-production data.

II. Malware Functionalities — Objectives and Behavioural Capabilities

A. Malware functionalities

Malware functionality describes what malicious software is designed to achieve after entering a system.

Initial execution: A malicious document, exploit, trojanised installer, script, or compromised account starts the malware’s code.
Discovery: Malware may enumerate concrete system properties such as hostnames, IP addresses, running processes, installed software, user accounts, domain membership, and security products.
Privilege escalation: The malware attempts to gain permissions beyond those of the initial account, such as administrator or SYSTEM privileges on Windows or root on Unix-like systems.
Credential access: Targets include browser credentials, password stores, authentication tokens, cookies, private keys, and credentials present in process memory.
Command and control (C2): An infected host exchanges commands and results with attacker infrastructure through protocols such as HTTPS, DNS, or custom TCP traffic.
Collection and exfiltration: Files, screenshots, clipboard contents, keystrokes, database records, or emails may be staged and transmitted outside the organisation.
Lateral movement: Stolen credentials and remote-management mechanisms may be used to reach additional hosts.
Impact: Ransomware encrypts data; wipers destroy it; botnets consume resources; sabotage malware alters physical or logical processes.
Evasion: Packing, encryption, anti-debugging, delayed execution, environment checks, log deletion, and trusted-process impersonation obstruct analysis.
Modularity: A loader may install separate modules for persistence, reconnaissance, credential theft, or exfiltration, allowing capabilities to change after infection.

B. Defensive analysis and limitations

Functionality is best inferred from multiple evidence sources rather than from filenames or isolated indicators.

Static evidence: Imports, strings, configuration data, embedded resources, and control-flow structures suggest capabilities without executing the sample.
Dynamic evidence: Process creation, registry modification, file writes, DNS requests, and network connections reveal actual behaviour.
Memory evidence: Decrypted configuration, injected code, credentials, and unpacked payloads may exist only in RAM.
Limitation: Sandbox-aware malware may remain dormant when it detects virtualisation, analysis tools, unrealistic user activity, or short execution windows.

III. Malware Persistence Methods — Surviving Reboots and Logons

A. Malware persistence methods

Persistence methods arrange for malicious code to execute again after reboot, logon, service restart, or another interruption.

Autostart locations: Windows Registry keys such as user or machine Run locations can launch a program during logon.
Startup directories: Files or shortcuts placed in designated startup folders are processed when a user signs in.
Scheduled execution: Windows Task Scheduler, Unix cron, and timer units can execute payloads at specified times or in response to events.
Services and daemons: A registered Windows service or modified systemd unit can start before user logon and may execute with elevated privileges.
Logon-related mechanisms: Authentication packages, shell settings, logon scripts, and related configuration can place malware in the sign-in path.
DLL loading abuse: Search-order hijacking or side-loading causes a legitimate application to load a malicious library with an expected filename.
Application extensions: Browser extensions, office add-ins, and application plug-ins can provide repeated execution within commonly used software.
Unix-like mechanisms: Shell profiles, systemd services, cron, and macOS LaunchAgents or LaunchDaemons are common persistence surfaces.
Boot and kernel persistence: Bootkits, malicious drivers, or firmware modification execute below ordinary applications, offering strong control but creating greater complexity and forensic risk.
Account persistence: Attackers may create users, add SSH authorised keys, or register access tokens so that persistence does not depend on one executable.

B. Detection and removal considerations

Persistence analysis must distinguish malicious entries from legitimate software that uses the same operating-system facilities.

Baseline comparison: Investigators compare services, tasks, drivers, startup entries, and account changes against an approved system baseline.
Contextual indicators: Strong warning signs include unsigned binaries, execution from user-writable directories, misleading names, unusual parent processes, and recently created autoruns.
Removal sequence: Analysts should first contain the host, preserve evidence, disable the launch mechanism, remove the payload, and reset compromised credentials.
Limitation: Deleting only the visible payload may fail because another persistence mechanism can recreate it.

IV. Virtual Memory — Address Translation and Process Isolation

A. Virtual memory

Virtual memory gives each process a private logical address space that the operating system maps to physical memory through page tables.

Pages and frames: Virtual memory is divided into pages, while RAM is divided into same-sized page frames; a common page size is 4 KiB.
Address translation: The memory-management unit uses page-table entries to translate virtual page numbers into physical frame numbers.

TEXT

VA = VPN × P + d
PA = PFN × P + d

Symbols: VA is virtual address, PA physical address, VPN virtual page number, PFN physical frame number, P page size, and d offset within the page.
Worked example: With P = 4096 bytes, virtual address 0x1234 lies on virtual page 1 with offset 0x234; the page table determines the physical frame.
Page permissions: Entries commonly specify readable, writable, executable, user-accessible, or kernel-only status. Data Execution Prevention restricts execution from non-executable pages.
Process isolation: Two processes may use the same virtual address while mapping it to different physical frames.
Demand paging: A page fault occurs when a referenced page is absent or violates permissions; the kernel then loads, maps, or rejects the access.
Address randomisation: Address Space Layout Randomisation changes the locations of stacks, libraries, heaps, and executables, complicating hard-coded attacks.
Malware relevance: Injection manipulates another process’s virtual memory, while memory-only malware may avoid writing its unpacked code to disk.

B. Analytical significance

Virtual-memory metadata can reveal malicious regions that are not apparent in ordinary file inspection.

Suspicious permissions: Pages that are simultaneously writable and executable, represented as RWX, deserve scrutiny because code can be written and immediately run there.
Private executable regions: Executable memory not backed by a normal image file may indicate unpacking, just-in-time compilation, or injection.
Qualification: Browsers, managed runtimes, and legitimate security tools can produce similar patterns, so permissions alone do not prove infection.

V. User Mode and Kernel Mode — Privilege Separation

A. User mode and kernel mode

Modern operating systems separate restricted application execution from privileged operating-system execution.

User mode
- Privilege level: On x86 systems, applications generally execute in ring 3.
- Restrictions: Code cannot directly access arbitrary physical memory, hardware devices, kernel data, or privileged processor instructions.
- System services: Applications request protected operations through system calls handled by the kernel.
- Failure scope: A user-process crash normally terminates that process rather than the operating system.
Kernel mode
- Privilege level: The operating-system kernel and authorised drivers generally execute in ring 0.
- Authority: Kernel code manages scheduling, virtual memory, devices, filesystems, networking, and security enforcement.
- Failure scope: Defective or malicious kernel code can crash or compromise the entire host.
- Malware value: A rootkit can hide processes, files, network connections, or registry objects by interfering with kernel data and interfaces.

Explicit contrast: User-mode malware is easier to deploy but remains constrained by access controls; kernel-mode malware gains broader control but must overcome driver-signing and integrity protections.

B. Defensive implications

The execution mode determines both the malware’s power and the evidence available to defenders.

User-mode monitoring: Endpoint tools can observe process creation, module loading, memory allocation, and API use, although user-mode hooks may be bypassed.
Kernel monitoring: Signed security drivers can inspect lower-level events but must be carefully protected because kernel defects affect system stability.
Trust problem: Once the kernel is compromised, user-mode tools may receive falsified results; offline disk analysis, secure boot evidence, or external memory acquisition may be required.

VI. Code Injection Techniques — Executing Inside Another Process

A. Code injection techniques

Code injection places executable content or a malicious module into another process’s address space and redirects a thread to execute it.

DLL injection: A target process is induced to load a specified dynamic-link library, making the library execute under the target’s identity.
Remote-thread injection: Malware allocates or identifies memory in another process, writes a payload, and causes a thread to begin at that location.
APC injection: Code is queued as an asynchronous procedure call and runs when a suitable target thread enters an alertable state.
Thread hijacking: An existing thread is suspended, its register context or instruction pointer is altered, and execution is redirected.
Process hollowing: A legitimate process is created in a suspended state, its original image is replaced or unmapped, and execution resumes in substituted code.
Reflective loading: A library is mapped from memory by custom loader logic rather than through the normal operating-system loader.
Shared-memory injection: A memory section is mapped into both attacker and target processes, allowing the payload to be transferred without a conventional cross-process write.
Purpose: Injection may inherit a trusted process name, privileges, network access, or security context while obscuring the original malicious process.

B. Detection and limitations

Injection detection focuses on abnormal relationships between memory, threads, modules, and process ancestry.

Behavioural sequence: Cross-process memory access followed by executable protection changes and thread creation is more meaningful than any single event.
Memory inspection: Analysts look for private executable pages, modules absent from loader lists, altered thread start addresses, and code inconsistent with the backing file.
Process context: A text editor opening another process with extensive memory rights is more suspicious than a debugger performing the same action.
Limitation: Security products, accessibility tools, debuggers, and application frameworks may legitimately inject code, requiring contextual correlation.

VII. Hooking Techniques — Redirecting Program Control Flow

A. Hooking techniques

Hooking intercepts a function, event, or system operation so that control passes through replacement or monitoring code.

Import Address Table hooking: A function pointer in a module’s import table is replaced so calls reach a hook function.
Inline hooking: Instructions at a function’s entry are overwritten with a branch to a detour; a trampoline may preserve displaced instructions before returning.
Export-table hooking: Exported function addresses are altered so future symbol resolution returns attacker-controlled locations.
Virtual-table hooking: Function pointers in C++ virtual method tables are replaced, redirecting object method calls.
Event and message hooks: Operating-system facilities can intercept keyboard, mouse, window, or message events; these support legitimate accessibility software but can also enable keylogging.
Kernel hooking: Rootkits may target system-call paths, dispatch tables, drivers, or callbacks, although modern integrity protections make direct kernel modification difficult.
Legitimate use: Profilers, debuggers, compatibility layers, monitoring agents, and security tools use hooks for instrumentation.