Unit 4: Linux and Network Forensics - Subjective Questions

Volatile data is information that may be altered or lost when a Linux system is shut down, restarted, or allowed to continue operating. Examples include:

• Contents of RAM
• Running processes and loaded kernel modules
• Active network connections and listening ports
• Logged-in users and current sessions
• Open files, command history held in memory, and clipboard data
• ARP cache and routing table

Non-volatile data persists after shutdown and is normally stored on disks or other storage media. Examples include:

• File-system files and directories
• System and application logs
• User account information
• Configuration files
• Shell history written to disk
• Deleted files that have not been overwritten
• Swap partitions and persistent journal files

Key distinction: volatile data is temporary and highly time-sensitive, whereas non-volatile data is comparatively persistent.

The order of volatility requires investigators to collect the most transient evidence first. A typical order is CPU and cache information, RAM, network state, running processes, temporary file systems, disk data, remote logs, and archival media. Following this order reduces the risk that valuable evidence will disappear or be modified before acquisition.

A forensically sound live-response procedure includes the following steps:

1. Obtain authorization: Confirm the legal scope and document the date, time, system identity, and investigator details.
2. Record the system state: Photograph or capture the screen and note visible applications, users, and error messages.
3. Use trusted tools: Run statically linked tools from verified external media whenever possible. Calculate hashes of the tools before use.
4. Collect evidence by volatility: Acquire system time, logged-in users, running processes, loaded modules, open files, active connections, routing data, ARP cache, mounted file systems, and RAM.
5. Minimize system changes: Avoid installing software or writing output to the suspect disk. Store collected data on external media or transmit it to a secured forensic server.
6. Hash acquired evidence: Generate cryptographic hashes such as SHA-256 for memory images and command outputs.
7. Maintain documentation: Record every command executed, its output location, hash, and any observed errors.
8. Preserve chain of custody: Seal and label the acquired data and record every transfer or access.

Live acquisition inevitably changes a small amount of system state. Therefore, every action and its expected impact must be documented so that the procedure remains reproducible and defensible.

Important sources of non-volatile Linux evidence include:

• File-system metadata: Inodes, permissions, ownership, timestamps, links, directory entries, and allocation information.
• System logs: Files under /var/log, including authentication, kernel, boot, audit, and application logs.
• Persistent journal: Binary logs maintained by systemd-journald, which can be examined using journalctl or forensic parsers.
• User artifacts: Shell histories, recently used files, SSH configuration, browser data, downloaded files, and desktop-environment artifacts.
• Account information: /etc/passwd, /etc/shadow, /etc/group, sudo configuration, and login records such as wtmp, btmp, and lastlog.
• Configuration files: Network settings, scheduled tasks, startup services, firewall rules, and application configurations.
• Persistence mechanisms: Cron jobs, systemd services, shell initialization files, kernel modules, and modified startup scripts.
• Swap and hibernation data: These may contain fragments of processes, credentials, commands, and network communication.
• Unallocated and slack space: These areas may contain remnants of deleted files.

These sources help reconstruct user activity, identify persistence, recover deleted information, and establish a timeline of events.

A Linux file-system image should be analyzed through a structured workflow:

1. Evidence verification: Calculate and compare cryptographic hashes to verify that the image has not changed since acquisition.
2. Identify the image and partitions: Determine the image format, partition table, partition offsets, file-system types, encryption, and logical volume layout.
3. Create a working copy: Preserve the original image and conduct analysis on a verified duplicate.
4. Read-only access: Mount the image read-only, preferably with options that prevent journal replay or metadata updates. Alternatively, use forensic tools that parse the image without mounting it.
5. File-system examination: Analyze directories, inodes, permissions, ownership, timestamps, symbolic links, extended attributes, and access control lists.
6. Artifact collection: Extract logs, account data, shell histories, SSH artifacts, browser data, cron jobs, startup services, and application files.
7. Deleted-file recovery: Examine unallocated space, deleted directory entries, file-system journals, and file signatures.
8. Keyword and hash analysis: Search for relevant terms and compare file hashes with known-good and known-malicious databases.
9. Timeline creation: Correlate file-system timestamps with logs and other artifacts.
10. Reporting: Document tools, versions, commands, hash values, findings, limitations, and conclusions.

The workflow protects evidence integrity while allowing investigators to reconstruct system activity and identify suspicious modifications.

Linux file systems store file content and metadata separately. These structures can provide evidence even after deletion:

• Inodes: An inode stores metadata such as file type, permissions, owner, size, timestamps, link count, and pointers to data blocks. If an inode has not been reused, it may still reveal metadata and locations of deleted content.
• Directory entries: A deleted file name may remain in a directory structure until the entry is overwritten or reused.
• File-system journal: Journaling file systems may retain records of recent metadata operations. These records can help determine when files were created, renamed, modified, or deleted.
• Unallocated space: After deletion, a file's data blocks are marked available, but their contents may remain until overwritten. File carving can recover data based on headers, footers, and internal structure.
• Slack space: Unused bytes within allocated blocks may contain remnants of previous file content.

Recovery is not always complete. Fragmentation, overwriting, encryption, solid-state drive garbage collection, and the reuse of inodes or data blocks can reduce recoverability. Recovered content should therefore be validated using metadata, file signatures, contextual artifacts, and hashes.

Read-only access prevents accidental or automatic modification of original evidence. Mounting a Linux file system normally may update access times, replay the journal, change file-system state, or create system files.

• Write blocker: A hardware or software write blocker prevents write commands from reaching the evidence device. Hardware write blockers are generally preferred for physical media because they provide control below the operating-system level.
• Mounting precautions: Investigators should avoid mounting original media directly. A verified image should be examined instead. If mounting is necessary, read-only options and precautions against journal replay should be used.
• Cryptographic hashes: Hashes such as SHA-256 provide a digital fingerprint of the evidence. A hash is calculated during acquisition and checked before and after analysis. Matching values support the conclusion that the image is unchanged.
• Working copies: Analysis should be conducted on one or more verified working copies while the original is securely preserved.
• Documentation: Device identifiers, commands, tools, options, dates, and hash values must be recorded.

Together, these controls preserve integrity, support chain of custody, and make the evidence more defensible in legal or disciplinary proceedings.

File-system timeline analysis arranges metadata events in chronological order to reconstruct activity on a Linux system. Important timestamps commonly include:

• Modification time: Indicates when file content was last changed.
• Access time: Indicates when file content was last read, although it may be disabled, delayed, or updated using relatime behavior.
• Change time: Indicates when inode metadata or file content was last changed. It is not a creation timestamp.
• Birth or creation time: Indicates when a file was created, but it is not supported or exposed consistently by every Linux file system and tool.

Investigators can combine these timestamps into a super-timeline with authentication logs, process events, shell history, browser activity, network logs, and application records.

Limitations include:

• Attackers may alter timestamps through timestomping.
• System clocks may be incorrect or changed.
• Time zones and daylight-saving rules may differ between sources.
• Mount options may suppress access-time updates.
• Copying, extracting, or restoring files may produce misleading timestamps.
• Deleted or overwritten metadata may be incomplete.

Therefore, timestamps should be normalized to a common time zone and corroborated with independent evidence rather than interpreted in isolation.

Linux memory forensics is the acquisition and analysis of RAM to identify operating-system state, user activity, malware, and other evidence that may not exist on disk.

Major artifacts obtainable from memory include:

• Running, hidden, and recently terminated processes
• Process identifiers, parent-child relationships, arguments, and environment variables
• Loaded kernel modules and shared libraries
• Open files, file descriptors, pipes, and sockets
• Active and recently used network connections
• Routing and interface information
• Process memory maps and executable code regions
• Command fragments and terminal activity
• Decrypted content and cryptographic material that happens to be resident in memory
• Malware configuration, injected code, and unpacked payloads
• Kernel structures, system-call modifications, and evidence of rootkits
• Cached file content and fragments of network communication

Memory analysis is valuable because sophisticated malware may operate only in RAM, remove its disk files, or keep important data encrypted on disk. However, memory contents are highly dynamic and must be collected promptly using an acquisition method appropriate to the Linux kernel and system configuration.

Linux memory forensics presents several technical and evidential challenges:

• Kernel variation: Data structures differ among kernel versions, distributions, architectures, and build configurations. Analysts need matching symbols or type information.
• Acquisition impact: Loading an acquisition module or running a tool modifies memory and may overwrite evidence.
• System instability: Rootkits, damaged kernels, or high system load may cause acquisition failure or crashes.
• Large memory size: Modern systems may contain hundreds of gigabytes of RAM, increasing acquisition, storage, and processing time.
• Encryption and protection: Secure boot, kernel lockdown, virtualization, and restricted device interfaces may prevent conventional acquisition.
• Malware interference: Malware may hide objects, falsify kernel responses, or detect acquisition tools.
• Incomplete captures: Memory can change while it is being copied, creating an internally inconsistent image.

Methods for addressing these issues include using validated tools, collecting through trusted external mechanisms, acquiring as early as possible, recording all system changes, generating hashes, obtaining correct kernel symbols, cross-validating findings with multiple plugins or tools, and correlating memory results with disk and network evidence. When possible, hypervisor-based acquisition can reduce reliance on the potentially compromised guest operating system.

Memory forensics detects malware by examining memory structures directly rather than trusting potentially compromised operating-system commands.

• Hidden processes: Investigators compare different process-enumeration methods, such as active task lists, process identifiers, scheduler structures, and memory scanning. A process found by scanning but absent from normal lists may have been unlinked by a rootkit.
• Process injection: Suspicious signs include executable anonymous memory, writable and executable regions, code without a corresponding file, unexpected shared libraries, altered memory permissions, and mismatches between mapped files and in-memory code.
• Kernel rootkits: Analysts inspect loaded and hidden kernel modules, system-call tables, interrupt structures, function pointers, hooks, and modified kernel code. Unknown modules or pointers directed into unexplained memory regions require investigation.
• Network correlation: Hidden processes may still own sockets or communicate with external command-and-control infrastructure.
• Cross-view analysis: Results from kernel structures, process memory, file-system images, and network logs are compared to identify inconsistencies.

No single anomaly proves compromise. Findings should be validated against the correct kernel version, legitimate security software, expected application behavior, and known-good system baselines.

Network forensics is the capture, preservation, and analysis of network communication and related metadata to investigate security incidents and reconstruct events.

• Full packet capture: Stores packet headers and, when available, payloads. It can support protocol analysis, file extraction, session reconstruction, and detection of malicious commands. Its disadvantages are high storage requirements, privacy concerns, and reduced visibility when traffic is encrypted.
• Flow records: Summarize communication using fields such as source and destination addresses, ports, protocol, start time, duration, packet count, and byte count. They require less storage and are useful for identifying scanning, beaconing, and unusual data transfer, but usually do not contain payloads.
• Network-device logs: Firewall, router, VPN, proxy, DNS, IDS, and wireless logs record decisions and events from particular devices. They provide contextual information such as blocked connections, authenticated users, alerts, and domain queries, but may be filtered, incomplete, or vendor-specific.

The strongest investigation combines these sources. Packet data provides depth, flow data provides broad and long-term visibility, and device logs provide security and identity context.

Linux logging records operating-system, authentication, service, kernel, and application events.

• Syslog: Applications submit messages with a facility and severity. A syslog daemon, such as rsyslog or syslog-ng, filters and stores messages locally or forwards them to a central server.
• systemd-journald: Collects structured events from services, the kernel, standard output, and other sources. The journalctl command can filter records by service, boot, process, priority, and time.
• Log rotation: Utilities such as logrotate rename, compress, archive, and eventually delete old log files. Investigators must collect rotated and compressed logs because the active file contains only recent events.
• Authentication records: Depending on the distribution, /var/log/auth.log or /var/log/secure records SSH, sudo, and authentication activity.
• Login databases: wtmp, btmp, and lastlog provide successful-login, failed-login, and last-login information.
• Audit logs: Linux Audit records system calls, policy events, account changes, and access to monitored resources.
• Kernel and service logs: These may reveal hardware events, module loading, crashes, service starts, and firewall activity.

Investigators should account for different distribution layouts, volatile journal storage, retention rules, missing events, and possible attacker tampering.

Effective log preservation requires technical and procedural controls:

• Centralized collection: Forward logs to a hardened remote collector so that an attacker who compromises one host cannot easily alter all copies.
• Time synchronization: Use an authenticated and monitored time source. Record time zones and clock offsets to support accurate correlation.
• Access control: Restrict log reading, deletion, and configuration changes using least privilege and strong authentication.
• Integrity protection: Apply cryptographic hashes, digital signatures, append-only storage, or write-once retention where appropriate.
• Secure transport: Protect log forwarding with authenticated encryption to reduce interception and spoofing.
• Adequate retention: Define retention periods based on legal, operational, and incident-response requirements.
• Redundancy and backup: Maintain protected copies at separate locations and test restoration procedures.
• Monitoring: Generate alerts for stopped logging services, unexpected gaps, altered configurations, clock changes, and storage exhaustion.
• Chain of custody: Document collection time, source, method, tool, custodian, hash, and every transfer.

Logs must also be interpreted carefully because they may be incomplete, incorrectly timestamped, generated by a compromised host, or based on spoofable network information.

Network forensic readiness is the ability of an organization to collect, preserve, and use network evidence efficiently before an incident occurs. Its goals are to improve incident response, reduce investigation cost, and increase the probability that reliable evidence will be available.

A readiness plan should include:

1. Requirements assessment: Identify legal obligations, likely threats, critical assets, investigation questions, and evidence sources.
2. Visibility design: Select capture points for internet gateways, critical network segments, cloud environments, VPNs, DNS, proxies, firewalls, and identity systems.
3. Collection strategy: Decide where full packets, flow records, alerts, and logs are required while balancing storage and privacy.
4. Time synchronization: Standardize clocks and time zones across hosts, network devices, and cloud services.
5. Central storage: Use secured, scalable, and redundant repositories with defined retention periods.
6. Integrity controls: Implement access restrictions, encryption, hashing, signing, and tamper-evident auditing.
7. Operational procedures: Define alert triage, evidence export, escalation, chain of custody, and legal approval processes.
8. Privacy and compliance: Minimize unnecessary personal data and regulate access to captured content.
9. Testing: Conduct exercises to confirm that evidence can be located, exported, verified, and interpreted.
10. Continuous improvement: Review sensor coverage, capacity, detection rules, and lessons learned after incidents.

Readiness must be maintained continuously because network architecture, threats, and legal requirements change over time.

Event correlation is the process of combining related events from multiple sources to identify patterns, reduce isolated alerts, and reconstruct an incident.

• Temporal correlation: Links events occurring within a defined time interval. For example, repeated failed SSH logins followed shortly by a successful login may indicate password guessing.
• Attribute-based correlation: Links events that share fields such as an IP address, user name, host, file hash, domain, or session identifier. A DNS query and firewall connection involving the same host and destination can be connected this way.
• Causal correlation: Establishes a cause-and-effect sequence. For example, a phishing download may lead to process execution, persistence creation, and outbound command-and-control traffic.
• Rule-based correlation: Uses predefined logic. An alert may be generated when a privileged login is followed by a new user account and firewall modification.
• Statistical or behavioral correlation: Detects deviations from a baseline, such as an internal server suddenly transferring unusually large amounts of data.

Reliable correlation requires normalized fields, synchronized timestamps, consistent identifiers, and awareness of NAT, DHCP, proxies, clock drift, and missing records. Correlation generates investigative hypotheses, which should be verified using original evidence.

Indicators of Compromise, or IoCs, are observable artifacts or patterns that suggest malicious activity or a security breach.

Network-log IoCs include:

• Known-malicious IP addresses, domains, URLs, and autonomous systems
• Repeated DNS queries for unusual or newly registered domains
• Domain-generation patterns and abnormally long subdomains
• Regular beaconing intervals to an external host
• Connections to rare countries, networks, ports, or protocols
• Unexpected inbound scanning or repeated failed connections
• Large or unusual outbound transfers indicating possible exfiltration
• Suspicious HTTP user agents, URI paths, response codes, or header patterns
• TLS certificate fingerprints, server-name values, and unusual handshake characteristics
• Internal lateral movement using administrative protocols

An investigator should validate IoCs against threat intelligence, historical baselines, asset roles, DNS records, and other evidence.

IoCs have limitations: IP addresses and domains can change, legitimate cloud infrastructure may host malicious content, shared addresses can create false positives, encryption hides payloads, and attackers can imitate normal traffic. Therefore, behavioral patterns and context are generally more durable than a single atomic indicator.

A systematic network traffic investigation can follow these stages:

1. Validate the alert: Determine the alert source, rule, affected asset, confidence level, and possible false positives.
2. Define scope: Establish the relevant systems, users, addresses, protocols, and time interval.
3. Preserve evidence: Export packet captures, flow data, DNS records, firewall logs, proxy logs, IDS alerts, and authentication records. Record hashes and chain of custody.
4. Normalize time: Convert timestamps to a common time zone and account for clock drift.
5. Identify endpoints: Map IP addresses to hosts and users using DHCP, VPN, NAT, asset inventory, and identity records.
6. Analyze traffic: Review sessions, packet direction, ports, protocols, byte counts, payloads, DNS behavior, certificates, and timing patterns.
7. Reconstruct activity: Determine initial access, command-and-control communication, lateral movement, downloads, and potential exfiltration.
8. Pivot on indicators: Search for related addresses, domains, hashes, user agents, certificates, and behavior across other data sources.
9. Correlate host evidence: Compare network observations with processes, memory artifacts, files, and Linux logs.
10. Assess impact: Identify compromised assets, exposed data, attacker persistence, and containment needs.
11. Report findings: Present a supported timeline, evidence references, conclusions, limitations, and recommended actions.

The analyst should distinguish confirmed facts from assumptions and preserve the original evidence throughout the investigation.

Packet capture analysis reconstructs communication by grouping packets into flows based on source and destination addresses, source and destination ports, protocol, and time.

A typical procedure is:

• Verify the capture file's integrity and identify the capture period and sensor location.
• Apply display filters to isolate the relevant host, protocol, port, or conversation.
• Reassemble fragmented IP packets where possible.
• Reconstruct transport streams by ordering TCP segments and accounting for retransmissions, duplicate packets, and missing segments.
• Decode the application protocol, such as HTTP, DNS, FTP, SMTP, or SMB.
• Inspect requests, responses, headers, status codes, commands, and authentication exchanges.
• Export transferred objects when complete payload data is available.
• Calculate hashes of recovered files and compare them with malware intelligence or known-good databases.
• Correlate the session with DNS queries, flow records, IDS alerts, and host activity.

Reconstruction may fail or be incomplete because of packet loss, asymmetric routing, insufficient capture length, unsupported protocols, tunneling, or encryption. Recovered objects must be treated as derived evidence and linked to the original capture through documentation and hash values.

Encrypted traffic still exposes useful metadata even when its application payload cannot be decrypted. Investigators can analyze:

• Source and destination addresses and ports
• Connection start time, duration, direction, packet count, and byte count
• Packet-size distributions and traffic bursts
• Periodicity that may indicate command-and-control beaconing
• DNS queries made before encrypted connections
• TLS versions, cipher suites, extensions, and server-name values when visible
• Certificate subjects, issuers, validity periods, fingerprints, and unusual self-signed certificates
• Client and server handshake fingerprints
• Failed handshakes, repeated reconnects, and connections to rare destinations
• Deviations from the normal communication profile of a host

Decryption may be possible using lawfully acquired session keys, endpoint key logs, proxy inspection records, or memory artifacts. Possession of a server's long-term private key does not necessarily decrypt modern sessions that use forward secrecy.

Metadata findings should be correlated with endpoint processes, DNS logs, proxy records, authentication events, and threat intelligence. Encryption alone is not evidence of malicious activity, so conclusions must rely on context and behavior.

The investigation should build a unified timeline and test whether the events represent a single compromise.

1. Preserve volatile evidence: Record system time, users, processes, open files, loaded modules, sockets, routes, and firewall state, then acquire memory using a documented method.
2. Acquire storage evidence: Create a verified forensic image and preserve relevant Linux logs, journal records, and remote network logs.
3. Analyze authentication: Review SSH, PAM, sudo, wtmp, btmp, and audit records. Identify source addresses, attempted accounts, authentication method, login time, and post-login commands.
4. Resolve identity and addressing: Correlate the source address with VPN, NAT, DHCP, firewall, and identity records.
5. Examine the unknown process: From memory, recover its arguments, parent process, executable regions, libraries, environment, open files, and network sockets. Compare the executable's memory and disk hashes.
6. Search for persistence: Inspect systemd units, cron jobs, startup scripts, SSH authorized keys, new accounts, shell profiles, and modified libraries or kernel modules.
7. Analyze outbound traffic: Use packet, flow, DNS, proxy, and TLS metadata to determine destination, interval, protocol, transferred volume, and potential command-and-control behavior.
8. Develop IoCs: Extract relevant addresses, domains, file hashes, process names, paths, keys, and certificate fingerprints, then search for them across the environment.
9. Create a correlated timeline: Link failed logins, successful access, process creation, persistence, and network communication while correcting for time zones and clock offsets.
10. Conclude and report: State what is confirmed, what remains inferred, the affected assets, likely entry point, persistence, network activity, and containment recommendations.

Strong attribution requires corroboration. For example, a successful login and outbound connection occurring close together do not alone prove that the login created the connection; process ownership, socket evidence, audit records, and packet timing should support that causal relationship.