Unit 3: Endpoint, SIEM, and Security Analytics

I. Security Monitoring and Evidence-Driven Defense

Endpoint security, SIEM, and security analytics form a connected defensive system: endpoints generate evidence, SIEM platforms centralize and correlate it, and analytics identify patterns requiring investigation or response.

• Governing principle — continuous visibility: Security is not a one-time control; telemetry must be collected, analyzed, acted upon, and retained throughout the attack lifecycle.
• Endpoint: A user or computing device—such as a workstation, server, mobile device, or virtual machine—that accesses organizational resources.
• Security telemetry: Evidence including process creation, authentication, network connections, registry changes, file hashes, DNS requests, and cloud audit events.
• Detection model: Events become alerts when rules, behavioral baselines, threat intelligence, or machine-learning models identify suspicious activity.
• Threat hunting: A proactive search for hidden compromise based on hypotheses rather than waiting for predefined alerts.
• SOC workflow: Security operations centers typically move through collection, detection, triage, investigation, containment, eradication, and recovery.
• Risk principle: Priority depends on both threat severity and business context; an alert on a domain controller normally carries more risk than the same alert on an isolated test machine.
• Core constraints: Defenders must balance coverage, false positives, storage cost, privacy, response speed, and regulatory obligations.

II. Endpoint Analysis and Threat Hunting — Device-Level Detection and Investigation

Endpoint analysis examines host activity at high resolution, while endpoint threat hunting searches that evidence for indicators, behaviors, and attack chains that automated controls may miss.

A. Endpoint Analysis and Threat Hunting: endpoint detection and response

Endpoint detection and response (EDR) continuously records endpoint behavior and supports detection, investigation, containment, and remediation.

• Telemetry: EDR agents capture process trees, command lines, file modifications, user logons, network destinations, loaded modules, and persistence mechanisms.
• Detection: Rules may identify concrete behavior such as winword.exe spawning powershell.exe, while behavioral models detect deviations without requiring a known hash.
• Investigation: Analysts reconstruct causal chains—parent process, child process, user, file, destination, and timestamp—to determine scope and root cause.
• Response: Common actions include isolating the host, killing a process, quarantining a file, blocking a hash, and collecting forensic artifacts.
• Limitation: Agent tampering, encrypted traffic, offline devices, and excessive alert volume can reduce visibility; EDR must complement identity, network, and cloud monitoring.

B. Endpoint Analysis and Threat Hunting: malware analysis and reverse engineering

Malware analysis determines what suspicious software does, how it persists, and which evidence can support detection or containment.

• Static analysis: Investigators inspect hashes, strings, imports, headers, digital signatures, and embedded resources without executing the sample.
  • SHA-256 provides a 256-bit identifier, but a changed file normally produces a different hash.
• Dynamic analysis: A controlled sandbox records processes, files, registry keys, API calls, network traffic, and persistence created during execution.
• Reverse engineering: Disassemblers and debuggers translate machine instructions into analyzable assembly and reveal control flow, encryption routines, or command-and-control logic.
• Safety: Samples require isolated networks, snapshots, restricted credentials, and controlled outbound access to prevent accidental propagation.
• Outputs: Analysis produces indicators of compromise (IOCs), behavioral detections, YARA rules, remediation steps, and mappings to techniques such as credential dumping.

C. Endpoint Analysis and Threat Hunting: data and asset-focused risk models

Data- and asset-focused models prioritize security work according to what a system stores, processes, or enables.

• Asset model: Assets are classified by business function, owner, exposure, dependencies, and criticality—for example, public web server versus certificate authority.
• Data model: Data classification distinguishes public, internal, confidential, and regulated information, with controls matched to sensitivity.
• Risk relationship: A practical prioritization model is:

Risk = Likelihood × Impact

• Likelihood represents exploitability, exposure, and observed threat activity.
• Impact reflects confidentiality, integrity, availability, financial, and legal consequences.
  • Contextual enrichment: EDR alerts gain priority when joined with asset inventories, vulnerability status, privileged-user data, and data ownership.
  • Limitation: Incomplete inventories and outdated classifications cause apparently precise risk scores to misrepresent real exposure.

D. Endpoint Analysis and Threat Hunting: principles of endpoint security

Endpoint security applies layered preventive, detective, and corrective controls directly to devices.

• Least privilege: Users and services receive only necessary permissions; routine work should not use local administrator accounts.
• Hardening: Disable unnecessary services, enforce secure configurations, restrict macros, enable host firewalls, and remove unsupported software.
• Patch management: Inventory, test, prioritize, deploy, and verify updates, especially for internet-facing or actively exploited vulnerabilities.
• Application control: Allowlisting limits execution to approved binaries, scripts, libraries, and interpreters.
• Defense in depth: Anti-malware, EDR, disk encryption, multifactor authentication, backups, and network segmentation address different failure modes.
• Assume compromise: Monitoring should remain active after preventive controls succeed because stolen credentials and trusted tools may bypass them.

III. Security Information and Event Management — Centralized Visibility and Response

A security information and event management (SIEM) platform ingests heterogeneous logs, normalizes them, correlates related activity, generates alerts, and supports investigation, compliance, and operational oversight.

A. Security Information and Event Management: fundamentals of SIEM

SIEM turns distributed machine records into searchable security evidence.

• Pipeline: Sources → collectors → parsers → normalized fields → storage → correlation → alerts and dashboards.
• Sources: Typical inputs include firewalls, EDR, Active Directory, DNS, VPNs, applications, cloud control planes, and intrusion-detection systems.
• Normalization: Vendor-specific fields are mapped to common concepts such as source_ip, user, action, and timestamp.
• Correlation: A rule might alert when five failed logons are followed by one success from the same address within ten minutes.
• Quality requirement: Accurate timestamps, synchronized through NTP, are essential for reconstructing event order.

B. Security Information and Event Management: distributed processing

Distributed processing allows SIEM systems to handle high event volumes across multiple collectors, workers, and storage nodes.

• Scale measure: Ingestion is commonly described using events per second (EPS) or data volume such as GB/day.
• Architecture: Load-balanced collectors receive data; message queues buffer bursts; parallel workers parse and enrich events; clustered storage supports searches.
• Resilience: Replication, checkpointing, retries, and local buffering reduce loss when links or nodes fail.
• Trade-off: Distribution improves throughput and availability but adds synchronization, consistency, and operational complexity.

C. Security Information and Event Management: accelerated threat hunting

SIEM accelerates hunting by making long-term, cross-source evidence searchable through reusable queries and enriched datasets.

• Hypothesis: A hunt begins with a proposition, such as “compromised accounts are authenticating from unusual regions.”
• Enrichment: Asset criticality, geolocation, user role, vulnerability data, and threat-intelligence indicators reduce manual lookup.
• Pivoting: Analysts move from a username to its devices, IP addresses, processes, and cloud actions.
• Optimization: Indexed fields, summary tables, time-bounded searches, and saved queries reduce search latency.

D. Security Information and Event Management: regulatory reporting with SIEM

SIEM supports regulatory reporting by preserving auditable records and demonstrating that required controls operate.

• Evidence: Authentication histories, administrator activity, access to sensitive data, and control changes support audits.
• Reports: Scheduled reports can document failed logons, privileged-account use, firewall changes, and incident handling.
• Integrity: Role-based access, immutable storage, checksums, and access logs help establish evidential reliability.
• Caution: SIEM reports assist compliance but do not prove security; report scope must match the applicable legal or contractual requirement.

E. Security Information and Event Management: infrastructure management

SIEM infrastructure must be engineered as a critical service rather than treated as a passive log repository.

• Capacity: Planning accounts for peak EPS, retention duration, indexing overhead, query concurrency, and replication.
• Health monitoring: Teams track collector outages, parsing failures, dropped events, queue depth, disk utilization, and search latency.
• Security: Administrative access requires least privilege, multifactor authentication, encryption, and separation of duties.
• Change control: Parser, rule, and platform updates should be tested because errors can silently suppress detections.

F. Security Information and Event Management: SIEM log retention strategies

Retention strategies preserve useful evidence while controlling cost, privacy exposure, and performance.

• Tiering: Recent searchable data may remain in hot storage; older data moves to warm, cold, or archived storage.
• Retention factors: Decisions consider investigation windows, regulatory duties, source value, storage expense, and data sensitivity.
• Calculation:

Storage ≈ Daily ingestion × Retention days × Replication factor

• Example: At 100 GB/day, 90 days, and two copies, raw capacity is approximately 18,000 GB before compression or indexing overhead.
• Governance: Deletion schedules, legal holds, encryption, and access controls must apply throughout the log lifecycle.

G. Security Information and Event Management: automated response and remediation

Automated response connects SIEM detections to predefined actions, often through security orchestration, automation, and response (SOAR).

• Playbooks: Automation can enrich an IP address, retrieve endpoint details, open a ticket, disable an account, or isolate a device.
• Control levels: Low-risk enrichment may run automatically; disruptive containment may require analyst approval.
• Safeguards: Confidence thresholds, allowlists, rollback procedures, rate limits, and audit trails reduce accidental damage.
• Measurement: Automation lowers mean time to respond (MTTR), but poorly tuned detections can automate false-positive consequences.

H. Security Information and Event Management: threat hunting with SIEM

Threat hunting with SIEM uses iterative queries to test hypotheses across identity, endpoint, network, and cloud evidence.

• Process: Define hypothesis → select data → query → pivot → validate → document → convert findings into detections.
• Behavioral focus: Hunts examine patterns such as rare parent-child processes, impossible travel, unusual service creation, or off-hours administration.
• Baseline comparison: Frequency and peer-group comparisons distinguish routine activity from outliers.
• Outcome: A successful hunt may find compromise, close a visibility gap, or establish that a suspected behavior is legitimate.

I. Security Information and Event Management: operational requirements

Effective SIEM operation requires people, processes, data governance, and measurable service objectives.

• Roles: Platform engineers maintain ingestion; detection engineers create rules; analysts triage alerts; incident responders contain threats.
• Runbooks: Each alert needs severity, required evidence, escalation criteria, response actions, and ownership.
• Metrics: Useful measures include ingestion availability, alert precision, mean time to detect, MTTR, and false-positive rate.
• Coverage: Use-case inventories should map critical assets and likely attack techniques to available log sources.

IV. Security Analytics and Machine Learning in SOC — Behavior-Based Detection

Security analytics applies statistical and machine-learning methods to telemetry, while SOC analysts provide context, validation, and response authority.

A. Security Analytics and Machine Learning in SOC: behavioral analytics and UEBA

User and entity behavior analytics (UEBA) models normal activity and identifies deviations involving users, hosts, applications, or service accounts.

• Features: Logon hour, source location, device count, data-transfer volume, accessed resources, and command frequency can form a behavioral profile.
• Peer groups: Comparing an administrator with other administrators is usually more meaningful than comparing that account with all employees.
• Risk aggregation: Several weak signals—new device, unusual country, and mass download—may combine into one high-risk case.
• Limitation: Job changes, travel, and seasonal workloads create legitimate deviations, so analyst validation remains necessary.

B. Security Analytics and Machine Learning in SOC: ML-based security analytics

ML-based analytics learns relationships from data to classify activity, rank risk, or discover clusters.

1. Supervised learning: Models such as logistic regression or random forests learn from labeled examples of benign and malicious activity.
2. Unsupervised learning: Clustering and isolation methods search for structure or outliers without complete labels.
   • Evaluation: Precision measures the proportion of alerts that are correct; recall measures the proportion of relevant threats detected.

Precision = TP / (TP + FP)
Recall = TP / (TP + FN)

• Constraint: Rare attacks, biased labels, adversarial adaptation, and changing environments can degrade model reliability.

C. Security Analytics and Machine Learning in SOC: deployment of predictive models

Deploying a predictive model requires controlled integration, monitoring, and lifecycle management beyond model training.

• Pipeline: Collect data → engineer features → train → validate → deploy → monitor → retrain or retire.
• Validation: Time-based test sets help prevent future information from leaking into training data.
• Operational mode: New models should first run in shadow mode, scoring events without triggering containment.
• Governance: Versioning must record model, features, training data, thresholds, owner, and approval history.
• Drift: Changes in users, systems, or attackers can alter feature distributions and reduce accuracy.

D. Security Analytics and Machine Learning in SOC: anomaly detection in SOC

Anomaly detection identifies observations that differ substantially from expected baselines, providing leads rather than proof of compromise.

• Statistical approach: A z-score measures distance from a mean in standard deviations:

z = (x - μ) / σ

• x is the observed value, μ the baseline mean, and σ the standard deviation.
  • Methods: Thresholds, time-series models, clustering, isolation forests, and autoencoders detect different forms of deviation.
  • Tuning: Thresholds must reflect asset criticality, seasonality, peer groups, and the SOC’s investigation capacity.
  • Interpretation: High anomaly scores require supporting evidence—such as identity, endpoint, and network events—before containment decisions are made.