Unit 5: Data Collection Rule (DCR)

INT328 — Network Virtualization And Cloud Security 11 min read

I. Orientation

Azure monitoring and cloud security depend on collecting activity, performance, and security data from resources into services that can analyze, retain, and correlate it. A Data Collection Rule (DCR) is an Azure Monitor configuration object that defines what data is collected, how it is transformed, and where it is sent. Log Analytics, Microsoft Defender for Cloud, and Microsoft Sentinel use related monitoring and security data flows to support detection, investigation, compliance, and response.

Central principle: Monitoring data should be collected deliberately, routed to an appropriate destination, and retained according to operational and compliance requirements.
DCR scope: A DCR describes data sources, data flows, destinations, optional transformations, and the resources to which the rule is associated.
Primary agent: The Azure Monitor Agent (AMA) commonly uses DCRs to collect guest operating-system events, performance counters, and text-based logs from virtual machines.
Common destination: A Log Analytics workspace stores data in Azure Monitor Logs and supports Kusto Query Language (KQL) queries.
Security relationship: Microsoft Defender for Cloud provides posture management and workload protection, while Microsoft Sentinel provides SIEM and SOAR capabilities.
Governance assumption: Collection must respect least privilege, data minimization, regional requirements, retention settings, and cost controls.
Terminology convention: A resource provider, diagnostic setting, agent, DCR, workspace, and security solution are separate configuration components even when they participate in one monitoring workflow.

II. Data Collection Rule (DCR) — Controlled Azure Monitor ingestion

A DCR is an Azure resource that specifies the relationship between an input data source and one or more output destinations. It separates collection policy from the individual resource, allowing administrators to apply consistent monitoring settings across virtual machines and other supported resources.

A. Definition and operating model

A DCR controls collection from supported Azure Monitor sources and sends selected records to destinations such as a Log Analytics workspace. The rule becomes effective when it is associated with an eligible resource.

Data sources: Examples include performanceCounters, windowsEventLogs, syslog, iisLogs, and logFiles.
Data flows: A data flow maps one or more data sources to a destination and can specify a target table such as Microsoft-Perf or Microsoft-Syslog.
Destinations: A DCR can reference Log Analytics workspaces and, for supported scenarios, Azure Monitor metrics or other destinations.
Transformations: A transformation can filter or reshape records before ingestion using a supported KQL transformation expression.
Association: The Azure Monitor Agent on a virtual machine requires a DCR association, linking the VM to the collection rule.
Security boundary: The identity creating or modifying the DCR needs permissions such as Microsoft.Insights/dataCollectionRules/*; the agent also requires appropriate connectivity and identity configuration.
Cost effect: Collecting every event at maximum verbosity increases ingestion and retention charges, so filtering noisy data at collection time is valuable.

B. DCR configuration sequence

A practical DCR implementation follows a predictable sequence from source selection to verification.

Select sources: Identify the exact event channels, syslog facilities, performance counters, or file paths required by operations and security teams.
Select destinations: Choose the Log Analytics workspace that will hold the data and confirm its region, retention, and access model.
Create data flows: Map each source to a destination table. For example, Windows Security events may flow to the WindowsEvent table.
Apply transformations: Use filtering where supported to discard irrelevant records or normalize fields before storage.
Associate resources: Link the DCR to virtual machines or other supported resources through the Azure portal, CLI, PowerShell, or ARM/Bicep.
Verify ingestion: Generate a known event, wait for ingestion, then query the destination table in Log Analytics.

Example CLI pattern:

BASH

az monitor data-collection rule association create \
  --name vm-dcr-association \
  --rule "$DCR_ID" \
  --resource "$VM_ID"

Here, DCR_ID is the full resource ID of the rule and VM_ID is the full resource ID of the virtual machine.

III. Log Analytics Workspace — Central log repository

A Log Analytics workspace is an Azure resource that stores and indexes log data for querying, alerting, visualization, and investigation. It is commonly the principal destination for Azure Monitor Agent data and the data store used by Microsoft Sentinel.

A. Workspace purpose and design

The workspace provides a logical and access-controlled boundary for Azure Monitor Logs. Its design affects data visibility, retention, query performance, and cost.

Data model: Records are stored in tables such as Heartbeat, Perf, Syslog, SecurityEvent, and resource-specific Azure tables.
Query language: KQL queries filter, aggregate, join, and summarize records. For example:

KUSTO

Heartbeat
| summarize LastSeen=max(TimeGenerated) by Computer

TimeGenerated identifies the event time, while Computer identifies the reporting machine.

Retention: Interactive retention determines how long data remains readily queryable; longer retention may use archive capabilities depending on table and workspace features.
Permissions: Azure RBAC controls workspace and resource access, while table-level or row-level controls may further restrict sensitive data.
Regional placement: Workspace location can affect data residency, service availability, and network design.
Billing: Charges can depend on ingested volume, retention, search, and commitment-tier choices.

B. Create Log Analytics workspace

Creating a workspace establishes the destination that DCRs, diagnostic settings, Defender solutions, and Sentinel will use.

Resource values: Choose an Azure subscription, resource group, workspace name, region, pricing or commitment option, and retention period.
Naming constraint: The workspace name must be unique within the relevant Azure context and should identify its environment, such as law-prod-security.
Portal sequence: Open Log Analytics workspaces, select Create, choose subscription and resource group, enter the workspace and region, configure pricing and retention, then review and create.
CLI example:

BASH

az monitor log-analytics workspace create \
  --resource-group rg-monitoring \
  --workspace-name law-prod-security \
  --location eastus

Post-creation values: Record the workspace resource ID and customer ID when required by integrations. Newer Azure APIs generally use resource IDs and managed identities instead of legacy shared keys.
Validation: Open Logs, run a basic query such as Heartbeat | take 10, and confirm that the workspace is reachable even before all agents are configured.

IV. Azure Storage account — Durable log and data retention

An Azure Storage account provides scalable object, file, queue, or table storage. In monitoring architectures, Blob Storage is frequently used for diagnostic-log export, long-term retention, archival, backup-related data, or investigation artifacts.

A. Storage role and security model

Storage complements Log Analytics: Log Analytics supports indexed operational queries, whereas Blob Storage is better suited to durable, lower-cost retention and raw exported data.

Service selection: Blob containers store objects; Azure Files provides SMB/NFS file shares; queues support messaging; tables provide key-value storage.
Replication: Locally redundant storage keeps copies in one region; zone-redundant or geo-redundant options improve resilience at additional cost and with different recovery characteristics.
Access control: Microsoft Entra ID with Azure RBAC is preferred for managed access. Shared access signatures should be narrow in scope, short-lived, and restricted by permissions and expiry.
Network controls: Storage firewalls, virtual network rules, private endpoints, and disabling public network access reduce exposure.
Encryption: Azure Storage encrypts data at rest by default; customer-managed keys may be required for particular governance policies.
Retention controls: Blob lifecycle management can move data to cool or archive tiers or delete it after a defined period.

B. Azure Storage account configuration

The account should be configured according to the type, sensitivity, and retention period of collected data.

Account kind: General-purpose v2 is the normal choice for Blob Storage and current storage features.
Performance: Standard performance is sufficient for most diagnostic exports; premium performance is reserved for latency-sensitive workloads.
Redundancy: Select LRS, ZRS, GRS, or another supported option based on availability and disaster-recovery requirements.
Secure transfer: Require HTTPS so exported data is protected in transit.
Blob container: Create a private container, for example diagnostic-logs, and avoid anonymous public access.
Diagnostic routing: Configure an Azure resource diagnostic setting to send selected categories to the storage account. This is distinct from a DCR and is commonly used for Azure resource control-plane or platform logs.
Validation: Check the container for newly created blobs and inspect their timestamps, resource paths, and category information.

A sound pattern is to retain high-value searchable data in Log Analytics while exporting compliance-required raw logs to restricted Blob Storage.

V. Microsoft Defender for Cloud — Cloud security posture and workload protection

Microsoft Defender for Cloud is an Azure security service that combines Cloud Security Posture Management (CSPM) with Cloud Workload Protection (CWP). It evaluates configuration risk, provides recommendations, and can detect threats in supported workloads.

A. Purpose and operating model

Defender for Cloud establishes security visibility across Azure and, through connectors, can extend assessment to other cloud environments and servers.

Posture assessment: Secure Score and recommendations identify weaknesses such as unrestricted management ports, missing encryption, or absent endpoint protection.
Plans: Defender plans can protect services such as servers, storage, databases, containers, and key management components; enabled plans can generate additional charges.
Agent integration: Server protection may use the Azure Monitor Agent and DCR-based collection for security events, vulnerability information, and related telemetry.
Regulatory mapping: Compliance dashboards map recommendations to standards and controls, but a dashboard result does not itself prove organizational compliance.
Alerts: Threat detections include severity, affected resource, evidence, and recommended response actions.
Identity principle: Defender requires permissions to assess resources and may use managed identities or service configurations for collection and integrations.

B. Configure Microsoft Defender for Cloud

Configuration begins with coverage and policy decisions, followed by data collection and alert validation.

Open environment settings: In Defender for Cloud, choose the relevant subscription or workspace and review security policy and Defender plans.
Enable plans: Activate only the workload protections required by the risk model, such as Defender for Servers for virtual machines.
Configure assessment: Set Microsoft Cloud Security Benchmark or customized policy initiatives and review the resulting recommendations.
Configure collection: For supported server scenarios, enable the required agent provisioning and security-event collection. Confirm the associated DCR sends data to the intended workspace.
Connect environments: Add non-Azure accounts when multicloud assessment is required, using the supported connector and least-privilege permissions.
Tune notifications: Configure continuous export, workflow automation, email notifications, or integrations with ticketing systems.
Validate: Create or identify a controlled test condition, confirm that Defender generates a recommendation or alert, and verify that the affected resource and evidence are correctly identified.

VI. Microsoft Sentinel — SIEM and SOAR for cloud security

Microsoft Sentinel is a cloud-native security information and event management (SIEM) and security orchestration, automation, and response (SOAR) service built on a Log Analytics workspace. It correlates data from Azure, applications, identities, devices, and external security products.

A. Sentinel architecture and workflow

Sentinel turns collected records into detections and coordinated response activities. It depends on an appropriately designed workspace and reliable connectors.

Workspace dependency: Sentinel is enabled on a Log Analytics workspace; its analytics rules and incidents operate over data stored there.
Data connectors: Connectors ingest signals from Microsoft Entra ID, Defender services, Azure resources, firewalls, Syslog, CEF, APIs, and partner products.
Analytics rules: Scheduled rules run KQL queries, while Microsoft security rules can use service-generated alerts. A rule can create an incident when conditions are met.
Incidents: Sentinel groups alerts into incidents containing entities such as accounts, IP addresses, hosts, and URLs.
Automation: Playbooks use Azure Logic Apps to perform actions such as notifying an analyst, disabling an account, or enriching an IP address.
Workbooks: Interactive dashboards summarize security posture, ingestion, incidents, and operational trends.
Threat intelligence: Indicators can enrich events and support detections, but indicators require validation to avoid false positives.

B. Microsoft Sentinel deployment and configuration

Sentinel configuration connects the workspace, data sources, detections, and response procedures into one operational workflow.

Enable Sentinel: Open the target Log Analytics workspace, select Microsoft Sentinel, and enable the service.
Add connectors: Select the required data connector, grant consent or permissions, configure the source, and verify that records arrive in the expected table.
Create detections: Write a KQL query, define its schedule and look-back period, select tactics and techniques, map entities, and configure incident creation.
Example query: