Unit 6: Understanding of Monitoring, Logging, Security & DevOps Best Practices

I. Orientation: The Feedback-Driven Operations Mindset

DevOps closes the loop between development and operations by treating running systems as continuous sources of feedback. Monitoring, logging, and observability supply the signals; security and reliability engineering govern how teams act on them.

• Core principle: Ship fast, but measure everything so failures surface in seconds, not customer complaints.
• Three signal types: Metrics (numbers over time), logs (discrete events), traces (request journeys) — collectively "observability."
• Shift-left security: Security controls move earlier into the pipeline (DevSecOps) rather than being bolted on at release.
• Reliability as a discipline: SRE applies engineering rigour and quantified targets (SLIs/SLOs) to operations.
• Automation over toil: Repetitive manual work is minimised or eliminated, culminating in GitOps, AIOps, and NoOps trends.

II. Need for Monitoring in DevOps

Monitoring answers "is the system healthy right now?" and is the precondition for fast, safe delivery.

A. Why Monitoring Is Essential

• Rapid detection: Alerts fire on threshold breaches (e.g., CPU > 85%, error rate > 1%) before users notice.
• Deployment validation: Canary and blue-green rollouts rely on live metrics to decide promote-or-rollback.
• The DORA metrics: Deployment frequency, lead time, change-failure rate, and MTTR (mean time to recovery) are all measured through monitoring data.
• Capacity planning: Trend lines on memory and request volume forecast when to scale.
• Accountability: Shared dashboards give Dev and Ops one version of truth, dissolving the "works on my machine" blame cycle.

III. Monitoring Tools: Prometheus & Grafana (Overview)

A near-standard open-source pairing: Prometheus collects and stores metrics; Grafana visualises them.

A. Prometheus — Metrics Collection and Storage

• Pull model: Prometheus scrapes HTTP /metrics endpoints on targets at fixed intervals (default 15s), unlike push-based agents.
• Time-series data: Each sample is metric_name{labels} value timestamp, e.g. http_requests_total{method="GET"} 1027.
• PromQL: Query language for aggregation and rates.

rate(http_requests_total[5m])

• Computes per-second request rate averaged over the last 5 minutes.
• Alertmanager: Companion service that deduplicates, groups, and routes alerts to email, Slack, or PagerDuty.

B. Grafana — Visualisation and Dashboards

• Data-source agnostic: Connects to Prometheus, Elasticsearch, InfluxDB, and more in one dashboard.
• Panels: Graphs, gauges, and heatmaps built from queries; e.g., a panel plotting 95th-percentile latency.
• Alerting UI: Visual alert rules with defined thresholds and notification channels.
• Together: Prometheus = brain (data + rules), Grafana = eyes (display) — a common exam contrast.

IV. Logging Fundamentals: ELK Stack

Logs record discrete, timestamped events; the ELK Stack ingests, stores, and searches them at scale.

A. Elasticsearch — Storage and Search

• Purpose: Distributed, JSON-document search engine indexing log data for sub-second full-text queries.
• Inverted index: Maps terms to documents, enabling fast lookups like "all logs containing NullPointerException."

B. Logstash — Ingestion and Transformation

• Pipeline stages: input → filter → output.
• Parsing: Grok filters convert unstructured lines into structured fields.

filter { grok { match => { "message" => "%{IP:client} %{WORD:method}" } } }

• Extracts client IP and HTTP method from a raw line.

C. Kibana — Visualisation and Exploration

• Discover view: Interactive search and filtering across indexed logs.
• Dashboards: Bar charts and maps built on Elasticsearch queries, e.g. errors grouped by service.
• Beats note: Lightweight shippers (Filebeat) often replace heavy Logstash agents at the edge, forming the "Elastic Stack."

V. Observability: Metrics, Logs, Traces

Observability is the ability to infer a system's internal state from its external outputs — the "three pillars."

A. Metrics

• Definition: Numeric measurements aggregated over time intervals.
• Traits: Cheap to store, ideal for alerting and trends, e.g. requests_per_second = 4200.
• Limitation: Tell you what is wrong (latency up) but not why.

B. Logs

• Definition: Immutable, timestamped records of discrete events.
• Traits: Rich context per event; good for root-cause detail, e.g. a stack trace at 14:03:22.
• Limitation: Volume and cost grow rapidly; hard to aggregate.

C. Traces

• Definition: End-to-end records of a single request as it crosses services.
• Spans: Each hop is a span with duration; assembled into a trace via a shared trace ID.
• Tools: Jaeger, Zipkin; standard: OpenTelemetry.
• Value: Pinpoints the slow service in a microservice chain — the gap metrics and logs alone leave.

VI. Introduction to DevSecOps

DevSecOps integrates security as a shared responsibility across the whole delivery lifecycle rather than a final gate.

A. Principles and Purpose

• Shift-left: Security testing starts at coding, not post-deployment.
• Everyone owns security: Developers, ops, and security engineers collaborate continuously.
• Automation: Security checks run automatically inside CI/CD pipelines.

B. Practices and Tooling

• SAST: Static Application Security Testing scans source code for flaws (e.g., SonarQube, Checkmarx).
• DAST: Dynamic testing probes the running app (e.g., OWASP ZAP).
• SCA: Software Composition Analysis flags vulnerable dependencies (e.g., Snyk, Dependabot).
• Secrets management: Vault or sealed secrets keep credentials out of code.
• Container scanning: Trivy or Clair inspect images before deployment.

VII. Introduction to Site Reliability Engineering (SRE)

SRE, originated at Google, applies software-engineering principles to operations to make systems scalable and reliable.

A. Core Concepts

• SLI (Service Level Indicator): A measured metric, e.g. request success ratio = 99.95%.
• SLO (Service Level Objective): The target for an SLI, e.g. 99.9% availability.
• SLA (Service Level Agreement): The contractual promise to customers, with penalties.
• Error budget: 1 − SLO; a 99.9% SLO permits 0.1% failure (~43 min/month) to "spend" on releases.

B. Operating Practices

• Toil reduction: Automate repetitive manual work; SREs cap toil at ~50% of time.
• Blameless postmortems: Failures are analysed for systemic cause, not individual fault.
• Release freeze rule: When the error budget is exhausted, feature launches pause until reliability recovers.

VIII. DevOps Best Practices

A consolidated set of habits that sustain the culture and pipeline.

A. Foundational Practices

• CI/CD: Automate build, test, and deploy so changes reach production continuously.
• Infrastructure as Code (IaC): Define infrastructure declaratively (Terraform, Ansible) for reproducibility.
• Version everything: Code, config, and infra live in Git for audit and rollback.

B. Cultural and Operational Practices

• Automate testing: Unit, integration, and security tests gate every merge.
• Monitor and alert: Instrument before shipping; no silent failures.
• Small, frequent releases: Reduce blast radius and simplify rollback.
• Blameless collaboration: Break Dev/Ops silos; share on-call and ownership.

IX. Real Industry Case Studies

Two landmark examples of reliability and resilience engineering in practice.

A. Google SRE

• Origin: Coined by Ben Treynor Sloss (~2003); "what happens when a software engineer runs operations."
• Error-budget model: Balances velocity against reliability using the 1 − SLO budget described above.
• Impact: Codified SLI/SLO/SLA and blameless postmortems now adopted industry-wide.

B. Netflix Chaos Engineering

• Definition: Deliberately injecting failures into production to prove resilience.
• Chaos Monkey: Tool that randomly terminates live instances to force redundancy and graceful degradation.
• Simian Army: Broader suite — Latency Monkey (delays), Chaos Kong (region outage) — testing larger failures.
• Principle: "The best way to avoid failure is to fail constantly," building antifragile systems.

X. Future of DevOps: GitOps, AIOps, NoOps

Emerging models that push automation further.

A. GitOps

• Definition: Git repository as the single source of truth for declarative infrastructure and apps.
• Mechanism: An operator (Argo CD, Flux) continuously reconciles cluster state to match Git.
• Benefit: Every change is a reviewed, auditable commit; rollback = git revert.

B. AIOps

• Definition: Applying AI/ML to operations data for automated detection and response.
• Capabilities: Anomaly detection, alert correlation to cut noise, and predictive failure warnings.
• Value: Turns overwhelming telemetry into actionable insight, reducing MTTR.

C. NoOps

• Definition: The aspirational state where operations are so automated that no dedicated ops team is needed.

• Enabler: Serverless and fully managed platforms (AWS Lambda) abstract away infrastructure.

• Reality check: Fully unattended operations remain largely theoretical; governance and edge cases still require humans.

• Trajectory: All three trends share one direction — shrinking manual toil while widening the reach of automated, code-defined, and increasingly intelligent operations.