Unit 1: Introduction to DevOps (Foundational Concepts)

I. Orientation: What DevOps Is

DevOps (coined around 2009 at the Velocity conference and the first "DevOpsDays" in Ghent, Belgium) is a cultural and technical movement that unites software Development and IT Operations to shorten the delivery cycle while keeping systems reliable. It is not a tool, a job title, or a team; it is a set of practices and a mindset that dissolve the traditional wall between those who build software and those who run it.

• Core definition: A collaborative approach where development, operations, and quality assurance share ownership of a product across its entire lifecycle.
• Central problem it solves: The historical "wall of confusion" where developers threw code over to operations, and each blamed the other for failures.
• Two pillars: Culture (shared responsibility, trust, blameless learning) and Automation (removing manual, error-prone handoffs).
• Measure of success: Fast, frequent, and reliable delivery of value to users, tracked with metrics rather than opinion.
• Scope: Spans code, build, test, release, deploy, operate, and monitor — a continuous loop, not a one-way pipeline.

II. Evolution of Software Development

The journey from rigid sequential models to continuous delivery, driven by the need for speed without sacrificing stability.

A. Waterfall

Sequential model where each phase completes before the next begins.

• Structure: Requirements → Design → Implementation → Testing → Deployment → Maintenance, executed once, top to bottom.
• Strength: Clear documentation and predictable milestones suit fixed-scope projects (e.g., regulated or contract-bound systems).
• Weakness: Feedback arrives only at the end; a requirement error found in testing is expensive to fix. Cycles run months to years.

B. Agile

Iterative model that delivers working software in short increments.

• Origin: Agile Manifesto, 2001 — "individuals and interactions over processes and tools", "working software over comprehensive documentation".
• Mechanism: Work split into sprints (typically 1–4 weeks); requirements evolve; continuous customer feedback.
• Gap it left: Agile accelerated development but stopped at the handoff to operations — deployment remained slow and manual.

C. DevOps

Extends Agile principles across the operations boundary to the running system.

• Continuity: Adds Continuous Integration, Continuous Delivery, and continuous monitoring so software flows from commit to production without stalling.
• Contrast with Agile: Agile optimises how a team builds; DevOps optimises how the organisation builds, ships, and operates together.
• Result: Release frequency rises from months to days or hours, with automated safeguards.

III. Origin and Rationale of DevOps

How mounting friction between development and operations produced a new discipline.

A. Understanding the Origin of DevOps

DevOps arose to end the structural conflict between two teams with opposing incentives.

• The conflicting goals: Developers rewarded for change (new features); operations rewarded for stability (no downtime). These pull in opposite directions.
• Catalysts: Patrick Debois organised the first DevOpsDays (2009) after John Allspaw and Paul Hammond's talk "10+ Deploys per Day: Dev and Ops Cooperation at Flickr".
• Enabling forces: Cloud computing (on-demand infrastructure), virtualization, and Agile's short cycles made frequent releases feasible and made manual operations the bottleneck.

B. Benefits of Cross-Functional Teams and Automation

Combining diverse skills and removing manual toil is what makes rapid, reliable delivery possible.

1. Cross-functional teams: One team owns development, testing, deployment, and operations.
   • Shared ownership: "You build it, you run it" — the team that writes code also supports it in production, aligning incentives.
   • Faster feedback: Operational problems reach developers directly, shortening the time to diagnose and fix.
2. Automation: Machines perform repeatable tasks previously done by hand.
   • Consistency: An automated build or deploy runs identically every time, eliminating "works on my machine" drift.
   • Speed and safety: Automated tests and deployments cut human error and free engineers for higher-value work.

IV. DevOps Goals

The three outcomes every DevOps initiative aims to deliver.

A. Improved Collaboration

Breaking silos so development and operations work as one unit.

• Shared tooling and visibility: Common dashboards, chat-ops, and version control give everyone the same information.
• Blameless culture: Post-incident reviews focus on system causes, not individual fault, so teams surface problems freely.

B. Faster Delivery

Reducing the time from idea to production.

• Small batch sizes: Frequent small releases are easier to test and roll back than large, infrequent ones.
• Lead time: DevOps targets low lead time for changes — the interval from code commit to running in production.

C. Reliability

Keeping systems stable even as change accelerates.

• Automated testing and monitoring: Continuous checks catch regressions before and after release.
• Fast recovery: Emphasis on low Mean Time To Recovery (MTTR); teams recover quickly rather than assuming failure never occurs.

V. CALMS Framework

A model (coined by Jez Humble) for assessing DevOps maturity across five dimensions.

A. Culture

The human foundation of DevOps.

• Shared responsibility: Dev and Ops jointly own outcomes; success and failure are collective.
• Trust and blamelessness: Psychological safety enables honest reporting and experimentation.

B. Automation

Removing manual steps from the delivery pipeline.

• CI/CD pipelines: Automated build, test, and deploy stages triggered by code commits.
• Infrastructure as Code: Servers and environments defined in version-controlled files.

C. Lean

Applying lean manufacturing principles to software.

• Eliminate waste: Remove waiting, rework, and unnecessary handoffs from the value stream.
• Continuous improvement: Deliver in small increments and refine constantly (kaizen).

D. Measurement

Deciding with data, not opinion.

• Key metrics: Deployment frequency, lead time, change-failure rate, MTTR (the four DORA metrics).
• Feedback loops: Metrics guide where to improve the pipeline next.

E. Sharing

Spreading knowledge across the organisation.

• Transparency: Open documentation, shared runbooks, and post-mortems.
• Collaboration: Successes and lessons flow between teams, preventing repeated mistakes.

VI. DevOps Principles and Practices

The guiding ideas and the concrete techniques that realise them.

A. Principles

Foundational beliefs that shape DevOps behaviour.

• Systems thinking: Optimise the whole delivery flow, not one isolated team.
• Amplify feedback loops: Get information from right (operations) to left (development) fast.
• Continuous experimentation and learning: Treat failures as learning opportunities and improve daily.

B. Practices

Repeatable techniques that implement the principles.

• Continuous Integration (CI): Developers merge code frequently; each merge triggers an automated build and test.
• Continuous Delivery/Deployment (CD): Validated code is released to production automatically or at the push of a button.
• Infrastructure as Code (IaC): Provision infrastructure with tools like Terraform or Ansible for repeatability.
• Monitoring and logging: Continuously observe system health and performance in production.

VII. Phases of the DevOps Lifecycle

The continuous loop, often drawn as an infinity symbol, through which software flows repeatedly.

• Plan: Define features and requirements; track work (e.g., Jira, backlog boards).
• Code: Write and manage source in version control (Git).
• Build: Compile and package the application (Maven, Gradle, Docker images).
• Test: Run automated unit, integration, and regression tests (Selenium, JUnit).
• Release: Prepare and approve builds for deployment through the pipeline.
• Deploy: Push the release to production, often automated (Kubernetes, Jenkins).
• Operate: Run and manage the live system and its infrastructure.
• Monitor: Observe performance, logs, and user experience (Prometheus, Grafana); insights feed back into Plan, closing the loop.

VIII. Key DevOps Roles in Industry

The specialised roles that staff a DevOps organisation.

• DevOps Engineer: Builds and maintains CI/CD pipelines and automation tooling that bridge Dev and Ops.
• Site Reliability Engineer (SRE): Applies software engineering to operations; uses error budgets to balance new features against reliability (a discipline formalised at Google).
• Release Manager: Coordinates the flow of releases through environments and manages deployment scheduling.
• Automation/Build Engineer: Focuses on scripting builds, tests, and infrastructure provisioning.
• Security Engineer (DevSecOps): Integrates security scanning and compliance into the pipeline so security is continuous, not a final gate.

IX. Use Cases in Companies

How leading firms applied DevOps at scale to gain speed and resilience.

A. Netflix

Pioneered resilience engineering for a globally streamed service.

• Chaos Engineering: The Chaos Monkey tool randomly terminates production instances to prove the system tolerates failure.
• Microservices and automation: Hundreds of independently deployable services allow frequent, isolated releases.

B. Amazon

Achieved extreme deployment frequency through automation.

• Continuous deployment: Amazon reported deploying code roughly every 11.7 seconds at peak across its fleet via automated pipelines.
• Two-pizza teams: Small autonomous teams own their services end to end, minimising coordination overhead.

C. Google

Formalised operations as an engineering discipline.

• Site Reliability Engineering: Codified SRE with error budgets and service-level objectives (SLOs) to quantify acceptable risk.
• Automation over toil: SREs cap manual operational work so engineering time goes to durable improvements.