Unit 2: AWS Storage; Amazon Virtual Private Cloud

AWS storage services provide durable, scalable data persistence, while Amazon VPC provides the isolated virtual network that hosts and connects those resources. This unit covers object, file and hybrid storage, then the networking primitives that control addressing, routing and security inside a VPC.

I. Foundations

AWS organises its infrastructure into Regions (independent geographic areas) and Availability Zones (isolated data centres within a Region). Storage and networking services are built on this hierarchy.

• Storage classes: three paradigms — object (S3, Glacier, flat namespace of keys), file (EFS, FSx, hierarchical with POSIX/SMB semantics), and block (EBS, not covered here).
• Durability vs availability: durability is the probability data survives (eleven 9s = 99.999999999%); availability is the probability the service responds when queried (e.g. 99.99%).
• VPC scope: a VPC is Region-scoped; subnets are AZ-scoped. CIDR notation (10.0.0.0/16) defines all address ranges.
• Shared responsibility: AWS secures the infrastructure; the customer configures encryption, IAM policies, security groups and NACLs.

II. Amazon S3 — Object Storage Service

Amazon S3 (Simple Storage Service) stores objects in buckets within a flat global namespace, addressable over HTTP(S).

A. S3 Service Architecture

S3 is a distributed object store, not a filesystem.

• Bucket: a Region-scoped container with a globally unique name: my-bucket.s3.us-east-1.amazonaws.com.
• Object: the stored entity — up to 5 TB, comprising the data, a key (full name), version ID, and metadata.
• Key: the flat identifier; prefixes like logs/2026/ simulate folders but are string prefixes only.
• Max single PUT: 5 GB; larger objects use multipart upload (parts of 5 MB–5 GB).
• Consistency: strong read-after-write consistency for all operations.

B. S3 Durability and Availability

S3 replicates each object across multiple devices and AZs automatically.

• Durability: 99.999999999% (11 nines) across S3 Standard — statistically one object lost per 10 million every 10,000 years.
• Availability by class:
  • S3 Standard: 99.99% availability, ≥3 AZs, hot data.
  • S3 Standard-IA: 99.9%, infrequent access, lower storage cost but a retrieval fee.
  • S3 One Zone-IA: 99.5%, single AZ, cheaper but no AZ-failure protection.
  • S3 Intelligent-Tiering: auto-moves objects between tiers based on access patterns.

C. S3 Object Lifecycle

Lifecycle rules automate transition and expiration of objects to cut cost.

• Transition action: move objects to a cheaper class after N days, e.g. Standard → Standard-IA at 30 days → Glacier at 90 days.
• Expiration action: permanently delete objects after a set age.
• Versioning interplay: rules can expire noncurrent versions separately from current ones.
• Example rule:
  
  JSON

    { "Transitions": [{ "Days": 90, "StorageClass": "GLACIER" }],
      "Expiration": { "Days": 365 } }

D. Accessing S3 Objects

Objects are retrieved by key through several interfaces.

• REST API / SDK / CLI: aws s3 cp s3://bucket/key ./file.
• Access control: IAM policies, bucket policies, and ACLs; Block Public Access overrides permissive settings by default.
• Presigned URL: a time-limited URL granting temporary access without credentials.
• Static website hosting and S3 Transfer Acceleration (edge-optimised uploads via CloudFront) are additional access paths.

III. Amazon S3 Glacier — Cold Archival Storage

S3 Glacier provides low-cost storage for long-term archives where retrieval latency is acceptable.

• Vault/archive model: data stored as archives inside vaults (or as a Glacier S3 storage class).
• Retrieval tiers:
  • Expedited: 1–5 minutes.
  • Standard: 3–5 hours.
  • Bulk: 5–12 hours, cheapest.
• Glacier Deep Archive: lowest cost of all, 12-hour retrieval, for 7–10 year retention (compliance records).
• Vault Lock: enforces WORM (write-once-read-many) compliance policies.

IV. Amazon Elastic File System — Managed NFS

Amazon EFS is a fully managed, elastic shared filesystem using the NFSv4 protocol for Linux workloads.

• Shared access: mounted concurrently by thousands of EC2 instances across multiple AZs.
• Elastic capacity: grows and shrinks automatically; pay only for storage used.
• Mount: mount -t nfs4 fs-xxxx.efs.region.amazonaws.com:/ /mnt/efs.
• Storage classes: Standard and One Zone, each with Infrequent Access tiers via lifecycle policy.
• Use case: content management, shared home directories, web-serving where POSIX semantics are needed.

V. Amazon FSx — Managed Native File Systems

Amazon FSx delivers fully managed third-party and high-performance filesystems.

• FSx for Windows File Server: native SMB protocol, Active Directory integration, NTFS — for Windows applications.
• FSx for Lustre: high-performance parallel filesystem for HPC, machine learning, and media processing; integrates with S3.
• FSx for NetApp ONTAP and OpenZFS: feature-rich managed variants.
• Distinction from EFS: FSx targets specific protocols/performance profiles; EFS is Linux NFS only.

VI. AWS Storage Gateway — Hybrid Storage Bridge

Storage Gateway connects on-premises environments to AWS cloud storage transparently.

• File Gateway: presents an NFS/SMB mount backed by S3.
• Volume Gateway: block storage (iSCSI); cached keeps hot data local, stored keeps full copies local with async backup to S3.
• Tape Gateway: a Virtual Tape Library (VTL) backed by S3/Glacier, replacing physical tape backups.
• Purpose: low-latency local access with durable cloud backing.

VII. VPC Addressing and Topology

A VPC is a logically isolated virtual network defined by an IP address range and partitioned across AZs.

A. VPC CIDR Blocks

A CIDR block defines the private IP range the VPC can allocate.

• Format: 10.0.0.0/16 provides 65,536 addresses; the /16 is the network prefix length.
• Allowed size: between /16 (largest) and /28 (smallest, 16 addresses).
• Reserved per subnet: AWS reserves the first four and last IP in every subnet (network, VPC router, DNS, future, broadcast).

B. Subnets

A subnet is a sub-range of the VPC CIDR bound to one Availability Zone.

• Public subnet: its route table has a route to an Internet Gateway.
• Private subnet: no direct IGW route; outbound internet via NAT.
• Example: 10.0.1.0/24 (256 addresses, 251 usable) in us-east-1a.

C. Availability Zones

AZs are physically separate data centres providing fault isolation within a Region.

• Multi-AZ design: spread subnets across ≥2 AZs so an AZ failure does not take down the application.
• Scope: each subnet lives in exactly one AZ; a VPC spans all AZs in its Region.

VIII. Elastic Network Interfaces

An Elastic Network Interface (ENI) is a virtual network card attachable to EC2 instances.

A. Elastic Network Interfaces

The ENI holds the networking attributes of an instance.

• Attributes: a MAC address, one primary private IP, security groups, and an optional Elastic IP.
• Persistence: an ENI's attributes survive instance stop/start when the ENI is detached and reattached.

B. Primary and Secondary Private IP Addresses

An ENI carries one primary and optional secondary private IPs.

• Primary private IP: permanent for the ENI's lifetime; cannot be reassigned.
• Secondary private IPs: movable between ENIs — used for hosting multiple services or seamless failover.

C. Attaching Elastic Network Interfaces

ENIs can be moved between instances in the same AZ.

• Hot attach: attach to a running instance.
• Use case: a "management network" ENI, or redirecting traffic by detaching an ENI from a failed instance and attaching it to a standby.

IX. VPC Routing and Internet Access

Routing determines where subnet traffic flows; gateways provide external connectivity.

A. Internet Gateway

An Internet Gateway (IGW) is a horizontally scaled component enabling VPC-internet communication.

• One per VPC: attached to the VPC; performs 1:1 NAT for instances with public IPs.
• Requirement: a subnet is public only when its route table sends 0.0.0.0/0 to the IGW.

B. Route Table

A route table contains rules directing network traffic by destination.

• Local route: 10.0.0.0/16 → local is implicit and cannot be removed.
• Association: each subnet links to exactly one route table; 0.0.0.0/0 → igw-xxxx makes it public.

C. Network Address Translation

NAT lets private-subnet instances reach the internet without being reachable from it.

• NAT Gateway: managed, in a public subnet, highly available per AZ; route 0.0.0.0/0 → nat-xxxx from private subnets.
• Behaviour: translates many private IPs to one source; inbound connections are blocked.

X. VPC Security Controls

Two layered firewalls govern VPC traffic — one at the instance, one at the subnet.

A. Security Group

A security group is a stateful firewall at the ENI/instance level.

• Stateful: return traffic is automatically allowed regardless of outbound rules.
• Allow-only: rules permit traffic; there are no deny rules. Default denies all inbound, allows all outbound.
• Example: allow TCP 443 from 0.0.0.0/0.

B. Network Access Control List

A NACL is a stateless firewall at the subnet boundary.

• Stateless: inbound and outbound rules evaluated independently; return traffic needs explicit rules.
• Numbered rules: evaluated lowest-first; supports both allow and deny.
• Contrast with SG:
  1. Security group: stateful, instance-level, allow-only.
  2. NACL: stateless, subnet-level, allow and deny.

XI. Elastic IP Addresses

An Elastic IP (EIP) is a static, public IPv4 address allocated to your account.

• Reassignable: remap to another instance/ENI to mask failures — the address persists.
• Cost: billed when allocated but not associated, discouraging waste.

XII. AWS Global Accelerator

Global Accelerator improves availability and performance using the AWS global network.

• Anycast static IPs: two fixed IPs route users to the nearest edge, then over AWS backbone to endpoints.
• Benefit vs CloudFront: optimises TCP/UDP traffic (not just HTTP caching) with fast regional failover.

XIII. VPC Peering

VPC peering connects two VPCs so resources communicate using private IPs.

• Direct link: no IGW, NAT, or gateway needed; works cross-Region and cross-account.
• No transitive routing: if A peers B and B peers C, A cannot reach C through B.
• Requirement: non-overlapping CIDR blocks; routes added to both route tables.

XIV. Hybrid Cloud Networking

Hybrid networking integrates on-premises data centres with AWS VPCs.

• VPN connection: IPsec tunnel over the internet via a Virtual Private Gateway and customer gateway.
• AWS Direct Connect: a dedicated private physical link bypassing the internet for consistent, low-latency bandwidth.
• Transit Gateway: a hub connecting many VPCs and on-premises networks, enabling transitive routing at scale.