Unit 3: Storage and Compute Services

I. Orientation: Cloud Storage and Compute Foundations

Cloud architecture separates compute, which executes applications, from storage, which preserves application data. AWS supplies these capabilities as managed, API-driven services that can be provisioned on demand across Regions and Availability Zones.

• Compute principle: Amazon EC2 provides resizable virtual machines called instances, allowing processing capacity to scale with workload demand.
• Storage principle: AWS offers storage according to access model:
  • Object storage: Amazon S3 stores objects in buckets.
  • Block storage: Amazon EBS provides volumes used like virtual disks.
  • File storage: Amazon EFS supplies shared, hierarchical file systems.
• Regional structure: A Region is a geographic area containing multiple isolated Availability Zones; architecture across Zones improves resilience.
• Elasticity: Capacity can expand or contract dynamically instead of being permanently sized for peak demand.
• Shared responsibility: AWS secures the underlying cloud infrastructure, while customers configure identities, permissions, data protection, operating systems, and applications.
• Service selection: Storage and compute choices depend on performance, durability, availability, access frequency, latency, compatibility, and cost.
• Automation convention: APIs, infrastructure as code, launch templates, user data, and lifecycle rules make deployments repeatable.

II. Amazon S3 — Scalable Object Storage

A. Amazon S3 fundamentals and storage classes

Amazon Simple Storage Service (S3) stores data as objects and is designed for high scalability, durability, and controlled access.

• Core structure: An object consists of data, metadata, and a key; objects reside inside buckets.
  • A bucket name must be globally unique within an AWS partition.
  • The combination of bucket, optional prefix, and object key identifies an object.
• Object model: S3 is not a block device or conventional mounted file system; applications normally access it through HTTPS APIs, SDKs, the CLI, or the AWS Management Console.
• Scale and size: A single object may range from zero bytes to 5 TB; multipart upload is required for objects larger than 5 GB.
• Durability: S3 storage classes are generally designed for 99.999999999% durability, achieved through redundant storage across infrastructure.
• Security controls: IAM policies, bucket policies, S3 Block Public Access, access points, and encryption govern object access.
  • New objects are encrypted at rest automatically using server-side encryption.
  • TLS protects data in transit.
• S3 Standard: Intended for frequently accessed data requiring low latency, high throughput, and multi-Availability-Zone resilience.
• S3 Intelligent-Tiering: Moves objects among access tiers according to changing access patterns, making it suitable when frequency is unpredictable.
• S3 Standard-Infrequent Access: Provides multi-Zone storage for less frequently accessed data, with lower storage cost but retrieval charges and minimum-duration rules.
• S3 One Zone-Infrequent Access: Stores data in one Availability Zone; it costs less but is unsuitable when data cannot be recreated after Zone loss.
• Archive classes: S3 Glacier Instant Retrieval, Glacier Flexible Retrieval, and Glacier Deep Archive trade progressively slower access for lower storage cost.
• S3 Express One Zone: Offers very low-latency, high-performance access in one Availability Zone through directory buckets.

B. Data transfer

S3 data-transfer methods should be selected according to object size, data volume, network capacity, distance, and required completion time.

• Online transfer: The S3 console, AWS CLI, SDKs, and REST API transfer data over a network connection.
  • aws s3 cp copies individual files or objects.
  • aws s3 sync compares source and destination and transfers differences.
• Multipart upload: A large object is divided into independently uploaded parts, improving throughput and permitting failed parts to be retried without restarting the entire upload.
• Transfer Acceleration: S3 Transfer Acceleration uses AWS edge locations and the AWS backbone to accelerate long-distance transfers into or out of a bucket.
• Private connectivity: An S3 gateway VPC endpoint allows supported VPC resources to reach S3 without traversing the public internet.
• Managed migration: AWS DataSync automates large online transfers between on-premises storage, S3, EFS, and other supported services.
• Offline migration: AWS Snow Family devices transport large datasets when available bandwidth is inadequate.
• Integrity: S3 supports checksums such as CRC-based algorithms and SHA algorithms so clients can verify that transferred content was not corrupted.
• Cost boundary: Uploading data to S3 is generally free, while internet egress and some inter-Region transfers incur charges.

aws s3 sync ./records s3://example-records/2025/ \
  --storage-class STANDARD_IA

The command synchronizes local records content to the specified prefix and assigns uploaded objects to S3 Standard-IA.

C. lifecycle policies

S3 Lifecycle policies automate transitions and expiration so that object retention matches access requirements at lower operational cost.

• Lifecycle configuration: A bucket configuration contains rules selected by prefix, object tags, object size, or combinations of filters.
• Transition action: A rule moves objects to a less expensive class after a specified age, such as from S3 Standard to Standard-IA and later to Glacier Flexible Retrieval.
• Expiration action: A rule deletes current objects after the defined retention period.
• Version-aware actions: Separate rules can transition or permanently remove noncurrent versions.
• Incomplete uploads: Rules can abort incomplete multipart uploads after a chosen number of days, preventing abandoned parts from accumulating charges.
• Constraints: Transitions must respect each class’s minimum storage duration and supported transition path; premature deletion can produce minimum-duration charges.
• Worked example: Log objects may remain in Standard for 30 days, transition to Standard-IA, move to Glacier Flexible Retrieval after 90 days, and expire after 365 days.

{
  "Rules": [{
    "ID": "LogRetention",
    "Status": "Enabled",
    "Filter": {"Prefix": "logs/"},
    "Transitions": [{"Days": 30, "StorageClass": "STANDARD_IA"}],
    "Expiration": {"Days": 365}
  }]
}

D. versioning in S3

S3 Versioning preserves multiple versions of an object, helping recover from accidental overwrites, application errors, and unintended deletion.

• Version identifier: Once enabled, each new object version receives a unique version ID; an ordinary retrieval returns the current version.
• Overwrite behavior: Uploading data under an existing key creates a new current version instead of destroying the previous version.
• Deletion behavior: A normal delete inserts a delete marker, which becomes current and hides earlier versions without immediately removing them.
• Permanent deletion: A specific version is permanently removed only when a delete request names its version ID.
• Versioning states: A bucket can be unversioned, versioning-enabled, or versioning-suspended; suspension does not erase versions already retained.
• Protection controls: Multi-factor authentication delete can add protection to versioning-state changes and permanent version deletions in supported workflows.
• Cost implication: Every retained version consumes storage, so lifecycle rules should expire obsolete noncurrent versions and remove expired delete markers.
• Replication dependency: S3 replication requires versioning on source and destination buckets and can copy objects across buckets or Regions.

III. Amazon EC2 — Configurable Virtual Compute

A. Launching EC2 instances and choosing AMIs

Launching an EC2 instance requires selecting a machine image, hardware capacity, network placement, storage, permissions, and security settings.

• Amazon Machine Image: An AMI is a Region-specific template containing an operating system, software configuration, architecture, launch permissions, and block-device mapping.
• AMI sources: Choices include AWS-provided images, AWS Marketplace products, community images, and private organizational images.
• Selection criteria: The AMI must match the required CPU architecture, operating system, virtualization support, licensing model, patch level, and application dependencies.
• Launch configuration: The workflow specifies an instance type, VPC subnet, security groups, IAM role, key pair or alternative access method, EBS volumes, and monitoring options.
• Network security: A security group is stateful; inbound rules permit required traffic, while response traffic is automatically allowed.
• Access principle: An EC2 IAM role supplies temporary application credentials and avoids embedding long-term access keys on the instance.
• Repeatability: Launch templates record versioned launch settings and support Auto Scaling groups, consistent rebuilds, and controlled configuration updates.
• Operational choice: Public IP addresses are unnecessary for private application tiers that use load balancers, NAT gateways, VPC endpoints, or Systems Manager.

B. EC2 instance types

An EC2 instance type defines a particular combination of virtual CPUs, memory, networking, storage capabilities, and processor architecture.

• Naming convention: In m7i.large, m identifies the family, 7 the generation, i an Intel-based variant, and large the size.
• General purpose: Families such as M provide balanced compute, memory, and networking for application servers and common enterprise workloads.
• Burstable performance: T-family instances accumulate and consume CPU credits, fitting workloads with low baseline CPU use and occasional bursts.
• Compute optimized: C-family instances provide a high compute-to-memory ratio for batch processing, media encoding, gaming servers, and scientific modeling.
• Memory optimized: R, X, and related families serve memory-intensive databases, caches, and real-time analytics.
• Accelerated computing: P, G, Inf, and Trn families provide GPUs or specialized accelerators for graphics, machine learning inference, and model training.
• Storage optimized: I and D families support workloads requiring high local I/O rates, dense storage, or low-latency instance-store access.
• Right-sizing: CloudWatch metrics such as CPU utilization, network throughput, and EBS performance reveal whether capacity should be resized.
• Purchase distinction: Instance type controls hardware capacity; On-Demand, Savings Plans, Reserved Instances, and Spot Instances control pricing or capacity commitments.

C. user data, and configuration

EC2 user data supplies launch-time instructions that bootstrap an instance into its intended operational role.

• Execution model: On Linux AMIs, cloud-init commonly runs user-data scripts as root during the first boot; exact behavior depends on the AMI.
• Typical tasks: Scripts install packages, retrieve configuration, register services, create files, and start application processes.
• Idempotence: Commands should tolerate retries and repeated execution, for example by testing whether a package or configuration already exists.
• Secrets boundary: User data should not contain plaintext passwords or permanent access keys because instance-authorized users may retrieve it.
• Configuration services: Systems Manager Parameter Store or Secrets Manager can supply controlled configuration and secrets through an IAM role.
• Troubleshooting: Linux cloud-init output is commonly available in /var/log/cloud-init-output.log.
• Metadata security: Instance Metadata Service Version 2 uses session-oriented tokens and should be required where compatible.

#!/bin/bash
dnf install -y nginx
systemctl enable --now nginx
echo "Healthy EC2 node" > /usr/share/nginx/html/index.html

This script installs NGINX, enables it at boot, starts it immediately, and creates a basic response page.

IV. EC2 Storage — Block and Shared File Services

A. Storage options such as EBS and EFS for EC2

EC2 applications choose between persistent block storage, shared file storage, and temporary local storage according to access pattern and durability requirements.

1. Amazon EBS

   • Block model: An EBS volume behaves like a virtual disk and is attached to EC2 through the operating system.
   • Availability Zone scope: A volume is created within one Availability Zone and normally attaches to instances in that Zone.
   • Volume families: General Purpose SSD (gp3) suits common workloads; Provisioned IOPS SSD (io2) serves latency-sensitive databases; HDD options support throughput-oriented workloads.
   • Persistence: EBS data persists independently of instance stopping; deletion at termination depends on the volume’s configured mapping.
   • Protection: Point-in-time EBS snapshots are incremental and stored through an AWS-managed regional service; snapshots can create volumes in other Zones.
   • Encryption: AWS KMS-backed EBS encryption protects volume data, snapshots, and data moving between supported instances and volumes.

2. Amazon EFS

   • File model: EFS provides a managed Network File System interface, allowing multiple Linux instances to access shared directories concurrently.
   • Regional design: Regional EFS storage classes distribute data across multiple Availability Zones; One Zone classes provide a lower-cost single-Zone alternative.
   • Elastic capacity: The file system grows and shrinks automatically as files are added or removed.
   • Use cases: Shared web content, content management, user directories, and container workloads benefit from common file access.
   • Connectivity: Mount targets provide VPC access, and security groups control NFS traffic, normally on TCP port 2049.

• Instance store contrast: Instance-store disks offer temporary, host-attached storage; data can be lost when the instance stops, terminates, or its host fails.
• Selection rule: Choose EBS for boot volumes and low-latency block access, EFS for concurrent shared file access, and instance store for disposable caches or temporary processing data.

V. Architectural Quality — AWS Well-Architected Guidance

A. Applying the Well-Architected Framework to storage and compute layers

The AWS Well-Architected Framework evaluates architecture through six pillars and converts workload requirements into measurable storage and compute decisions.

• Operational excellence: Manage AMIs, launch templates, lifecycle configurations, alarms, and infrastructure through version-controlled automation; use observability to understand workload behavior.
• Security: Apply least-privilege IAM roles, private networking, security groups, encryption with KMS, S3 Block Public Access, patch management, and auditable API activity through CloudTrail.
• Reliability: Deploy stateless EC2 capacity across multiple Availability Zones behind load balancing; use Auto Scaling, S3 versioning, EBS snapshots, and tested recovery procedures.
• Performance efficiency: Select instance families from measured CPU, memory, network, and storage demand; match S3 classes, EBS volume types, and EFS performance settings to access patterns.
• Cost optimization: Right-size instances, stop unused development capacity, use suitable purchase models, delete unattached EBS volumes, and transition aging S3 objects through lifecycle rules.
• Sustainability: Increase utilization, adopt efficient current-generation processors, scale capacity with demand, and remove unnecessary stored data and idle compute.
• Trade-off analysis: Architecture must balance objectives rather than maximize one metric; multi-Zone redundancy increases resilience but may increase storage and transfer costs.
• Continuous review: CloudWatch metrics, AWS Config evaluations, Cost Explorer data, Compute Optimizer recommendations, and recovery tests provide evidence for recurring architectural improvement.