Unit 6: AWS Storage and Database Services

I. Orientation — Storage and Data Management in AWS

AWS storage and database services separate data from physical infrastructure, allowing capacity, durability, performance, availability, and cost to be selected according to workload requirements.

• Core service models:
  • Block storage: Presents raw volumes to an operating system; represented by Amazon EBS.
  • Object storage: Stores objects in buckets and accesses them through APIs; represented by Amazon S3 and S3 Glacier classes.
  • File storage: Provides shared hierarchical file systems; represented by Amazon EFS.
  • Managed databases: Supply database engines without requiring customers to manage underlying servers.
• Shared responsibility: AWS manages physical facilities and service infrastructure, while customers manage data classification, permissions, encryption choices, backups, and workload configuration.
• Selection dimensions: Access pattern, latency, throughput, consistency, durability, availability, scalability, protocol, data model, and cost determine the appropriate service.
• Availability scope: Some resources, such as EBS volumes, belong to one Availability Zone; others, such as standard S3 buckets and regional EFS file systems, distribute data across multiple Availability Zones.
• Security controls: IAM policies, resource policies, network controls, TLS, and AWS Key Management Service encryption are common protection mechanisms.

II. Amazon Elastic Block Store — Persistent Block Storage

A. Amazon Elastic Block Store (EBS)

Amazon EBS provides persistent block-level volumes primarily for Amazon EC2 instances, making it suitable for boot disks, databases, and applications requiring low-latency storage.

• Storage model: An EBS volume appears to an instance as a block device, which can be partitioned and formatted with a file system such as ext4, XFS, or NTFS.
• Availability Zone scope: A volume and its attached EC2 instance normally must be in the same Availability Zone; a snapshot can create a replacement volume in another zone.
• Volume families:
  • General Purpose SSD (gp3): Balances price and performance for boot volumes and common applications.
  • Provisioned IOPS SSD (io2): Supports demanding transactional databases requiring predictable IOPS and durability.
  • Throughput Optimized HDD (st1): Suits large, sequential workloads such as logs and data processing.
  • Cold HDD (sc1): Supports infrequently accessed sequential data at lower cost.
• Snapshots: Point-in-time EBS snapshots are incremental, so after the first snapshot only changed blocks are stored; snapshots are maintained through AWS-managed S3 infrastructure.
• Performance measures: SSD workloads are commonly evaluated in IOPS, while HDD workloads depend heavily on throughput measured in MiB/s.
• Protection: Encryption using AWS KMS covers data at rest, data moving between the volume and instance, and snapshots created from the volume.
• Limitation: EBS is not inherently a shared file system; multi-host applications generally require EFS or a clustered system designed for shared block access.

III. Amazon S3 — Scalable Object Storage

A. Amazon S3

Amazon Simple Storage Service stores data as objects inside regional buckets and is designed for massive scale, high durability, and API-based access.

• Object structure: Each object has a key, data, and metadata; the combination of bucket and key identifies it, such as reports/2025/sales.csv.
• Scale and durability: An individual object can be as large as 5 TB, and S3 is engineered for 99.999999999% annual object durability.
• Consistency: S3 provides strong read-after-write consistency for object writes, overwrites, deletes, and listing operations.
• Storage classes:
  • S3 Standard: Frequent access with multi-Availability-Zone resilience.
  • S3 Intelligent-Tiering: Automatically moves objects among access tiers according to observed use.
  • Standard-IA and One Zone-IA: Lower storage cost with retrieval charges for infrequent access.
• Data management: Versioning retains earlier object versions, lifecycle rules transition or expire objects, and replication copies data within or between Regions.
• Access control: IAM policies, bucket policies, access points, and Block Public Access govern access; new buckets and objects are private by default.
• Common uses: Static website assets, media, backups, logs, data lakes, and application objects fit S3 better than operating-system disks.
• Limitation: S3 does not supply normal block semantics or a traditional file-system interface; applications usually access it through REST APIs or SDKs.

IV. Amazon EFS — Shared Elastic File Storage

A. Amazon EFS

Amazon Elastic File System provides managed, elastic file storage that multiple compute resources can mount concurrently.

• Interface: EFS uses the Network File System protocol, including NFSv4.1, and preserves directories, file names, and standard file permissions.
• Shared access: Multiple EC2 instances, containers, and supported serverless workloads can access the same files, making EFS useful for content repositories and shared application data.
• Regional design: Regional EFS file systems store data across multiple Availability Zones; mount targets provide network access within selected VPC subnets.
• Capacity: Storage grows and shrinks automatically as files are added or removed, avoiding manual volume provisioning.
• Performance options: Throughput can scale with stored data or be configured through elastic and provisioned throughput modes.
• Storage classes: EFS Standard supports active data, while EFS Infrequent Access and Archive classes reduce cost for colder files; lifecycle policies automate movement.
• Security: Security groups control network access, access points can enforce application identities and root directories, and encryption is available in transit and at rest.
• Limitation: EFS is principally designed for Linux-style NFS workloads; Windows-native SMB workloads generally require Amazon FSx for Windows File Server.

V. Amazon S3 Glacier — Long-Term Archival Storage

A. Amazon S3 Glacier

Amazon S3 Glacier storage classes reduce the cost of retaining rarely accessed data by accepting longer retrieval times or minimum storage durations.

• Glacier Instant Retrieval: Provides millisecond retrieval for archives that still require immediate access and has a 90-day minimum storage duration.
• Glacier Flexible Retrieval: Supports expedited, standard, and bulk retrieval options; standard retrieval commonly takes several hours.
• Glacier Deep Archive: Targets very long-lived records, uses a 180-day minimum storage duration, and commonly restores data within 12 to 48 hours.
• Lifecycle integration: An S3 lifecycle rule can transition an object from S3 Standard to a Glacier class after a specified age, such as 90 days.
• Archive use cases: Compliance records, historical media, scientific datasets, and disaster-recovery copies suit Glacier when retrieval is rare.
• Cost constraint: Retrieval charges, request charges, and early-deletion charges must be considered alongside the low storage price.
• Operational point: Archived objects in Flexible Retrieval or Deep Archive normally require a restore operation before their contents can be accessed.

VI. Storage Technology Selection — Matching Service to Workload

A. Storage technology selection

Storage should be selected by matching the application’s access interface and service-level requirements rather than choosing only by price.

• Interface decision:
  1. Block: Choose EBS for an EC2 boot volume or a database requiring block-level I/O.
  2. Object: Choose S3 for API-accessed files, backups, static assets, or data lakes.
  3. File: Choose EFS when multiple Linux clients require concurrent NFS access.
  4. Archive: Choose S3 Glacier classes when retention is long and access is uncommon.
• Latency and access frequency: Active transactional data favors EBS; frequently accessed objects favor S3 Standard; cold records favor Glacier Flexible Retrieval or Deep Archive.
• Sharing requirement: An EBS volume normally serves resources in one Availability Zone, whereas EFS supports shared file access across Availability Zones.
• Durability and recovery: Snapshots, S3 Versioning, replication, lifecycle rules, and cross-Region copies address different recovery-point and geographic-resilience requirements.
• Cost model: Compare provisioned capacity, requests, IOPS, throughput, data transfer, retrieval, and minimum-duration charges rather than storage cost per GB alone.
• Example: A web application might use EBS for its server boot disk, EFS for shared uploaded files, S3 for static images, and Deep Archive for seven-year compliance retention.

VII. Amazon RDS — Managed Relational Databases

A. Amazon RDS

Amazon Relational Database Service operates relational database engines while AWS handles routine infrastructure tasks such as provisioning, patching, backup, and hardware replacement.

• Supported engines: RDS supports PostgreSQL, MySQL, MariaDB, Oracle Database, Microsoft SQL Server, IBM Db2, and Amazon Aurora.
• Relational model: Tables, keys, constraints, joins, and SQL make RDS suitable for structured data and ACID transactions.
• High availability: A Multi-AZ deployment maintains a synchronous standby for failover; the standby is primarily an availability mechanism, not a read-scaling resource.
• Read scaling: Read replicas use asynchronous replication and can serve read-heavy traffic, although replication lag may occur.
• Backup and recovery: Automated backups support point-in-time recovery within the configured retention window, while manual snapshots remain until deleted.
• Security: RDS operates within a VPC and supports security groups, encryption with KMS, TLS connections, and engine-specific authentication controls.
• Limitation: Customers receive database-level control but not normal access to the underlying host operating system; specialized administrative requirements may therefore require self-managed databases on EC2.

VIII. Amazon DynamoDB — Serverless NoSQL Database

A. Amazon DynamoDB

Amazon DynamoDB is a serverless key-value and document database designed for predictable, low-latency performance at large scale.

• Primary keys: A table uses either a partition key alone or a composite key consisting of a partition key and sort key.
• Partitioning: DynamoDB hashes the partition key to distribute items; high-cardinality keys help prevent a “hot” partition from receiving disproportionate traffic.
• Items: Each item can contain flexible attributes and can be up to 400 KB.
• Capacity modes: On-demand mode charges per request, while provisioned mode configures read and write capacity with optional automatic scaling.
• Read behavior: Eventually consistent reads are the default; strongly consistent reads are available for tables and local secondary indexes within a Region.
• Indexes: Global secondary indexes provide alternative partition and sort keys, while local secondary indexes retain the table’s partition key.
• Features: DynamoDB Streams records item changes, transactions provide all-or-nothing operations, and global tables support multi-Region replication.
• Limitation: DynamoDB does not provide arbitrary SQL joins; table and key design must begin with known access patterns.

IX. Amazon Redshift — Analytical Data Warehousing

A. Amazon Redshift

Amazon Redshift is a managed, massively parallel processing data warehouse optimized for large-scale SQL analytics rather than frequent row-level transactions.

• Columnar storage: Values from the same column are stored together, improving compression and reducing reads for queries such as SUM(revenue).
• MPP architecture: A leader coordinates work across compute nodes and slices so large scans and aggregations execute in parallel.
• Physical design: Distribution choices place rows across nodes, while sort keys improve range filtering and reduce unnecessary block scanning.
• RA3 architecture: RA3 nodes separate managed storage from compute capacity, allowing frequently used data to be cached locally.
• External analysis: Redshift Spectrum queries structured data directly in S3 without loading every row into warehouse tables.
• Deployment choices: Provisioned clusters offer selected node capacity, while Redshift Serverless automatically supplies warehouse resources and charges according to use.
• Workload fit: Business intelligence, reporting, log analysis, and star-schema analytics suit Redshift.
• Limitation: High-volume, single-row transactional processing is better handled by RDS, Aurora, or DynamoDB.

X. Amazon Aurora — Cloud-Native Relational Database

A. Amazon Aurora

Amazon Aurora is an AWS-designed relational database engine compatible with MySQL or PostgreSQL and built around distributed, fault-tolerant storage.

• Compatibility: Existing MySQL or PostgreSQL drivers and many database tools can connect to the corresponding Aurora edition.
• Cluster architecture: Writer and reader database instances share a distributed cluster volume rather than maintaining independent full data copies.
• Resilience: Aurora stores six copies of data across three Availability Zones and can repair storage failures automatically.
• Scalability: Cluster storage grows automatically up to 128 TiB, and a cluster can use up to 15 Aurora Replicas for read scaling.
• Endpoints: The cluster endpoint directs writes to the current writer, while the reader endpoint distributes read connections among replicas.
• Availability: If the writer fails, Aurora can promote a replica, reducing recovery time compared with rebuilding a database server.
• Deployment options: Aurora Serverless v2 adjusts database capacity in fine-grained increments, while Aurora Global Database supports cross-Region reads and disaster recovery.
• Positioning: Aurora combines relational SQL and ACID transactions with cloud-oriented scalability, but it remains more schema-dependent than DynamoDB and less specialized for analytics than Redshift.