Unit 3: Storage and Compute Services - Subjective Questions

Amazon Simple Storage Service (Amazon S3) is a highly scalable object storage service used to store and retrieve data over the internet.

• Bucket: A logical container in which objects are stored. Each bucket has a globally unique name and is created in a specific AWS Region.
• Object: The actual data stored in S3, such as an image, document, video, or backup file. An object consists of data, metadata, and an object key.
• Object key: The unique name or path used to identify an object within a bucket. For example, documents/reports/report.pdf is an object key.
• Region: The geographical AWS location in which the bucket and its objects reside. Selecting a nearby Region can reduce latency and may support compliance requirements.
• Metadata: Information associated with an object, such as content type, creation details, and custom attributes.

S3 provides high durability, virtually unlimited scalability, access control, encryption, versioning, lifecycle management, and integration with many other AWS services.

Amazon S3 offers storage classes optimized for different access patterns and cost requirements:

• S3 Standard: Designed for frequently accessed data. It provides low latency, high throughput, and multi-Availability Zone resilience. Suitable for websites, mobile applications, and active datasets.
• S3 Intelligent-Tiering: Automatically moves objects between access tiers based on usage. It is useful when access patterns are unknown or change over time.
• S3 Standard-Infrequent Access: Intended for data accessed less frequently but requiring rapid retrieval. Suitable for backups and disaster recovery files.
• S3 One Zone-Infrequent Access: Stores data in one Availability Zone. It costs less but is less resilient than multi-AZ classes. Suitable for reproducible or secondary backup data.
• S3 Glacier Instant Retrieval: Intended for rarely accessed archival data that still requires millisecond retrieval.
• S3 Glacier Flexible Retrieval: Suitable for archives that can tolerate retrieval times ranging from minutes to hours.
• S3 Glacier Deep Archive: The lowest-cost class for long-term retention, where retrieval may take several hours.
• S3 Express One Zone: Provides very low latency and high request performance in a single Availability Zone for performance-sensitive workloads.

The correct choice depends on access frequency, retrieval-time requirements, resilience, minimum storage duration, and total cost.

Amazon S3 is designed to provide reliable and secure object storage:

• Durability: Most S3 storage classes are designed for 99.999999999% durability, commonly called eleven nines. Data is redundantly stored across multiple devices, and for most classes, across multiple Availability Zones.
• Availability: Availability represents the expected percentage of time during which the service remains accessible. It varies by storage class and is lower than durability.
• Consistency: S3 provides strong read-after-write consistency for object creation, overwrite, and deletion operations. After a successful write, subsequent reads return the latest data.
• Access control: Access can be managed using IAM policies, bucket policies, access points, and, where required, access control lists.
• Encryption in transit: HTTPS protects data while it travels between clients and S3.
• Encryption at rest: S3 supports server-side encryption using S3-managed keys, AWS KMS keys, or customer-provided keys. Client-side encryption is also possible.
• Public access protection: S3 Block Public Access settings help prevent accidental public exposure.
• Auditing: AWS CloudTrail and S3 access logging can record activity for governance and investigation.

Data can be transferred to and from S3 through several methods:

• AWS Management Console: Suitable for simple manual uploads and downloads of relatively small datasets.
• AWS CLI and SDKs: Appropriate for scripted, automated, or application-based transfers.
• Multipart upload: Divides a large object into parts that can be uploaded independently and in parallel. Failed parts can be retried without restarting the entire transfer.
• S3 Transfer Acceleration: Uses AWS edge locations and an optimized network path to accelerate long-distance internet transfers.
• AWS DataSync: Automates and accelerates transfers between on-premises storage, other storage services, and AWS.
• AWS Transfer Family: Provides managed SFTP, FTPS, FTP, and AS2 endpoints backed by S3.
• AWS Snow Family: Uses physical devices for very large datasets or locations with limited network connectivity.
• Direct Connect or VPN: Provides private or controlled connectivity between an organization and AWS.

Selection should consider:

1. Dataset size and number of files
2. Available bandwidth and transfer deadline
3. Geographical distance and latency
4. Security and compliance requirements
5. Need for online or offline transfer
6. Automation, retry, and validation requirements
7. Transfer, networking, and device costs

An S3 lifecycle policy is a set of rules that automatically manages objects throughout their lifetime. Rules can apply to an entire bucket or to selected objects based on prefixes, tags, or other filters.

• Transition action: Moves objects to a lower-cost storage class after a specified number of days. For example, objects may move from S3 Standard to S3 Standard-IA after 30 days and then to S3 Glacier Flexible Retrieval after 90 days.
• Expiration action: Deletes objects after a specified retention period. For example, log files may be deleted after 365 days.
• Version-related actions: A rule can transition or delete noncurrent object versions and remove expired delete markers.
• Incomplete multipart upload cleanup: A lifecycle rule can abort unfinished multipart uploads after a configured period.

Example policy design:

• Store application logs in S3 Standard initially.
• Transition them to S3 Standard-IA after 30 days.
• Transition them to S3 Glacier Flexible Retrieval after 90 days.
• Permanently delete them after 7 years.

Lifecycle policies reduce storage cost and operational effort, but minimum storage durations and retrieval charges must be considered.

S3 versioning preserves multiple variants of an object in the same bucket. Each stored variant receives a unique version ID.

• When an object is overwritten, S3 creates a new version instead of replacing the old version permanently.
• When a versioned object is deleted without specifying a version ID, S3 normally inserts a delete marker. The object appears deleted, but earlier versions remain stored.
• To recover the object, list its versions and delete the current delete marker, or copy a previous version so that it becomes the latest version.
• To permanently remove a particular version, the user must explicitly delete that version by its version ID.
• Versioning can be enabled and later suspended, but it cannot be returned to the original never-enabled state.

Benefits:

• Recovery from accidental deletion or overwrite
• Preservation of object history
• Improved data protection and auditability

Considerations: Every version consumes storage and may increase cost. Lifecycle rules should therefore be used to transition or expire old versions.

The three mechanisms solve related but different problems:

• S3 versioning: Retains multiple versions of an object within a bucket. It protects against accidental deletion and overwrite.
• S3 replication: Automatically copies eligible objects to another destination bucket. Same-Region Replication can support isolation and aggregation, while Cross-Region Replication can support geographical resilience and compliance. Versioning must be enabled on source and destination buckets.
• S3 Object Lock: Uses a write-once-read-many model to prevent object versions from being deleted or overwritten during a retention period. It supports governance and compliance retention modes, as well as legal holds.

Key distinction:

• Versioning provides history and recoverability.
• Replication provides additional copies in another bucket or Region.
• Object Lock provides immutability and retention enforcement.

A strong protection strategy may combine all three: version objects, replicate them to an isolated bucket, and apply Object Lock where regulatory retention is required.

A typical EC2 launch process includes the following steps:

1. Select a Region: Choose a Region based on latency, service availability, cost, and compliance.
2. Choose an AMI: Select an operating system and software image suitable for the workload.
3. Choose an instance type: Select the required combination of vCPU, memory, network capacity, storage capabilities, and processor architecture.
4. Configure networking: Choose a VPC, subnet, public IP settings, and, if needed, an IAM instance profile.
5. Configure storage: Add and size EBS volumes, choose volume types, and enable encryption.
6. Add user data: Provide a bootstrap script to install software and perform initial configuration.
7. Configure security groups: Allow only necessary inbound and outbound traffic.
8. Select an access method: Configure AWS Systems Manager Session Manager or, where needed, a key pair for SSH or Windows access.
9. Review and launch: Confirm the configuration and create the instance.
10. Verify status: Wait for instance and system status checks to pass.
11. Test the workload: Connect securely, inspect logs, and test the application endpoint.
12. Enable operations: Configure CloudWatch monitoring, backups, patching, alarms, and tags.

For production systems, infrastructure as code and Auto Scaling are generally preferred over repeated manual launches.

An Amazon Machine Image, or AMI, is a template used to launch EC2 instances. It commonly contains an operating system, boot volume configuration, software packages, and launch permissions.

Factors for selecting an AMI include:

• Operating system: Linux, Windows, or another supported platform.
• Processor architecture: The AMI must support the selected architecture, such as x86_64 or Arm64.
• Region availability: AMI IDs are Region-specific, although images can be copied between Regions.
• Source and trust: Prefer official AWS images, verified vendors, or organization-approved golden images.
• Security: Check patch level, hardening, known vulnerabilities, and unnecessary installed software.
• Licensing: Marketplace and commercial software images may include additional charges and usage terms.
• Virtualization and boot mode: Ensure compatibility with the intended instance family.
• Storage configuration: Review root-device type, volume size, encryption, and additional snapshots.
• Maintenance: Select an AMI with a defined update and deprecation process.

Organizations often build secured golden AMIs to achieve consistent, repeatable, and governed deployments.

EC2 instance families are grouped according to resource characteristics:

• General purpose: Provide a balanced ratio of compute, memory, and networking. Suitable for web servers, development environments, and small databases.
• Compute optimized: Provide a high ratio of CPU to memory. Suitable for batch processing, media encoding, scientific modeling, gaming servers, and high-performance web servers.
• Memory optimized: Provide large amounts of RAM. Suitable for in-memory databases, real-time analytics, and memory-intensive enterprise applications.
• Storage optimized: Provide high local storage throughput or IOPS. Suitable for distributed file systems, data warehousing, log processing, and high-performance NoSQL workloads.
• Accelerated computing: Include GPUs or specialized accelerators. Suitable for machine learning, graphics rendering, video processing, and scientific simulations.
• Burstable performance: Accumulate and spend CPU credits. Suitable for workloads with low baseline usage and occasional CPU bursts.

Selection should evaluate vCPU, memory, architecture, network bandwidth, EBS bandwidth, local storage, accelerator needs, and price. Benchmarking with realistic traffic is more reliable than selecting an instance based only on theoretical specifications.

EC2 user data is information supplied at instance launch to automate initial configuration. On Linux, it is commonly a shell script or cloud-init configuration. On Windows, it can contain PowerShell commands.

A simple Linux bootstrap script is:

#!/bin/bash

dnf install -y httpd

systemctl enable --now httpd

echo 'Application server is running' > /var/www/html/index.html

The script performs the following actions:

• Installs the Apache HTTP server.
• Configures the service to start during boot.
• Starts the service immediately.
• Creates a basic web page.

Important considerations:

• User-data scripts normally run with administrative privileges, so they must be reviewed carefully.
• Scripts should be idempotent, meaning repeated execution should not produce an incorrect state.
• Passwords and permanent credentials should not be placed in user data. Use IAM roles and AWS Secrets Manager or Parameter Store.
• Bootstrapping results should be checked through cloud-init or system logs.
• User data is commonly executed only during the first boot unless the instance is explicitly configured otherwise.

These controls secure different aspects of EC2 operation:

• Security groups: Stateful virtual firewalls attached to network interfaces. They should permit only required protocols, ports, and trusted sources. For example, a web server may accept HTTPS from users while allowing administrative traffic only through approved management paths.
• IAM roles: Supply temporary AWS credentials to applications running on an instance. They avoid embedding long-term access keys and should follow least privilege.
• Key pairs: Support public-key authentication for operating-system access, such as SSH. Private keys must be protected and rotated according to organizational policy.
• AWS Systems Manager Session Manager: Provides auditable shell access without opening inbound SSH or RDP ports. It can reduce exposure to the internet and centralize session logging.
• Patch Manager and State Manager: Help automate patching and enforce desired configuration.

A secure design combines restricted security groups, least-privilege IAM roles, encrypted storage, current patches, centralized logging, and controlled administrative access through Systems Manager.

Selection guidance:

• Choose EBS when an application requires durable low-latency block storage.
• Choose EFS when multiple Linux-based compute resources require a shared file system.
• Choose instance store when very fast temporary storage is useful and data loss can be tolerated or the data is replicated elsewhere.

The major EBS volume categories are:

• General Purpose SSD: Designed for a broad range of transactional workloads, boot volumes, development systems, and many databases. The current general-purpose option allows performance to be provisioned independently of capacity within supported limits.
• Provisioned IOPS SSD: Intended for critical workloads that require high, consistent IOPS and low latency, such as large relational or NoSQL databases.
• Throughput Optimized HDD: Designed for frequently accessed, throughput-intensive workloads such as big-data processing, log processing, and data warehouses. It is not suitable as a boot volume.
• Cold HDD: A low-cost option for less frequently accessed, throughput-oriented data. It is not intended for workloads requiring high random IOPS.

Selection should consider:

• Required IOPS and throughput
• Random versus sequential access
• Latency sensitivity
• Volume capacity
• Boot-volume requirements
• Expected burst behavior
• Cost per month and cost of provisioned performance

The selected volume should be monitored using CloudWatch, and performance should be adjusted when utilization or application latency indicates a bottleneck.

EBS snapshots are incremental point-in-time backups of EBS volumes stored and managed by AWS.

• The first snapshot contains all used blocks, while later snapshots store changed blocks relative to existing snapshot data.
• Snapshots can be used to create new EBS volumes in any Availability Zone within the Region.
• They can be copied to another Region or AWS account to support disaster recovery and isolation.
• Snapshot lifecycle policies or AWS Backup can automate creation, retention, copying, and deletion.
• Restored volumes may experience initial read latency because blocks are loaded as accessed. Fast Snapshot Restore can provide fully initialized performance where required.

Encryption:

• EBS encryption protects volume data, snapshots, and data moving between supported instances and volumes.
• AWS KMS keys manage encryption access and auditing.
• Volumes created from encrypted snapshots remain encrypted.

A complete recovery plan must define backup frequency, retention, cross-account or cross-Region copies, recovery procedures, and regular restoration tests. A snapshot that has never been successfully restored is not sufficient evidence of recoverability.

Amazon Elastic File System, or EFS, is a managed elastic file service that provides shared file storage through the Network File System protocol.

• Regional accessibility: A regional EFS file system can be accessed from multiple Availability Zones.
• Mount targets: A mount target is created in selected VPC subnets and provides an IP address through which clients mount the file system. High availability normally requires mount targets in each relevant Availability Zone.
• Elastic capacity: Storage capacity grows and shrinks automatically as files are added or removed.
• Performance: EFS offers performance and throughput options for general workloads and applications with higher parallel throughput requirements.
• Storage classes and lifecycle management: Infrequently accessed files can be moved to lower-cost tiers based on lifecycle policies.
• Network security: Security groups control network access to mount targets.
• Identity and authorization: IAM authorization and EFS access points can enforce application-specific identities and directories.
• Encryption: EFS supports encryption at rest and encryption in transit.

EFS is appropriate for shared content repositories, web-serving fleets, analytics, home directories, and containerized applications that need concurrent file access.

Only the being moved contributes to the saving.

Convert the transferred capacity to gigabytes:

Calculate the cost difference per GB-month:

Therefore, the estimated monthly saving is:

The estimated annual saving is:

Thus, the organization saves approximately 1008 per year in storage charges.

This is only a simplified estimate. A real decision must also consider:

• Retrieval fees
• Transition request charges
• Minimum storage-duration charges
• Data-access frequency
• Early deletion fees
• Monitoring or automation charges associated with the selected class

A Well-Architected storage design addresses all six pillars:

• Operational Excellence: Define storage resources through infrastructure as code, use consistent tagging, automate backups and lifecycle policies, and create operational runbooks.
• Security: Block unintended public S3 access, apply least-privilege policies, encrypt S3, EBS, and EFS data, protect KMS keys, log access, and use private connectivity where appropriate.
• Reliability: Enable S3 versioning where recovery is needed, use replication for additional isolation, create EBS snapshots, use multi-AZ EFS mount targets, and test data restoration.
• Performance Efficiency: Select suitable S3 storage and request patterns, match EBS volume performance to IOPS and throughput needs, and choose the correct EFS performance configuration.
• Cost Optimization: Apply S3 and EFS lifecycle policies, delete obsolete snapshots and unattached volumes, right-size EBS performance, and monitor storage growth.
• Sustainability: Remove redundant data, use managed elastic services, reduce unnecessary copies, and select efficient architectures that match demand.

The design must also classify data by sensitivity, access pattern, recovery point objective, recovery time objective, and retention requirement. Controls can then be selected according to measurable business needs.

A Well-Architected EC2 design for a production web application can include:

• Operational Excellence: Deploy instances using launch templates and infrastructure as code. Automate bootstrapping, patching, monitoring, and deployment. Store logs centrally.
• Security: Place instances in appropriate private subnets, restrict security groups, use IAM roles instead of static keys, encrypt volumes, harden AMIs, and use Systems Manager for administration.
• Reliability: Run instances across multiple Availability Zones behind a load balancer. Use Auto Scaling, health checks, automated replacement, and tested backup and recovery procedures.
• Performance Efficiency: Benchmark instance families, use current-generation processors, select appropriate networking and EBS performance, and scale horizontally according to demand.
• Cost Optimization: Right-size instances, use Auto Scaling to remove idle capacity, apply Savings Plans for stable usage, and use Spot Instances for fault-tolerant workloads.
• Sustainability: Increase utilization, remove idle resources, use efficient instance generations, and scale resources according to actual demand.

The application should avoid storing critical session or application state only on an individual instance. Externalizing state allows failed instances to be replaced safely and improves elasticity.

A suitable architecture can be designed as follows:

1. Static content: Store images, style sheets, scripts, and downloads in Amazon S3. Use appropriate encryption, bucket policies, versioning, and lifecycle rules. A content delivery service can cache content near users.
2. Dynamic traffic: Place an Application Load Balancer in public subnets across multiple Availability Zones.
3. Compute layer: Run EC2 instances in private subnets across at least two Availability Zones. Use an Auto Scaling group and a launch template containing the approved AMI, instance type, security settings, IAM role, and user data.
4. Block storage: Use encrypted EBS volumes for boot disks and application data requiring block-level access. Automate snapshots according to recovery objectives.
5. Shared files: Use EFS when all application instances need simultaneous access to the same file hierarchy. Create mount targets in each active Availability Zone.
6. Security: Apply least-privilege IAM permissions, tightly scoped security groups, private access paths where appropriate, KMS encryption, and centralized auditing.
7. Operations: Use CloudWatch metrics, logs, dashboards, and alarms. Use Systems Manager for patching and administrative sessions.
8. Reliability: Configure health checks, automatic replacement, multi-AZ deployment, versioned infrastructure, and tested restoration procedures.
9. Cost control: Right-size instances, scale with demand, transition old S3 and EFS data, and remove obsolete EBS snapshots.

This architecture separates static, block, and shared-file storage according to access requirements while making the compute layer replaceable and horizontally scalable.