> For the complete documentation index, see [llms.txt](https://mamawhocode.gitbook.io/aws/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://mamawhocode.gitbook.io/aws/services/storage/s3.md).

# S3

[FAQs](https://aws.amazon.com/s3/faqs/?saa=sec\&sec=prep) | [SSE-S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html) |

## Overview

![](https://www.awsgeek.com/Amazon-S3/Amazon-S3.jpg)

## Features

* Support LIST/GET/PUT/COPY/POST/DELETE

### S3 - Static Website Hosting

* S3 can host *static* websites and have them accessible on the Internet.
* The website endpoint will be: <http://bucket-name.s3-website-aws-region.amazonaws.com>
  * none SSL
  * included bucket-name & region-name in the endpoint link.

### S3 - Bucket Versioning

* Helps you recover accidental, such as: overwrite & delete.
* Can be enabled at *bucket level*.
* Turn ON versioning is a best practice.
* When not enable = `Suspend`
* Before enabling versioning, all the `Version ID` of objects are `NULL`

```bash
# list all objects
aws s3 ls s3://bucketname

# list all versioning objects
aws s3api list-object-versions --bucket bucketname
```

{% hint style="info" %}
After enabling Bucket Versioning, you might need to update your lifecycle rules to manage previous versions of objects -> Only apply to newly create objects.
{% endhint %}

{% hint style="danger" %}
After suspend Bucket Versioning

* Lifecycle rules set for previous object versions will still apply.
* Existing objects in your bucket do not change.
* Newly added objects with the same name as an existing object replaces the existing object.
  {% endhint %}

### **S3 - Analytics**

Used to ***analyze storage access patterns*** to help you decide when to transition the right data to the right storage class.

### S3 - Replication

* MUST enable [Versioning](#bucket-versioning) in source & destination buckets.
* Copy is ASYNC

### S3 - CORS

* Need to enable CORS headers if client does cross-origin request.

![](https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2Fgit-blob-e6efc82fcad776437e419b23de6e02701348dd8c%2Ffigure_20230513184234.png?alt=media)

### S3 - Event Notifications

to receive notifications when an event happen in your S3 bucket&#x20;

<figure><img src="https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2Fgit-blob-a9fa868f2ac082fdf5c52fbd90042a574dbec14f%2FS3_20230523234200.png?alt=media" alt=""><figcaption></figcaption></figure>

### S3 - Pre-signed URLs

Provide *<mark style="color:red;">temporary access</mark>* URL to your private bucket.

Url expiration varies by the way you generate the URLs

* S3 console: 1 min \~ 12 hours
* AWS CLI: 3600sec \~ 168 hours

**Use cases**: allow only logged-in users to see you premium videos.

### S3 - Object Lock

* Object versioning must be `enabled`.
* Block an object version deletion for a specified amount of time.
* Retention mode
  * Compliance (strick mode): can't be overwritten or deleted by any user, even root user
  * Governance (softer): most user can't overwrite or delete an object.

### S3 Glacier - Vault Lock

refer [Vault Lock](/aws/services/storage/s3/glacier.md).

### S3 Storage Lens

A fully managed S3 storage <mark style="color:red;">analytics solution</mark> that provides a comprehensive view of&#x20;

* object storage usage&#x20;
* activity trends
* recommendations to optimize costs.&#x20;

Storage Lens allows you to analyze object *<mark style="color:red;">access patterns</mark>* across all of your S3 buckets and generate detailed metrics and reports.

### S3 - Object Lambda

Allows you to add your own code to S3 GET requests to modify and process data as it's being returned to an application.

***Use cases***: data needs to be transformed on-the-fly, redact the PII from the data.

<figure><img src="https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2FCKlwBLW0aTI9clu09qHv%2Fimage.png?alt=media&amp;token=de9da2bd-7f4c-46f7-b37a-ea2e7c4ad8f1" alt="" width="563"><figcaption><p>S3 Object Lambda</p></figcaption></figure>

## Pricing

### S3 Classes

Below is the pricing order. The first one is the most expensive one.

<figure><img src="https://media.tutorialsdojo.com/public/lifecycle-transitions-v2.png" alt=""><figcaption><p>Cost-effective classes</p></figcaption></figure>

* Standard
* Standard Infrequent Access
  * lower cost than `Standard`
  * use cases: disaster recovery, backups
* Intelligent-Tiering: `automatically move` your data to infrequent access tier S3 Standard-IA
* One Zone IA
  * 11 9's in single AZ, but data lost when AZ is detroyed.
  * use cases: storing backup data of your on-premises, or data that can recreated.
* Glacier: low cost storage used for achiving/backup
  * Glacier Instant Retrieval
  * Glacier Flexible Retrieval
* Glacier Deep Archive: long term storage

![S3 classes](https://zesty.co/wp-content/uploads/2022/04/amazon-s3-aws-storage-classes.png)

### S3 Lifecycle policy

* Help optimize S3 storage cost
* Ex: transit all objects of a bucket from Standard class -> Standard-IA after 6 months uploading.

### Storage class analysis

* Help to decide when to transit to the right class
* Recommendation for `Standard` & `Standard-IA`. Not work for One-Zone-IA or Glacier.

### S3 Requester Pay

* In requester pay pattern, the owner still in charge for storage cost. ![](https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2Fgit-blob-aa5a55622f4c1b051a368903677afe425d9166f5%2FS3_20230524050658.png?alt=media)

## Performance

* S3 scales per `prefix`. Request per second as below:
  * 3,500 PUT/COPY/POST/DELETE
  * 5,500 GET/HEAD
* Latency between 100-200ms
* Support FOLDER concept to `group` objects.
* Use as many prefixes as posible to achieve the required throughput and disired performance

![](https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2Fgit-blob-ce8992e0ca54f69c641be88c91ef6ac6b00783d0%2FS3_20220501051913.png?alt=media)

Object may be replicated accross AZs, but within a single region. S3-IA object can be in 1 AZ.

### S3 - Transfer acceleration

* A bucket-level feature.
* Use **S3 Transfer Acceleration** to enable fast, easy and secure transfer of files over <mark style="background-color:yellow;">LONG distance</mark>. It will transfer files to an `AWS Edge location` of target S3 bucket.\
  -> *<mark style="color:red;">Speed up 50-500%</mark>*
* ***Use cases***:
  * Need to collect data from various locations.

### S3 Select & Glacier select

* Using Server-side filtering (***simple SQL***) for better performance & less transfer, CPU cost at client. \
  You can perform S3 Select to query only the necessary data inside the CSV files based on the bucket's name and the object's key.

  <figure><img src="https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2Fgit-blob-da98e1c63bd132f98579796666a9037757c789b7%2FS3_20230524053636.png?alt=media" alt=""><figcaption></figcaption></figure>
* For more complex queries, consider using Athena.
* Supports CSV, JSON, and Parquet.
* Use cases: retrieve only a subset of data, best for simple SQL (no JOIN, no function, no array...)

### S3 Batch operation

* Perform batch operations on existing S3 objects.
* Can use `S3 Inventory` to get the list of object. Then use [S3 Select](#s3-select--glacier-select) to filter objects. ![](https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2Fgit-blob-b3b8b62cb9997a6ee00e6d0658b890f9444f2257%2FS3_20230524054708.png?alt=media)

### S3 Inventory

S3 Inventory provides a report of your S3 objects and their corresponding metadata on a daily or weekly basis for a specified S3 bucket or a shared prefix.&#x20;

These reports include the type of server-side encryption each object is using, along with its replication status.

#### **Features**

* Configurable to include all or specific object versions.
* Can report on various metadata fields such as size, last modified date, storage class, and encryption status.
* Supports output in CSV, ORC, and Parquet formats, enabling straightforward integration with analytics tools.

## Security

* User-based: IAM policies
* Resource-based:
  * [Bucket policy](#glossary): ex: allow cross-account access
  * Object ACL (can be disable)
  * Bucket ACL (can be disable)
* Encryption data at rest: `SSE-S3`, `SSE-KMS`, `SSE-C (Customer provided key)`
* Encryption data in transit: TLS
* Access to the `most recent` data immediately after a write (create or overwrite)

### S3 - MFA delete

To avoid accidental deletion in S3 bucket:

* Enable [versioning](#s3-bucket-versioning)
* Enable MFA delete

### S3 - Encryption

<table><thead><tr><th width="137">Method</th><th width="167">Key management</th><th width="182">Encryption process</th><th>Extras</th></tr></thead><tbody><tr><td>Client-side</td><td>you</td><td>you</td><td>None</td></tr><tr><td>SSE-C</td><td>you</td><td>S3</td><td>None</td></tr><tr><td><a href="#trivia"><mark style="color:red;">SSE-S3</mark></a></td><td>S3</td><td>S3</td><td>AES-256</td></tr><tr><td>SSE-KMS</td><td>S3 &#x26; KMS</td><td>S3</td><td><ul><li>Rotation control</li><li>Role seperation</li></ul></td></tr></tbody></table>

By providing ***S3 object key*** and the ***encryption key***, you can use GetObject API to download encrypted object.

{% hint style="danger" %}
Amazon S3 now applies server-side encryption with Amazon S3 managed keys (<mark style="color:red;">**SSE-S3**</mark>) as the ***base level of encryption*** for every bucket in Amazon S3. You only need to use the `x-amz-server-side-encryption` header if you want to override the default SSE-S3 encryption and use a different encryption option like SSE-KMS or SSE-C
{% endhint %}

{% hint style="info" %}
**Note**

* Server-side encryption encrypts only the object data, not the object metadata.
* SSE-KMS supports symmetric keys, not asymmetric keys.
  {% endhint %}

### SSE-KMS limitation

<img src="https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2Fgit-blob-797cc6c01f2d167c6579716a504fde2314a16f15%2FS3_20230524052712.png?alt=media" alt="" width="326">

* KMS have limitation of request per second
  * When download, it call Decrypt KMS API
  * When upload, it calls `GenerateDataKey` KMS API
* KMS quota different between region: 5500, 10000, 30000
  * increase the quota by making request at Service Quota console

## Best practices

* Encrypt your data either on the client-side
* Turn on `versioning`
* Consider using multipart uploads (divide in parts & parallel uploads) for
  * object that are over 100MB.
  * if > 5GB, `multi-part upload` is a MUST.

## Trivia

* Bucket name is `GLOBALY` unique. But bucket are defined at REGIONAL level.
* Maximum of an object is 5TB.&#x20;
* Multi-part upload is recommended for file > 100MB. If a file larger than 5GB, `multi-part upload is a must.`
* S3 provides <mark style="color:red;">no API</mark> that can search for objects based on object metadata.
* Naming convention
  * No UPPERCASE, no underscore
  * 3-63 chars long
  * Not an IP
  * Must start with lowercase letter or number
  * Must NOT start with prefix xn--
  * Must NOT end with suffix -s3alias
* Using <mark style="background-color:orange;">Athena</mark> to query data in S3 by *standard* SQL.
* To achieve more requests per second, *increase prefixes* in your bucket.
* `"aws:SecureTransport":"false"` is a condition and `Deny` effect to a bucket policy to force the request using SSL or TLS -> force using HTTPS.
* When an application running on the EC2 instance makes a call to the S3 `ListObjects` API, the request will be allowed by IAM if the attached role has the `s3:ListBucket` permission for the relevant bucket.

## Concepts

* [Key](/aws/services/storage/s3.md): FULL path of an object
  * ex: s3://mybucket/myfolder/subfolder/myfile.txt
* [Bucket policy](/aws/services/storage/s3.md): JSON-based access policy that determines who can access your bucket & what operations they can perform.

<img src="https://2259236002-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fuh9xZDZ53qGqmMCM44PU%2Fuploads%2Fgit-blob-619bab3dac9d78e51c2f2ece4efe7e571a5b36ac%2Ffigure_20230513183755.png?alt=media" alt="a bucket policy example" width="311">

* ***Hot storage***: frequently accessed data
* ***Warm storage***: less frequently accessed data
* ***Cold storage***: rarely accessed data

## Pricing