AWS Storage - S3 Cross Region Replication

https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html

Replication requires the following:

• The source bucket owner must have the source and destination AWS Regions enabled for their account. The destination bucket owner must have the destination Region enabled for their account.

• Both source and destination buckets must have versioning enabled.

• Amazon S3 must have permissions to replicate objects from the source bucket to the destination bucket or buckets on your behalf.

• If the owner of the source bucket doesn't own the object in the bucket, the object owner must grant the bucket owner READ and READ_ACP permissions with the object access control list (ACL). 

• If the source bucket has S3 Object Lock enabled, the destination buckets must also have S3 Object Lock enabled.

What is replication?

• Replication enables automatic, asynchronous copying of objects across Amazon S3 buckets. 
• Buckets with object replication configuration is owned by the same AWS account or by different accounts. 
• You can replicate objects to a single destination bucket or to multiple destination buckets. 
• The destination buckets can be in different or same AWS Regions.

Same-Region Replication (SRR) or Cross-Region Replication (CRR): To automatically replicate new objects as they are written to the bucket use live replication.

S3 Batch Replication: To replicate existing objects to a different bucket on demand, use .

To enable SRR or CRR:

Step1: add a replication configuration to your source bucket and the minimum configuration must provide the following:

• The destination bucket or buckets where you want Amazon S3 to replicate objects
• An AWS Identity and Access Management (IAM) role that Amazon S3 can assume to replicate objects on your behalf

To make S3 Batch Replication:

Cross-Region Replication enables an automatic, asynchronous copying process between the buckets
Configured Buckets can be owned by a single account or they can be from a different account.
Destination bucket can be the different or same region

Step 1: Create a source bucket (srcbucket) in a region ( Mumbai ap-south-1) and Enable the versioning.
Step 2: Create a destination bucket (destbucket) in a different region ( North Virginia us-east-1) and enable the versioning.
Step 3: Select Source bucket and Click on Management Tab.
Step 4: Click on Create Replication Rule and Provide the below information

• Replication rule name: ReplicaRule1
• Status: Enable
• Priority: 0
• Source bucket: Apply to all objects in the bucket.

Step 5: Select the destination bucket.

• Select option choose a bucket from this account
• Click on browse s3 button
• Select destination bucket

Step 6: IAM Role: Choose from existing IAM role:

• Create your own role (this will create something like S3crr_bucket_irfan)

Step 7: Encryption :- unchecked
Step 8: Destination Storage class: unchecked
Step 9: Click on save button

Step 9.1: if asked if want to replicate existing content: click no for now.
Step 10: Upload a file in the source bucket and it should be copied in the destination bucket as well.


Experiment:
1. Delete the file from the source bucket and check whether it is deleted in the destination bucket or not.

Ans: The file still persists in the destination bucket.
2. Try to disable the versioning of either of these buckets.

Ans: disable versioning in source bucket is not allowed as crr rule is present in the source bucket.

but disable versioning in destination bucket is allowed.

when versioning is disabled in bucket, no new files will be replicated in the destination bucket.

AWS Storage - S3 Storage Classes & Lifecycles

Class 1: S3 Standard 

Class 2: S3 IA (Infrequent Access)

Class 3: S3 IT (Intelligent Tiering)

Class 4: S3 One-Zone IA

Class 5: S3 Glacier (suitable for archiving students data of 20 years ago)

***Storage classes are dependent on the S3 storage lifecycles.

1. Amazon S3 Standard (S3 Standard)

S3 Standard offers high durability, availability, and performance object storage for frequently accessed data. Because it delivers low latency and high throughput.

S3 Storage Classes can be configured at the object level and a single bucket can contain objects stored across S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, and S3 One Zone-IA. You can also use S3 Lifecycle policies to automatically transition objects between storage classes without any application changes.

Use Cases:

• cloud applications
• dynamic websites
• content distribution
• mobile and gaming applications
• big data analytics 

Key Features:

• Low latency and high throughput performance
• Designed for durability of 99.999999999% of objects across multiple Availability Zones
• Resilient against events that impact an entire Availability Zone
• Designed for 99.99% availability over a given year
• Backed with the Amazon S3 Service Level Agreement for availability
• Supports SSL for data in transit and encryption of data at rest
• S3 Lifecycle management for automatic migration of objects to other S3 Storage Classes

2. Amazon S3 Standard-Infrequent Access (S3 Standard-IA)
S3 Standard-IA is for data that is accessed less frequently, but requires rapid access when needed. S3 Standard-IA offers the high durability, high throughput, and low latency of S3 Standard, with a low per GB storage price and per GB retrieval charge. This combination of low cost and high performance make S3 Standard-IA ideal for long-term storage, backups, and as a data store for disaster recovery files. 

S3 Storage Classes can be configured at the object level and a single bucket can contain objects stored across S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, and S3 One Zone-IA. You can also use S3 Lifecycle policies to automatically transition objects between storage classes without any application changes.

Use Cases:

• Students data or learning materials and resources on cloud. When the students finish their course, the materials are still stored but using the IA S3 storage class.
• long-term storage.
• data store for disaster recovery files.

Key Features:

• Same low latency and high throughput performance of S3 Standard
• Designed for durability of 99.999999999% of objects across multiple Availability Zones
• Resilient against events that impact an entire Availability Zone
• Data is resilient in the event of one entire Availability Zone destruction
• Designed for 99.9% availability over a given year
• Backed with the Amazon S3 Service Level Agreement for availability
• Supports SSL for data in transit and encryption of data at rest
• S3 Lifecycle management for automatic migration of objects to other S3 Storage Classes

3. Amazon S3 Intelligent-Tiering (S3 Intelligent-Tiering)

Amazon S3 Intelligent-Tiering (S3 Intelligent-Tiering) is the first cloud storage that automatically reduces your storage costs on a granular object level by automatically moving data to the most cost-effective access tier based on access frequency, without performance impact, retrieval fees, or operational overhead. S3 Intelligent-Tiering delivers milliseconds latency and high throughput performance for frequently, infrequently, and rarely accessed data in the Frequent, Infrequent, and Archive Instant Access tiers. You can use S3 Intelligent-Tiering as the default storage class for virtually any workload, especially 
For a small monthly object monitoring and automation charge, S3 Intelligent-Tiering monitors access patterns and automatically moves objects that have not been accessed to lower-cost access tiers. S3 Intelligent-Tiering automatically stores objects in three access tiers: 

1. one tier that is optimized for frequent access, 
2. a 40% lower-cost tier that is optimized for infrequent access
3. a 68% lower-cost tier optimized for rarely accessed data. 

S3 Intelligent-Tiering monitors access patterns and moves objects that have not been accessed for 30 consecutive days to the Infrequent Access tier and after 90 days of no access to the Archive Instant Access tier. For data that does not require immediate retrieval, you can set up S3 Intelligent-Tiering to monitor and automatically move objects that aren’t accessed for 180 days or more to the Deep Archive Access tier to realize up to 95% in storage cost savings.

There are no retrieval charges in S3 Intelligent-Tiering. If an object in the Infrequent or Archive Instant Access tier is accessed later, it’s automatically moved back to the Frequent Access tier. If the object you’re retrieving is stored in the optional Deep Archive tiers, before you can retrieve the object, you must first restore a copy using RestoreObject. For information about restoring archived objects, see Restoring Archived Objects. No additional tiering charges apply when objects are moved between access tiers within the S3 Intelligent-Tiering storage class.

Use Cases:

• data lakes
• data analytics
• new applications
• user-generated content.

Key Features:

• Frequent, Infrequent, and Archive Instant Access tiers have the same low-latency and high-throughput performance of S3 Standard
• The Infrequent Access tier saves up to 40% on storage costs
• The Archive Instant Access tier saves up to 68% on storage costs
• Opt-in asynchronous archive capabilities for objects that become rarely accessed
• Deep Archive Access tier has the same performance as Glacier Deep Archive and saves up to 95% for rarely accessed objects
• Designed for durability of 99.999999999% of objects across multiple Availability Zones and for 99.9% availability over a given year
• Backed with the Amazon S3 Service Level Agreement for availability
• Small monthly monitoring and auto tiering charge
• No operational overhead, no lifecycle charges, no retrieval charges, and no minimum storage duration
• Objects smaller than 128KB can be stored in S3 Intelligent-Tiering but will always be charged at the Frequent Access tier rates, and are not charged the monitoring and automation charge.

4. Amazon S3 One Zone-Infrequent Access (S3 One Zone-IA)

S3 One Zone-IA is for data that is accessed less frequently, but requires rapid access when needed. Unlike other S3 Storage Classes which store data in a minimum of three Availability Zones (AZs), S3 One Zone-IA stores data in a single AZ and costs 20% less than S3 Standard-IA. S3 One Zone-IA is ideal for customers who want a lower-cost option for infrequently accessed data but do not require the availability and resilience of S3 Standard or S3 Standard-IA. 

S3 One Zone-IA offers the same high durability, high throughput, and low latency of S3 Standard, with a low per GB storage price and per GB retrieval charge. S3 Storage Classes can be configured at the object level, and a single bucket can contain objects stored across S3 Standard, S3 Intelligent-Tiering, S3 Standard-IA, and S3 One Zone-IA. You can also use S3 Lifecycle policies to automatically transition objects between storage classes without any application changes.

Use Cases:

• It’s a good choice for storing secondary backup copies of on-premises data
• easily re-creatable data. 
• You can also use it as cost-effective storage for data that is replicated from another AWS Region using S3 Cross-Region Replication.

Key Features:

• Same low latency and high throughput performance of S3 Standard
  
• Designed for durability of 99.999999999% of objects in a single Availability Zone†
  
• Designed for 99.5% availability over a given year
  
• Backed with the Amazon S3 Service Level Agreement for availability
• Supports SSL for data in transit and encryption of data at rest
• S3 Lifecycle management for automatic migration of objects to other S3 Storage Classes

† Because S3 One Zone-IA stores data in a single AWS Availability Zone, data stored in this storage class will be lost in the event of Availability Zone destruction.

5. The Amazon S3 Glacier storage classes

It is purpose-built for data archiving, and are designed to provide you with the highest performance, the most retrieval flexibility, and the lowest cost archive storage in the cloud. 

You can choose from three archive storage classes optimized for different access patterns and storage duration:

1. S3 Glacier Instant Retrieval storage class: For archive data that needs immediate access, such as medical images, news media assets, or genomics data, it is a storage class that delivers the lowest cost storage with milliseconds retrieval. 
2. S3 Glacier Flexible Retrieval (formerly S3 Glacier): For archive data that does not require immediate access but needs the flexibility to retrieve large sets of data at no cost, such as backup or disaster recovery use cases with retrieval in minutes or free bulk retrievals in 5—12 hours. 
3. S3 Glacier Deep Archive: To save even more on long-lived archive storage such as compliance archives and digital media preservation, it is the lowest cost storage in the cloud with data retrieval from 12—48 hours.

Use Cases:

• deep-archive 
• archive medical reports
• backup that does not rquire immediate access but needs flexibility
• digital media preservation (radio, tv)

AWS Storage - S3 Lifecycle Management

 

THIS IS CONTINUATION OF S3 STORAGE CLASSES & LIFECYCLE. PLS CHECK THAT OUT FIRST BEFORE YOU CONTINUE THIS ARTICLE.

Management for the Lifecycle of S3 bucket objects depends upon their requirement like frequent access or Infrequently accessing or archiving the data.

Prerequisite: Bucket should be already created in a specified region.

Create Lifecycle Rule (to manage the costs and resources needed effectively):
Step 1: Select Management Tab in the bucket .
Step 2: Click on create Lifecycle Rule
Step 3: Provide the below information about lifecycle rule.

• Life Cycle Rule name: lcr1
• Choose Rule Scope: Apply to all objects in the bucket
• Life Rule Action: select below option
• Move current versions of objects between storage classes
• Define the transition actions
• Standard IA---> 30 days
• Intelligent Tiering ---> 60 days
• One Zone ---> 90 days
• Glacier:--->180 days
• Deep Archive:---> 365 days

Step 4: Click on Create Rule button and Rule should be created and Should be enabled.

EXTRA INFOS: AWS Storages, their Types and Block vs Object Storage

AWS has the following types of Storage types.

1. Simple Storage Services (S3)
2. Elastic File System (EFS)
3. Elastic Block Storage (EBS)
4. SNOW

Block Storage vs Object Storage:

Block Storage	Object Storage
Block Storage divides the data to be data stored evenly sized blocks.	Object storage stores the file as a whole and does not divide them.
Data Blocks stored in Block Storage would not contain the metadata ( data created, data modified, content type etc).	In Object storage, an object is the file data itself. it has metadata and object Global unique ID. *Object Global unique ID is a unique identifier for the object.
Block Storage only keeps the address( index) where data blocks are stored and can be mounted as a drive	Object Storage can't be mounted as a drive.

Example of Block Storage is EBS

Example of Object Storage is S3,Dropbox etc. 

Block Storage is suitable for transaction databases, random read/write loads, and structured. databases.

AWS Storage - S3 Storage Naming Convention and Bucket Sub Resources

 

Simple Storage Service -S3
It is storage for the internet (HTTP or HTTPS) and it has a web service interface.
It is object-based storage.
You can't install OS on S3.
It has a distributed data store architecture where objects are redundantly stored in multiple locations(min 3 locations in the same region).
Data is stored in Bucket.
A bucket is a flat container of Objects.
Max capacity of the bucket is 5 TB.
You can create folders in a bucket.
You can't create nested buckets.
Bucket ownership is non-transferable.
S3 bucket is region-specific.
You can have 100 buckets per account but you can expand it.


S3 Naming Rules (Naming Convention).
S3 Bucket names(keys) are globally unique across all AWS regions.
Bucket name can't be changed.
If Bucket is deleted then its name will be available to use.
Bucket name is >3 and <=63 characters long. (within 4 -63 char)
Bucket name is the part of Bucket URL.
Bucket name must be a series of one or more labels like (my.bucket).
Bucket name can container lowercase letters, numbers and hyphens. Upper case is not allowed.
Bucket name should not be IP address.
By default Bucket's objects are private.



S3 Bucket Subresources
Lifecycle: It decides on object's life cycle
Static Website: S3 can host a static website.
Access Control list: Bucket policies are defined to access the bucket.

Bucket name is simply in two parts Bucket Region endpoint/bucket name. For example s3 bucket named mybucket in Europe west region.
https://s3-eu-west1.amazonaws.com/mybucket

AWS Storage - S3 Static Web Hosting

S3 supports hosting a static website (HTML, HTML with CSS, JavaScript), it does not support dynamic website (server-side processing like JAVA, PHP, ASP.NET) hosting.


Prerequisite:
S3 Bucket : A valid bucket should be already created.
Html Pages: Create index.html(Landing Page),errors.html (Error Page) file with valid html code.
Access Level: Public Access using bucket policy Bucket Policy

Step 1: Enable versioning for the bucket.
Step 2: Upload index.html and error.html files to the bucket.
Step 3: Go to bucket Properties (Properties Tab)
Step 4: Select the Static Web Hosting option and click on the Edit button
Step 5: Select Enable Option.
Step 6: Select option Select Hosting type--> Host a static website.
Step 7: In Index document specify index.html and in Error document specify error.html.
Step 8: Click on Save Changes button.
Step 9: There will be a Bucket Website endpoint created.
Step 10: Click on this endpoint, you should be able to access the index page.


Experiments
Upload the HTML files which is other than index.html and error.html
Disable the versioning and check are you able to host the website

ASSIGNMENT: Create S3 Bucket and Versioning

Simple Storage Service (s3) Bucket

Step 1: Create a Bucket

1. Search for S3 service OR Storage---> S3
   https://s3.console.aws.amazon.com/s3/home?region=ap-southeast-2#
   
2. Click Create Bucket Button
3. Provide details about bucket information
4. Bucket name : mybucket908
5. Aws Region :- ap-south-1
6. ACL Disabled
7. Block all public access: unchecked it
8. Click on Create Bucket button.

Step 2: Add some files into S3 bucket

1. Select bucket mybucket908
2. Click on the upload button
3. Add files/Add Folders Upload some files from your system.
4. I am adding the file say index.html

Step 3: Download the file

1. select index.html and click on Download/open
2. You should be able to download

Step 4: Make index.html public

1. Select Object URL and try to access it on the browser
2. You will get the Access denied error
3. Change the permission by Clicking on the Permission tab
4. ACL enabled option and Acknowledge it.
5. select index.html file
6. Object Actions --> Make public using ACL and click on Make public button.

Step 5: Delete the Bucket

1. You can only delete the bucket if there is no object in the bucket.

Enable versioning for S3 Objects

For example, if you make some changes in the index.html on your system and upload it again on the s3 bucket, then you will lose your previous version of the application. So we need to Enable versioning on Bucket level.

    Step 1: Select the Bucket and select Properties.

    Step 2:  Click on Bucket Versioning Edit button

    Step 3: Enable Bucket Version

    Step 4: Save Changes.

    Step 5: Make change in index.html file and Upload it to mybucket908.

    Step 6: Select index.html file and click on Version Tab.

    Step 7: You can see multiple versions of your application(index.html)

Experiments

• Access the current version by using Object URL if it gives access denied error then make it public and access it.
• Check are you able to switch between Current version and older version of index.html

Recover a deleted file

You can can't recover a file until the versioning is enabled on bucket level. Delete marker with unique ID shows the files which got deleted.

    1. Select index.html file

    2. Click on Delete button and Delete object

    3. Click on Show Versions, it will show the index.html file with its versions.

Docker Storage with Examples

SUMMARY:

A) WHAT IS STORAGE AND THE TYPES

B) EXAMPLES FOR NON-PERSISTENT DATA

C) EXAMPLES FOR PERSISTENT DATA

 Docker Storage

       To keep data for the container is called container's storage. 

       Docker storage is only available on Linux.

Types of Storage

Non Persistent        In this type of storage, the data will be lost if the container is deleted. tmpfs In this file system, the data will be stored in memory and it is only available during the container's lifetime, which means if the container is stopped or deleted the data will be lost. It is more suitable for the in-memory calculation.

Persistent       In this type of storage, the data will be persisted even though the container gets deleted. Docker Volume    It is the storage that is maintained by the docker daemon in the docker area.. The default storage location is /var/lib/docker/volumes folder. Bind Mount   It is the storage that is managed by the admin of the system. It is a file or directory which is maintained on the host machine.

Docker Volume
It is the storage that is maintained by the docker daemon in the docker area.. The default storage location is /var/lib/docker/volumes folder

Docker volume Commands:

create	Create a volume
inspect	Display detailed information on one or more volumes
ls	List volumes
prune	Remove all unused local volumes
rm	Remove one or more volumes

Bind Mount
It is the storage that is managed by the admin of the system. It is a file or directory which is maintained on the host machine. It is having fewer features as compared to docker volume and it does not have any formal commands to manage these volumes by docker. You can use -v or --mount option to mount a directory of your host machine to the container

Example of Non-Persistent Storage:

Example 1:-

Step 1: Create a container using ubuntu docker image

         docker container run -it --name tmpcontainer -d ubuntu

Step 2: Go inside the container

         docker exec -it tmpcontainer bash

Step 3: In this container, create a directory call test and store some files into it.

•             mkdir test
•       cd test
•       touch file1 file2 file3 file4
•       ls

Step 4:  Stop the container.

            docker stop tmpcontainer

Step 5:  Start the container and check the test director still persist with all its files.

•              docker start tmpcontainer
•       docker exec -it tmpcontainer bash
•       ls  test

Step 6: Delete the container and think is there any way to get your test directory again.

            docker rm -f tmpcontainer

            No there is no way to get back the test directory data, because the container is deleted so storage in this container was non-persistent.

Example 2:- (tmpfs)

Step 1: Create a container using ubuntu docker image.

         docker container run -it --name tmpcontainer --mount type=tmps,destination=/test -d ubuntu

Step 2: Go inside the container

         docker exec -it tmpcontainer bash

Step 3: In this container, create a directory call test and store some files into it.

•     cd test
•     touch file1 file2 file3 file4
•     ls

Step 4:  Stop the container.

            docker stop tmpcontainer

Step 5:  Start the container and check the test director still persist with all its files or not. The files should be removed means you will not get any data inside test folder..

             docker start tmpcontainer
             docker exec -it tmpcontainer bash
             ls  test

Step 6: Delete the container and think is there any way to get your test directory again.

            docker rm -f tmpcontainer

            No there is no way to get back the test directory data, because the container is deleted so storage in this container was non-persistent.

Example 3:- (tmps with tmpfs-mode)

Create a container with tmpfs and change the permission of destination folder.

    docker container run -it --name c1 --mount type=tmpfs,destination=/tmp1,tmpfs-mode=1700 -d ubuntu




Example of Persistence Storage

Example 1:- (docker volume)

To delete all unused docker volumes:

docker volume prune

Step 1: To create a docker volume (demo-vol)

docker volume create demo-vol

Step 2: List all docker volumes

docker volume ls

Step 3: By default the driver is local and path for this docker volume is /var/lib/docker/volume.

ls /var/lib/docker/volumes/demo-vol/_data

Step 4: Run a container which points its /app directory to demo-vol

docker container run -it --name c1 --mount source=demo-vol,destination=/app -d ubuntu

Step 5: Go inside the container

docker exec -it c1 bash

Step 6: Create some files under /app directory

cd app

touch file1 file2 file3 file4

exit

Step 7: Check these files are available under demo-volume's directory

ls /var/lib/docker/volumes/demo-vol/_data

Step 8: Delete file4 from _data directory and check in c1 container's /app directory, file4 should not be available under this directory.

rm /var/lib/docker/volumes/demo-vol/_data/file4

docker exec -it c1 bash

ls /app

Step 9: Add a new file file5 in-app directory of the container and check this file should be available in demo-volume's data directory.

cd app

touch file5

exit

ls /var/lib/docker/volumes/demo-vol/_data

Step 10: Delete container c1.

docker rm -f c1

Step 11: Make sure your demo-vol data is not deleted.

ls /var/lib/docker/volumes/demo-vol/_data

Step 12: Create a new container and attach demo-vol to that container's /demo directory.

docker container run -it --name c2 --mount source=demo-vol,destination=/demo -d centos

Step 13: Check in the container's demo directory whether all these files exist or not. 

docker exec -it c2 bash

ls demo

Step 14: Delete the volume. Deletion of volume will delete the data as well.

docker rm -f c2

docker volume rm demo-vol

Note: You can use -v option instead of --mount with docker volume to mount a volume eg.

docker container run -it --name c3 -v demo-vol:/demo -d centos

Note: You can also refer to an existing directory of containers that points to docker volume.

docker container run -it --name c3 -v demo-vol:/root -d centos

In the above example, root directory's all files will be stored under demo-vol's data directory.

Example 2:- (Bind Mount)

Step 1: Create a directory (/home/vagrant/myfiles) that you want to map with the container's target directory

mkdir /home/vagrant/myfiles

Step 2: Run a container which maps myfiles directory to container's /app1 directory.

docker container run -it --name bindmountcontainer -v /home/vagrant/myfiles:/app1 -d ubuntu

OR

docker container run -it --name bindmountcontainer --mount type=bind,source=/home/vagrant/myfiles,target=/app1 -d ubuntu

Step 3: Go inside the container and add some files in app1 directory

docker exec -it bindmountcontainer bash

cd app1

touch file1 file2 file3 file4

exit

Step 4: Go to myfiles directory and find all the files that exist in this folder or not.

ls myfiles