AWS Compute - EBS Volume (Snapshots and Volume Types)

EBS Volume Snapshots

Volumes are AZ specific means the volumes created in an AZ can be attached to EC2 instances which get created in that AZ

To utilize volumes in other AZ you need to create the snapshot of the volume and then create a new volume in the different AZ from the snapshot and attach to the instance which are available in different AZ

Copy the snapshots from one Region to another Region

When we create a volume from a snapshot then the size of volume can be increased but cannot be decreased.

We can create a root volume by taking a snapshot of the root volume of any instance.

EBS Volume Types

Amazon EBS provides the following volume types:
1. SSD
2. Provisioned IOPS SSD

3. Previous Generation HDD


1. Solid state drives (SSD)
Optimized for transactional workloads involving frequent read/write operations with small I/O size, where the dominant performance attribute is IOPS.
Solid state drives (SSD)
The SSD-backed volumes provided by Amazon EBS fall into these categories:

General Purpose SSD provides a balance of price and performance. We recommend these volumes for most workloads.

    gp3 (type of general purpose SSD): Durability:- 99.8% - 99.9% durability(0.1% - 0.2% annual failure rate)

    gp2 (type of general purpose SSD): Durability:- 99.8% - 99.9% durability(0.1% - 0.2% annual failure rate)

Use Case:

• Low-latency interactive apps
• Development and test environments

1. Volume Size:   1 GB - 16 TB
2. Max IOPS (input Output per Second) per volume (16 KiB I/O) : 16000
3. Max throughput (data transfer) per volume: For gp3: 1,000 MB/s and for gp2: 250 MB/s*
4. Amazon EBS Multi-attach(can attach multiple EBS volume): Not Supported
5. Boot volume (OS system volume): Supported

2. Provisioned IOPS SSD
Provides high performance for mission-critical, low-latency, or high-throughput workloads. io2 Block Express a.k.a io2, io1 are the types of Provisioned IOPS SSD.

Durability: 99.999% durability (0.001% annual failure rate), for io1 - 99.8% - 99.9% durability (0.1% - 0.2% annual failure rate)

Use Case:
Workloads that require:

• Sub-millisecond latency
• Sustained IOPS performance
• More than 64,000 IOPS or 1,000 MB/s of throughput

1. Volume Size : 4 GB - 64 TB, for io1 - 4 GB - 16 TB
2. Max IOPS per volume (16 KiB I/O) : 256,000 , for io1: 64,000
3. Max throughput per volume : 4000MB/s, for io1: 1000/MiB/s
4. Amazon EBS Multi-attach : Supported

3. Hard disk drives (HDD)
Optimized for large streaming workloads where the dominant performance attribute is throughput.

Use Cases:

• Big data
• Data warehouses
• Log processing

1. Volume: 125 GB - 16 TB


4. Previous generation
Hard disk drives that can be used for workloads with small datasets where data is accessed infrequently and performance is not of primary importance. We recommend that you consider a current generation volume type instead.

Use Case:

• Workloads where data is infrequently accessed

1. Volume size: 1GB to 1 TB

Docker Storage with Examples

SUMMARY:

A) WHAT IS STORAGE AND THE TYPES

B) EXAMPLES FOR NON-PERSISTENT DATA

C) EXAMPLES FOR PERSISTENT DATA

 Docker Storage

       To keep data for the container is called container's storage. 

       Docker storage is only available on Linux.

Types of Storage

Non Persistent        In this type of storage, the data will be lost if the container is deleted. tmpfs In this file system, the data will be stored in memory and it is only available during the container's lifetime, which means if the container is stopped or deleted the data will be lost. It is more suitable for the in-memory calculation.

Persistent       In this type of storage, the data will be persisted even though the container gets deleted. Docker Volume    It is the storage that is maintained by the docker daemon in the docker area.. The default storage location is /var/lib/docker/volumes folder. Bind Mount   It is the storage that is managed by the admin of the system. It is a file or directory which is maintained on the host machine.

Docker Volume
It is the storage that is maintained by the docker daemon in the docker area.. The default storage location is /var/lib/docker/volumes folder

Docker volume Commands:

create	Create a volume
inspect	Display detailed information on one or more volumes
ls	List volumes
prune	Remove all unused local volumes
rm	Remove one or more volumes

Bind Mount
It is the storage that is managed by the admin of the system. It is a file or directory which is maintained on the host machine. It is having fewer features as compared to docker volume and it does not have any formal commands to manage these volumes by docker. You can use -v or --mount option to mount a directory of your host machine to the container

Example of Non-Persistent Storage:

Example 1:-

Step 1: Create a container using ubuntu docker image

         docker container run -it --name tmpcontainer -d ubuntu

Step 2: Go inside the container

         docker exec -it tmpcontainer bash

Step 3: In this container, create a directory call test and store some files into it.

•             mkdir test
•       cd test
•       touch file1 file2 file3 file4
•       ls

Step 4:  Stop the container.

            docker stop tmpcontainer

Step 5:  Start the container and check the test director still persist with all its files.

•              docker start tmpcontainer
•       docker exec -it tmpcontainer bash
•       ls  test

Step 6: Delete the container and think is there any way to get your test directory again.

            docker rm -f tmpcontainer

            No there is no way to get back the test directory data, because the container is deleted so storage in this container was non-persistent.

Example 2:- (tmpfs)

Step 1: Create a container using ubuntu docker image.

         docker container run -it --name tmpcontainer --mount type=tmps,destination=/test -d ubuntu

Step 2: Go inside the container

         docker exec -it tmpcontainer bash

Step 3: In this container, create a directory call test and store some files into it.

•     cd test
•     touch file1 file2 file3 file4
•     ls

Step 4:  Stop the container.

            docker stop tmpcontainer

Step 5:  Start the container and check the test director still persist with all its files or not. The files should be removed means you will not get any data inside test folder..

             docker start tmpcontainer
             docker exec -it tmpcontainer bash
             ls  test

Step 6: Delete the container and think is there any way to get your test directory again.

            docker rm -f tmpcontainer

            No there is no way to get back the test directory data, because the container is deleted so storage in this container was non-persistent.

Example 3:- (tmps with tmpfs-mode)

Create a container with tmpfs and change the permission of destination folder.

    docker container run -it --name c1 --mount type=tmpfs,destination=/tmp1,tmpfs-mode=1700 -d ubuntu




Example of Persistence Storage

Example 1:- (docker volume)

To delete all unused docker volumes:

docker volume prune

Step 1: To create a docker volume (demo-vol)

docker volume create demo-vol

Step 2: List all docker volumes

docker volume ls

Step 3: By default the driver is local and path for this docker volume is /var/lib/docker/volume.

ls /var/lib/docker/volumes/demo-vol/_data

Step 4: Run a container which points its /app directory to demo-vol

docker container run -it --name c1 --mount source=demo-vol,destination=/app -d ubuntu

Step 5: Go inside the container

docker exec -it c1 bash

Step 6: Create some files under /app directory

cd app

touch file1 file2 file3 file4

exit

Step 7: Check these files are available under demo-volume's directory

ls /var/lib/docker/volumes/demo-vol/_data

Step 8: Delete file4 from _data directory and check in c1 container's /app directory, file4 should not be available under this directory.

rm /var/lib/docker/volumes/demo-vol/_data/file4

docker exec -it c1 bash

ls /app

Step 9: Add a new file file5 in-app directory of the container and check this file should be available in demo-volume's data directory.

cd app

touch file5

exit

ls /var/lib/docker/volumes/demo-vol/_data

Step 10: Delete container c1.

docker rm -f c1

Step 11: Make sure your demo-vol data is not deleted.

ls /var/lib/docker/volumes/demo-vol/_data

Step 12: Create a new container and attach demo-vol to that container's /demo directory.

docker container run -it --name c2 --mount source=demo-vol,destination=/demo -d centos

Step 13: Check in the container's demo directory whether all these files exist or not. 

docker exec -it c2 bash

ls demo

Step 14: Delete the volume. Deletion of volume will delete the data as well.

docker rm -f c2

docker volume rm demo-vol

Note: You can use -v option instead of --mount with docker volume to mount a volume eg.

docker container run -it --name c3 -v demo-vol:/demo -d centos

Note: You can also refer to an existing directory of containers that points to docker volume.

docker container run -it --name c3 -v demo-vol:/root -d centos

In the above example, root directory's all files will be stored under demo-vol's data directory.

Example 2:- (Bind Mount)

Step 1: Create a directory (/home/vagrant/myfiles) that you want to map with the container's target directory

mkdir /home/vagrant/myfiles

Step 2: Run a container which maps myfiles directory to container's /app1 directory.

docker container run -it --name bindmountcontainer -v /home/vagrant/myfiles:/app1 -d ubuntu

OR

docker container run -it --name bindmountcontainer --mount type=bind,source=/home/vagrant/myfiles,target=/app1 -d ubuntu

Step 3: Go inside the container and add some files in app1 directory

docker exec -it bindmountcontainer bash

cd app1

touch file1 file2 file3 file4

exit

Step 4: Go to myfiles directory and find all the files that exist in this folder or not.

ls myfiles

ASSIGNMENTS: Docker Volume

 

ASSIGNMENT 1:-

1. Create a docker apache image using Dockerfile.

 2. Create a deploy-vol docker volume.

 3. Run a container using docker apacheimg with port forwarding 80:80 and map /var/www/html folder to deploy-vol.

4. Check default apache page is accessible on the browser.

5. Copy/Create a test.html file under /var/lib/docker/volumes/deploy-vol/_data

6. Check on the browser that you are able to access this webpage (test.html)

7. Go inside the container and create a file newtest.html under /var/www/html and verify that you are able to access it with the browser.

0. mkdir dockvolassignment

0.1 cd dockvolassignment

1. vi Dockerfile

1.2 in Dockerfile:
FROM ubuntu

#from is getting the base image  

ARG DEBIAN_FRONTEND=noninteractive
#noninteractive for automation so it does not ask questions such as geographical..

RUN apt-get update
#run is adding instruction on the instruction layer

RUN apt-get -y install apache2

ADD . /var/www/html
#add is copy the file from host machine to docker image folder

ENTRYPOINT apachectl -D FOREGROUND
#entrypoint whenever docker image is used in a container, some commands need to be executed automatically

ENV name DEVOPS
#environment variable is used on the global level

1.3 docker build . -t apacheimg

2. docker volume create deploy-vol

3. docker container run --name assignment1 -v deploy-vol:/var/www/html -dit -p 80:80 apacheimg

4. curl localhost

5. echo "<h1> Welcome to test.html page! </h1>" > /var/lib/docker/volumes/deploy-vol/_data/test.html

6. curl localhost/test.html

7. docker exec -it assignment1 bash

7.2 in container:

echo "<h1> This is newtest page and NOT test page! </h1>" > /var/www/html/newtest.html

7.3 exit container:

exit

7.4 curl localhost/newtest.html

ASSIGNMENT 2:-

1. Attach a docker volume to multiple containers and add or modify the files in any of these containers and check that the changes are getting reflected in all the containers or not.

2. Point #1 is to be repeated for Bind Mount

3. Delete the Bind mount directory which is mapped to a container's destination directory and check the destination behavior means try to create some file or list the files on the destination folder.

4. Continue to Point 3, you may get some error that the source directory does not exist. Create source directory again and check the destination folder and find out whether you are able to create new files in this folder or not.

ANSWERS:

1. yes

2. yes

3. for bind mount: 

touch: cannot touch '/app1/devops3fromc3.txt': No such file or directory

3.2 for volume (means prune the docker volume):

4. for bind mount: 

touch: cannot touch '/app1/devops3fromc3.txt': No such file or directory (but if you create a container after that, it can sync the files) this means that by using bind, once your source is changed, your target will never find the source again even if its the same name.

4.2 for volume (means create a docker volume with same name after pruning it):

5. Restart the container which you are using for #3 and #4 and again check the destination folder to store some data and find out whether you are able to see it in the source folder or not.