AWS Networking & Content Delivery - VPC creating public and private subnet

VPC is region-wise, by default you can create max 5 VPC. Range of IP Address are defined for each VPC.

CIDR Range: Range of IP address in a network.
Example
CIDR:- 192.168.0.0/16
Because we are using /16 so 192.168 will be constant whereas other octates can be changed from 0-255 which means it has 256x256 = 65536 IP Addresses.

Valid IP Address in this Range
192.168.10.20, 192.168.200.10,192.168.2.0/24, 192.168.3.10/21

Invalid IP in the above CIDR
192.168.10.278, 192.169.0.0/24

Reserved IP Addresses in a network
These reserve IP addresses can not be used for EC2 instances:
1. Network IP
2. Broadcast IP
3. Future use IP


Lab

Create a VPC

1. Search for VPC service
2. Click on Your VPC Link
3. Click Create VPC button
4. On this Page define properties related to VPC

Select VPC only option
Name: irfan-vpc
IPV4 CIDR: 10.10.0.0/16
Tenancy : Default

5. Click on Create VPC button

Create Subnets (By Default no subnet get created in the VPC)

1. Click on Subnets link
2. Click on Create subnet

--------------------------------------------------------------------------
3. Set following properties for Public Subnet

• VPC ID: irfan-vpc
• Subnet Name: irfan-public-subnet-2a
• Availability Zone: 2a
• CIDR: 10.10.1.0/24

4. Click on Create Subnet
----------------------------------------------------------------------------------------
5. Set following properties for Private subnet

• VPC ID: irfan-vpc
• Subnet Name: irfan-private-subnet-2b
• Availability Zone: 2b
• CIDR: 10.10.2.0/24

6. Click on Create Subnet

Create an Internet Gateway to provide Internet connectivity to Public subnet

1. Click on Internet Gateways

2. Name: IGW1

• Click on Create Internet Gateway

3. Select Internet Gateway IGW1 and Action--->Attach to VPC and select myvpc-1

Now Internet connectivity is available to myvpc-1 VPC.


Create Route Tables (A default Route table get created for VPC)

1. Click on Route Tables link

2. Click on Create Route table button

--------------------------------------------------------------------------------------------
3. Set the properties

• Name: Public Route
• VPC: irfan-vpc

4. Click on Create Route table button

-----------------------------------------------------------------------------------------------
5. Set the properties

• Name: Private Route
• VPC: irfan-vpc

6. Click on Create Route Table

Attach subnets to Route tables

1. Select Public Route

2. Select Subnet Associations

3. Click on Edit subnet associations and select Public Subnet 1-a

4. Click on Routes --> Edit Routes--> Add Route

5. Destination: 0.0.0.0/0

6. Target: IGW1

7. click on Save Changes button.

---------------------------------------------------------------------------------------------

1. Select Private Route

2. Select Subnet Associations

3. Click on Edit subnet associations and select Private subnet 1-b




Create EC2 instance in irfan-vpc

1. Create a Public EC2 instance Amazon Linux Image

2. Select VPC as irfan-vpc

3. Select subnet as Public Subnet 1-a

4. Auto assign Public IP: Enable

5. Create new Security Groups for SSH let's call it sshSG.

--------------------------------------------------------------------------------------------

1. Create a Private EC2 instance Amazon Linux Image

2. Select VPC as myvpc-1

3. Select subnet as Private Subnet 1-b

4. Auto assign Public IP: Disable

5. Select sshSG security group

Experiment

1. Private IP addresses are in the defined range of subnets or not.

Ans: yes

2. Both instances can ping to each other with private ip address.

Ans: no UNLESS you add the security group - ICMP:IPV4 !!!

Now if you want to connect to the private subnet,

you need to connect to it via the public subnet:

1. copy the key-pair info (.pem)

2. create the key-pair info into your public subnet:

    vi IrfanSydney.pem

    copy the content over

    save :wq!

3. go to the private instance and click connect, choose via ssh, then it will tell you the steps:

    chmod 400 IrfanSydney.pem

    ssh -i "IrfanSydney.pem" ec2-user@10.10.2.184

AWS Networking & Content Delivery - VPC Natgateway and Elastic IP for private subnet

To check internet connectivity is available for Private VM (ping 8.8.8.8), you will need to establish a NAT gateway.

NAT Gateway 

A NAT gateway is a Network Address Translation (NAT) service. You can use a NAT gateway so that instances in a private subnet can connect to services outside your VPC but external services cannot initiate a connection with those instances.

Creating a NAT Gateway

1. Click on create a NAT (Network Address Translation) Gateway

2. Select Public Subnet 1-a

3. Connectivity type: Public

4. Click on Allocate Elastic IP button

5. Click on the Nat Gateway

6. It takes some time to create, once it is created select it and Go to Action --> Attach to VPC

7. Click on Route Table and Select private Route table

8. Select Routes --> Edit Route

9. Click on Add Route , Destination 0.0.0.0/0 and Target NAT Gateway and Click on save changes.

10. Now the private VM should have internet connectivity.

to connect to the private VM:

go to the public VM and connect into it.

then follow this steps on the public VM:

you should now be able to connect to the private VM via the public VM cause NatGateway is established.

Check if the private VM has internet connection:

after connecting to the private VM, use the curl command.

Experiments

1.  Delete NAT Gateway, Check if the EIP also get deleted.

Ans: EIP did not get deleted.

2. Find the EIP address allocated to NAT Gateway.

Ans: under the Elastic IP tab

3. How to delete a VPC.

Ans: delete any instances and any network interfaces.

# Actually just terminate the instances and the network interface will auto delete by themselves. now you can delete your VPC.

Docker Private Registry (local or remote) (secure or insecure)

SUMMARY:

A. Making a Docker Registry on a host machine and Pulling from that same registry.

B. Pulling a Docker Registry from a registry of another host machine (1st method: insecurely)

C. Pulling a Docker Registry from a registry of another host machine (2nd method: securely)

WHAT is DOCKER PRIVATE REGISTRY (LOCAL / REMOTE SERVER)?

Firstly, we will make a private registry storing our docker image either in local or remote server.

Secondly, we will make a certificate for this registry with openssl.

A) Making a Docker Registry on a host machine and Pulling from that same registry.

A registry is a storage and content delivery system in a host machine, holding secured Docker images (you can know the storage path later below), and can be available in different tagged versions.

1. Create a container with registry docker image (this will download registry image from hub.docker.com) 

       docker container run --name local_registry -d -p 5000:5000 registry

why 5000? because docker registry is running at port 5000.

2. Check if the container is running:

        docker ps -a / docker container ls

Access the container on 5000 port with your serverip ( system's IP) http://<serverip>:5000/v2/_catalog
on CLI: curl localhost:5000/v2/_catalog
on browser: just paste and change to localhost:5000/v2/_catalog

you will not be able to do this unless in vagrant file, you add in this line:
config.vm.network "forwarded_port", guest: 5000, host: 5000

3. Inspect the container (cause we are interested in the Mounts) docker container inspect local_registry



4. check the source by typing ls then the highlighted above. this will show blank.
for my case: /var/lib/docker/volumes/4de3c67ac5843db5fe70a4d28bfc95e97339bde8900a78a60041f48eb20b8c0b/_data

it will show here:
/var/lib/docker/volumes/4de3c67ac5843db5fe70a4d28bfc95e97339bde8900a78a60041f48eb20b8c0b/_data/docker/registry/v2/repositories/myalpine#


5. Clone the ubuntu image to localhost:5000/ubuntu:latest

    docker image tag localhost:5000/myalpine
        OR
    docker image tag ubuntu 127.0.0.1:5000/myalpine

notice how this is different from ifanrahman/hisalpine:latest? this is because the ifanrahman one is to push to docker hub. but localhost is to push to private registry.


6. Push the image to docker registry 

        docker image push localhost:5000/myalpine

can check here: /var/lib/docker/volumes/4de3c67ac5843db5fe70a4d28bfc95e97339bde8900a78a60041f48eb20b8c0b/_data/docker/registry/v2/repositories/myalpine#

7. then delete the image

        docker rmi localhost:5000/myalpine


8. Pull the image from the local/private registry with following command

        docker image pull localhost:5000/myalpine

B) Pulling a Docker Registry from a registry of another host machine (1st method: insecurely)

Store Docker Images into Docker Registry (insecurely)
Consideration for this example

IP address of registry server is 192.168.33.10

1. Tag the docker image (alpine) with 192.168.33.10:5000

IF you haven't pull alpine, follow below. if yes skip:

   1) remove all the containers

                    docker rm -f $(docker ps -a -q)

            2) pull the docker image alpine ( you can take any image)

                    docker pull alpine

2. tag the image with private IP address of Registry server

        docker image tag alpine 192.168.33.10:5000/prvalpine

3. verify the tagged docker image got created

        docker images

4. create Docker registry container (if you haven't create it)

        docker container run -d -p 5000:5000 --name local_registry registry

5. push the tagged docker image (it will throw an error because the repository is not secure)

        docker push 192.168.33.10:5000/prvalpine

Error:-> Get https://192.168.33.10:5000/v2/: http: server gave HTTP response to HTTPS client

Remedy: If you want to push the insecure registry then create a file /etc/docker/daemon.json and enter below lines and save the file (Remember to change your IP as per your docker host system IP)


{

"insecure-registries": ["192.168.33.10:5000"]

}



6. restart the Docker daemon

        systemctl restart docker

7. start  the Docker registry Container (cause once you restart, containers will be exited)

        docker start local_registry

8. push the tagged image (this time it should be pushed to docker registry without any error)

        docker push 192.168.33.10:5000/prvalpine

Pull the insecure private registry on a different remote system
Take another Virtual Machine that is in the same network and install docker into that remote machine

1. Install docker

        apt update && apt install docker.io -y

2. If you want to push the insecure registry then create a file /etc/docker/daemon.json and enter below lines and save the file (Please change IP as per your docker host system IP)

{

"insecure-registries": ["192.168.33.10:5000"]

}

3. restart the Docker daemon

        systemctl restart docker

4. pull the Docker Registry image from the private registry

        docker pull 192.168.33.10:5000/prvalpine

5. verify image is available on this system

        docker images

C) Pulling a Docker Registry from a registry of another host machine (2nd method: securely)

Creating a secure Registry

1. remove daemon.json file on Docker Registry and Remote System

        rm /etc/docker/daemon.json

2. restart docker service

        systemctl restart docker

3. remove local_registry Container on Docker Registry Server ( if it is in running state)

        docker rm -f local_registry

4. create a directory to keep the certificates on Docker Registry Server

        mkdir /certs

5. create a directory certs in /etc/docker directory (when docker container run, it will first search any certification in this directory)

        mkdir /etc/docker/certs.d

6. create a directory for images

        mkdir /my_repo

7. create a self signed certificate with openssl utility. (this will create public key .crt and private key .key)

        openssl req -newkey rsa:4096 -nodes -sha256 -keyout /certs/domain.key -x509 -days 365 -out /certs/domain.crt

it asks some optional questions but the mandatory step is to provide common name
common Name :- repo.docker.kmit ( you can give any name)
it will ask email address too, just press enter

check:

ls /certs (domain.key and domain.crt will be created)

8. create a directory with repo.docker.kmit:5000 under /etc/docker/certs.d directory (-p is used if certs.d is not created (parent directory))

        mkdir -p /etc/docker/certs.d/repo.docker.kmit:5000

9. go to /certs directory

        cd /certs

10. copy /certs/domain.crt file to /etc/docker/certs.d/repo.docker.kmit:5000 with name ca.crt

        cp domain.crt /etc/docker/certs.d/repo.docker.kmit:5000/ca.crt

11. run a secure registry on a container and config the container (-v is volume, whatever is in my_repo map it into /var/lib/registry -e is environment variable)

docker run -d -p 5000:5000 -v /my_repo:/var/lib/registry -v /certs:/certs -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key --restart on-failure --name myregistry registry

Remedy: provide repo.docker.kmit name by adding entry in /etc/hosts file (192.168.33.10  is docker host ip change it as necessary)

        vi /etc/hosts        

                        192.168.33.10 repo.docker.kmit

12. download any image and tag it with the common name: repo.docker.kmit:5000

        docker pull mysql

        docker image tag mysql repo.docker.kmit:5000/mysql

13. push it to docker registry

        docker push repo.docker.kmit:5000/mysql

Pulling Images securely on Client or Remote System

1. login to remote system which is on same network and docker is installed on it.

Resolve repo.docker.kmit name by adding entry in /etc/hosts file (  192.168.33.10 is docker registry ip )

        192.168.33.10 repo.docker.kmit

Create a directory /etc/docker/certs.d/repo.docker.kmit:5000

Copy valid certificate domain.crt file from docker Registry server and keep it at /etc/docker/certs.d/repo.docker.kmit:5000/ (hmm..how?)

        ANSIBLE!

ansible 192.168.33.11 -m copy -a "src=/certs/domain.crt dest=/etc/docker/certs.d/repo.docker.kmit:5000/domain.crt"

Pull docker image from docker registry and it will be sucessfull

        docker pull repo.docker.kmit:5000/mysql