DEVSECOPS: Prometheus Monitoring on Kubernetes

 

Prometheus

Prometheus is an open-source monitoring framework. It provides out-of-the-box monitoring capabilities for the Kubernetes container orchestration platform.

Metric Collection: Prometheus uses the pull model to retrieve metrics over HTTP.

Metric Endpoint: The systems that you want to monitor using Prometheus should expose the metrics on an /metrics endpoint. Prometheus uses this endpoint to pull the metrics in regular intervals

PromQL: Prometheus comes with PromQL, a very flexible query language that can be used to query the metrics in the Prometheus dashboard. Also, the PromQL query will be used by Prometheus UI and Grafana to visualize metrics.

Prometheus Exporters: Exporters are libraries that convert existing metrics from third-party apps to Prometheus metrics format. 

TSDB (time-series database): Prometheus uses TSDB for storing all the data. By default, all the data gets stored locally. However, there are options to integrate remote storage for Prometheus TSDB.

Prometheus Monitoring Setup on Kubernetes

I assume that you have a Kubernetes cluster up and running with kubectl setup on your workstation.

Connect to your Kubernetes cluster and make sure you are having admin privileges. 

Step 1:

First, we will create a Kubernetes namespace for all our monitoring components. If you don’t create a dedicated namespace, all the Prometheus kubernetes deployment objects get deployed on the default namespace.

kubectl create namespace monitoring

Step 2:

Prometheus uses Kubernetes APIs to read all the available metrics from Nodes, Pods, Deployments, etc. For this reason, we need to create an RBAC policy with read access to required API groups and bind the policy to the monitoring namespace.

Create a file clusterRole.yaml

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRole

metadata:

  name: prometheus

rules:

- apiGroups: [""]

  resources:

  - nodes

  - nodes/proxy

  - services

  - endpoints

  - pods

  verbs: ["get", "list", "watch"]

- apiGroups:

  - extensions

  resources:

  - ingresses

  verbs: ["get", "list", "watch"]

- nonResourceURLs: ["/metrics"]

  verbs: ["get"]

---

apiVersion: rbac.authorization.k8s.io/v1

kind: ClusterRoleBinding

metadata:

  name: prometheus

roleRef:

  apiGroup: rbac.authorization.k8s.io

  kind: ClusterRole

  name: prometheus

subjects:

- kind: ServiceAccount

  name: default

  namespace: monitoring

Execute the file which creates role and role binding on cluster level for default service account in monitoring namespace.

kubectl create -f clusterRole.yaml

Step 3:

Create a Config Map To Externalize Prometheus Configurations

All configurations for Prometheus are part of prometheus.yaml file and all the alert rules for Alertmanager are configured in prometheus.rules

prometheus.yaml: This is the main Prometheus configuration which holds all the scrape configs, service discovery details, storage locations, data retention configs, etc).

prometheus.rules: This file contains all the Prometheus alerting rules.

By externalizing Prometheus configs to a Kubernetes config map, you don’t have to build the Prometheus image whenever you need to add or remove a configuration. You need to update the config map and restart the Prometheus pods to apply the new configuration.

The config map with all the Prometheus scrape config and alerting rules gets mounted to the Prometheus container in /etc/prometheus location as prometheus.yaml and prometheus.rules files.

 Create a file called config-map.yaml and copy the file contents from here Prometheus Config file

 Execute the following command to create the config map in Kubernetes.

kubectl create -f config-map.yaml

Step 4:

Create a Prometheus Deployment

 Create a file named prometheus-deployment.yaml and copy the following contents onto the file. In this configuration, we are mounting the Prometheus config map as a file inside /etc/prometheus

apiVersion: apps/v1

kind: Deployment

metadata:

  name: prometheus-deployment

  namespace: monitoring

  labels:

    app: prometheus-server

spec:

  replicas: 1

  selector:

    matchLabels:

      app: prometheus-server

  template:

    metadata:

      labels:

        app: prometheus-server

    spec:

      containers:

        - name: prometheus

          image: prom/prometheus

          args:

            - "--storage.tsdb.retention.time=12h"

            - "--config.file=/etc/prometheus/prometheus.yml"

            - "--storage.tsdb.path=/prometheus/"

          ports:

            - containerPort: 9090

          resources:

            requests:

              cpu: 500m

              memory: 500M

            limits:

              cpu: 1

              memory: 1Gi

          volumeMounts:

            - name: prometheus-config-volume

              mountPath: /etc/prometheus/

            - name: prometheus-storage-volume

              mountPath: /prometheus/

      volumes:

        - name: prometheus-config-volume

          configMap:

            defaultMode: 420

            name: prometheus-server-conf

 

        - name: prometheus-storage-volume

          emptyDir: {}

Execute above deployment

kubectl create  -f prometheus-deployment.yaml 

You can check the created deployment using the following command.

kubectl get deployments --namespace=monitoring

Step 5:

Access the deployment by using NodePort service

Create a file named prometheus-service.yaml and copy the following contents. We will expose Prometheus on all Kubernetes node IPs on port 30000.

apiVersion: v1

kind: Service

metadata:

  name: prometheus-service

  namespace: monitoring

  annotations:

      prometheus.io/scrape: 'true'

      prometheus.io/port:   '9090'

spec:

  selector:

    app: prometheus-server

  type: NodePort  

  ports:

    - port: 8080

      targetPort: 9090

      nodePort: 30000

Execute this service to Access Prometheus Dashboard

kubectl create -f prometheus-service.yaml --namespace=monitoring

Step 6: 

Go to browser and access the Prometheus Dashboard on port number 30000

DEVSECOPS: Kubernetes with Grafana

 

Grafana:

It is a visualization tool to provide graphical representation for monitoring services like Prometheus.

Steps to install Grafana on the Kubernetes environment.

Step 1:

Create a file named grafana-datasource-config.yaml

apiVersion: v1

kind: ConfigMap

metadata:

  name: grafana-datasources

  namespace: monitoring

data:

  prometheus.yaml: |-

    {

        "apiVersion": 1,

        "datasources": [

            {

               "access":"proxy",

                "editable": true,

                "name": "prometheus",

                "orgId": 1,

                "type": "prometheus",

                "url": "http://prometheus-service.monitoring.svc:8080",

                "version": 1

            }

        ]

    }

Execute the script

kubectl create -f grafana-datasource-config.yaml

Step 2: 

Create a file named deployment.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

  name: grafana

  namespace: monitoring

spec:

  replicas: 1

  selector:

    matchLabels:

      app: grafana

  template:

    metadata:

      name: grafana

      labels:

        app: grafana

    spec:

      containers:

      - name: grafana

        image: grafana/grafana:latest

        ports:

        - name: grafana

          containerPort: 3000

        resources:

          limits:

            memory: "1Gi"

            cpu: "1000m"

          requests:

            memory: 500M

            cpu: "500m"

        volumeMounts:

          - mountPath: /var/lib/grafana

            name: grafana-storage

          - mountPath: /etc/grafana/provisioning/datasources

            name: grafana-datasources

            readOnly: false

      volumes:

        - name: grafana-storage

          emptyDir: {}

        - name: grafana-datasources

          configMap:

              defaultMode: 420

              name: grafana-datasources

Execute the script

kubectl create -f deployment.yaml

Step 3:

Create a service file named service.yaml

apiVersion: v1

kind: Service

metadata:

  name: grafana

  namespace: monitoring

  annotations:

      prometheus.io/scrape: 'true'

      prometheus.io/port:   '3000'

spec:

  selector:

    app: grafana

  type: NodePort  

  ports:

    - port: 3000

      targetPort: 3000

      nodePort: 32000

Execute the script

kubectl create -f service.yaml

Now you should be able to access the Grafana dashboard using any node IP on port 32000. Make sure the port is allowed in the firewall to be accessed from your workstation.

http://<your-node-ip>:32000

DEVSECOPS - Cloud Architect Skills

To become a cloud architect, you need to have more people skills in contrast to technical skills, while tech skill is important too as we are dealing with the tech, people skills are essential in becoming a good architect.

so what do architects do?

• design, document, sell
• lead, translate, customer relationship management
• project management, client entertainment

1. Design (by understanding the business)

system design, end-to-end design. build based on inputs from client:

1. business goals/objectives?
2. pain points?
3. vision?
4. are there other competitors and what are they doing?
5. budget?

2. Documentation

25% of the work is documenting.

easy to understand document for non-tech executives.

deep technology document for tech people.

3. Sell

Must have sales skills. selling solutions to customers. selling internal management to say we need manpower. Sell to upper management to implement solution. 

Risk, Cost, Propose, get Buy Ins

4. Lead

architects are director levels or above positions.

leading and managing other people's employees.

5. Translate

need to have the business acumen/knowledge to be able to translate the input from client side into tech solution.

ask questions, get the baseline of the client's organization current systems.

bring in a team of cloud engineers to baseline the systems (what are their CPU and Memory utilization, how many systems, etc.) therefore cloud engineers need more knowledge on how to do the things

 

6. Customer relationship management

You're only as strong as your weakest link in your team.

7. Project Management

as an architect, you basically are managing alot of different projects.

each project will have a PoC, a it needs to be done, thus you will require good project managment skills because on top of PoC, you might be needing to do manage a third-party solution such as Cisco or Fortinet to help design for your customer, a better solution.

8. Entertain Clients

Lunch and dinners to entertain clients. develop deep relationship with clients cause sometimes you are on a 16-hour project so you will need some time to talk.