Category: DevOps Tools

Anything related to Devops Tools such as Docker, puppet ..etc

K8s Types of probe 

Posted on by Osama Mustafa in DevOps Tools

A probe is a diagnostic performed periodically by the kubelet on a container. To perform a diagnostic, the kubelet either executes code within the container, or makes a network request.

Probe outcome

Each probe has one of three results:

  • Success :- The container passed the diagnostic.
  • Failure :- The container failed the diagnostic.
  • Unknown :- The diagnostic failed (no action should be taken, and the kubelet will make further checks).

Types of probe 

The kubelet can optionally perform and react to three kinds of probes on running containers:

  • livenessProbe

Indicates whether the container is running. If the liveness probe fails, the kubelet kills the container, and the container is subjected to its restart policy. If a container does not provide a liveness probe, the default state is Success.

  • readinessProbe

Indicates whether the container is ready to respond to requests. If the readiness probe fails, the endpoints controller removes the Pod’s IP address from the endpoints of all Services that match the Pod. The default state of readiness before the initial delay is Failure. If a container does not provide a readiness probe, the default state is Success.

  • startupProbe

Indicates whether the application within the container is started. All other probes are disabled if a startup probe is provided, until it succeeds. If the startup probe fails, the kubelet kills the container, and the container is subjected to its restart policy. If a container does not provide a startup probe, the default state is Success.

When should you use a liveness probe

If the process in your container is able to crash on its own whenever it encounters an issue or becomes unhealthy, you do not necessarily need a liveness probe; the kubelet will automatically perform the correct action in accordance with the Pod’s restartPolicy.

If you’d like your container to be killed and restarted if a probe fails, then specify a liveness probe, and specify a restartPolicy of Always or OnFailure.

When should you use a readiness probe

If you’d like to start sending traffic to a Pod only when a probe succeeds, specify a readiness probe. In this case, the readiness probe might be the same as the liveness probe, but the existence of the readiness probe in the spec means that the Pod will start without receiving any traffic and only start receiving traffic after the probe starts succeeding.

If you want your container to be able to take itself down for maintenance, you can specify a readiness probe that checks an endpoint specific to readiness that is different from the liveness probe.

If your app has a strict dependency on back-end services, you can implement both a liveness and a readiness probe. The liveness probe passes when the app itself is healthy, but the readiness probe additionally checks that each required back-end service is available. This helps you avoid directing traffic to Pods that can only respond with error messages.

If your container needs to work on loading large data, configuration files, or migrations during startup, you can use a startup probe. However, if you want to detect the difference between an app that has failed and an app that is still processing its startup data, you might prefer a readiness probe.

When should you use a startup probe

Startup probes are useful for Pods that have containers that take a long time to come into service. Rather than set a long liveness interval, you can configure a separate configuration for probing the container as it starts up, allowing a time longer than the liveness interval would allow.

If your container usually starts in more than initialDelaySeconds + failureThreshold × periodSeconds, you should specify a startup probe that checks the same endpoint as the liveness probe. The default for periodSeconds is 10s. You should then set its failureThreshold high enough to allow the container to start, without changing the default values of the liveness probe. This helps to protect against deadlocks.

Cheers
Osama

Passing Configuration Data to a Kubernetes Container

Posted on by Osama Mustafa in DevOps Tools

One of these applications is a simple Nginx web server. This server is used as part of a secure backend application, and the company would like it to be configured to use HTTP basic authentication.

This will require an htpasswd file as well as a custom Nginx config file. In order to deploy this Nginx server to the cluster with good configuration practices, you will need to load the custom Nginx configuration from a ConfigMap (this already exists) and use a Secret to store the htpasswd data.

Create a Pod with a container running the nginx:1.19.1 image. Supply a custom Nginx configuration using a ConfigMap, and populate an htpasswd file using a Secret.

Generate an htpasswd file:

htpasswd -c .htpasswd user

View the file’s contents:

cat .htpasswd

Create a Secret containing the htpasswd data:

kubectl create secret generic nginx-htpasswd --from-file .htpasswd

Delete the .htpasswd file:

rm .htpasswd

Create the Nginx Pod

Create pod.yml:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.19.1
    ports:
    - containerPort: 80
    volumeMounts:
    - name: config-volume
      mountPath: /etc/nginx
    - name: htpasswd-volume
      mountPath: /etc/nginx/conf
  volumes:
  - name: config-volume
    configMap:
      name: nginx-config
  - name: htpasswd-volume
    secret:
      secretName: nginx-htpasswd

View the existing ConfigMap:

kubectl get cm

we need to create the configMap for nginx-config:-

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-config
data:
 nginx.conf: |
   user  nginx;
    worker_processes  1;

    error_log  /var/log/nginx/error.log warn;
    pid        /var/run/nginx.pid;

     events {
       worker_connections  1024;
      }
     http {
      server {
        listen       80;
        listen  [::]:80;
        server_name  localhost;
        location / {
            root   /usr/share/nginx/html;
            index  index.html index.htm;
        }
        auth_basic "Secure Site";
        auth_basic_user_file conf/.htpasswd;
      }
     }

Get more info about nginx-config:

kubectl describe cm nginx-config

Apply the pod

kubectl apply -f pod.yml

Within the existing busybox pod, without using credentials, verify everything is working:

kubectl exec busybox -- curl <NGINX_POD_IP>

We’ll see HTML for the 401 Authorization Required page — but this is a good thing, as it means our setup is working as expected.

curl: (6) Couldn't resolve host 'user'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   179  100   179    0     0  62174      0 --:--:-- --:--:-- --:--:-- 89500
<html>
<head><title>401 Authorization Required</title></head>
<body>
<center><h1>401 Authorization Required</h1></center>
<hr><center>nginx/1.19.1</center>
</body>
</html>

Run the same command again using credentials (including the password you created at the beginning of the lab) to verify everything is working:

kubectl exec busybox -- curl -u user:<PASSWORD> <NGINX_POD_IP>

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   612  100   612    0     0  48846      0 --:--:-- --:--:-- --:--:-- 51000
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Cheers

Osama

role vs rolebinding in kubernetes

Posted on by Osama Mustafa in DevOps Tools

You need to know the difference between

  • Role.
  • Rolebinding.
  • ClusterRole.

Please refer the Kubernetes documentation here

A Role always sets permissions within a particular namespace; when you create a Role, you have to specify the namespace it belongs in.

ClusterRole, by contrast, is a non-namespaced resource. The resources have different names (Role and ClusterRole) because a Kubernetes object always has to be either namespaced or not namespaced; it can’t be both.

A rolebinding is namespace scoped and clusterrolebinding is cluster scoped i.e across all namespace.

ClusterRoles and ClusterRoleBindings are useful in the following cases:

  1. Give permissions for non-namespaced resources like nodes
  2. Give permissions for resources in all the namespaces of a cluster
  3. Give permissions for non-resource endpoints like /healthz

A RoleBinding can also reference a ClusterRole to grant the permissions defined in that ClusterRole to resources inside the RoleBinding’s namespace. This kind of reference lets you define a set of common roles across your cluster, then reuse them within multiple namespaces.

example

Create a Role for the dev User

  1. Create a role spec file role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: beebox-mobile
  name: pod-reader
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "watch", "list"]

2. Save and exit the file by pressing Escape followed by :wq.

3. apply the role.

kubectl apply -f file-name.yml

Bind the Role to the dev User and Verify Your Setup Works

  1. Create the RoleBinding spec file:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: pod-reader
  namespace: beebox-mobile
subjects:
- kind: User
  name: dev
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

2. Apply the role, by running

kubectl apply -f file-name.yml

Cheers

Osama

Backing up and Restoring Kubernetes Data in etcd

Posted on by Osama Mustafa in DevOps Tools

Backups are an important part of any resilient system. Kubernetes is no exception. In this post , I will show you how to backup/restore kubernetes data.

Back Up the etcd Data

  1. Look up the value for the key cluster.name in the etcd cluster:
ETCDCTL_API=3 etcdctl get cluster.name \
  --endpoints=https://10.0.1.101:2379 \
  --cacert=/home/cloud_user/etcd-certs/etcd-ca.pem \
  --cert=/home/cloud_user/etcd-certs/etcd-server.crt \
  --key=/home/cloud_user/etcd-certs/etcd-server.key

2. Back up etcd using etcdctl and the provided etcd certificates:

ETCDCTL_API=3 etcdctl snapshot save /home/cloud_user/etcd_backup.db \
  --endpoints=https://10.0.1.101:2379 \
  --cacert=/home/cloud_user/etcd-certs/etcd-ca.pem \
  --cert=/home/cloud_user/etcd-certs/etcd-server.crt \
  --key=/home/cloud_user/etcd-certs/etcd-server.key

3. Reset etcd by removing all existing etcd data

Note: you don’t have to do this step if this is production, I am only doing this to show how to restore the data.

sudo systemctl stop etcd
sudo rm -rf /var/lib/etcd

Restore the etcd Data from the Backup

  1. Restore the etcd data from the backup (this command spins up a temporary etcd cluster, saving the data from the backup file to a new data directory in the same location where the previous data directory was):
sudo ETCDCTL_API=3 etcdctl snapshot restore /home/cloud_user/etcd_backup.db \
  --initial-cluster etcd-restore=https://10.0.1.101:2380 \
  --initial-advertise-peer-urls https://10.0.1.101:2380 \
  --name etcd-restore \
  --data-dir /var/lib/etcd

2. Set ownership on the new data directory

sudo chown -R etcd:etcd /var/lib/etcd

3. Start etcd

sudo systemctl start etcd

4. Verify the restored data is present by looking up the value for the key cluster.name again:

ETCDCTL_API=3 etcdctl get cluster.name \
  --endpoints=https://10.0.1.101:2379 \
  --cacert=/home/cloud_user/etcd-certs/etcd-ca.pem \
  --cert=/home/cloud_user/etcd-certs/etcd-server.crt \
  --key=/home/cloud_user/etcd-certs/etcd-server.key

Cheers

Osama

Upgrade k8s using kubeadm

Posted on by Osama Mustafa in DevOps Tools

First, upgrade the control plane node

Drain the control plane node.

kubectl drain master-node-name --ignore-daemonsets

Upgrade kubeadm.

sudo apt-get update && \
sudo apt-get install -y --allow-change-held-packages kubeadm=version

kubeadm version

Plan the upgrade.

sudo kubeadm upgrade plan v-version (for example v1.24.2)

Upgrade the control plane components.

sudo kubeadm upgrade apply v1.22.2

Upgrade kubelet and kubectl on the control plane node.

sudo apt-get update && \
sudo apt-get install -y --allow-change-held-packages kubelet=version kubectl=version

Restart kubelet.

sudo systemctl daemon-reload
sudo systemctl restart kubelet

Uncordon the control plane node.

kubectl uncordon master-node-name

Verify that the control plane is working

Note:- you should not perform upgrades on all worker nodes at the same time. Make sure enough nodes are available at any given time to provide uninterrupted service.

Worker nodes

Run the following on the control plane node to drain worker node 1:

kubectl drain worker1-node-name --ignore-daemonsets --force

Log in to the first worker node, then Upgrade kubeadm.

sudo apt-get update && \ sudo apt-get install -y --allow-change-held-packages kubeadm=version

Upgrade the kubelet configuration on the worker node.

sudo kubeadm upgrade node

Upgrade kubelet and kubectl on the worker node.

sudo apt-get update && \ sudo apt-get install -y --allow-change-held-packages kubelet=version kubectl=verion

Restart kubelet.

sudo systemctl daemon-reload 
sudo systemctl restart kubelet

From the control plane node, uncordon worker node 1.

kubectl uncordon worker1-node-name

Repeat the upgrade process for worker nodes.

Cheers

Osama

k8s management tools

Posted on by Osama Mustafa in DevOps Tools

There is a variery of management tools that allow you to manage k8s and make your life much easier to proivde extra additional features.

  • kubectl

the official command line interface for k8s, this is the main method to interact you will use.

  • kubeadm

tools that allow you to setup control plane.

  • MiniKube

tool that runs a single-node Kubernetes cluster locally on your workstation for development and testing purposes.

Very simple tools you can find it here.

  • Helm

tool for managing packages of pre-configured Kubernetes resources. These packages are known as Helm charts.

Use Helm to:

  • Find and use popular software packaged as Kubernetes charts
  • Share your own applications as Kubernetes charts
  • Create reproducible builds of your Kubernetes applications
  • Intelligently manage your Kubernetes manifest files
  • Manage releases of Helm packages

  • Kompose

a tool to help Docker Compose users move to Kubernetes.

Use Kompose to:

  • Translate a Docker Compose file into Kubernetes objects
  • Go from local Docker development to managing your application via Kubernetes
  • Convert v1 or v2 Docker Compose yaml files or Distributed Application Bundles

and the last one which is kustomize.

Cheers

Osama

Install k8s as one control plane and one worker node

Posted on by Osama Mustafa in DevOps Tools

The first thing you will need to do which is configure the two servers, either you can choose one of the following options:-

  • VMWARE
  • Cloud

Master Node Setup

Step #1

Create configuration file for containerd: 

cat <<EOF | sudo tee /etc/modules-load.d/containerd.conf
overlay
br_netfilter
EOF

After the above step, you need to load the modules.

sudo modprobe overlay
sudo modprobe br_netfilter

Step #2

Set system configurations for Kubernetes networking

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

Apply new settings

sudo sysctl --system

Step #3

Install Containerd

sudo apt-get update && sudo apt-get install -y containerd

Step #4

Create default configuration file for containerd

sudo mkdir -p /etc/containerd

Generate default containerd configuration and save to the newly created default file

sudo containerd config default | sudo tee /etc/containerd/config.toml

Load the new configuration

sudo systemctl restart containerd
sudo systemctl status containerd

Step #5

Disable Swap

sudo swapoff -a

Step #6

Install dependency packages:

sudo apt-get update && sudo apt-get install -y apt-transport-https curl

Download and add GPG key

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

Add Kubernetes to repository list

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://apt.kubernetes.io/ kubernetes-xenial main
EOF

sudo apt-get update

Step #6

Install Kubernetes packages (Note: If you get a dpkg lock message, just wait a minute or two before trying the command again):

sudo apt-get install -y kubelet=1.24.0-00 kubeadm=1.24.0-00 kubectl=1.24.0-00

Just in case Turn off automatic updates

sudo apt-mark hold kubelet kubeadm kubectl

The above steps should be done on the worker node even if you have 3 or 4.

Initialize the Cluster

Initialize the Kubernetes cluster on the control plane node using kubeadm (Note: This is only performed on the Control Plane Node):

sudo kubeadm init --pod-network-cidr 192.168.0.0/16 --kubernetes-version 1.24.0

Set kubectl access:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

You can test your cluster by run 

kubectl get nodes

Install the Calico Network Add-On

On the control plane node, install Calico Networking

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Check status of the control plane node:

kubectl get nodes

Join the Worker Nodes to the Cluster

In the control plane node, create the token and copy the kubeadm join command (NOTE:The join command can also be found in the output from kubeadm init command):

kubeadm token create --print-join-command

Copy the output

Worker node Setup.

from the above command of Kubeadm join run it using sudo command.

In the control plane node, view cluster status (Note: You may have to wait a few moments to allow all nodes to become ready)

kubectl get nodes

Cheers

Enjoy the DevOps

infrastructure as code tools

Posted on by Osama Mustafa in DevOps Tools

Infrastructure as code is one of the most common uses to set up a cloud environment, either Cloudformation, Oracle resource stack, or 3rd party such as Pulumi or terraform.

For this, I would like to share the tools I use for the perfect IaC tools that could be useful for someone

Enjoy

Osama

New Cloud Project

Posted on by Osama Mustafa in DevOps Tools

The Idea of this project the following :

You need to develop and deploy a python app that writes a new file to S3 on every execution. These files need to be maintained only for 24h.

The content of the file is not important, but add the date and time as prefix for you files name.

The name of the buckets should be the following ones for QA and Staging respectively:

qa-FIRSTNAME-LASTNAME-platform-challenge

staging-FIRSTNAME-LASTNAME-platform-challenge

The app will be running as a docker container in a Kubernetes cluster every 5 minutes. There is a Namespace for QA and a different Namespace for Staging in the cluster. You don’t need to provide tests but you need to be sure the app will work.

Github HERE

Cheers

Osama

Creating a Helm Chart

Posted on by Osama Mustafa in DevOps Tools

Helm is the first application package manager running atop Kubernetes. It allows describing the application structure through convenient helm-charts and managing it with simple commands. Because it’s a huge shift in the way the server-side applications are defined, stored and managed.

Helm Charts provide “push button” deployment and deletion of apps, making adoption and development of Kubernetes apps easier for those with little container or microservices experience. Apps deployed from Helm Charts can then be leveraged together to meet a business need, such as CI/CD or blogging platforms.

Install Helm

  • Use curl to create a local copy of the Helm install script
 curl https://raw.githubusercontent.com/helm/helm/master/scripts/get > /tmp/get_helm.sh
cat /tmp/get_helm.sh
  • Use chmod to modify access permissions for the install script
chmod 700 /tmp/get_helm.sh

Set the version to v2.8.2

 DESIRED_VERSION=v2.8.2 /tmp/get_helm.sh

Ensure Helm uses the correct stable chart repo (the default one used by Helm has been decommissioned)

helm init --stable-repo-url https://charts.helm.sh/stable

Initialize Helm:

helm init --wait

Give Helm the permissions it needs to work with Kubernetes

kubectl --namespace=kube-system create clusterrolebinding add-on-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default

Make sure our configuration is working properly

Create a Helm Chart

mkdir charts

cd charts
  • Create the chart for httpd
helm create httpd
  • Verify our directory was created correctly by running ls command
  • Navigate to the httpd directory by using cd command “cd httpd
  • view the files and directory cd httpd/
  • This directory contains two files: Chart.yaml and values.yaml. We need to edit the values.yaml file.
  • Open values.yaml
Under image, change the repository to httpd.
Change the tag to latest.
Under service, change type to NodePort.
replicaCount: 1
image:
  repository: httpd
  tag: latest
  pullPolicy: IfNotPresent
service:
  type: NodePort
  port: 80

ingress:
  enabled: false
  annotations: {}
  path: /
  hosts:
    - chart-example.local

  tls: []
resources: {}
nodeSelector: {}
tolerations: []
affinity: {}
  • Create Your Application Using Helm
  • Back to directory httpd and run the command
helm install --name my-httpd ./httpd/

Copy the commands listed under the NOTES section of the output, and then paste and run them. It should return the private IP address and port number of our application.

  • Let’s check to see if our pods have come online
kubectl get pods
kubectl get services

Finished

Thank you for reading

Osama