From Disk Partitions in EC2/Linux to Pods in EKS/Kubernetes
Table of Contents
- Introduction
- Scenario
- Partition to Pod
- Real-World Scenarios
- Scale & Maintain the Solution
- Conclusion
- About the Author ✍🏻
Introduction
In the dynamic world of containerized workloads orchestrated by Kubernetes, tracing issues back to their source can sometimes be tricky. Traditional system insights, often rooted in OS or cloud layers, don’t always speak the language of Kubernetes. This short blog is the story of how we trace a seemingly opaque clue like a Linux partition ID back to the Kubernetes workload residing within it.
Scenario
Imagine this scenario: a Cloud Native Application Protection Platform (CNAPP) scans your EBS volumes/snapshots for security vulnerabilities. It reports the location of these vulnerabilities — files nested within specific filesystem paths. In a typical Linux VM, you go to the location & fix the issue but if this VM is a node of a Kubernetes cluster, these files might belong to a pod in your cluster.
The file path alone might not be enough to pinpoint the pod requiring remediation. While traditional VMs typically hold entries for all mounted volumes in /etc/fstab, Kubernetes nodes won’t always do this, since the EBS volumes “travel” from node to node as pods migrate. So the CNAPP resorts to using the partition ID as part of the vulnerable file path.
Partition to Pod
One approach (among many) of tracing back the partition ID to the K8s workload is as follows:
Open up a terminal in your K8s node & use lsblk to find the partition:
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 200G 0 disk
├─nvme0n1p1 259:1 0 200G 0 part /
└─nvme0n1p128 259:2 0 1M 0 part
nvme1n1 259:3 0 20G 0 disk /var/lib/kubelet/pods/60058b56-ee0f-4c5b-ad66-d6588516aed2/volumes/kubernetes...
nvme2n1 259:4 0 70G 0 disk /var/lib/kubelet/pods/9cb1e483-54a8-46ca-8636-b9713c20ea4b/volumes/kubernetes...
nvme3n1 259:5 0 20G 0 disk /var/lib/kubelet/pods/cad5df0f-c1ea-4730-bbfb-fe6b0a04e225/volumes/kubernetes...To view only the partition ID to PV mapping:
$ lsblk -o UUID,MOUNTPOINT
UUID MOUNTPOINT
d34ae088-88d0-43f5-8270-dd09f73ee4c3 /
918c69b7-0909-4e5b-a755-a65a52dff147 /var/lib/kubelet/pods/60058b56-ee0f-4c5b-ad66-d6588516aed2/volumes/kubernetes...
fb707be4-2541-4cc9-a3af-2ca5fb2fb446 /var/lib/kubelet/pods/9cb1e483-54a8-46ca-8636-b9713c20ea4b/volumes/kubernetes...
6c444a88-4ae6-4025-aa02-3baec5d64583 /var/lib/kubelet/pods/cad5df0f-c1ea-4730-bbfb-fe6b0a04e225/volumes/kubernetes...To strip down everything unwanted & just see the partition ID & PV ID:
$ lsblk -o UUID,MOUNTPOINT | grep kubernetes | \
sed 's/\/mount//g' | sed 's/-.* / /g' | \
sed 's/\/var\/lib\/kubelet\/pods\/.*\/volumes\/kubernetes\.io~csi\///g'
918c69b7 pvc-5606a62f-1f31-46c2-9f60-74c72f6b0e77
fb707be4 pvc-3f0ef402-a2d1-47b1-848f-b8c58b4e682a
6c444a88 pvc-33953429-3de0-4575-be39-24d820275e42Now, a simple kubectl get pv in your cluster is enough to map the above PVs to their respective workloads & the target of your remediation efforts.
Real-World Scenarios
There are a few scenarios where the ability to trace Linux disk partition IDs back to their Kubernetes pods might be useful:
- Vulnerability Response: This is the use case we covered above. Identify the pod harboring the vulnerability detected by the CNAPP & fix it.
- Performance Bottlenecks: If system monitoring tools identify slow I/O operations traced back to a specific partition, leverage this technique to identify the impacted pod & address the resource constraints hindering its performance.
- Forensic Analysis: In case of an incident, tracing suspicious file modifications back to the responsible pod through partition IDs can be a crucial step in forensic analysis, helping to isolate the affected workload & expedite remediation efforts.
Scale & Maintain the Solution
If workloads in your cluster are fairly static, it’s a one-time effort to collect all partition IDs from all your nodes & map them all to their PVCs & pods in the cluster. Since dynamically-provisioned PVs won’t be recreated unless you delete their PVCs, the underlying EBS volumes also stay unchanged. And since a partition ID is inherent to the EBS volume, it doesn’t change when the volume moves to other nodes as its pod moves.
Conclusion
While Kubernetes offers a powerful abstraction layer for managing containerized workloads, it can sometimes obscure the underlying infrastructure. This article highlights one such case where tooling outside of Kubernetes can be made to work with K8s when needed.
About the Author ✍🏻
Harish KM is a Principal DevOps Engineer at QloudX & a top-ranked AWS Ambassador since 2020. 👨🏻💻
With over a decade of industry experience as everything from a full-stack engineer to a cloud architect, Harish has built many world-class solutions for clients around the world! 👷🏻♂️
With over 20 certifications in cloud (AWS, Azure, GCP), containers (Kubernetes, Docker) & DevOps (Terraform, Ansible, Jenkins), Harish is an expert in a multitude of technologies. 📚
These days, his focus is on the fascinating world of DevOps & how it can transform the way we do things! 🚀
Linkedin