You're reading for free via bob van der staak's Friend Link. Become a member to access the best of Medium.
Member-only story
Penetration testing a Kubernetes environment
Searching for weaknesses in the configuration.
To read for free/Friends Link https://bobvanderstaak.medium.com/72719f9e1010?source=friends_link&sk=2fddd1540d46a67aa517b210aea1313e
In this blog, which is probably a series, I want to share some steps you can take while performing a penetration test on a Kubernetes environment. It will share some techniques and tools you can use to exploit a pod or a cluster-based environment when parts of the weak default configuration are used.
Retrieving Information
The first step, the basics. With a good foundation, we can extract lots of valuable information from a cluster. Which can help pinpoint the weakness.
For starters, there are multiple ways to retrieve information about pods inside a cluster by making use of kubectl.
The following command is probably the most used but returns a limited amount of information:
kubectl get pods -AThis command requests to retrieve all pods and the property -A specifies we want to retrieve them across all namespaces.
TIP: If you don’t have access to one of the namespaces the request will not work and the command will fail.
As you can see not a lot of information is returned. We can use the -o property to return more information (like which node it is running on and the internal IPs the are using.)
But you will receive lots of information when you perform the command kubectl get pods -A -o json. Which will return a complete json object for all pods across all namespaces. The information density difference between the two is immense and the latter results in lots of valuable knowledge that can be used during your security assessment.
For starters, it will return the implemented and used securityContext for every individual pod (in the namespace you are authorized to access). In the example above we see for example a good implementation where all capabilities are dropped and only the required capabilities are added in.
A deep dive in the different capabilities and the dangerous ones I will do in another blog
Also, limits and requests are implemented to be sure that this pod will not be vulnerable to resource starvation attacks. The limits make sure that this pod can not be used for a starvation attack. it can never take more resources then that is described there. the requests property makes sure that it will always have that amount of resources available for itself. So if other pods require a lot of resources those amounts are at least always reserved for microsoft-defender-publisher pod.
Lots of other valuable information can be extracted using this.
I wrote a few powershell modules that can help to get a better understanding of the Kubernetes cluster:
You can install it with:
git clone https://github.com/beserkerbob/KubernetesEnumerationTool.git
cd KubernetesEnumerationTool
Import-Module .\KubernetesEnumerationTool -ForceThen you can use the following command:
Get-PodBestPracticeAnalysisIt will use your kubectl session and determine the best practices for each pod for every namespace you have access rights to.
It will check for now:
+================================================+=========================+================+
| What is being checked: | Possible security issue | Best practices |
+================================================+=========================+================+
| If for the pod readonlyfilesystem are required | | x |
+------------------------------------------------+-------------------------+----------------+
| if the pods are running privileged containers | x | x |
+------------------------------------------------+-------------------------+----------------+
| if there are pods not dropping all capability | | x |
+------------------------------------------------+-------------------------+----------------+
| If AppArmor is implemented | | x |
+------------------------------------------------+-------------------------+----------------+
| if SeccompProfile is implemented | | x |
+------------------------------------------------+-------------------------+----------------+
| If limits are implemented | x | x |
+------------------------------------------------+-------------------------+----------------+
| If there are unsafe capabilities used | x | x |
+------------------------------------------------+-------------------------+----------------+
| If the pod allowed privilege escalation | x | x |
+------------------------------------------------+-------------------------+----------------+
| If there are orphaned pods | | x |
+------------------------------------------------+-------------------------+----------------+
| If readiness probes are used | | x |
+------------------------------------------------+-------------------------+----------------+
| If startup probes are used | | x |
+------------------------------------------------+-------------------------+----------------+
| if pre-stop checks are implemented | | x |
+------------------------------------------------+-------------------------+----------------+
| if runAsUser is implemented | x | |
+------------------------------------------------+-------------------------+----------------+
| If pods are running with HostPID enabled | x | |
+------------------------------------------------+-------------------------+----------------+
| If pods are running with HostIPC enabled | x | |
+------------------------------------------------+-------------------------+----------------+
| If pods are running with HostNetwork enabled | x | |
+------------------------------------------------+-------------------------+----------------+
| If pods are pods with sensitive mount paths | x | |
+------------------------------------------------+-------------------------+----------------+
Especially the following items are dangerous:
If you allow pods to run with HostPID, HostIPC, HostNetwork, Privileged, allowing pods to perform a host mount and allow pods to run without strong limits and implementing requests.
For this blog, we will look at a combination of errors. Where by default you have high access rights on the cluster to deploy pods and everything without strong policy enforcement.
Exploitation through the node with kubectl debug
When you have access to Kubernetes with kubectl you can quickly validate if your current access rights are too much and should be tweaked to implement least privilege.
you can for example perform the following command:
kubectl auth can-i --listThis will showcase what access rights you have on the provided cluster as you can see we have for the default namespace lots of access rights.
the *.* indicates we have access rights on all resources and we can perform anything on any verb. (so we can create, update, patch, get, delete …)
This is the wrong implementation and should in almost all cases not be the case. The least privilege isn’t implemented here correctly.
based on this information we can try and quickly get access to the node. You can use one of my Powershell modules by running
Test-ExploitabilityToNodeWhich will try to perform this action and retrieve for you automatically some sensitive information as proof. It will also validate if there are some constraints implemented that would block these types of attacks.
Or you can do it manually (it is always good to understand and know what you are doing)
First, we need to know the names of the Kubernetes nodes. which can be performed by doing:
kubectl get nodes
We need the name of the node we want to debug. and then we can debug it with the following command
kubectl debug node/<nodeName> -it -q - image=<imagename we want to use> -n <namespace>When performing that command you are root on the node and can immediately access the host (because of a host mount) and retrieve valuable information like the service account token and the ssh keys for the root user:
cat /host/var/run/secrets/kubernetes.io/serviceaccount/token
cat /host/root/.ssh/authorized_keys
This is one of the fastest ways to get to the node. However, it is noisy and lots of privileges are required to do this. If we investigate the pod which is created (called node-debugger also a clear indicator of something strange is going on for the blue team).
We see that there is a mountPath for /host (which is the hostmount which is already a big issue). But also the hostIPC, HostNetwork and HostPID are allowed.
When node debugging isn’t possible because of access rights or permissions we can look into other methods, like patching, using hostPID etc. Which i will cover in another blog.
Conclusion
Default Kubernetes configurations are most of the time not great and are vulnerable to exploitation. This example is using AKS and we can see that we need to implement strong RBAC rules to be sure to limit these attacks.
For now, you can use the tools provided and quickly analyze for any exploitation possibilities in your system. In the net blogs, I will go into more details of other exploitation possibilities and how you can secure yourself against them.
Feel free to use my opensource tool which will automate this process. The kubernetesEnumerationTool: https://github.com/beserkerbob/KubernetesEnumerationTool
Happy testing!
If you want to discuss anything related to infosec I’m on LinkedIn: https://www.linkedin.com/in/bobvanderstaak/
