When your fancy observability tools have failed you there’s still trusty tpcdump
The was done on Ubuntu. YMMV on other distros.
Exec into a pod
kubectl exec -it my-pod-name sh
Run
cat /sys/class/net/eth0/iflink
> 588 # container eth id
It should return a number, the container eth id
Now run
kubectl describe po my-pod-name | grep Node
To find out the node it’s running on
SSH into the node then run below to find the eni id
ip link | grep 588
> eni89aabc12345
Now use tcpdump to sniff the requests coming in
tcpdump -A -i eni89aabc12345
To capture a specific header
tcpdump -A -i eni89aabc12345 | grep -i X-Real-IP -C 5