kubernetes too many pods when scheduling pod to run on a tainted node

 

When i was trying to schedule a pod to run on a node, i bump into this message - "too many pods" and always stuck on pending state.

Then i look at the pods by node, it has less than 20 pods and i can't get it to schedule on those pods.

Then i started review my nodeSelector to make sure everything is fine. Then i look at my node taint and pods tolerations. Seems to be fine. 

Getting the node labels

kubectl get node --show-labels

Getting the nodes taint 

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints --no-headers

Listing out the pods used by a node 

kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=<your-node-name>

Then i started to check my deployment to verify everything is working as expected

apiVersion: apps/v1

kind: Deployment

metadata:

  name: with-node-affinity

spec:

  replicas: 4

  selector:

    matchLabels:

      app: with-node-affinity

  template:

    metadata:

      labels:

        app: with-node-affinity

    spec:

      containers:

      - name: with-node-affinity

        image: ubuntu

      tolerations:

      - effect: NoSchedule

        key: unsafe

        operator: Equal

        value: applications

      affinity:

        nodeAffinity:

          requiredDuringSchedulingIgnoredDuringExecution:

            nodeSelectorTerms:

            - matchExpressions:

              - key: agentpool

                operator: In

                values:

                - appsystlpool

As you can see I have my tolerations setup exactly to match the node taints. 

This affinity specification is the same as using the nodeSelector. 

All these still doesn't explain why my pods doesn't get scheduled correctly. 

When i do a kubectl describe pod/my-pod, i get alot of message saying 29 node with taint  doesn't match, 4 node with non matching toleration, 4 node too many pods. 

Then decided to look the the pods configurations on the nodes 

kubectl get nodes -o yaml | grep pods

When i look at the output, i noticed that pods is configure 10 which is really know. Together with the system pods, that would have been easily exceeded value.