gke using spot instances

To ensure your workload can work and uses spot instances (instead of either one), you should use the following configuration. Notice here, we have both tolerations and nodeSelector.  

    spec:

      nodeSelector:

        cloud.google.com/gke-spot: "true"

      tolerations:

      - key: cloud.google.com/gke-spot

        operator: Equal

        value: "true"

        effect: NoSchedule

      containers:

      - name: helloworld

        image: docker.io/istio/examples-helloworld-v2:1.0

If you're using autopilot, then a new node gets created which uses spot instances. You can check this using the following command 

kubectl get node/your-node-name -o yaml | grep cloud.google.com/gke-spot=true

Example output are shown here:- 

If you would like to configure spot vm preferences, you can use the following setup. 

apiVersion: cloud.google.com/v1

kind: ComputeClass

metadata:

  name: prefer-l4-spot

spec:

  priorities:

  - machineFamily: n2

    spot: true

    minCores: 64

  - machineFamily: n2

    spot: true

  - machineFamily: n2

    spot: false`

I wasn't able to get this to run under autopilot, as GKE warden complaint about "Error from server (GKE Warden constraints violations): error when creating "compute-class.yaml": admission webhook "warden-validating.common-webhooks.networking.gke.io" denied the request: GKE Warden rejected the request because it violates one or more constraints.

Violations details: {"[denied by custom-compute-class-limitation]":["compute-class \"prefer-l4-spot\" must have NodePoolAutoCreation enabled in Autopilot. ComputeClass 'prefer-l4-spot'."]}"