istio circuit breaker - setup and analysis

We will lable our default namespace to turn on ambient support. 

kubectl label ns default istio.io/dataplane-mode=ambient

Next we will deploy our httpbin workload

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.26/samples/httpbin/httpbin.yaml

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.26/samples/curl/curl.yaml

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.26/samples/httpbin/sample-client/fortio-deploy.yaml

Deploying waypoint for default namespace

istioctl waypoint apply -n default --enroll-namespace --wait

And we will apply this to turn on telemetry and logging 

apiVersion: telemetry.istio.io/v1

kind: Telemetry

metadata:

  name: enable-access-logging

  namespace: default

spec:

  accessLogging:

    - providers:

      - name: envoy

Next, lets verify the logs are coming through 

kubectl logs --follow deploy/waypoint

When we run the command below, we should be see some logs appearing 

kubectl exec deploy/curl -- curl -s httpbin:8000/get

Now lets configure our circuit breaker by adding a DestinationRule 

apiVersion: networking.istio.io/v1

kind: DestinationRule

metadata:

  name: httpbin

spec:

  host: httpbin.default.svc.cluster.local

  trafficPolicy:

    connectionPool:

      tcp:

        maxConnections: 1

      http:

        http1MaxPendingRequests: 1

        maxRequestsPerConnection: 1

Let's generate some load with this command and we expect to see 503. 

kubectl exec deploy/fortio-deploy -- fortio load -c 3 -qps 0 -n 100 \

  -quiet http://httpbin:8000/get

Sample output could look like this 

[2024-11-27T09:13:39.078Z] "GET /get HTTP/1.1" 503 UO upstream_reset_before_response_started{overflow} - "-" 0 81 0 - "-" "fortio.org/fortio-1.66.5" "6df1d0fb-30cc-4908-b3c4-c9c47ac1bf82" "httpbin:8000" "-" inbound-vip|8000|http|httpbin.default.svc.cluster.local - 10.96.127.222:8000 10.244.2.25:57528 - default

If you have prometheus setup in your cluster, then you will be able to get some stats out. 

Reviewing the metrics 

The key metrics that we would like to see are 

cluster.inbound-vip|8000|http|httpbin.default.svc.cluster.local;.upstream_cx_overflow: 59

cluster.inbound-vip|8000|http|httpbin.default.svc.cluster.local;.upstream_cx_pool_overflow: 0

cluster.inbound-vip|8000|http|httpbin.default.svc.cluster.local;.upstream_rq_pending_overflow: 37

cluster.inbound-vip|8000|http|httpbin.default.svc.cluster.local;.upstream_rq_retry_overflow: 0

We can also query our envoy admin portal using the following command :-

istioctl dash envoy deployment/waypoint.default