istio - monitoring istiod key metrics

Monitoring the health and status of istiod and its key metrics can be quite benefitials to ensure our production system run smoothly. 

To get started, istiod exposes metrics. To view it we can simply do this 

kubectl -n istio-system get pods -l app=istiod

kubectl -n istio-system port-forward istiod-86db895df-ltll2 15014:15014

And then we can expose it by hitting this url.

http://localhost:15014/metrics

And you probably end up with the following screen

The key metrics for our istiod are 

- pilot_xds_push_time - time spend in pushing discovery service 

- memory usages 

  go_memstats_heap_alloc_bytes

  go_memstats_heap_inuse_bytes

Let's go and get thus on prometheus and grafana. 

First we need to expose our istiod locally via kubectl. Todo that we will use this command

kubectl port-forward --address 0.0.0.0 svc/istiod 15041:15014 -n istio-system

Setting up promethus

Let's create a pm.yaml file to host all our configuration. Please note: 192.168.1.79 is my local IP. 

global:

  scrape_interval: 15s

scrape_configs:

  - job_name: 'keycloak'

    metrics_path: '/metrics'

    static_configs:

      - targets: ['192.168.1.79:15041']

To run proemethus we run the following command on powershell. 

docker run --rm --name Prometheus -p 9090:9090 --add-host=host.docker.internal:host-gateway -v ${pwd}/pm.yaml:/etc/prometheus/prometheus.yml prom/prometheus

Ensure your target is up.

Setup grafana. 

We will run grafana with this command:-

docker run --rm --name grafana -p 3000:3000  grafana/grafana

And then we will go and create a new connection. In the promethus server url, ensure you are using "http://host.docker.internal:9090"

Next, go to explore and then you will see all those exposed metrics. You can select those metrics above.