istio debugging and troubleshooting

To get an understanding on a cluster side level if your proxy is behaving as it sync. This typically means if proxy and istiod are in sync. 

istioctl proxy-status

In this example, if you see it is in sync, that's a good start

To see what istioctl proxy-status is really doing, then you can run the following command

istioctl ps --vklog=9

Listerner

To see what port is currently connected or to see if there's an outgoing connection for your application to Azure SQL Server for example, you can use listener options

istioctl pc listener httpbin-f66f49b6f-jhmmz

You can see there's outgoing connection to DNS, port 53. The port of interest is 8000 - since i am running httpbin. It is important to note that httpbin uses port 8000.

So you can try to narrow down the scope to port 8000 

istioctl pc listener httpbin-f66f49b6f-jhmmz --address 0.0.0.0 --port 8000

Route - Is there a way for my traffic to get out 

So let's say i deliberate change httpbin deployment to use port 8099, then you will see the routes does not match on the port - hence no connectivity and no correct route to the pods. To see this you can run

istioctl pc route httpbin-f66f49b6f-jhmmz 

You can also cross validate this against the listener too - if my pod would accept incoming request on the erroneous port 8000. In the route information above we can that traffic is coming in on port 8000.

Let's see if our pod would accept it by running 

istioctl pc listener httpbin-f66f49b6f-jhmmz 

You can see that it is only listening to port 8099. So no match there. 

Noticed that we have a proper route defined from virtual service, vhost (service) and domain.

Also notice that 8099 ports still remain in the route. 

Taken from 

https://github.com/istio/istio/wiki/Troubleshooting-Istio

To get configuration and stats from a proxy (gateway or sidecar):

• Stats: kubectl exec $POD -c istio-proxy -- curl 'localhost:15000/stats' > stats
• Config Dump: kubectl exec $POD -c istio-proxy -- curl 'localhost:15000/config_dump' > config_dump.json OR istioctl proxy-config all $POD -ojson > config_dump.json
• Clusters Dump: kubectl exec $POD -c istio-proxy -- curl 'localhost:15000/clusters' > clusters
• Logs: kubectl logs $POD -c istio-proxy > proxy.log

To enable debug logging, which may be useful if the default log does not provide enough information:

• At runtime: istioctl proxy-config log POD --level=debug
• For a pod, set annotation: sidecar.istio.io/logLevel: "debug"
• For the whole mesh, install with --set values.global.proxy.logLevel=debug