otel tail sampling intro
Otel tail sampling is a really powerful feature that allow us to configure what tracses, log and metrics that we're going to be pushing to Application insights or other monitoring tools. I had a go with filtering where I tried to send traces only for transactions greater than 2 secs. Soon I realized that is quite limiting and not usable because we need those for performance analysis. All we ever see is slowness in our system. Tail sampling is a more customizable feature that allow us to configure stuff like 1. send error traces always 2. send request that has high latency times 3. high value transactions to ensure we are able to trace it 4. we can also impose sampling percentage to send over like 10% or 50% These are the configuration that we keep on the otel collector. processors : tail_sampling : decision_wait : 10s # Time to wait for all spans of a trace to arrive num_trac...