otel tail sampling intro

Otel tail sampling is a really powerful feature that allow us to configure what tracses, log and metrics that we're going to be pushing to Application insights or other monitoring tools. 

I had a go with filtering where I tried to send traces only for transactions greater than 2 secs. Soon I realized that is quite limiting and not usable because we need those for performance analysis. All we ever see is slowness in our system.  

Tail sampling is a more customizable feature that allow us to configure stuff like 

1. send error traces always

2. send request that has high latency times 

3. high value transactions to ensure we are able to trace it 

4. we can also impose sampling percentage to send over like 10% or 50% 

These are the configuration that we keep on the otel collector.





processors:
  tail_sampling:
    decision_wait: 10s           # Time to wait for all spans of a trace to arrive
    num_traces: 100000           # Max number of active traces kept in memory
    expected_new_traces_per_sec: 1000
    policies:
      # 1. Always keep traces that contain errors
      - name: error-policy
        type: status_code
        status_code:
          status_codes: [ERROR]

      # 2. Always keep slow requests (e.g., latency > 500ms)
      - name: latency-policy
        type: latency
        latency:
          threshold_ms: 500

      # 3. Keep traces matching custom business attributes (e.g., high-value checkouts)
      - name: high-value-transactions
        type: numeric_attribute
        numeric_attribute:
          key: transaction.amount
          min_value: 10000

      # 4. Fallback: Keep a small probabilistic percentage (e.g., 5%) of normal traffic
      - name: probabilistic-fallback
        type: probabilistic
        probabilistic:
          sampling_percentage: 5.0

# Sample 2% of everything else (normal, healthy traffic)
      - name: healthy-traffic-fallback
        type: probabilistic
        probabilistic:
          sampling_percentage: 2.0






Comments

Post a Comment

Popular posts from this blog

NodeJS: Error: spawn EINVAL in window for node version 20.20 and 18.20

Encounter this issue when running autorest (that uses nodejs).  I am using node version:  v18.20.2 After i revert to use node v18.12.0, then i was able to get my app to run without this error. Since i am working with Azure devops yaml, i was able to use the following to resolve my issue.  - task : NodeTool@0   inputs :     versionSource : 'spec'     versionSpec : '18.18.0'
Read more

llama cpp running it in google colab

Image
There are 2 options to do this in Google  colab.  Option 1 (Easiest)  With this option, we can just install the relevant package and then run the model.  First we need to install the required packages using the following command:-  ! pip install -U llama-cpp-python Next,  we will try to get the model using the following command # !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained(   repo_id= "unsloth/Qwen3-0.6B-GGUF" ,   filename= "Qwen3-0.6B-IQ4_NL.gguf" , ) We can see the model being downloaded:- And then we will run the following command to test this model llm.create_chat_completion(   messages = [     {       "role" : "user" ,       "content" : "What is the capital of France?"     }   ] ) And the output will look something like this:-  Option 2 llama.cpp is a powerful inference engine and if you wanted to get it running in google colab, you co...
Read more