mitzen

Understanding otel collector and how it works is important for us to configure and push relevant metrics over. The focus of this article is to provide a simple sraight forward configurations. There are more complex setup available, we will cover those later.  This is an example of the OTEL architecture  Receiver First we have the " Receiver " - where data comes in. In this setup, data can be pushed via port 4318 (over http) and 4317 (over gRPC). # collector-config.yaml receivers :   otlp :     protocols :       grpc :         endpoint : 0.0.0.0:4317       http :         endpoint : 0.0.0.0:4318 Exporter  Here we have configure our otel collector to log metric to console (debug) and prometheus on port 8888. So we are exposing a new promethus service here and any prometheus services (if configured correctly) would be able to scrap these out. The endpoint exposed is /metrics exporters : ...

In this example, we will be pushing or in better term exporting metric to our otel-collector. Generally there are a couple of approach for otel collector to obtain metrics data 1. push - this is where application push metric data to it  2. pull - this is where it scrap for metrics  3. others In this example, we will be setting up for scenario 1 - where app will push metrics. Let's setup our otel collector first Here is our config. # collector-config.yaml receivers :   otlp :     protocols :       grpc :         endpoint : 0.0.0.0:4317       http :         endpoint : 0.0.0.0:4318 exporters :   debug :     verbosity : detailed service :   pipelines :     traces :       receivers : [ otlp ]       exporters : [ debug ]     metrics :       receivers : [ otlp ]       exporters : [ debug ]   ...

  We can create a vectorized search for our Azure AI Search without going through setting up Azure Foundry embedding model. We will just use standard embedding "hnsw" and we also don't require a indexer for now, probably when we have more document to index.  The setup process would be  1. Create index 2. Embed and upload your docs - we require this to show how we can vectorized our document so we can test it out later 3. Perform vector search Creating index We can create our vector index called "index-vector" using the following code. As you can see here, we are also embedding and uploading the document  from azure . identity import DefaultAzureCredential from azure . core . credentials import AzureKeyCredential from azure . search . documents . indexes import SearchIndexClient from azure . search . documents import SearchClient from azure . search . documents . indexes . models import (     ComplexField ,     SimpleField ,   ...

  To setup Azure AI Search for markdown or other document type, we need to be setting up the following resources:-  - Index  - Data source - Indexer Then we are going to bring it together :- Creating our index This is what our index json setup looks like :- {   "@odata.etag" : "\"0x8DF054C596E2837\"" ,   "name" : "markdown-index" ,   "purviewEnabled" : false ,   "fields" : [     {       "name" : "id" ,       "type" : "Edm.String" ,       "searchable" : true ,       "filterable" : true ,       "retrievable" : true ,       "stored" : true ,       "sortable" : true ,       "facetable" : true ,       "key" : true ,       "synonymMaps" : []     },     {       "name" : "content" ,       "type" : "Edm.String" ,       "searchable" : true ,...

Besides hooking up your search service to an AI, you might want to connect to your search service to retrieve data you already indexed. We will be creating index for our json data stored in a storage account via portal - it will automatically create index and indexer for us.  Besides JSON, you can index other data such as SQL database, mark down and even Apache Spark. Index allow our document to be ' structure and define for faster search '. Indexer perform actual indexing to our data and we can specify or schedule when we wants it. Pre-requisite  AI search  Storage acccount with hierarchical name enabled python code  First we click on " Import data " and select " Data Lake " And this is what our storage account looks like :- And then let's configure our json:- click " next " :- and then accept the default :- And then you can configure the schedule for our index to run. Use the default and configure it to run "Once". And finally provi...

One of the most overlook skills in cloud is designing or allocating subnet. Let's say your  vnet is created with address space of 10.0.0.0/16 - about 65,526 - we like to keep our VNET optimium with less fragmentations as possible.  Here is a list that we can use to ensure we are able to size our subnet for different setup /29, /28, /27 and /26. This make our life easier.  This would really depends on how your workload grows in a given subnet. For application that you would like to place them in a specific subnet and plan for certain expansion. If it grows beyond that, then you might need to provision a larger subnet. 

Cline provide free version of Ox Alpha too.  

We can use packages from a git repository, for example -  https://github.com/kepungnzai/dbt-tutorial-setup.git To do that we can update our packages.yml in the root folder - packages.yml  packages :   - package : dbt-labs/dbt_utils     version : ">=1.3.0"   - package : dbt-labs/audit_helper     version : 0.14.0   - git : https://github.com/kepungnzai/dbt-tutorial-setup.git     revision : main then run  dbt deps And dbt get the packages from git And as you can see, the package apear under our folder - dbt_packages:-  

In dbt we can check and assess our data to see if it is stale or has not been updated for some time.  We can use the following configuration here to do it - config-> freshness. # dbt test --select "test_type:data" version : 2 sources :   - name : SNOWFLAKE_SAMPLE_DATA     database : SNOWFLAKE_SAMPLE_DATA       schema : TPCDS_SF10TCL     config :       freshness :         error_after :           count : 2           period : day     tables :       - name : store_sales We can run " dbt source freshness ".  In the first section, it error out because the day last update is 1 year 8 moths ago  We can easily bump up the number here to 3285 (9 years).  # dbt test --select "test_type:data" version : 2 sources :   - name : SNOWFLAKE_SAMPLE_DATA     database : SNOWFLAKE_SAMPLE_DATA       schema :...