Posts

terraform working with for and for_each

terraform "for" is used for data transformation while for_each is used to create resource for_each  locals {   servers = {     "web-1" = { cpu = 2 }     "web-2" = { cpu = 4 }   } } resource "null_resource" "servers" {   for_each = local . servers   # Creates 2 resources   triggers = { cpu = each . value . cpu } } for loop usage can be seen here locals {   names = [ "alice" , "bob" , "charlie" ]     # Transform to uppercase (just data)   upper_names = [ for name in local . names : upper (name)]     # Create map (just data)   name_map = { for name in local . names : name => upper (name)}     # Filter (just data)   short_names = [ for name in local . names : name if length (name) < 5 ] } output "result" {   value = local . upper_names   # Just prints ["ALICE", "BOB", "CHARLIE"] } Both for_each and for can be used ...
Post a Comment
Read more

vscode unable to resolve - openai.types.eval_create_params when package is already installed azure-ai-projects

Image
Bump into this error while trying to run my python code that requires some openai evaluation modules. from openai . types . evals . create_eval_jsonl_run_data_source_param import CreateEvalJSONLRunDataSourceParam , SourceFileID from openai . types . eval_create_params import DataSourceConfigCustom from azure . identity import DefaultAzureCredential from azure . ai . projects import AIProjectClient  Then when I checked my vscode python interpreter, I noticed that my setup wasn't quite right, I had to go back to my .venv folder and start up my vscode from there. Then it was able to resolve the python modules.
Post a Comment
Read more

aws cloudwatch - log analytics options used to query logs

We can use multiple approach to query a logs in cloudwatch - log analytics. 1. Logs Insights QL (the native language)   Example of query would look similiar to this  SOURCE logGroups ( namePrefix : [], class : "STANDARD" ) START =- 3600s END = 0s | fields @timestamp , @message | filter @message like /Error/ | sort @timestamp desc | limit 10000 2. OpenSearch Structured Query Language (SQL)  For teams that prefer industry-standard database syntax, CloudWatch supports OpenSearch SQL. It is highly useful if you need to perform relational actions like JOIN commands across logs. SOURCE "arn:aws:logs:ap-southeast-2:042005083034:log-group:/aws/lambda/my-function" START =- 3600s END = 0s | SELECT status , COUNT ( * ) FROM log_group_name 3. OpenSearch Piped Processing Language (PPL) OpenSearch PPL is an alternative pipeline-based query language. It allows you to process data sequentially through a series of chained commands.  SOURCE "arn:aws:logs:a...
Post a Comment
Read more

OTEL Collector Architecture

Image
Understanding otel collector and how it works is important for us to configure and push relevant metrics over. The focus of this article is to provide a simple sraight forward configurations. There are more complex setup available, we will cover those later.  This is an example of the OTEL architecture  Receiver First we have the " Receiver " - where data comes in. In this setup, data can be pushed via port 4318 (over http) and 4317 (over gRPC). # collector-config.yaml receivers :   otlp :     protocols :       grpc :         endpoint : 0.0.0.0:4317       http :         endpoint : 0.0.0.0:4318 Exporter  Here we have configure our otel collector to log metric to console (debug) and prometheus on port 8888. So we are exposing a new promethus service here and any prometheus services (if configured correctly) would be able to scrap these out. The endpoint exposed is /metrics exporters : ...
Post a Comment
Read more

pushing metric to otel collector

Image
In this example, we will be pushing or in better term exporting metric to our otel-collector. Generally there are a couple of approach for otel collector to obtain metrics data 1. push - this is where application push metric data to it  2. pull - this is where it scrap for metrics  3. others In this example, we will be setting up for scenario 1 - where app will push metrics. Let's setup our otel collector first Here is our config. # collector-config.yaml receivers :   otlp :     protocols :       grpc :         endpoint : 0.0.0.0:4317       http :         endpoint : 0.0.0.0:4318 exporters :   debug :     verbosity : detailed service :   pipelines :     traces :       receivers : [ otlp ]       exporters : [ debug ]     metrics :       receivers : [ otlp ]       exporters : [ debug ]   ...
Post a Comment
Read more

Azure AI Search with vectorized search

Image
  We can create a vectorized search for our Azure AI Search without going through setting up Azure Foundry embedding model. We will just use standard embedding "hnsw" and we also don't require a indexer for now, probably when we have more document to index.  The setup process would be  1. Create index 2. Embed and upload your docs - we require this to show how we can vectorized our document so we can test it out later 3. Perform vector search Creating index We can create our vector index called "index-vector" using the following code. As you can see here, we are also embedding and uploading the document  from azure . identity import DefaultAzureCredential from azure . core . credentials import AzureKeyCredential from azure . search . documents . indexes import SearchIndexClient from azure . search . documents import SearchClient from azure . search . documents . indexes . models import (     ComplexField ,     SimpleField ,   ...
Post a Comment
Read more

Azure AI search for Markdown document

Image
  To setup Azure AI Search for markdown or other document type, we need to be setting up the following resources:-  - Index  - Data source - Indexer Then we are going to bring it together :- Creating our index This is what our index json setup looks like :- {   "@odata.etag" : "\"0x8DF054C596E2837\"" ,   "name" : "markdown-index" ,   "purviewEnabled" : false ,   "fields" : [     {       "name" : "id" ,       "type" : "Edm.String" ,       "searchable" : true ,       "filterable" : true ,       "retrievable" : true ,       "stored" : true ,       "sortable" : true ,       "facetable" : true ,       "key" : true ,       "synonymMaps" : []     },     {       "name" : "content" ,       "type" : "Edm.String" ,       "searchable" : true ,...
Post a Comment
Read more