Posts

Azure AI Search with vectorized search

Image
  We can create a vectorized search for our Azure AI Search without going through setting up Azure Foundry embedding model. We will just use standard embedding "hnsw" and we also don't require a indexer for now, probably when we have more document to index.  The setup process would be  1. Create index 2. Embed and upload your docs - we require this to show how we can vectorized our document so we can test it out later 3. Perform vector search Creating index We can create our vector index called "index-vector" using the following code. As you can see here, we are also embedding and uploading the document  from azure . identity import DefaultAzureCredential from azure . core . credentials import AzureKeyCredential from azure . search . documents . indexes import SearchIndexClient from azure . search . documents import SearchClient from azure . search . documents . indexes . models import (     ComplexField ,     SimpleField ,   ...
Post a Comment
Read more

Azure AI search for Markdown document

Image
  To setup Azure AI Search for markdown or other document type, we need to be setting up the following resources:-  - Index  - Data source - Indexer Then we are going to bring it together :- Creating our index This is what our index json setup looks like :- {   "@odata.etag" : "\"0x8DF054C596E2837\"" ,   "name" : "markdown-index" ,   "purviewEnabled" : false ,   "fields" : [     {       "name" : "id" ,       "type" : "Edm.String" ,       "searchable" : true ,       "filterable" : true ,       "retrievable" : true ,       "stored" : true ,       "sortable" : true ,       "facetable" : true ,       "key" : true ,       "synonymMaps" : []     },     {       "name" : "content" ,       "type" : "Edm.String" ,       "searchable" : true ,...
Post a Comment
Read more

Azure search AI - connecting to the service and search our Index

Image
Besides hooking up your search service to an AI, you might want to connect to your search service to retrieve data you already indexed. We will be creating index for our json data stored in a storage account via portal - it will automatically create index and indexer for us.  Besides JSON, you can index other data such as SQL database, mark down and even Apache Spark. Index allow our document to be ' structure and define for faster search '. Indexer perform actual indexing to our data and we can specify or schedule when we wants it. Pre-requisite  AI search  Storage acccount with hierarchical name enabled python code  First we click on " Import data " and select " Data Lake " And this is what our storage account looks like :- And then let's configure our json:- click " next " :- and then accept the default :- And then you can configure the schedule for our index to run. Use the default and configure it to run "Once". And finally provi...
Post a Comment
Read more

vnet subnet sizing

Image
One of the most overlook skills in cloud is designing or allocating subnet. Let's say your  vnet is created with address space of 10.0.0.0/16 - about 65,526 - we like to keep our VNET optimium with less fragmentations as possible.  Here is a list that we can use to ensure we are able to size our subnet for different setup /29, /28, /27 and /26. This make our life easier.  This would really depends on how your workload grows in a given subnet. For application that you would like to place them in a specific subnet and plan for certain expansion. If it grows beyond that, then you might need to provision a larger subnet. 
Post a Comment
Read more

Ox Alpha is available in Cline

Image
Cline provide free version of Ox Alpha too.  
Post a Comment
Read more

dbt packages using git

Image
We can use packages from a git repository, for example -  https://github.com/kepungnzai/dbt-tutorial-setup.git To do that we can update our packages.yml in the root folder - packages.yml  packages :   - package : dbt-labs/dbt_utils     version : ">=1.3.0"   - package : dbt-labs/audit_helper     version : 0.14.0   - git : https://github.com/kepungnzai/dbt-tutorial-setup.git     revision : main then run  dbt deps And dbt get the packages from git And as you can see, the package apear under our folder - dbt_packages:-  
Post a Comment
Read more

dbt freshness

Image
In dbt we can check and assess our data to see if it is stale or has not been updated for some time.  We can use the following configuration here to do it - config-> freshness. # dbt test --select "test_type:data" version : 2 sources :   - name : SNOWFLAKE_SAMPLE_DATA     database : SNOWFLAKE_SAMPLE_DATA       schema : TPCDS_SF10TCL     config :       freshness :         error_after :           count : 2           period : day     tables :       - name : store_sales We can run " dbt source freshness ".  In the first section, it error out because the day last update is 1 year 8 moths ago  We can easily bump up the number here to 3285 (9 years).  # dbt test --select "test_type:data" version : 2 sources :   - name : SNOWFLAKE_SAMPLE_DATA     database : SNOWFLAKE_SAMPLE_DATA       schema :...
Post a Comment
Read more

dbt using macro

In dbt we can easily create macro to automate common use tasks. One of that would be a audit log operation.  In  your folder called macro, please create a file called "audit_operation.sql". {% macro audit_operation(activity_type) %}   insert into raw . audit (run_id, TYPE_ACITIVITY) values ( '{{ 1 }}' , '{{ activity_type }}' ) {% endmacro %} And then we can use it in our dbt project like so. Instead of placing the SQL code in the commented code, we use macro.  name : jaffle_shop profile : jaffle_shop seed-paths : [ "seeds" ] model-paths : [ "models" ] macro-paths : [ "macros" ] test-paths : [ "tests" ]           # ← Make sure this exists clean-targets :   - "target"   - "dbt_packages" seeds :   jaffle_shop :     +schema : raw # on-run-start:  "insert into raw.audit (run_id, TYPE_ACITIVITY) values ('1', 'pre-hook')" on-run-start : "{{ audit_operation('macro-pre-hoo...
Post a Comment
Read more

dbt using variable to make your sql more flexible

 dbt uses jinja as a template engine to work with file. Therefor we can configure our variable 2 ways. 1. dbt_project.yaml or vars.yml (on the project root) - we can introduce variable by using the followings - as you can see here we are introducing a variable call person.  name : jaffle_shop profile : jaffle_shop seed-paths : [ "seeds" ] model-paths : [ "models" ] macro-paths : [ "macros" ] test-paths : [ "tests" ]           # ← Make sure this exists clean-targets :   - "target"   - "dbt_packages" seeds :   # Builds seeds into '<your_schema_name>_raw'   jaffle_shop :     +schema : raw vars :   person : JohnDoe this is what your vars.yml looks like  vars :   person : JohnDoe2   sold_date_sk : 20160901 2. passing variable in from the command line using --var argument. Please note variable are passed in as json - so we need to do something like this. Please note: it is important to use double quote ...
Post a Comment
Read more

dbt - going enterprise

This is quite a good place to get started with enterprising your dbt   https://www.phdata.io/blog/accelerating-and-scaling-dbt-for-the-enterprise/
Post a Comment
Read more

dbt package using dependencies in your project

Image
We can extend our dbt capabilities through packages just like packages in other languages like nuget in dotnet. In fact, we might already been using packages already, if you are following previous tutorials where we use materialization and sources keyword - this package is called dbt_utils.  Let's see how we can confugure it :- packages :   - package : dbt-labs/dbt_utils     version : " >=1.3.0 "   - package : dbt-labs/audit_helper     version : 0.14.0 And them run " dbt deps " That's how we can install it If you're looking out for more dependencies then please go to dbt hub.https://hub.getdbt.com/ So you have the dbt util install how can you use it? Let's try    dbt_utils.equal_rowcount - this is quite a simple example to compare 2 table in your test if they have the same row count. In your model directory and look for your model that you like to compare. In my case it is model/snowflake_sample_data and then add the tests in  ...
Post a Comment
Read more