mitzen

We can extend our dbt capabilities through packages just like packages in other languages like nuget in dotnet. In fact, we might already been using packages already, if you are following previous tutorials where we use materialization and sources keyword - this package is called dbt_utils.  Let's see how we can confugure it :- packages :   - package : dbt-labs/dbt_utils     version : " >=1.3.0 "   - package : dbt-labs/audit_helper     version : 0.14.0 And them run " dbt deps " That's how we can install it If you're looking out for more dependencies then please go to dbt hub.https://hub.getdbt.com/ So you have the dbt util install how can you use it? Let's try    dbt_utils.equal_rowcount - this is quite a simple example to compare 2 table in your test if they have the same row count. In your model directory and look for your model that you like to compare. In my case it is model/snowflake_sample_data and then add the tests in  ...

We can use python in our dbt project as well. Here is an example of code that you can place under model folder def model ( dbt , session ):     dbt . config ( materialized = ' table ' )         source_store_sales = dbt . source ( ' SNOWFLAKE_SAMPLE_DATA ' , ' store_sales ' )         return source_store_sales This is equilvalent to : {{ config( materialized = ' view ' )}} with source_store_sales as (     select * from {{ source ( ' SNOWFLAKE_SAMPLE_DATA ' , ' store_sales ' ) }} ), renamed as (     select       *     from source_store_sales ) select * from renamed There are limitation when using python code where it only supports table and incremental. 

dbt data test is used to test validity of the data during the Transform and load stage and that can be easily setup using the followings example Please note: The model is your file name which is  " snowflake_sample_data_store_sales ". Remember dbt always use the filename as model name.  Data test models :   - name : snowflake_sample_data_store_sales     description : List of store sales records with basic cleaning and transformation applied.     columns :       - name : SS_ITEM_SK         description : The unique key for each location.         data_tests :           - not_null if you run dbt test now, you will get the following output here: Unit test  We also have unit test in dbt - that allow us to run unit test against our data. This is useful to ensure our sql used for our TL is working as expected.  You need to ensure the model name matches your filenam...

We can find the actual path of your executable by using powershell command. Normal we added some exec somewhere and forgot about it.  The powershell command can help to trace where it is, save me bunch of time trying to look for this exec. ( Get-Command dbt ) .Source 

Let's say you do not want certain resources to be created. This can be controlled via Azure policy and can be scope to subscription or resource group level. We will prevent storage account from being created in a resource group and here is how we can do it. First goto Azure Policy - Authoring - Definition and look for ' Not allowed resource types '.  Then click on "Assign Policy" and for my test, I will scope this to Resource group level. And then provide a name to it like so, Next, is where we specify "Resource Type" - let's select storageAccounts and then click on "Next".  And now if you switch over to "Assignments", you will notice your policy gets created.  Now it is time for test out your policy by creating a storage account in myfdrydev-rg . And then you will be hit with an error.

To kick my very own LLM model that can be train and learned from simple dataset - i started off with a bare minimum model that allows me to learn really simple stuff like 1 + 1 = 2, 2 + 2 = 4.   So first we outline our vocabulary and size - how we are representing this information to the LLM model Vocabulary and token Then we tokenize those input for training and inference. This model only understands these vocab Special: <pad> , <start> , <eos> Operators: + , - , = Numbers: 1 , 2 , 3 , 4 Simple transformer with: Embedding Layer : Converts token IDs to 32-dim vectors and this is the layer where we handle our vocab      # Token embedding   self . embedding = nn . Embedding ( vocab_size , d_model , padding_idx = 0 ) Positional Encoding : Learnable position embeddings      # Positional encoding (learnable)   self . pos_embedding = nn . Embedding ( seq_len , d_model ) Transformer Encoder : 1 layer with 2 attentio...