Posts

dbt: running unit test and data test

Image
dbt data test is used to test validity of the data during the Transform and load stage and that can be easily setup using the followings example Please note: The model is your file name which is  " snowflake_sample_data_store_sales ". Remember dbt always use the filename as model name.  Data test models :   - name : snowflake_sample_data_store_sales     description : List of store sales records with basic cleaning and transformation applied.     columns :       - name : SS_ITEM_SK         description : The unique key for each location.         data_tests :           - not_null if you run dbt test now, you will get the following output here: Unit test  We also have unit test in dbt - that allow us to run unit test against our data. This is useful to ensure our sql used for our TL is working as expected.  You need to ensure the model name matches your filenam...

windows - finding your executable file when "where" command didn't do good

We can find the actual path of your executable by using powershell command. Normal we added some exec somewhere and forgot about it.  The powershell command can help to trace where it is, save me bunch of time trying to look for this exec. ( Get-Command dbt ) .Source 

Azure policy in action : preventing type resources from being created

Image
Let's say you do not want certain resources to be created. This can be controlled via Azure policy and can be scope to subscription or resource group level. We will prevent storage account from being created in a resource group and here is how we can do it. First goto Azure Policy - Authoring - Definition and look for ' Not allowed resource types '.  Then click on "Assign Policy" and for my test, I will scope this to Resource group level. And then provide a name to it like so, Next, is where we specify "Resource Type" - let's select storageAccounts and then click on "Next".  And now if you switch over to "Assignments", you will notice your policy gets created.  Now it is time for test out your policy by creating a storage account in myfdrydev-rg . And then you will be hit with an error.

LLM from Scratch - 1

To kick my very own LLM model that can be train and learned from simple dataset - i started off with a bare minimum model that allows me to learn really simple stuff like 1 + 1 = 2, 2 + 2 = 4.   So first we outline our vocabulary and size - how we are representing this information to the LLM model Vocabulary and token Then we tokenize those input for training and inference. This model only understands these vocab Special: <pad> , <start> , <eos> Operators: + , - , = Numbers: 1 , 2 , 3 , 4 Simple transformer with: Embedding Layer : Converts token IDs to 32-dim vectors and this is the layer where we handle our vocab      # Token embedding   self . embedding = nn . Embedding ( vocab_size , d_model , padding_idx = 0 ) Positional Encoding : Learnable position embeddings      # Positional encoding (learnable)   self . pos_embedding = nn . Embedding ( seq_len , d_model ) Transformer Encoder : 1 layer with 2 attentio...

github dependabot configuration and simple demo

Image
Github dependabot scans your repository for dependencies that are outdated, vulnerable package, run tests automatically and can even auto-merge if it is safe to do so.  It is different from CodeQL or CQL where CQL would do security code scanning for coding and vulnearabilities.  How to enable dependabot?  Goto your repository -> Settings -> Advanced Security -> under the tab Dependabot and then turn it on. To enable your package malware alerts, you can click on the "Dependabot rules" and then you will see this layout here where you can enable it by clicking on the "pencil" button. When will you get the scan results?  You typically get the scan results minutes.  In my repo here, https://github.com/kepungnzai/dotnet-dependabot-test - we have old and vulnerable packages and then dependabot come back with a PR for me which looks something like this - which is amazing! 

transitioning pipelines to a devsecops pipeline design

  Traditional DevOps focuses on speed and reliability but often treats security as an afterthought — vulnerabilities are caught too late in production. DevSecOps integrates security at every stage of the pipeline. Here are the critical additions: The "Shift Left" Principle Instead of finding security issues after deployment, DevSecOps catches them early : Pre-commit : Scan for secrets before code is pushed Build time : SAST analysis, dependency checks, container scanning Pre-deploy : Verify signatures, compliance gates, IaC security Runtime : Intrusion detection, vulnerability monitoring Post-deploy : DAST testing, regression checks # DevSecOps vs DevOps Pipelines in GitHub Actions ## Core Differences ### Traditional DevOps Pipeline - ** Focus ** : Speed and reliability - ** Security ** : Added at the end (security testing after deployment) - ** Approach ** : "Shift right" - security concerns are addressed late ### DevSecOps Pipeline - ** Focus ...