Posts

Showing posts from November, 2019

Azure function weird way of setting up logger

No really sure why we need to specify a type for creating / initializing logger (as indicated by code below)


azure app insights :- understanding Performance the charts

Image
Sometimes when things are not so clear, and it suddenly turns mysterious when another person say else. All you ever get is a huge mess. This is one of those cases

Operation Tab

In this tab, you get to see "What and how long an operations are running on my server"




Operation times give you how long a spike last. What is the peak operation time happened on the server.

Just underneath the "Operation time", you can sub-divide by Request, CPU, Available Memory, Storage.

The story it is telling is, during those time when the spike occurs, how is my resources holding up?
Does memory got sucked out and is the CPU reaching certain limit (since it is showing a number like 0, 1,2) i am thinking it means, average limit between available CPU.

The distribution chart, shows where percentage of operations lies. In my case, 50% of the time, there are over 100 request processed within 64 miliseconds, 90% of the time, under 10 request served within 6.8 seconds window.

Roles tab
Role ta…

Simple QA with BERT

This is the easiest Q and A example that you can find.
Workflow goes something like this,

1. tokenizer to convert into torch tensor. You need to encode beginning with [CLS] and [SEP]. Statement or knowledge is encode with 1. Question is 0.

2. create BERT QA model and run. 

3. get the score out. Interpreting the score is key. You get start and end index of a answer. Represented by start_scores and end_scores below. It is like using a highlighter and crossing out answer to our face.


## initialize the model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased') model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
### Please note here that encoding are given as   #                 0 0 0 0 0 0 0 (denote question) while 1 1 1 1 1 1 1 (denotes answers / knowledge ) question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet" input_text = "[CLS] " + question + " [SEP] "…

Azure function v2 - Logger injection

Here is the code for startup function for doing logger injection





And then in your function app constructor, this is where you start to get instances of your logger.

function app v2 - appsettings local and remote

Azure function app v2 uses standarized approach for getting  app settings / app configuration information.

This is what your start up function looks like :




Metric settings is place to hold your settings info:



And finally, wiring it up to your function app



This is what your configuration looks like :-






Method not found: 'Microsoft.Extensions.DependencyInjection.IServiceCollection

I faced this issues after i upgrade Microsoft.Logging.Extensions. Jezz man, just trying to add more visibility.

Remove this and you're set to go but if you really need to do logging and stuff, probably add reference to the correct nuget package.

For my case it is

"Microsoft.ApplicationInsights.AspNetCore" Version="2.8.2"

COLA - Benchmark top model ranking

BERT for dummies

I will keep this short. BERT is a pre-trained model (trained on wikipedia) that means we skip all those time consuming stuff of tokenizing and encoding those models. Everything here is a ripped off without modification from Google Colab example (please refer to link below for all the codes)

This example is a classification problem and it will classify if a sentence is grammatically correct or not.

Sample used :- COLA - a dataset for benchmarking grammar.

Summary of tasks before getting lost in the crazy codes maze.


- Load all the necessary library - which includes hugging face's BERT and pytorch

- Load this COLA dataset and transform it into a correct format

- Load BERT model

- Predict and Evaluate

The results is true or false.


Done :)


Reference link

https://colab.research.google.com/drive/1ywsvwO6thOVOrfagjjfuxEf6xVRxbUNO


Configuring Azure diagnostic logging retention period

Image
It is just confusing sometimes for developer to specify the correct retention period for Azure diagnostic settings, especially when there is a retention period option next to different types of logs / metrics.
Which one is which?

If you decided to send it to a storage account, then the retention period make sense.
If you sending it to azure diagnostic, then retention period is whatever you specify for the workspace.

How do you go about changing this?

Go to -> Azure diagnostic logging workspace -> Cost and billing -> Data Retention.





Google open IA natural dataset and model ranking

Link to google open IA natural dataset and model ranking


https://ai.google.com/research/NaturalQuestions/dataset

Appinsights : Do you know what you're doing

This link here offers great practical approach to train yourself with appinsights.

https://www.azuredevopslabs.com/labs/azuredevops/appinsights/

Conversational Question Answering Challenge : CoQa

Check out the leader board rankings here :

https://stanfordnlp.github.io/coqa/

azure devops - build the proper way

Been a while since i did anything with Azure Devops build pipeline

A successful and proven deployment way to do build is using the following yml. This can be easily extracted and use by webdeploy for releases

Git - Getting and changing remote url

I happened to have updated my username and password for a git repo that i worked with. The thing is, info is persisted in the git repository and giving exceptions whenever i tried to push stuff across.


To view config url 

git config --get remote.origin.url


To Update config url for your repo

git remote set-url origin https://username:password@dev.azure.com/someorg/repository_url

powerbi service - creating streaming dataset

Image
Goto -> My Workspace -> (you will see your dashboard) -> goto Datasets (tab) and then  look for "+ Create" on top right of your screen.

Please hav a look at the screenshots below :-





patch - python unit test

To use patch (to spy on arguments and methods) called, here is a code example :-


simple mocking in python

There are basically 2 main mock object, mock and magicmock.
Here is a simple example of python mocking