Posts

Showing posts from May, 2019

Getting spark to read kafka data

Setup kafka bin /zookeeper-server-start .sh config /zookeeper .properties bin /kafka-server-start .sh config /server .properties Create a topic bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test To write something to your topic bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test Working with spark You can get the script from here . To submit spark python code spark-submit --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.1 kafka.py localhost subscribe test

Running pyspark code / script from cli

In this example, we are running a spark python file called test.py.  spark-submit test.py /home/jeremy/test/test.dat

pyspark start with a package from command line

pyspark --packages com.microsoft.azure:azure-eventhubs-spark_2.11:2.3.10

Cloud resource provisioning

Static cloud based resource provisioning is bad. Tools such as terraform needs a dynamic scripting language. There are so many resources created with any kinda of FQDN. And there is not alot of choices to choose from and conflict often occurs. We need a better cloud provision tool that resolve conflict better or provides a scripting interface.

Azure function to access key vault

You can easily configure Azure function to pull secret value from KeyVault. First you need to setup :- 1. Azure function function -> Platform feature -> Identity -> Go ahead and turn on the identity. This is what the called Managed Identity setup as you will get an App Id (guid). Save it and then setup your vault below :- 2. In your Key Vault, Access policy -> Add New -> Select Principal -> Add the name of your Azure function app Id 3. Restart your azure function app. 4. In your settings file, you can have some an configuration entry say, CONN_WEB_APP with the following value, @Microsoft.KeyVault(SecretUri=https://blahblahblah.vault.azure.net/secrets/Conn/e74dfd34e8e4415fbd09c540de2cf556) You're all sorted! :)

Azure function tool publish : Can't determine Project to build. Expected 1 .csproj but found 0

If you encounter this issue, you are probably using older version of azure function tool. To upgrade, use the following command :- sudo npm i -g azure-functions-core-tools@core --unsafe-perm true Then try to publish your app again :- func azure functionapp publish ClaimSubmissionFunction

Vscode - when working with projects

To add new solution dotnet new sln To add existing project into the solution dotnet sln AzureFunctionApp.sln add ClaimSubmission/ClaimSubmission.csproj

Angular : Adding store + reducer

Added reducer or store pretty much the same thing, to me. To setup the required libraries, npm install @ngrx/store --save First you need some sort of actions import { Action } from '@ngrx/store' ; export enum ActionTypes { Login = 'Login' , Logout = 'Logout' , Reset = 'Reset' , } export class Login implements Action { readonly type = ActionTypes . Login ; } export class Logout implements Action { readonly type = ActionTypes . Logout ; } export class Reset implements Action { readonly type = ActionTypes . Reset ; } export type ActionsUnion = Login | Logout | Reset ; Then you setup your reducer :- import { ActionReducer , ActionReducerMap , createFeatureSelector , createSelector , MetaReducer } from '@ngrx/store' ; import { environment } from '../../environments/environment' ; import { ActionsUnion , ActionTypes , Login } from ...

Angular : Adding effects to your project.

To install effects, use the following cli :- npm install @ngrx/effects --save To add effect, you need to setup your Store reducer first. After that everything is so much easier. You only requires this piece of code to setup your effects :- import { Injectable } from '@angular/core' ; import { Actions , Effect , ofType } from '@ngrx/effects' ; import { ActionTypes } from './actions/Login' ; import { map , mergeMap } from 'rxjs/operators' ; @ Injectable () export class AppEffects { constructor ( private actions$ : Actions ) { } @ Effect () loginEffects$ = this . actions$ . pipe ( ofType ( ActionTypes . Login ), map ( action => { console . log ( "login effect taking place...." ) return ({ type: "tested" }); })); } And then configure it, in the app.module.ts imports: [ BrowserModule , FormsModule , AppRoutingModule , StoreModule . forRoot ({ login :...

angular adding routing table

Create a file called "app-routing.module.ts" with the following content :- import { NgModule } from '@angular/core' ; import { Routes , RouterModule } from '@angular/router' ; import { AppComponent } from './app.component' ; //This is my case const routes : Routes = [ { path: '' , component: AppComponent }, { path: 'about' , component: AppComponent } ]; @ NgModule ({ imports: [ RouterModule . forRoot ( routes )], exports: [ RouterModule ] }) export class AppRoutingModule { } Next,  open app.module.ts import { AppRoutingModule } from './app-routing.module' ; And add "AppRoutingModule" @ NgModule ({ declarations: [ AppComponent , CarInsuranceComponent , HouseInsuranceComponent , HealthInsuranceComponent , ], imports: [ BrowserModule , Forms...

CSS grid as a landing page

< div id = "container" > </ div > #container { display : grid ; text-align : right ; grid-template-columns : auto auto auto ; height : 90vh ; background : url ( "assets/background.png" ) no-repeat ; background-position : bottom right ; }

Post quantum encrytion last 17 algorithms

BIKE Classic McEliece CRYSTALS-KYBER FrodoKEM HQC LAC LEDAcrypt (merger of LEDAkem/LEDApkc) NewHope NTRU (merger of NTRUEncrypt/NTRU-HRSS-KEM) NTRU Prime NTS-KEM ROLLO (merger of LAKE/LOCKER/Ouroboros-R) Round5 (merger of Hila5/Round2) RQC SABER SIKE Three Bears The 9 Second Round Candidates for digital signatures are: CRYSTALS-DILITHIUM FALCON GeMSS LUOV MQDSS Picnic qTESLA Rainbow SPHINCS+