Posts

gcp running serverless dataproc with a hello world python script

Image
Pyspark Pi calculation To get thing started, we need a pyspark code - this is a simple pi.py andvthen upload this to your google bucket.  import sys from random import random from operator import add from pyspark . sql import SparkSession if __name__ == " __main__ " :     """         Usage: pi [partitions]     """     spark = SparkSession \         . builder \         . appName ( " PythonPi " )\         . getOrCreate ()     partitions = int ( sys . argv [ 1 ]) if len ( sys . argv ) > 1 else 2     n = 100000 * partitions     def f ( _ : int ) -> float :         x = random () * 2 - 1         y = random () * 2 - 1         return 1 if x ** 2 + y ** 2 <= 1 else 0     count = spark . sparkContext . parallelize ( range ( 1 , n...
Post a Comment
Read more

ADK using Vertex AI RAG

Using RAG with ADK engine, we need to setup our RAG engine corpus and tell our agent to use that as a point of reference. As you can see here RAG_CORPUS contains details of this endpoint.   Then we use a tool called VertexAiRagRetrieval  from google . adk . tools . retrieval . vertex_ai_rag_retrieval import (     VertexAiRagRetrieval , ) Then we hook it up to our agent and make these knowledge accessible  # Initialize tools list tools = [] # Only add RAG retrieval tool if RAG_CORPUS is configured rag_corpus = os . environ . get ( " RAG_CORPUS " ) if rag_corpus :     ask_vertex_retrieval = VertexAiRagRetrieval (         name = " retrieve_rag_documentation " ,         description =(             " Use this tool to retrieve documentation and reference materials for the question from the RAG corpus, "         ),         rag_resources ...
Post a Comment
Read more

ADK app deployment to Vertex AI and consuming it

This is for formality purposes and we constantly need to deploy and consumer agentic app that we deployed to Vertex AI.  To deploy,  we can use the following code. import logging import os import vertexai from dotenv import set_key from vertexai import agent_engines from vertexai . preview . reasoning_engines import AdkApp from rag . agent import root_agent logging . basicConfig ( level = logging . DEBUG ) logger = logging . getLogger ( __name__ ) GOOGLE_CLOUD_PROJECT = os . getenv ( " GOOGLE_CLOUD_PROJECT " ) GOOGLE_CLOUD_LOCATION = os . getenv ( " GOOGLE_CLOUD_LOCATION " ) STAGING_BUCKET = os . getenv ( " STAGING_BUCKET " ) # Define the path to the .env file relative to this script ENV_FILE_PATH = os . path . abspath (     os . path . join ( os . path . dirname ( __file__ ), " .. " , " .env " ) ) vertexai . init (     project = GOOGLE_CLOUD_PROJECT ,     location = GOOGLE_CLOUD_LOCATION ,     staging_bucket = S...
Post a Comment
Read more

VertexAiSessionService creating client getting error -ValueError: Project/location and API key are mutually exclusive in the client initializer.

Bump into this error while trying to test out the adk service that is deployed to vertex AI. It seems like vertex agent can't decide it should use GOOGLE_API_KEY or another.  ValueError: Project/location and API key are mutually exclusive in the client initializer. To resolve this, just have to set   GOOGLE_API_KEY="". Then re-run  your script.
Post a Comment
Read more

Postgres database and table storage and index optimizations

Image
Optimizating database storage is always a key focus area when running databases. Lets get started with some basics.  Database storage used Run the following query to see how much space your database currently using SELECT datname AS database_name , pg_size_pretty(pg_database_size(datname)) AS total_size FROM pg_database ORDER BY pg_database_size(datname) DESC ; And you may get further break down here:- Table size spaces used  Let's see how much space used between data and index  SELECT relname AS table_name , pg_size_pretty(pg_total_relation_size(relid)) AS total_size, pg_size_pretty(pg_relation_size(relid)) AS data_size, pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) AS index_size FROM pg_catalog . pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC ; And we may get the following outputs. We can see how much actual data stored in the data (data_size) and how much taken up by the index_size .  It is important to keep...
Post a Comment
Read more

azure keyvault role assigment for secret

This is a common workflow where user would be granted acceptes to a secret.  However role assignment to secret + version level can still happened like what we see here. But it is invalid. You won't be able to access the secret and the role assignment won't appear either Command will be successful but don't do this  az role assignment create assignee 868f2c6b - a124 - 45b5 - b0d1 - ae6e408a7098   --role "4633458b-17de-408a-b874-0445c86b69e6"   --scope /subscriptions/tenand-id/resourceGroups/mytest-kv-rg/providers/ Microsoft.KeyVault/vaults/pv-test-kv-dev/secrets/mytestsecet/ 550070619b634bee99d2b72d8e3616d8 Instead we should grant role assignment to secret level as the lowest level, for example. Then your role assignment magically appears as well az role assignment create assignee 868f2c6b - a124 - 45b5 - b0d1 - ae6e408a7098   --role "4633458b-17de-408a-b874-0445c86b69e6" --scope /subscriptions/tenantid/resourceGroups/mytest-kv-rg/providers/ Microso...
Post a Comment
Read more

Google ADK - adding A2A support for your agent rapidly

Image
In this post, I am going to show how easy it is to convert your agentic app created using ADK to support A2A. What we will do is setup an agentic app that supports A2A. Then we will bring it up to serve new requests. Finally we will call it from ADK app. Ensure we have the proper module installed uv add google-adk[a2a] And then once we have the dependencies, we have a agent.py file contains code to your hello world agent or get weather agent - something really basic. Notice we purposely put the core code in agent.py so we can have another other deployment options for example AppEngine we can create another file called app_engine.py.  But in our case, we will call it a2a_agent.py as we wrapping our agent.py and expose it as a A2A server.   agent.py code (your typical get weather sample) import datetime from zoneinfo import ZoneInfo from google . adk . agents import Agent from google . adk . apps import App from google . adk . models import Gemini from google . g...
Post a Comment
Read more