Posts

google agentic deploying

We can use the following python to perform deployment. Also notice that this is more configurable compare to glcoud command line.  So we will be deploying an agent to google agent runtime and it will use an gateway. This gateway is an ingress (client-to-agent)  and it is called my-first-gateway. If you have another setup client-to-everywhere else (egress) setup would be slightly different. import asyncio from urllib import response from google . adk . agents import Agent from google . adk . runners import Runner from google . adk . sessions import InMemorySessionService from google . genai import types from google . adk . agents . callback_context import CallbackContext from google . adk . models . llm_request import LlmRequest from vertexai import types as vertex_types from google . genai import types import vertexai # Ensure this matches your actual GCP Project ID, not a placeholder vertexai . init ( project = "project-015f2405-16e7-4b36-996" ,...
Post a Comment
Read more

local llm image understanding using GLM OCR

Image
GLM OCR is pretty lightweight and fit for purpose when i tried it out on invoice images.  Here is the code for setting up and running it locally.  """ Extract line items from an invoice image using a local model served at http://localhost:8888/v1, called through LiteLLM. Setup:     pip install litellm pydantic     export LOCAL_API_KEY="your-local-server-key" Usage:     python extract_invoice_local.py invoice.jpg """ import base64 import json import mimetypes import os import re import sys from typing import Optional import litellm from pydantic import BaseModel # Same settings as your LiteLlm config --------------------------------------- MODEL = "openai/ggml-org/GLM-OCR-GGUF" API_BASE = "http://localhost:8888/v1" API_KEY = "sk-unsloth-4d0a1b198bd177a2a72ee1954585342a"   # keep the key out of source code TEMPERATURE = 0.0 # Ask the server to constrain output to the schema (llama.cpp-based server...
Post a Comment
Read more

ai platform using llama-4-maverick 17b for image understanding

Image
We have been dealing with alot of text completion and it is time that we try to use model with image. This is a sample code that we can use to extract contents from an image (invoice) and return line item in json format - so we can proceed it or fill up a form.  I assume you have enable llama-4-maverick serveless instance.  """ Extract line items from an invoice image using Llama 4 Maverick on Vertex AI, through the OpenAI-compatible Chat Completions endpoint. Setup:     pip install openai pydantic google-auth requests     gcloud auth application-default login      # or use a service account Usage:     python extract_invoice_vertex.py invoice.jpg """ import base64 import json import mimetypes import re import sys from typing import Optional import google . auth import google . auth . transport . requests from openai import OpenAI from pydantic import BaseModel PROJECT_ID = "YOUR-PROJECT-ID" LOCATION = "us-ea...
Post a Comment
Read more

google adk using model from AI platform via endpoint

Sometimes we would like to make use of a deployed model in AI platform and this is what post intent to achieve. We wrap our class in BaseLLM and then implement the relevant code in this case override the based method generate_content_async. And plug it into our root agent and use runner to execute this code. This make it really easy to use. import asyncio from typing import AsyncGenerator from google . cloud import aiplatform from google . adk . agents import Agent from google . adk . models import BaseLlm , LlmRequest , LlmResponse from google . genai import types from google . adk . runners import Runner from google . adk . sessions import InMemorySessionService from google . genai import types class VertexEndpointLlm ( BaseLlm ):     project : str     location : str     endpoint_id : str     async def generate_content_async (         self , llm_request : LlmRequest , stream : bool = False  ...
Post a Comment
Read more

gcp agentic platform consuming llama4 serverless

Image
You don't necessary have to deploy llama-4 to use it, all we need to do is 'enable' and then we will get an endpoint where we can consume it.  We can use curl command :- export ENDPOINT=us-east5-aiplatform.googleapis.com exort REGION=us-east5 export PROJECT_ID="project-id"  curl \   - X POST \   - H "Authorization: Bearer $(gcloud auth print-access-token)" \   - H "Content-Type: application/json" https : // $ { ENDPOINT } / v1 / projects / $ { PROJECT_ID } / locations / $ { REGION } / endpoints / openapi / chat / completions \   - d '{"model":"meta/llama-4-maverick-17b-128e-instruct-maas", "stream":true, "messages":[{"role": "user", "content": "Summer travel plan to Paris"}]}'  Or we can just use to consume it via python OpenAI endpoint :- from openai import OpenAI   import google . auth , google . auth . transport . requests creds , _ = ...
Post a Comment
Read more

google ai platform model evaluation (not agent evaluation) - static and dynamic response

Image
Your AI customer service agent passed every test on your local machine. Then it went live and started promising out-of-stock inventory and inventing fake refund policies. We perform evaluation on the target model to ensure it working as expected. To set it up, goto AI platform and then under "Evaluation" click on "New evaluation". We have options to setup eval set for completed dataset (where you have prompt and response) - this response is of course generated by your AI.  Data format required is jsonl with your own pre-defined column - as long as you know which column your prompt and response lies.  In this setup, we are going to use static approach - where we are dealing with eval set from a static data file.  And then when prompt to upload a jsonl file, we can choose "Upload" Then we will use a sample jsonl file here :- { "eval_id" : "product_info_check" , "prompt" : "Do you have wireless headphones in stock?" , ...
Post a Comment
Read more