commander using llm model openai/XHToken/Spark-X2.5-4B-GGUF

When running my commander app with XHToken/Spark-X2.5-4B-GGUF the result is pretty good. The model is able to follow instruction and extracts out the right container image to run and the commands too.  

This is my codebase for setting up my model XHToken/Spark-X2.5-4B-GGUF.



import os
from dotenv import load_dotenv
from google.adk.agents import LlmAgent
from google.adk.models.lite_llm import LiteLlm
from google.adk.tools import google_search


# Create a LiteLLM model pointing to your local server
model = LiteLlm(
    model="openai/XHToken/Spark-X2.5-4B-GGUF",
    api_base="http://localhost:8888/v1",  # Your local server
    api_key="sk-unsloth-4d0a1b198bd177a2a72ee1954585342a" ,
    temperature=0.0,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

from app.prompt import ROOT_AGENT_INSTRUCTION
from app.tools.container_toolset import (
    container_image_finder_tool, run_container_command_tool
)
load_dotenv()

root_agent = LlmAgent(
    name="commander",
    description="Agent that read, understand and execute commands specified in a README or external HTTPS URL document.",
    model=model,
    instruction="You are a command execution agent. You are given a README.",
    tools=[container_image_finder_tool, run_container_command_tool]
)



And this is the output from my run using this model here - which is pretty good :-





Comments

Post a Comment

Popular posts from this blog

NodeJS: Error: spawn EINVAL in window for node version 20.20 and 18.20

Encounter this issue when running autorest (that uses nodejs).  I am using node version:  v18.20.2 After i revert to use node v18.12.0, then i was able to get my app to run without this error. Since i am working with Azure devops yaml, i was able to use the following to resolve my issue.  - task : NodeTool@0   inputs :     versionSource : 'spec'     versionSpec : '18.18.0'
Read more

llama cpp running it in google colab

Image
There are 2 options to do this in Google  colab.  Option 1 (Easiest)  With this option, we can just install the relevant package and then run the model.  First we need to install the required packages using the following command:-  ! pip install -U llama-cpp-python Next,  we will try to get the model using the following command # !pip install llama-cpp-python from llama_cpp import Llama llm = Llama.from_pretrained(   repo_id= "unsloth/Qwen3-0.6B-GGUF" ,   filename= "Qwen3-0.6B-IQ4_NL.gguf" , ) We can see the model being downloaded:- And then we will run the following command to test this model llm.create_chat_completion(   messages = [     {       "role" : "user" ,       "content" : "What is the capital of France?"     }   ] ) And the output will look something like this:-  Option 2 llama.cpp is a powerful inference engine and if you wanted to get it running in google colab, you co...
Read more