google adk with local llm agent

 We all wanted our agent to be running 24 hours a day to boost productivity and that normally requires a local LLM. agent. Today we are going to look into setting Qwen3.8 local model using unsloth studio. 

First we get unsloth (under the hood it is using llama.cpp) and generate an API token.

Then we will proceed to create python env and setup our packages. 

uv venv

uv init

uv add google-adk litellm litert-llm ollama openai

Lets' go and create your app name by running the command below :-

adk create app 

This will create a directory and some basic scaffolding files

 

Next we will update agent.py with the following code :-

from google.adk.agents.llm_agent import LlmAgent

from google.adk.models.lite_llm import LiteLlm

# Create a LiteLLM model pointing to your local server

model = LiteLlm(

    model="openai/empero-ai/Qwen3.8-2B-Distill-GGUF",

    api_base="http://localhost:8888/v1",  # Your local server

    api_key="sk-unsloth-4d0a1b198bd177a2a72ee1954585342a"

)

# Create your agent

root_agent = LlmAgent(

    model=model,

    name="my_local_agent",

    description="An agent using a local LLM",

    instruction="You are a helpful assistant with access to tools.",

    tools=[],  # Add your tools here if needed

)

Ensure unsloth is running your model - in my case i am running "openai/empero-ai/Qwen3.8-2B-Distill-GGUF"

After that, from the root folder run "adk web".  You will be able to see that we have our web app running that looks like this :- 

Ensure you have seleted app and then you can start to prompt the agent. From here we can go crazy with our agent.

Code can be found here

https://github.com/kepungnzai/google-adk-local-llm