google adk eval set runnng from the command line

Eval is important to ensure our model and apps works as expected - or atleast probablistically. 

With google adk we can start running our eval set with some environment setup (powershell)



$env:GOOGLE_GENAI_USE_VERTEXAI="TRUE"
$env:GOOGLE_CLOUD_PROJECT="project-xxx"
$env:GOOGLE_CLOUD_LOCATION="global"


To get started we need 2 things - the dataases and evaluation criteria. 


Getting the evalset.json - where we would like to test for a scenario - this is similiar to providing input and defining the expected output like how we would do it in a unit test. 


{
  "eval_set_id": "hello_world",
  "name": "hello world smoke test",
  "description": "No provider named, so intake should default to gcp and mark it as assumed.",
  "eval_cases": [
    {
      "eval_id": "no_provider_defaults_to_gcp",
      "conversation": [
        {
          "invocation_id": "inv-1",
          "user_content": {
            "parts": [{ "text": "Build a hello world web service." }],
            "role": "user"
          },
          "final_response": {
            "parts": [{ "text": "Delivery note for a hello world service on GCP." }],
            "role": "model"
          },
          "intermediate_data": {
            "tool_uses": [
              { "name": "set_target_cloud", "args": { "cloud": "gcp", "assumed": true } }
            ],
            "intermediate_responses": []
          }
        }
      ],
      "session_input": {
        "app_name": "loop_agent",
        "user_id": "eval_user",
        "state": {}
      }
    }
  ]
}

This is the test_config.json and we are evaluating the model based on these. Tool_trajectory_avg_score measures tool call usage.



{
  "criteria": {
    "tool_trajectory_avg_score": { "threshold": 1.0, "match_type": "IN_ORDER" }
  }
}



To execute this eval run, use 

uv run adk eval loop_agent loop_agent/evals/hello_world.evalset.json --config_file_path=loop_agent/evals/test_config.json --print_detailed_results

And this is an example of my runs :- 



Comments

Popular posts from this blog

NodeJS: Error: spawn EINVAL in window for node version 20.20 and 18.20

llama cpp running it in google colab