Eval is important to ensure our model and apps works as expected - or atleast probablistically.
With google adk we can start running our eval set with some environment setup (powershell)
$env:GOOGLE_GENAI_USE_VERTEXAI="TRUE"
$env:GOOGLE_CLOUD_PROJECT="project-xxx"
$env:GOOGLE_CLOUD_LOCATION="global"
To get started we need 2 things - the dataases and evaluation criteria.
Getting the evalset.json - where we would like to test for a scenario - this is similiar to providing input and defining the expected output like how we would do it in a unit test.
{
"eval_set_id": "hello_world",
"name": "hello world smoke test",
"description": "No provider named, so intake should default to gcp and mark it as assumed.",
"eval_cases": [
{
"eval_id": "no_provider_defaults_to_gcp",
"conversation": [
{
"invocation_id": "inv-1",
"user_content": {
"parts": [{ "text": "Build a hello world web service." }],
"role": "user"
},
"final_response": {
"parts": [{ "text": "Delivery note for a hello world service on GCP." }],
"role": "model"
},
"intermediate_data": {
"tool_uses": [
{ "name": "set_target_cloud", "args": { "cloud": "gcp", "assumed": true } }
],
"intermediate_responses": []
}
}
],
"session_input": {
"app_name": "loop_agent",
"user_id": "eval_user",
"state": {}
}
}
]
}
This is the test_config.json and we are evaluating the model based on these. Tool_trajectory_avg_score measures tool call usage.
{
"criteria": {
"tool_trajectory_avg_score": { "threshold": 1.0, "match_type": "IN_ORDER" }
}
}
To execute this eval run, use
uv run adk eval loop_agent loop_agent/evals/hello_world.evalset.json --config_file_path=loop_agent/evals/test_config.json --print_detailed_results
And this is an example of my runs :-
Comments