google ai platform model evaluation (not agent evaluation) - static and dynamic response
Your AI customer service agent passed every test on your local machine. Then it went live and started promising out-of-stock inventory and inventing fake refund policies.
We perform evaluation on the target model to ensure it working as expected. To set it up, goto AI platform and then under "Evaluation" click on "New evaluation".
We have options to setup eval set for completed dataset (where you have prompt and response) - this response is of course generated by your AI.
Data format required is jsonl with your own pre-defined column - as long as you know which column your prompt and response lies.
In this setup, we are going to use static approach - where we are dealing with eval set from a static data file.
And then when prompt to upload a jsonl file, we can choose "Upload"
Then we will use a sample jsonl file here :-
{"eval_id": "product_info_check", "prompt": "Do you have wireless headphones in stock?", "reference": "Yes, we have wireless headphones in stock! They are priced at $120.00 and feature noise-canceling with a 20-hour battery life. 🎧", "response": "Yes, we have wireless headphones in stock! They are priced at $120.00 and feature noise-canceling with a 20-hour battery life. 🎧"}
{"eval_id": "purchase_history_check", "prompt": "What did I buy recently? My customer ID is CUST001.", "reference": "Here's your recent purchase history for Customer ID CUST001:\n\n* **Order ORD-101** (October 15, 2023): Wireless Headphones, Status: delivered, Total: $120\n* **Order ORD-102** (November 1, 2023): USB-C Cable, Phone Case, Status: refunded, Total: $35\n\nLet me know if you have any other questions or need further assistance! 🛍️", "response": "No idea"}
{"eval_id": "refund_request", "prompt": "I want a refund for order ORD-102 because it was damaged.", "reference": "Your refund for order **ORD-102** due to \"damaged\" has been successfully processed! Refund amount: **$35.0**. Your order status has been updated to **refunded**. \nIs there anything else I can help you with today? 🛍️", "response": "Your refund for order **ORD-102** due to \"damaged\" has been successfully processed! Refund amount: **$35.0**. Your order status has been updated to **refunded**. \nIs there anything else I can help you with today? 🛍️"}
A summary of what we are testing
Then we will add metric as shown here: General Quality metric or you can choose other metric such as Instruction following.
And now we just want to see how well it did and having it recorded in Google cloud.
A Google-hosted Gemini model acting as the judge (autorater). It gets each metric’s prompt template filled with your row and returns a score and an explanation.
And what if you like to do that with your code? This is an example how you can do that. The framework is smart enough to review your jsonl to see it has a column called response. In that case, we don't have to specify target model who will be doing those responses.
Once we have all our settings configure, next we can hit Evaluate button. And we can see an example output here
A Google-hosted Gemini model acting as the judge (autorater). It gets each metric’s prompt template filled with your row and returns a score and an explanation.
And what if you like to do that with your code? This is an example how you can do that. The framework is smart enough to review your jsonl to see it has a column called response. In that case, we don't have to specify target model who will be doing those responses.
The code example is default to BYO eval set.
If you needed to use deployed model endpoint, you can uncomment the respective code. As you may have noticed my location is different for vertex ai and ai platform. The eval is available in us-central and obviously not available in asia-southeast1 - but my model is deployed to asia-southeast-1
import pandas as pd
import vertexai
from google.cloud import aiplatform
from vertexai.evaluation import EvalTask, PointwiseMetric, MetricPromptTemplateExamples
import vertexai
from google.cloud import aiplatform
PROJECT = "your-project-number-not-id"
EVAL_LOCATION = "us-central1" # used by EvalTask / judge
ENDPOINT_LOCATION = "asia-southeast1" # where Nemotron is deployed
ENDPOINT_ID = "mg-endpoint-xxxxxxxxxx"
# Global default: evaluation service region. Call this once, and never call
# aiplatform.init() with a different location afterwards.
vertexai.init(project=PROJECT, location=EVAL_LOCATION)
# Endpoint pinned to its own region, independent of the global setting
endpoint = aiplatform.Endpoint(
endpoint_name=f"projects/{PROJECT}/locations/{ENDPOINT_LOCATION}/
endpoints/{ENDPOINT_ID}",
project=PROJECT,
location=ENDPOINT_LOCATION,
)
# 1. The model under test: prompt (str) -> response (str)
def nemotron_fn(prompt: str) -> str:
# Same ChatML-style prompt that worked in your test; verify against the real template
text = f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
resp = endpoint.predict(
instances=[{"text": text}],
parameters={"sampling_params": {
"max_new_tokens": 256, "temperature": 0.2, "stop": ["<|im_end|>"],
}},
use_dedicated_endpoint=True,
)
return resp.predictions[0]["text"].strip()
# 2. Dataset
eval_df = pd.read_json("./eval_dataset/eval_dataset.jsonl", lines=True)
# 3. Metrics: computation-based (need reference) + model-based (judged by Gemini)
custom_helpfulness = PointwiseMetric(
metric="helpfulness",
metric_prompt_template=(
"Rate the response from 1 to 5 for how well it answers the prompt.\n"
"Prompt: {prompt}\nResponse: {response}\n"
"Give a short explanation, then the score."
),
)
eval_task = EvalTask(
dataset=eval_df,
metrics=[
"exact_match",
"rouge_l_sum",
"fluency",
"coherence",
MetricPromptTemplateExamples.Pointwise.INSTRUCTION_FOLLOWING,
custom_helpfulness,
],
experiment="nemotron-eval", # optional: logs to Vertex AI Experiments
)
result = eval_task.evaluate(model=nemotron_fn)
# bring your own evaluation function (model under test) to the eval task
#result = eval_task.evaluate()
print(result.summary_metrics) # aggregate scores (means, std)
print(result.metrics_table) # per-row: prompt, response, score, explanation
result.metrics_table.to_csv("eval_results.csv", index=False)
And sample output from running the code :-
And this is our outputs for eval_results.csv
eval_id,prompt,reference,response,exact_match/score,rouge_l_sum/score,fluency/explanation,fluency/score,coherence/explanation,coherence/score,instruction_following/explanation,instruction_following/score,helpfulness/explanation,helpfulness/score
product_info_check,Do you have wireless headphones in stock?,"Yes, we have wireless headphones in stock! They are priced at $120.00 and feature noise-canceling with a 20-hour battery life. 🎧","Yes, I have a variety of wireless headphones available. I can help you find the perfect pair based on your needs and budget. Let me know what you're looking for!",0.0,0.18518518,"The response is grammatically perfect, uses appropriate and clear word choices, and flows naturally and seamlessly from one idea to the next, making it completely fluent.",5.0,"The response is perfectly coherent; it directly answers the question, expands on the answer with relevant information, and then offers further assistance in a logical and flowing manner, using clear language and structure.",5.0,"The AI response directly and clearly answers the user's question about whether wireless headphones are in stock with a 'Yes,' and then offers further assistance, which completely fulfills the prompt's intent.",5.0,"The response directly answers the question, confirms availability with a positive note about variety, and proactively offers further assistance, making it a perfect and helpful reply.",5.0
purchase_history_check,What did I buy recently? My customer ID is CUST001.,"Here's your recent purchase history for Customer ID CUST001:
* **Order ORD-101** (October 15, 2023): Wireless Headphones, Status: delivered, Total: $120
* **Order ORD-102** (November 1, 2023): USB-C Cable, Phone Case, Status: refunded, Total: $35
Let me know if you have any other questions or need further assistance! 🛍️",Based on the context: I bought a laptop.,0.0,0.0,"The response is grammatically perfect, uses clear and appropriate word choice, and flows naturally despite its brevity. There are no errors or awkward phrasings.",5.0,"The response is incoherent because it refers to a non-existent 'context' for the information, and then uses the first-person pronoun 'I' to state a purchase, making it unclear who bought the laptop or where this information came from, rather than addressing the user's query about *their* purchase.",1.0,"The AI completely misunderstands the user's query. It cannot know what the user bought and incorrectly states 'I bought a laptop' as if it were the user or had made a purchase itself, which is nonsensical. It also refers to 'context' that was not provided.",1.0,"The response hallucinates an answer ('a laptop') without any information being provided in the prompt about past purchases, making it completely unhelpful and incorrect, despite claiming 'Based on the context' when no such context was given.",1.0
refund_request,I want a refund for order ORD-102 because it was damaged.,"Your refund for order **ORD-102** due to ""damaged"" has been successfully processed! Refund amount: **$35.0**. Your order status has been updated to **refunded**.
Is there anything else I can help you with today? 🛍️","I'm sorry to hear that your order arrived damaged. I can help you with a refund request.
To process this, I'll need a few details:
1. **Order Details**: You've provided order number ORD-102. Is this correct?
2. **Damage Description**: Could you briefly describe the damage? (e.g., broken item, torn packaging, missing parts)
3. **Photos**: Do you have photos of the damaged item and packaging? If so, please upload them.
4. **Reason for Refund**: You mentioned it was damaged - is this the only reason, or are there other issues?
Once I have this information, I can initiate the refund process for you.",0.0,0.22695035,"The response is completely fluent, free of grammatical errors, demonstrates nuanced word choice, and has a natural, seamless flow. It effectively addresses the user's prompt while requesting necessary information in a clear and well-structured manner.",5.0,"The response exhibits a seamless logical flow, expertly organized information, and exceptional cohesion throughout. It starts by acknowledging the user's issue, transitions smoothly to requesting necessary details via a clear numbered list, and concludes by stating the next steps, all while maintaining relevance to the main point of processing a refund for a damaged item.",5.0,"The AI response completely fulfills the user's implicit request to initiate a refund process. It acknowledges the issue, confirms it can help, and then clearly outlines the next steps and information needed to proceed, which is exactly what a user would expect in this scenario.",5.0,"The response perfectly addresses the user's refund request for a damaged item by acknowledging the issue, confirming the ability to help, and clearly outlining all necessary information (order details, damage description, photos, reason) needed to proceed with the refund process.",5.0
Comments