Auto-discover test_config.json only for single-file adk eval input

[ftnext]

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

• Related: Consistency improvements between adk eval and AgentEvaluator #4410

Problem:
adk eval behavior was inconsistent with the expected config discovery flow.
When --config_file_path was omitted, CLI always fell back to default criteria,
instead of using test_config.json located next to an eval file.

Solution:
Added config path resolution in adk eval:

• If --config_file_path is provided, use it as-is.
• If omitted and input is a single eval file, look for <eval_file_dir>/test_config.json.
• If omitted and input is multiple eval files or eval set IDs, do not auto-discover and keep default criteria behavior.

This keeps behavior explicit for mixed-directory multi-file runs while enabling
convenient per-file config discovery for single-file usage.

Testing Plan

Unit Tests:

• I have added or updated unit tests for my change.
• All unit tests pass locally.

Passed locally:

% pytest tests/unittests/cli

========================= 272 passed, 140 warnings in 6.60s =========================

Manual End-to-End (E2E) Tests:

Ran adk eval home_automation_agent tests/fixtures/home_automation_agent/simple_test.test.json
and verified that it uses tests/fixtures/home_automation_agent/test_config.json.

Checklist

• I have read the CONTRIBUTING.md document.
• I have performed a self-review of my own code.
• I have commented my code, particularly in hard-to-understand areas.
• I have added tests that prove my fix is effective or that my feature works.
• New and existing unit tests pass locally with my changes.
• I have manually tested my changes end-to-end.
• Any dependent changes have been merged and published in downstream modules.

Additional context

This PR intentionally limits auto-discovery to single-file input to avoid ambiguous behavior when multiple eval files are provided from different directories.

[gemini-code-assist]

Summary of Changes

Hello @ftnext, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines the configuration discovery mechanism for the adk eval command, addressing an inconsistency where test_config.json was not automatically found for single evaluation files. The changes introduce a more intuitive and explicit resolution process, allowing for convenient per-file configuration discovery in single-file scenarios while ensuring predictable behavior for multi-file or eval set ID inputs.

Highlights

• Enhanced adk eval Configuration Discovery: Implemented a new logic for the adk eval command to intelligently discover test_config.json files, addressing previous inconsistencies.
• Prioritized Explicit Configuration: The --config_file_path argument now explicitly takes precedence if provided, ensuring user-specified paths are always honored.
• Single-File Auto-Discovery: For single evaluation file inputs, adk eval will automatically look for test_config.json in the same directory as the eval file, simplifying configuration for common use cases.
• Consistent Multi-File Behavior: When multiple evaluation files or eval set IDs are provided, auto-discovery is intentionally disabled to prevent ambiguous behavior, maintaining the previous default configuration loading.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• src/google/adk/cli/cli_tools_click.py
  • Introduced a new private helper function _resolve_eval_config_file_path to encapsulate the logic for determining the correct configuration file path based on provided arguments and evaluation input.
  • Refactored the cli_eval command to utilize _resolve_eval_config_file_path for resolving the config_file_path before loading evaluation criteria.
  • Removed the previous direct call to get_evaluation_criteria_or_default and related print statement, integrating the new resolution logic.
• tests/unittests/cli/utils/test_cli_tools_click.py
  • Added comprehensive unit tests for the new _resolve_eval_config_file_path function, covering scenarios such as explicit path preference, auto-discovery for single eval files, and correct handling for eval set IDs and multiple eval files (where no auto-discovery should occur).

Activity

• Unit tests have been added or updated to cover the new configuration resolution logic, and all existing unit tests pass locally.
• Manual end-to-end testing was performed, verifying that adk eval correctly uses test_config.json when invoked with a single evaluation file.
• The author has confirmed adherence to the contribution guide, performed a self-review, added comments to the code, and ensured all checklist items are complete.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a useful feature to auto-discover test_config.json when running adk eval on a single file. The implementation is straightforward, with the core logic encapsulated in a new _resolve_eval_config_file_path helper function. The changes are well-supported by new unit tests that cover the various config file resolution scenarios. I have one suggestion to simplify the logic in the new helper function.

[ryanaiagent]

Hi @ftnext , Thank you for your contribution! We appreciate you taking the time to submit this pull request.
I have left comments for this request in #4410