GV

GeezerVison

The World Wide Web, restored to 800 pixels of glory.

Home|About|Gaming cabinet|Archives|How it works|Starter sites

On this site

  • Sign in
  • ActionsCI/CD and automation
  • CodespacesCloud development environments
  • All Features
  • Documentation
  • Training
  • Certifications
  • Build with Copilot
  • Customize Copilot
  • Integrations
  • Trending repos
  • GitHub Sponsors
  • Developer articles
  • Enterprises

This view

  • Original page
  • New address

GeezerVison is showing github.com — fix(gooddata-eval): read set_skills' result rather than its arguments to determine active skills by cobanfurkanx · Pull Request #1865 · gooddata/gooddata-python-sdk

Conversation

@cobanfurkanx

@cobanfurkanx cobanfurkanx commented Oct 10, 2026 •
edited by coderabbitai Bot
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Closes #1779

Description

_set_skills_declarations() / _final_skill_declaration() in packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py and _check_visualization_skill_activated in core/evaluators/visualization.py previously derived the active skill set solely from the set_skills tool call's arguments (what the agent requested).

However, what the agent requests is not always what actually becomes active:

  • The service drops requested names it does not recognise.
  • The service pulls in skills declared as dependencies of the ones requested.

The tool call's own result echoes back the authoritative post-replacement active set (e.g. skills_to_activate). Reading the result is strictly truer to what actually became active than re-deriving it from the request arguments.

Changes

  • Implemented _extract_active_skills(tc) to prefer the authoritative post-replacement set in tc.parsed_result() (skills_to_activate, skill_names, skills), falling back to tc.parsed_arguments().get("skill_names") when the result is absent, unparseable, or errored.
  • Updated _set_skills_declarations() in conversation.py to use _extract_active_skills().
  • Updated _check_visualization_skill_activated() in visualization.py to use _extract_active_skills().
  • Added unit tests in both test_agentic_conversation.py and test_visualization_evaluator.py covering:
    1. Unrecognised name requested (dropped by result -> not credited).
    2. Implicit dependency pulled in by result (not in arguments -> credited).
    3. Result missing (fallback path).
    4. Result unparseable (fallback path).
    5. Result errored (fallback path).
    6. Explicit empty result (deactivates skills).

Summary by CodeRabbit

  • Bug Fixes
    • Skill activation and evaluation now reflect the skills reported by a successful tool result, including dependencies and explicit empty results.
    • When results are missing, unreadable, or indicate failure, evaluation falls back to the requested skills.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions
@cobanfurkanx
fix(eval): read set_skills result rather than arguments to determine …
dc12ddc
…active skills

Prefer the authoritative post-replacement active skills echoed back in the set_skills tool result over its request arguments in both conversation evaluation and visualization evaluation.

When the tool result is missing, unparseable, or indicates an error, fall back to the requested skill_names arguments to preserve compatibility with legacy traces.

Closes gooddata#1779
@cobanfurkanx
cobanfurkanx requested review from hkad98, lupko and pcerny as code owners October 10, 2026 11:02
@coderabbitai

coderabbitai Bot commented Oct 10, 2026 •
edited
Loading

Uh oh!

There was an error while loading. Please reload this page.

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: cfc95777-c865-445d-89d0-3e74dd4ce263

📥 Commits

Reviewing files that changed from the base of the PR and between 8168136 and dc12ddc.


📒 Files selected for processing (4)
  • packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py
  • packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py
  • packages/gooddata-eval/tests/test_agentic_conversation.py
  • packages/gooddata-eval/tests/test_visualization_evaluator.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.



📝 Walkthrough

Walkthrough

The conversation evaluator and visualization evaluator now derive active skills from set_skills results when usable. Both fall back to requested skill arguments when results are missing, unparseable, or indicate an error. Tests cover dropped requested skills, included dependencies, and fallback cases.

Changes

Active Skill Evaluation

Layer / File(s) Summary
Conversation skill declarations and coverage
packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py, packages/gooddata-eval/tests/test_agentic_conversation.py
Conversation declarations use skills from the tool result when available, including an explicit empty list. Tests cover argument fallback and how result skills affect routing and coverage.
Visualization activation checks
packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py, packages/gooddata-eval/tests/test_visualization_evaluator.py
The visualization evaluator checks result skills before falling back to requested skills. Tests cover included and excluded skills, missing or unparseable results, and error results.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Low


Merge Risk: ⚪ Minimal · up to dc12d

No actionable issue remains from this review; the change is mergeable after normal checks.

Pre-merge checks | Passed 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and accurately summarizes the main change: active skills now come from the set_skills result instead of only from the requested arguments.
Linked Issues check Passed The changes satisfy the coding requirements in #1779. Both conversation skill declarations and visualization activation now read the parsed set_skills result first. They support skills_to_activate, sk…
Out of Scope Changes check Passed The pull request changes only the two active-skill detection paths and their unit tests. The test changes directly verify the behavior requested by #1779. No unrelated source behavior or unrelated fil…
Docstring Coverage Passed Docstring coverage is 88.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 25 functions across 4 files.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit checks the skill list twice,
The result sets the names precise.
If results fail or go away,
The request can guide the way.
Soft paws cheer as tests agree.

Comment @coderabbitai help to get the list of available commands.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lupko lupko Awaiting requested review from lupko lupko is a code owner
@pcerny pcerny Awaiting requested review from pcerny pcerny is a code owner
@hkad98 hkad98 Awaiting requested review from hkad98 hkad98 is a code owner

At least 1 approving review is required to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

gooddata-eval: read set_skills' result rather than its arguments to determine active skills

Uh oh!

There was an error while loading. Please reload this page.

1 participant

@cobanfurkanx
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.