GeezerVison is showing github.com — Allow to run utPLSQL without certain tags (ut.run(a_tags=>'-notThisTag')) · Issue #983 · utPLSQL/utPLSQL
Allow to run utPLSQL without certain tags (ut.run(a_tags=>'-notThisTag')) #983DescriptionIs your feature request related to a problem? Please describe.
Playing around with tags, for me it is often much easier to tag several problematic (or long-running) tests which I don't want to run in several situtations.
Describe the solution you'd like
Would be great to enable ut-runner Tags-parameter to allow -tag:
begin
ut.run(a_tags=>'-long_running');
end;
This would run all tests which don't have the long_running tag.
begin
ut.run(a_tags=>'wip,-long_running');
end;
This would run all tests that have the wip tag but not the long_running tag
Additional context
We discussed an alternative approach with logic AND/OR operators (a_tags => 'tag1&tag2|tag3) but the - seems much more useful. Activitylwasylow commented on Aug 20, 2019 Hi @pesse
So we would have to do some validation to ensure that tag is not starting with '-' as such in annotation cache and treat the ^-.% as reserved.
We would basically execute all tests ( tagged or not ) that do not have a tag -tag or have no tag at all.
Is that correct ? jgebal commented on Aug 20, 2019 @lwasylow
I think we should only allow single_word tag names.
It is to be considered if they should be case sensitive or not.
to be honest I'm not sure if they are considered case-sensitive today.
Also, it looks like we silently remove/ignore tags that are not single_words with this line of code:
a_list := ut_utils.convert_collection( ut_utils.filter_list(l_tag_list,ut_utils.gc_word_no_space) );
We should probably put warnings on tag names that are not valid (before we filter them out).
We should also put warning and ignore tag names that start with a "-" symbol. alexeyhimself commented on Oct 20, 2019
Also, it looks like we silently remove/ignore tags that are not single_words with this line of code:
a_list := ut_utils.convert_collection( ut_utils.filter_list(l_tag_list,ut_utils.gc_word_no_space) );
This is due to:
Tags should adhere to following rules:
* tags are case sensitive
* tags cannot be an empty string
* tags cannot contain spaces e.g. to create a multi-word tag please use underscores,dashes, dots etc. e.g. test_of_batch
* tags with empty spaces will be ignored during execution
* tags can contain special characters
(given from: http://utplsql.org/utPLSQL/v3.1.7/userguide/annotations.html) lwasylow commented on Oct 20, 2019 Yes, will fix that and add a warning about it. alexeyhimself commented on Oct 20, 2019 I support @pesse 's approach of adding "minus" as a prefix to the tags for the tests that must be excluded:
ut.run(a_tags=>'wip,-long_running');
We would basically execute all tests ( tagged or not ) that do not have a tag -tag or have no tag at all.
Is that correct ?
Yes, as to me this is correct. Consider a use case scenario: test case is marked as
and other test cases have other tags or have no tags at all.
When I call:
ut.run(a_tags=>'-long_running');
then I expect that will be executed ALL test cases EXCEPT those marked with "long_running" tag.
When I call:
ut.run(a_tags=>'wip,-long_running');
then I expect that will be executed ALL test cases EXCEPT those marked with "long_running" tag PLUS those that marked with "wip".
In a conflict situation, when we have intersecting sets:
--%tag(wip)
procedure test_wip;
--%tag(wip,long_running)
procedure test_wip_long;
--%tag(long_running)
procedure test_long;
and when we call:
ut.run(a_tags=>'wip,-long_running');
or
ut.run(a_tags=>'-long_running,wip');
then I expect that will be ALL test cases EXCEPT those marked with "long_running" tag PLUS those that marked with "wip" EXCEPT those marked with "long_running" again. I.e. in example above will be run only
--%tag(wip)
procedure test_wip;
But now this MY expectation goes against the "OR" rule of annotations (http://utplsql.org/utPLSQL/v3.1.7/userguide/annotations.html):
Framework applies OR logic when resolving a tags so any tests / suites that match at least
one tag will be included in the test run.
That's why I'd considered a change to the "OR" rule for annotations and made it this way:
Framework applies OR logic when resolving "positive" tags (without "minus" prefix) and
at the end applies AND NOT logic when resolving "negative" tags (with "minus" prefix).
So processing for the scope for the test run is in the following order:
1. ("OR" rule, legacy): any tests / suites that match at least one "positive" tag will be
included in the test run.
2. ("AND NOT" rule, new feature): any tests / suites that match at least one "negative" tag
will be excluded from the test run.
3. (special case caused by new feature): all "positive" vs. "negative" conflicts will be
resolved to satisfy "negative" tags because the rule "AND NOT" is applied at the end
of the scope formation.
What would you say? jgebal commented on Oct 20, 2019 I would say:
Exclusion is stronger than inclusion and so in your scenario:
--%tag(wip)
--%test
procedure test_wip;
--%tag(wip,long_running)
--%test
procedure test_wip_long;
--%tag(long_running)
--%test
procedure test_long;
and when we call ut.run(a_tags=>'wip,-long_running'); or ut.run(a_tags=>'-long_running,wip');, only the test_wip test gets executed.
This is because of the exclusion logic.
So we say:
- include only
wip items tagged as wip and exclude all long_running
The or logic is applied to inclusion/exclusion list only.
When applying inclusions/exclusions we would use set-based logic that is similar to this operation
select item from items where tag = 'wip'
minus
select item from items where tag = 'long_running'
When you combine filtering by suitepath and filtering by tag and operator is used.
So when you run a suite, only suites that are on the specified suitepath AND contain any of specified tags will be executed. jgebal commented on Oct 20, 2019 @lwasylow - I'm already making changes for tags alexeyhimself commented on Oct 20, 2019 @jgebal, I totally agree with you saying:
I would say:
Exclusion is stronger than inclusion and so in your scenario:
...
And thank you very much for making changes for tags already! Tags are very important for automation tools usages. added 3 commits that reference this issue on Oct 20, 2019This issue will close once commit fa64751 is merged into the 'develop' branch.fa64751 This issue will close once commit 56f6282 is merged into the 'develop' branch.56f6282 This issue will close once commit cdb6d3e is merged into the 'develop' branch.cdb6d3e jgebal commented on Oct 23, 2019 @alexeyhimself
Changes are now merged into 3.1.9 develop
Will be leased soon. alexeyhimself commented on Oct 23, 2019 pesse commented on Oct 23, 2019 Also, cli already supports it. You can just go
utplsql run user/pw@connection --tags=-long_running
with 3.1.8 cli against 3.1.9 core alexeyhimself commented on Oct 23, 2019 @pesse, yes, I've seen that in utPLSQL-cli repo! And I really appreciate that! Thank you!
We will try the feature as soon as utPLSQL v3.1.9 appear!
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|