Description

alisevych

opened on Oct 10, 2022

Issue body actions

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• Build from the unit-test-bot/rc10102022 branch
• Download plugin
• Check downloaded zip-file size < 100 MB
• Open IntelliJ IDEA 2022.2.2
• Remove previously installed UTBot plugin
• Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• Open the project in the IDE
• Install the downloaded plugin

Go through manual scenarios

Manual scenario #1

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/algorithms/ArraysQuickSort.java file
• Generate tests for the class
• Remove results
• Generate and Run test for a method

Manual scenario #2

• Use default plugin settings
• Open the utbot-sample/src/main/java/org/utbot/examples/mock/CommonMocksExample.java file
• Generate tests with different Mocking options combinations

Manual scenario #3

• Create a new Gradle project with JDK 17
• Add a simple java file to test
• Generate a test in the existing test root

Manual scenario #4

• Create a new Maven project with JDK 8
• Add a simple java file to test
• Generate a test with a new test root

Manual scenario #5

• Create a new Idea project with JDK 11
• Add a simple java file to test
• Generate tests for several classes

Regression testing

• IDEA 2022.1.4 (JRE 11)
• IDEA 2022.2.2 (JRE 17)
• Sandbox
• Nested calls
• Recursion
• Cycles
• Mocks
• UtUtils class generation
• Summaries: tags, plain text
• Summaries: regions
• Exceptions: thrown, caught, check regions

Activity

1. alisevych

   added

   ctg-qaIssue is related to QA processIssue is related to QA process

   on Oct 10, 2022
2. alisevych

   self-assigned this

   on Oct 10, 2022
3. alisevych commented on Oct 10, 2022

   

   alisevych

   on Oct 10, 2022

   MemberAuthor

   More actions

   reproducing #1028
   reproducing #286
   reproducing #747
4. alisevych

   added this to the 2022.10.1 Release milestone on Oct 10, 2022
5. korifey

   added this to UTBot Javaon Oct 10, 2022
6. korifey

   moved this to Todo in UTBot Javaon Oct 10, 2022
7. alisevych commented on Oct 10, 2022

   

   alisevych

   on Oct 10, 2022

   MemberAuthor

   More actions

   new bug with Sandboxing and Fuzzing - many tests with same coverage are generated and no minimized
   #1129
8. alisevych commented on Oct 10, 2022

   

   alisevych

   on Oct 10, 2022

   MemberAuthor

   More actions

   new bug with Recursion - empty test class and
   java.lang.Exception
   at com.jetbrains.rd.util.LoggerKt.defaultLogFormat(Logger.kt:80)
   at org.utbot.rd.loggers.UtRdConsoleLogger.log(UtRdConsoleLogger.kt:24)
   at com.jetbrains.rd.util.SwitchLogger.log(Logger.kt:47)
   at org.utbot.instrumentation.instrumentation.et.RuntimeTraceStorage.visit(TraceHandler.kt:162)
   at org.utbot.examples.recursion.Recursion.fib(Recursion.java:21)
   at org.utbot.examples.recursion.Recursion.fib(Recursion.java:22)
   at org.utbot.examples.recursion.Recursion.fib(Recursion.java:22)
   ...
   at org.utbot.instrumentation.instrumentation.InvokeInstrumentation$invoke$2$result$1.invoke-IoAF18A(InvokeInstrumentation.kt:60)
   #1130 : reproduced once with Recursion, not reproducing with Cycles
9. alisevych commented on Oct 10, 2022

   

   alisevych

   on Oct 10, 2022

   MemberAuthor

   More actions

   Reopened #273 - please see the comment with stack trace
10. alisevych commented on Oct 11, 2022

    

    alisevych

    on Oct 11, 2022

    MemberAuthor

    More actions

    reproducing #711 - static field and static method
11. alisevych commented on Oct 11, 2022

    

    alisevych

    on Oct 11, 2022

    MemberAuthor

    More actions

    new bug : sandboxed test is not compiling because commented stack trace contains "illegal unicode escape" in "jkmqh\uhauh"
    #1134
12. alisevych commented on Oct 11, 2022

    

    alisevych

    on Oct 11, 2022

    MemberAuthor

    More actions

    reproducing #1135
13. alisevych commented on Oct 11, 2022

    

    alisevych

    on Oct 11, 2022

    MemberAuthor

    More actions

    clarification task for Summaries:
    regions with Explicitly thrown exception region #1133
    to be discussed and corrected
14. alisevych commented on Oct 11, 2022

    

    alisevych

    on Oct 11, 2022

    MemberAuthor

    More actions

    reproducing #1141
15. alisevych commented on Oct 12, 2022

    

    alisevych

    on Oct 12, 2022

    MemberAuthor

    More actions

    reproducing #1098 - on different code, when no tests are generated (timeout is minimal for example)
16. alisevych commented on Oct 12, 2022

    

    alisevych

    on Oct 12, 2022

    MemberAuthor

    More actions

    proceeding with RC testing of build#430

    Check with different settings

    • JDK 17 - Gradle
    • JDK 17 - Maven
    • empty test class generation - no sarif report save failure JDK 11
    • empty test class generation - sarif report save failure JDK 17
      failed -> new fix in build#434
    • performance checks:
      reproducing StringExamples generation exceeds timeout in 2-3 times #1141

    Check other parts

    • CLI, JDK 11
    • CLI, JDK 17
17. alisevych

    changed the title [-]Manual testing of RC build#417[/-] [+]Manual testing of RC build#417 - RC build#434[/+] on Oct 12, 2022
18. alisevych commented on Oct 13, 2022

    

    alisevych

    on Oct 13, 2022

    MemberAuthor

    More actions

    proceeding with RC testing of build#434

    Check with different settings - in IDEA 2022.2.3 (new version 05-Oct-2022)

    • JDK 17 - Gradle
    • JDK 17 - Maven
    • JDK 17 - IntelliJ
    • existing test sources root
    • new test sources root
    • Test frameworks: JUnit 4, JUnit 5, TestNG
    • Parametrized
    • Mocks on / mock static off
    • Mocks on / mock static on
    • Performance checks:
    • reproducing StringExamples generation exceeds timeout in 2-3 times #1141
    • reproducing Test generation timeout is exceeded several times for BaseStreamExample #1027

    Check other parts

    • CLI, JDK 11
    • CLI, JDK 17
    • CLI, JDK 8

    Environmental checks

    • macOs, last IDEA, JDK 17,
    • IDEA 2022.2.3 (new version 05-Oct-2022)
19. alisevych

    changed the title [-]Manual testing of RC build#417 - RC build#434[/-] [+]Manual testing of RC build#417 - RC build#430 - RC build#434[/+] on Oct 13, 2022
20. alisevych

    changed the title [-]Manual testing of RC build#417 - RC build#430 - RC build#434[/-] [+]Manual testing of RC[/+] on Oct 13, 2022
21. alisevych commented on Oct 13, 2022

    

    alisevych

    on Oct 13, 2022

    MemberAuthor

    More actions

    new bug #1149 - reproducing on latest main:
    Generate and Run for several classes: tests are generated correctly, but run is executed only for one test class.
22. alisevych commented on Oct 13, 2022

    

    alisevych

    on Oct 13, 2022

    MemberAuthor

    More actions

    new bug: #1160
    IDEA dead lock -trying to save SARIF report - when resources folder is removed before generations
23. alisevych commented on Oct 14, 2022

    

    alisevych

    on Oct 14, 2022

    MemberAuthor

    More actions

    new bug: test generated by Symbolic engine for StringExamples.isNotBlank has long string with 6000+ "\u0000"
    #1157
24. alisevych

    changed the title [-]Manual testing of RC[/-] [+]Manual testing of RC 2022.10.1[/+] on Oct 14, 2022
25. alisevych

    closed this as completedon Oct 14, 2022
26. Repository owner moved this from Todo to Done in UTBot Javaon Oct 14, 2022
27. alisevych

    changed the title [-]Manual testing of RC 2022.10.1[/-] [+]Manual testing of Release 2022.10.1[/+] on Oct 19, 2022

Issue actions