GeezerVison

The World Wide Web, restored to 800 pixels of glory.

GeezerVison is showing github.com — Introduce object creation using builders for fuzzers by Markoutte · Pull Request #2583 · UnitTestBot/UTBotJava

Conversation

Markoutte commented Sep 8, 2023

Copy link

Copy Markdown

Collaborator

Description

We introduce new BuilderObjectValueProvider for Java that finds any method, that can create a value of given type. Together with AbstractObjectValueProvider it works only when ObjectValueProvider and others fails to build a value.

The provider searches any method from any class that can return object of this type or any type that can be inferenced from the given one. It doesn't matter if this method static or not, but if this method is not static, that we suppose that we have a builder class for it. As a builder class we try to find other methods that returns an instance of this builder. This logic added because of default way to create such builder, where any set or add method returns the instance of builder to support chain calling. This builder should work to find methods for factories and singletones as well.

To support this provider, we introduce new routine in fuzzer's API called Modify (the name maybe is not final). It is added for Seed.Recurisve and works as follows: it accepts a sequence of routines that changes or creates new object with given this instance and fuzzed values. For example, we firstly create builder's object and update it with builder method, then finally we call the method, that creates an object using this builder. This sequence is always called when creating an object and never changed its ordering.

In addition, you can use this method, apply some method to the instance and return the instance object itself, for example, to always close a Closable object.

As a result, ContestEstimator covers 53% of guava project code, before it covers only 43%.

How to test

Automated tests

All fuzzer tests should pass.

Manual tests

You can try to generate tests for this little project:

public interface Muzzi {
    int getA();
}

public class RealMuzzi implements Muzzi {

    private final int a;

    private RealMuzzi(int a) {
        this.a = a;
    }

    @Override
    public int getA() {
        return a;
    }

    public static class Builder {

        private int a;

        public Builder setA(int a) {
            this.a = a;
            return this;
        }

        public Muzzi build() {
            return new RealMuzzi(a);
        }
    }
}

public class MuzziTest {
    public int size(Muzzi muzzi) {
        if (muzzi.getA() < 0) {
            return -1;
        } else if (muzzi.getA() > 0) {
            return 1;
        }
        return 0;
    }
}

Self-check list

Check off the item if the statement is true. Hint: [x] is a marked item.

Please do not delete the list or its items.

I've set the proper labels for my PR (at least, for category and component).
PR title and description are clear and intelligible.
I've added enough comments to my code, particularly in hard-to-understand areas.
The functionality I've repaired, changed or added is covered with automated tests.
Manual tests have been provided optionally.
The documentation for the functionality I've been working on is up-to-date.

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

All reactions

Markoutte added 4 commits

September 8, 2023 07:10


          Introduce Builder value provider

a21fcfb


          Rename finally to transformers

e5be238


          Simplify code

4a45ad4


          Fix problem when synthetic methods are used for test generation

510099e

Markoutte added ctg-enhancement

New feature, improvement or change request

comp-fuzzing Issue is related to the fuzzing labels

Sep 8, 2023

Markoutte requested review from IlyaMuravjov and egiptipavel

September 8, 2023 06:58

IlyaMuravjov mentioned this pull request

Sep 8, 2023

Add executable generic parameter type resolving #2584

Merged

5 of 6 tasks

IlyaMuravjov requested changes

Sep 8, 2023

There was an error while loading. Please reload this page.

Markoutte added 2 commits

September 11, 2023 08:29


          Add fuzzed summary

84d51c6


          Remove unnecessary check

0852a76

Markoutte requested a review from IlyaMuravjov

September 11, 2023 05:50


          Preserve transformers when copying Seed.Recursive

59c99e5

IlyaMuravjov requested changes

Sep 11, 2023

View reviewed changes

Comment thread

utbot-java-fuzzing/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt Resolved

Uh oh!

There was an error while loading. Please reload this page.

Comment thread

utbot-java-fuzzing/src/main/kotlin/org/utbot/fuzzing/providers/Objects.kt Resolved

Uh oh!

There was an error while loading. Please reload this page.

Markoutte added 2 commits

September 11, 2023 14:43


          Handle CNFE when class from method parameter is not present in classl…

1caa8de

…oader


          Unify hierarchy calling method

19a9298

Markoutte requested a review from IlyaMuravjov

September 11, 2023 11:49

IlyaMuravjov commented Sep 11, 2023

Copy link

Copy Markdown

Collaborator

Now it finds bridge method org.springframework.format.number.money.CurrencyUnitFormatter.print(Object, Locale): String and latter crashes with ClassNotFoundException: javax.money.CurrencyUnit when trying to create a value for CurrencyUnitFormatter.

I think the codebase is not ready to deal with arbitrary classes that can be found on the classpath and the best solution for now is to make sure that we only create FuzzedType instances from classes that only mention classes that are on the classpath.

In other words some ValueProviders expect that using fuzzedType.classId.allMethods, fuzzedType.classId.allConstructors, and fuzzedType.classId.allDeclaredFieldIds won't throw an exception.

The easiest way to achieve that, that I can see is to make runJavaFuzzing pass ClassDependenciesCheckingValueProvider(delegate = ValueProvider.of(providers) to runFuzzing where ClassDependenciesCheckingValueProvider.generate(description, type):

returns sequenceOf(Seed.Simple(nullFuzzedValue(classClassId)) if type.classId.allMethods, type.classId.allConstructors, or type.classId.allDeclaredFieldIds throws any exceptions
otherwise it returns delegate.generate(description, type), preferably filtering out routines that accept classes, mentioning classes that are not on the classpath

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

IlyaMuravjov added a commit that referenced this pull request

Sep 11, 2023


          Avoid using anyObjectValueProvider that is removed in #2583

b130123

Markoutte commented Sep 12, 2023

Copy link

Copy Markdown

Collaborator Author

@IlyaMuravjov I think that there're a many of other problems that we won't find by this review. Let's create separate issues with reproducers and stack traces for them and fix them. The main goal of this PR is to add this builder for Java, not for Spring. And I'd like to keep Java part as simple as possible without additional wrappers and delegators.

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

IlyaMuravjov mentioned this pull request

Sep 12, 2023

Java fuzzer crashes when referencing classes not in the classpath #2594

Open

IlyaMuravjov added 2 commits

September 12, 2023 22:11


          Use null for partially unresolvable classes in fuzzed Spring tests

bbb102b


          Merge remote-tracking branch 'origin/main' into pelevin/builders-prov…

b9de69d

…iders

# Conflicts:
#	utbot-spring-framework/src/main/kotlin/org/utbot/framework/context/spring/SpringApplicationContextImpl.kt

IlyaMuravjov mentioned this pull request

Sep 12, 2023

Multiple time out tests are generated by fuzzer #2595

Open

IlyaMuravjov commented Sep 12, 2023

Copy link

Copy Markdown

Collaborator

@Markoutte
I've pushed a commit that adds a decorator letting Spring test generation to work with BuilderObjectValueProvider without crashing with ClassNotFoundException. However, this problem is not Spring-specific, so I've also made an issue #2594 showing how it can be reproduced without Spring.

I merged main to resolve conflicts instead of rebasing onto main, because I didn't want to force push into your branch.

Also, this PR reveled a problem in instrumnted-process that marks executions as timed out even if the thing that actually timed out is the argument creation prior to method under test invocation, see #2595.

Next, it seems that Scene.v().classes doesn't contain all classes on the classpath, rather it only includes:

classes that are reachable from classes located in the .class files (as opposed to .jar files)
some classes from the JDK
classes that are reachable from org.utbot.framework.util.classesToLoad

Meaning that for large projects BuilderObjectValueProvider can get significantly slower and the result of 53% stated in the PR description depends on the project under test being provided as .jar file (similarly to how it's done in the ContestEstimator) and not .class files (similarly to how it's done when UtBot is used via Intellij plugin).

Finally, I tried to check by how much providing guava in form of .class files instead of .jar file affects performance, but when I tried generating tests for the following method, the fuzzer crashed with the following exception.

public static boolean isNull(LocalDateTime localDateTime) {
    if (localDateTime == null) return true;
    return false;
}

23:32:56.036 | INFO  | JavaLanguage              | Fuzzing is stopped because of an error
java.lang.ClassNotFoundException: com.google.common.io.TempFileCreator$JavaNioCreator$lambda_userPermissions_3__261
	at java.net.URLClassLoader.findClass(URLClassLoader.java:445) ~[?:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:587) ~[?:?]
	at org.utbot.common.FallbackClassLoader.loadClass(FallbackClassLoader.kt:41) ~[utbot-core-2023.09-SNAPSHOT.jar:?]
	at java.lang.ClassLoader.loadClass(ClassLoader.java:520) ~[?:?]
	at org.utbot.framework.plugin.api.util.IdUtilKt.getJClass(IdUtil.kt:105) ~[utbot-framework-api-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.providers.BuilderObjectValueProvider$generate$1.invokeSuspend(Objects.kt:237) ~[utbot-java-fuzzing-2023.09-SNAPSHOT.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequenceBuilderIterator.hasNext(SequenceBuilder.kt:129) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at org.utbot.fuzzing.ValueProvider$Combined$generate$1.invokeSuspend(Providers.kt:231) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequenceBuilderIterator.hasNext(SequenceBuilder.kt:129) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at org.utbot.fuzzing.ValueProvider$Combined$generate$1.invokeSuspend(Providers.kt:231) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequenceBuilderIterator.hasNext(SequenceBuilder.kt:129) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.FlatteningSequence$iterator$1.ensureItemIterator(Sequences.kt:307) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.FlatteningSequence$iterator$1.hasNext(Sequences.kt:303) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequencesKt___SequencesKt.toCollection(_Sequences.kt:787) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequencesKt___SequencesKt.toMutableList(_Sequences.kt:817) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlin.sequences.SequencesKt___SequencesKt.toList(_Sequences.kt:808) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at org.utbot.fuzzing.FuzzingApi$produce$seeds$1.apply(Api.kt:427) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi$produce$seeds$1.apply(Api.kt:426) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1220) ~[?:?]
	at org.utbot.fuzzing.FuzzingApi.produce(Api.kt:426) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:399) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.reduce(Api.kt:484) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.produce(Api.kt:438) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:399) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.reduce(Api.kt:519) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.produce(Api.kt:437) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:399) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.reduce(Api.kt:519) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.produce(Api.kt:437) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:399) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.reduce(Api.kt:539) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.produce(Api.kt:437) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:399) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz$fuzzOne(Api.kt:331) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.fuzz(Api.kt:350) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi.access$fuzz(Api.kt:1) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at org.utbot.fuzzing.FuzzingApi$fuzz$2.invokeSuspend(Api.kt) ~[utbot-fuzzing-2023.09-SNAPSHOT.jar:?]
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) ~[kotlin-stdlib-1.8.10.jar:1.8.10-release-430(1.8.10)]
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:284) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:85) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.6.3.jar:?]
	at org.utbot.common.ConcurrencyKt.runBlockingWithCancellationPredicate(Concurrency.kt:38) ~[utbot-core-2023.09-SNAPSHOT.jar:?]
	at org.utbot.framework.plugin.api.TestCaseGenerator$generate$2$3.invoke(TestCaseGenerator.kt:204) ~[utbot-framework-2023.09-SNAPSHOT.jar:?]
	at org.utbot.framework.plugin.api.TestCaseGenerator$generate$2$3.invoke(TestCaseGenerator.kt:203) ~[utbot-framework-2023.09-SNAPSHOT.jar:?]
	at org.utbot.common.ConcurrencyKt.runIgnoringCancellationException(Concurrency.kt:47) ~[utbot-core-2023.09-SNAPSHOT.jar:?]
	at org.utbot.framework.plugin.api.TestCaseGenerator.generate(TestCaseGenerator.kt:203) ~[utbot-framework-2023.09-SNAPSHOT.jar:?]
	at org.utbot.framework.process.EngineProcessMainKt$setup$4.invoke(EngineProcessMain.kt:127) ~[utbot-framework-2023.09-SNAPSHOT.jar:?]
	at org.utbot.framework.process.EngineProcessMainKt$setup$4.invoke(EngineProcessMain.kt:112) ~[utbot-framework-2023.09-SNAPSHOT.jar:?]
	at org.utbot.rd.IdleWatchdog$measureTimeForActiveCall$1$2$1.invoke(ClientProcessUtil.kt:115) ~[utbot-rd-2023.09-SNAPSHOT.jar:?]
	at org.utbot.rd.IdleWatchdog.wrapActive(ClientProcessUtil.kt:88) ~[utbot-rd-2023.09-SNAPSHOT.jar:?]
	at org.utbot.rd.IdleWatchdog$measureTimeForActiveCall$1.invoke(ClientProcessUtil.kt:114) ~[utbot-rd-2023.09-SNAPSHOT.jar:?]
	at com.jetbrains.rd.framework.IRdEndpoint$set$1.invoke(TaskInterfaces.kt:182) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.IRdEndpoint$set$1.invoke(TaskInterfaces.kt:182) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.impl.RdCall.onWireReceived(RdTask.kt:362) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.MessageBroker$invoke$2$2.invoke(MessageBroker.kt:57) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.MessageBroker$invoke$2$2.invoke(MessageBroker.kt:56) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.impl.ProtocolContexts.readMessageContextAndInvoke(ProtocolContexts.kt:148) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.MessageBroker$invoke$2.invoke(MessageBroker.kt:56) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.framework.MessageBroker$invoke$2.invoke(MessageBroker.kt:54) ~[rd-framework-2023.1.2.jar:?]
	at com.jetbrains.rd.util.threading.SingleThreadSchedulerBase.queue$lambda-3(SingleThreadScheduler.kt:41) ~[rd-core-2023.1.2.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]

All reactions

Sorry, something went wrong.

Uh oh!

There was an error while loading. Please reload this page.

IlyaMuravjov mentioned this pull request

Sep 15, 2023

Support mock strategies and type replacement in Spring unit test fuzzing #2561

Merged

5 of 6 tasks

IlyaMuravjov added a commit that referenced this pull request

Sep 19, 2023


          Avoid using anyObjectValueProvider that is removed in #2583

f64a5b7

EgorkaKulikov pushed a commit that referenced this pull request

Sep 19, 2023


          Support mock strategies and type replacement in Spring unit test fuzz…

36c23ff

…ing (#2561)

* Support mock strategies and type replacement in Spring unit test fuzzing

* Avoid mocking replaced types

* Only use `InjectMockValueProvider` for `thisInstance`

* Disallow non-mocks when mock can be used

* Move all value provider decorators to one package

* Improve adding of `ReplacedFuzzedTypeFlag`

* Make `map` and `except` apply transformer/filter to ALL value providers

* Avoid using `anyObjectValueProvider` that is removed in #2583

* Make `AnyObjectValueProvider` not extend any interfaces, remove unrelated comment

* Fix compilation after rebase

IlyaMuravjov mentioned this pull request

Sep 20, 2023

Fuzzer ignores generics of class StringMap extends HashMap<String, String> #2571

Open

Reviewers

egiptipavel

Awaiting requested review from egiptipavel

IlyaMuravjov

Awaiting requested review from IlyaMuravjov

Requested changes must be addressed to merge this pull request.

Uh oh!

There was an error while loading. Please reload this page.

Assignees

No one assigned

Labels

comp-fuzzing

Issue is related to the fuzzing

ctg-enhancement

New feature, improvement or change request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Uh oh!

There was an error while loading. Please reload this page.

2 participants

Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

You are visitor 01402503 — GeezerVison does not store the pages you look up.
Best viewed with Netscape Navigator 4.7 or Internet Explorer 6 at 800×600. No cookies required.