GeezerVison is showing github.com — pr05 Visual Regression Testing #1: Initial Visual Testing Framework by Vaivaswat2244 · Pull Request #1261 · processing/processing4
Conversation
Visual Testing Framework for Processing
Summary
This PR introduces a comprehensive visual regression testing framework for Processing, enabling automated detection of visual changes in rendering output across different platforms. This also opens up the scope for further discussions and modifications to the approach.
Implementation Details
Core Components
1. Sketch Runner with Controlled Execution
- Executes Processing sketches in a controlled test environment
- Captures single frame after
setup() and one draw() call
- Automatic window cleanup after screenshot capture
- Configurable render wait times for complex sketches
2. Test Runner (VisualTestRunner.java)
- Captures screenshots from Processing sketches
- Manages baseline images with platform-specific naming
- Compares actual output against saved baselines
- Automatically creates baselines on first run
3. JUnit 5 Integration (VisualTest.java)
- Base test class extending JUnit 5
- Provides
assertVisualMatch() helper method
- Supports test tagging and suite organization
- Handles first-run baseline creation gracefully
4. Image Comparison Algorithm (ImageComparator.java)
- Implements a sophisticated pixel-matching algorithm
- Detects and filters out minor rendering artifacts (line shifts, anti-aliasing differences)
- Configurable thresholds for pass/fail criteria
- Generates diff images for failed tests showing exact pixel differences
Project Structure
Key Features
1. Hierarchical Screenshot Organization
- Screenshots organized by test suite:
__screenshots__/suite-name/test-name-platform.png
2. Test Suite Organization
- Tests organized by feature area (shapes, rendering, text, etc.)
- JUnit 5
@Tag annotations for flexible test selection
- For now, only two suites have been added, i.e., shapes and rendering. These will obviously be extended as tests grow
3. Intelligent Pixel Matching
- Handles minor anti-aliasing differences
- Detects and ignores line-shift artifacts
4. Gradle Tasks
./gradlew :visual-tests:test # Run all tests
./gradlew :visual-tests:visualTests # Run only visual tests
./gradlew :visual-tests:testBasicShapes # Run specific suite Usage Example
@Tag("shapes")
public class BasicShapesTest extends VisualTest {
@Test
@DisplayName("Red circle renders correctly")
public void testRedCircle() {
assertVisualMatch("basic-shapes/red-circle", new ProcessingSketch() {
@Override
public void setup(PApplet p) {
p.noStroke();
}
@Override
public void draw(PApplet p) {
p.background(255);
p.fill(255, 0, 0);
p.ellipse(p.width/2, p.height/2, 100, 100);
}
});
}
}Testing Strategy
First Run:
- Test captures screenshot from sketch
- No baseline exists → saves as baseline
- Test marked as "BASELINE CREATED"
Subsequent Runs:
- Test captures screenshot
- Compares against platform-specific baseline
- Reports PASS/FAIL with mismatch details
- Saves diff image if failed
cc @catilac @Stefterv @mingness @SableRaf
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Updated README to reflect migration to Jetpack Compose and strategy for replacing JEditTextArea with RSyntaxTextArea. Added insights on LSP-based editor research and the need for user feedback on Tweak Mode and autocompletion features.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added a few comments related to the Gradle setup. I will look into the behaviour and functionality at another stage
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Vaivaswat2244 and Annmool reacted with thumbs up emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
|
@Vaivaswat2244 It seems that the running of the tests is orchestrated via the gradle file, when instead, I think it should be orchestrated by JUnit. I'm not sure how the reporting system might work - maybe it needs gradle for that, but for the tests themselves, there should be a way of structuring them so they are simply the tests, any data needed by them, and they you invoke JUnit to run the tests. In our call tomorrow, let's look closer at this.
|
Stefterv, catilac, and Vaivaswat2244 reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Hey @Stefterv and @mingness, I've made the changes and also ensured the tests are orchestrated using JUnit.
Also, I was in the process of porting tests from p5 (currently done for the shapes) and have added those in these commits.
|
mingness reacted with heart emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Looks good to me! I only have a few formatting suggestions: There are changes in the root level gradle files that are actually just formatting - I suggest to revert these files. Also in other places where there are substantial formatting only changes (some lines in ./core/build.gradle.kts) I suggest to revert the formatting changes, so that all the changes are functional.
Sure, I'll revert the formatting changes.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Looking good! Thank you @Vaivaswat2244
One thing that I think would be useful would be some way to know if there differences between the different platforms, as a warning or something similar
I also noticed that Intellij gives a link to see the differences between the asserts, if you see a chance to make that the two images that would be awesome!
@catilac is it an idea to also move this to a feature branch for now?
Sure, I think the differences between platforms can be identified, we can compare the running sketch to all the baselines which we'll be storing (linux, mac and win), though this can increase the test time significantly, maybe we can do this periodically, using a cron job or something similar.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
great question @Stefterv! @Vaivaswat2244 if this project will require multiple phases or multiple PRs, it could be helpful to create a feature branch for it. If it's all self contained in this, then no problem.
And in general, small PRs around certain logical boundaries are nice for digesting changes when it's a large effort.
up to you @Stefterv !
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Hey @catilac, Addition to this pr, there should be atleast 3 more prs till the completion.
1- adding more tests
2- CI integration and report generation (p5 version of reporting system)
3- documentation
Changing to feature branch or staying in main, I am open to both the options.
|
Stefterv reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
@Vaivaswat2244 can you fix the last comment and merge the latest changes from main into your branch?
|
Vaivaswat2244 reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
Hey @Stefterv , I just fixed the last comment, but in order to merge the latest changes from main to my branch, I had to resolve the merge conflicts between the branches. Please take a look and let me know if I've made a git mess...
|
Stefterv reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Add configuration for dependencyUpdates task to manage non-stable versions.
More details surrounding `JEditTextArea`
I started cleaning up some of `Base`'s startup sequence for clarity of what is being started when. Nowhere near completion and I think a lot of this class will need to be refactored in the future.
Also removed some of the timing measurement comments
Added some comments to the Processing CLI class
Created CONTRIBUTORS.md and updated .all-contributorsrc to reference the new file instead of README.md. This will reduce the size of the README and improve loading times.
Added troubleshooting steps for build failures related to permissions.
Add configuration for dependencyUpdates task to manage non-stable versions.
|
Hey @catilac @Stefterv Can we squash merge this pull request to the feature branch visual-tests as we rebased this branch to main branch and it has the approvals.
If you'd like me change something, please let me know.
|
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
|
@Vaivaswat2244 thanks! Im in transit but will take a look and squash to the feature branch
|
Vaivaswat2244 reacted with thumbs up emoji
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Sorry, something went wrong.
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Assignees
No one assigned
Milestone
No milestone
Development
Successfully merging this pull request may close these issues.
Uh oh!
There was an error while loading. Please reload this page.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
|
