GeezerVison

Coverage increased (+1.06%) to 90.343% when pulling 6f1eb6c on fsteeg:1.1 into bbae6c3 on jsonld-java:master.

Added setup for skipping tests in 79fdafa (before this, coverage went down, now it's up).

• Skip only failing tests to make sure we run 1.1 code paths in CI
• Use resolvable URLs in skip files, they can act like TODO lists
• Delete *-skip files to re-generate them during next test run

Now mvn test runs all tests that are not listed in the files.

I think this is could be a good starting point to implement json-ld 1.1 step-by-step. What do you think @ansell ?

To clarify what this does compared to the current test setup in master:

• Run all original 1.0/master tests (nothing skipped here)
• Additionally run the lson-ld-api tests in test/resources/json-ld-api-tests
• Additionally run the lson-ld-framing tests in test/resources/json-ld-framing-tests

For all of these:

• If there is no reports/*-skip file (e.g. reports/json-ld-api-tests-skip), run all tests, writing failing tests URIs (e.g. https://w3c.github.io/json-ld-api/tests/compact-manifest.html#t0037) to a file
• If there is such a file, skip all tests listed in that file
• State of skipped tests in this branch is 0 (of 472) for 1.0, 366 (of 805) for 1.1-api, and 59 (of 86) for 1.1-framing

The propsed workflow towards 1.1 would be:

• Merge this into master, keep releasing from master, because all current master tests still work
• When implementing new 1.1 features, pick an entry from a skip file (e.g. https://w3c.github.io/json-ld-api/tests/compact-manifest.html#t0037 from 1.1-api), implement suport for that test case
• Remove the reports/*-skip file, rerun tests to regenerate a (hopefully) smaller skip file, with less tests to skip, and commit changed skip file together with the implementation

A quick status update: I've pushed some tweaks to the tests and report setup, updated the tests and looked into integrating with https://w3c.github.io/json-ld-api/reports/ (see 6f1eb6c). With current spec tests and implementation (master + 1.1) we currently get this test suite compliance summary:

I discussed our next steps with @dr0i and we think we should split up this branch/PR into two branches: one for the changes in the test setup only (1.1-tests) and one, branched off from the first, with implementation changes only (1.1). That way, we hope we'll avoid a giant, un-reviewable PR while working towards 1.1 compliance.

Any report would be a good start; you can always submit an update and we can regenerate the report.

Time permitting, I'll have a look at your code and offer some suggestions for improving conformance.

Replaced by / moved to #283 & #284, closing.