GeezerVison is showing github.com — Update JHDF5 library to 19.04.0 · Issue #181 · scijava/pom-scijava
Update JHDF5 library to 19.04.0 #181DescriptionRight now we are managing the CISD JHDF5 library at 14.12.6. We would like to upgrade to 19.04.0, to take advantage of new capabilities in HDF5. However, there are two obstacles:
-
The H5D class was retired in favor of the H5 class from upstream HDF Group's HDF5 Java bindings. Usages of H5D in consumer libraries will need to be updated to their H5-based equivalents.
-
The JHDF5 library has been deployed with two different groupIds, cisd and ch.systems.cisd, with the SciJava community later agreeing to unify around cisd and stop using the ch.systems.cisd groupId (Reconcile jhdf5 artifact usages #87). But some components are still using ch.systems.cisd; fortunately, updating them to 19.04.0 will naturally fix the groupId because we have not deployed 19.04.0 to maven.scijava.org under the old ch.systems.cisd groupId. But we still need to be careful to upgrade everything across the board at once, and/or exclude the coordinate with old groupId from dependency trees in cases where it leaks in.
Here is a table of which components on my radar use JHDF5 in its various incarnations:
ch.systems.cisd:jhdf5
cisd:jhdf5
org.scala-saddle:jhdf5_2.10
The checkboxes here can be checked when a particular component has migrated to JHDF5 19.04.0. And once all checkboxes are checked (except vcell-bioformats, which is probably out of scope here), this issue can be closed.
See also this and that and now also that chat thread.
CC @mkitti Reacted by Mark Kittisopikul, Christian Tischer, Tobias Pietzsch and Sébastien Besson Activitychanged the title [-]Update JHDF5 library to 19.10[/-] [+]Update JHDF5 library to 19.04.0[/+] on Nov 10, 2021tischi commented on Nov 10, 2021 @ctrueden thanks a lot for announcing this! I also have a couple of repos that are using hdf5:
Sorry, I did not get exactly what I need to do? Update/add a dependency in the pom and then change the code accordingly? mkitti commented on Nov 10, 2021 imagesc-bot commented on Nov 10, 2021 ctrueden commented on Nov 11, 2021 @tischi wrote:
Sorry, I did not get exactly what I need to do? Update/add a dependency in the pom and then change the code accordingly?
The first thing to do would be to make sure you are depending on:
<properties>
<cisd.jhdf5.version>19.04.0</cisd.jhdf5.version>
</properties>
<dependency>
<groupId>cisd</groupId>
<artifactId>jhdf5</artifactId>
<version>${cisd.jhdf5.version}</version>
</dependency>and not any other version of JHDF5. Once you've made that switch, you are likely to have compile errors around the no-longer-present H5D utility class. I defer to @mkitti's advice above on how best to fix these errors. Perhaps people can comment here with specific broken things and how they fix them, to help each other? Reacted by Christian Tischer tischi commented on Nov 11, 2021 Some experiences:
- after changing the jhdf5 dependency, I had to simply find/replace
ncsa. with nothing in all the import statements (e.g. import ncsa.hdf.hdf5lib.H5 -> import hdf.hdf5lib.H5)
- many/all H5 functions that used to return an
int now return a long, e.g. H5.H5Screate_simple(...) and H5.H5Acreate(...) and H5.H5Fopen(...) now return a long.
- also many/all of the constants are of type
long now, e.g., HDF5Constants.H5T_C_S1
Here are the changes that I made to imaris-writer. Reacted by Curtis Rueden, Mark Kittisopikul and Jan Eglinger tischi commented on Nov 11, 2021 Probably as expected I am not yet able to actually run the tests in imaris-writer, because it depends on bigdataviewer-core, which is not yet compatible with the new jdhf5.
Screenshot of the HDF5AccessHack class in bigdataviewer-core-10.2.0 from within IntelliJ:
ping @tpietzsch ctrueden commented on Nov 11, 2021 Reacted by Josh Moore and John Bogovic sbesson commented on Nov 11, 2021 Thanks for starting the discussion @ctrueden. Several of us are currently on leave but we'll discuss this next week and get back to you.
As a preliminary outcome, a naive bump of the cisd:jhdf5:14.12.6 library used in Bio-Formats to cisd:jhdf5:19.04.0 results in compilation issues of type
[ERROR] /opt/ome/bioformats/components/formats-bsd/src/loci/formats/in/BDVReader.java:[34,37] package ch.systemsx.cisd.base.mdarray does not exist
Looking briefly at the Javadoc and the source code links pasted by @mkitti above, it looks like this package is indeed no longer shipped as part of the library. Like above, this API might have been retired in favor of the equivalent API in the upstream Java bindings but I have not investigated further. mkitti commented on Nov 11, 2021 ctrueden commented on Nov 12, 2021
ch.systemsx.cisd.base.mdarray was never part of JHDF5 as far as I can tell, but may have been a dependency.
According to a classname search, it was embedded in cisd:jhdf5 through 14.12.6, and is no longer present in cisd:jhdf5:19.04.0, as @sbesson notes.
That same search shows that cisd:base:18.09.0 also contains that package. And the POM of cisd:jhdf5:19.04.0 depends on it.
What surprises me is that @sbesson's "naive bump" to cisd:jhdf5:19.04.0 would yield such compile errors, because cisd:base:18.09.0 should have been brought in also as a transitive dependency. ctrueden commented on Nov 12, 2021 @sbesson I tried updating the JHDF5 version in ome/bioformats as well, and everything builds fine. Here is the patch I used:
diff --git a/components/formats-bsd/pom.xml b/components/formats-bsd/pom.xml
index f8be993e9b..7d10ac168f 100644
--- a/components/formats-bsd/pom.xml
+++ b/components/formats-bsd/pom.xml
@@ -73,6 +73,11 @@
<artifactId>kryo</artifactId>
<version>${kryo.version}</version>
</dependency>
+ <dependency>
+ <groupId>commons-lang</groupId>
+ <artifactId>commons-lang</artifactId>
+ <version>2.6</version>
+ </dependency>
<dependency>
<groupId>org.perf4j</groupId>
<artifactId>perf4j</artifactId>
@@ -100,7 +105,7 @@
<dependency>
<groupId>cisd</groupId>
<artifactId>jhdf5</artifactId>
- <version>14.12.6</version>
+ <version>19.04.0</version>
</dependency>
<dependency>
<groupId>com.drewnoakes</groupId>All the formats-bsd tests also still pass (though they take 90 seconds to run).
The commons-lang dependency was being brought in transitively by the old jhdf5, but the new jhdf5 no longer depends on it... but it is an actual dependency of formats-bsd, so should be declared here regardless.
So early signs point to formats-bsd not struggling too much with this update! 👍 sbesson commented on Nov 12, 2021 Thanks @ctrueden, great to see that you managed to compile. The fact that cisd:base:18.09.0 would show up as a dependency was also my expectation.
Looking again, I think there is issue come from the cisd:jhdf5 POM that got downloaded locally. A few years ago, we added a repository in the OME Artifactory mirroring the subset of https://maven.scijava.org/content/groups/public/ for the cisd group and this is the primary repository used when building Bio-Formats.
Looking at the content of my local POM, which was downloaded from this cache rather than upstream Maven Scijava:
(base) sbesson@ls30630:~ $ cat ~/.m2/repository/cisd/jhdf5/19.04.0/jhdf5-19.04.0.pom
<?xml version="1.0" encoding="UTF-8"?>
<project xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<groupId>cisd</groupId>
<artifactId>jhdf5</artifactId>
<version>19.04.0</version>
<description>POM was created by Sonatype Nexus</description>
</project>
differs from the https://maven.scijava.org/content/groups/public/cisd/jhdf5/19.04.0/jhdf5-19.04.0.pom which declares the dependency.
Comparing the timestamps of https://maven.scijava.org/content/groups/public/cisd/jhdf5/19.04.0/ and adds to my confusion. While most of the components were indeed replicated the next day, the POM and others seemed to have been uploaded on Sep 2020.
I suspect we will clear and regenerate this cache on our side but I'll look into the logs to figure out what happens. ctrueden commented on Nov 12, 2021
I suspect we will clear and regenerate this cache on our side
👍
I'll look into the logs to figure out what happens.
If I had to hazard a guess: perhaps when I first uploaded 19.04.0 I did it directly without a custom POM, then realized it would need to have cisd:base a dependency and deleted and reuploaded it, and your cache was generated during the intervening time. I do not specifically recall that happening, but it seems like the sort of thing I would do.
If you are concerned about more general discrepancies beyond only these components, you could write a script to compare the md5sums across all of your cached components, just to be safe. tpietzsch commented on Feb 4, 2022 @ctrueden which version property should we use? You mentioned in a snippet above cisd.jhdf5.version, but pom-scijava manages
<dependency>
<groupId>cisd</groupId>
<artifactId>jhdf5</artifactId>
<version>${jhdf5.version}</version>
</dependency>
Should we just override the jhdf5.version property, or is there some reason to use the
<properties>
<cisd.jhdf5.version>19.04.0</cisd.jhdf5.version>
</properties>
<dependency>
<groupId>cisd</groupId>
<artifactId>jhdf5</artifactId>
<version>${cisd.jhdf5.version}</version>
</dependency>
construct? ctrueden commented on Feb 4, 2022
which version property should we use?
TL;DR: Either one should be fine! Overriding the short-name property will keep the two values aligned, though.
Background: To avoid future name clashes, pom-scijava has switched over to fully qualified version properties for all version management. However, it also defines a short style property as well. For example:
<ui-behaviour.version>2.0.5</ui-behaviour.version>
<org.scijava.ui-behaviour.version>${ui-behaviour.version}</org.scijava.ui-behaviour.version>And then the actual dependency block uses the fully qualified property.
JHDF5 is no different; this is how it's set up in pom-scijava now.
Caveat emptor: As far as overriding it in downstream POMs goes, it does not matter which one you use. But either way, it is important that all version property overrides in specific POMs be transient. Ideally, a component release should not contain any version property overrides, although I appreciate this this is not always possible. (If a release has a component override to a newer version than its associated BOM, then that newer version will NOT be brought in when depending on that component from another pom-scijava-inheriting project, unfortunately, because the dependency's properties are not inherited—properties only inherit from your parent POM.) mkitti commented on Feb 7, 2022
Probably as expected I am not yet able to actually run the tests in imaris-writer, because it depends on bigdataviewer-core, which is not yet compatible with the new jdhf5.
@tischi BDV-core has now been updated: bigdataviewer/bigdataviewer-core#131
Perhaps your tests on imaris-writer may work now? Reacted by Jan Eglinger and Christian Tischer tischi commented on Feb 15, 2022
Perhaps your tests on imaris-writer may work now?
Thanks, yes, using <bigdataviewer-core.version>10.3.1</bigdataviewer-core.version> works. I guess I will remove this however again from the pom and wait for the corresponding scijava parent pom release? Reacted by Mark Kittisopikul mkitti commented on Feb 15, 2022 @tischi , if your code works either way, then I think that makes sense. If it only works with the newer JHDF5 then, perhaps it is good to keep it for now until the parent pom is updated?
@ctrueden , any thoughts on what to do in the transition? ctrueden commented on Feb 20, 2022 As of 2f7d665, the entire SciJava component collection is free of ch.systems.cisd:jhdf5 dependencies, and the version of cisd:jhdf5 is now 19.04.0. This gets us much of the way toward the goal of this issue. In particular, it means a new pom-scijava can be released that manages jhdf5 at 19.04.0. Downstream non-SciJava-managed components can then update to the new parent, and fix any JHDF5-related issues as described above, in order to complete their upgrade to 19.04.0. Reacted by Mark Kittisopikul, John Bogovic and Tobias Pietzsch imagesc-bot commented on Jun 28, 2022 imagesc-bot commented on Jun 29, 2022 imagesc-bot commented on Jul 25, 2022 ctrueden commented on Aug 11, 2022 ctrueden commented on Aug 27, 2022 Today I released pom-scijava 32.0.0, with a harmonious BOM across all JHDF5-related components. We are now on jhdf5 19.04.1.
Any components above which are not managed by pom-scijava are on their own to follow suit. If you want to be part of the update process in future, you can file a PR adding version management of your components to pom-scijava, and your components will become part of the "mega-melt" BOM harmony testing.
MetadataMetadataDevelopmentNo branches or pull requestsIssue actions
|
