This repository documents a reproducibility analysis performed while preparinga manuscript to describe the Bioconductor goSorensen package.

The analysis was motivated by numerical discrepancies observed when examples from the manuscript were rerun under a different Bioconductor release. To investigate their origin, the same R workflow was executed under three Bioconductor releases:

• Bioconductor 3.21
• Bioconductor 3.22
• Bioconductor 3.23

using the official Bioconductor Docker images.

The comparison shows that differences across releases already appear at the enrichment stage, before the calculation of Sørensen-Dice dissimilarities. In particular, the size of the gene universe, the number of GO terms considered, and the resulting enrichment tables may change across releases. These differences subsequently propagate to the Sørensen-Dice dissimilarities and equivalence tests.

The repository contains:

.
├── README.md
├── compare_bioc_versions.R
├── Dockerfile.3.21
├── Dockerfile.3.22
├── Dockerfile.3.23
├── LICENSE
├── .gitignore
└── results
    ├── results_3_21
    │   ├── cont_all_BP4.rds
    │   ├── cont_atlas_sanger_BP4.rds
    │   ├── dissMatrx_BP4.rds
    │   ├── enrichedInBP4.rds
    │   ├── eqTest_atlas_sanger_BP4.rds
    │   └── results-3.21.txt
    ├── results_3_22
    │   └── ...
    └── results_3_23
        └── ...

Each results_3_xx directory contains a text summary of the complete execution and the main intermediate and final R objects produced by the workflow. The .rds files make it possible to inspect these objects without rerunning the complete analysis.

The Docker images themselves are not stored in this repository. They are downloaded from the official Bioconductor Docker repository and extended locally with the packages required for the analysis.

To reproduce the analysis, Docker is required. A local installation of R or Bioconductor is not necessary.

The analysis was originally performed using Docker Desktop under Windows with the WSL2 backend, but the same procedure can also be used under Linux or macOS.

Clone the repository and move to its root directory:

git clone https://github.com/ASPresearch/goSorensen-Bioc-Reproducibility.git
cd goSorensen-Bioc-Reproducibility

All commands below use paths relative to the repository directory. No user-specific or absolute host paths are required.

docker pull bioconductor/bioconductor_docker:RELEASE_3_21
docker pull bioconductor/bioconductor_docker:RELEASE_3_22
docker pull bioconductor/bioconductor_docker:RELEASE_3_23

These base images do not necessarily contain all packages required by the analysis. The Dockerfiles included in this repository create small derived images containing the corresponding versions of goSorensen, GO.db, and org.Hs.eg.db.

For example, Dockerfile.3.21 contains:

FROM bioconductor/bioconductor_docker:RELEASE_3_21

RUN R -e "BiocManager::install(c('goSorensen','org.Hs.eg.db','GO.db'), ask=FALSE, update=FALSE)"

Equivalent Dockerfiles are provided for Bioconductor 3.22 and 3.23.

From the repository root:

docker build -f Dockerfile.3.21 -t gosorensen-bioc:3.21 .
docker build -f Dockerfile.3.22 -t gosorensen-bioc:3.22 .
docker build -f Dockerfile.3.23 -t gosorensen-bioc:3.23 .

The environments used in this comparison contained:

The same script, compare_bioc_versions.R, is executed in each environment. It records the R/Bioconductor/package versions, gene-universe size, number of GO Biological Process terms evaluated at level 4, enrichment results, the atlas versus sanger contingency table and equivalence test, the pairwise equivalence-threshold dissimilarity matrix, and a complete sessionInfo().

It also saves the principal intermediate and final R objects as .rds files.

docker run --rm \
  -v "${PWD}:/work" \
  gosorensen-bioc:3.21 \
  Rscript --vanilla /work/compare_bioc_versions.R /work/results/results_3_21

docker run --rm \
  -v "${PWD}:/work" \
  gosorensen-bioc:3.22 \
  Rscript --vanilla /work/compare_bioc_versions.R /work/results/results_3_22

docker run --rm \
  -v "${PWD}:/work" \
  gosorensen-bioc:3.23 \
  Rscript --vanilla /work/compare_bioc_versions.R /work/results/results_3_23

docker run --rm `
  -v "${PWD}:/work" `
  gosorensen-bioc:3.21 `
  Rscript --vanilla /work/compare_bioc_versions.R /work/results/results_3_21

docker run --rm `
  -v "${PWD}:/work" `
  gosorensen-bioc:3.22 `
  Rscript --vanilla /work/compare_bioc_versions.R /work/results/results_3_22

docker run --rm `
  -v "${PWD}:/work" `
  gosorensen-bioc:3.23 `
  Rscript --vanilla /work/compare_bioc_versions.R /work/results/results_3_23

These commands mount the repository root as /work inside the container and write the outputs directly into the corresponding subdirectory under results/.

The three executions produced different results already at the enrichment stage.

These results show that the differences between releases are not introduced only during the final Sørensen-Dice calculation. The inputs to that calculation already differ across Bioconductor releases.

                 Enriched in sanger
Enriched in atlas TRUE FALSE
            TRUE   201   187
            FALSE   30  2948

                 Enriched in sanger
Enriched in atlas TRUE FALSE
            TRUE   204   186
            FALSE   27  2964

                 Enriched in sanger
Enriched in atlas TRUE FALSE
            TRUE   131   161
            FALSE   24  3007

Changes in these enrichment tables propagate directly to the corresponding Sørensen-Dice dissimilarities and equivalence tests.

For example, the atlas versus sanger equivalence-threshold dissimilarity was:

Bioconductor 3.21: 0.4508853
Bioconductor 3.22: 0.4606465
Bioconductor 3.23: 0.4962106

The complete matrices are stored in the corresponding dissMatrx_BP4.rds files and printed in the text output of each run.

This comparison illustrates an important reproducibility issue in enrichment-based workflows.

goSorensen operates on enrichment results derived from Gene Ontology and organism-specific annotations. These resources, together with the surrounding Bioconductor software environment, evolve across releases.

Consequently, rerunning the same workflow with a different Bioconductor release may lead to a different gene universe, a different set of GO terms, different enrichment results and contingency tables, and therefore different downstream Sørensen-Dice dissimilarities and equivalence-test results.

The observed differences are already visible before the Sørensen-Dice calculations and are consistent with changes in the annotation resources and software environment across Bioconductor releases.

This behaviour does not necessarily indicate a lack of reproducibility of the statistical method itself. It shows, however, that numerical reproducibility of an enrichment-based workflow depends on the computational and annotation environment in which it is run.

For this reason, the Bioconductor release and the relevant package versions should be reported whenever numerical reproducibility is important.

The numerical results reported by the reviewer were reproduced using Bioconductor 3.22, which was the current Bioconductor release at the time of the review.

For the revised manuscript, all examples were recalculated consistently using Bioconductor 3.23, the current release at the time the revised manuscript was prepared.

The revised manuscript therefore reports the computational environment used for the examples and notes that rerunning them under another Bioconductor release may yield different numerical results.

This repository provides the code, environment specifications, and outputs supporting that reproducibility assessment.

Running compare_bioc_versions.R in each environment reproduces the outputs stored in:

results/results_3_21/
results/results_3_22/
results/results_3_23/

For full package and dependency information, see the sessionInfo() output included in each results-3.xx.txt file.

The analysis uses R, Bioconductor, goSorensen, GO.db, org.Hs.eg.db, and Docker.

If this reproducibility analysis is used in connection with the goSorensen manuscript, please cite the corresponding article and the goSorensen package.