Pipeline packaging

Some systems don't allow access to internet making it impossible to use a pipeline that is dependent on resources hosted on the web, like docker hub and github. This is solved by packing the pipeline and all its dependencies.

Package pipeline

On local computer or unrestricted cluster

The following instructions on packaging the pipeline for an offline environment should be performed on your local computer or a cluster with internet access.

Use a build script that will package all files needed to run the pipeline:

  • The virtual environment
  • Github repositories (Pipeline, hydra-genetics modules, Snakemake-wrappers)
  • Singularities / Apptainers

Preparations

Requires:

# On Marvin conda can be loaded by
module load miniconda3

Package and download all files with the build script

Set pipeline version and pipeline name which must match an existing version and name found on github List all reference config files that should be used as arguments to the build script

An example build script can be found in the hydra-genetics github repo

# Example: TAG_OR_BRANCH="v0.18.3" PIPELINE_NAME="Twist_Solid" PIPELINE_GITHUB_REPO="https://github.com/genomic-medicine-sweden/Twist_Solid.git" bash build/build_conda.sh config/references/design_files.hg19.yaml config/references/novaseq.hg19.pon.yaml config/references/references.hg19.yaml
TAG_OR_BRANCH="vX.Y.X" PIPELINE_NAME="Your_pipeline_name" PIPELINE_GITHUB_REPO="pipeline_github_repo.git" bash build/build_conda.sh config/references/<file1>.yaml config/references/<file2>.yaml

Resulting files and folders

The following files and folders have been created and need to be moved to your server:

  1. file: design_and_ref_files.tar.gz
  2. file: {PIPELINE_NAME}_{TAG_OR_BRANCH}.tar.gz
  3. folder: apptainer_cache

Copy pipeline package files

Copy the pipeline packaged files to the compute cluster.

# Commands to be run from the cluster where the pipeline was packaged
# <USER>@<cluster>:<basepath> are relative to the offline cluster where the package pipeline should be transferred
# For example: run the `rsync` command from marvin with your credentials on miarka
rsync -Pav design_and_ref_files.tar.gz <USER>@<cluster>:<basepath>/${PIPELINE_SHORT_NAME}/
rsync -Pav ${PIPELINE_NAME}_${TAG_OR_BRANCH}.tar.gz apptainer_cache <USER>@<cluster>:<basepath>/${PIPELINE_SHORT_NAME}/${TAG_OR_BRANCH}/