Skip to content

CLI reference

ob

OmniBenchmark Command Line Interface (CLI).

Usage:

ob [OPTIONS] COMMAND [ARGS]...

Options:

  --debug / --no-debug  Enable debug mode
  --version             Show the version and exit.
  --help                Show this message and exit.

Subcommands

  • info: List benchmarks and/or information about them.
  • run: Run benchmarks or benchmark modules.
  • software: Manage and install benchmark-specific software.
  • storage: Manage remote storage.

info

List benchmarks and/or information about them.

Usage:

ob info [OPTIONS] COMMAND [ARGS]...

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

Subcommands

  • computational: Export computational graph to dot format.
  • diff: Show differences between 2 benchmark versions.
  • list-versions: List all available benchmarks versions at a specific endpoint.
  • topology: Export benchmark topology to mermaid diagram format.

computational

Export computational graph to dot format.

Usage:

ob info computational [OPTIONS]

Options:

  -b, --benchmark PATH  Path to benchmark yaml file or benchmark id.
                        [required]
  --help                Show this message and exit.

diff

Show differences between 2 benchmark versions.

Usage:

ob info diff [OPTIONS]

Options:

  -b, --benchmark PATH  Path to benchmark yaml file or benchmark id.
                        [required]
  -v1, --version1 TEXT  Reference version.  [required]
  -v2, --version2 TEXT  Version to compare with.  [required]
  --help                Show this message and exit.

list-versions

List all available benchmarks versions at a specific endpoint.

Usage:

ob info list-versions [OPTIONS]

Options:

  -b, --benchmark PATH  Path to benchmark yaml file or benchmark id.
                        [required]
  --help                Show this message and exit.

topology

Export benchmark topology to mermaid diagram format.

Usage:

ob info topology [OPTIONS]

Options:

  -b, --benchmark PATH  Path to benchmark yaml file or benchmark id.
                        [required]
  --help                Show this message and exit.

run

Run benchmarks or benchmark modules.

Usage:

ob run [OPTIONS] COMMAND [ARGS]...

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

Subcommands

  • benchmark: Run a benchmark as specified in the yaml.
  • module:
  • validate: Validate a benchmark yaml.

benchmark

Run a benchmark as specified in the yaml.

Usage:

ob run benchmark [OPTIONS]

Options:

  --debug / --no-debug            Enable debug mode
  -b, --benchmark PATH            Path to benchmark yaml file or benchmark id.
                                  [required]
  -c, --cores INTEGER             Use at most N CPU cores in parallel. Default
                                  is 1.
  -u, --update                    Force re-run execution for all modules and
                                  stages.
  -d, --dry                       Dry run.
  -y, --yes                       Automatically confirm all prompts.
  -k, --continue-on-error         Go on with independent jobs if a job fails
                                  (--keep-going in snakemake).
  -l, --local                     Execute and store results locally. Default
                                  False.
  --keep-module-logs / --no-keep-module-logs
                                  Keep module-specific log files after
                                  execution.
  --task-timeout TEXT             Timeout for each separate task execution
                                  (local only). Do note that total runtime is
                                  not additive.
  --help                          Show this message and exit.

module

Run a specific module that is part of the benchmark.

Usage:

ob run module [OPTIONS]

Options:

  -b, --benchmark PATH            Path to benchmark yaml file or benchmark id.
                                  [required]
  -m, --module TEXT               Module id to execute  [required]
  -i, --input_dir PATH            Path to the folder with the appropriate
                                  input files.
  -d, --dry                       Dry run.
  -u, --update                    Force re-run execution for all modules and
                                  stages.
  -k, --continue-on-error         Go on with independent jobs if a job fails
                                  (--keep-going in snakemake).
  --keep-module-logs / --no-keep-module-logs
                                  Keep module-specific log files after
                                  execution.
  --help                          Show this message and exit.

validate

Validate a benchmark yaml.

Usage:

ob run validate [OPTIONS]

Options:

  -b, --benchmark PATH  Path to benchmark yaml file or benchmark id.
  --help                Show this message and exit.

software

Manage and install benchmark-specific software.

Usage:

ob software [OPTIONS] COMMAND [ARGS]...

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

Subcommands

  • check: Check whether the component {what} is available.
  • conda: Manage and install software using conda.
  • module: Manage and install software using Easybuild
  • singularity: Manage and install software using Singularity.

check

Check whether the component {what} is available.

Usage:

ob software check [OPTIONS]

Options:

  -w, --what TEXT  Binary/functionality to check:

                   --what singularity : singularity

                   --what module      : module tool, typically lmod

                   --what easybuild   : easybuild

                   --what conda       : conda
  --help           Show this message and exit.

conda

Manage and install software using conda.

Usage:

ob software conda [OPTIONS] COMMAND [ARGS]...

Options:

  --help  Show this message and exit.

Subcommands

  • pin: Pin all conda env-related dependencies versions using snakedeploy.
  • prepare: Pin all conda envs needed for a given benchmark YAML.
pin

Pin all conda env-related dependencies versions using snakedeploy.

Usage:

ob software conda pin [OPTIONS]

Options:

  -e, --env PATH  Conda env YAML.  [required]
  --help          Show this message and exit.
prepare

Pin all conda envs needed for a given benchmark YAML.

Usage:

ob software conda prepare [OPTIONS]

Options:

  -b, --benchmark PATH  Benchmark YAML.  [required]
  --help                Show this message and exit.

module

Manage and install software using Easybuild

Usage:

ob software module [OPTIONS] COMMAND [ARGS]...

Options:

  --help  Show this message and exit.

Subcommands

  • build: Build a given easyconfig (and generates the relevant envmodules).
  • prepare: Build all envmodules needed for a given benchmark YAML.
build

Build a given easyconfig (and generates the relevant envmodules).

Usage:

ob software module build [OPTIONS]

Options:

  -e, --easyconfig TEXT  Easyconfig.  [required]
  -p, --threads INTEGER  Number of threads.
  --help                 Show this message and exit.
prepare

Build all envmodules needed for a given benchmark YAML.

Usage:

ob software module prepare [OPTIONS]

Options:

  -b, --benchmark PATH   Benchmark YAML.  [required]
  -p, --threads INTEGER  Number of threads.
  --help                 Show this message and exit.

singularity

Manage and install software using Singularity.

Usage:

ob software singularity [OPTIONS] COMMAND [ARGS]...

Options:

  --help  Show this message and exit.

Subcommands

  • build: Build a singularity (fakeroot) image for a given easyconfig.
  • prepare: Build all singularity (fakeroot) images needed for a benchmark.
  • push: Pushes a singularity SIF file to an ORAS-compatible registry.
build

Build a singularity (fakeroot) image for a given easyconfig.

Usage:

ob software singularity build [OPTIONS]

Options:

  -e, --easyconfig TEXT  Easyconfig.  [required]
  --help                 Show this message and exit.
prepare

Build all singularity (fakeroot) images needed for a benchmark.

Usage:

ob software singularity prepare [OPTIONS]

Options:

  -b, --benchmark PATH  Benchmark YAML.  [required]
  --help                Show this message and exit.
push

Pushes a singularity SIF file to an ORAS-compatible registry.

Usage:

ob software singularity push [OPTIONS]

Options:

  -u, --docker_username TEXT  Docker username.  [required]
  -p, --docker_password TEXT  Docker password.  [required]
  -s, --sif PATH              Path to the Singularity SIF file.  [required]
  -o, --oras TEXT             Registry's ORAS static URL, for instance oras://
                              registry.mygitlab.ch/myuser/myproject:mytag.
                              [required]
  --help                      Show this message and exit.

storage

Manage remote storage.

Usage:

ob storage [OPTIONS] COMMAND [ARGS]...

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

Subcommands

  • archive: Archive a benchmark
  • checksum: Generate md5sums of all benchmark outputs
  • create-policy: Create a new policy for a benchmark.
  • create-version: Create a new benchmark version.
  • download: Download all or specific files for a benchmark.
  • list: List all or specific files for a benchmark.

archive

Archive a benchmark

Usage:

ob storage archive [OPTIONS]

Options:

  -b, --benchmark PATH            Path to benchmark yaml file or benchmark id.
                                  [required]
  -c, --code                      Archive benchmarking code (repos).
  -s, --software                  Archive software environments.
  -r, --results                   Archive results files.
  --compression [none|deflated|bzip2|lzma]
                                  Compression method.  [default: none]
  --compresslevel INTEGER         Compression level.
  -n, --dry-run                   Do not create the archive, just show what
                                  would be done.
  -l, --local                     Execute and store results locally. Default
                                  False.
  --help                          Show this message and exit.

checksum

Generate md5sums of all benchmark outputs

Usage:

ob storage checksum [OPTIONS]

Options:

  --debug / --no-debug  Enable debug mode
  -b, --benchmark PATH  Path to benchmark yaml file or benchmark id.
                        [required]
  --help                Show this message and exit.

create-policy

Create a new policy for a benchmark.

Usage:

ob storage create-policy [OPTIONS]

Options:

  -b, --benchmark PATH  Path to benchmark yaml file or benchmark id.
                        [required]
  --help                Show this message and exit.

create-version

Create a new benchmark version.

Usage:

ob storage create-version [OPTIONS]

Options:

  --debug / --no-debug  Enable debug mode
  -b, --benchmark PATH  Path to benchmark yaml file or benchmark id.
                        [required]
  --help                Show this message and exit.

download

Download all or specific files for a benchmark.

Usage:

ob storage download [OPTIONS]

Options:

  --debug / --no-debug  Enable debug mode
  -b, --benchmark PATH  Path to benchmark yaml file or benchmark id.
                        [required]
  -t, --type TEXT       File types. Options: all, code, inputs, outputs, logs,
                        performance.
  -s, --stage TEXT      Stage to download files from.
  -m, --module TEXT     Module to download files from.
  -i, --id TEXT         File id to download.
  -o, --overwrite       Overwrite existing files.
  --help                Show this message and exit.

list

List all or specific files for a benchmark.

Usage:

ob storage list [OPTIONS]

Options:

  --debug / --no-debug  Enable debug mode
  -b, --benchmark PATH  Path to benchmark yaml file or benchmark id.
                        [required]
  -t, --type TEXT       File types. Options: all, code, inputs, outputs, logs,
                        performance.
  -s, --stage TEXT      Stage to list files for.
  -i, --id TEXT         File id/type to list.
  --help                Show this message and exit.