CLI reference

ob

OmniBenchmark Command Line Interface (CLI).

Usage:

ob [OPTIONS] COMMAND [ARGS]...

Options:

  --debug / --no-debug  Enable debug mode
  --version             Show the version and exit.
  -h, --help            Show this message and exit.

Subcommands

• archive: Archive a benchmark and its artifacts.
• cite: Extract citation metadata from CITATION.cff files in benchmark modules.
• create: Create new benchmarks or modules from templates.
• dashboard: Generate a dashboard from benchmark results.
• describe: Describe benchmarks and/or information about them.
• remote: Manage remote storage.
• run: Run a benchmark.
• validate: Validate benchmarks and modules.

archive

Archive a benchmark and its artifacts.

BENCHMARK: Path to benchmark YAML file.

Usage:

ob archive [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug            Enable debug mode
  -c, --code                      Archive benchmarking code (repos).
  -s, --software                  Archive software environments.
  -r, --results                   Archive results files.
  --compression [none|deflated|bzip2|lzma|gzip]
                                  Compression method. 'gzip' creates a tar.gz
                                  archive, others create ZIP archives.
                                  [default: gzip]
  --compresslevel INTEGER         Compression level (1-9, higher is better
                                  compression).  [default: 9]
  -n, --dry-run                   Do not create the archive, just show what
                                  would be done.
  --use-remote-storage            Download results from remote storage for
                                  archiving. Default False.
  --out-dir TEXT                  Local directory name to archive from (local-
                                  only mode). Default: `out`
  -o, --output-file PATH          Custom output file path for the archive.
                                  Extension must match compression type:
                                  none/deflated (.zip), bzip2 (.bz2), lzma
                                  (.xz), gzip (.tar.gz).
  --help                          Show this message and exit.

cite

Extract citation metadata from CITATION.cff files in benchmark modules.

Automatically clones repositories to temporary directories if not found locally. By default, runs in warn mode and converts errors to warnings. Use --strict to fail on errors instead.

Usage:

ob cite [OPTIONS]

Options:

  --debug / --no-debug            Enable debug mode
  -b, --benchmark PATH            Path to benchmark yaml file or benchmark id.
                                  [required]
  -f, --format [json|yaml|bibtex]
                                  Output format for citation information.
  --strict                        Fail on errors instead of converting them to
                                  warnings (default: warn mode).
  --out PATH                      Output file to write results to.
  --help                          Show this message and exit.

create

Create new benchmarks or modules from templates.

Usage:

ob create [OPTIONS] COMMAND [ARGS]...

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

Subcommands

• benchmark: Create a new benchmark from a template.
• module: Create a new module from a template.

benchmark

Create a new benchmark from a template.

This command scaffolds a new benchmark project structure using copier templates. It creates all necessary configuration files, directory structure, and example modules to get you started with your benchmark.

Usage:

ob create benchmark [OPTIONS] [PATH]

Options:

  --debug / --no-debug            Enable debug mode
  --no-input                      Do not prompt for parameters and only use
                                  defaults
  --non-interactive               Non-interactive mode (requires all mandatory
                                  parameters as flags)
  --name TEXT                     Name of the benchmark
  --author-name TEXT              Author name
  --author-email TEXT             Author email
  --license [MIT|Apache-2.0|GPL-3.0|BSD-3-Clause|CC0-1.0]
                                  License for the benchmark
  --description TEXT              Description of the benchmark
  --help                          Show this message and exit.

module

Create a new module from a template.

This command scaffolds a new OmniBenchmark module project structure using copier templates. It creates the necessary configuration files (CITATION.cff, omnibenchmark.yaml), a sample entrypoint script, and documentation to get you started with your module development.

When using --benchmark and --for-stage, the generated entrypoint will include CLI argument parsing for the specific inputs required by that stage.

Usage:

ob create module [OPTIONS] PATH

Options:

  --debug / --no-debug            Enable debug mode
  -b, --benchmark TEXT            Path or URL to benchmark YAML file (for
                                  stage-specific input parsing)
  --for-stage TEXT                Stage ID to generate module for (requires
                                  --benchmark)
  --no-input                      Do not prompt for parameters and only use
                                  defaults
  --non-interactive               Non-interactive mode (requires all mandatory
                                  parameters as flags)
  --name TEXT                     Name of the module
  --author-name TEXT              Author name
  --author-email TEXT             Author email
  --license [MIT|Apache-2.0|GPL-3.0-or-later|BSD-3-Clause|CC0-1.0]
                                  License for the module
  --description TEXT              Description of the module
  --entrypoint [run.R|run.py|run.sh]
                                  Main entrypoint script for the module
  --help                          Show this message and exit.

dashboard

Generate a dashboard from benchmark results.

This command generates a dashboard from the performance data collected during benchmark execution.

Usage:

ob dashboard [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug  Enable debug mode
  -f, --format [bettr]  Dashboard format to generate (default: bettr).
  -o, --out-dir PATH    Output directory containing benchmark results
                        (default: out).
  --help                Show this message and exit.

describe

Describe benchmarks and/or information about them.

Usage:

ob describe [OPTIONS] COMMAND [ARGS]...

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

Subcommands

• snakemake: Export a snakemake computational graph to dot format.
• status: Show the status of a benchmark.
• topology: Export benchmark topology to MERMAID diagram format.

snakemake

Export a snakemake computational graph to dot format.

BENCHMARK: Path to benchmark YAML file.

Usage:

ob describe snakemake [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

status

Show the status of a benchmark.

BENCHMARK: Path to benchmark YAML file.

Usage:

ob describe status [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug  Enable debug mode
  --out-dir TEXT        Output folder name (local only).
  --missing_files       Show missing files in the status report.
  --reason              Show reason for missing files in the status report.
  --logs                Show logs for missing files in the status report.
  --json                Return the status report as JSON.
  --html                Return the status report as HTML.
  --html-file TEXT      Output HTML file name.
  --force               If HTML file exists, overwrite it.
  --help                Show this message and exit.

topology

Export benchmark topology to MERMAID diagram format.

BENCHMARK: Path to benchmark YAML file.

Usage:

ob describe topology [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug            Enable debug mode
  --show-params / --no-show-params
                                  Whether to write parameters to the mermaid
                                  output.
  --compact-params / --no-compact-params
                                  Whether to write parameters compactly (only
                                  used if --show-params).
  --help                          Show this message and exit.

remote

Manage remote storage.

Usage:

ob remote [OPTIONS] COMMAND [ARGS]...

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

Subcommands

• files: Manage files in remote storage.
• policy: Manage storage policies. DEPRECATED: use your cloud provider's IAM tooling directly.
• version: Manage benchmark versions.

files

Manage files in remote storage.

Usage:

ob remote files [OPTIONS] COMMAND [ARGS]...

Options:

  --help  Show this message and exit.

Subcommands

• checksum: Generate md5sums of all benchmark outputs.
• download: Download all or specific files for a benchmark.
• list: List all or specific files for a benchmark.

checksum

Generate md5sums of all benchmark outputs.

BENCHMARK: Path to benchmark YAML file.

Usage:

ob remote files checksum [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

download

Download all or specific files for a benchmark.

BENCHMARK: Path to benchmark YAML file.

Usage:

ob remote files download [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug  Enable debug mode
  -t, --type TEXT       File types. Options: all, code, inputs, outputs, logs,
                        performance.
  -s, --stage TEXT      Stage to download files from.
  -m, --module TEXT     Module to download files from.
  -i, --id TEXT         File id to download.
  -o, --overwrite       Overwrite existing files.
  --help                Show this message and exit.

list

List all or specific files for a benchmark.

BENCHMARK: Path to benchmark YAML file.

Usage:

ob remote files list [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug  Enable debug mode
  -t, --type TEXT       File types. Options: all, code, inputs, outputs, logs,
                        performance.
  -s, --stage TEXT      Stage to list files for.
  -m, --module TEXT     Module to list files for.
  -i, --id TEXT         File id/type to list.
  --help                Show this message and exit.

policy

Manage storage policies. DEPRECATED: use your cloud provider's IAM tooling directly.

Usage:

ob remote policy [OPTIONS] COMMAND [ARGS]...

Options:

  --help  Show this message and exit.

Subcommands

• create: Generate an S3/MinIO IAM policy for a benchmark bucket.

create

Generate an S3/MinIO IAM policy for a benchmark bucket.

This command generates a least-privilege AWS IAM policy JSON that can be used to create access keys in MinIO or AWS. The policy allows full S3 operations on the bucket but denies deletion and governance bypass.

You can either: - Provide a benchmark YAML file (-b/--benchmark) to read the bucket name from storage.bucket_name - Provide the bucket name directly (--bucket)

Usage:

ob remote policy create [OPTIONS]

Options:

  --debug / --no-debug  Enable debug mode
  -b, --benchmark PATH  Path to benchmark YAML file.
  --bucket TEXT         S3 bucket name. If not provided, reads from benchmark
                        YAML.
  --help                Show this message and exit.

version

Manage benchmark versions.

Usage:

ob remote version [OPTIONS] COMMAND [ARGS]...

Options:

  --help  Show this message and exit.

Subcommands

• create: Create a new benchmark version.
• diff: Show differences between 2 benchmark versions.
• list: List all available benchmark versions.

create

Create a new benchmark version.

BENCHMARK: Path to benchmark YAML file.

Usage:

ob remote version create [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

diff

Show differences between 2 benchmark versions.

BENCHMARK: Path to benchmark YAML file.

Usage:

ob remote version diff [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug  Enable debug mode
  -v1, --version1 TEXT  Reference version.  [required]
  -v2, --version2 TEXT  Version to compare with.  [required]
  --help                Show this message and exit.

list

List all available benchmark versions.

BENCHMARK: Path to benchmark YAML file.

Usage:

ob remote version list [OPTIONS] BENCHMARK

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

run

Run a benchmark.

BENCHMARK: Path to benchmark YAML file.

This command: 1. Fetches and caches all module repositories 2. Resolves modules and generates an explicit Snakefile 3. Runs snakemake on the generated Snakefile

Any arguments after -- are passed directly to snakemake.

Examples:

ob run benchmark.yaml # Run full benchmark ob run benchmark.yaml --cores 8 # Run with 8 cores ob run benchmark.yaml --dry # Generate Snakefile only ob run benchmark.yaml --dirty # Allow local paths with uncommitted changes ob run benchmark.yaml --unpinned # Allow branch refs on remote repos ob run benchmark.yaml -m M1 # Dev mode: run only module M1 ob run benchmark.yaml -- --rerun-triggers mtime # Pass flags to snakemake ob run benchmark.yaml -- --forceall # Force re-run all rules

Usage:

ob run [OPTIONS] BENCHMARK [SNAKEMAKE_ARGS]...

Options:

  --debug / --no-debug     Enable debug mode
  -c, --cores INTEGER      Use at most N CPU cores in parallel. Default is 1.
  -d, --dry                Dry run (only generate Snakefile, don't execute).
  -k, --continue-on-error  Go on with independent jobs if a job fails (--keep-
                           going in snakemake).
  --out-dir TEXT           Output folder name. Default: `out`
  --dirty                  Allow local path module references with uncommitted
                           changes. Use for development only.
  --unpinned               Allow unpinned branch references on remote repos
                           (resolved to HEAD at run time). Use for development
                           only.
  --use-remote-storage     Execute and store results remotely using S3 storage
                           configured in the benchmark YAML.
  -m, --module TEXT        Run only the sub-graph needed for a single module
                           (development mode). Prunes all stages after the
                           target module's stage and keeps only the first
                           upstream input × parameter expansion for each
                           module.
  --help                   Show this message and exit.

validate

Validate benchmarks and modules.

Usage:

ob validate [OPTIONS] COMMAND [ARGS]...

Options:

  --debug / --no-debug  Enable debug mode
  --help                Show this message and exit.

Subcommands

• module: Validate module (metadata, try to run).
• plan: Validate benchmark YAML plan structure.

module

Validate module (metadata, try to run).

Validates a module repository at PATH. Defaults to current directory if not specified.

Checks: - Required files exist (CITATION.cff, LICENSE, omnibenchmark.yaml) - CITATION.cff is valid and has required fields - License consistency between CITATION.cff and LICENSE file

By default, shows warnings but doesn't fail. Use --strict to fail on warnings.

Usage:

ob validate module [OPTIONS] [PATH]

Options:

  --strict  Treat warnings as errors and fail on any validation issue.
  --help    Show this message and exit.

plan

Validate benchmark YAML plan structure.

Checks: - YAML syntax is valid - Required fields are present - Data types are correct - References between stages/modules are valid - Software environment backends are properly configured

Usage:

ob validate plan [OPTIONS] BENCHMARK

Options:

  --help  Show this message and exit.