exacb Framework
exaCB is a framework for continuous benchmarking on HPC systems, designed to scale up to exascale-class machines. It enables CI-driven benchmarking workflows by combining lightweight command-line tooling with reusable CI/CD building blocks (e.g., GitLab CI/CD templates) and by integrating established benchmarking harnesses. It forms part of a suite of tools at JSC, alongside JUBE and LLView, to support systematic and reproducible benchmarking studies on JSC systems.
exaCB connects its components through a shared protocol and data model, allowing tools and workflows to communicate in a standardized way. This modular design lets sites and application teams swap components as needed, supporting a broad range of benchmarking scenarios while remaining extensible.
After execution, benchmark results are post-processed by the exaCB command-line tools into structured, machine-readable reports. These reports are intended to be easy to analyze and compare across systems and over time. The framework supports multiple benchmarking modes, including:
Performance benchmarking (time-to-solution and related metrics)
Scalability studies
Energy-efficiency evaluations
A key design principle of exaCB is incremental adoption. Benchmarks can be onboarded at a minimal “runnability” level and then progressively evolve toward richer capabilities such as instrumented measurements and full reproducibility. This lowers the barrier to entry for application teams, enables rapid onboarding, and provides an auditable path toward producing structured results that remain trackable across system updates and software changes.
Usage in Practice
We demonstrate the practicality of this approach through its use in JUREAP, an early-access program preparing applications for the JUPITER exascale system at JSC. Using exaCB, we executed benchmarks for a collection of 70+ applications spanning diverse domains. Despite heterogeneous benchmark maturity and an evolving system environment, runs were orchestrated automatically via shared CI/CD components and results were collected in a uniform format—enabling system-wide analyses with minimal additional effort.
Beyond runtime metrics, exaCB supports framework-level workload augmentation. For example, shared platform configurations can inject non-invasive instrumentation (such as wrappers for energy measurement) consistently across a benchmark collection.
We are also integrating the JUPITER Benchmark Suite (sixteen application benchmarks and seven synthetic benchmarks) into exaCB. The suite includes extensive JUBE workflows and reference results and played a central role in the procurement of the JUPITER system. Its integration into exaCB aims to ensure that procurement-level benchmarks can be reproduced continuously in CI/CD workflows, using the same declarative and reproducible methodology as application-centric studies.
Getting started and reference
Note
If you need help, please open an issue on our help repository.
Contents: