exacb Framework

Note

See specific notes on usage of exacb in the context of JUREAP on the dedicated page.

The exacb framework is a set of tools and configurations to enable Continuous Benchmarking (CB) through GitLab on JSC systems. With a given target application, it provides an easy common entry point to monitor application performance and collect data in a uniform manner.

The name, exacb, originates from the fact that the framework was set up in the scope of the JUPITER system, the EuroHPC JU Exascale system to be hosted at JSC. The framework is used for the JSC-internal set of applications benchmarking JUPITER and for key user applications, for example, applications participating in JUREAP (the JUPITER Research and Early Access Program).

The exacb framework primarily consists of command-line reporting tool exacb and a set of Gitlab CI/CD components.

• Tutorial: Instructions on getting started with the exacb framework.

• Reference: API Reference for the various tools.

Note

If you need help, please open an issue on our help repository.

Contents:

• Tutorial
  • Ensure that the CI Timeout is Sufficient
  • Private Token Generation
  • Setting Default Project and Budget Variables
  • Adding a CI/CD Job
  • Disabling Specific CI/CD Jobs
  • Using Reservations for Benchmarking
  • Recording Benchmark Results in the Repository
  • Canceling a Benchmark Job
• Reference
  • jureap/jube CI/CD Component
  • jureap/jedi CI/CD Component
  • jureap/scaling CI/CD Component
• JUREAP
  • Designing your Benchmarks
  • Reproducible Benchmarks using JUBE
  • Continuous Benchmarking using exacb
• FAQs
  • Plotting Large Scaling Results Manually
  • Partially Failed CI Pipeline
  • Empty CI Pipeline
  • Partition Split on JUBE
  • Increase CI Timeout
  • Switch to v3 Branch
  • Incorrect SLURM Project
  • Disabling Specific CI/CD Jobs
  • Manually Scheduling Jobs
• Changelogs
  • 2024-10-04 : 3.0.0 (v3)
  • 2024-07-25
  • 2024-07-04
• Checklist
  • Step 0 : Setup and Preparation
  • Step 1: exacb Integration
  • Step 2 : SPEP Meeting
  • Step 3 : JEDI Performance
  • Step 4: Large-scale scaling
  • Step 5: Documentation
  • Step 6: Certificate
• How-To Guides
  • How-To Guide : JEDI Evaluation Run