Tutorial

Note

Please see the quickstart repository for a demonstration of the steps described in this tutorial.

This tutorial will guide you through the process of setting up a benchmark repository and running a benchmark on the JUWELS Booster system using the exacb framework, using the JUREAP components as an example. In particular, we will use the jureap/jube Gitlab CI/CD components to introduce jobs in the Gitlab CI/CD workflow

We assume that you have a JUBE-based benchmark script that you want to run on the JUWELS Booster. Please look at the exacb quickstart repository for a demonstration of the steps described in this tutorial. We further assume that the benchmark script is present at path_to_benchmark.yml. Then, one needs to follow the following steps:

Ensure that the CI Timeout is Sufficient

Please ensure that the CI timeout is set to 7d in the GitLab CI/CD settings (Settings -> CI/CD -> General pipelines -> Timeout). This is to ensure that the benchmark job does not get killed due to a timeout which can happen if the SLURM queue is full and the job is waiting for a long time to get scheduled.

Private Token Generation

We will need a way to push the execution results of the benchmark to an orphan branch of your benchmark repository. For this, generate a Project Access Token for your repository with the permission to write to the repository write_repository. Add this token to the variables in the GitLab CI/CD settings (Settings -> CI/CD -> Variables) of your repository with the following details:

Type: Variable

Environments: All

Role -> Maintainer

Flags:

Protect Variable: Yes

Mask Variable: Yes

Expand Variable: No

Key: EXACB_STORAGE_ACCESS_TOKEN

Value: <Your Project Access Token>

Setting Default Project and Budget Variables

To allow for users to run benchmarks using different project and budget settings, we set the default project and budget variables in the GitLab CI/CD variables instead of hardcoding them in the component code. In order to do this, we first need to set the default project and budget variables in the GitLab CI/CD settings (Settings -> CI/CD -> Variables) of your repository with the following details:

Variable for Compute Budget
- Type: Variable
- Environments: All
- Visible/Masked -> Visible
- Flags:
  - Project Variable: No
  - Expand variable reference: No
- Description: Compute Budget
- Key: EXACB_BUDGET
- Value: <Your Budget Name>
Variable for Project Environment
- Type: Variable
- Environments: All
- Visible/Masked -> Visible
- Flags:
  - Project Variable: No
  - Expand variable reference: No
- Description: Project Environment
- Key: EXACB_PROJECT
- Value: <Your Project Name>

Adding a CI/CD Job

Next, we register a CI/CD job in the GitLab CI/CD pipeline to run the benchmark. The step consists of two parts: specifying a setup template for the job and then adding the job to the GitLab CI/CD pipeline.

Setting up the Benchmark Fixture

If you need to set up a benchmark fixture before running the benchmark, you can add a setup job template to the .gitlab-ci.yml file. Due to current current CI/CD component design, one needs to add the setup job template to the .gitlab-ci.yml file with a before_script section even if you do not need to set up a fixture. One can add an empty before_script section to the setup job template in such as case

.setup.template:
  before_script:
    - echo "No Setup Needed"

Adding the Benchmark Job

To add a benchmark job, we include the JUREAP CI/CD compoenent and pass some variables to the component as an input.

The following inclusion of the component will run the single node variant of the benchmark script in benchmark/jube/shell.yml on the JUWELS Booster system using the exalab compute budget and the project environment cexalab. Both the benchmark code and the results are assumed to be public. It uses the empty benchmark fixture specified as as .setup job template at the bottom.

include:
  - component: gitlab.jsc.fz-juelich.de/exacb/catalog/jureap/jube@main
    inputs:
      prefix: <namespace you prefer> # Example: quickstart.juwels.booster.single
      machine: <machine> # refer to
      queue: booster
      variant: single
      is_public: all
      jube_file: benchmark/jube/shell.yml
      setup: .setup
      project: $EXACB_PROJECT
      budget: $EXACB_BUDGET

.setup:
  before_script:
    - echo "No Setup Needed"

Adding Multiple Benchmark Jobs

Multiple benchmark jobs can be added by including the JUREAP CI/CD component multiple times with different inputs. The following example shows how to add two benchmark jobs, one for the single node variant and the other for the the strong.tiny variant with two different (but still empty) benchmark fixtures.

include:
  - component: gitlab.jsc.fz-juelich.de/exacb/catalog/jureap/jube@main
    inputs:
      prefix: "quickstart.exacb.single"
      variant: "single"
      machine: "juwels_booster"
      queue: "booster"
      project: $EXACB_PROJECT
      budget: $EXACB_BUDGET
      is_public: "all"
      jube_file: "benchmark/jube/shell.yml"
      fixture: .setup.single
  - component: gitlab.jsc.fz-juelich.de/exacb/catalog/jureap/jube@main
    inputs:
      prefix: "quickstart.exacb.strong.tiny"
      variant: "strong.tiny"
      machine: "juwels_booster"
      queue: "booster"
      project: $EXACB_PROJECT
      budget: $EXACB_BUDGET
      is_public: "all"
      jube_file: "benchmark/jube/shell.yml"
      fixture: .setup.strong.tiny

.setup.single:
  before_script:
    - echo "No Setup Needed"

.setup.strong.tiny:
  before_script:
    - echo "No Setup Needed even for strong.tiny variant"

Disabling Specific CI/CD Jobs

Please refer to the FAQ on disabling jobs to see how to disable jobs without having to change the component code.

Using Reservations for Benchmarking

If you want to use a specific reservation for benchmarking, you can specify the reservation name in the reservation input of the JUREAP CI/CD component as shown here.

Recording Benchmark Results in the Repository

If you want to record the benchmark results in the repository, you can specify the record input of the JUREAP CI/CD component as shown here. An example usage is shown below. Note that the component used here has the branch v3 since that is the first branch that introduced the record input.

include:
  - component: gitlab.jsc.fz-juelich.de/exacb/catalog/jureap/jube@v3
    inputs:
      prefix: "quickstart.exacb.single"
      variant: "single"
      machine: "juwels_booster"
      queue: "booster"
      project: $EXACB_PROJECT
      budget: $EXACB_BUDGET
      is_public: "all"
      jube_file: "benchmark/jube/shell.yml"
      fixture: .setup.single
      record: "true"

.setup.single:
  before_script:
    - echo "No Setup Needed"

Canceling a Benchmark Job

Due to the current design of the CI/CD components, one cannot cancel a slurm job from the GitLab CI/CD pipeline. In such cases, the user has to manually cancel the job from the compute systems using scancel command.