Reference

The primary way for users to interact with the exacb framework is through the GitLab CI/CD Components.. The components introduce jobs in the GitLab CI/CD pipeline that run the benchmarks, gather the results, deploy them to the storage, and publish them to the GitLab Pages of the project. Currently, the exacb framework provides the following components:

jureap/jube
jureap/jedi

jureap/jube CI/CD Component

The jureap/jube is the primary component of the exacb framework for the JUREAP program. It inserts a selection of jobs into the GitLab CI/CD pipeline that are used to run benchmarks on the JSC machines, deploy the results to the storage, and publish the results to the Gitlab Pages of the project. The visibility of the complete set of data and Gitlab Pages is identical to the visibility of the benchmark repository.

Usage

The jube component accepts the above variables as input and run the JUBE benchmark on the specified machine with the specified parameters.

module load JUBE
jube-autorun -r \
  "-e --tag $[[inputs.usecase]] $[[inputs.queue]] $[[inputs.variant]] --outpath ../outpath" $[[inputs.jube_file]]

The above command runs the JUBE benchmark on the specified machine with the

include:
  - component: gitlab.jsc.fz-juelich.de/exacb/catalog/jureap/jube@main
    inputs:
      prefix: <prefix>
      variant: <variant{.subvariant}>
      machine: <machine name>
      queue: <slurm queue to use>
      project: <slurm project name>
      budget: <budget to use for the benchmark>
      reservation: <reservation to use for the benchmark>
      is_public: all
      jube_file: benchmark/jube/shell.yml
      fixture: <gitlab CI job template for job fixture>
      usecase: <the use case you want to run, empty by default>
      record: <record the benchmark results, default is true>

.fixture:
  before_script:
    - # ... code to setup the benchmark

**Inputs accepted by the jureap/jube CI/CD component**
name	description	required	default	allowed values
prefix	namespace prefix for CI jobs	yes	-	any
variant	variant and subvariants for the benchmark	yes	-	any
machine	machine for the benchmark	yes	-	juwels,juwels_booster,jureca
queue	queue for the benchmark	yes	-	any
project	project for the benchmark	yes	-	any
budget	budget account for the benchmark	yes	-	any
reservation	reservation for the benchmark	no	empty	any
is_public	publicity of the benchmark	yes	-	all,resultonly,private
jube_file	path to the jube file	yes	-	any
fixture	fixture template for the benchmark	yes	empty	any

The components mentioned above are specified in more detail below.

prefix

Prefix to namespace the CI/CD jobs introduced by the component.

One must use a unique prefix for each use of a component in your pipeline.

For example, if you set the prefix to mylib.exacb, and the component adds a job with basename build, the full job name will be mylib.exacb.build. Adding a prefix to your jobs names ensures that you can include a component multiple times in your pipeline without causing name conflicts. It also makes it easier to include the components in your production pipeline without conflicting with other jobs.

The component reserves the right to add any jobs starting with `prefix` , therefore, user should not manually any jobs starting with the `prefix` anywhere in the pipeline.

record (optional)

since: v3
default value: false

Record the benchmark results. The variable is optional and is false by default. The variable is used to control whether the benchmark results should be recorded or not. The variable is useful when the user wants to run the benchmark but does not want to record the results. The user can set the variable to false in such cases.

Note

The user should set the variable to false only when the user does not want to record the benchmark results. The user should set the variable to true when the user wants to record the benchmark results.

variant

The variant of the benchmark that you want to run on the machine.

Options: single, strong.micro, strong.tiny, strong.small, strong.medium, weak.tiny, jedi.evaluation.jedi (since v3), jedi.evaluation.jureca (since v3), jedi.evaluation.juwels_booster (since v3)

machine

The JSC machine on which to execute the CI job.

Options: juwels, juwels_booster, jureca, jedi

queue

The slurm queue that the benchmark should run on.

project

The name of the slurm project you want to use to run the benchmark. The slurm-based project namespace will be used to run the benchmarks. In particular, the benchmarks will be executed in /p/scratch/<project>/$GITLAB_USER_LOGIN.exacb/$CI_JOB_ID. The rest of the paths will be created automatically by the Jacamar CI component as needed.

Here, <project> is the value of the project variable. $GITLAB_USER_LOGIN is the Gitlab/JUDoor username for the user who has triggered the pipeline. $CI_JOB_ID is the unique identifier for the job which can be found by the user in the Gitlab CI/CD pipeline’s web UI.

The user triggering the pipeline should have access to the project specified here.
Note

The jureap/jube component is designed to run in the location dependent on <project> given by the path below. You can also find this information in the output of the CI/CD job as shown here .
/p/scratch/<project>/$GITLAB_USER_LOGIN.exacb/$CI_JOB_ID/$GITLAB_USER_LOGIN/<...>

label (optional)

since: v3
default value: <empty>

The label to use for the benchmark. The label is used to identify the benchmark in the benchmark results. The label is optional and is empty by default.

In JUREAP, the label is used as a caption for the benchmark table and for the name in the legend of the benchmark plot.

budget

Compute budget to use for the benchmark.

The user triggering the benchmark should have access to the budget specified here.

reservation (optional)

default value: <empty>

The reservation to use for the benchmark. The user triggering the benchmark should have access to the reservation specified here. The variable is optional, and in case no variable is provided, the benchmark will be executed without a reservation.

is_public

The visibility of the benchmarks. The JUREAP program might want to use and publish the results from the benchmarks for various purposes. In the process, it is important to ensure that publicity of various aspects of the benchmark results is enforced.

While the actual choice of the visibility is done offline in the JUREAP program, we provide a way for the applications to control the visibility of the benchmark results in the CI/CD pipelines as well. The extra control here acts as an additional check for the automated workflows.

Options:

all: Both results and benchmark code can be published publically.

resultonly: Only the results are public, while the benchmark code is private.

private: Neither the results nor the benchmark code is public.

jube_file

The path to the JUBE file to use for the benchmark. The path should be specified relative to the root of the repository.

usecase (optional)

default value: <empty>

The use case you want to run. The use case is a string that can be used to differentiate between different usecases you want to run from the same jube file. The use case is optional and is empty by default, which is what would be the case for most single benchmarks.

The string would be passed verbatim to jube as additional tags. See the invocation below for an example.

fixture

By default the CI/CD component will run something equivalent to the code presented at the start of the section

It is possible that a user might want to do more setup before and after the benchmark run is executed, in essence attach a fixture to the benchmark run. In the exacb framework, we provide a way for the user to specify the fixture in form of a gitlab CI/CD job template.

A user can provide gitlab CI/CD job template with a before_script section containing the setup script. The before_script section runs according to the policies of gitlab in the benchmark job. In particular, the before_script section runs before the actual benchmark. The environment setup in the before_script section is available to the benchmark job as well.

Additionally, the user can optionally provide a after_script section in the .fixture template. The after_script section runs after the benchmark job complete according to the policies of gitlab. In particular, the after_script section do not affect the job’s exit code, and runs in a new shell, separate from the before_script section and the benchmark job. The after_script section is useful for any tear down that the user might want to do.
Note

Due to the technical limitations of the Gitlab CI/CD components, the user must provide the input to the fixture variable even when the user does not need to set up anything. In such cases, the user can provide an empty before_script section as shown below. One can look at how it is used in the quickstart repository.
.setup.template:
  before_script:
    - echo "No Setup Needed"

jureap/jedi CI/CD Component

The jureap/jedi is the evaluation component of the exacb framework for the JUREAP JEDI evaluation milestone. The component takes in data from an existing pipeline runs, analyses the data, generates the report, stores the report in the pages branch of the repository.

IMPORTANT: The jureap/jedi component is designed to run on the data that already exists in the exacb.data branch of the repository from the previous pipeline runs. It CANNOT run on the data that is generated by the current pipeline run.

In order to run the jureap/jedi component, the user must have run the jureap/jube component before as specified in Evaluation Instructions in the recording mode.

The generated reports are stored in the pages branch of the repository. The component creates a directory containing both the report and the raw data used to generate the report. The directory is named using the scheme as follows:

<pipelineid>-<date-time>-<prefix>

where <pipelineid> is the pipeline id of the pipeline run which was used to generate the report, <date-time> is the date and time when the pipeline was created, and <prefix> is the prefix used to namespace the CI/CD jobs introduced by the component.

Usage

include:
  - component: gitlab.jsc.fz-juelich.de/exacb/catalog/jureap/jedi@v3
    inputs:
      prefix: "report.jedi.evaluation"
      pipeline: [ "221418", "221415" ] # Array of pipeline ids
      selector: [ "jedi.strong.tiny" , "jureca.strong.tiny" ] # Array of corresponding selectors
      ### WORKLOAD FACTOR IS DEPRECATED!!! IT IS NOT USED IN THE JEDI EVALUATION
      workload_factor: [ "1.2" , "1.0" ] # Array of corresponding workload factors

**Inputs accepted by the jureap/jube CI/CD component**
name	description	required	default	allowed values
pipeline	id of pipeline to include in results	yes	-	integer number of pipeline id
selector	prefix of the results to use in the pipeline	yes	-	string of prefix
workload_factor (DEPRECATED)	no use	no	1.0	float number

The jube component accepts the above variables as input. The components mentioned above are specified in more detail below.

pipeline

The array of pipeline ids to use for the evaluation. The pipeline id is used to select the data from a particular pipeline run for evaluation. The entries of the pipeline array correspond in a one-to-one manner with the selector array. One can use as many pipeline ids as needed for the plotting. The data of different pipelines from the same machine will be aggregated into a single data set when plotting.

selector

The selector array to use for the evaluation. The selector is used to select the data from a particular prefix in the pipeline for evaluation. The entries of the selector array correspond in a one-to-one manner with the entries of the pipeline array.

In the example usage above, the selector jedi.strong.tiny is used to select the data from the jedi.strong.tiny instance of pipeline [ “221418”] for evaluation, while the selector jureca.strong.tiny is used to select the data from the jureca.strong.tiny instance of pipeline [ “221415”] for evaluation.

workload_factor (DEPRECATED!)

The workload factor is a DEPRECATED parameter here, not used in the JEDI evaluation. The parameter is kept for backward compatibility with the previous versions of the component. Please set the parameter to an array of the same length as the pipeline array with the value 1.0 for each entry in the array as shown in the example.

jureap/scaling CI/CD Component

The jureap/scaling is the evaluation component of the exacb framework for the JUREAP scaling evaluation milestone. The component takes in data from an existing pipeline runs, analyses the data, generates the report, stores the report in the pages branch of the repository.

IMPORTANT: The jureap/scaling component is designed to run on the data that already exists in the exacb.data branch of the repository from the previous pipeline runs. It CANNOT run on the data that is generated by the current pipeline run.

In order to run the jureap/scaling component, the user must have run the jureap/jube component before as specified in :ref:`Evaluation Instructions <https://gitlab.jsc.fz-juelich.de/jureap/jureap/-/issues/21>`_ in the recording mode.

The generated reports are stored in the pages branch of the repository. The component creates a directory containing both the report and the raw data used to generate the report. The directory is named using the scheme as follows:

<pipelineid>-<date-time>-<prefix>

where <pipelineid> is the pipeline id of the pipeline run which was used to generate the report, <date-time> is the date and time when the pipeline was created, and <prefix> is the prefix used to namespace the CI/CD jobs introduced by the component.

Usage

include:
  - component: gitlab.jsc.fz-juelich.de/exacb/catalog/jureap/scaling@v3
    inputs:
      prefix: "report.scaling.evaluation"
      pipeline: [ "221418", "221415" ] # Array of pipeline ids
      selector: [ "scaling.strong.tiny" , "jureca.strong.tiny" ] # Array of corresponding selectors
      workload_factor: [ "1.2" , "1.0" ] # Array of corresponding workload factors
      skip_workload_factor: false # Skip the workload factor error bars in the plot
      legend_position: "0.9,0.15" # Position of the legend in the plot (x,y)

The jube component accepts the above variables as input. The components mentioned above are specified in more detail below.

pipeline

The array of pipeline ids to use for the evaluation. The pipeline is used to

selector

The selector array to use for the evaluation. The selector is used to select the data from a particular prefix in the pipeline for evaluation. The entries of the selector array correspond in a one-to-one manner with the entries of the pipeline array.

In the example usage above, the selector scaling.strong.tiny is used to select the data from the scaling.strong.tiny instance of pipeline [ “221418”] for evaluation, while the selector jureca.strong.tiny is used to select the data from the jureca.strong.tiny instance of pipeline [ “221415”] for evaluation.

workload_factor

The workload factor to use for the evaluation. The workload_factor input parameter is used to assign a workload parameter for benchmarks. This is meant to be used in cases where the workloads of the two benchmarks (the one on scaling and the one on JURECA for example) are different. Different benchmark workload means that the benchmarks are not directly comparable. The workload factor is then used to make the benchmarks comparable.

Assume you are doing the following scaling:

Tiny : 1,2,4 nodes

Small : 8,16,32 nodes

Medium : 64,128,256 nodes

and assume that the computational workload to execute the simulation is given by

Tiny : 1 unit

Small : 8 units

Medium : 64 units

Then, for a perfectly weak scaling workload, the simulation on 1, 8 and 64 nodes should take the same time. The workload factors are then given by

Tiny : 1

Small : 8

Medium : 64

skip_workload_factor (optional)

since: v3
default value: false

Skip the workload factor error bars in the plot. The variable is optional and is false by default. The variable is used to control whether the workload factor error bars should be plotted or not. The variable is useful when the user wants to plot the workload factor error bars. The user can set the variable to false in such cases.

Note

The user should set the variable to false only when the user wants to plot the workload factor error bars. The user should set the variable to true when the user does not want to plot the workload factor error bars.

legend_position (optional)

since: v3
default value: 0.9,0.15 (x,y)

The position of the legend in the plot in (x,y) co-ordinates. The variable is optional and is 0.9,0.15 by default. The variable is used to control the position of the legend in the plot. The variable is useful when the user wants to change the position of the legend in the plot. The user can set the variable to the desired position in such cases.

Note

The user should set the variable to the desired position of the legend in the plot. The user should set the variable to the default value when the user does not want to change the position of the legend in the plot.