Developing a Workflow Extension with a Custom Registry

Kaapana workflow extensions can be developed to use processing-container images from a custom container registry instead of the default registry configured for the platform.

This guide demonstrates how to prepare and upload such a workflow extension using a custom registry, based on the otsus-method-workflow example. The guide consists of two steps

  1. Providing the processing-container images for the operators

  2. Preparing the extension-chart with the Airflow code such as the dag file and operator files

Prerequisites

The following prerequisites are assumed:

  • Access to the Kaapana source code directory

  • Docker installed and configured locally

  • Helm installed

  • Permissions to create and push images to a custom container registry at <protocol>://<registry>:<port>/<repository>

  • Access to the Kaapana Web Interface

Provide processing-container images

Build and Push the Processing-Container Image

Build and push the processing-container images for your operators to the custom registry:

docker build -t <registry>:<port>/<repository>/otsus-method:1.0.0 .
docker push <registry>:<port>/<repository>/otsus-method:1.0.0

Launch a New Registry extension with scope Project-Secret

For each project that should be able to use these processing-container images launch a New Registry extension in Kaapana. Configure it as follows:

  • display_name: Choose a unique and identifiable name for this New Registry, e.g. REGISTRY-<project-name>-<registry-name>

  • secret_scope: Project-Secret

  • custom_registry_url: <protocol>://<registry>:<port>/<repository>

  • credentials_curstom_registry_username: The username to access the registry

  • credentials_curstom_registry_password: The password to access the registry

  • test_pull_image: <registry>:<port>/<repository>/otsus-method:1.0.0

This will create the registry-secret in the selected active project.

Important

The Absolute Registry URL for the New Registry has to contain the protocol, either http or https. For more information about installation errors and configuration options checkout the documentation <extensions_new_registry>.

Prepraing the extension-chart

Prepare the Airflow Code and Build the DAG Installer Image

  1. Set the image parameter of your operators according to the processing-container images you pushed to your registry without the protocol.

    Set image="<registry>:<port>/<repository>/otsus-method:1.0.0", in OtsusNotebookOperator.py and OtsusMethodOperator.py.

  2. Build and push the DAG installer image containing the Airflow code:

    docker build -t <registry>:<port>/<repository>/dag-otsus-method:0.0.0 .
docker push <registry>:<port>/<repository>/dag-otsus-method:0.0.0

Important

The image version 0.0.0 of the dag-otsus-method image is necessary as it matches with the default version in the dag-installer chart.

Prepare, Package, and Upload the Workflow Helm Chart

  1. Update the workflow Helm chart configuration.

Set the custom registry URL in otsus-method/extension/otsus-method-workflow/values.yaml.

---
global:
    image: "dag-otsus-method"
    action: "copy"
    custom_registry_url: "<protocol>://<registry>:<port>/<repository>"

  1. Specify the version of the chart in Chart.yaml

    ---
...
name: otsus-method-workflow
version: "1.0.0"
...

  2. Specify the relative path to the dag-installer-chart in requirements.yaml

    ---
dependencies:
- name: dag-installer-chart
version: 0.0.0
repository: file://<relative-path-to-dag-installer-chart>

    You find the dag-installer chart at /services/utils/dag-installer-chart in the kaapana repository.

  3. Package the Helm chart:

    cd otsus-method/extension/otsus-method-workflow
helm dep up
helm package .

  4. Upload the packaged Helm chart otsus-method-workflow-1.0.0.tgz via the Extensions View in the Kaapana Web Interface.

Note

The version of the workflow corresponds to the version in the Chart.yaml file.

Launch a New Registry Extension with scope Service-Secret

Launch a New Registry extension in Kaapana. Configure it as follows:

  • display_name: Choose a unique and identifiable name for this New Registry, e.g. REGISTRY-service-<registry-name>

  • secret_scope: Service-Secret

  • custom_registry_url: <protocol>://<registry>:<port>/<repository>

  • credentials_curstom_registry_username: The username to access the registry

  • credentials_curstom_registry_password: The password to access the registry

  • test_pull_image: <registry>:<port>/<repository>/dag-otsus-method:0.0.0

This will create a registry-secret that will be used to pull the image with the Airflow code.

Important

The registry-url for the New Registry has to contain the protocol either http or https. For more information about installation errors and configuration options checkout the documentation.