Runtime Deployment

This section describes how to install and configure Aether Edge Runtime including Kubernetes and system level applications listed below.

  • sealed-secrets

  • rancher-monitoring

  • fluent-bit

  • opendistro-es

  • hostpath-provisioner

  • edge-maintenance-agent

  • sriov-device-plugin

  • uedns

For this, we will be using Aether’s GitOps based CI/CD systems and what you will need to do is create patches in Aether GitOps repositories, aether-pod-configs and aether-app-configs, to provide cluster and application configurations to the CI/CD systems.

Attention

If you skipped VPN bootstrap step and didn’t add the deployment jobs for the new edge, go to Add deployment jobs step and finish it first before proceeding.

Prepare System Application Configuration

In this step, you will create system application configurations for the new cluster so that the new cluster can start with proper configurations as soon as it is deployed. For the K8S application configuration and deployment, Aether leverages Rancher’s built-in GitOps tool, Fleet, and aether-app-configs is the repository where all Aether applications are defined.

Most of the Aether system applications listed above do not require cluster specific configurations, except rancher-monitoring and uedns. For these applications, you will have to manually create custom configurations and commit them to aether-app-configs.

First, download aether-app-configs if you don’t have it already in your development machine.

$ cd $WORKDIR
$ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-app-configs"

Configure rancher-monitoring

Open fleet.yaml under infrastructure/rancher-monitoring, add a custom target with the new cluster name as a selector, and provide cluster specific Helm values and kustomize overlay directory path like below.

$ cd $WORKDIR/aether-app-configs/infrastructure/rancher-monitoring
$ vi fleet.yaml
# add following block at the end
- name: ace-test
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: ace-test
  helm:
    values:
      prometheus:
        prometheusSpec:
          additionalAlertRelabelConfigs:
            - source_labels: [__address__]
              target_label: cluster
              replacement: ace-test
  kustomize:
    dir: overlays/prd-ace

Note

Above step will not be required in Rancher v2.6 as it supports using cluster labels as helm values in a list.

Configure ue-dns

For UE-DNS, it is required to create a Helm values file for the new cluster. You’ll need cluster domain and kube-dns ClusterIP. Both can be found in aether-pod-configs/production/cluster_map.tfvars. Be sure to replace [ ] in the example configuration below to the actual cluster values.

$ cd $WORKDIR/aether-app-configs/infrastructure/coredns
$ mkdir overlays/prd-ace-test
$ vi overlays/prd-ace-test/values.yaml
# SPDX-FileCopyrightText: 2021-present Open Networking Foundation <info@opennetworking.org>

serviceType: ClusterIP
service:
  clusterIP: [next address of the kube-dns ip]
servers:
  - zones:
      - zone: .
    port: 53
    plugins:
      - name: errors
      - name: health
        configBlock: |-
          lameduck 5s
      - name: ready
      - name: prometheus
        parameters: 0.0.0.0:9153
      - name: forward
        parameters: . /etc/resolv.conf
      - name: cache
        parameters: 30
      - name: loop
      - name: reload
      - name: loadbalance
  - zones:
      - zone: aetherproject.net
    port: 53
    plugins:
      - name: errors
      - name: rewrite continue
        configBlock: |-
          name regex (.*)\.aetherproject.net {1}.svc.[cluster domain]
          answer name (.*)\.svc\.[cluster domain] {1}.aetherproject.net
      - name: forward
        parameters: . [kube-dns ip]
        configBlock: |-
          except kube-system.svc.[cluster domain] aether-sdcore.svc.[cluster domain] tost.svc.[cluster domain]
      - name: cache
        parameters: 30

Next, update fleet.yaml under infrastructure/coredns so that Fleet can use the custom configuration you just created when deploying UE-DNS to the cluster.

$ cd $WORKDIR/aether-app-configs/infrastructure/coredns
$ vi fleet.yaml
# add following block at the end
- name: prd-ace-test
  clusterSelector:
    matchLabels:
      management.cattle.io/cluster-display-name: ace-test
  helm:
    valuesFiles:
      - overlays/prd-ace-test/values.yaml

Submit your changes.

$ cd $WORKDIR/aether-app-configs
$ git status
$ git add .
$ git commit -m "Add test ACE application configs"
$ git review

Now, it’s ready to deploy K8S.

K8S cluster deployment

Download aether-pod-configs repository if you don’t have it already in your development machine.

$ cd $WORKDIR
$ git clone "ssh://[username]@gerrit.opencord.org:29418/aether-pod-configs"

Attention

If you skipped VPN bootstrap step and didn’t update global resource maps for the new edge, go to Update global resource maps step and finish cluster_map.tfvars update first before proceeding.

Run the following commands to automatically generate Terraform configurations needed to create a new cluster in Rancher and add the servers and switches to the cluster.

# Create ace_cofig.yaml file if you haven't yet
$ cd $WORKDIR/aether-pod-configs/tools
$ cp ace_config.yaml.example ace_config.yaml
$ vi ace_config.yaml
# Set all values

$ make runtime
Created ../production/ace-test/provider.tf
Created ../production/ace-test/cluster.tf
Created ../production/ace-test/rke-bare-metal.tf
Created ../production/ace-test/addon-manifests.yml.tpl
Created ../production/ace-test/project.tf
Created ../production/ace-test/backend.tf
Created ../production/ace-test/cluster_val.tfvars

Attention

If the cluster has an even number of compute nodes, edit cluster_val.tfvars file so that only the odd number of nodes have etcd and controlplane roles.

Create a review request.

$ cd $WORKDIR/aether-pod-configs
$ git add .
$ git commit -m "Add test ACE runtime configs"
$ git review

Once your review request is accepted and merged, Aether CI/CD system starts to deploy K8S. Wait until the cluster status changes to Active in Rancher. It normally takes 10 - 15 minutes depending on the speed of the container images download at the edge.

It is also a good idea to check the system pod status after successful K8S deployment. To do so, login to Rancher, open the cluster that you just deployed in the Global view, and click Launch kubectl button. You can interact with the cluster using the window that opens. Run the following commands and make sure all pods are Running.

# Run kubectl commands inside here
# e.g. kubectl get all
> kubectl get po -A

Attention

Ignore BESS UPF failure at this point if BESS UPF is enabled. We’ll fix it in BESS UPF step.

Disable deployment jobs

After confirming the K8S cluster is ready, disable the deployment job.

$ cd $WORKDIR/aether-ci-management
$ vi jjb/repos/cd-pipeline-terraform.yaml

# Add jobs for the new cluster
diff jjb/repos/cd-pipeline-terraform.yamll
--- a/jjb/repos/cd-pipeline-terraform.yaml
+++ b/jjb/repos/cd-pipeline-terraform.yaml
@@ -227,3 +227,9 @@
       - 'cd-pipeline-terraform-postmerge-cluster':
           cluster: 'ace-eks'
       - 'cd-pipeline-terraform-premerge-cluster':
           cluster: 'ace-test'
-          disable-job: false
       - 'cd-pipeline-terraform-postmerge-cluster':
           cluster: 'ace-test'
-          disable-job: false

Submit your change and wait for the job is updated.