Aether-in-a-Box FAQs and Troubleshooting

FAQs

RKE2 vs. Kubespray Install

The AiaB installer will bring up Kubernetes on the server where it is run. By default it uses RKE2 as the Kubernetes platform. However, older versions of AiaB used Kubespray and that is still an option. To switch to Kubespray as the Kubernetes platform, edit the Makefile and replace rke2 with kubespray on this line:

node0:~/aether-in-a-box$ git diff Makefile
diff --git a/Makefile b/Makefile
index 5f2c186..608c221 100644
--- a/Makefile
+++ b/Makefile
@@ -35,7 +35,7 @@ ENABLE_GNBSIM ?= true
 ENABLE_SUBSCRIBER_PROXY ?= false
 GNBSIM_COLORS ?= true

-K8S_INSTALL ?= rke2
+K8S_INSTALL ?= kubespray
 CTR_CMD     := sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io

 PROXY_ENABLED   ?= false
node0:~/aether-in-a-box$

You may wish to use Kubespray instead of RKE2 if you want to use locally-built images with AiaB (e.g., if you are developing SD-CORE services). The reason is that RKE2 uses containerd instead of Docker and so cannot access images in the local Docker registry.

How to use Local Image

Note that RKE2 (the default Kubernetes installer) is based on containerd rather than Docker. Containerd has its own local image registry that is separate from the local Docker Registry. With RKE2, if you have used docker build to build a local image, it is only in the Docker registry and so is not available to run in AiaB without some additional steps. An easy workaround is to use docker push to push the image to a remote repository (e.g., Docker Hub) and then modify your Helm values file to pull in that remote image. Another option is to save the local Docker image into a file and push the file to the containerd registry like this:

docker save -o /tmp/lte-uesoftmodem.tar omecproject/lte-uesoftmodem:1.1.0
sudo /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io \
    images import /tmp/lte-uesoftmodem.tar

The above commands save the local Docker image omecproject/lte-uesoftmodem:1.1.0 in a tarball, and then upload the tarball into the containerd registry where it is available for use by RKE2. Of course you should replace omecproject/lte-uesoftmodem:1.1.0 with the name of your image.

If you know that you are going to be using AiaB to test locally-built images, probably the easiest thing to do is to use the Kubespray installer. If you have already installed using RKE2 and you want to switch to Kubespray, first run make clean before following the steps in the Getting Help section above.

Restarting the AiaB Server

AiaB should come up in a mostly working state if the AiaB server is rebooted. If any pods are stuck in an Error or CrashLoopBackoff state they can be restarted using kubectl delete pod. It might also be necessary to power cycle the Sercomm eNodeB in order to get it to reconnect to the SD-CORE.

Enabling externalIP at MME

You can enable externalIP service in the MME by providing following config in the override file:

node0:~/aether-in-a-box$ git diff sd-core-4g-values.yaml
diff --git a/sd-core-4g-values.yaml b/sd-core-4g-values.yaml
index 0939739..f240f89 100644
--- a/sd-core-4g-values.yaml
+++ b/sd-core-4g-values.yaml
@@ -24,6 +24,11 @@ omec-control-plane:
       bootstrap:
         users: []
         staticusers: []
+    mme:
+      s1ap:
+        serviceType: ClusterIP
+        externalIP: 10.1.1.1
+
     spgwc:
       pfcp: true
       cfgFiles:
node0:~/aether-in-a-box$

Enabling externalIP at AMF

You can enable externalIP service in the AMF by providing following config in the override file:

node0:~/aether-in-a-box$ git diff sd-core-5g-values.yaml
diff --git a/sd-core-5g-values.yaml b/sd-core-5g-values.yaml
index e513e1f..fc1c684 100644
--- a/sd-core-5g-values.yaml
+++ b/sd-core-5g-values.yaml
@@ -34,6 +34,9 @@

     amf:
       cfgFiles:
+        ngapp:
+          serviceType: ClusterIP
+          externalIp: "10.1.1.2"
+          port: 38412
         amfcfg.conf:
           configuration:
             enableDBStore: false
@@ -176,6 +179,7 @@ omec-user-plane:
           cpiface:
             dnn: "internet"
             hostname: "upf"

 5g-ran-sim:
   enable: ${ENABLE_GNBSIM}

node0:~/aether-in-a-box$

Troubleshooting

NOTE: Running both 4G and 5G SD-CORE simultaneously in AiaB is currently not supported.

Proxy Issues

When working with AiaB behind a proxy, it may be possible to experience certain issues due to security policies. That is, the proxy may block a domain (e.g., opencord.org) and you may see messages like these ones when trying to clone or get a copy of aether-in-a-box:

ubuntu18:~$ git clone https://gerrit.opencord.org/aether-in-a-box
Cloning into 'aether-in-a-box'...
fatal: unable to access 'https://gerrit.opencord.org/aether-in-a-box/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none

or:

ubuntu18:~$ wget https://gerrit.opencord.org/plugins/gitiles/aether-in-a-box/+archive/refs/heads/master.tar.gz
--2022-06-01 13:13:42--  https://gerrit.opencord.org/plugins/gitiles/aether-in-a-box/+archive/refs/heads/master.tar.gz
Resolving proxy.company-xyz.com (proxy.company-xyz.com)... w.x.y.z
Connecting to proxy.company-xyz.com (proxy.company-xyz.com)|w.x.y.z|:#... connected.
ERROR: cannot verify gerrit.opencord.org's certificate, issued by 'emailAddress=proxy-team@company-xyz.com,... ,C=US':
 Self-signed certificate encountered.

To address this issue, you need to talk to your company’s proxy admins and request to unblock (re-classify) the opencord.org domain

“make” fails immediately

AiaB connects macvlan networks to DATA_IFACE so that the UPF can communicate on the network. To do this it assumes that the systemd-networkd service is installed and running, DATA_IFACE is under its control, and the systemd-networkd configuration file for DATA_IFACE ends with <DATA_IFACE>.network, where <DATA_IFACE> stands for the actual interface name. It tries to find this configuration file by looking in the standard paths. If it fails you’ll see a message like:

FATAL: Could not find systemd-networkd config for interface foobar, exiting now!
make: *** [Makefile:112: /users/acb/aether-in-a-box//build/milestones/interface-check] Error 1

In this case, you can specify a DATA_IFACE_PATH=<path to the config file> argument to make so that AiaB can find the systemd-networkd configuration file for DATA_IFACE. It’s also possible that your system does not use systemd-networkd to configure network interfaces (more likely if you are running in a VM), in which case AiaB is currently not able to install in your setup. You can check that systemd-networkd is installed and running as follows:

$ systemctl status systemd-networkd.service
● systemd-networkd.service - Network Service
    Loaded: loaded (/lib/systemd/system/systemd-networkd.service; disabled; vendor preset: enabled)
    Active: active (running) since Tue 2022-07-12 13:42:18 CDT; 2h 26min ago
TriggeredBy: ● systemd-networkd.socket
    Docs: man:systemd-networkd.service(8)
Main PID: 13777 (systemd-network)
    Status: "Processing requests..."
    Tasks: 1 (limit: 193212)
    Memory: 6.4M
    CGroup: /system.slice/systemd-networkd.service
            └─13777 /lib/systemd/systemd-networkd

Data plane is not working

The first step is to read Understanding AiaB networking`_understanding_aiab_networking, which gives a high level picture of the AiaB data plane and how the pieces fit together. In order to debug the problem you will need to figure out where data plane packets from the eNodeB are dropped. One way to do this is to run ``tcpdump` on (1) DATA_IFACE to ensure that the data plane packets are arriving, (2) the access interface to see that they make it to the UPF, and (3) the core to check that they are forwarded upstream.

If the upstream packets don’t make it to DATA_IFACE, you probably need to add the static route on the eNodeB so packets to the UPF have a next hop of DATA_IFACE. You can see these upstream packets by running:

tcpdump -i <data-iface> -n udp port 2152

If they don’t make it to access you should check that the kernel routing table is forwarding a packet with destination 192.158.252.3 to the access interface. You can see them by running:

tcpdump -i access -n udp port 2152

If they don’t make it to core then they are being dropped by the UPF for some reason. This may be a configuration issue with the state loaded in the ROC / SD-CORE – the UPF is being told to discard these packets. You should check that the device’s IMSI is part of a slice and that the slice’s policy settings allow traffic to that destination. You can view them via the following:

tcpdump -i core -n net 172.250.0.0/16

That command will capture all packets to/from the UE subnet.

If you cannot figure out the issue, see Getting Help.

Getting Help

Please introduce yourself and post your questions to the #aether-dev channel on the ONF Community Slack. Details about how to join this channel can be found on the ONF Wiki. In your introduction please state your institution and position, and describe why you are interested in Aether and what is your end goal.

If you need help debugging your setup, please give as much detail as possible about your environment: the OS version you have installed, are you running on bare metal or in a VM, how much CPU and memory does your server have, are you installing behind a proxy, and so on. Also list the steps you have performed so far, and post any error messages you have received. These details will aid the community to understand where you are and how to help you make progress.