Aether-in-a-Box with External 5G Radio
This document describes how to set up an Aether-in-a-Box (AiaB) with a external gNodeB and connect real devices (e.g., 5G phones). It assumes that you are already familiar with running AiaB with emulated gNodeB/UE. AiaB on Hardware Radios is suitable for laboratory experiments and proof-of-concept deployments. Its goals are to provide an easy-to-install environment where Aether’s features can be explored with real devices. To create this setup you will need the following equipment:
Server for running AiaB (SD-CORE / UPF / ROC)
Ubuntu 18.04 or 20.04 clean install
Haswell CPU family or newer
At least 4 CPUs and 12GB RAM
Ability to run “sudo” without a password
No firewall running on the AiaB host
External 5G small cell gNodeB or simulator
If real phones are used then you need SIM card writer and blank SIM cards
IMPORTANT: AiaB is for simple deployment scenarios and so makes some simplifying assumptions:
AiaB assumes either 4G or 5G SD-CORE is deployed. Running both 4G and 5G SD-CORE simultaneously in AiaB is currently not supported. However, running both 4G and 5G SD-CORE simultaneously is supported by Aether.
Performance and scalability are not goals. AiaB does not support I/O acceleration (e.g., with SR-IOV). However, performance and scalability are goals of the Aether project.
AiaB assumes that the server and the gNodeB are connected to the same LAN and share the same IP subnet; in other words they can reach each other in a single hop and there is no IP router between them. This simplifies communication between the gNodeB and the UPF, which is running inside a container and has a private IP address that is not necessarily routable on the local network. However, this is not a requirement for all Aether deployments.
AiaB also assumes that the AiaB server’s network is configured using systemd-networkd, which is the default for Ubuntu, and copies some files into /etc/systemd/network; the reason for this is to enable persistence of AiaB’s networking configuration across server reboots. This configuration method is specific to AiaB.
The server will run Aether-in-a-Box. The gNodeB will connect to the server over the local network. Perform these steps to prepare the server for the AiaB install:
Connect the server to the local network
Perform a clean install of Ubuntu 18.04 or Ubuntu 20.04 on the server
Verify that systemd-networkd is being used to configure networking (e.g., run
systemctl status systemd-networkd.service)
Set up password-less sudo for the user that will install Aether-in-a-Box
After the steps above have been completed, install Aether-in-a-Box as follows:
sudo apt install git make git clone "https://gerrit.opencord.org/aether-in-a-box" cd aether-in-a-box
Next, modify the file sd-core-5g-values.yaml. Under
add an IMSI range for the SIM cards you created, with the Transport Key
and OPc values you used earlier. For example, the following will add
IMSIs between 315010999912301 and 315010999912303:
subscribers: - ueId-start: "315010999912301" ueId-end: "315010999912303" plmnId: "315010" opc: "69d5c2eb2e2e624750541d3bbc692ba5" key: "000102030405060708090a0b0c0d0e0f" sequenceNumber: 135
Determine which is the interface that has L3 connectivity to the
gNodeB – this will be
DATA_IFACE in the configuration later. If
the gNodeB will also be connected to the local network, then this is just the
server’s primary interface. If the gNodeB will be connected via an
isolated L2/L3 network segment, then
DATA_IFACE refers to the server
interface on that network. Remember this interface for later.
Option 1: Configure Aether with ROC
The Aether ROC provides a GUI and API for dynamically configuring Aether. If you don’t wish to use the ROC to configure AiaB, you can skip to the next section.
Aether Monitoring is available only when Aether is deployed using ROC. Monitoring support for 5G is still in progress.
Install AiaB as follows (specifying
DATA_IFACE from above):
ENABLE_GNBSIM=false DATA_IFACE=<iface> CHARTS=latest make roc-5g-models 5g-core
Next, use the ROC to add information about your SIM cards. The ROC GUI is available at http://<server-ip>:31194.
Configuration > Site from the drop-down at top right and edit
AiaB site. Change the following values and click
Sim Cards from the drop-down at top right. Edit the
existing entries to reflect the SIM cards you are adding to devices
by replacing their IMSI values. Click
Update after each edit.
If you want to connect more than two devices, consult the ROC
Finally, click the Basket icon at top right and click the
Now jump to the Verifying the AiaB installation section.
Option 2: Configure Aether without ROC
It is possible to configure Aether without the ROC, using static YAML files and the SimApp service. If you have already installed the ROC, you should skip this section.
Edit sd-core-5g-values.yaml. Change
mnc as follows:
plmn: mcc: "315" mnc: "010"
Also add the IMSIs of your devices under
imsis, for example:
device-groups: - name: "5g-gnbsim-user" imsis: - "315010999912301" - "315010999912302" - "315010999912303"
Install AiaB as follows (specifying
DATA_IFACE from above):
ENABLE_GNBSIM=false DATA_IFACE=<iface> CHARTS=latest make 5g-core
Verifying the AiaB installation
Installing AiaB will take about 20 minutes with a fast Internet
connection. If you see any errors / timeouts, try running the
command again. The build will finish with a message:
“Your MME IP address is… ” This is just the IP address assigned to
DATA_IFACE. Remember this for the gNodeB setup.
When the install is complete, check that the 5G SD-CORE is running as follows:
$ kubectl get pods -n omec NAME READY STATUS RESTARTS AGE amf-6d9d8f44c8-2d7f7 1/1 Running 0 4m6s ausf-9fcbfb6b-b52rm 1/1 Running 0 4m6s mongodb-0 1/1 Running 0 4m6s mongodb-1 1/1 Running 0 3m48s mongodb-arbiter-0 1/1 Running 0 4m6s nrf-5b49c74c7-g7f7f 1/1 Running 0 4m6s nssf-57d6dbc7f8-42ch4 1/1 Running 0 4m6s pcf-dd8b976d4-wwqgm 1/1 Running 0 4m5s simapp-6d7dc8875c-gkvxk 1/1 Running 0 4m6s smf-6476786686-6ptjr 1/1 Running 0 4m6s udm-864ffdf49b-x4gcj 1/1 Running 0 4m6s udr-dc5bf7f5b-xqkvc 1/1 Running 0 4m6s upf-0 5/5 Running 0 4m6s webui-6dc76b5f85-6c65j 1/1 Running 0 4m6s $
You should see all pods in Running status.
If you have installed the ROC, check that all its pods are running as follows:
$ kubectl -n aether-roc get pod NAME READY STATUS RESTARTS AGE aether-roc-api-78cc548bb9-7vjs2 1/1 Running 0 4m16s aether-roc-gui-v2-6d674fd446-tttb5 1/1 Running 0 4m16s aether-roc-umbrella-grafana-74f8489c8f-s9p45 2/2 Running 0 4m16s aether-roc-websocket-855d64549b-44fnc 1/1 Running 0 4m16s onos-cli-5d448ff6c4-stq5t 1/1 Running 0 4m16s onos-config-7f4df96b88-vtp5s 6/6 Running 0 4m16s onos-consensus-store-0 1/1 Running 0 4m15s onos-topo-585c7c8976-6jq7b 3/3 Running 0 4m16s sdcore-adapter-v2-5646d455b9-2d6zl 1/1 Running 0 4m15s
You should see all pods in Running status.
The UPF pod connects to the
DATA_IFACE specified above using macvlan networks called
access. Next, check that these have been successfully created, e.g. using
$ ifconfig core core: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.250.1 netmask 255.255.255.0 broadcast 192.168.250.255 ether 16:9d:c1:0f:19:3a txqueuelen 1000 (Ethernet) RX packets 513797 bytes 48400525 (48.4 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 102996 bytes 26530538 (26.5 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 $ ifconfig access access: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.252.1 netmask 255.255.255.0 broadcast 192.168.252.255 ether 7a:9f:38:c0:18:15 txqueuelen 1000 (Ethernet) RX packets 558162 bytes 64064410 (64.0 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 99553 bytes 16646682 (16.6 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Understanding AiaB networking
Why does AiaB create the
access interfaces? These are necessary to enable
the UPF to exchange packets with the eNodeB (access) and Internet (core); they correspond to
the last two network interfaces below inside the UPF’s bessd container:
$ kubectl -n omec exec -ti upf-0 bessd -- ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 8a:e2:64:10:4e:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.84.19/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::88e2:64ff:fe10:4ebe/64 scope link valid_lft forever preferred_lft forever 4: access@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 82:b4:ea:00:50:3e brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.252.3/24 brd 192.168.252.255 scope global access valid_lft forever preferred_lft forever inet6 fe80::80b4:eaff:fe00:503e/64 scope link valid_lft forever preferred_lft forever 5: core@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default link/ether 4e:ac:69:31:a3:88 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.250.3/24 brd 192.168.250.255 scope global core valid_lft forever preferred_lft forever inet6 fe80::4cac:69ff:fe31:a388/64 scope link valid_lft forever preferred_lft forever
In other words, there are interfaces named
core both inside and outside the UPF. All four
are MACVLAN interfaces
bridged with DATA_IFACE. There are two subnets on this bridge: the two
access interfaces are on 192.168.252.0/24
and the two
core interfaces are on 192.168.250.0/24. It is helpful to think of two links, called
core, connecting the AiaB host and UPF. AiaB sets up IP routes on the AiaB host and inside the UPF
to forward packets into and out of the UPF as explained below.
access interface inside the UPF has an IP address of 192.168.252.3; this is the destination IP address
of GTP-encapsulated data plane packets from the gNodeB. In order for these packets to actually find their way
to the UPF, they must arrive on the DATA_IFACE interface and then be forwarded on the
outside the UPF.
The next section describes how to configure a static route on the gNodeB in order to send the GTP packets to
DATA_IFACE. Forwarding the packets to the
access interface is done by the following kernel route on the
AiaB host (which should be present if your AiaB installation was successful):
$ route -n | grep "Iface\|access" Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.252.0 0.0.0.0 255.255.255.0 U 0 0 0 access
The high-level behavior of the UPF is to forward packets between its
core interfaces, while
at the same time removing/adding GTP encapsulation on the
access side. Upstream packets
arriving on the
access side from a UE have their GTP headers removed and the raw IP packets are
forwarded to the
core interface. The routes inside the UPF’s bessd container will look something
$ kubectl -n omec exec -ti upf-0 -c bessd -- ip route default via 169.254.1.1 dev eth0 default via 192.168.250.1 dev core metric 110 184.108.40.206/22 via 192.168.252.1 dev access 220.127.116.11 via 169.254.1.1 dev eth0 169.254.1.1 dev eth0 scope link 192.168.250.0/24 dev core proto kernel scope link src 192.168.250.3 192.168.252.0/24 dev access proto kernel scope link src 192.168.252.3
The default route via 192.168.250.1 is directing upstream packets to the Internet via the
with a next hop of the
core interface outside the UPF.
These packets undergo source NAT in the kernel (also configured by AiaB) and are sent to the IP destination
in the packet. The return (downstream) packets undergo reverse NAT and now have a destination IP address of the UE.
They are forwarded by the kernel to the
core interface by these rules on the AiaB host:
$ route -n | grep "Iface\|core" Destination Gateway Genmask Flags Metric Ref Use Iface 18.104.22.168 192.168.250.3 255.255.0.0 UG 0 0 0 core 192.168.250.0 0.0.0.0 255.255.255.0 U 0 0 0 core
The first rule above matches packets to the UEs (on 22.214.171.124/16 subnet). The next hop for these
packets is the
core IP address inside the UPF. The second rule says that next hop address is
reachable on the
core interface outside the UPF. As a result the downstream packets arrive in the
UPF where they
are GTP-encapsulated with the IP address of the gNodeB. Inside the UPF these packets will match a route
like this one (see above; 126.96.36.199/22 in this case is the DATA_IFACE subnet):
188.8.131.52/22 via 192.168.252.1 dev access
These packets are forwarded to the
access interface outside the UPF and out DATA_IFACE to the eNodeB.
Recall that AiaB assumes that the eNodeB is on the same subnet as DATA_IFACE, so in this case it also has an
IP address in the 184.108.40.206/22 range.
Note that If you are not finding
core interfaces on outside the UPF, following commands
can be used to create these two interfaces manually:
$ ip link add core link <DATA_IFACE> type macvlan mode bridge 192.168.250.3 $ ip link add access link <DATA_IFACE> type macvlan mode bridge 192.168.252.3
We expect external gNodeB configuration is carried out separately.
Connect the gNodeB LAN port to a free Ethernet port on a Linux machine (say, a laptop) that will be used for the initial configuration of the gNodeB.
Test connectivity from gNodeB to the UPF. If required add a static route to the UPF address (192.168.252.3)
Test connectivity from the gNodeB to the AMF
If connectivity results are success, then you are ready to try to connect devices to the network.
Documenting how to configure different types of devices to work with Aether is work-in-progress, but here are some basic guidelines.
Create SIM cards by following the instructions for your SIM card writer. Of course you are free to use any values for IMSI, etc. that you choose, but these are the values that will work with the rest of the configuration in this document:
IMSI: each one is unique, matching pattern
If you choose different values for your SIM cards, you will need to modify subsequent configuration steps appropriately.
Insert the SIM cards in devices that you wish to be able to connect to the Aether network.
The values of IMSI, OPc, and Transport Key you have configured on your SIM cards
must be entered into the
subscribers block under
omec-sub-provision in the
sd-core-4g-values.yaml file. If you are not using the ROC, the IMSIs must also be
device-groups, and the relevant device group added under
If you are using the ROC, then your devices must be configured there and the associated
device group added to a slice. In either case it is necessary to configure the basic info
Be aware that not all phones support the CBRS frequency bands. AiaB is known to work with recent iPhones (11 and greater) and Google Pixel phones (4 and up). CBRS may also be supported by recent phones from Samsung, LG Electronics and Motorola Mobility, but these have not been tested with AiaB. If you successfully test a phone on AiaB, please post details on Slack so we can add it to the list.
The APN to configure on your phone is
To easily monitor your AiaB deployment, you can run the following command to deploy the Prometheus/Grafana monitoring stack on your AiaB server:
After this step, Grafana is available at http://<server-ip>:30950. You will see a number of system dashboards for monitoring Kubernetes, as well as a simple AiaB_5g dashboard that enables inspection of the local Aether state.
The dashboard shows whether the gNodeB and UPF are connected to the core, how many active UEs there are, including UE’s IMSI and Slice information, and the uplink (Tx Bitrate) and downlink (Rx Bitrate) throughput at the UPF. SMF and AMF service metrics work is in progress and shall be available soon.
To create a new dashboard for Aether, you can login to Grafana as user admin with password prom-operator. You can also modify an existing dashboard by making a copy of the dashboard and editing that. Note that any dashboard changes will be lost if Grafana is restarted; if you want to make your dashboard more permanent, you can save its JSON to a file, add the file to directory aether-in-a-box/resources/5g-monitoring, and edit kustomization.yaml in that directory to add the file to the ConfigMapGenerator block. After these steps, re-running make monitoring-5g should install the dashboard in a ConfigMap and Grafana should automatically load it.
Please refer Aether Troubleshooting Guide