.. vim: syntax=rst Aether-in-a-Box with External 5G Radio ======================================= This document describes how to set up an Aether-in-a-Box (AiaB) with a external gNodeB and connect real devices (e.g., 5G phones). It assumes that you are already familiar with running AiaB with emulated gNodeB/UE. AiaB on Hardware Radios is suitable for laboratory experiments and proof-of-concept deployments. Its goals are to provide an easy-to-install environment where Aether's features can be explored with real devices. To create this setup you will need the following equipment: * Server for running AiaB (SD-CORE / UPF / ROC) * Ubuntu 18.04 or 20.04 clean install * Haswell CPU family or newer * At least 4 CPUs and 12GB RAM * Internet connection * Ability to run "sudo" without a password * No firewall running on the AiaB host * External 5G small cell gNodeB or simulator * If real phones are used then you need SIM card writer and blank SIM cards **IMPORTANT**: AiaB is for simple deployment scenarios and so makes some simplifying assumptions: * AiaB assumes either 4G or 5G SD-CORE is deployed. Running both 4G and 5G SD-CORE simultaneously in AiaB is currently not supported. However, running both 4G and 5G SD-CORE simultaneously is supported by Aether. * Performance and scalability are not goals. AiaB does not support I/O acceleration (e.g., with SR-IOV). However, performance and scalability are goals of the Aether project. * AiaB assumes that the server and the gNodeB are connected to the **same LAN** and share the **same IP subnet**; in other words they can reach each other in a single hop and there is no IP router between them. This simplifies communication between the gNodeB and the UPF, which is running inside a container and has a private IP address that is not necessarily routable on the local network. However, this is not a requirement for all Aether deployments. * AiaB also assumes that the AiaB server's network is configured using *systemd-networkd*, which is the default for Ubuntu, and copies some files into `/etc/systemd/network`; the reason for this is to enable persistence of AiaB's networking configuration across server reboots. This configuration method is specific to AiaB. Preparation ----------- Server setup ------------ The server will run Aether-in-a-Box. The gNodeB will connect to the server over the local network. Perform these steps to prepare the server for the AiaB install: * Connect the server to the local network * Perform a clean install of Ubuntu 18.04 or Ubuntu 20.04 on the server * Verify that systemd-networkd is being used to configure networking (e.g., run ``systemctl status systemd-networkd.service``) * Set up password-less sudo for the user that will install Aether-in-a-Box After the steps above have been completed, install Aether-in-a-Box as follows:: sudo apt install git make git clone "https://gerrit.opencord.org/aether-in-a-box" cd aether-in-a-box Next, modify the file *sd-core-5g-values.yaml*. Under ``subscribers``, add an IMSI range for the SIM cards you created, with the Transport Key and OPc values you used earlier. For example, the following will add IMSIs between 315010999912301 and 315010999912303:: subscribers: - ueId-start: "315010999912301" ueId-end: "315010999912303" plmnId: "315010" opc: "69d5c2eb2e2e624750541d3bbc692ba5" key: "000102030405060708090a0b0c0d0e0f" sequenceNumber: 135 Determine which is the interface that has L3 connectivity to the gNodeB -- this will be ``DATA_IFACE`` in the configuration later. If the gNodeB will also be connected to the local network, then this is just the server's primary interface. If the gNodeB will be connected via an isolated L2/L3 network segment, then ``DATA_IFACE`` refers to the server interface on that network. Remember this interface for later. Option 1: Configure Aether with ROC ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Aether ROC provides a GUI and API for dynamically configuring Aether. If you don't wish to use the ROC to configure AiaB, you can skip to the next section. .. note:: Aether Monitoring is available only when Aether is deployed using ROC. Monitoring support for 5G is still in progress. Install AiaB as follows (specifying ``DATA_IFACE`` from above):: ENABLE_GNBSIM=false DATA_IFACE= CHARTS=latest make roc-5g-models 5g-core Next, use the ROC to add information about your SIM cards. The ROC GUI is available at `http://:31194`. Choose ``Configuration > Site`` from the drop-down at top right and edit the ``AiaB site``. Change the following values and click ``Update``: * MCC: 315 * MNC: 010 Choose ``Sim Cards`` from the drop-down at top right. Edit the existing entries to reflect the SIM cards you are adding to devices by replacing their IMSI values. Click ``Update`` after each edit. If you want to connect more than two devices, consult the :ref:`ROC documentation `. Finally, click the Basket icon at top right and click the ``Commit`` button. Now jump to the `Verifying the AiaB installation`_ section. Option 2: Configure Aether without ROC ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ It is possible to configure Aether without the ROC, using static YAML files and the SimApp service. If you have already installed the ROC, you should skip this section. Edit *sd-core-5g-values.yaml*. Change ``mcc`` and ``mnc`` as follows:: plmn: mcc: "315" mnc: "010" Also add the IMSIs of your devices under ``imsis``, for example:: device-groups: - name: "5g-gnbsim-user" imsis: - "315010999912301" - "315010999912302" - "315010999912303" Install AiaB as follows (specifying ``DATA_IFACE`` from above):: ENABLE_GNBSIM=false DATA_IFACE= CHARTS=latest make 5g-core Verifying the AiaB installation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Installing AiaB will take about 20 minutes with a fast Internet connection. If you see any errors / timeouts, try running the ``make`` command again. The build will finish with a message: “Your MME IP address is… ” This is just the IP address assigned to the ``DATA_IFACE``. Remember this for the gNodeB setup. When the install is complete, check that the 5G SD-CORE is running as follows:: $ kubectl get pods -n omec NAME READY STATUS RESTARTS AGE amf-6d9d8f44c8-2d7f7 1/1 Running 0 4m6s ausf-9fcbfb6b-b52rm 1/1 Running 0 4m6s mongodb-0 1/1 Running 0 4m6s mongodb-1 1/1 Running 0 3m48s mongodb-arbiter-0 1/1 Running 0 4m6s nrf-5b49c74c7-g7f7f 1/1 Running 0 4m6s nssf-57d6dbc7f8-42ch4 1/1 Running 0 4m6s pcf-dd8b976d4-wwqgm 1/1 Running 0 4m5s simapp-6d7dc8875c-gkvxk 1/1 Running 0 4m6s smf-6476786686-6ptjr 1/1 Running 0 4m6s udm-864ffdf49b-x4gcj 1/1 Running 0 4m6s udr-dc5bf7f5b-xqkvc 1/1 Running 0 4m6s upf-0 5/5 Running 0 4m6s webui-6dc76b5f85-6c65j 1/1 Running 0 4m6s $ You should see all pods in Running status. If you have installed the ROC, check that all its pods are running as follows:: $ kubectl -n aether-roc get pod NAME READY STATUS RESTARTS AGE aether-roc-api-78cc548bb9-7vjs2 1/1 Running 0 4m16s aether-roc-gui-v2-6d674fd446-tttb5 1/1 Running 0 4m16s aether-roc-umbrella-grafana-74f8489c8f-s9p45 2/2 Running 0 4m16s aether-roc-websocket-855d64549b-44fnc 1/1 Running 0 4m16s onos-cli-5d448ff6c4-stq5t 1/1 Running 0 4m16s onos-config-7f4df96b88-vtp5s 6/6 Running 0 4m16s onos-consensus-store-0 1/1 Running 0 4m15s onos-topo-585c7c8976-6jq7b 3/3 Running 0 4m16s sdcore-adapter-v2-5646d455b9-2d6zl 1/1 Running 0 4m15s You should see all pods in Running status. The UPF pod connects to the ``DATA_IFACE`` specified above using macvlan networks called ``core`` and ``access``. Next, check that these have been successfully created, e.g. using ``ifconfig``:: $ ifconfig core core: flags=4163 mtu 1500 inet 192.168.250.1 netmask 255.255.255.0 broadcast 192.168.250.255 ether 16:9d:c1:0f:19:3a txqueuelen 1000 (Ethernet) RX packets 513797 bytes 48400525 (48.4 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 102996 bytes 26530538 (26.5 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 $ ifconfig access access: flags=4163 mtu 1500 inet 192.168.252.1 netmask 255.255.255.0 broadcast 192.168.252.255 ether 7a:9f:38:c0:18:15 txqueuelen 1000 (Ethernet) RX packets 558162 bytes 64064410 (64.0 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 99553 bytes 16646682 (16.6 MB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 Understanding AiaB networking ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Why does AiaB create the ``core`` and ``access`` interfaces? These are necessary to enable the UPF to exchange packets with the eNodeB (access) and Internet (core); they correspond to the last two network interfaces below inside the UPF's `bessd` container:: $ kubectl -n omec exec -ti upf-0 bessd -- ip addr 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if30: mtu 1450 qdisc noqueue state UP group default link/ether 8a:e2:64:10:4e:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.84.19/32 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::88e2:64ff:fe10:4ebe/64 scope link valid_lft forever preferred_lft forever 4: access@if2: mtu 1500 qdisc noqueue state UP group default link/ether 82:b4:ea:00:50:3e brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.252.3/24 brd 192.168.252.255 scope global access valid_lft forever preferred_lft forever inet6 fe80::80b4:eaff:fe00:503e/64 scope link valid_lft forever preferred_lft forever 5: core@if2: mtu 1500 qdisc noqueue state UP group default link/ether 4e:ac:69:31:a3:88 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 192.168.250.3/24 brd 192.168.250.255 scope global core valid_lft forever preferred_lft forever inet6 fe80::4cac:69ff:fe31:a388/64 scope link valid_lft forever preferred_lft forever In other words, there are interfaces named ``access`` and ``core`` **both inside and outside** the UPF. All four are MACVLAN interfaces bridged with DATA_IFACE. There are two subnets on this bridge: the two ``access`` interfaces are on 192.168.252.0/24 and the two ``core`` interfaces are on 192.168.250.0/24. It is helpful to think of two links, called ``access`` and ``core``, connecting the AiaB host and UPF. AiaB sets up IP routes on the AiaB host and inside the UPF to forward packets into and out of the UPF as explained below. The ``access`` interface **inside the UPF** has an IP address of 192.168.252.3; this is the destination IP address of GTP-encapsulated data plane packets from the gNodeB. In order for these packets to actually find their way to the UPF, they must arrive on the DATA_IFACE interface and then be forwarded on the ``access`` interface **outside the UPF**. The next section describes how to configure a static route on the gNodeB in order to send the GTP packets to DATA_IFACE. Forwarding the packets to the ``access`` interface is done by the following kernel route on the AiaB host (which should be present if your AiaB installation was successful):: $ route -n | grep "Iface\|access" Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.252.0 0.0.0.0 255.255.255.0 U 0 0 0 access The high-level behavior of the UPF is to forward packets between its ``access`` to ``core`` interfaces, while at the same time removing/adding GTP encapsulation on the ``access`` side. Upstream packets arriving on the ``access`` side from a UE have their GTP headers removed and the raw IP packets are forwarded to the ``core`` interface. The routes inside the UPF's `bessd` container will look something like this:: $ kubectl -n omec exec -ti upf-0 -c bessd -- ip route default via 169.254.1.1 dev eth0 default via 192.168.250.1 dev core metric 110 128.105.144.0/22 via 192.168.252.1 dev access 128.105.145.141 via 169.254.1.1 dev eth0 169.254.1.1 dev eth0 scope link 192.168.250.0/24 dev core proto kernel scope link src 192.168.250.3 192.168.252.0/24 dev access proto kernel scope link src 192.168.252.3 The default route via 192.168.250.1 is directing upstream packets to the Internet via the ``core`` interface, with a next hop of the ``core`` interface **outside the UPF**. These packets undergo source NAT in the kernel (also configured by AiaB) and are sent to the IP destination in the packet. The return (downstream) packets undergo reverse NAT and now have a destination IP address of the UE. They are forwarded by the kernel to the ``core`` interface by these rules on the AiaB host:: $ route -n | grep "Iface\|core" Destination Gateway Genmask Flags Metric Ref Use Iface 172.250.0.0 192.168.250.3 255.255.0.0 UG 0 0 0 core 192.168.250.0 0.0.0.0 255.255.255.0 U 0 0 0 core The first rule above matches packets to the UEs (on 172.250.0.0/16 subnet). The next hop for these packets is the ``core`` IP address **inside the UPF**. The second rule says that next hop address is reachable on the ``core`` interface **outside the UPF**. As a result the downstream packets arrive in the UPF where they are GTP-encapsulated with the IP address of the gNodeB. Inside the UPF these packets will match a route like this one (see above; 128.105.144.0/22 in this case is the DATA_IFACE subnet):: 128.105.144.0/22 via 192.168.252.1 dev access These packets are forwarded to the ``access`` interface **outside the UPF** and out DATA_IFACE to the eNodeB. Recall that AiaB assumes that the eNodeB is on the same subnet as DATA_IFACE, so in this case it also has an IP address in the 128.105.144.0/22 range. Note that If you are not finding ``access`` and ``core`` interfaces on **outside the UPF**, following commands can be used to create these two interfaces manually:: $ ip link add core link type macvlan mode bridge 192.168.250.3 $ ip link add access link type macvlan mode bridge 192.168.252.3 gNodeB setup ------------ We expect external gNodeB configuration is carried out separately. Connect the gNodeB LAN port to a free Ethernet port on a Linux machine (say, a laptop) that will be used for the initial configuration of the gNodeB. - Test connectivity from gNodeB to the UPF. If required add a static route to the UPF address (192.168.252.3) - Test connectivity from the gNodeB to the AMF If connectivity results are success, then you are ready to try to connect devices to the network. Connecting Devices ------------------ Documenting how to configure different types of devices to work with Aether is work-in-progress, but here are some basic guidelines. Create SIM cards by following the instructions for your SIM card writer. Of course you are free to use any values for IMSI, etc. that you choose, but these are the values that will work with the rest of the configuration in this document: * IMSI: each one is unique, matching pattern ``315010*********`` (15 digits) * OPc: ``69d5c2eb2e2e624750541d3bbc692ba5`` * Transport Key: ``000102030405060708090a0b0c0d0e0f`` If you choose different values for your SIM cards, you will need to modify subsequent configuration steps appropriately. Insert the SIM cards in devices that you wish to be able to connect to the Aether network. The values of IMSI, OPc, and Transport Key you have configured on your SIM cards must be entered into the ``subscribers`` block under ``omec-sub-provision`` in the ``sd-core-4g-values.yaml`` file. If you are not using the ROC, the IMSIs must also be added under ``device-groups``, and the relevant device group added under ``network-slices``. If you are using the ROC, then your devices must be configured there and the associated device group added to a slice. In either case it is necessary to configure the basic info under ``subscribers``. Be aware that not all phones support the CBRS frequency bands. AiaB is known to work with recent iPhones (11 and greater) and Google Pixel phones (4 and up). CBRS may also be supported by recent phones from Samsung, LG Electronics and Motorola Mobility, but these have not been tested with AiaB. If you successfully test a phone on AiaB, please post details on Slack so we can add it to the list. The APN to configure on your phone is ``internet``. Enable Monitoring ----------------- To easily monitor your AiaB deployment, you can run the following command to deploy the Prometheus/Grafana monitoring stack on your AiaB server:: make monitoring-5g After this step, Grafana is available at `http://:30950`. You will see a number of system dashboards for monitoring Kubernetes, as well as a simple AiaB_5g dashboard that enables inspection of the local Aether state. .. figure:: images/5g-aiab-grafana-dashboard.png :align: center :width: 100 % *5G Grafana AiaB Dashboard* The dashboard shows whether the gNodeB and UPF are connected to the core, how many active UEs there are, including UE's IMSI and Slice information, and the uplink (Tx Bitrate) and downlink (Rx Bitrate) throughput at the UPF. SMF and AMF service metrics work is in progress and shall be available soon. To create a new dashboard for Aether, you can login to Grafana as user `admin` with password `prom-operator`. You can also modify an existing dashboard by making a copy of the dashboard and editing that. Note that any dashboard changes will be lost if Grafana is restarted; if you want to make your dashboard more permanent, you can save its JSON to a file, add the file to directory `aether-in-a-box/resources/5g-monitoring`, and edit `kustomization.yaml` in that directory to add the file to the ConfigMapGenerator block. After these steps, re-running `make monitoring-5g` should install the dashboard in a ConfigMap and Grafana should automatically load it. Troubleshooting --------------- Please refer :ref:`Aether Troubleshooting Guide`