System Testing

The Aether system is tested using Robot Framework. The tests are located inside the aether-system-tests repository and they are run nightly using a Jenkins job.

Test Structure

At a high level, functional tests are ran by:

Configuring ROC with various slices required for tests using pre-defined input data.
ROC updates using pre-defined input data with patch updates built with the test framework.
Control plane and data plane validated using NG40.

5G

Functional testing includes multiple slice creations, enable/disable of device groups, QoS validations, rate limiting tests (at UE, slice, application), application filtering tests, container restart tests.

4G

Functional testing includes multiple slice creations, enable/disable of device groups, QoS validations, rate limiting tests (at UE, slice, application), application filtering tests, container restart tests.

Development Prerequisites

Deploying each component is done with the use of Helm. Instructions can be found on this page.

The following steps are needed to deploy the Aether system:

If applicable, remove ROC, CRDs, and other resources. Fetch Kubernetes config.
Deploy SD-Core components
Deploy SD-Fabric components
Deploy ROC
Wait for pods to be up
Sync with SD-Core adapter

Running Tests

System tests are executed in the following Jenkins views:

Automated Test Jobs (Nightly)

Tests are executed in the following order after deployment:

Functional
Failure/Restart
Cleanup

Test Setup

# Set up virtual environment
cd aether-system-tests/
make ast-venv
source ast-venv/bin/activate; set -u;

Running Tests (4G)

Functional, Failure/Restart, and Cleanup are executed similarly. Replace ${TEST_FILE} with the test file to execute. This runs all tests in the suite. Add -i ${TEST_TAG} to robot arguments to run a specific test case.

cd ${WORKSPACE_DIR}
robot -d ${WORKSPACE_DIR}/${TEST_TYPE}/robotlogs \
    --debugfile ${WORKSPACE_DIR}/functionality/robotlogs/${TEST_FILE}_debug \
    -o ${log_xml} \
    -l ${log_html} \
    -r ${report_html} \
    -e notready \
    -v ACC_CONTEXT:${accContext} \
    -v AMP_CONTEXT:${rocContext} \
    -v ACE_CONTEXT:${aceContext} \
    -v UPF_NAMESPACE:${upfNamespace} \
    -v UPF_NAMESPACE2:${upfNamespace2} \
    -v UPF_NAMESPACE3:${upfNamespace3} \
    -v PFCP_NAMESPACE:${pfcpNamespace} \
    -v CORE_TYPE:${coreType} \
    -v UPF_TYPE:${upfType} \
    -v UPF_TYPE_NAME:${upfTypeName}\
    -v NG40_HOST:${ng40Host} \
    -v NG40_USER:${ng40User} \
    -v NG40RAN_DIR:${ng40RanDir} \
    -v NG40TEST_DIR:${ng40TestDir} \
    -v CSV_DIR:${ng40CsvDir} \
    -v PCAP_DIR:${WORKSPACE_DIR}/functionality/${pcapDir} \
    -v NG40_LOG_DIR:${WORKSPACE_DIR}/functionality/${ng40LogDir} \
    -v CDR_LOG_DIR:${WORKSPACE_DIR}/functionality/${cdrLogDir} \
    -v CONTAINER_LOG_DIR:${WORKSPACE_DIR}/functionality/${containerLogDir} \
    -v POD_NAME:${pod} \
    aether-system-tests/system-tests/tests/4G/functional/${TEST_FILE}.robot || true

Variables

${WORKSPACE_DIR} - Current workspace directory for Aether System Tests. Used for output logs and PCAPs.
${TEST_TYPE} - Directories for logs for each test type Ex: functional, failure, cleanup.
${accContext} - SD-Core context.
${rocContext} - ROC context.
${aceContext} - SD-Fabric and UPF context.
${upfNamespace} - UPF namespace used in tests. Ex: aether-sdcore-upf1.
${pfcpNamespace} - PFCP namespace used in tests. Ex: tost.
${coreType} - 4G or 5G.
${upfType} - UPF type used in tests. Ex: bess. Can also be ${None} or empty.
${upfTypeName} - UPF type name used in tests.
This affects which input files are used. Ex: p4.
Can also be ${None} or empty.
${ng40Host} - IP address for NG40 host.
${ng40User} - NG40 User.
${ng40RanDir} - NG40. RAN directory.
${ng40CsvDir} - NG40 CSV directory.
${ng40TestDir} - NG40 tests directory.
${pcapDir} - Saved PCAP directory.
${cdrLogDir} - NG40 CDR directory.
${containerLogDir} - Log directory for each deployed component.
${pod} - Pod name used in tests. This affects which input files are used. Ex: qa, qa2.

Running Tests (5G)

5G tests are executed similar to 4G, except using the tests located in the 5G directory.

cd ${WORKSPACE_DIR}
robot -d ${WORKSPACE_DIR}/${TEST_TYPE}/robotlogs \
    --debugfile ${WORKSPACE_DIR}/functionality/robotlogs/${TEST_FILE}_debug \
    -o ${log_xml} \
    -l ${log_html} \
    -r ${report_html} \
    -e notready \
    -v ACC_CONTEXT:${accContext} \
    -v AMP_CONTEXT:${rocContext} \
    -v ACE_CONTEXT:${aceContext} \
    -v UPF_NAMESPACE:${upfNamespace} \
    -v UPF_NAMESPACE2:${upfNamespace2} \
    -v UPF_NAMESPACE3:${upfNamespace3} \
    -v PFCP_NAMESPACE:${pfcpNamespace} \
    -v CORE_TYPE:${coreType} \
    -v UPF_TYPE:${upfType} \
    -v UPF_TYPE_NAME:${upfTypeName}\
    -v NG40_HOST:${ng40Host} \
    -v NG40_USER:${ng40User} \
    -v NG40RAN_DIR:${ng40RanDir} \
    -v NG40TEST_DIR:${ng40TestDir} \
    -v CSV_DIR:${ng40CsvDir} \
    -v PCAP_DIR:${WORKSPACE_DIR}/functionality/${pcapDir} \
    -v NG40_LOG_DIR:${WORKSPACE_DIR}/functionality/${ng40LogDir} \
    -v CDR_LOG_DIR:${WORKSPACE_DIR}/functionality/${cdrLogDir} \
    -v CONTAINER_LOG_DIR:${WORKSPACE_DIR}/functionality/${containerLogDir} \
    -v POD_NAME:${pod} \
    aether-system-tests/system-tests/tests/5G/functional/${TEST_FILE}.robot || true