Skip to content

Code organization and installation

Jump to bottom
Daniel Suzuki edited this page May 3, 2022 · 33 revisions

Reference

If you use any data or code, please cite: "5G MIMO Data for Machine Learning: Application to Beam-Selection using Deep Learning", Aldebaro Klautau, Pedro Batista, Nuria Gonzalez-Prelcic, Yuyang Wang and Robert W. Heath Jr, ITA'2018 (available at http://ita.ucsd.edu/workshop/18/files/paper/paper_3313.pdf).

Available data

If just looking for the data, jump to datasets.

Machine learning

If just looking for the machine learning codes, jump to Beam selection.

Basic information

The motivation is having signals and channels used in telecommunications to perform machine learning (ML). The adopted process is as follows: Python code repeatedly invokes both a traffic simulator (SUMO) and a ray-tracing simulator (Remcom Wireless InSite) to obtain raw data, for example, representing MIMO channels. The simulations with SUMO and InSite compose the first stage of the process, as depicted in the block diagram below. In the second stage, the raw data is organized into a 5GMdata database (we use SQLAlchemy - https://www.sqlalchemy.org with SQLite) according to the target application, which can be channel estimation, beam training, etc. A third (we call it post-processing) stage is to convert the 5GMdata into a format suitable for machine learning. The fourth stage regards running the ML experiment using, e.g., deep learning with Keras.

5GM data generation process

We provide code for executing all 4 stages, and also some of the generated datasets to promote reproducible research. The user may simply use the datasets, skipping all stages but the fourth or using his/her own ML code. The provided data can be in 5GMdata format (output of stage 2) or in a format convenient for Python (numPy arrays in a .npz file) and Matlab (hdf5 files), as the output of stage 3. The information provided here concerns all stages but someone that wants to simply use a given dataset can find it here.

Summary of code organization

All related projects start with prefix 5gm at Github. The scripts that concern interaction with Remcom Wireless Insite (RWI) are identified by the acronym rwi and are used in stage 1. For example, the 5gm-rwi-parsing project has code to automatically parse the output of a ray-tracing simulation. Note the name of the project at Github differs from the Python package name. For example, the project 5gm-rwi-parsing has the code for the Python package rwiparsing. In case you need to modify a software in one of these packages, note that the main scripts are:

  1. simulation.py => located in the 5gm-rwi-simulation Github project (you should clone 5gm-rwi-simulation), it repeatedly calls the traffic simulator SUMO and ray-tracing tool InSite. Before using it, make sure you edit the config.py and inform the correct path for key files

  2. todb.py => also located in the 5gm-data Github project, it converts from raw data to a database (organizes 5GMdata as a SQLAlchemy database)

  3. collectFixedChannels.py => Located in the 5gm-data Github project (you should clone 5gm-rwi-simulation), it performs post-processing of the data, generating .HDF5 and .NPZ files for Fixed simulations

  4. convert5gmv1ToChannels.py => Located in the 5gm-data Github project (you should clone 5gm-data), it performs post-processing of the data, generating .HDF5 and .NPZ files for Mobile simulations

  5. convert5gmv1ForBeamSelectionInPython.py => located in the 5gm-data Github project (you should clone 5gm-data), it performs post-processing of the data, creating data structures that can be used for machine learning. Modifying this code, other data structures and signal processing can be obtained

  6. deep_ann_classifier.py => located in the 5gm-beam-selection project, which should be cloned, it is an example of application that uses data organized as a npz (numPy arrays) binary file.

  7. deep_convnet_regression.py => located in the 5gm-beam-selection project, which should be cloned, it is an example of application that uses data organized as a npz (numPy arrays) binary file.

Getting started: Installation

Linux and Python3 is assumed.

Quick and dirty installation

# If you have a Python virtual environment use workon to enable it
# Install all used packages
pip install Shapely sqlalchemy h5py tensorflow scipy numpy cheetah3 matplotlib
# Choose a folder and cd to that folder, which is assumed here to be /home/myname
cd /home/myname
git clone https://github.com/lasseufpa/5gm-rwi-simulation
cd 5gm-rwi-simulation
python3 setup.py install
cd ..
git clone https://github.com/lasseufpa/5gm-data
cd 5gm-data
python3 setup.py install

After a successful installation, the folders 5gm-data and 5gm-rwi-simulation have been created at /home/myname and four Python packages installed. For example, if you use pip freeze | grep rwi, you should find:

rwimodeling==1.0.3
rwiparsing==1.1
rwisimulation==1.2.1

(the version numbers may differ). The fourth installed package is 5gmdata.

Detailed installation procedure and associated information

This assumes the steps required for all 4 stages in the process, including running the ray-tracing and traffic simulators. Basically, one needs to clone the Github projects: 5gm-rwi-simulation (for stage 1 and 2), 5gm-data (for stage 3) and an application such as 5gm-beam-selection, which is an example of stage 4. As mentioned, one may simply use the provided datasets.

All the configuration was tested on OS X 10.13.2 and ArchLinux with Kernel 4. But if the dependencies are installed on Windows it should also work. The easiest way on Windows is to have Windows 10 and install the "Linux subsystem" that Microsoft provides (instructions at https://docs.microsoft.com/en-us/windows/wsl/install-win10). In this case, to have access to files in both systems (Windows and its Linux subsystem), when creating files on "Linux", use the Windows file system (for example: /mnt/c or /mnt/d on Linux for C:\ and D:\ on Windows, respectively).

It is assumed that Python 3 is used. You can see if Python is installed issuing the command python or python3. You will also need pip to install software (or pip3, which can be installed with sudo apt install python3-pip for example).

It is advised to run the commands bellow inside a Python virtualenv. See virtualenvwrapper install for installation. If you will not be using virtualenvwrapper you can skip the next commands session.

# create a virtualenv to work on
mkvirtualenv -p python3 insite
# if you close the terminal, when restarting later on, run the command below to get back into the virtualenv
workon insite

Download and install the project 5gm-rwi-simulation

If you are not going to run your own simulations (you will skip stage 1) and use 5GMdata already available, you can skip the steps below and simply install the project 5gm-data as described in the next subsection.

To execute stage 1 and create a database running simulations, clone the following project: 5gm-rwi-simulation. The following commands can be issued:

# Choose a folder and cd to that folder, which is assumed here to be /home/myname
cd /home/myname
git clone https://github.com/lasseufpa/5gm-rwi-simulation
cd 5gm-rwi-simulation

After retrieving the code, to properly install 5gm-rwi-simulation and its dependencies in your computer, issue the following command:

python3 setup.py install

The setup.py script will automatically install 5gm-rwi-simulation mandatory dependencies (such as 5gm-rwi-parsing) in your virtual environment (e.g., at folder /home/myname/.environments/insite/lib/python3.5/site-packages/). See additional details in the cheatsheet in the bottom of this page.

(Obs: in case you are using both Windows 10 and Linux (subsystem), you need to issue the installation commands on the system that you will use to execute the code.)

Download and install the project 5gm-data

Clone the following project: 5gm-data and install the package:

# Choose a folder and cd to that folder, which is assumed here to be /home/myname
cd /home/myname
git clone https://github.com/lasseufpa/5gm-data
cd 5gm-data
python3 setup.py install

Other packages and softwares

Some users may want to simply invoke InSite and, in this case, the following dependencies are not needed. But for a complete install, that enables you to run all scripts, issue:

pip3 install Shapely sqlalchemy h5py tensorflow scipy numpy cheetah3 matplotlib

For flexibility on dealing with mobility of users (cars, etc.), the open source SUMO is needed. Its installation instructions are provided at http://www.sumo.dlr.de/userdoc/Installing.html In summary, the commands are:

# for Ubuntu
sudo add-apt-repository ppa:sumo/stable
sudo apt-get update
sudo apt-get install sumo sumo-tools sumo-doc

To check the installation, you can try to invoke sumo from command line (on Windows, the executable is sumo.exe and sumo on Linux). For other command line applications in Sumo's package, see http://www.sumo.dlr.de/userdoc/Basics/Using_the_Command_Line_Applications.html

It is assumed that InSite for Linux is properly installed and its command line tools are in your system PATH. InSite GUI runs only on Windows but we use its tools wibatch and calcprop from a Linux command line.

After Installation: Running Simulation

Stage 1: Running the ray-tracing and traffic simulators

To run the simulation, a config file is needed, an example is provided in the 5gm-rwi-simulation's "example" folder. Take a look at it and configure for using your paths. Use your favorite (ASCII) text editor (notepad, vi, etc.) to edit config.py. Make sure the wibach_bin variable in config.py matches your InSite paths and similar things. Once you configure the paths and how you want to run the simulation, execute rwi-simulation with the following (or similar) commands:

cd 5gm-rwi-simulation/example
# assuming you are inside the `example` directory, set PYTHONPATH so `config.py` can be found
# Find your SUMO's tools folder and also add it to PYTHONPATH
PYTHONPATH=$(pwd)

The command above update the PYTHONPATH environment variable such that the current folder (obtained with the pwd command)

Edit your bashrc file, vim is assumed:

vim ~/.bashrc

Add this lines in the end of file:

export SUMO_HOME=/usr/share/sumo
export PYTHONPATH=$PYTHONPATH:/usr/share/sumo/tools

To run a new simulation we need edit the config.py file and check if the variable base_insite_project_path has the right path for the "insite base" folder. And the sumo_cfg variable has the right path for "sumo base" folder. after this run the command:

python3 -m rwisimulation.simulation -po
python3 -m rwisimulation.simulation -rj

Stage 2: Organizing raw data into a 5GMdata database

Mobile Simulations The results of the ray-tracing & SUMO simulations can be stored in an SQLite database (episodedata.db). move the simulation files to the folder of the project 5gm-data and use the command:

cd 5gm-data
python3 todb.py rt_results

In file convert5gmv1ToChannels.py, make sure you edit the variables "numScenesPerEpisode" and "fileNamePrefix", and use the command

python3 convert5gmv1ToChannels.py

Fixed Simulations

Move the simulation files to the folder of the project 5gm-rwi-simulation and use the command:

python3 collectFixedChannels.py rt_results

will generate files in format convenient for Python (numPy arrays in a .npz file) and Matlab (hdf5 files)

V2V Simulations

Move the simulation files from the folder of the project 5gm-rwi-simulation, make sure to change the variables numScenesPerEpisode, numTx, numRx in collectChannels_from_V2V.py then use the command:

python3 collectChannels_from_V2V.py rt_results output

This will generate files in format convenient for Python (numPy arrays in a .npz file) and Matlab (hdf5 files)

To easily interpret this files, access: Raymobtime

Stage 3: Converting the database into a file suitable to machine learning packages such as Keras

The database can then be used to further process the data, and generate a binary NumPy file for example. The script convert5gmv1ForBeamSelectionInPython.py is an example of exhaustively reading almost all the data in the database and saving it to a binary file. It works as a reference for accessing the data. Another useful description of the database is obtained by reading save5gmdata.py.

git clone https://github.com/lasseufpa/5gm-data.git
cd 5gm-data
python3 convert5gmv1ForBeamSelectionInPython.py

You may need to edit convert5gmv1ForBeamSelectionInPython.py in order to inform where the input 5GMdata database previously obtained is located.

Stage 4: Running the code of a machine learning application

Access: Beam Selection

Cheatsheet

Update all projects

cd 5gm-rwi-simulation/
git pull
python3 setup.py install
pip3 install --upgrade git+https://github.com/lasseufpa/5gm-rwi-parsing@master#egg=rwiparsing
pip3 install --upgrade git+https://github.com/lasseufpa/5gm-rwi-3d-modeling@master#egg=rwimodeling

See what packages are installed

When installing 5gm-data, mandatory dependencies are automatically installed in your system. To see them, one can type pip freeze. If there are many files, use grep to filter. For example, to see all rwi packages: pip freeze | grep rwi.

To see where a specific package was installed pip show:

Example:

$ pip show rwisimulation
Name: rwisimulation
Version: 1.2.1
Summary: Run simulations in Remcom Wireless InSite
Home-page: https://gitlab.lasse.ufpa.br/software/python-machine-learning/rwi-simulation
Author: LASSE
Author-email: pedosb@gmail.com
License: UNKNOWN
Location: /home/aldebaro/.environments/insitev2/lib/python3.5/site-packages/rwisimulation-1.2.1-py3.5.egg
Requires: numpy, rwiparsing, rwimodeling

Do not modify code or update a file that is part of an "automatically" installed dependency:

The suggestion is that you do not directly edit a dependency file. As an example, as a dependency of 5gm-data, the files in the 5gm-rwi-parsing project may have been automatically installed in a "system" folder. If you want to modify code from 5gm-rwi-parsing, instead of finding the file in your system / dependencies folder and modifying it, you should obtain the specific project source code from Github (clone 5gm-rwi-parsing), modify it and then install the new version with:

git clone https://gitlab.lasse.ufpa.br/software/python-machine-learning/rwi-parsing
# do your changes and then install with your modifications
cd rwi-parsing
python setup.py install

Note the procedure above is different from simply installing a package as a dependency. It is the correct procedure to edit / modify a file within a package. Bottom line: a dependency must not be edited directly on the dependencies folder.

Remove a package to make sure you are using the right version:

Hint from Pedro Batista: Sometimes if I think that a file/package is not being properly overwritten, I remove it and then install it again. To uninstall I use pip show to see where it is installed (see "Location:") and remove the folder.

See inside Python from where a given Python module is being imported:

import inspect
import rwiparsing
inspect.getsourcefile(rwiparsing)
Clone this wiki locally