David Pablo Cohn; DRAFT: 2019-08-10
OpenRVDAS is a yet-nascent Open Source Research Vessel Data Acquisition System. A few links for context:
- Project website: http://openrvdas.org
- Code home: http://github.com/davidpablocohn/openrvdas
For a general introduction to intent and design philosophy of the system, see thetop-level README.md for the code, and/or the 2018 INMARTECH Presentation. This document focuses on USAP/NBP/LMG-specific issues.
Overview
OpenRVDAS has a set of installation scripts in the utils/ directory, including one designed to install onto a clean CentOS7 machine. USAP machines running RHEL7 use almost identical code, but tend to have additional security measures that have to be compensated for. The current recommended approach for USAP RHEL machines is to use the utils/build_openrvdas_centos7.sh script (as root, as described in the code base's INSTALL.md document), with a few extra commands before and after the script.
At present, we have things working on RHEL 7.6 Server. RHEL Workstation seems to install properly, but network writes (e.g. UDP) fail silently and mysteriously.
Pre-Installation
On the NBP, AD "realms" are causing havoc; user rvdas exists and interferes with all sorts of things. It also has default group 'rvdas_group' (instead of 'rvdas'), which breaks the script. To get around this, before fetching/running the script, do:
sudo useradd rvdas # (assuming your user is called rvdas)
sudo groupadd rvdas
sudo usermod -G rvdas rvdas
sudo gpasswd -a rvdas wheel
The default configuration files all try to log to /data/logger. You'll want to make sure that directory exists and is writeable by user rvdas:
sudo mkdir /data
sudo mkdir /data/logger
sudo chown rvdas /data/logger
sudo chmod –R 755 /data
Installation
Grab the script:
# Set the proxy depending on which ship we're on
SHIP=nbp
#SHIP=lmg
export http_proxy=proxy.${SHIP}.usap.gov:3128
export https_proxy=proxy.${SHIP}.usap.gov:3128
# Get build script from Github
wget \
https://raw.githubusercontent.com/davidpablocohn/openrvdas/master/utils/build\_openrvdas_centos7.sh
Now run the script as sudo
chmod 755 ./build_openrvdas_centos7.sh
sudo ./build_openrvdas_centos7.sh
The script will ask a lot of questions and provide default answers in parens that will be filled in if you hit "return"; without any other input:
############################################################################
OpenRVDAS configuration script
Do you wish to continue? **y**
Name to assign to host (lmg-dast-s1-t)?
Hostname will be 'lmg-dast-s1-t'
Install root? (/opt)
Install root will be '/opt'
Script will next ask which code repo and branch to use. Use the default repo, but specify branch "usap"
Repository to install from? (http://github.com/davidpablocohn/openrvdas)
Repository branch to install? (master) **usap**
HTTP/HTTPS proxy to use (http://proxy.lmg.usap.gov:3128)?
Setting up proxy http://proxy.lmg.usap.gov:3128
Will install from github.com
Repository: 'http://github.com/davidpablocohn/openrvdas'
Branch: 'usap'
Script will try to create the rvdas user under which the system will run. It won't mind if the user already exists:
OpenRVDAS user to create? (rvdas)
Checking if user rvdas exists yet
User exists, skipping
Script will next ask about database options. If this is the first time you've run the script and MariaDB has not already been installed, the current root database password will be empty:
Database password to use for rvdas? (rvdas)
Current database password for root? (if one exists - hit return if not)
New database password for root? () **rvdas**
############################################################################
The OpenRVDAS server can be configured to start on boot. Otherwise
you will need to either run it manually from a terminal (by running
server/logger_manager.py from the openrvdas base directory) or
start as a service (service openrvdas start).
Do you wish to start the OpenRVDAS server on boot? **y**
The script will run a while and ask at the end if you want to reboot. If you need to do the post-installation step(s) below, say "no", perform those steps, then reboot. Otherwise, say yes, then log in afterwards and check /var/log/openrvdas/openrvdas.log to make sure things have come up properly.
Post-Installation
The installation should allow you to connect via http to the server at the name you specified at the start of the script (e.g. lmg-dast-s1-t). If you want to connect using any other names, e.g. the fully-qualified domain name lmg-dast-s1-t.lmg.usap.gov, you'll need to add it to the Django server settings file in django_gui/settings.py:
ALLOWED_HOSTS = [HOSTNAME, 'localhost', HOSTNAME + 'lmg.usap.gov']
If you didn't reboot immediately after installation, reboot now.
Running on live data
Assuming your machine has MOXA ports connected, you should be able to run on live data.
Point your browser at port 8000 of your server to get at the control console (e.g.: http://lmg-dast-s1-t:8000). Scroll to the bottom of the page and log in as rvdas (or whatever user you create), then hit the "Load configuration file" button.
What configuration you should load depends on which ship and machine you're on. The configurations depend by ship (LMG vs NBP) in which ports and loggers they have defined. On a given ship, different machines have different IP addresses, and these must be encoded in the configurations.
The sample configurations available at the moment are:
NBP
- nbp-dast-02-t : local/usap/nbp/nbp_cruise.yaml
LMG
- openrvdas : local/usap/lmg/lmg_cruise.yaml Pre-production test server. Config has basic loggers, plus "timeout loggers" that inform the cached data server if a logger hasn't produced any data in N seconds. Also write timeout information to /var/log/openrvdas/timeouts.log
- lmg-dast-s1-t : local/usap/lmg/lmg_s1_cruise.yaml Development server. As above, basic loggers, plus "timeout loggers" that inform the cached data server if a logger hasn't produced any data in N seconds. Also write timeout information to /var/log/openrvdas/timeouts.log
Note that each of these configurations has some network stuff specific to the machine in question (specifically the machine's UDP broadcast address) and may not work on your newly-built machine. To create a configuration for your machine, see "Creating your own cruise definitions," below, then come back to this point in the document.
Whichever machine you're working on, you'll need to scp the appropriate file to your local machine so that the browser's file picker can load it. There's a feature request to replace the current file loader with a server-side file loader, which will both simplify loading and close the gaping security hole of allowing an authenticated user to run an arbitrary config which could, if they wanted, read/write/delete any file on the server that user rvdas had access to.
Once you have identified a cruise definition to use, it may also be worth opening another terminal, logging in as rvdas, going to /opt/openrvdas and starting a listener on port 6224 to monitor the raw logger output:
logger/listener/listen.py –-udp 6224
Opening a browser window at
http://lmg-dast-s1-t:8000/static/widgets/lmg_basic.html
should get you to a rudimentary display widget. Returning to the control page and setting the s330, mwx1 and true_winds loggers to their "->net" configurations should bring the display widget to life.
Our limited experimentation suggests that this will produce a fully-functional RHEL7 installation of the system, from which you can proceed with the various introductory documents (OpenRVDAS Introduction and Overview, the Introduction to Loggers and Controlling Loggers).
USAP/LMG/NBP-specific Code
By convention, program- and ship-specific code will be stored under the top-level local/ directory. It is likely that this code will only be visible in the usap-specific branch, which is called (surprisingly) "usap".
Creating your own cruise definitions
At the moment, the YAML cruise definitions are almost unreadably verbose. Eventually we want to have a concise way to view and graphically construct them. At the moment, they are constructed via one of a few scripts in the local/usap/ hierarchy.
local/usap/create_cruise.py \
local/usap/lmg/lmg_port_defs-2019-08-07.yaml \
> local/usap/nbp/nbp_cruise_2019-08-07.yaml
WARNING: the scripts are ugly. Very. Don't judge me. They work, for now.
The idea is that the "port defs" file encodes machine-specific variables, such as the machine's broadcast address, what ports should be used for raw/parsed record broadcast, and which instruments are connected to which serial ports. So given the right "port defs" file, you can use the same script for an NBP or LMG test setup or live data. Have a look at the port def files in:
test/LMG1903/LMG1903_port_defs.yaml
test/NBP1406/NBP1406_port_defs.yaml
local/usap/lmg/lmg_port_defs-2019-08-07.yaml
local/usap/nbp/nbp_port_defs_2019-08-07.yaml
for examples.
Troubleshooting
Naturally, stuff won't work as hoped/expected. We do need some better diagnostic tools, but this section discusses Pablo's current strategies for debugging.
Nothing is happening
See if you can do run basic command line listeners. In one terminal:
cd /opt/openrvdas
listener.py --udp 6225
In another (157.132.133.255 is the broadcast address for the machine in question):
cd /opt/openrvdas
listen.py --file LICENSE --write_udp 157.132.133.255::6225
The first terminal should receive the sent copy of the LICENSE file. If this doesn't work, maybe ports/permissions are broken, and nothing else will work.
The server doesn't appear to be running
- Run systemctl and search for "openrvdas", "nginx" and "uwsgi" services to make sure they're all running . The uswgi service should say "loaded active exited":
loaded active running The nginx HTTP and reverse proxy server
loaded active running Run openrvdas/server/logger_manager.py as service
loaded active exited Run uWSGI as a daemon
If one of them, say openrvdas, has failed, run sudo journalctl -u openrvdas to see why it isn't running.
If that proves unenlightening, stop the service and try running the server manually:
sudo systemctl stop openrvdas
-
Cat the file openrvdas/scripts/start_openrvdas.sh and copy/paste its contents into a terminal window to see if you can watch things start up and die.
-
It may also be a problem with whatever cruise definition file you've loaded, or tried to load. Again, better on-screen diagnostics are needed, but you can force a load from the command line. Alter the final line of the startup script to include the argument "--config ". e.g.:
server/logger_manager.py --database django \
--no-console -v --stderr_file \
$OPENRVDAS_LOGFILE \
--data_server_websocket $DATA_SERVER_WEBSOCKET \
$DATA_SERVER_LISTEN_ON_UDP \
$START_DATA_SERVER \
--config local/usap/lmg/lmg_s1_cruise.yaml
If there's something wrong with the config file, you should get a front seat to the diagnostic dump. If you further alter the command line to remove the "--no-console" flag above, you can manipulate the logger_manager you've just started from the console. Type "help" to get the full list of commands, but you can load cruise definitions, set modes and set individual logger configs.
A logger is dying mysteriously
Newly-added functionality (as of 2019-08-12) makes this a little easier, but it's still a mess. Right now, the best way to debug this is to run the logger in isolation, using listen.py:
listen.py --config_file my_cruise_file:'my_logger_config_name' -v
e.g.
listen.py --config_file test/NBP1406/NBP1406_cruise.yaml:'gyr1->net' -v
And (hopefully) watch it die in a comprehensible and easy-to-fix way. You can add an extra -v to further increase logger verbosity.
Adding a new Cruise/Logger/Instrument
Adding a new cruise
[Make a copy of the current port defs file, changing the dates; run create_cruise.py or other similar script to create a new NBPXXXX_cruise.yaml file. Load and run]
Adding a new logger
[Mucking with create_cruise_definition.py or one of its variants]
Adding a new instrument
[Adding device_type definition to appropriate place in local/devices/, local/usap/devices, local/usap/[nbp,lmg]/devices, and adding device definition to local/usap/lmg/devices/lmg_devices.yaml. Then adding new logger, as above]
Note: when we screw up root mysql password: https://robbinespu.github.io/eng/2018/03/29/Reset_mariadb_root_password.html