Merge pull request #281 from NTIA/calibrate_to_antenna

Sensor failure recovery, support calibration to the antenna, save onboard cals separately, cal on startup

.github/workflows/github-actions-test.yml

@@ -75,8 +75,8 @@ jobs:
           source ./env.template
           export MOCK_SIGAN=1
           export MOCK_SIGAN_RANDOM=1
-          docker-compose build --no-cache
-          docker-compose up -d
+          docker compose build --no-cache
+          docker compose up -d
 
       - name: Wait for containers # wait for containers to finish starting
         run: sleep 45

.pre-commit-config.yaml

@@ -19,24 +19,24 @@ repos:
       - id: end-of-file-fixer
       - id: trailing-whitespace
   - repo: https://github.com/asottile/pyupgrade
-    rev: v3.15.0
+    rev: v3.15.2
     hooks:
       - id: pyupgrade
         args: ["--py38-plus"]
   - repo: https://github.com/pycqa/isort
-    rev: 5.12.0
+    rev: 5.13.2
     hooks:
       - id: isort
         name: isort (python)
         types: [file, python]
         args: ["--profile", "black", "--filter-files", "--gitignore"]
   - repo: https://github.com/psf/black
-    rev: 23.10.1
+    rev: 24.3.0
     hooks:
       - id: black
         types: [file, python]
   - repo: https://github.com/igorshubovych/markdownlint-cli
-    rev: v0.37.0
+    rev: v0.39.0
     hooks:
       - id: markdownlint
         types: [file, markdown]

README.md

@@ -292,7 +292,9 @@ scos-sensor website from the same computer as where it is hosted.
 When running in a production environment or on a remote system, various settings will
 need to be configured.
 
-## docker-compose.yml
+### Compose File
+
+This section details configuration which takes place in `compose.yaml`:
 
 - shm_size: This setting is overriding the default setting of 64 mb. If using
   scos-sensor on a computer with lower memory, this may need to be decreased. This is
@@ -321,6 +323,9 @@ settings in the environment file:
   By default, this is configured to
   use a version of `ghcr.io/ntia/scos-tekrsa/tekrsa_usb` to use a Tektronix
   signal analyzer.
+- CALIBRATION_EXPIRATION_LIMIT: Number of seconds elapsed for a calibration result to
+  become expired. On startup, if existing calibration is expired, the action defined by
+  STARTUP_CALIBRATION_ACTION will be run to generate new calibration data.
 - CALLBACK_AUTHENTICATION: Sets how to authenticate to the callback URL. Supports
   `TOKEN` or `CERT`.
 - CALLBACK_SSL_VERIFICATION: Set to “true” in production environment. If false, the SSL
@@ -353,6 +358,7 @@ settings in the environment file:
   certificate.
 - POSTGRES_PASSWORD: Sets password for the Postgres database for the “postgres” user.
   Change in production. The env.template file sets to a randomly generated value.
+- RAY_INIT: If set to true, SCOS Sensor will ensure initializaiton of the Ray library.
 - REPO_ROOT: Root folder of the repository. Should be correctly set by default.
 - SCOS_SENSOR_GIT_TAG: The scos-sensor branch name. This value may be used in action
   metadata to capture the version of the software that produced the sigmf archive.
@@ -379,6 +385,8 @@ settings in the environment file:
   scos-sensor repository with a valid certificate in production.
 - SSL_KEY_PATH: Path to server SSL private key. Use the private key for your valid
   certificate in production.
+- STARTUP_CALIBRATION_ACTION: The name of an available action which will run on startup
+  if no unexpired calibration data is already present.
 - USB_DEVICE: Optional string used to search for available USB devices. By default,
   this is set to Tektronix to see if the Tektronix signal analyzer is available. If
   the specified value is not found in the output of lsusb, scos-sensor will attempt
@@ -420,95 +428,80 @@ specific to the sensor you are using.
 }
 ```
 
-### Sensor Calibration File
+### Calibration Files
 
-By default, scos-sensor will use `configs/default_calibration.json` as the sensor
-calibration file. However, if`configs/sensor_calibration.json` or
-`configs/sigan_calibration.json` exist they will be used instead of the default
-calibration file. Sensor calibration files allow scos-sensor to apply a gain based
-on a laboratory calibration of the sensor and may also contain other useful
-metadata that characterizes the sensor performance. For additional
-information on the calibration data, see the
-[NTIA-Sensor SigMF Calibration Object](https://github.com/NTIA/sigmf-ns-ntia/blob/master/ntia-sensor.sigmf-ext.md#08-the-calibration-object).
-The default calibration file is shown below:
+Calibration files allow SCOS Sensor to scale data based on a laboratory and/or
+in-field calibration of the sensor, and may also contain other useful metadata that
+characterizes the sensor performance. Two primary types of calibration files are used:
+sensor calibration files and differential calibration files.
 
-```json
-{
-  "calibration_data":{
-    "datetime": "1970-01-01T00:00:00.000000Z",
-    "gain": 0,
-    "noise_figure": null,
-    "1db_compression_point": null,
-    "enbw": null,
-    "temperature": 26.85
-  },
-  "last_calibration_datetime": "1970-01-01T00:00:00.000000Z",
-  "calibration_parameters": [],
-  "clock_rate_lookup_by_sample_rate": [
-  ],
-  "sensor_uid": "DEFAULT CALIBRATION",
-  "calibration_reference": "noise source output"
-}
-```
+Sensor calibration files may be provided upon sensor deployment or generated by onboard
+calibration actions. If both exist, onboard calibration data takes priority. The
+sensor will first attempt to load `configs/onboard_sensor_calibration.json`, and fall
+back to `configs/sensor_calibration.json` if the first option fails. Next, SCOS determines
+whether the loaded calibration data is expired, based on the threshold set by the
+CALIBRATION_EXPIRATION_LIMIT threshold. If calibration data is expired, the sensor will
+attempt to run the calibration action defined by STARTUP_CALIBRATION_ACTION.
+
+SCOS Sensor also supports an additional calibration file, called a differential
+calibration. The differential calibration is used to provide additional scaling factors
+to shift the reference point of data from the onboard calibration terminal to elsewhere
+in the signal path. For instance, a differential calibration file can be provided with
+scaling factors to shift the data reference point from the onboard calibration terminal
+to the antenna port. The differential calibration is loaded separately, and used in
+addition to, either the onboard or lab-provided sensor calibration file.
+
+#### Calibration File Contents
+
+In sensor calibration files, the unit of calibration data is defined by the
+[NTIA-Sensor SigMF Calibration Object](https://github.com/NTIA/sigmf-ns-ntia/blob/master/ntia-sensor.sigmf-ext.md#08-the-calibration-object).
+In differential calibration files, the only key in the calibration data is `loss`.
 
 The `calibration_parameters` key lists the parameters that will be used to obtain
-the calibration data. In the case of the default calibration, there are no
-`calibration_parameters` so the calibration data is found directly within the
-`calibration_data` element and by default scos-sensor will not apply any additional
-gain. Typically, a sensor would be calibrated at particular
-sensing parameters. The calibration data for specific parameters should be listed
-within the calibration_data object and accessed by the values of the settings
-listed in the calibration_parameters element. For example, the calibration below
-provides an example of a sensor calibrated at a sample rate of 14000000.0 samples
-per second at several frequencies with a signal analyzer reference level setting of -25.
+the calibration data. In the case of onboard calibration being generated from scratch,
+the startup calibration action's signal analyzer settings are used as the `calibration_parameters`.
+The calibration data is found directly within the `calibration_data` element and by
+default SCOS Sensor will not apply any additional gain. Typically, a sensor would be
+calibrated at particular sensing parameters. The calibration data for specific parameters
+should be listed within the `calibration_data` object and accessed by the values of the
+settings listed in the calibration_parameters element. For example, the sensor
+calibration below provides an example of a sensor calibrated at a sample rate of
+140000000 samples per second at several frequencies with a signal analyzer reference
+level setting of -25.
 
 ```json
 {
   "last_calibration_datetime": "2023-10-23T14:39:13.682Z",
   "calibration_parameters": [
     "sample_rate",
     "frequency",
-    "reference_level",
-    "preamp_enable",
-    "attenuation"
+    "reference_level"
   ],
   "clock_rate_lookup_by_sample_rate": [],
   "calibration_data": {
     "14000000.0": {
       "3545000000.0": {
         "-25": {
-          "true": {
-            "0": {
-              "datetime": "2023-10-23T14:38:02.882Z",
-              "gain": 30.09194805857024,
-              "noise_figure": 4.741521295220736,
-              "temperature": 15.6
-            }
-          }
+          "datetime": "2023-10-23T14:38:02.882Z",
+          "gain": 30.09194805857024,
+          "noise_figure": 4.741521295220736,
+          "temperature": 15.6
         }
       },
       "3555000000.0": {
         "-25": {
-          "true": {
-            "0": {
-              "datetime": "2023-10-23T14:38:08.022Z",
-              "gain": 30.401008416406599,
-              "noise_figure": 4.394893979804061,
-              "temperature": 15.6
-            }
-          }
+          "datetime": "2023-10-23T14:38:08.022Z",
+          "gain": 30.401008416406599,
+          "noise_figure": 4.394893979804061,
+          "temperature": 15.6
         }
       },
       "3565000000.0": {
         "-25": {
-          "true": {
-            "0": {
-              "datetime": "2023-10-23T14:38:11.922Z",
-              "gain": 30.848049817892105,
-              "noise_figure": 4.0751785215495819,
-              "temperature": 15.6
-            }
-          }
+          "datetime": "2023-10-23T14:38:11.922Z",
+          "gain": 30.848049817892105,
+          "noise_figure": 4.0751785215495819,
+          "temperature": 15.6
         }
       }
     }
@@ -518,7 +511,7 @@ per second at several frequencies with a signal analyzer reference level setting
 
 When an action is run with the above calibration, SCOS will expect the action to have
 a sample_rate, frequency, and reference_level specified in the action config. The values
-specified for these parameters will then be used to retrieve the calibration data.
+specified for these parameters will then be used to retrieve the calibration data entry.
 
 ## Security
 
@@ -844,7 +837,7 @@ for additional information. Be sure to re-source the environment file, update th
 requirements files, and prune any existing containers
 before rebuilding scos-sensor.
 
-## Preselector Support
+### Preselector Support
 
 Scos-sensor can be configured to support
 [preselectors](http://www.github.com/ntia/Preselector).
@@ -858,7 +851,7 @@ in docker-compose.yaml to the python module that contains the
 preselector implementation you specify in PRESELECTOR_CLASS in
 docker-compose.yaml.
 
-## Relay Support
+### Relay Support
 
 Scos-sensor can be configured with zero or more [network controlled relays](https://www.controlbyweb.com/webrelay/).
 The default relay configuration directory is configs/switches.

docker-compose.yml → compose.yaml

@@ -1,5 +1,3 @@
-version: '3'
-
 services:
   db:
     image: postgres:15-alpine
@@ -45,6 +43,7 @@ services:
       - ADDITIONAL_USER_NAMES
       - ADDITIONAL_USER_PASSWORD
       - AUTHENTICATION
+      - CALIBRATION_EXPIRATION_LIMIT
       - CALLBACK_AUTHENTICATION
       - CALLBACK_SSL_VERIFICATION
       - CALLBACK_TIMEOUT
@@ -70,6 +69,8 @@ services:
       - SIGAN_CLASS
       - SIGAN_POWER_SWITCH
       - SIGAN_POWER_CYCLE_STATES
+      - STARTUP_CALIBRATION_ACTION
+      - RAY_INIT
       - RUNNING_MIGRATIONS
       - USB_DEVICE
     expose:

env.template

@@ -18,8 +18,7 @@ ADMIN_PASSWORD=password
 
 # set to CERT to enable scos-sensor certificate authentication
 AUTHENTICATION=TOKEN
-
-BASE_IMAGE=ghcr.io/ntia/scos-tekrsa/tekrsa_usb:0.2.3
+CALIBRATION_EXPIRATION_LIMIT=360
 # Default callback api/results
 # Set to CERT for certificate authentication
 CALLBACK_AUTHENTICATION=TOKEN
@@ -34,8 +33,6 @@ CALLBACK_TIMEOUT=2
 # Use either true or false
 DEBUG=true
 
-DEVICE_MODEL=RSA507A
-
 # Use latest as default for local development
 DOCKER_TAG=latest
 
@@ -75,8 +72,12 @@ SCOS_SENSOR_GIT_TAG="$(git describe --tags)"
 # SECURITY WARNING: generate unique key with `manage.py generate_secret_key`
 SECRET_KEY="$(python3 -c 'import secrets; print(secrets.token_urlsafe(64))')"
 
+# Signal analyzer selection/setup
 SIGAN_CLASS=TekRSASigan
 SIGAN_MODULE=scos_tekrsa.hardware.tekrsa_sigan
+USB_DEVICE=Tektronix
+DEVICE_MODEL=RSA507A
+BASE_IMAGE=ghcr.io/ntia/scos-tekrsa/tekrsa_usb:latest
 
 # SECURITY WARNING: You should be using certs from a trusted authority.
 #                   If you don't have any, try letsencrypt or a similar service.
@@ -86,8 +87,11 @@ SSL_CA_PATH=scos_test_ca.crt
 SSL_CERT_PATH=sensor01.pem
 SSL_KEY_PATH=sensor01.pem
 
-USB_DEVICE=Tektronix
-
+# Calibration action selection
+#    The action specified here will be used to attempt an onboard
+#    sensor calibration on startup, if no onboard calibration data
+#    is available on startup. The specified action must be available.
+STARTUP_CALIBRATION_ACTION=SEA_CBRS_Calibrate_Baseline
 
 # Debug dependant settings
 if $DEBUG; then

gunicorn/config.py

@@ -2,7 +2,6 @@
 import sys
 from multiprocessing import cpu_count
 
-
 bind = ":8000"
 workers = 1
 worker_class = "gthread"

src/handlers/apps.py

@@ -5,7 +5,7 @@
 from scos_actions.signals import (
     location_action_completed,
     measurement_action_completed,
-    trigger_api_restart
+    trigger_api_restart,
 )
 
 logger = logging.getLogger(__name__)
@@ -20,7 +20,6 @@ def ready(self):
             db_location_deleted,
             db_location_updated,
             location_action_completed_callback,
-
         )
         from handlers.measurement_handler import measurement_action_completed_callback
 
@@ -40,5 +39,3 @@ def ready(self):
 
         trigger_api_restart.connect(trigger_api_restart_callback)
         logger.debug("trigger_api_restart_callback registered to trigger_api_restart")
-
-