Store/retrieve device calibration data #169
Comments
|
For what it's worth I do not think this should be handled within a data acquisition system and instead should be handled by a dedicated metadata tool. I see where it would be a slick feature for sensors that are handled by OpenRVDAS, however not all data systems on a research vessel are run through OpenRVDAS. My personal preference would be to develop a stand-alone sensor metadata tracking system that could be used to manage metadata for all sensor systems on a vessel. I've actually been thinking recently that this metadata tool might be the next project OpenOceanTools takes on. |
|
It sounds reasonable to me to have an external system manage metadata. But
I think OpenRVDAS components need to be able to retrieve/parse/work with
it. Would be interesting to consider whether simply having a
"MetadataReader" component that could retrieve things like calibration
values (to be used, e.g., in a DerivedDataTransform) would be sufficient.
…On Mon, Oct 21, 2019 at 11:36 AM Webb Pinner ***@***.***> wrote:
For what it's worth I do not think this should be handled within a data
acquisition system and instead should be handled by a dedicated metadata
tool. I see where it would be a slick feature for sensors that are handled
by OpenRVDAS, however not all data systems on a research vessel are run
through OpenRVDAS.
My personal preference would be to develop a stand-alone sensor metadata
tracking system that could be used to manage metadata for all sensor
systems on a vessel. I've actually been thinking recently that this
metadata tool might be the next project OpenOceanTools takes on.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#169?email_source=notifications&email_token=AFO7V3SKJOCYPNV3GHNQVWTQPYAE5A5CNFSM4JC7BXBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB3Q3TI#issuecomment-544673229>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFO7V3SHYAYBYQACLC533VLQPYAE5ANCNFSM4JC7BXBA>
.
|
|
Been thinking about this a bit more. I still believe a DAS should just deal with reading and writing data. Transforms for things like prepending timestamps and true-wind calculations are a necessary complication but I believe anything beyond that should be handled by an external mechanism. All that said, I am now in favor of having a transform that can apply a user-defined lambda function to a data stream. This would allow an operator to perform static corrections (i.e. adding the distance from a depth sensor to the water line for proper depth-from-surface) and unit conversions (knots <--> kph, C <--> F, etc). This type of transform could also be used to apply a calibration curve if the operator so desired. I can't take credit for this idea. An older version of the DAS used on WHOI ships included this functionality. I think SCS can do this as well. My apprehension about getting into automated corrections based on calibration data comes from several bad experiences and was recently reinforced by a incident where a piece of mid-water equipment made contact with the seafloor because the wrong correction equation was being unknowingly applied in real-time to a depth sensor. With regards to metadata tracking... I hope you can talk with Chris Romsos about work he's done in this area that I just uncovered yesterday. |
|
Yup - this kind of conversion is what the DerivedDataTransform is all
about. Of course, OpenRVDAS should (always) be storing the raw values that
come out of the sensors; I just want to make sure it has access to the
various calibration factors that let it turn those raw values into numbers
that are meaningful to the scientists and crew on board.
…On Tue, Oct 22, 2019 at 8:38 AM Webb Pinner ***@***.***> wrote:
Been thinking about this a bit more. I still believe a DAS should just
deal with reading and writing data. Transforms for things like prepending
timestamps and true-wind calculations are a necessary complication but I
believe anything beyond that should be handled by an external mechanism.
All that said, I am now in favor of having a transform that can apply a
user-defined lambda function to a data stream. This would allow an operator
to perform static corrections (i.e. adding the distance from a depth sensor
to the water line for proper depth-from-surface) and unit conversions
(knots <--> kph, C <--> F, etc). This type of transform could also be used
to apply a calibration curve if the operator so desired. I can't take
credit for this idea. An older version of the DAS used on WHOI ships
included this functionality. I think SCS can do this as well.
My apprehension about getting into automated corrections based on
calibration data comes from several bad experiences and was recently
reinforced by a incident where a piece of mid-water equipment made contact
with the seafloor because the wrong correction equation was being
unknowingly applied in real-time to a depth sensor.
With regards to metadata tracking... I hope you can talk with Chris Romsos
about work he's done in this area that I just uncovered yesterday.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#169?email_source=notifications&email_token=AFO7V3QVEDQLHVNF3O5EBILQP4UBJA5CNFSM4JC7BXBKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEB6MX2I#issuecomment-545049577>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFO7V3WDT2CHLSUGFPMNFZDQP4UBJANCNFSM4JC7BXBA>
.
|
|
Yes.... I see Webb's point, but I consider it apples and oranges. Both tasty fruit, but different. Let's take a common example: A transmissometer. You've got calibration factors you copy from the vendor's calibration sheet. And WAY YES! It would be cool to have a tool where I could upload that and it would get indexed with make/model/serial-number/date, when I installed it, etc. Especially if it had a API I could query. But to get the actual beam transissance (is that the right word?) I need those factors, plus ideally in-situ dark and air measurements (which are data points sent to the DAS via the sea-water wall data stream so that the values used are in the raw data somewhere). The meta-data system wouldn't necessarily know about those since they'll change every time we clean the instrument and sample the air/dark values, and let's face it, tech's get busy and don't always have time to futz with metadata. So yes... a function using (basically) the derived data transform is great. I think it might be hard to figure out how to enter that into the existing config structure though. I'd almost suggest a separate config file and a special-case thing like the true-winds transform that reads it. That way you could describe the metadata, the equation(s), the required data from the DAS, and the data you're providing TO the das. Since such a transform might apply to more than one logger stream, you maybe can't just it in the chain for one instrument. I've actually been working on something like that, and can't wait for Pablo to show up so I can have him tell me I'm doing it wrong and show me the OpenRVDAS way. :-) |
|
Sounds good - looking forward to seeing what you've got in mind.
…On Tue, Jul 20, 2021, 4:26 PM LMG-ET ***@***.***> wrote:
Yes.... I see Webb's point, but I consider it apples and oranges. Both
tasty fruit, but different. Let's take a common example: A transmissometer.
You've got calibration factors you copy from the vendor's calibration
sheet. And WAY YES! It would be cool to have a tool where I could upload
that and it would get indexed with make/model/serial-number/date, when I
installed it, etc. Especially if it had a API I could query. But to get the
actual beam transissance (is that the right word?) I need those factors,
plus ideally in-situ dark and air measurements (which are data points sent
to the DAS via the sea-water wall data stream so that the values used are
in the raw data somewhere). The meta-data system wouldn't necessarily know
about those since they'll change every time we clean the instrument and
sample the air/dark values, and let's face it, tech's get busy and don't
always have time to futz with metadata.
So yes... a function using (basically) the derived data transform is
great. I think it might be hard to figure out how to enter that into the
existing config structure though. I'd almost suggest a separate config file
and a special-case thing like the true-winds transform that reads it. That
way you could describe the metadata, the equation(s), the required data
from the DAS, and the data you're providing TO the das. Since such a
transform might apply to more than one logger stream, you maybe can't just
it in the chain for one instrument.
I've actually been working on something like that, and can't wait for
Pablo to show up so I can have him tell me I'm doing it wrong and show me
the OpenRVDAS way. :-)
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#169 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFO7V3XH2EYU25B4XXFUROLTYXLZDANCNFSM4JC7BXBA>
.
|
We need to be able to store/retrieve/use device calibration data:
Transforms should have access to these metadata so that they can perform the adjustment (and pass the metadata of how the adjustment was made along with the new, adjusted values).
There should be mechanisms for associating those data with data streams, such as outputting as header in a log file or including in JSON or DASRecord messages.
The text was updated successfully, but these errors were encountered: