Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Read returned a bad value while running dstools-create-model #18

Open
Adirana27 opened this issue Dec 18, 2024 · 2 comments
Open

Read returned a bad value while running dstools-create-model #18

Adirana27 opened this issue Dec 18, 2024 · 2 comments
Assignees
@joshoewahp
Labels
bug Something isn't working

Comments

@Adirana27
Copy link

Adirana27 commented Dec 18, 2024 •
edited
Loading

I am trying to create a model by running "dstools-create-model --imsize 10000 --cell 2.5 --nterms 2 --robust 0.5 --name SB61103.EMU_1227-69.beam00 scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage.ms". And Encountering the following error:

INFO 2024-12-18 20:38:52 - dstools.imaging - Imaging scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage.ms with 10000x10000 pixels, 2.5asec cellsize, and 2 Taylor terms.

  • An exception occured:

  • FiledesIO::read /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage.ms/table.mf - read returned a bad value

Am I missing something? Any leads would be much appreciated.
@joshoewahp
Copy link
Collaborator

Hi, thanks for raising this.

It looks like there is something not quite right with your MeasurementSet. I've previously seen this type of error when trying to open a MeasurementSet with casacore after running certain CASA tasks (e.g. uvsub), though I'm not 100% sure of the cause.

One very silly fix that has worked for me is to run mstransform without applying any actual transformations,

from casatasks import mstransform

mstransform(
    vis="scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage.ms",
    datacolumn="all",
    outputvis="scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage.fixed.ms",
)

and then try running dstools-create-model on the output of that,

dstools-create-model --imsize 10000 --cell 2.5 --nterms 2 --robust 0.5 --name SB61103.EMU_1227-69.beam00 scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage.fixed.ms

This seems to fix whatever corruption has occurred in the MS format by copying things over to a new MS, though is not a particularly satisfying solution.

Did you run any other processing on this data? If so that information would be helpful to debug the cause, as I can't reproduce this on the MS downloaded straight from CASDA, nor after correcting the flux scale / beam pointing with dstools-askap-preprocess.

BTW it looks like you're using the v2.0.0-dev branch of DStools, which is still in development (expected to release around week 1 of Jan). That shouldn't be causing this issue, but just a heads up in case you run into any other bugs that I've yet to iron out before release.

@joshoewahp joshoewahp added the bug Something isn't working label Dec 23, 2024
@joshoewahp joshoewahp self-assigned this Dec 23, 2024
@Adirana27
Copy link
Author

Hi Josh,

Thank you for looking into this—I appreciate your help.

I reran the dstools-create-model command on the output of mstransform, and it resolved the specific error I mentioned earlier. However, when running dstools-create-model on the corrected measurement set (MS), a new error occurred, which appears to be related to cfitsio.

For context, I also ran the dstools-askap-preprocess tool beforehand to correct the MS.

I’ve attached the full output below for your reference.

Command: dstools-create-model --imsize 10000 --cell 2.5 --nterms 2 --robust 0.5 --name SB61103.EMU_1227-69.beam00 --verbose scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms

Detailed output:

WSClean version 3.5 (2024-07-30)
This software package is released under the GPL version 3.
Author: André Offringa (offringa@gmail.com).

No corrected data in first measurement set: tasks will be applied on the data column.
=== IMAGING TABLE ===
# Pol Ch JG ²G FG FI In Freq(MHz)
| Independent group:
+-+-J- 0 I 0 0 0 0 0 0 799-835 (36)
| +-J- 4 I 1 0 4 4 0 0 835-871 (36)
| +-J- 8 I 2 0 8 8 0 0 871-907 (36)
| +-J-12 I 3 0 12 12 0 0 907-943 (36)
| +-J-16 I 4 0 16 16 0 0 943-979 (36)
| +-J-20 I 5 0 20 20 0 0 979-1015 (36)
| +-J-24 I 6 0 24 24 0 0 1015-1051 (36)
| +-J-28 I 7 0 28 28 0 0 1051-1087 (36)
| Independent group:
+-+-J- 1 Q 0 1 1 1 0 0 799-835 (36)
| +-J- 5 Q 1 1 5 5 0 0 835-871 (36)
| +-J- 9 Q 2 1 9 9 0 0 871-907 (36)
| +-J-13 Q 3 1 13 13 0 0 907-943 (36)
| +-J-17 Q 4 1 17 17 0 0 943-979 (36)
| +-J-21 Q 5 1 21 21 0 0 979-1015 (36)
| +-J-25 Q 6 1 25 25 0 0 1015-1051 (36)
| +-J-29 Q 7 1 29 29 0 0 1051-1087 (36)
| Independent group:
+-+-J- 2 U 0 2 2 2 0 0 799-835 (36)
| +-J- 6 U 1 2 6 6 0 0 835-871 (36)
| +-J-10 U 2 2 10 10 0 0 871-907 (36)
| +-J-14 U 3 2 14 14 0 0 907-943 (36)
| +-J-18 U 4 2 18 18 0 0 943-979 (36)
| +-J-22 U 5 2 22 22 0 0 979-1015 (36)
| +-J-26 U 6 2 26 26 0 0 1015-1051 (36)
| +-J-30 U 7 2 30 30 0 0 1051-1087 (36)
| Independent group:
+-+-J- 3 V 0 3 3 3 0 0 799-835 (36)
+-J- 7 V 1 3 7 7 0 0 835-871 (36)
+-J-11 V 2 3 11 11 0 0 871-907 (36)
+-J-15 V 3 3 15 15 0 0 907-943 (36)
+-J-19 V 4 3 19 19 0 0 943-979 (36)
+-J-23 V 5 3 23 23 0 0 979-1015 (36)
+-J-27 V 6 3 27 27 0 0 1015-1051 (36)
+-J-31 V 7 3 31 31 0 0 1051-1087 (36)
WARNING: This measurement set has no or an invalid WEIGHT_SPECTRUM column; will use less informative WEIGHT column.
Reordering /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms into 8 x 4 parts.
Reordering: 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
Initializing model visibilities: 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
Precalculating MF weights for Briggs'(0.5) weighting...
Opening reordered part 0 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 0 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 0 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 0 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 1 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 1 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 1 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 1 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 2 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 2 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 2 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 2 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 3 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 3 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 3 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 3 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 4 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 4 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 4 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 4 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 5 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 5 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 5 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 5 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 6 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 6 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 6 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 6 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 7 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 7 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 7 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Opening reordered part 7 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
== Constructing PSF ==
Detected 376.5 GB of system memory, usage not limited.
Opening reordered part 0 spw 0 for /fred/oz309/Fix_MS/SB61103/scienceData.EMU_1227-69.SB61103.EMU_1227-69.beam00_averaged_cal.leakage_fixed.ms
Determining min and max w & theoretical beam size... DONE (w=[0.0027168:14871.8] lambdas, maxuvw=17943.3 lambda)
Theoretic beam = 11.5''
Minimal inversion size: 5220 x 5220, using optimal: 5292 x 5292
Loading data in memory...
Gridding 2279340 rows...
Gridded visibility count: 70355705
Fitting beam... major=nan masec, minor=nan masec, PA=nan masec, theoretical=11.5''.
Writing psf image... An exception occurred, writing back will be skipped. Cleaning up...

**+ An exception occured:

  • CFITSIO reported error when performing IO on file 'SB61103.EMU_1227-69.beam00-0000-psf.fits':bad float to string conversion (Error in ffd2e: double value is a NaN or INDEF)**

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@joshoewahp joshoewahp

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@joshoewahp @Adirana27