Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Is it possible to add take_dark_lib and check_dark_lib api through event server? #1211

Open
jilaqi-le-gao opened this issue Jun 12, 2024 · 17 comments
Open

Is it possible to add take_dark_lib and check_dark_lib api through event server? #1211

jilaqi-le-gao opened this issue Jun 12, 2024 · 17 comments

Comments

@jilaqi-le-gao
Copy link

If I want to use phd2 only through event server api, I found there is no api to take dark lib (using default configure is enough) and check whether this profile setting has a dark lib.
I have checked the event server wiki and event_server.cpp, failed to find dark lib related functions.
Is it possible to add those two api in event server? Or is there any easy way to add those two api in event server?
@agalasso
Copy link
Contributor

agalasso commented Jun 12, 2024 •
edited
Loading

We could potentially add this to the api, but could you elaborate on your use case for why you might need it?
The expectation is that a dark library is created once when the camera is initially setup and reused automatically whenever the camera's equipment profile is selected. For purposes of guiding, it should not be necessary to rebuild the dark library more than once or twice per year, if at all. For guiding, the dark library is only used for helping avoid locking onto hot pixels, unlike the dark frame calibration that is important for image camera frames.

@AstroAir

This comment was marked as abuse.

Sign in to view
@AstroAir
Copy link

@agalasso And I think the biggest problem now is that we can only operate phd2 by simulating clicks inside the program, without a good interface

@bwdev01
Copy link
Contributor

bwdev01 commented Jun 13, 2024

It isn't a question of how to do it, it's the more important question of why we would want to. This strikes me as a very marginal use-case for the reasons Andy already mentioned. Further, I question how many typical guide cameras have a mechanical shutter that would support remote dark library building. We have never signed up for making every interactive operation in PHD2 controllable through the API nor do I think we should. I think your implementation is probably better left to a private version of PHD2, which is one of the benefits of having it be open-source.

Bruce

@AstroAir
Copy link

@bwdev01
If there is such an interface that can better adapt to remote operations, I can operate through the client (such as writing a few quick scripts), without the need for VNC

@jilaqi-le-gao
Copy link
Author

@agalasso @bwdev01
Let me explain why we need it.
Because we are developing a remote control software, which operates on a mobile phone, remotelly controls phd2(through a python backend). This meansing our software does not have a chance to use the GUI because everything is on http or websocket. Therefore, we could not use the gui to generate the dark lib.
The current event server api provides almost everything to get phd2 work remotely, except the dark lib.
We also tried to not use the dark lib to guide. But in recent test, even we try to select star manually, it still has chance to select star onto a hot-pixel. Even the min-HFR is larger than 1.5 does not solve the case.
That's why i come to this issue, just to make the remote control more perfect.
Also, if it is necessary to make our software open-source, i would be glad to do so after we finish it.

@agalasso
Copy link
Contributor

@jilaqi-le-gao thanks for explaining the motivation. I am not opposed to adding this feature.
Here's an implementation question: do you expect the client application to interact with the user about covering the camera?
We probably need an additional server api to allow the client application to ask PHD2 whether the guide camera has a shutter. That way the workflow would go something like this:

  1. client application asks PHD2 whether the camera has a mechanical shutter (new server api)
  2. if there is no shutter, the client application asks the user to cover the camera
  3. the client application asks phd2 to take darks (new server api). PHD2 will not ask the user to cover the camera.
  4. phd2 responds to the api request immediately, acknowledging darks are in progress
  5. takes dark frames, reporting progress as event notifications (new event type)
  6. phd2 sends a "darks done" event
  7. client application can tell the user to uncover the camera

To summarize, phd2 would need:

  • an api to report whether the camera has a mechanical shutter
  • an api to start taking darks; phd2 will not prompt user to cover/uncover the camera
  • new notification events to report the progress and completion of darks

@bwdev01
Copy link
Contributor

bwdev01 commented Jun 15, 2024 via email

@jilaqi-le-gao
Copy link
Author

@agalasso
Well, the shutter is new case for me. In our program, becasue we run phd2 with indi server (we make everything controlled by a python server), I am not sure whether the shutter is controllable through indi. Just in my experience, I haven't seen guider cameras with shutter yet.
But, yes, the workflow you metioned is correct.

@jilaqi-le-gao
Copy link
Author

I would really like to see the supporting data here, I doubt seriously that the problem – whatever it was – has been analyzed correctly. In the last 11 years, I can count on one hand the number of times we’ve ever had to tell someone to replace their dark library. I would like to avoid the recurring theme of 1) encounter a problem, 2) mis-analyze the problem but don’t ask for any help, 3) request a new feature that will purportedly “fix” the problem. I’m flabbergasted that anyone would want to provide a server interface to fiddle around with dark libraries while ignoring other far more useful capabilities like Calibration Assistant, Guiding Assistant, polar alignment, etc. From: Gao Le @.> Sent: Saturday, June 15, 2024 6:45 AM To: OpenPHDGuiding/phd2 @.> Cc: bwdev01 @.>; Mention @.> Subject: Re: [OpenPHDGuiding/phd2] Is it possible to add take_dark_lib and check_dark_lib api through event server? (Issue #1211) @agalasso https://github.com/agalasso @bwdev01 https://github.com/bwdev01 Let me explain why we need it. Because we are developing a remote control software, which operates on a mobile phone, remotelly controls phd2(through a python backend). This meansing our software does not have a chance to use the GUI because everything is on http or websocket. Therefore, we could not use the gui to generate the dark lib. The current event server api provides almost everything to get phd2 work remotely, except the dark lib. We also tried to not use the dark lib to guide. But in recent test, even we try to select star manually, it still has chance to select star onto a hot-pixel. Even the min-HFR is larger than 1.5 does not solve the case. That's why i come to this issue, just to make the remote control more perfect. Also, if it is necessary to make our software open-source, i would be glad to do so after we finish it. — Reply to this email directly, view it on GitHub <#1211 (comment)> , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDHSV7T3I3ZVOMEDHWTYVLZHRANHAVCNFSM6AAAAABJFZNWCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRZGY3DGNRYHA . You are receiving this because you were mentioned. https://github.com/notifications/beacon/ADDHSVYWJ7ME3BYQIRR3T2TZHRANHA5CNFSM6AAAAABJFZNWCCWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUBKJYMQ.gif Message ID: @.*** @.***> >

Well, you are right. I perfer to using the exsiting solution to solve this problem than developing a new api. But our key problem is not the guider error, but mis-select a star.
We can provide some data to see whether the dark lib is exactly necessary or not to start an auto-select, which avoiding selecting hot-pixel by setting a reasonable min-HDF. But as we do not have a lot of good weather for testing for almost half years, probbaly we need to wait some time, as the next week may raining all days.

I think maybe we could provide data like this:
1, Our guider camera type (in last testing is QHY 585), and guider telescope info, mount info
2, the origin guider captured fits files, which the star selected to a hot-pixel, even if:

  • we send a roi when we start auto guide. (this is an existing event server api). Inside the roi, there is a very bright star.
  • we set the min-HFR to at least 2.

if the phd2 has other logs or data can export for better analysis this problem, please let me know.

However, in my opinion, setting a roi region before start guide is not always the solution, because we can't expect the light star to be always inside this area. So, we could not let the roi region to be very small. Aslo, the roi is only one rect area, which may easily contain a hot-pixel. But anyway, maybe more testing can tell us more.

@bwdev01
Copy link
Contributor

bwdev01 commented Jun 16, 2024 via email

@jilaqi-le-gao
Copy link
Author

@bwdev01
Well, one more comment is, because we do not have dark lib api, so all our tests are run under no dark lib situations, to find out if we ingore the dark lib database, can we still use auto-star select.

As you suggested, I will try with a much larger HFD value. I will post the data here as soon as we tested.

@bwdev01
Copy link
Contributor

bwdev01 commented Jun 17, 2024 via email

@jilaqi-le-gao
Copy link
Author

Don’t just guess about MinHFD, you need to be systematic about this. Another possibility other than hot pixels is that you might have reflected light on the guide camera sensor that is masquerading as a star. That is also fixable with the MaxHFD setting. Just do the following: 1. Under reasonable night sky conditions, start looping the guide camera using 2-sec exposures. 2. Capture several representative guide camera images using the File/Save menu option 3. Run the Star Profile tool while you are looping exposures and look at the typical HFD values 4. Do an auto-find and see if you get a list of reasonable stars 5. If you think auto-find has selected a hot pixel, post the debug log file using the Help/Upload Logs function and send me that resultant link Why can’t you also build a legitimate dark library the usual way, making sure that the guide scope is covered when the exposures are being taken? Bruce From: Gao Le @.> Sent: Sunday, June 16, 2024 5:48 PM To: OpenPHDGuiding/phd2 @.> Cc: bwdev01 @.>; Mention @.> Subject: Re: [OpenPHDGuiding/phd2] Is it possible to add take_dark_lib and check_dark_lib api through event server? (Issue #1211) @bwdev01 https://github.com/bwdev01 Well, one more comment is, because we do not have dark lib api, so all our tests are run under no dark lib situations, to find out if we ingore the dark lib database, can we still use auto-star select. As you suggested, I will try with a much larger HFD value. I will post the data here as soon as we tested. — Reply to this email directly, view it on GitHub <#1211 (comment)> , or unsubscribe https://github.com/notifications/unsubscribe-auth/ADDHSV6DR7MGOGQGRXRG56LZHYW47AVCNFSM6AAAAABJFZNWCCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZRHE3TIOBZGI . You are receiving this because you were mentioned. https://github.com/notifications/beacon/ADDHSV6WQDJUSAX345BHXULZHYW47A5CNFSM6AAAAABJFZNWCCWGG33NNVSW45C7OR4XAZNMJFZXG5LFINXW23LFNZ2KUY3PNVWWK3TUL5UWJTUBOW2OY.gif Message ID: @.*** @.***> >

Because as our application is using phd2 without the desktop of linux. Every control signal is done by event server api.
Therefore, as the event server doesn't contain 'generate dark lib' api, we must assume our application is run under the situation with no dark lib. Even though under tesing, we can attach a screen to see what is happening in phd2, and export some data. But screen is not reachable under deploy environment in our application.
Besides, as a lot of settings do not have an event server api, such as min-HFD, we have to prewrite this value into .PHD2Guding setting file, before starting a phd2 process. Which means we can't dynamically change the min-HFD unless we restart the whole phd2 process.

As just come in to my mind, I am not sure whether we can prewrite dark lib data into .PHD2Guding file if dark lib data is written in such file. If it is feasible, I need to know the data format of dark lib data, path and format in config file. Then, we can generate dark lib through our own way and prewrite into .PHD2Guding file before we start process. In such way, there is no need to change event server api.

@jilaqi-le-gao
Copy link
Author

@agalasso at last several days, we have tested running without dark lib and setting min-HFD to 7.
Actually, it works, it will auto select to real star, ignoring the dark noise.
However, when I testing with indi guide simulator to solve other problems. I faced another problem, if all stars HFD inside the view is too small, it may cause an too small HFD error and report star lost, even at the beginning it locked a star. Due the star HFD may changing due to the seeing, It may cause the HFD change and become lower than given min-HFD value, and cause star lost.

In short, setting a higher min-HFD value to avoid selecting hot-pixel is feasible. But the best value depends on the stars in view. If the stars are too dim, it may cause a too small HFD error and leads to an star-lost.

Is there any other ways to generate dark lib by myself, and add the result into phd2, by files or settings?

@agalasso
Copy link
Contributor

agalasso commented Aug 3, 2024

@jilaqi-le-gao PHD2 will automatically load a dark library if it finds one as follows:

  • file location (linux): ~/.phd2/darks_defects/PHD2_dark_lib_<PROFILE_ID>.fit where PROFILE_ID is the profile numeric id. You can get the profile ids with the get_profiles API. If there is just one profile the profile id is 1.
  • file format: FITS file with one 16-bit (USHORT) image HDU per exposure time. Each HDU's header must have an EXPOSURE key with the exposure time in seconds (float).
  • relevant code: load_multi_darks in myframe.cpp

When PHD2 generates a dark library FITS file, it takes a series of exposures and combines them using the mean pixel value for each pixel (CreateMasterDarkFrame in darks_dialog.cpp)

When PHD2 does dark subtraction, it selects the HDU from the dark library that has an exposure time less than or equal to the exposure time of the current frame, so dark subtraction works best if there is a dark frame for each exposure time used during guiding.

@jilaqi-le-gao
Copy link
Author

@jilaqi-le-gao PHD2 will automatically load a dark library if it finds one as follows:

* file location (linux): `~/.phd2/darks_defects/PHD2_dark_lib_<PROFILE_ID>.fit`  where PROFILE_ID is the profile numeric id. You can get the profile ids with the `get_profiles` API.  If there is just one profile the profile id is `1`.

* file format: FITS file with one 16-bit (USHORT) [image HDU](https://fits.gsfc.nasa.gov/fits_primer.html) per exposure time.  Each HDU's header must have an [EXPOSURE](https://heasarc.gsfc.nasa.gov/docs/fcg/common_dict.html) key with the exposure time in seconds (float).

* relevant code: `load_multi_darks` in `myframe.cpp`

When PHD2 generates a dark library FITS file, it takes a series of exposures and combines them using the mean pixel value for each pixel (CreateMasterDarkFrame in darks_dialog.cpp)

When PHD2 does dark subtraction, it selects the HDU from the dark library that has an exposure time less than or equal to the exposure time of the current frame, so dark subtraction works best if there is a dark frame for each exposure time used during guiding.

Thanks a lot, this information really helps a lot.
We are current trying to do both ways: 1, add api to event_server.cpp, trying to call start take dark function through socket. If success, we may push an PR.
2, in case we failed to run the take dark through socket, we want to generate the dark lib into the format you metioned, as using indi to call guider to take darks does the same thing. We only need to make the data format the same and put the file at the exact place.

Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@agalasso @bwdev01 @jilaqi-le-gao @AstroAir