possibly generate new index files for astrometry_net correction

[mcclearyj]

Default ones might have wrong scale, or "not be good enough" for what need. James might have instructions on how to do this... We need to investigate this.

[jrmullaney]

Instructions on how to build astrometry.net index files for
the LSST stack based on UCAC4 data.
(Modified from: https://github.com/dstndstn/astrometry.net/tree/master/doc/UCAC4_guide)

1. Install astrometry.net and make sure you can run hsplit.

2. Create a new directory called anetIndex and cd to that directory:

mkdir anetIndex
cd anetIndex

3. Clone the astrometry.net repository from github into this new directory

git clone https://github.com/dstndstn/astrometry.net

4. To convert the raw UCAC files to fits files you need to 'make' the ucac4tofits script.
   When I did this, I ran into an error in astrometry.net/includ/astrometry/os-features.h,
   so I had to edit that file to comment out the following lines:
   #ifndef DONT_INCLUDE_OS_FEATURES_CONFIG_H
   #include "astrometry/os-features-config.h"
   #endif
   After doing that, I was able to make ucac4tofits by:

cd astrometry.net/catalogs
make ucac4tofits

5. In anetIndex/ , make a new directory to contain the raw UCAC4 data files and cd there:

cd ../../
mkdir UCAC4
cd UCAC4

6. Grab the download file (for me, the one that cloned was not the latest version, oddly):

wget https://github.com/dstndstn/astrometry.net/blob/master/doc/UCAC4_guide/get_ucac4.py

7. Run the download file:

python get_ucac4.py

8. Two of the files z???.bz2 files that are downloaded by get_ucac4.py are corrupted, and no
   matter how much re-downloading I did seemed to fix them (corrupted at source?). As such, you
   need to make sure that they are ignored by the fits builder:

mv z431.bz2 z431_bad.bz2
mv z664.bz2 z664_bad.bz2

9. Copy the fits file generator you created in step 4 to the UCAC4 directory:

cp ../astrometry.net/catalogs/ucac4tofits

10. Run ucac4tofits to convert the binary z files to fits files:

./ucac4tofits -N 1 z???.bz2

11. For the LSST, don't require all the columns in the UCAC4 tables, but we do need:
    id, ra, dec, some mags and a star-not-galaxy flag.
    So, I wrote my own python script to strip out what the LSST stack needs and add the
    starnotgal flag. Run it by:

python trimUCAC4.py

12. Run this command to split the fits tables into 12 healpix tiles with
    2 deg overlap (-m 2):

hpsplit -o split-%02i.fits -n 1 -m 1 trim_???.fits

13. For GOTO, I find that index scale 5 works well, but I also include some
    larger scales as well. It turnes out that the default scales in build_index.py
    are already ok, so you don't need to edit them for GOTO. If you found you needed
    some other scales, then edit:
    scale_range = [5, 7]

You may also need to edit the column that the index-builder sorts on, do that by
editing:
sortcol = 'v'

14. After editing (if needed), then run:

python build-index.py
(which takes some time)

15. Once build_index.py has done its thing, you'll have a lovely suite of
    UCAC4-based astrometry.net index files suitable for working with the LSST
    stack. You'll need to copy or move them over to the stack:

mkdir /path/to/astrometry_net_data/UCAC4
cp index-ucac4* /path/to/astrometry_net_data/UCAC4/

16. Write an andConfig.py file for the stack to refer to and put it in:
    /path/to/astrometry_net_data/UCAC4/
    That file should contain the following:
    root.starGalaxyColumn = "starnotgal"
    filters = ('b','v','m')
    root.magColumnMap = dict([(f,f) for f in filters])
    root.magErrorColumnMap = dict([(f, f + '_err') for f in filters])
    root.indexFiles = [
    'index-ucac4-5-00.fits',
    'index-ucac4-5-01.fits',
    'index-ucac4-5-02.fits',
    ...all the other index files... (replace this with filenames, obviously!)
    ''index-ucac4-7-11.fits'']

17. Create a ups directory in UCAC4:

mkdir mkdir /path/to/astrometry_net_data/UCAC4/ups
and in it create a file called:
astrometry_net_data.table
which contains a single line:
setupRequired(sconsUtils)

18. With the stack set up, add the new astrometry index files to eups:
    eups declare astrometry_net_data UCAC -r /path/to/astrometry_net_data/UCAC4/

19. Then, finally, make sure you set up the new index files when setting up your obs_package
    by adding the following line to your obs_package's eups table:
    setupRequired(astrometry_net_data UCAC)

[jrmullaney]

Trim UCAC4:
'''
from astropy.io import fits
import glob
import numpy as np

files = glob.glob('ucac4*.fits')

for file in files:
hdu = fits.open(file)
dat = hdu[1].data
ind = np.where(dat['OBJTYPE']<=1)
id = fits.Column(name='id', array=dat['RNM'][ind], format='K')
ra = fits.Column(name='RA', array=dat['RA'][ind], format='D')
de = fits.Column(name='DEC', array=dat['DEC'][ind], format='D')
mm = fits.Column(name='m', array=dat['MAG'][ind], format='F')
me = fits.Column(name='m_err', array=dat['SIGMAG'][ind], format='F')
vm = fits.Column(name='v', array=dat['VMAG'][ind], format='F')
ve = fits.Column(name='v_err', array=dat['VMAG_ERR'][ind], format='F')
bm = fits.Column(name='b', array=dat['BMAG'][ind], format='F')
be = fits.Column(name='b_err', array=dat['BMAG_ERR'][ind], format='F')
sng = fits.Column(name='starnotgal', array=np.squeeze(np.ones_like(ind)), format='K')

t = fits.BinTableHDU.from_columns([id, ra, de, mm, me, vm, ve, bm, be, sng])

newname = file.replace('ucac4', 'trim')
t.writeto(newname, overwrite=True)

'''