Bug fixes and updates required by pipeline for SSDR2  - 20130917

Last Updated: 20140318 (DRY)

Bug Fixes Required in Pipeline

Images and Spectra

• 1. Set origin = "ESO" for all files      DONE

  2. In the FITS headers that contain a cards with keyword a value > 68 characters, we need to add the card LONGSTRN = T. This permits the use of the CONTINUE keyword automatically included by pyfits whenever this 68 character limit is broken. This occurred for files that recorded the names of some intermediate file with very long filenames. NOTE ESO have since change their mind and the LONGSTRN keyword is no longer permitted!! They now want us to truncate keyword values > 68 characters. See the file MASTERJ093953.18+165_20130208_Gr13_Free_slit1.5_56456_1_si.fits attached to this page as an example. The ARC keyword reads:

                ARC     = 'arc_MASTEROTJ093953.18+165516.4_20130208_Gr13_Free_slit1.5_56456_1.f'

        that means we need to cut the value whenever is longer that 68 char ? or the value is truncated automatically ?

    3. MJD Keywords need updated -- for images and spectra 

        mjd_end = mjd_obs + (ndit*single_image_exptime+1.8)/(60.*60.*24.)   DONE for SOFI SPECTRA         telapse = mjd_end-mjd_obs         tmid = telapse/2.

        where 1.8 sec is the overhead between one DIT and the next

Images Only (now including EFOSC acquisition images)

• 1. Some files have ABMAGLIM = INF.0 or ABMAGLIM = 9999.0

  2. Fix: Use equations A.7 and A.8 from the PESSTO survey description paper to set ABMAGLIM and ABMAGSAT for all EFOSC and SOFI images. DONE Some of the images will still have 9999, is astrometry fail

  3. Having confirmed with Alberto, the NCOMBINE attribute of the images should always be the number of on-

     1. source frames that went into the final image. For SOFI images where the sky is measured away from the

     2. source we were setting NCOMBINE = number of on-source & off-source frames. This needs to be changed to NCOMBINE = number of on-source only 

     3. The problem was only with SOFIimages ONN/OFF. DONE 

     4. EXPTIME = TEXPTIME for all images. The science data product standard requires EXPTIME to be the total integration time per pixel

     5. “obtained in at least 50% of the image array taking into account the chosen offset pattern”. Therefore it is expected that EXPTIME = DIT * NDIT * NCOMBINE,

     6. (and hence identical to the TEXPTIME, which is set correctly).

  4. Please remove the CDELT1 and CDELT2 from all images. 

• • 1. DONE

     5. We noticed that we had been recording the SOFI DETRON value in ADU and not electron counts  -- set DETRON = 12.0 in sofi images. Also need to update sofi EFFRON:

          effron = 12.*sqrt(ncombine)/sqrt(ndit)

          I don't understand this. For EFOSC we have DETRON=READ OUT NOISE = 11.6

          for sofi at the moment we have     READ OUT NOISE = 2.1   , but we want DETRON=12.0 ?

         For SOFI we had set DETRON = 2.1 ADU, but we need it in electron counts. Therefore DETRON = adu*gain = 2.1*5.4 ~ 12.

         The value is confirmed here : http://www.eso.org/sci/facilities/lasilla/instruments/sofi/inst/setup/Detector_characteristic.html

      6. We need to make sure the GAIN (e-/DN) is scaled according to the stacking process we've adopted. Our SOFI images are median combined, which means:

            effective gain = 2/3 * NCOMBINE * gain   DONE

     7. There's an issue with ZP measured in a very small percentage of images (zeropoint_corrections_20140523.tgz - https://db.tt/9XbCCSfX). Cosimo measure the ZPs by hand and Dave updated the values in the fits headers before send to ESO. Find the SOFI/EFOSC spreadsheets of corrections attached below (CI_mjd_obs_vs_photzp_EFOSC2_20140514t172734.xlsx, CI_mjd_obs_vs_photzp_SOFI_20140522t155631.xlsx)

        8. Wrong datatype or undefined value for some DETRON, PHOTSYS, PHOTZPER, PSF_FWHM keywords (best to look at the keywords in the 10 example images here and for some notes on what Dave did to fix these issues see here) 

      9. EFFRON for SOFI images. The SOFI images were median-combined so we are to use this formula:

                    EFFRON = RON / sqrt( ncombine * ndit ) * sqrt( pi/2 )

                instead of

                    EFFRON = 12.*sqrt(ncombine)/sqrt(ndit)

    10. SOFI zeropiints. ESO want the zeropoint corrected so that:

              MAG = -2.5 log10( data ) + PHOTZP

         and not as we have it:

            MAG = -2.5 log10( data / TEXPTIME) + PHOTZP

     Note we should be using DIT and not TEXPTIME as the images are median combined, so to fix our PHOTZP we need to apply this correction:

            PHOTZP_new = PHOTZP_old + 2.5log10(DIT)

    See here for more notes. 

        Also note, we have decided not to submit EFOSC images as 'science images', so we don't need to worry about the format of PHOTZP for EFOSC (at this time).

Spectra Only

• 1. 860 efosc spectra from April 2012 with in incorrect SPEC_BIN value set (may no longer be an issue):

     • Gr11 files with SPEC_BIN = 1.27 when it should be 0.41

     • Gr13 files with SPEC_BIN = 1.52 when it should be 0.55

     • Gr16 files with SPEC_BIN = 1.88 when it should be 0.43

      if the issue was only in the April data, I'm not doing anything unless the problem is found in more recent data

• 1. Set SPEC_SYE = 1 for EFOSC and SOFI spectra

     1. DONE but probably this should be 0.1 instead of 1.0. I have changed and now the wavelength check has a step of .1 A

  2. The exposure time for the combined SOFI spectra should be the TOTAL time spent on exposures, i.e.

• • 1. exptime = texptime = single_exposure_time * NDIT * NCOMBINE

• 3. SPEC_RES incorrect for some EFOSC spectra - values should be in the 200–600 but for some spectral files the value is < 80. An example file is attached (SSS130404-102043-062_20130419_Gr13_Free_slit1.0_56457_2_si.fits) with spec_res = 18.33

  4. Some EFOSC spectra have PRODCATG = "SCIENCE.IMAGE" but it should be PRODCATG = "SCIENCE.SPECTRUM". This happened in ~1000 efosc spectra (see the PS1-12baa_20120808_Gr16_OG530_slit1.0_56446_1.fits example below).