Phase 3 data reduction and keyword work to do

Reference : any referral to the SPD is the "ESO Science Data Product Standard"  January 2013 release:

EFOSC2 spectroscopy files - to do

1. SPEC_RES=    359.5606862048376 / [nm] Reference spectral resolution (FWHM)                  DONE
the comment should be "Spectral resolving power", the number is correct - just the  comment needs updated. 

SPECSYS = 'HELIOCEN'           / Observed frame                                                                       DONE
change to 
SPECSYS = 'TOPOCENT'           / Observed frame
Sorry, I was wrong previously. Should be topocentric (i.e. as observed from La Silla, 
no correction for earth or solar motion). 

2. Can we delete these :                                                                                                         DONE
HIERARCH ESO DPR CATG        = 'SCIENCE '   / Observation category
HIERARCH ESO DPR TECH        = 'SPECTRUM'   / Observation technique
HIERARCH ESO DPR TYPE        = 'OBJECT  '   / Observation type

As the SDP manual now says : "However, the hierarchical products from the DPR category, like HIERARCH ESO DPR CATG, are an exception. Actually, science data products shall not contain keywords from the DPR category [AD3]. The value of the keyword DPR TECH is normally propagated to the keyword OBSTECH in the product tile."

3. The unit, or number here is wrong. The unit is required to be in degrees !                   DONE
i.e. should be 2.778e-4 in degrees, for 1 arcsec slit 
APERTURE=                  1.0 / [deg] Aperture diameter

4. Unit here is wrong. You have it defined correctly, i.e.                                                    DONE
But this is in days of course. 
TELAPSE = 0.001620376155187842 / Total elapsed time [s]

You need to multiply by 24*3600 = 86400  to get it into seconds. 

5. CCD parameters :

there is a discrepancy between what is written in the header for CCD40 and what is on the EFSOC2 webpages. The header value of 0.91 e-/ADU is wrong. Morgan has measured the gain and readnoise from April 2013 and October 2012 and finds GAIN = 1.18  e- per ADU. Hence I think we should change 

HIERARCH ESO DET OUT1 GAIN   =         0.91 / Conversion from electrons to ADU                  DONE
HIERARCH ESO DET OUT1 GAIN   =         1.18 / Conversion from electrons to ADU

and also we need to add a header keyword :

GAIN = 1.18 / Number of electrons per data unit                                                                  DONE


"Detector Readout noise in electrons"
We need to add this (Section 2.5 in the SDP manual). It is the detector read noise. Again there is a difference in the ESO header. From the same runs in April 2013 and October 2012, Morgan finds DETRON = 11.00 e- (October 2012) and 12.20 e- for April 2013. Hence I think we should change 

HIERARCH ESO DET OUT1 RON    =         7.80 / Readout noise per output (e-)                       DONE
HIERARCH ESO DET OUT1 RON    =         11.6 / Readout noise per output (e-)

and also we need to add a header keyword :

DETRON = 11.6/ Detector readout noise in electrons                                                              DONE

"Effective readout noise in electrons"                                                                                SET as DETRON (at the moment)
set this equal to DETRON, as we have single science frames (not
combined) and typically we have a large number of bias and flat frames
which will reduce their contribution to EFFRON. SJS to CHECK THIS DURING REDUCTIONS.

6.  Add keyword for LACOSMIC :                                                                                                    DONE
vwe need to add a header keyword to note if cosmic ray cleaning has been applied
this should be boolean (T/F). We are encouraged to add our own keywords, as long as
they are standard, are useful and are described in the data release paper 
I suggest :
LACOSMIC = T          /TRUE if Laplacian cosmic ray rejection has been applied to the image

7. The 2D wavelength calibrated file
There's a revised way of submitting this, and making the fitsheaders compatible. It's a little
different to what we have, and needs some minor changes 
The 2D file is an "associated ancillary data" product, not a product in itself.  It needs 
specifically linked from the 1D spectum file (i.e. in our  *_1_e.fits file)

In this 1D spectrum file (*_1_e.fits), we must point to this, so we
need these headers in the original 
*_1_e.fits file :

ASSON1 ='NAME of 2D file goes here....'          / Name of associated file                                DONE
ASSOC1 ='ANCILLARY.2DSPECTRUM'        / Category of associated file

Important - *delete* this header keyword from the 2D files (and only the 2D files):                           DONE
PRODCATG= 'SCIENCE.IMAGE'      / Data product category

Delete it completely - as ESO don't want it. They just want the 1D spectrum file to reference the 2D "ancillary file. 

EFOSC2 image files  (i.e. photometry, not 2D spectral images) 

1.  Keyword value "ORIGIN" is set wrong in the image files. It should be "ESO"

FILE                                                               ORIGIN                              
LSQ12dyw_20120808_R642_56432_1.fits                                 NOAO-IRAF FITS Image Kernel July 2003
SN2012ca_20120824_r784_56432_1.fits                                 NOAO-IRAF FITS Image Kernel July 2003
SN2012ec_20120818_r784_56432_1.fits                                 NOAO-IRAF FITS Image Kernel July 2003
tCHASEJ20032484-5557192_20120807_Gr13_Free_slit1.0_56432_1.fits     ESO                                  
tCHASEJ20032484-5557192_20120807_Gr13_Free_slit1.0_56432_1_2df.fits ESO                                  
tCHASEJ20032484-5557192_20120807_Gr13_Free_slit1.0_56432_1_e.fits   ESO                                  
tCHASEJ20032484-5557192_20120807_Gr13_Free_slit1.0_56432_1_ex.fits ESO                                  
tCHASEJ20032484-5557192_20120807_Gr13_Free_slit1.0_56432_1_f.fits   ESO                                  
tCHASEJ21253403-4016432_20120807_Gr13_Free_slit1.5_56432_1.fits     ESO   

2.  Keyword TEXPTIME is missing in the imaging (but it is there in the spectral files). For EFOSC2 this is just set to EXPTIME, as we are not submitting merged products. 

3.  Following keywords are missing in the imaging (but are there OK in the spectral files)
OBSTECH  (should be set to "IMAGE")
PRODCATG  (should be set to SCIENCE.IMAGE)
BUNIT  (should be set to "ADU") 

These also need added (as above for spectral files)

4.  FLUXCAL is completely missing in the file LSQ12dyw_20120808_R642_56432_1.fits but it is set correctly in the others 

/Users/sjs/data/pessto/Phase3/May2013-test1>dfits *.fits | fitsort FLUXCAL
FILE                                                               FLUXCAL    
SN2012ca_20120824_r784_56432_1.fits                                 UNCALIBRATED
SN2012ec_20120818_r784_56432_1.fits                                 UNCALIBRATED
tCHASEJ20032484-5557192_20120807_Gr13_Free_slit1.0_56432_1.fits     ABSOLUTE    
tCHASEJ20032484-5557192_20120807_Gr13_Free_slit1.0_56432_1_2df.fits ABSOLUTE    
tCHASEJ20032484-5557192_20120807_Gr13_Free_slit1.0_56432_1_e.fits   ABSOLUTE    
tCHASEJ20032484-5557192_20120807_Gr13_Free_slit1.0_56432_1_ex.fits ABSOLUTE    
tCHASEJ20032484-5557192_20120807_Gr13_Free_slit1.0_56432_1_f.fits   ABSOLUTE    
tCHASEJ21253403-4016432_20120807_Gr13_Free_slit1.5_56432_1.fits     ABSOLUTE  

6.  Similarly, the following are completely missing in the file LSQ12dyw_20120808_R642_56432_1.fits
but set correctly in the others 

7.  DATAMIN and DATAMAX are both set to 0 in the image files. Should be "the minimum/maximum valid physical value across the pixel array in units defined by BUNIT." I guess this should be 0 and 65536

Submission of photometric imaging files 

See the new ESO Phase3 faq,  for some new information

i.e. see the specific answers to the following two questions at the bottom :

Q: How to submit image data without accurate astrometric/photometric calibration via Phase 3?

Q: How to submit an image without photometric and astrometric calibration?

Hence we need to decide if we will submit our images as "calibrated" or "uncalibrated" images. We can do either, but the submission process and file linkage is different depending on which route we choose. 

Here is the proposed method :

1. Acquisition images : 

Should all be run through PESSTOEFOSCPHOT. Always submit them as an associated file of a main SCIENCE product (i.e. our fully reduced SCIENCE.SPECTRUM fits binary table spectrum). This means that the header in the spectrum file should point to 

ASSONi = 'name_of_acquistion_image__should-contain-object-name.fits'.

And this is important - we must delete the header item PRODCATG from the acquisition image. As this file is not a SCIENCE product, it is just an association. 

This part of the header keyword work will be done at QUB, during the creation of the binary table file. Dave will do the association between the fits binary table and the images (Morgan/Cosimo don't need to do anything, Stefano does not need to adjust pipeline for this). 

As this is an associated file, it is not necessary that we have astrometrically and photometrically calibrated images. But as PESSTOEFOSCPHOT will usually do both (to some level of accuracy), we will always run them through this process and submit with the pipeline calibration of astrometry and photometry in the header items (i.e.according to Sect 2.6 WCS keywords and Sect. 2.9 Image photometric zeropoint  in the SPD). 

To keep things homogeneous, I suggest we set the relevant keywords exactly the same as below for the "Multi-colour follow-up imaging"

2. Multi-colour follow-up imaging  :

Should all be run through PESSTOEFOSCPHOT. These will all be submitted as SCIENCE products. Hence they must have 


They must have WCS and photometric information header keywords, or else they will not qualify as standard ESO products. But these header keywords can be set in an appropriate way if we do not have reliable astrometry or photometry (see the FAQ above). 

2.1 If pipeline determines a ZP from reference stars in the field, then :

PHOTZP - set to a measured ZP if the pipeline finds suitable reference stars in the field.
PHOTZPER - This is "an optional keyword to indicate the (1 sigma) uncertainty of the photometric zeropoint." Stefano, does the pipeline set this sensibly ? 

SV: The PHOTZPER in not set at the moment since it is optional

PHOTSYS - set to AB, Vega etc. 

and set 


2.2  If there is no way to set a ZP from reference field stars
can be done looking the zeropoint of a full run at the end of the reduction at the marshall level ?

PHOTZP - set to the instrumental zeropint, plus an extinction correction based on the filter, airmass and standard La Silla extinction curve. See
SV: At the moment is set to 9999

 -set this to 2.00
PHOTZP  -  set to AB, Vega etc. 

and set 

FLUXCAL = ‘ABSOLUTE’                                      SV: At the moment it is set to UNCALIBRATED
In the data release paper, we will discuss the cases where PHOTZPER is set to 2.00 and explain that this is a nominal zerpoint from instrumental characteristics. 

In addition, the following keywords must be set if FLUXCAL = 'ABSOLUTE' 


I think these are all handled fine in the pipeline, as long as there is a value for ZPT. Correct, Stefano ?? 
As far as the astrometry is correct PSF_FWHM,ELLIPTIC,CRDER1,CRDER2,CUNIT1,CUNIT2,CSYER1,CSYER2 are set

as far as the zeropoint is calculated PHOTZP,PHOTSYS,ABMAGSAT,ABMAGLIM are set

if we are using a nominal zeropoint PHOTZP,PHOTSYS,ABMAGSAT,ABMAGLIM can be set using this nominal zeropoint (as far as the astrometry works)

3. WCS Astrometric calibration 
Handled within the pipeline - we always set to calibrated and Stefano has the keywords set appropriately. If astrometry completely fails, we handle on case by case basis
e.g. do not submit the data file, or make it an association.