Submission issues from ESO email 20130808

Problem reports ESO highlighted like this.  

Solutions that Dave (or Stephen) will implement now, and in the future SSDR releases are in purple

Updates to the pipeline required by Stefano. These are not needed for SSDR1 reductions, but should be done for the next release of the pipeline

> Dear Stephen and PESSTO team,

> Hence please regard the current feedback as still preliminary. A more extended

> validation will be carried out only once the release is closed.

> 

> Preliminary feedback:

> 

> 1) As agreed, please re-submit using the collection named PESSTO.

> 

> 2) Please remember to change that name (was SSDR) also in the release description.

> 

> 3) Please, as agreed, use the release description template

SJS will deal with the above.

> 4) Automatic validation failed for 2 reasons:

> 

> 4.1) Some spectra were missing, hence some referenced files were orphans

> 

>       I think you already fixed this.

The reason this failed was due to spectra failing to be verified and therefore leaving their associated files as `orphaned` files.

> 4.2) Please add LONGSTRN = T in the headers of these files:

> 

>  - OGLE-2013-SN-019_20130310_Gr13_Free_slit1.0_56457_1_sb.fits

>  - MASTEROTJ093953.18+1_20130208_Gr13_Free_slit1.5_56456_1_si.fits

>  - MASTEROTJ093953.18+1_20130208_Gr13_Free_slit1.5_56456_1_sb.fits

>  - OGLE-2013-SN-019_20130310_Gr13_Free_slit1.0_56457_1_si.fits

> 

>  That is because fitsverify 4.16 reports the following problem:

>    "The OGIP long string keyword convention is used without the

>    recommended LONGSTRN keyword.  (HEASARC Convention)"

Solution: look for the CONTINUE keyword in **ALL** fits headers - if found add the `LONGSTRN = T` keyword. This should be done just before converting to binary table format.

> 5) You submitted some flux calibrated images as ancillary files, with the

>    purpose of associating them to the relevant 1d spectra.

>    May we please strongly recommend you to change this, that is:

> 

> 5.1) Publish those calibrated images not as ancillary files but as main scientific products

> This is to make them searchable via the Phase 3 user interfaces,

> at the benefit of the user community.

> 

> 5.2) Please do not forget to remove from the spectra the ASSO* keywords

> that described those ANCILLARY.IMAGEs that are now promoted to self-reliant

> scientific products.

Solution: Remove the `ASSO*` keywords from all acquisition images. 

Solution: Add the `PRODCATG = SCIENCE.IMAGE` keyword.

> 5.3) Please make sure that both science spectra and science images

> use the same value for the OBJECT keyword. That provides the functionality

> you were after, that is, the association between spectra and images.

> (cf. ESO Science Data Product standard (§2.1):

>  "For spectroscopic public surveys, the value of the OBJECT keyword shall be

>  set to the survey source identifier, which shall be unique within the survey.").

This is already the case.

> 6) 1D spectral keywords found in the ANCILLARY.2DSPECTRUM

> 

> As you already clarified (thanks!), those are pertinent to the 2D spectra as well,

> so we will be pleased to accept this extra and very useful descriptors.

No action required.

> 7) Please remove BUNIT, DATAMIN, and DATAMAX from the PHU of the spectra

> 

>   E.g.: OGLE2013-SN-016_20130310_Gr11_Free_slit1.0_56448_1_sb.fits

> 

>  Those 3 keywords are unusable given that no IMAGE is to be found in that file, and given that it is impossible to know to which array in the BINTABLE they refer to.

Solution: Remove the keywords `BUNIT`, `DATAMIN` and `DATAMAX` from **ALL** binary tables Primary Headers. Question for Dave/Stefano - can anything be done at pipeline level, or best left afterwards to the tidy up? Dave's Response: Best done at the conversion stage.

> 8) Please remove duplicate keywords as per instruction below

> 

> Some keywords are found in both the primary and extension headers,

> while the ESO Science Data Product (SDP) standard expect them to be

> present only in one of the two.

> For a cleaner header, can you please consider the following fixes:

> APERTURE  Please remove it from the PHU

> LAMRMS       remove it from the bitable

> SPEC_BW     remove it from the PHU

> SPEC_ERR   remove it from the bintable

> SPEC_SYE   remove it from the bintable

> SPEC_VAL   remove it from PHU

> TELAPSE      remove it from PHU

> TITLE            remove it from PHU

> TMID             remove it from PHU

> VOCLASS     remove it from PHU

> VOPUB         remove it from PHU

    

> 9) Please fix the inconsistencies found on 6 keywords:

>  - ABMAGLIM

>  - LAMLIN

>  - MJD-END

>  - ORIGIN

>  - TELAPSE

>  - TMID

> 

> Example:   2013am_20130403_GB_merge_56478_1_sb.fits

> 

> 9.1) TELAPSE expected 2636.56988802831, found: 2776.569888065569

> 9.2) MJD-END expected 56386.1149523952, found: 56386.1165727655

> 9.3) TMID is also wrong (given the wrong value of MJD-END).

Here are the values of the keywords in the fits header of 2013am_20130403_GB_merge_56478_1_sc.fits:

TELAPSE =    2776.569888065569 / Total elapsed time [s]

MJD-OBS =       56386.08443654 / MJD start

EXPTIME =                560.0                    

MJD-END =    56386.11657276555 / MJD end  

TEXPTIME=               1680.0 / Total integration time of all exposures (s)

TMID    =    56386.10050465277 /  [d] MJD mid exposure   

NCOMBINE=                    8   

HIERARCH ESO DET DIT         =   70.0000000 / Integration Time   

HIERARCH ESO DET NDIT         =   3 

And here are Dave’s findings:

EXPTIME is incorrect.

The definition for EXTIME in the documentation:

Total integration time per pixel (in seconds).

For an imaging data product resulting from the co-addition of multiple exposures pointing at the same sky position (with a tolerance given by a small fraction of the instrumental field of view), EXPTIME should represent the total integration time per pixel obtained in the centre of the image.

… Note that EXPTIME as given in the original raw data almost never represents the proper number of EXPTIME for the product, specifically if detector sub-integrations are involved.

The pipeline currently uses:

EXPTIME=NCOMBINE*DIT = 560.

But having checked with ESO we should always have EXPTIME equal to TEXPTIME i.e. we assume TEXPTIME is used to refer to stacking of individual combined spectra/images. Therefore

EXPTIME=NCOMBINE*DIT*NDIT

Solutions:

MJD-END is incorrect

Having queried eso about how MJD-END is calulated, here is their response:

The MJD-END for a combined SOFI spectrum from 8 raw files should be 

 1. take MJD-OBS time for the last observed raw file 

 2. add to it the following number of seconds: NDIT*(EXPTIME+1.8 sec^) 

^1.8 sec is the overhead between one DIT and the next [from the SOFI usermanual]. 

TELAPSE is incorrect (due to incorrect MJD-END value)

TELAPSE=MJD−END−MJD−OBS

TMID is incorrect (due to incorrect MJD-END value)

TMID=(MJD−END+MJD−OBS)/2.

> 9.4) ORIGIN expected "ESO", found: "NOAO-IRAF FITS Image Kernel July 2003"

> 9.5) LAMLIN expected 14 (integer), found 14.0

Solution: 

> 9.6) ABMAGLIM expected a real, found a string: 'INF.0   '

>        see e.g.: acq_2013am_20130405_V641_56463_19.fits

> 

> (The first three are inconsistent with the values found in the raw files.

> The last three are inconsistent with the values and data types defined in the

> Phase 3 Science Data Product Standard).

PHOTZP missing in 8 acquisition images.

Cosimo: FIXED, it was probably a problem in the pipeline when I run multiple reduction in different terminal.

PSF_FWHM missing in one acquisition image (acq_LSQ12hja_20121212_V641_56463_5.fits).

No obvious reason about the lacking of that.