Embedded Files
Problem reports ESO highlighted like this.
Solutions that Dave (or Stephen) will implement now, and in the future SSDR releases are in purple
Updates to the pipeline required by Stefano. These are not needed for SSDR1 reductions, but should be done for the next release of the pipeline
> Dear Stephen and PESSTO team,
> Hence please regard the current feedback as still preliminary. A more extended
> validation will be carried out only once the release is closed.
>
> Preliminary feedback:
>
> 1) As agreed, please re-submit using the collection named PESSTO.
>
> 2) Please remember to change that name (was SSDR) also in the release description.
>
> 3) Please, as agreed, use the release description template
SJS will deal with the above.
> 4) Automatic validation failed for 2 reasons:
>
> 4.1) Some spectra were missing, hence some referenced files were orphans
>
> I think you already fixed this.
The reason this failed was due to spectra failing to be verified and therefore leaving their associated files as `orphaned` files.
> 4.2) Please add LONGSTRN = T in the headers of these files:
>
> - OGLE-2013-SN-019_20130310_Gr13_Free_slit1.0_56457_1_sb.fits
> - MASTEROTJ093953.18+1_20130208_Gr13_Free_slit1.5_56456_1_si.fits
> - MASTEROTJ093953.18+1_20130208_Gr13_Free_slit1.5_56456_1_sb.fits
> - OGLE-2013-SN-019_20130310_Gr13_Free_slit1.0_56457_1_si.fits
>
> That is because fitsverify 4.16 reports the following problem:
> "The OGIP long string keyword convention is used without the
> recommended LONGSTRN keyword. (HEASARC Convention)"
Solution: look for the CONTINUE keyword in **ALL** fits headers - if found add the `LONGSTRN = T` keyword. This should be done just before converting to binary table format.
> 5) You submitted some flux calibrated images as ancillary files, with the
> purpose of associating them to the relevant 1d spectra.
> May we please strongly recommend you to change this, that is:
>
> 5.1) Publish those calibrated images not as ancillary files but as main scientific products
> This is to make them searchable via the Phase 3 user interfaces,
> at the benefit of the user community.
>
> 5.2) Please do not forget to remove from the spectra the ASSO* keywords
> that described those ANCILLARY.IMAGEs that are now promoted to self-reliant
> scientific products.
Solution: Remove the `ASSO*` keywords from all acquisition images.
Solution: Add the `PRODCATG = SCIENCE.IMAGE` keyword.
> 5.3) Please make sure that both science spectra and science images
> use the same value for the OBJECT keyword. That provides the functionality
> you were after, that is, the association between spectra and images.
> (cf. ESO Science Data Product standard (§2.1):
> "For spectroscopic public surveys, the value of the OBJECT keyword shall be
> set to the survey source identifier, which shall be unique within the survey.").
This is already the case.
> 6) 1D spectral keywords found in the ANCILLARY.2DSPECTRUM
>
> As you already clarified (thanks!), those are pertinent to the 2D spectra as well,
> so we will be pleased to accept this extra and very useful descriptors.
No action required.
> 7) Please remove BUNIT, DATAMIN, and DATAMAX from the PHU of the spectra
>
> E.g.: OGLE2013-SN-016_20130310_Gr11_Free_slit1.0_56448_1_sb.fits
>
> Those 3 keywords are unusable given that no IMAGE is to be found in that file, and given that it is impossible to know to which array in the BINTABLE they refer to.
Solution: Remove the keywords `BUNIT`, `DATAMIN` and `DATAMAX` from **ALL** binary tables Primary Headers. Question for Dave/Stefano - can anything be done at pipeline level, or best left afterwards to the tidy up? Dave's Response: Best done at the conversion stage.
> 8) Please remove duplicate keywords as per instruction below
>
> Some keywords are found in both the primary and extension headers,
> while the ESO Science Data Product (SDP) standard expect them to be
> present only in one of the two.
> For a cleaner header, can you please consider the following fixes:
> APERTURE Please remove it from the PHU
> LAMRMS remove it from the bitable
> SPEC_BW remove it from the PHU
> SPEC_ERR remove it from the bintable
> SPEC_SYE remove it from the bintable
> SPEC_VAL remove it from PHU
> TELAPSE remove it from PHU
> TITLE remove it from PHU
> TMID remove it from PHU
> VOCLASS remove it from PHU
> VOPUB remove it from PHU
> 9) Please fix the inconsistencies found on 6 keywords:
> - ABMAGLIM
> - LAMLIN
> - MJD-END
> - ORIGIN
> - TELAPSE
> - TMID
>
> Example: 2013am_20130403_GB_merge_56478_1_sb.fits
>
> 9.1) TELAPSE expected 2636.56988802831, found: 2776.569888065569
> 9.2) MJD-END expected 56386.1149523952, found: 56386.1165727655
> 9.3) TMID is also wrong (given the wrong value of MJD-END).
Here are the values of the keywords in the fits header of 2013am_20130403_GB_merge_56478_1_sc.fits:
TELAPSE = 2776.569888065569 / Total elapsed time [s]
MJD-OBS = 56386.08443654 / MJD start
EXPTIME = 560.0
MJD-END = 56386.11657276555 / MJD end
TEXPTIME= 1680.0 / Total integration time of all exposures (s)
TMID = 56386.10050465277 / [d] MJD mid exposure
NCOMBINE= 8
HIERARCH ESO DET DIT = 70.0000000 / Integration Time
HIERARCH ESO DET NDIT = 3
And here are Dave’s findings:
EXPTIME is incorrect.
The definition for EXTIME in the documentation:
Total integration time per pixel (in seconds).
For an imaging data product resulting from the co-addition of multiple exposures pointing at the same sky position (with a tolerance given by a small fraction of the instrumental field of view), EXPTIME should represent the total integration time per pixel obtained in the centre of the image.
… Note that EXPTIME as given in the original raw data almost never represents the proper number of EXPTIME for the product, specifically if detector sub-integrations are involved.
The pipeline currently uses:
EXPTIME=NCOMBINE*DIT = 560.
But having checked with ESO we should always have EXPTIME equal to TEXPTIME i.e. we assume TEXPTIME is used to refer to stacking of individual combined spectra/images. Therefore
EXPTIME=NCOMBINE*DIT*NDIT
Solutions:
DRY will correct EXPTIME in the data to be submitted to ESO
SV to correct the values at the Pipeline level (not needed for SSDR1, but should be corrected in next version of pipeline)
MJD-END is incorrect
Having queried eso about how MJD-END is calulated, here is their response:
The MJD-END for a combined SOFI spectrum from 8 raw files should be
1. take MJD-OBS time for the last observed raw file
2. add to it the following number of seconds: NDIT*(EXPTIME+1.8 sec^)
^1.8 sec is the overhead between one DIT and the next [from the SOFI usermanual].
DRY will correct in the data to be submitted to ESO : these appear to be in the SOFI 2d spectra and binaray tables.
SV to correct the values at the Pipeline level (not needed for SSDR1, but should be corrected in next version of pipeline)
TELAPSE is incorrect (due to incorrect MJD-END value)
TELAPSE=MJD−END−MJD−OBS
DRY will correct in the data to be submitted to ESO : these appear to be in the SOFI 2d spectra and binaray tables.
SV to correct the values at the Pipeline level (not needed for SSDR1, but should be corrected in next version of pipeline)
TMID is incorrect (due to incorrect MJD-END value)
TMID=(MJD−END+MJD−OBS)/2.
DRY will correct in the data to be submitted to ESO : these appear to be in the SOFI 2d spectra and binaray tables.
SV to correct the values at the Pipeline level (not needed for SSDR1, but should be corrected in next version of pipeline)
> 9.4) ORIGIN expected "ESO", found: "NOAO-IRAF FITS Image Kernel July 2003"
> 9.5) LAMLIN expected 14 (integer), found 14.0
Solution:
DRY will correct in the data to be submitted to ESO : these appear to be in the SOFI 2d spectra and binaray tables.
SV to correct the values at the Pipeline level (not needed for SSDR1, but should be corrected in next version of pipeline)
> 9.6) ABMAGLIM expected a real, found a string: 'INF.0 '
> see e.g.: acq_2013am_20130405_V641_56463_19.fits
>
> (The first three are inconsistent with the values found in the raw files.
> The last three are inconsistent with the values and data types defined in the
> Phase 3 Science Data Product Standard).
PHOTZP missing in 8 acquisition images.
Cosimo: FIXED, it was probably a problem in the pipeline when I run multiple reduction in different terminal.
PSF_FWHM missing in one acquisition image (acq_LSQ12hja_20121212_V641_56463_5.fits).
No obvious reason about the lacking of that.
Page updated
Report abuse