eso phase III data validation report 20131125.

We received the following email from Alberto on 20131125:

Your data (spectra and images) submitted under:

> Data Collection PESSTO
> Release Number 1
> Data Provider: Stephen Smartt
> Close Date: 24/Oct/2013

have now been reviewed regarding the overall consistency and compliance
with the ESO/SDP standard in order to ensure their successful integration
into the ESO archive. The purpose of this message is to provide you with
the results of this process and to request feedback where needed.

SUMMARY:

The data substantially meet the required ESO standard. However
a few problems have been found in the course of the validation
process, which must be addressed before the data can be published
through the ESO science archive to the community.

LIST OF PROBLEMS:

METADATA

[R01]. NCOMBINE vs PROVn inconsistencies must be cured

It is expected that the NCOMBINE value is equal to the number of PROVn keywords in your data.
This is not the case for 10 spectra (see list here below) and for more than 150 images.
Either the PROVn list is incomplete, or the counts in NCOMBINE are incorrect.

Example of affected images:
SN2013ak_20130417_Ks_merge_56475_1.fits
LSQ13fn_20130207_Ks_merge_56475_1.fits
etc…

The list of 10 spectra affected by this problem are:
SN2009ip_20130417_GB_merge_56478_1_sb.fits
SN2012ca_20120809_GB_merge_56476_1_sb.fits
SN2012ca_20121113_GB_merge_56477_1_sb.fits
SN2012ca_20130311_GB_merge_56478_1_sb.fits
SN2012ca_20130311_GR_merge_56478_1_sb.fits
SN2012ec_20121204_GB_merge_56477_1_sb.fits
SN2012ec_20130103_GB_merge_56478_1_sb.fits
SN2012ec_20130113_GB_merge_56478_1_sb.fits
SN2012ec_20130207_GB_merge_56478_1_sb.fits
SN2012hs_20130303_GB_merge_56478_1_sb.fits

Please revise and correct those 10 spectra, and please check, and when
needed correct, all SOFI images.

Reminder: if the correction to this problem requires the modification
of the PROVn information, please remember to update consistently
the MJD-END, EXPTIME, and ALL other affected keywords.

@done: Dave to fix this issue  [SJS to check also]

Notes: 

There was an issue with the writing out of the fits files so that although the PROVn keywords are all recorded in the database, they were not all printed to the FITS headers. As this was only a print-out issue no other keywords are affected. 

For the 1D Spectral files, PROVn was capped at 1 resulting in 95 files with incorrect PROV keywords

For the 2D Spectral files, PROVn was capped at 4 resulting in 57 files with incorrect PROV keywords

For the 1D Binary Table Spectra files, PROVn was capped at 8 resulting in 10 files with incorrect PROV keywords (the same 10 referenced by Alberto)

---

[R02]. MJD-END information must be revised

In August we advised you to adopt the following algorithm for the MJD-END of SOFI spectra:
MJD-END = last of the MJD-OBS of the raw files used + NDIT * (DIT + 1.8)/86400.
The current MJD-END value appears to be 1.8 * NDIT seconds shorter, given that you did not consider the overhead. Please revise those values.

@done: Code was in place but was not running under my crontab. It is running now and MJD-END values have been corrected.

Notes:

For the file SN2012ht_20130113_GR_merge_56478_1_sb.fits

We have the following values:

    mjd_obs: 56306.350798

    ndit: 3

    ncombine: 4

    prov1: SOFI.2013-01-14T08:25:08.947.fits

    prov2: SOFI.2013-01-14T08:33:50.283.fits

    prov3: SOFI.2013-01-14T08:42:21.781.fits

    prov4: SOFI.2013-01-14T08:51:03.115.fits

Here are the details of the last raw file used to create the final file:

    mjd_obs: 56306.368786

    exptime: 120

    filename: SOFI.2013-01-14T08:51:03.115.fits

Therefore:

        MJD-END = MJD-OBS + NDIT * (DIT + 1.8)/86400

            = 56306.368786 + 3 * (120 + 1.8)/86400

            = 56306.373015167

---

[R03]. EXPTIME of SOFI images must be corrected

Spot checks for SOFI images showed:
EXPTIME = DIT
TEXPTIME= DIT * NDIT * NCOMBINE

(Images used for the spot check:
BD174708003_20120915_J_merge_56474_1.fits and
LSQ12hxg_20130103_J_merge_56475_1.fits
)
The science data product standard requires EXPTIME to be the total integration time per pixel
“obtained in at least 50% of the image array taking into account the chosen offset pattern”;
In the cases I checked the offsets between the coadded images were not too big (less than half the fov)
and therefore I would expect EXPTIME to be DIT * NDIT * NCOMBINE,
(and hence identical to the TEXPTIME, which is set correctly).

Please make sure the EXPTIMEs comply with the ESO Data Product Standard definition (section 2.1).

The definition for NCOMBINE:

    Number of raw science data files that were combined to generate this data product. Calibration data files do not contribute to this count.

and the fits header comment

    # of combined raw science data files

Having confirmed with Alberto, the NCOMBINE of the images should always be the number of on-source frames that went into the final image. We were setting NCOMBINE = number of on-source & off-source frames for those SOFI images where the sky is measured away from the source.

@done: NCOMBINE has been set to always be the number of on-source frames only - 164 images & weights corrected.

    update view_sofi_imaging_reduced set NCOMBINE = 8 where NCOMBINE = 16 and prov9 is null;

    update view_sofi_imaging_reduced set NCOMBINE = 8 where NCOMBINE = 8 and prov5 is null;

@done: set exptime = texptime

    update view_sofi_imaging_reduced set exptime = ndit*dit*ncombine;

Notes:

For BD174708003_20120915_J_merge_56474_1.fits:

texptime: 24

 exptime: 1.5

    ndit: 4

ncombine: 4

So for the exptime:

    exptime = DIT * NDIT * NCOMBINE

                        = 1.5 * 4 * 4

                        = 24

                        = TEXPTIME

---

[R04]. OBJECT names containing spurious “.dat” must be cured

List of affected objects:
eg131.dat
eg21.dat
eg274.dat
feige110.dat
gd153.dat
gd71.dat
l745a.dat
ltt3218.dat
ltt3864.dat
ltt7379.dat
Please remove the .dat from both the object name and the file name.

@done

---

[R05]. OBJECT name without a sign?

I found an object name (MASTEROTJ081031.62111450.2) which does not show
the sign; was this meant to be: MASTEROTJ081031.62+111450.2?
If so, please correct.

@done

---

[R06]. Removal of CDELT1 and CDELT2

All EFOSC2 and SOFI image headers contain both the CD matrix and the CDELT1 and CDELT2 keywords.
The CDELTn are not compatible with the CD matrix nor in their values (CDELTn = 1 or 2 degrees),
nor formally (it is against the ESO Data Interface Control Document (DICD) to have both CD and CDELT
in the header).
Please remove the CDELT1 and CDELT2 from all images.

@done

---

[R07]. Submitted FITS files that do not qualify as science data products

There are 42 images with undetermined or improbable values
for relevant keywords such as ABMAGLIM, PSF_FWHM,CRDERn, etc.:

• Images with wrong ABMAGLIM (with values as low as 1.3)
  (eg, ABMAGLIM=3.058 for SN2009ip_20121204_Ks_merge_56475_1.fits),

• Images with negative ABMAGSAT
  (e.g. ABMAGSAT= –11.2 for LSQ12dwl_20120915_H_merge_56474_1.fits)

• Images with invalid PSF_FWHM or ELLIPTIC = 9999
  (e.g. PSF_FWHM= 9999.0 for SN2012dy_20120916_H_merge_56474_1.fits)

• Images with CRDERn or CSYERn = 9999

SJS to check all images before we decide what to do 

There are 19 EFOSC Images with CRDER1/CRDER2/CSYER1/CSYER2 = 9999. Having pulled out the number of sources used to determine the astrometric solution from the ASTROMET keyword, these images are the same images that have less than 3 sources to use for the solution.

@review - Dave: I suggest trashing the 19 images above.

22 SOFI images had PSF_FWHM = 9999   - I didn’t realised PSF_FWHM hadn’t been updated in the fits headers of these images. Cosimo had sent through the images with an ascii file of the PSF measurements, which I incorrectly assumed was just for reference (see email with subject “SOFI files to fix” from 26th August 2013). BUT these images still have ABMAGSAT < 0 and ELLIPTIC = 9999 as they failed PESSTOASTRO - Cosimo had to measure the PSF manually. 

SJS to check all images before we decide what to do 

Those images do not qualify as science data products and cannot be ingested within Phase 3.

Please consider providing the correct metadata or, if not possible, consider
submitting them as ancillary files associated to the main SCIENCE product.
In the latter case, please then:

• use the ASSOC mechanism setting ASSOCi = ‘ANCILLARY.IMAGE’ in the SCIENCE product
  (see: http://www.eso.org/sci/observing/phase3/faq.html#non_calibrated_image);

• drop the PRODCATG keyword (it cannot be present in ancillary files);

• Add a paragraph to the release description describing them
  and explaining what these files could be good for (e.g. could photometry be
  derived from them? etc.);

• Update accordingly Table 4 of the release description.

The list of the 42 affected files is here provided:
CSS121015–004244+132_20121212_U640_56463_1.fits
CSS121015–004244+132_20121212_i705_56463_1_fr.fits
CSS121015–004244+132_20121212_r784_56463_1.fits
LSQ12dlf_20130104_B639_56463_1.fits
LSQ12dwl_20120915_H_merge_56474_1.fits
LSQ13sj_20130403_Ks_merge_56475_1.fits
SN2009ip_20121008_U640_56462_1.fits
SN2009ip_20121204_Ks_merge_56475_1.fits
SN2012dy_20120916_H_merge_56474_1.fits
SN2012dy_20121121_H_merge_56475_1.fits
SN2012ec_20120818_z623_56462_1.fits
SN2012fr_20121113_H_merge_56475_1.fits
SN2012fr_20121113_J_merge_56475_1.fits
SN2012fr_20130318_H_merge_56475_1.fits
SN2012hs_20130119_R642_56463_3.fits
SN2012hs_20130119_i705_56463_2_fr.fits
SN2012hs_20130119_i705_56463_3_fr.fits
SN2012ht_20130103_H_merge_56475_1.fits
SN2013F_20130113_Ks_merge_56475_1.fits
SN2013F_20130128_Ks_merge_56475_1.fits
SN2013K_20130128_H_merge_56475_1.fits
SN2013K_20130207_J_merge_56475_1.fits
SN2013K_20130221_H_merge_56475_1.fits
SN2013K_20130305_H_merge_56475_1.fits
SN2013K_20130318_H_merge_56475_1.fits
SN2013K_20130318_J_merge_56475_1.fits
SN2013K_20130417_H_merge_56475_1.fits
SN2013am_20130403_H_merge_56475_1.fits
SN2013am_20130411_H_merge_56475_1.fits
SN2013am_20130411_Ks_merge_56475_1.fits
SN2013am_20130417_J_merge_56475_1.fits
SN2013am_20130417_Ks_merge_56475_1.fits
acq_CSS120328–104902–072_20120412_g782_56462_16.fits
acq_CSS121015–004244+132_20121212_R642_56463_2.fits
acq_LSQ12cer_20120428_g782_56462_2.fits
acq_LSQ12cer_20120428_g782_56462_3.fits
acq_LSQ12ezn_20120925_R642_56462_8.fits
acq_LSQ12fjf_20121016_V641_56462_5.fits
acq_LSQ12gpw_20121222_R642_56463_12.fits
acq_LSQ12gpw_20121222_R642_56463_13.fits
acq_LSQ12hja_20121212_V641_56463_5.fits
acq_PSNJ13371970–2953486_20130119_V641_56463_6.fits

---

DATA

[R08]. Inaccurate astrometric solutions must be cured.

Two sources were considered in a spot check: CSS121015–004244+132 and SN2013am.
The position of each source was measured onto the various pertaining images,
and a scatter was noticed.
For the first source, the shift in position varied from a sub-arcsec level to 45“ on EFOSC2 images.
For the second source, the shift in position reached 70” on SOFI images.
In some cases the scatter was inline with the astrometric error recorded in the header
(CRDERn and CSYERn), in some other cases the scatter was much larger than
the recorded astrometric error.

EFOSC examples:
The following table contains 4 images, their astrometric error (from the CRDERn and CSYERn keywords),
and the measured spurious shifts between the individual image and image #1.
1) acq_CSS121015–004244+132_20121205_R642_56463_2.fits astrom.err. ~0.3“ Shift = 0.0
2) acq_CSS121015–004244+132_20121212_R642_56463_2.fits astrom.err. = 9999.0 Shift ~ 30”
3) acq_CSS121015–004244+132_20121212_R642_56463_3.fits astrom.err. ~1.5“ Shift ~ 45”
4) CSS121015–004244+132_20121112_B639_56462_1.fits astom.err. ~0.4“ Shift ~ 1.5”
Excluding image 2 which is not a valid data product (as per [R07]),
the 3rd and 4th images show a shift that is much larger than the provided astrometric error.

SOFI examples: SN2013am_201304*1.fits images show big relative shifts of more than 1 arcmin.
Once excluded the images with an unknown (9999.0) astrometric precision, there is still one image
SN2013am_20130411_J_merge_56475_1.fits which shows the SN off by ~57" from its true position;
That image is characterized by a large astrometric error of more than 0.5 degrees (more than 6 times
the field of view of the camera!).
In total there are 35 images with CRDER1 and/or CRDER2 greater than the field of view of the camera,
and up to 5 degrees.

There's a bug in the pipeline so that CRDER1 and CRDER2 are incorrectly reported a factor of sqrt(2) below their true values.