eso phase III data validation report 20131125.We received the following email from Alberto on 20131125:
@done: Dave to fix this issue [SJS to check also] Notes: There was an issue with the writing out of the fits files so that although the PROVn keywords are all recorded in the database, they were not all printed to the FITS headers. As this was only a print-out issue no other keywords are affected. For the 1D Spectral files, PROVn was capped at 1 resulting in 95 files with incorrect PROV keywords For the 2D Spectral files, PROVn was capped at 4 resulting in 57 files with incorrect PROV keywords For the 1D Binary Table Spectra files, PROVn was capped at 8 resulting in 10 files with incorrect PROV keywords (the same 10 referenced by Alberto)
@done: Code was in place but was not running under my crontab. It is running now and MJD-END values have been corrected. Notes: For the file SN2012ht_20130113_GR_merge_56478_1_sb.fits We have the following values: mjd_obs: 56306.350798 ndit: 3 ncombine: 4 prov1: SOFI.2013-01-14T08:25:08.947.fits prov2: SOFI.2013-01-14T08:33:50.283.fits prov3: SOFI.2013-01-14T08:42:21.781.fits prov4: SOFI.2013-01-14T08:51:03.115.fits Here are the details of the last raw file used to create the final file: mjd_obs: 56306.368786 exptime: 120 filename: SOFI.2013-01-14T08:51:03.115.fits Therefore: MJD-END = MJD-OBS + NDIT * (DIT + 1.8)/86400 = 56306.368786 + 3 * (120 + 1.8)/86400 = 56306.373015167
The definition for NCOMBINE: Number of raw science data files that were combined to generate this data product. Calibration data files do not contribute to this count. and the fits header comment # of combined raw science data files Having confirmed with Alberto, the NCOMBINE of the images should always be the number of on-source frames that went into the final image. We were setting NCOMBINE = number of on-source & off-source frames for those SOFI images where the sky is measured away from the source. @done: NCOMBINE has been set to always be the number of on-source frames only - 164 images & weights corrected. update view_sofi_imaging_reduced set NCOMBINE = 8 where NCOMBINE = 16 and prov9 is null; update view_sofi_imaging_reduced set NCOMBINE = 8 where NCOMBINE = 8 and prov5 is null; @done: set exptime = texptime update view_sofi_imaging_reduced set exptime = ndit*dit*ncombine; Notes: For BD174708003_20120915_J_merge_56474_1.fits: texptime: 24 exptime: 1.5 ndit: 4 ncombine: 4 So for the exptime: exptime = DIT * NDIT * NCOMBINE = 1.5 * 4 * 4 = 24 = TEXPTIME
@done
@done
@done
SJS to check all images before we decide what to do There are 19 EFOSC Images with CRDER1/CRDER2/CSYER1/CSYER2 = 9999. Having pulled out the number of sources used to determine the astrometric solution from the ASTROMET keyword, these images are the same images that have less than 3 sources to use for the solution. mysql> select currentFilename, crder1, crder2, ELLIPTIC, PSF_FWHM, abmaglim, ABMAGSAT, astromet_sources from view_efosc_imaging_reduced where astromet_sources < 3 and DATA_REL = "SSDR1" order by astromet_sources; +------------------------------------------------------+--------+--------+----------+----------+----------+----------+------------------+ | currentFilename | CRDER1 | CRDER2 | ELLIPTIC | PSF_FWHM | ABMAGLIM | ABMAGSAT | astromet_sources | +------------------------------------------------------+--------+--------+----------+----------+----------+----------+------------------+ | SN2012ec_20120818_z623_56462_1.fits | 9999 | 9999 | 9999 | 9999 | 8.83 | -9.97 | 0 | | acq_LSQ12fjf_20121016_V641_56462_5.fits | 9999 | 9999 | 9999 | 9999 | 11.93 | -8.50 | 0 | | acq_CSS120328-104902-072_20120412_g782_56462_16.fits | 9999 | 9999 | 9999 | 9999 | 11.21 | -8.52 | 0 | | acq_LSQ12cer_20120428_g782_56462_2.fits | 9999 | 9999 | 9999 | 9999 | 12.45 | -8.52 | 0 | | LSQ12dlf_20130104_B639_56463_1.fits | 9999 | 9999 | 9999 | 9999 | 11.15 | -8.55 | 0 | | acq_LSQ12hja_20121212_V641_56463_5.fits | 9999 | 9999 | 9999 | 9999 | 12.45 | -8.50 | 1 | | acq_CSS121015-004244+132_20121212_R642_56463_2.fits | 9999 | 9999 | 9999 | 9999 | 12.00 | -8.40 | 1 | | acq_LSQ12cer_20120428_g782_56462_3.fits | 9999 | 9999 | 9999 | 9999 | 13.46 | -8.52 | 1 | | SN2012hs_20130119_i705_56463_2_fr.fits | 9999 | 9999 | 9999 | 9999 | 11.59 | -9.25 | 1 | | acq_LSQ12gpw_20121222_R642_56463_13.fits | 9999 | 9999 | 9999 | 9999 | 11.31 | -8.40 | 1 | | CSS121015-004244+132_20121212_U640_56463_1.fits | 9999 | 9999 | 9999 | 9999 | 13.10 | -10.69 | 1 | | acq_LSQ12gpw_20121222_R642_56463_12.fits | 9999 | 9999 | 9999 | 9999 | 11.31 | -8.40 | 2 | | CSS121015-004244+132_20121212_r784_56463_1.fits | 9999 | 9999 | 9999 | 9999 | 13.07 | -9.11 | 2 | | CSS121015-004244+132_20121212_i705_56463_1_fr.fits | 9999 | 9999 | 9999 | 9999 | 12.38 | -9.23 | 2 | | SN2012hs_20130119_R642_56463_3.fits | 9999 | 9999 | 9999 | 9999 | 12.45 | -8.40 | 2 | | SN2009ip_20121008_U640_56462_1.fits | 9999 | 9999 | 9999 | 9999 | 10.49 | -10.69 | 2 | | acq_LSQ12ezn_20120925_R642_56462_8.fits | 9999 | 9999 | 9999 | 9999 | 11.73 | -8.40 | 2 | | acq_PSNJ13371970-2953486_20130119_V641_56463_6.fits | 9999 | 9999 | 9999 | 9999 | 13.33 | -8.50 | 2 | | SN2012hs_20130119_i705_56463_3_fr.fits | 9999 | 9999 | 9999 | 9999 | 11.60 | -9.25 | 2 | +------------------------------------------------------+--------+--------+----------+----------+----------+----------+------------------+ 19 rows in set (0.07 sec) @review - Dave: I suggest trashing the 19 images above. 22 SOFI images had PSF_FWHM = 9999 - I didn’t realised PSF_FWHM hadn’t been updated in the fits headers of these images. Cosimo had sent through the images with an ascii file of the PSF measurements, which I incorrectly assumed was just for reference (see email with subject “SOFI files to fix” from 26th August 2013). BUT these images still have ABMAGSAT < 0 and ELLIPTIC = 9999 as they failed PESSTOASTRO - Cosimo had to measure the PSF manually. mysql> select currentFilename, crder1, crder2, ELLIPTIC, PSF_FWHM, abmaglim, ABMAGSAT, astromet_sources from view_sofi_imaging_reduced where crder1 = 9999 or crder2 = 9999 or ELLIPTIC =9999 or PSF_FWHM = 9999 and DATA_REL = "SSDR1" order by astromet_sources; +-----------------------------------------+------------------+------------------+----------+----------+---------------+----------------+------------------+ | currentFilename | CRDER1 | CRDER2 | ELLIPTIC | PSF_FWHM | ABMAGLIM | ABMAGSAT | astromet_sources | +-----------------------------------------+------------------+------------------+----------+----------+---------------+----------------+------------------+ | SN2012fr_20121113_J_merge_56475_1.fits | 9999 | 9999 | 9999 | 1.675 | 9.17401740549 | 9999 | 3 | | LSQ12dwl_20120915_H_merge_56474_1.fits | 0.00283333333333 | 0.00402777777778 | 9999 | 2.03 | 5.21133444414 | -11.2384189664 | 12 | | SN2013F_20130113_Ks_merge_56475_1.fits | 0.00341666666667 | 0.00235555555556 | 9999 | 1.55 | 5.09747048342 | -10.6207962701 | 13 | | SN2012dy_20120916_H_merge_56474_1.fits | 0.004 | 0.00291666666667 | 9999 | 2.373 | 4.00403377736 | -12.3386947497 | 15 | | LSQ13sj_20130403_Ks_merge_56475_1.fits | 0.00338888888889 | 0.00397222222222 | 9999 | 1.501 | 3.47515008939 | -12.625218593 | 15 | | SN2013am_20130411_Ks_merge_56475_1.fits | 0.00391666666667 | 0.00255555555556 | 9999 | 1.15 | 3.63595520193 | -12.5253840644 | 15 | | SN2013am_20130403_H_merge_56475_1.fits | 0.00325 | 0.0035 | 9999 | 1.3383 | 3.69820525013 | -12.621870425 | 16 | | SN2012fr_20121113_H_merge_56475_1.fits | 0.00363888888889 | 0.00261944444444 | 9999 | 9999 | 4.79572610235 | -11.6744222026 | 16 | | SN2013am_20130417_J_merge_56475_1.fits | 0.00369444444444 | 0.00433333333333 | 9999 | 1.695 | 6.18151464493 | -11.2671403862 | 17 | | SN2012ht_20130103_H_merge_56475_1.fits | 0.0035 | 0.00291666666667 | 9999 | 1.8075 | 4.59102742578 | -12.1525633791 | 17 | | SN2012dy_20121121_H_merge_56475_1.fits | 0.00363888888889 | 0.00338888888889 | 9999 | 1.8087 | 3.83496683545 | -12.1821713557 | 18 | | SN2013am_20130411_H_merge_56475_1.fits | 0.00297222222222 | 0.0045 | 9999 | 1.36 | 4.49081617551 | -11.9452312935 | 18 | | SN2013am_20130417_Ks_merge_56475_1.fits | 0.00352777777778 | 0.00411111111111 | 9999 | 1.415 | 4.00760891997 | -11.7792217574 | 18 | | SN2009ip_20121204_Ks_merge_56475_1.fits | 0.00344444444444 | 0.00294444444444 | 9999 | 1.4762 | 3.05823872228 | -12.8473434521 | 19 | | SN2013F_20130128_Ks_merge_56475_1.fits | 0.00355555555556 | 0.00202777777778 | 9999 | 2.1716 | 5.08293601165 | -10.7597446795 | 19 | | SN2012fr_20130318_H_merge_56475_1.fits | 0.00422222222222 | 0.00327777777778 | 9999 | 2.1683 | 1.95286088311 | -13.3352789192 | 21 | | SN2013K_20130221_H_merge_56475_1.fits | 0.00358333333333 | 0.00311111111111 | 9999 | 1.46 | 4.3973112019 | -11.7492837102 | 22 | | SN2013K_20130417_H_merge_56475_1.fits | 0.0035 | 0.00327777777778 | 9999 | 1.5566 | 4.37375004615 | -11.89049238 | 23 | | SN2013K_20130305_H_merge_56475_1.fits | 0.00319444444444 | 0.00375 | 9999 | 1.81 | 4.44049825535 | -11.8158976648 | 24 | | SN2013K_20130318_H_merge_56475_1.fits | 0.00352777777778 | 0.00386111111111 | 9999 | 1.8775 | 4.13022047254 | -11.4069768211 | 26 | | SN2013K_20130318_J_merge_56475_1.fits | 0.00366666666667 | 0.00388888888889 | 9999 | 2.0667 | 5.14621809123 | -11.864888123 | 27 | | SN2013K_20130207_J_merge_56475_1.fits | 0.00388888888889 | 0.00336111111111 | 9999 | 2.025 | 4.69180801318 | -12.7431269977 | 27 | | SN2013K_20130128_H_merge_56475_1.fits | 0.00380555555556 | 0.00336111111111 | 9999 | 1.85 | 4.75205872593 | -11.2571493358 | 28 | +-----------------------------------------+------------------+------------------+----------+----------+---------------+----------------+------------------+ 23 rows in set (0.04 sec) @review - Dave: I suggest Cosimo/Stefano try and rerun PESSTOASTRO on these files to workout why they are failing. SJS to check all images before we decide what to do
ISSUE 1. Should be: CRDER1 = rmsx * 3600 but instead: CRDER1 = rmsx * 3600 / sqrt(2) @done: dave extracted the rmsx and rmsy values out of the ASTROMET keyword, converted this to degrees and overwrote the incorrect CRDERn values. ISSUE 2 There are 23 SOFI images with CRDER1/CRDER2 = 9999. Same 23 SOFI images from [R07] above. These are the only images with astromet rmsx/rmsy > 10.0 arcsec. The pipeline must decide these images have failed astrometric calibrations? ISSUE 3. There are 34 SOFI images with CRDER1 and/or CRDER2 greater than the field of view of the camera. All the images have the sqrt(2) bug when converting from astromet rmsx/rmsy to CRDER1/CRDER2 - but the 34 images with the huge CRDER1/CRDER2 also have a compounded 3600 scaling factor error (arcsec --> degree obviously). The conversion wasn't done when Cosimo ran PESSTOASTRO for some reason. So CRDER1 = rmsx * 3600 but instead CRDER1 = rmsx / sqrt(2) @done: extract out rmsx and rmsy and convert correctly into CRDER1 and CRDER2 ISSUE 4. For SOFI I manually checked a few images for SN2013am and SN2013ec and found some images had reasonable values reported in the (corrected) CRDER1 and CRDER2 values, but many images had huge measured shifts ( > 1 arcmin) which were not reflected in the reported CRDER1, CRDER2 values. These massive shifts seem to have some correlation with the telescope pointing of the image. So for 1 telescope pointing the shifts are reasonable, but for another pointing they are catastrophic. I did the same measurements for EFOSC images of CSS121015-004244 and found the same issue. See the spreadsheet below (sheets "sofi imaging SN2013am checks" and "efosc imaging CSS121015 checks ").
@review: what should we do with these images? SJS to check all images before we decide what to do
@action: Dave will check these numbers again before submitting.
Here is the image refered to: BD174708003_20120915_J_merge_56474_1.fits. You can see stripping in the south west quadrant of the image. @action: SJS to add details to release description.
Table 3 has has 41 “followup” objects with a total of 516 efosc + 95 sofi spectra (661 total). Technically these included the classification spectrum and the subsequent followup spectra for each object. Of the remaining 208 objects which do not appear in table 3 (total 257 objects - 41 followup objects = 208) we have 298 spectra. Please note that in some cases 2 or more classification spectra are taken per object – all files are uploaded to ESO so there should be more GR13 spectral classification files present than the number of objects classified. This makes a total of 814 EFOSC spectra uploaded to ESO (516 + 298). @action: SJS or Dave to change the text to make it clearer.
@review: We may need to add another keyword in the FITS headers - there’s no other way unless we have a public webpage listing objects classified and objects followed.
@action: SJS to change text
@action: SJS to add to text
@action: SJS to add to text
|