Embedded Files
eso phase III data validation report 20131125.
eso phase III data validation report 20131125.
We received the following email from Alberto on 20131125:
Your data (spectra and images) submitted under:
> Data Collection PESSTO
> Release Number 1
> Data Provider: Stephen Smartt
> Close Date: 24/Oct/2013
have now been reviewed regarding the overall consistency and compliance
with the ESO/SDP standard in order to ensure their successful integration
into the ESO archive. The purpose of this message is to provide you with
the results of this process and to request feedback where needed.
SUMMARY:
The data substantially meet the required ESO standard. However
a few problems have been found in the course of the validation
process, which must be addressed before the data can be published
through the ESO science archive to the community.
LIST OF PROBLEMS:
METADATA
[R01]. NCOMBINE vs PROVn inconsistencies must be cured
It is expected that the NCOMBINE value is equal to the number of PROVn keywords in your data.
This is not the case for 10 spectra (see list here below) and for more than 150 images.
Either the PROVn list is incomplete, or the counts in NCOMBINE are incorrect.
Example of affected images:
SN2013ak_20130417_Ks_merge_56475_1.fits
LSQ13fn_20130207_Ks_merge_56475_1.fits
etc…
The list of 10 spectra affected by this problem are:
SN2009ip_20130417_GB_merge_56478_1_sb.fits
SN2012ca_20120809_GB_merge_56476_1_sb.fits
SN2012ca_20121113_GB_merge_56477_1_sb.fits
SN2012ca_20130311_GB_merge_56478_1_sb.fits
SN2012ca_20130311_GR_merge_56478_1_sb.fits
SN2012ec_20121204_GB_merge_56477_1_sb.fits
SN2012ec_20130103_GB_merge_56478_1_sb.fits
SN2012ec_20130113_GB_merge_56478_1_sb.fits
SN2012ec_20130207_GB_merge_56478_1_sb.fits
SN2012hs_20130303_GB_merge_56478_1_sb.fits
Please revise and correct those 10 spectra, and please check, and when
needed correct, all SOFI images.
Reminder: if the correction to this problem requires the modification
of the PROVn information, please remember to update consistently
the MJD-END, EXPTIME, and ALL other affected keywords.
@done: Dave to fix this issue [SJS to check also]
Notes:
There was an issue with the writing out of the fits files so that although the PROVn keywords are all recorded in the database, they were not all printed to the FITS headers. As this was only a print-out issue no other keywords are affected.
For the 1D Spectral files, PROVn was capped at 1 resulting in 95 files with incorrect PROV keywords
For the 2D Spectral files, PROVn was capped at 4 resulting in 57 files with incorrect PROV keywords
For the 1D Binary Table Spectra files, PROVn was capped at 8 resulting in 10 files with incorrect PROV keywords (the same 10 referenced by Alberto)
[R02]. MJD-END information must be revised
In August we advised you to adopt the following algorithm for the MJD-END of SOFI spectra:
MJD-END = last of the MJD-OBS of the raw files used + NDIT * (DIT + 1.8)/86400.
The current MJD-END value appears to be 1.8 * NDIT seconds shorter, given that you did not consider the overhead. Please revise those values.
@done: Code was in place but was not running under my crontab. It is running now and MJD-END values have been corrected.
Notes:
For the file SN2012ht_20130113_GR_merge_56478_1_sb.fits
We have the following values:
mjd_obs: 56306.350798
ndit: 3
ncombine: 4
prov1: SOFI.2013-01-14T08:25:08.947.fits
prov2: SOFI.2013-01-14T08:33:50.283.fits
prov3: SOFI.2013-01-14T08:42:21.781.fits
prov4: SOFI.2013-01-14T08:51:03.115.fits
Here are the details of the last raw file used to create the final file:
mjd_obs: 56306.368786
exptime: 120
filename: SOFI.2013-01-14T08:51:03.115.fits
Therefore:
MJD-END = MJD-OBS + NDIT * (DIT + 1.8)/86400
= 56306.368786 + 3 * (120 + 1.8)/86400
= 56306.373015167
[R03]. EXPTIME of SOFI images must be corrected
Spot checks for SOFI images showed:
EXPTIME = DIT
TEXPTIME= DIT * NDIT * NCOMBINE
(Images used for the spot check:
BD174708003_20120915_J_merge_56474_1.fits and
LSQ12hxg_20130103_J_merge_56475_1.fits
)
The science data product standard requires EXPTIME to be the total integration time per pixel
“obtained in at least 50% of the image array taking into account the chosen offset pattern”;
In the cases I checked the offsets between the coadded images were not too big (less than half the fov)
and therefore I would expect EXPTIME to be DIT * NDIT * NCOMBINE,
(and hence identical to the TEXPTIME, which is set correctly).
Please make sure the EXPTIMEs comply with the ESO Data Product Standard definition (section 2.1).
The definition for NCOMBINE:
Number of raw science data files that were combined to generate this data product. Calibration data files do not contribute to this count.
and the fits header comment
# of combined raw science data files
Having confirmed with Alberto, the NCOMBINE of the images should always be the number of on-source frames that went into the final image. We were setting NCOMBINE = number of on-source & off-source frames for those SOFI images where the sky is measured away from the source.
@done: NCOMBINE has been set to always be the number of on-source frames only - 164 images & weights corrected.
update view_sofi_imaging_reduced set NCOMBINE = 8 where NCOMBINE = 16 and prov9 is null;
update view_sofi_imaging_reduced set NCOMBINE = 8 where NCOMBINE = 8 and prov5 is null;
@done: set exptime = texptime
update view_sofi_imaging_reduced set exptime = ndit*dit*ncombine;
Notes:
For BD174708003_20120915_J_merge_56474_1.fits:
texptime: 24
exptime: 1.5
ndit: 4
ncombine: 4
So for the exptime:
exptime = DIT * NDIT * NCOMBINE
= 1.5 * 4 * 4
= 24
= TEXPTIME
[R04]. OBJECT names containing spurious “.dat” must be cured
List of affected objects:
eg131.dat
eg21.dat
eg274.dat
feige110.dat
gd153.dat
gd71.dat
l745a.dat
ltt3218.dat
ltt3864.dat
ltt7379.dat
Please remove the .dat from both the object name and the file name.
@done
[R05]. OBJECT name without a sign?
I found an object name (MASTEROTJ081031.62111450.2) which does not show
the sign; was this meant to be: MASTEROTJ081031.62+111450.2?
If so, please correct.
@done
[R06]. Removal of CDELT1 and CDELT2
All EFOSC2 and SOFI image headers contain both the CD matrix and the CDELT1 and CDELT2 keywords.
The CDELTn are not compatible with the CD matrix nor in their values (CDELTn = 1 or 2 degrees),
nor formally (it is against the ESO Data Interface Control Document (DICD) to have both CD and CDELT
in the header).
Please remove the CDELT1 and CDELT2 from all images.
@done
[R07]. Submitted FITS files that do not qualify as science data products
There are 42 images with undetermined or improbable values
for relevant keywords such as ABMAGLIM, PSF_FWHM,CRDERn, etc.:
Images with wrong ABMAGLIM (with values as low as 1.3)
(eg, ABMAGLIM=3.058 for SN2009ip_20121204_Ks_merge_56475_1.fits),
Images with negative ABMAGSAT
(e.g. ABMAGSAT= –11.2 for LSQ12dwl_20120915_H_merge_56474_1.fits)
Images with invalid PSF_FWHM or ELLIPTIC = 9999
(e.g. PSF_FWHM= 9999.0 for SN2012dy_20120916_H_merge_56474_1.fits)
Images with CRDERn or CSYERn = 9999
SJS to check all images before we decide what to do
There are 19 EFOSC Images with CRDER1/CRDER2/CSYER1/CSYER2 = 9999. Having pulled out the number of sources used to determine the astrometric solution from the ASTROMET keyword, these images are the same images that have less than 3 sources to use for the solution.
mysql> select currentFilename, crder1, crder2, ELLIPTIC, PSF_FWHM, abmaglim, ABMAGSAT, astromet_sources from view_efosc_imaging_reduced where astromet_sources < 3 and DATA_REL = "SSDR1" order by astromet_sources;
+------------------------------------------------------+--------+--------+----------+----------+----------+----------+------------------+
| currentFilename | CRDER1 | CRDER2 | ELLIPTIC | PSF_FWHM | ABMAGLIM | ABMAGSAT | astromet_sources |
+------------------------------------------------------+--------+--------+----------+----------+----------+----------+------------------+
| SN2012ec_20120818_z623_56462_1.fits | 9999 | 9999 | 9999 | 9999 | 8.83 | -9.97 | 0 |
| acq_LSQ12fjf_20121016_V641_56462_5.fits | 9999 | 9999 | 9999 | 9999 | 11.93 | -8.50 | 0 |
| acq_CSS120328-104902-072_20120412_g782_56462_16.fits | 9999 | 9999 | 9999 | 9999 | 11.21 | -8.52 | 0 |
| acq_LSQ12cer_20120428_g782_56462_2.fits | 9999 | 9999 | 9999 | 9999 | 12.45 | -8.52 | 0 |
| LSQ12dlf_20130104_B639_56463_1.fits | 9999 | 9999 | 9999 | 9999 | 11.15 | -8.55 | 0 |
| acq_LSQ12hja_20121212_V641_56463_5.fits | 9999 | 9999 | 9999 | 9999 | 12.45 | -8.50 | 1 |
| acq_CSS121015-004244+132_20121212_R642_56463_2.fits | 9999 | 9999 | 9999 | 9999 | 12.00 | -8.40 | 1 |
| acq_LSQ12cer_20120428_g782_56462_3.fits | 9999 | 9999 | 9999 | 9999 | 13.46 | -8.52 | 1 |
| SN2012hs_20130119_i705_56463_2_fr.fits | 9999 | 9999 | 9999 | 9999 | 11.59 | -9.25 | 1 |
| acq_LSQ12gpw_20121222_R642_56463_13.fits | 9999 | 9999 | 9999 | 9999 | 11.31 | -8.40 | 1 |
| CSS121015-004244+132_20121212_U640_56463_1.fits | 9999 | 9999 | 9999 | 9999 | 13.10 | -10.69 | 1 |
| acq_LSQ12gpw_20121222_R642_56463_12.fits | 9999 | 9999 | 9999 | 9999 | 11.31 | -8.40 | 2 |
| CSS121015-004244+132_20121212_r784_56463_1.fits | 9999 | 9999 | 9999 | 9999 | 13.07 | -9.11 | 2 |
| CSS121015-004244+132_20121212_i705_56463_1_fr.fits | 9999 | 9999 | 9999 | 9999 | 12.38 | -9.23 | 2 |
| SN2012hs_20130119_R642_56463_3.fits | 9999 | 9999 | 9999 | 9999 | 12.45 | -8.40 | 2 |
| SN2009ip_20121008_U640_56462_1.fits | 9999 | 9999 | 9999 | 9999 | 10.49 | -10.69 | 2 |
| acq_LSQ12ezn_20120925_R642_56462_8.fits | 9999 | 9999 | 9999 | 9999 | 11.73 | -8.40 | 2 |
| acq_PSNJ13371970-2953486_20130119_V641_56463_6.fits | 9999 | 9999 | 9999 | 9999 | 13.33 | -8.50 | 2 |
| SN2012hs_20130119_i705_56463_3_fr.fits | 9999 | 9999 | 9999 | 9999 | 11.60 | -9.25 | 2 |
+------------------------------------------------------+--------+--------+----------+----------+----------+----------+------------------+
19 rows in set (0.07 sec)
@review - Dave: I suggest trashing the 19 images above.
22 SOFI images had PSF_FWHM = 9999 - I didn’t realised PSF_FWHM hadn’t been updated in the fits headers of these images. Cosimo had sent through the images with an ascii file of the PSF measurements, which I incorrectly assumed was just for reference (see email with subject “SOFI files to fix” from 26th August 2013). BUT these images still have ABMAGSAT < 0 and ELLIPTIC = 9999 as they failed PESSTOASTRO - Cosimo had to measure the PSF manually.
mysql> select currentFilename, crder1, crder2, ELLIPTIC, PSF_FWHM, abmaglim, ABMAGSAT, astromet_sources from view_sofi_imaging_reduced where crder1 = 9999 or crder2 = 9999 or ELLIPTIC =9999 or PSF_FWHM = 9999 and DATA_REL = "SSDR1" order by astromet_sources;
+-----------------------------------------+------------------+------------------+----------+----------+---------------+----------------+------------------+
| currentFilename | CRDER1 | CRDER2 | ELLIPTIC | PSF_FWHM | ABMAGLIM | ABMAGSAT | astromet_sources |
+-----------------------------------------+------------------+------------------+----------+----------+---------------+----------------+------------------+
| SN2012fr_20121113_J_merge_56475_1.fits | 9999 | 9999 | 9999 | 1.675 | 9.17401740549 | 9999 | 3 |
| LSQ12dwl_20120915_H_merge_56474_1.fits | 0.00283333333333 | 0.00402777777778 | 9999 | 2.03 | 5.21133444414 | -11.2384189664 | 12 |
| SN2013F_20130113_Ks_merge_56475_1.fits | 0.00341666666667 | 0.00235555555556 | 9999 | 1.55 | 5.09747048342 | -10.6207962701 | 13 |
| SN2012dy_20120916_H_merge_56474_1.fits | 0.004 | 0.00291666666667 | 9999 | 2.373 | 4.00403377736 | -12.3386947497 | 15 |
| LSQ13sj_20130403_Ks_merge_56475_1.fits | 0.00338888888889 | 0.00397222222222 | 9999 | 1.501 | 3.47515008939 | -12.625218593 | 15 |
| SN2013am_20130411_Ks_merge_56475_1.fits | 0.00391666666667 | 0.00255555555556 | 9999 | 1.15 | 3.63595520193 | -12.5253840644 | 15 |
| SN2013am_20130403_H_merge_56475_1.fits | 0.00325 | 0.0035 | 9999 | 1.3383 | 3.69820525013 | -12.621870425 | 16 |
| SN2012fr_20121113_H_merge_56475_1.fits | 0.00363888888889 | 0.00261944444444 | 9999 | 9999 | 4.79572610235 | -11.6744222026 | 16 |
| SN2013am_20130417_J_merge_56475_1.fits | 0.00369444444444 | 0.00433333333333 | 9999 | 1.695 | 6.18151464493 | -11.2671403862 | 17 |
| SN2012ht_20130103_H_merge_56475_1.fits | 0.0035 | 0.00291666666667 | 9999 | 1.8075 | 4.59102742578 | -12.1525633791 | 17 |
| SN2012dy_20121121_H_merge_56475_1.fits | 0.00363888888889 | 0.00338888888889 | 9999 | 1.8087 | 3.83496683545 | -12.1821713557 | 18 |
| SN2013am_20130411_H_merge_56475_1.fits | 0.00297222222222 | 0.0045 | 9999 | 1.36 | 4.49081617551 | -11.9452312935 | 18 |
| SN2013am_20130417_Ks_merge_56475_1.fits | 0.00352777777778 | 0.00411111111111 | 9999 | 1.415 | 4.00760891997 | -11.7792217574 | 18 |
| SN2009ip_20121204_Ks_merge_56475_1.fits | 0.00344444444444 | 0.00294444444444 | 9999 | 1.4762 | 3.05823872228 | -12.8473434521 | 19 |
| SN2013F_20130128_Ks_merge_56475_1.fits | 0.00355555555556 | 0.00202777777778 | 9999 | 2.1716 | 5.08293601165 | -10.7597446795 | 19 |
| SN2012fr_20130318_H_merge_56475_1.fits | 0.00422222222222 | 0.00327777777778 | 9999 | 2.1683 | 1.95286088311 | -13.3352789192 | 21 |
| SN2013K_20130221_H_merge_56475_1.fits | 0.00358333333333 | 0.00311111111111 | 9999 | 1.46 | 4.3973112019 | -11.7492837102 | 22 |
| SN2013K_20130417_H_merge_56475_1.fits | 0.0035 | 0.00327777777778 | 9999 | 1.5566 | 4.37375004615 | -11.89049238 | 23 |
| SN2013K_20130305_H_merge_56475_1.fits | 0.00319444444444 | 0.00375 | 9999 | 1.81 | 4.44049825535 | -11.8158976648 | 24 |
| SN2013K_20130318_H_merge_56475_1.fits | 0.00352777777778 | 0.00386111111111 | 9999 | 1.8775 | 4.13022047254 | -11.4069768211 | 26 |
| SN2013K_20130318_J_merge_56475_1.fits | 0.00366666666667 | 0.00388888888889 | 9999 | 2.0667 | 5.14621809123 | -11.864888123 | 27 |
| SN2013K_20130207_J_merge_56475_1.fits | 0.00388888888889 | 0.00336111111111 | 9999 | 2.025 | 4.69180801318 | -12.7431269977 | 27 |
| SN2013K_20130128_H_merge_56475_1.fits | 0.00380555555556 | 0.00336111111111 | 9999 | 1.85 | 4.75205872593 | -11.2571493358 | 28 |
+-----------------------------------------+------------------+------------------+----------+----------+---------------+----------------+------------------+
23 rows in set (0.04 sec)
@review - Dave: I suggest Cosimo/Stefano try and rerun PESSTOASTRO on these files to workout why they are failing.
SJS to check all images before we decide what to do
Those images do not qualify as science data products and cannot be ingested within Phase 3.
Please consider providing the correct metadata or, if not possible, consider
submitting them as ancillary files associated to the main SCIENCE product.
In the latter case, please then:
use the ASSOC mechanism setting ASSOCi = ‘ANCILLARY.IMAGE’ in the SCIENCE product
(see: http://www.eso.org/sci/observing/phase3/faq.html#non_calibrated_image);
drop the PRODCATG keyword (it cannot be present in ancillary files);
Add a paragraph to the release description describing them
and explaining what these files could be good for (e.g. could photometry be
derived from them? etc.);
Update accordingly Table 4 of the release description.
The list of the 42 affected files is here provided:
CSS121015–004244+132_20121212_U640_56463_1.fits
CSS121015–004244+132_20121212_i705_56463_1_fr.fits
CSS121015–004244+132_20121212_r784_56463_1.fits
LSQ12dlf_20130104_B639_56463_1.fits
LSQ12dwl_20120915_H_merge_56474_1.fits
LSQ13sj_20130403_Ks_merge_56475_1.fits
SN2009ip_20121008_U640_56462_1.fits
SN2009ip_20121204_Ks_merge_56475_1.fits
SN2012dy_20120916_H_merge_56474_1.fits
SN2012dy_20121121_H_merge_56475_1.fits
SN2012ec_20120818_z623_56462_1.fits
SN2012fr_20121113_H_merge_56475_1.fits
SN2012fr_20121113_J_merge_56475_1.fits
SN2012fr_20130318_H_merge_56475_1.fits
SN2012hs_20130119_R642_56463_3.fits
SN2012hs_20130119_i705_56463_2_fr.fits
SN2012hs_20130119_i705_56463_3_fr.fits
SN2012ht_20130103_H_merge_56475_1.fits
SN2013F_20130113_Ks_merge_56475_1.fits
SN2013F_20130128_Ks_merge_56475_1.fits
SN2013K_20130128_H_merge_56475_1.fits
SN2013K_20130207_J_merge_56475_1.fits
SN2013K_20130221_H_merge_56475_1.fits
SN2013K_20130305_H_merge_56475_1.fits
SN2013K_20130318_H_merge_56475_1.fits
SN2013K_20130318_J_merge_56475_1.fits
SN2013K_20130417_H_merge_56475_1.fits
SN2013am_20130403_H_merge_56475_1.fits
SN2013am_20130411_H_merge_56475_1.fits
SN2013am_20130411_Ks_merge_56475_1.fits
SN2013am_20130417_J_merge_56475_1.fits
SN2013am_20130417_Ks_merge_56475_1.fits
acq_CSS120328–104902–072_20120412_g782_56462_16.fits
acq_CSS121015–004244+132_20121212_R642_56463_2.fits
acq_LSQ12cer_20120428_g782_56462_2.fits
acq_LSQ12cer_20120428_g782_56462_3.fits
acq_LSQ12ezn_20120925_R642_56462_8.fits
acq_LSQ12fjf_20121016_V641_56462_5.fits
acq_LSQ12gpw_20121222_R642_56463_12.fits
acq_LSQ12gpw_20121222_R642_56463_13.fits
acq_LSQ12hja_20121212_V641_56463_5.fits
acq_PSNJ13371970–2953486_20130119_V641_56463_6.fits
DATA
[R08]. Inaccurate astrometric solutions must be cured.
Two sources were considered in a spot check: CSS121015–004244+132 and SN2013am.
The position of each source was measured onto the various pertaining images,
and a scatter was noticed.
For the first source, the shift in position varied from a sub-arcsec level to 45“ on EFOSC2 images.
For the second source, the shift in position reached 70” on SOFI images.
In some cases the scatter was inline with the astrometric error recorded in the header
(CRDERn and CSYERn), in some other cases the scatter was much larger than
the recorded astrometric error.
EFOSC examples:
The following table contains 4 images, their astrometric error (from the CRDERn and CSYERn keywords),
and the measured spurious shifts between the individual image and image #1.
1) acq_CSS121015–004244+132_20121205_R642_56463_2.fits astrom.err. ~0.3“ Shift = 0.0
2) acq_CSS121015–004244+132_20121212_R642_56463_2.fits astrom.err. = 9999.0 Shift ~ 30”
3) acq_CSS121015–004244+132_20121212_R642_56463_3.fits astrom.err. ~1.5“ Shift ~ 45”
4) CSS121015–004244+132_20121112_B639_56462_1.fits astom.err. ~0.4“ Shift ~ 1.5”
Excluding image 2 which is not a valid data product (as per [R07]),
the 3rd and 4th images show a shift that is much larger than the provided astrometric error.
SOFI examples: SN2013am_201304*1.fits images show big relative shifts of more than 1 arcmin.
Once excluded the images with an unknown (9999.0) astrometric precision, there is still one image
SN2013am_20130411_J_merge_56475_1.fits which shows the SN off by ~57" from its true position;
That image is characterized by a large astrometric error of more than 0.5 degrees (more than 6 times
the field of view of the camera!).
In total there are 35 images with CRDER1 and/or CRDER2 greater than the field of view of the camera,
and up to 5 degrees.
ISSUE 1.
There's a bug in the pipeline so that CRDER1 and CRDER2 are incorrectly reported a factor of sqrt(2) below their true values.
Should be:
CRDER1 = rmsx * 3600
but instead:
CRDER1 = rmsx * 3600 / sqrt(2)
@done: dave extracted the rmsx and rmsy values out of the ASTROMET keyword, converted this to degrees and overwrote the incorrect CRDERn values.
ISSUE 2
There are 23 SOFI images with CRDER1/CRDER2 = 9999. Same 23 SOFI images from [R07] above. These are the only images with astromet rmsx/rmsy > 10.0 arcsec. The pipeline must decide these images have failed astrometric calibrations?
ISSUE 3.
There are 34 SOFI images with CRDER1 and/or CRDER2 greater than the field of view of the camera. All the images have the sqrt(2) bug when converting from astromet rmsx/rmsy to CRDER1/CRDER2 - but the 34 images with the huge CRDER1/CRDER2 also have a compounded 3600 scaling factor error (arcsec --> degree obviously). The conversion wasn't done when Cosimo ran PESSTOASTRO for some reason.
So
CRDER1 = rmsx * 3600
but instead
CRDER1 = rmsx / sqrt(2)
@done: extract out rmsx and rmsy and convert correctly into CRDER1 and CRDER2
ISSUE 4.
For SOFI I manually checked a few images for SN2013am and SN2013ec and found some images had reasonable values reported in the (corrected) CRDER1 and CRDER2 values, but many images had huge measured shifts ( > 1 arcmin) which were not reflected in the reported CRDER1, CRDER2 values.
These massive shifts seem to have some correlation with the telescope pointing of the image. So for 1 telescope pointing the shifts are reasonable, but for another pointing they are catastrophic.
I did the same measurements for EFOSC images of CSS121015-004244 and found the same issue. See the spreadsheet below (sheets "sofi imaging SN2013am checks" and "efosc imaging CSS121015 checks ").
pessto_ssdr1_astrometry_issues.xlsx
Those were just only spot checks, so it is essential that you verify the offsets for all EFOSC2
and SOFI images.
We recommend you to revise the astrometric solution of all images, and that you lower
the astrometric errors in order to higher the quality of this important dataset.
Please make sure to register properly all images, so that
(1) they do not show big relative offsets
(2) their astrometry matches with the coordinate of the corresponding spectra
Should this be not possible, please add to the release description a list of the affected objects
with their relative offset amount.
@review: what should we do with these images?
SJS to check all images before we decide what to do
Data Release Description
— Potential inconsistencies —
[R09]. Some inconsistencies found wrt uploaded data
i- Both Par. 4 of Release content and Table 4 mention 18 GB of data, while we actually received 17GB.
ii- Table 4 lists:
EFOSC2 images (of which ACQ): NumFiles: 2681 (1072), Volume: 12 GB (4.2 GB)
while the data uploaded on the phase 3 staging areas are:
EFOSC2 images (of which ACQ): NumFiles: 2671 (1072), Volume: 11 GB (4.2 GB)
Notice the difference in NumFiles and in Volume.
iii- Finally, and consistently with the above, Table 4 shows TOTAL number of files 4981, while
the number of files are 4971 on the phase 3 ftp.
Please clarify if the data submitted to ESO are complete.
@action: Dave will check these numbers again before submitting.
— Clarifications, typos, etc. —
[R10]. Artifacts in images: various spot checks revealed extended artifacts in SOFI images.
For an example see: BD174708003_20120915_J_merge_56474_1.fits
There are certainly more such cases.
Please report in the release description the fact that some images show these artifacts.
You could describe under which circumstances this happens, how often, what is
the potential effect of those when analyzing the data (e.g. impact at on photometry), etc.
Here is the image refered to: BD174708003_20120915_J_merge_56474_1.fits. You can see stripping in the south west quadrant of the image.
@action: SJS to add details to release description.
[R11]. In Release Content, 4th paragraph 4, second sentence:
“In total there are 814 EFOSC2 spectra released, and this number is greater than
the sum of 516 and 257 (773)…” should be revised and made more explicit.
Note: I am not fully sure to understand its meaning.
I would have thought that those 516 and 257 are not independent, but if they are, then,
please clarify in the text that Table 3 does not cover the spectra used for the initial
classification (correct?).
Table 3 has has 41 “followup” objects with a total of 516 efosc + 95 sofi spectra (661 total). Technically these included the classification spectrum and the subsequent followup spectra for each object.
Of the remaining 208 objects which do not appear in table 3 (total 257 objects - 41 followup objects = 208) we have 298 spectra. Please note that in some cases 2 or more classification spectra are taken per object – all files are uploaded to ESO so there should be more GR13 spectral classification files present than the number of objects classified. This makes a total of 814 EFOSC spectra uploaded to ESO (516 + 298).
@action: SJS or Dave to change the text to make it clearer.
Also, please specify in the release description how one could identify the spectra used
for the initial classification.
At this effect, please change the last sentence of the 2nd paragraph
from:
“The total number of spectra released are 516 EFOCS2 spectra and 95 SOFI spectra.”
to:
“The number of spectra released for these 41 targets are 516 EFOSC2 spectra and
95 SOFI spectra as shown in Table 3. The classification spectra, not included in Table 3,
can be identified by……. ”
@review: We may need to add another keyword in the FITS headers - there’s no other way unless we have a public webpage listing objects classified and objects followed.
Don't add any new header keywords - SJS to deal with this in the release description. Will simply be a statement that the earliest spectra have been used, often it is not just one spectrum that is used, so there is no hard coding that will work.
[R12]. Please state in the release description that the majority
of the objects have both spectra and images, and highlight the fact
that some images (in the current submission: 65) have no spectrum counterpart,
and that some objects (in the current submission: 12) have spectra but no image
counterpart.
This could be a useful piece of information to the archive users
that might try to locate non existing images for a given spectrum,
and vice versa.
@action: SJS to change text
For the record, I provide some examples here below:
65 images with no spectrum counterpart:
- acq_LSQ12bua_20120412_g782_56462_2.fits
- gd71.dat_20130310_R642_56463_3.fits
- ltt3864.dat_20130418_R642_56463_1.fits
- etc.
and 12 objects with spectra and no image counterpart:
1. CSS130303–105206–133424
2. LSQ12bue
3. LSQ12cbv
4. LSQ12drz
5. LSQ12ecu
6. OGLE2013-SN–017
7. PSNJ12393328+1525520
8. SN2013L
9. SN2013ap
10. SN_IC35
11. SNhunt119
12. SSS130304–114144–171349
[R13]. Please describe how the weight maps are built, if flat/illumination variations are propagated or not, etc.
@action: SJS to add to text
[R14]. Typos
- Typo in Overview of Observations: EFFOSC2 spelt with double F.
- The end of the first sentence of “Data Reduction, Calibration and Quality” (page 4) is not at all clear.
- In section “Arc frames and wavelength calibrations” of page 5 (in the middle), please correct 13–17th
to 13–17 Angstrom.
- Caption of Fig 1: the first mentioning of object SN2013ai should be dropped.
@action: SJS to add to text
For your information
C01. Weight maps
The weight maps (for example BD174708003_20120915_J_merge_56474_1.weight.fits),
seem to take into account the different exposure levels within the mosaic, but not the flat
field nor the illumination information.
Also, the weight maps provided do not try to mask away some big artifacts one can spot
in some SOFI images.
We invite you to consider adding the extra flat/illumination propagation to the weight
maps, plus also the masking of bad artefacts (so the spurious detections could be avoided
in such regions, eg. when running sextractor or similar), in the next releases.
PROPOSED ACTIONS:
A. Please address each of the reported issues and requests for clarification (R01 - R14).
It is useful to remind you that the emphasis on this first data release
is on the spectra. The Phase 3 policies do not require the submission
of images for DR1; you could postpone their publication as independent
science data products until all problems are solved.
@action SJS to check all images before we decide what to do
B. Take note of the suggestion (C01) for improving weight maps in the next data release.
C. In order to allow you to address the problems reported
above and to upload a new version of the release description,
the Phase 3 release has been reverted to the “open” state.
In this way you have again full read and write access permission
on the Phase 3 ftp server.
D. Once you have completed the re-submission of data products including
the revised release description to the ESO Phase 3 ftp server, please
close the release (PESSTO/batch_1) using the Phase 3 release manager,
so to signal to ASG that the action is again on our side.
E. ASG would like to request that you send us a written reply with the list of
actions taken to resolve the fixes R01-R16, that would greatly help our
verification process.
As usual, in case you have questions or should you need any additional
information, please do not hesitate to contact us via email
at “usd-help@eso.org, subject: Phase 3”.
Best Regards,
Alberto - ESO/ASG
Page updated
Report abuse