PESSTO Year 2 - Final Data Reductions

Test Data from Year 1

Here you'll find some raw data from PESSTO Year 1 which can be used to test the Pipeline. The data consists of 20 observations and their associated calibration frames; 5 observations for each EFOSC/SOFI imaging/spectra setup (see README file companying the data).

Outstanding Tasks:
  • cosimo and erkki to reduce this data using the latests version of the PESSTO Pipeline (v2.2.2)
CI: 
- PESSTOFASTSPEC, shift values are not integer any more (ex. 9.1 Ang in my first reduction)
SV:  this is not an issue, but a new feature

- PESSTOEFOSCPHOT: as in the previous version is not possible to remove bad bias during the process. Surprisingly the pipeline didn't reject bias with standard deviation higher than 400.
SV: this is not true, I downloaded the data sample, and the pipeline reject all the bias frame with std>400. It does not reject 2 or 3 frames with std ~ 300 (the good one have std ~ 10. Anyhow these few frames can be rejected manually if instead of giving 'G' (all good) the user goes trough all the bias frame one by one
giving 'g' (good) and 'b' (bad) for the rest of the frames. 
attached is a plot of the std of all the frames:
Anyhow I change the algorithm to reject the bad frame. I'm using now the percentile instead of the standard deviation to compute the sigma.
This give a better rejection in cases like this one where there is a significant number of outliers.
I also added an option in the PESSTOEFOSCPHOT to change the number of sigma to use in the rejection (default is 5 sigma)
Last comment, these are bias frames from 9 months of data. Usually it would be better to divide the data month by month

CI: My PESSTOEFOSCPHOT version (2.2.2) doesn't give the option 'G' (all good) but ask only for the masterbias, if it is OK.
However it is not a major issue since we can go back and check them individually.


- PESSTOEFOSCPHOT: OK, I didn't get any problems running it
- PESSTOEFOSC2DSPECTRA: No issues
- PESSTOEFOSC1DSPECTRA: No issues

EK:
- PESSTOSOFIPHOT: There are imaging flats in the test data set, however the pipeline is using the library flats. The pipeline gives a warning that the flat images are corrupted if --listflat option is used. 
We have only one set of good flat and illumination frames that are already in the pipeline archive, there is no need to use flat with option --listflat.
There was anyhow a bug and the --listflat option was not working
I fixed the bug, not the option --list flat should work (List should include the pre-reduced flat, not the raw flat)
just to remind, for each flat field used, the illumination correction obtained at the same time should be also used

Alignment/background problems in the merged image of SN2013ak (and BD174708003, due to the bad upper right quadrant in the raw files). 
SN2013ak, still investigating the problem. I don't understand what is happening. If I use the new flat field the image looks good.
https://sites.google.com/a/pessto.org/wiki/pessto-operation-groups/data-reduction-and-quality-control-team/pessto-year-2---final-data-reductions/Screen%20Shot%202014-11-14%20at%2011.50.37%20PM.png

OK something more on this case. I tried to reduce the images with old and new flat field (old flat field=Apr 2012, new flat=Apr 2014) data are from Apr 2013.
Even though both flat field are +-1 year from observations, the problem seems to be the crosstalk correction (not the flat field)  


BD174708003 is corrupted, nothing to fix
         
Astrometry not correct in the merged SN2012ec image. Otherwise the task is running fine. 

for images where astrometry crash I added the possibility to run astrometry and/or zeropoint in interactive mode using the command PESSTOASTRO

PESSTOASTRO SN2012ec_20120825_H_merge_56975_1.fits -i -a -z

- PESSTOSOFI2dSPEC: Failed wavelength calibration of the arc lamp file SOFI.2013-04-17T20:12:38.334.fits . Task was running fine after deleting the file.
SV: this is happening some time. The solution is to eliminate the file (as you did), or identify the lines in the interactive mode using these plots

I made some changes that should make the line identification a bit more stable, however SOFI.2013-04-17T20:12:38.334.fits  still make problems (for the automatic identification)
I manage to use it identifying the lines manually. If you see that most of the time the identification is wrong, we can replace the reference arc in the pipeline archive.
Last note: Usually it is better to use both LAMP ON and OFF to eliminate the strong continuum in the lamp. The test dataset include only lamp ON.


- PESSTOSOFI1dSPEC: The task seems to be running fine.    

Pipeline Bug Tracking

DRY: Experimenting with BitBucket Problem Tracker (should be identical to the one on Github).
KWS: Installed Mantis on PSWeb.  Probably the least irritating install - and config of tasks still to do. But installation can be found here: https://psweb.mp.qub.ac.uk/mantisbt/.  It's not clear how code config management tools (svn/github) work with Mantis.
KWS: Installing Bugzilla - bit of a nightmare install requiring many, many updates to the SYSTEM config of Perl.  However, I did eventually get it installed here: https://psweb.mp.qub.ac.uk/bugzilla/.
KWS: Redmine came very highly recommended - and I really liked the svn/git plugins, but the default CentOS 6 installation of Ruby can't get this going.  Had to install ruby from scratch and also rubygems from scratch.  This worked, and got it going with the internal temporary server, but still having difficulty with Apache config, since cgi no longer supported.  The documentation for Apache config is extremely vague, to say the least.  Is it mod_fastcgi or mod_fcgid?  Do I need both??  If not, please just give us ONE set of config options...
KWS: Trac.  I never completed the original installation - partly because of language/character set issues on the database.  Time to revisit.  I'd really like to install inside a virtualenv, but got bogged down trying to do this.

Installation of Pipeline v2.2.2 on popular OSs

KWS: Update info about spaces and Ureka.
KWS: Update the simple data reduction tests with the new data.
KWS: Feed back to Stefano about the PESSTO -v, --vers, --version output and get it to stop reporting PyRAF version!

Mavericks
CI: It went everything OK, no major issues
EK: Installation was fine. 

SJS: OK. One minor reporting bug. Typing "PESSTO --version" reports the current pyraf version, not the version of PESSTO pipeline. But "PESSTO --versio" or any short version of the word version reports the correct version of PESSTO pipeline. 

Yosemite
DRY: Installation OK.  As of Ureka v1.4.1.2 Yosemite is supported and Sextractor 2.8.6 is now included, so no need to provide separate binary.

Ubuntu
KWS: v14.04.  Works OK, but suffers from same problems as CentOS6.6 - i.e. you MUST install legacy 32bit libraries for Ureka to function properly.  Sextractor comes with Ureka, and wget is pre-installed anyway.
For info, the procedure for getting the libraries is as follows...

sudo -i
apt-get install libc6:i386
cd /etc/apt/sources.list.d
echo "deb http://old-releases.ubuntu.com/ubuntu/ raring main restricted universe multiverse" >ia32-libs-raring.list
apt-get update
apt-get install ia32-libs
rm /etc/apt/sources.list.d/ia32-libs-raring.list
apt-get update
apt-get install gcc-multilib
exit

The only other issue I'm having is getting swarp to work. Installing the SWarp Ubuntu package did absolutely nothing!  So best option is to download from here: http://www.astromatic.net/download/swarp/swarp-2.38.0.tar.gz and then cd to unpacked code directory and ./configure; make; sudo make install.


CentOS
KWS: Did fresh install on CentOS 6.6. Errors if you try to run IRAF!  Still investigating.  However, if I log onto another machine and execute the SAME CODE on a CentOS 6.5 machine all seems to work OK.  Again, confirmed that Sextractor is included with the install.

>>> import pyraf
**execvp failed, '[Errno 2] No such file or directory'**
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/__init__.py", line 122, in <module>
    iraf.Init(doprint=0, hush=1)
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraffunctions.py", line 261, in Init
    userpkg.run(_doprint=0, _hush=hush, _save=1)
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 359, in run
    self._run(redirKW, specialKW)
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 1667, in _run
    self._runCode()
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 1470, in _runCode
    self._clFunction(*parList, **kw)
  File "<CL script clpackage.user>", line 57, in login
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 767, in __call__
    return self.run(*args, **kw)
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 359, in run
    self._run(redirKW, specialKW)
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 808, in _run
    irafexecute.IrafExecute(self, pyraf.iraf.getVarDict(), **redirKW)
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/irafexecute.py", line 339, in IrafExecute
    irafprocess.run(task, pstdin=stdin, pstdout=stdout, pstderr=stderr)
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/irafexecute.py", line 504, in run
    self.writeString(taskname+redir_info+'\n')
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/irafexecute.py", line 571, in writeString
    self.write(Asc2IrafString(s))
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/irafexecute.py", line 595, in write
    tobytes(dsection))
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/subproc.py", line 229, in write
    if bytes_write(self.toChild, strval) != len(strval):
  File "/home/sne/Ureka/python/lib/python2.7/site-packages/stsci.tools-3.2.2-py2.7.egg/stsci/tools/for2to3.py", line 114, in bytes_write
    return os.write(fd, tobytes(bufstr))
OSError: [Errno 32] Broken pipe

Turns out this is one of a whole raft of issues with the CentOS 6.6 install.  E.g.:
sne::psweb { ~ }-> which xgterm
/home/sne/Ureka/bin/xgterm
sne::psweb { ~ }-> xgterm &
[1] 32109
sne::psweb { ~ }-> bash: /home/sne/Ureka/bin/xgterm: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

[1]+  Exit 126                xgterm
sne::psweb { ~ }->

UPDATE - it seems that 32 bit "glibc" libraries are not installed by default with CentOS 6.6.  Need to iterate with Robert.



ą
Cosimo Inserra,
17 Nov 2014, 05:05
Comments