PESSTO Year 2 - Final Data Reductions

Test Data from Year 1

Here you'll find some raw data from PESSTO Year 1 which can be used to test the Pipeline. The data consists of 20 observations and their associated calibration frames; 5 observations for each EFOSC/SOFI imaging/spectra setup (see README file companying the data).

Outstanding Tasks:

CI: 

- PESSTOFASTSPEC, shift values are not integer any more (ex. 9.1 Ang in my first reduction)

SV:  this is not an issue, but a new feature

- PESSTOEFOSCPHOT: as in the previous version is not possible to remove bad bias during the process. Surprisingly the pipeline didn't reject bias with standard deviation higher than 400.

SV: this is not true, I downloaded the data sample, and the pipeline reject all the bias frame with std>400. It does not reject 2 or 3 frames with std ~ 300 (the good one have std ~ 10. Anyhow these few frames can be rejected manually if instead of giving 'G' (all good) the user goes trough all the bias frame one by one

giving 'g' (good) and 'b' (bad) for the rest of the frames. 

attached is a plot of the std of all the frames:

Anyhow I change the algorithm to reject the bad frame. I'm using now the percentile instead of the standard deviation to compute the sigma.

This give a better rejection in cases like this one where there is a significant number of outliers.

I also added an option in the PESSTOEFOSCPHOT to change the number of sigma to use in the rejection (default is 5 sigma)

Last comment, these are bias frames from 9 months of data. Usually it would be better to divide the data month by month

CI: My PESSTOEFOSCPHOT version (2.2.2) doesn't give the option 'G' (all good) but ask only for the masterbias, if it is OK.

However it is not a major issue since we can go back and check them individually.

- PESSTOEFOSCPHOT: OK, I didn't get any problems running it

- PESSTOEFOSC2DSPECTRA: No issues

- PESSTOEFOSC1DSPECTRA: No issues

EK:

- PESSTOSOFIPHOT: There are imaging flats in the test data set, however the pipeline is using the library flats. The pipeline gives a warning that the flat images are corrupted if --listflat option is used. 

We have only one set of good flat and illumination frames that are already in the pipeline archive, there is no need to use flat with option --listflat.

There was anyhow a bug and the --listflat option was not working

I fixed the bug, not the option --list flat should work (List should include the pre-reduced flat, not the raw flat)

just to remind, for each flat field used, the illumination correction obtained at the same time should be also used

Alignment/background problems in the merged image of SN2013ak (and BD174708003, due to the bad upper right quadrant in the raw files). 

SN2013ak, still investigating the problem. I don't understand what is happening. If I use the new flat field the image looks good.

OK something more on this case. I tried to reduce the images with old and new flat field (old flat field=Apr 2012, new flat=Apr 2014) data are from Apr 2013.

Even though both flat field are +-1 year from observations, the problem seems to be the crosstalk correction (not the flat field)  

BD174708003 is corrupted, nothing to fix

         

Astrometry not correct in the merged SN2012ec image. Otherwise the task is running fine. 

for images where astrometry crash I added the possibility to run astrometry and/or zeropoint in interactive mode using the command PESSTOASTRO

PESSTOASTRO SN2012ec_20120825_H_merge_56975_1.fits -i -a -z

- PESSTOSOFI2dSPEC: Failed wavelength calibration of the arc lamp file SOFI.2013-04-17T20:12:38.334.fits . Task was running fine after deleting the file.

SV: this is happening some time. The solution is to eliminate the file (as you did), or identify the lines in the interactive mode using these plots

http://csp1.lco.cl/~cspuser1/OBS/SPEC/SOFI_Xe_arc_blue.gif

http://csp1.lco.cl/~cspuser1/OBS/SPEC/SOFI_Xe_arc_red.gif

I made some changes that should make the line identification a bit more stable, however SOFI.2013-04-17T20:12:38.334.fits  still make problems (for the automatic identification)

I manage to use it identifying the lines manually. If you see that most of the time the identification is wrong, we can replace the reference arc in the pipeline archive.

Last note: Usually it is better to use both LAMP ON and OFF to eliminate the strong continuum in the lamp. The test dataset include only lamp ON.

- PESSTOSOFI1dSPEC: The task seems to be running fine.    

Pipeline Bug Tracking

DRY: Experimenting with BitBucket Problem Tracker (should be identical to the one on Github).

KWS: Installed Mantis on PSWeb.  Probably the least irritating install - and config of tasks still to do. But installation can be found here: https://psweb.mp.qub.ac.uk/mantisbt/.  It's not clear how code config management tools (svn/github) work with Mantis.

KWS: Installing Bugzilla - bit of a nightmare install requiring many, many updates to the SYSTEM config of Perl.  However, I did eventually get it installed here: https://psweb.mp.qub.ac.uk/bugzilla/.

KWS: Redmine came very highly recommended - and I really liked the svn/git plugins, but the default CentOS 6 installation of Ruby can't get this going.  Had to install ruby from scratch and also rubygems from scratch.  This worked, and got it going with the internal temporary server, but still having difficulty with Apache config, since cgi no longer supported.  The documentation for Apache config is extremely vague, to say the least.  Is it mod_fastcgi or mod_fcgid?  Do I need both??  If not, please just give us ONE set of config options...

KWS: Trac.  I never completed the original installation - partly because of language/character set issues on the database.  Time to revisit.  I'd really like to install inside a virtualenv, but got bogged down trying to do this.

Installation of Pipeline v2.2.2 on popular OSs

KWS: Update info about spaces and Ureka.

KWS: Update the simple data reduction tests with the new data.

KWS: Feed back to Stefano about the PESSTO -v, --vers, --version output and get it to stop reporting PyRAF version!

Mavericks

CI: It went everything OK, no major issues

EK: Installation was fine. 

SJS: OK. One minor reporting bug. Typing "PESSTO --version" reports the current pyraf version, not the version of PESSTO pipeline. But "PESSTO --versio" or any short version of the word version reports the correct version of PESSTO pipeline. 

Yosemite

DRY: Installation OK.  As of Ureka v1.4.1.2 Yosemite is supported and Sextractor 2.8.6 is now included, so no need to provide separate binary.

Ubuntu

KWS: v14.04.  Works OK, but suffers from same problems as CentOS6.6 - i.e. you MUST install legacy 32bit libraries for Ureka to function properly.  Sextractor comes with Ureka, and wget is pre-installed anyway.

For info, the procedure for getting the libraries is as follows...

sudo -i

apt-get install libc6:i386

cd /etc/apt/sources.list.d

echo "deb http://old-releases.ubuntu.com/ubuntu/ raring main restricted universe multiverse" >ia32-libs-raring.list

apt-get update

apt-get install ia32-libs

rm /etc/apt/sources.list.d/ia32-libs-raring.list

apt-get update

apt-get install gcc-multilib

exit

The only other issue I'm having is getting swarp to work. Installing the SWarp Ubuntu package did absolutely nothing!  So best option is to download from here: http://www.astromatic.net/download/swarp/swarp-2.38.0.tar.gz and then cd to unpacked code directory and ./configure; make; sudo make install.

CentOS

KWS: Did fresh install on CentOS 6.6. Errors if you try to run IRAF!  Still investigating.  However, if I log onto another machine and execute the SAME CODE on a CentOS 6.5 machine all seems to work OK.  Again, confirmed that Sextractor is included with the install.

>>> import pyraf

**execvp failed, '[Errno 2] No such file or directory'**

Traceback (most recent call last):

  File "<stdin>", line 1, in <module>

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/__init__.py", line 122, in <module>

    iraf.Init(doprint=0, hush=1)

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraffunctions.py", line 261, in Init

    userpkg.run(_doprint=0, _hush=hush, _save=1)

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 359, in run

    self._run(redirKW, specialKW)

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 1667, in _run

    self._runCode()

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 1470, in _runCode

    self._clFunction(*parList, **kw)

  File "<CL script clpackage.user>", line 57, in login

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 767, in __call__

    return self.run(*args, **kw)

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 359, in run

    self._run(redirKW, specialKW)

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/iraftask.py", line 808, in _run

    irafexecute.IrafExecute(self, pyraf.iraf.getVarDict(), **redirKW)

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/irafexecute.py", line 339, in IrafExecute

    irafprocess.run(task, pstdin=stdin, pstdout=stdout, pstderr=stderr)

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/irafexecute.py", line 504, in run

    self.writeString(taskname+redir_info+'\n')

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/irafexecute.py", line 571, in writeString

    self.write(Asc2IrafString(s))

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/irafexecute.py", line 595, in write

    tobytes(dsection))

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/pyraf-2.1.6-py2.7-linux-x86_64.egg/pyraf/subproc.py", line 229, in write

    if bytes_write(self.toChild, strval) != len(strval):

  File "/home/sne/Ureka/python/lib/python2.7/site-packages/stsci.tools-3.2.2-py2.7.egg/stsci/tools/for2to3.py", line 114, in bytes_write

    return os.write(fd, tobytes(bufstr))

OSError: [Errno 32] Broken pipe

Turns out this is one of a whole raft of issues with the CentOS 6.6 install.  E.g.:

sne::psweb { ~ }-> which xgterm

/home/sne/Ureka/bin/xgterm

sne::psweb { ~ }-> xgterm &

[1] 32109

sne::psweb { ~ }-> bash: /home/sne/Ureka/bin/xgterm: /lib/ld-linux.so.2: bad ELF interpreter: No such file or directory

[1]+  Exit 126                xgterm

sne::psweb { ~ }->

UPDATE - it seems that 32 bit "glibc" libraries are not installed by default with CentOS 6.6.  Need to iterate with Robert.