Durham Postgraduate Data Reduction Course

Timetable
Lectures are held in OCW017. Practicals will be in the computer rooms CM001 (maths building) and ER156 (Elvet Riverside) -- check the timetable.

Any issues, see Mark Swinbank (office OCW113) or Julie Wardlow (office OCW111).

Introduction and Setting Up


Setting up

You are required to do these steps before attending the first workshop. If you have any difficulties come and see Julie (OCW111) or Mark (OCW113).

We will be using the University desktop computers for the practicals. However, if you have your own linux or MAC laptop and are able to set this up with IRAF, wcstool, ds9 and GAIA then you can use this as an alternative. However, for this guide we will assume that you are using the University computers. You will need a few Gb of space on the XGAL disks (this is likely to be inside the /obs/r2 directory). If you do not have this already speak to Alan Lotts. (OR you can use the temporary space on /hot/2, although your work data will not be forever saved there. To set up temporary space type "mkdir /hot/2/$USER" and then "cd /hot/2/$USER" to access the space.)

When on the University machines:
  1. When starting the computer, chose to open a Linux environment (rather than windows). -OR- if on a Windows only computer search in the AppHub of "X2Go-Mira" and launch it to get a Linux environment, witha command line and terminal etc.
  2. In the command line type "xhost +" to allow X forwarding
  3. ssh into nova32 and then zwicky: "ssh -XC username@nova32"; "ssh -XC username@zwicky". For some reason, some of you may need to use -YC rather than -XC to get the forwarding to work. If you are outside of the astro network you may need to first ssh to stargate rather than nova32.
  4. Switch to a cshell terminal (this is needed for IRAF to work properly) by typing "csh" (ot tcsh if you prefer).
  5. Load the software modules that you need (wcstools, ds9, gaia, iraf): "module load wcstools ds9 starlink iraf" (NB: starlink contains gaia). You will need to do this every time you log in, but only load the modules that you need. If you later need additional modules use "module load [name]". To see available modules/software type "module available"
  6. Create a working directory for the data reduction course. This is where you will need disk space and may be in best placed in /obs/r2/USENAME if you have space here, or /hot/2/$USER otherwise. If in doubt ask Alan Lotts.
  7. Download these two tarballs: (1) www.astro.dur.ac.uk/~knpv27/outgoing/pg_dr_spec.tar.gz; (2) www.astro.dur.ac.uk/~knpv27/outgoing/pg_dr_phot.tar.gz. The easiest way to do this is to type "wget URL" when you are in the working directory.
  8. Unpack the tarballs by typing "tar -zxvf pg_dr_spec.tar.gz" and "tar -zxvf pg_dr_phot.tar"
  9. Try opening some of the fits files using "ds9 file.fits" and "gaia file.fits". If nothing displays, you may need to log-in again using -YC (see above). You need to be able to run at least gaia for the workshop. If you have an error message in regards to the GAIA_DIR directory then you need to add this text to your .cshrc file: "source /usr/local/etc/DOT_CSHRC". (to open you .cshrc file type: "emacs ~/.cshrc"). Once you have added the text, type: "source ~/.cshrc". Hopefully gaia will now work.
  10. Try running a wcstools command (see below) on a fits file. e.g. "imhead file.fits". If this doesn't work, add the following line to your .cshrc file: "setenv PATH /obs/perm/ams/bin/wcstools/wcstools/bin/:{$PATH}" (open this file by typing: emacs ~/.cshrc and you will need to "compile" it after editting by typing: source ~/.cshrc ). You need to be able to run wcstools for the workshop.
  11. Make a directory in your home space called iraf: "mkdir iraf"
  12. When in your iraf directory type: "mkiraf" and follow the instructions (choose "xgterm" when asked).
  13. You now need to set-up IRAF (see below). This is already installed but needs setting up on your user environment. Add the following lines to you .cshrc file: (1)"alias iraf '\cp /home/USERNAME/iraf/login.cl ./ ; xgterm -fg black -bg white -fn fixed -sb -e cl -title IRAF '" and (2) "set path = ($path /iraf/local/bin)". Make sure you put your username in the appropriate place and don't forget to compile(source) the .cshrc file after editing it.
  14. In your home directory type "source .cshrc"
  15. Now type "iraf" and hopefully a new cl window will open. In this window, test everything appears to be working by typing "epar imarith" and then ":q" to get out of that. If you get no error messages we should be ready to go!



Course Format

The course consists of two introductory lectures and two workshops covering: (1) Imaging and photometry, and (2) Spectroscopy. In the workshop you will go through example exercises to learn some basic data reduction skills. At the end of each part you will be given Hand-in Problems that need to be completed. These will be marked and go towards your final first year course grades. These are due in before Friday November 24th, 2017. You should email us or come and speak to us (Mark & Julie) if you have any problems. We would rather you asked questions than struggled on your own, or worse, produced nothing!.



Introduction: fits files and wcstools

Hopefully by now you are already familiar with the fits file format. This format allows one file to have multiple extensions (or "layers") of data. It is extremely useful for large tables of data and astronomical images or datacubes. Fits files also have "headers" that (can) contain extensive information about the file. For example for astronomical images the header is likely to contain information on the target, the instrumental set-up, the observing conditions, the pixel scale of the image etc. etc. This information can be absolutely vital for reducing data.

wcstools is a set of tasks that can be run from the command line for displaying and manipulating the headers of fits files. For example:
cmhmac> imhead filename.fits
This will display the header in the terminal. Sometimes you need to specify which extension you wish to display the header for: e.g., imhead filename.fits"[0]". You sometimes will want to display the value of a particular keyword in a header for one or several files. You can do this using gethead. For example:
cmhmac > gethead OBJECT *.fits
arc.fits [0] CuAr
std.fits [0] BD+28 4211
flat.fits [0] GCALflat



Introduction: IRAF

You will be using IRAF to do some basic data reduction during the workshops. Although IRAF is some-what out dated for most purposes, it can still be extremely powerful for performing basic data-reduction tasks and is (surprisingly!) still widely used by many astronomers (the Gemini Observatory still use this as their basis for data reduction!). The main reason we will be using IRAF is that you can do each step of the data reduction "by hand" so you can learn some of the fundamental steps. In reality you will all be using different datasets from different telescopes. Most telescopes now have automated pipelines; however, these sometimes can break, give odd results/mistakes or do not meet the level of accuracy required for some products. It is therefore useful to be able understand the basic steps and have the ability to "hack" or improve automated pipelines.

IRAF consists of several packages of tasks. For example, the images.imutil package contains the imarith task for basic image arithmatic. You can navigate to this package by typing:
cl> images
cl> imutil
If you want to move to the previous level type:
cl> bye
To find out what a task does and to get help on running the task type:
cl> help taskname
Each IRAF task has a set of parameters that need to be chosen carefully. You can see and edit the parameters of a task by typing:
cl> epar taskname
Once you have edited the parameters you can run the tak by typing ":go". If you want to leave the epar screen type ":q". Tasks can also be run directly from the IRAF terminal, for example:
cl> imarith image1.fits * image2.fits new_image.fits
This runs the imarith task and multiplies image1 by image2 to make a new image called new_image.

If you are ever in interactive mode (i.e., have a plot displayed), you can use a variety of key strokes and typed commands. You can find out what these are by typing "?" when the plot is displayed. The list of instructions will display in the IRAF terminal. To get out of this type "q"

Some annoying quirks and get-arounds for IRAF:

  • IRAF does not behave well when trying to overwrite an existing file of the same name. Therefore, it is usually best to delete/remove all files that you want IRAF to overwrite before running the task.
  • When in interactive mode (when you have plots open and are interactively clicking on them) IRAF crashes if you try to close the windows by clicking the "x" button. DO NOT DO THIS. You will need to press 'q' to leave interactive mode first (in most cases).
  • When things are going wrong it can sometimes be useful (or required) to reset to a task's default parameters. This can be done by typing "unlearn taskname". In drastic situations, this sometimes doesn't work OR tasks use sub-tasks that also need to be "unlearned". In your iraf directory there is a sub-directory called uparm. Inside this directory are a bunch of files that store chosen parameters. In times of crisis I have deleted all of these files, but be warned, this will forget all of your previous choices of parameter values for all IRAF tasks.
  • From experience I have found that IRAF doesn't cope well if you simultaneously run multiple tasks inside the same working directory. I believe this is because IRAF sometimes writes out tempory files that get mixed up if multiple tasks are running.

Some documentation:



Introduction: ds9 and GAIA

ds9 and GAIA (part of the starlink package) are two extremely powerful programs for displaying, analysing and manipulating astronomy frames/data cube. Each program has its own strengths and weakenesses with respect to the other. You should familiarise yourself with both. There is good online documentation available for these packages.