AR 11258 — STIS/CCD Side-2 pattern noise removal

Documentation and Software

Getting the software

All source code — IDL procedures, Fortran77 code, and C-shell scripts — developed to remove the superimposed "herring-bone" pattern-noise from raw, unbinned, full-frame as well as sub-frame 1060×N pixel (for even N ≥ 16) "Side-2" STIS/CCD frames can be downloaded as a single gzip-compressed tar-archive [version: 22Apr2014].

For those only interested in the IDL procedures, these can be downloaded here:

Note, that versions of autofilet.pro prior to v1.2 could handle only full-frame read-out STIS/CCD data.

The above IDL routines will suffice for those wishing to remove the herring-bone pattern-noise from single frames and who do not require merging the results back into the Multi-Extension FITS (MEF) format compatible with calstis for subsequent offline reprocessing and calibration, or who have their own procedure(s) to do so.

Compiling the software

For those who wish to remove the pattern-noise from large numbers of STIS datasets and/or who need to merge resulting frames back into calstis compatible MEF format, Fortran77 programs arg and fits2mef need to be compiled with a standard compatible compiler. Compilation of these programs was recently tested for the gfortran compiler only on Redhat/CentOS 5.x (i386 and x86_64) and 6.x (x86_64 only) Enterprise Linux machines, and for the g77 compiler only on Redhat/CentOS 5.x (i386 and x86_64) Enterpise Linux machines, although various versions of arg are known to have compiled fine on SunOS, Solaris, HPUX, AIX, and MacOS (PPC) architectures from the mid-1990s on with f77 and g77.

Note, that the compiler that is native to XCODE on MacOSX (and/or installed through fink) and is called "gfortran", is incomplete and incompatible, but that compliant MacOSX versions of the real GNU gcc and gfortran are available from: http://gcc.gnu.org/wiki/GFortran and in case of problems, see: http://www.webmo.net/support/fortran_osx.html. On Macs, also read and possibly edit the arg install script and any makefiles in the fits2mef tree. Check that compiler flags are appropriate and sufficient for your system.
I would welcome edits that made the software compile on your system (I don't have access to different OS's for testing; working binaries for architectures other than Redhat/CentOS linux are therefore welcome too, so I could share them with users that do not have administrator privileges or who are at a loss with makefiles or Fortran).

Throughout the following, $> denotes a tcsh prompt (in a xterm terminal/console window). The STIS data and all software are assumed to reside under a project working directory symbolically denoted by $stis2/. The date will be in a subdirectory thereof named ORIG/. There should also be (empty) subdirectories named PSUB/ and TABLES/ under this project directory. The software to be compiled is assumed to reside in $stis2/src/Fortran/arg2.3/ and $stis2/src/Fortran/fits2mef/.

Compiling arg

$> cd $stis2/src/Fortran/arg2.3
$> ./arg.install.private

In case of a compilation error, check/edit file "tmparg.f" and search for the definition of variable 'mfile' in subroutine "manual" (your directory path may have caused the line to exceed 72 characters; if so, simply define it as "arg.man" and compile "tmparg.f": $> gfortran tmparg.f -o arg ).

$> cd $stis2/ ; ln -s src/Fortran/arg2.3/arg arg
$> ln -s src/Fortran/arg2.3/narg narg

In case you manually redefined 'mfile', also:   $> ln -s src/Fortran/arg2.3/man/arg.man arg.man

Compiling fits2mef

Enter the fits2mef source directory:
$> cd $stis2/src/Fortran/fits2mef

Check whether the 'makefile' points to an existing compiler (F77 entry) and that the compiler flags make sense for that compiler. Then run:
$> make

Should the F77 entry not point to an existing compiler on your system, edit the three 'makefile's in the present sub-directory and in the generic/ and rjfits/ subdirectories thereof to point to a compatible standard-compliant Fortran77 compiler and edit the FFLAGS entries to specify the compiler flags appropriate for that compiler.

Once successfully compiled using make (i.e., there where no, or there remain no unresolved compilation errors, executable fits2mef will have been created and there will be two libraries in the lib/ subdirectory), return to the main project directory and create a couple of symbolic links:
$> cd $stis2/ ; ln -s src/Fortran/fits2mef/fits2mef fits2mef
$> ln -s src/Fortran/fits2mef/fits2mef.man fits2mef.man

Compiling cfitsio and listhead

Finally, in the detailed step by step instructions below, we will use the listhead utility that comes with W.D. Pence's cfitsio library, available from: http://heasarc.gsfc.nasa.gov/fitsio/fitsio.html.
This library therefore needs to be installed and this demonstration utility compiled and within the standard search path on your machine.

Detailed step-by-step instructions for processing a large number of "Side-2" STIS/CCD frames

Detailed instructions describing how one may remove the pattern-noise from a large number of datasets and merge the individual frames back into calstis-compatible MEF files are given in this AA_README.txt file.

Note, that for the smaller 1060×N sub-frame read-outs, there is much less information in the frequency power spectrum than in the case of full-frame read-out. While the pattern-noise after running autofilet.pro may still be significantly reduced compared to the original raw frame, the pattern may not have been removed completely and/or everywhere throughout the frame.

While the author (Rolf Jansen) is familiar and comfortable with C-shell scripts and a command line interface, other researchers might prefer more monolithic end-to-end software. Azalee Bostroem (STScI) has created a Python wrapper around autofilet.pro that might be useful to others and is available at: https://github.com/abostroem/science_programs/tree/master/herringbone_correction.
Note, however, that the version of autofilet.pro on that site is out of date, but can simply be replaced by the v1.2 at the top of this page.


In case of problems with this page, contact: Rolf.Jansen @ asu.edu , tel.: (480)727-7119
Last updated: Apr 22, 2014