##################### A U T O F I L E T H O W - T O #################### # # Cycle 16 HST Archival Calibration Legacy Program #11258 (PI: R.A. Jansen) # "Removing the Herring-bone Pattern-noise from *all* STIS Side-2 CCD data: # a factor ~3 enhancement in sensitivity" # -- IDL version of 'autofilet'; compiled version of 'fits2mef'+'arg' -- ############################################################################## ----------------------------------------------------------------------------- * throughout the following, '$>' denotes a tcsh prompt (in a xterm window), and 'cl>' an IRAF prompt (with IRAF running in a xgterm window). The data is assumed to reside under a project directory $stis2/ in a subdirectory thereof named ORIG/ . There should also be (empty) subdirectories named PSUB/ and TABLES/ ----------------------------------------------------------------------------- * compile executables 'arg' and 'fits2mef'. Compilation of these Fortran77 programs has been recently tested only on Redhat/CentOS 32- and 64-bit Enterprise Linux machines for compilers with GNU heritage: 'f77' / 'g77' / 'gfortran', although versions of 'arg' are known to have compiled fine on SunOS, Solaris, HPUX, and AIX machines from the mid-1990s on. Note, that the compiler that is native to MacOSX or installs through 'fink' and that is called "gfortran" is botched, but that compliant MacOSX versions of the real 'gfortran' are available from: http://gcc.gnu.org/wiki/GFortran and in case of problems, see: http://www.webmo.net/support/fortran_osx.html On Macs, also read and possibly edit any makefiles and check that compiler flags are appropriate/sufficient for your system and, please, share with me edits that worked for you (I do not have a Mac for testing; binaries for architectures other than Redhat/CentOS linux are welcome too, so I could share them with users that do not have administrator privileges)! $> cd $stis2/src/Fortran/arg2.2 $> ./arg.install.private In case of a compilation error, check/edit file "tmparg.f" and search for the definition of variable 'mfile' in subroutine "manual" (your directory path may have caused the line to exceed 72 characters; if so, simply define it as "arg.man" and compile "tmparg.f": gfortran tmparg.f -o arg ). $> cd $stis2/ ; ln -s src/Fortran/arg2.2/arg arg $> ln -s src/Fortran/arg2.2/narg narg (In case you manually redefined 'mfile', also: $> ln -s src/Fortran/arg2.2/man/arg.man arg.man ) $> cd $stis2/src/Fortran/fits2mef Check whether the 'makefile' points to an existing compiler (F77 entry) and that the compiler flags make sense for that compiler. Then run: $> make Should the F77 compiler not point to an existing compiler on your system, edit the three 'makefile's in the present sub-directory and in the "generic/" and "rjfits/" subdirectories thereof to point a compatible standard-compliant Fortran77 compiler and edit the FFLAGS entries to specify the compiler flags appropriate for that compiler. Once successfully compiled using make (i.e., there where no, or there remain no unresolved compilation errors, executable "fits2mef" will have been created and there will be two libraries in the "lib/" subdirectory), return to the main project directory and create a couple of symbolic links: $> cd $stis2/ ; ln -s src/Fortran/fits2mef/fits2mef fits2mef $> ln -s src/Fortran/fits2mef/fits2mef.man fits2mef.man Finally, we will use the 'listhead' utility that comes with W.D. Pence's 'cfitsio' library (http://heasarc.gsfc.nasa.gov/fitsio/fitsio.html) of routines. CFITSIO therefore needs to be installed and this demonstration utility compiled and within the standard search path on your machine. (I haven't written the equivalent utility within my 'rjfits' library yet -- on my todo list... Once done, we no longer need CFITSIO) Note, that if there remained unresolved compilation/linking errors, none of the following is likely to get you very far... ----------------------------------------------------------------------------- $> cd $stis2/ Create a listing of the raw datasets in ORIG/ : $> \ls -1 ORIG/*_raw.fits ORIG/*_wav.fits > f.lis Extract selected information from the headers of the science exposures and create a human- and machine-readable inventory: If not present already, create a template script named "mkinventory.tem" that looks as follows: imextensions ("%", output="file", index="0-", extname="SCI", lindex=yes, lname=no, lver=no, ikparams="", > "tmpf.lis") hselect ("@tmpf.lis", s1, "yes", >> "tmpINVENTORY") delete ("tmpf.lis", yes, verify=no) If not present already, create a file "mkinventory.def" with the following contents: s1="$I,date-obs,time-obs,exptime,ccdgain,naxis1,naxis2,binaxis1,binaxis2" s1=s1//",opt_elem,aperture,minwave,maxwave,obstype,targname,ra_targ,dec_targ" s1=s1//",ra_aper,dec_aper,pa_aper,orientat,sunangle,moonangl,sun_alt,occdhtav" Replicate "mkinventory.tem" while replacing each and every occurence of a '%' sign with the next file name in "f.lis" to create an IRAF script. This is most easily done automatically using the 'arg/narg' utility: cl> cd stis2$ cl> !narg mkinventory.tem f.lis '%' > mkinventory.tmp cl> !cat mkinventory.def mkinventory.tmp > mkinventory.cl cl> head mkinventory.cl nlines=20 <-- check that script is correct cl> cl < mkinventory.cl <-- execute iraf script Reformat "tmpINVENTORY" such that the dataset names become lower case, and such that the "dateobs" and "timeobs" are concatenated with a "T" to create a standard Y2K-compliant date-string on which we can subsequently sort the inventory. All other field need to be of fixed format (in particular: of constant width). This is easiest done using 'sed', 'awk' and 'sort' via the following script: If not present already, create a file named "fminventory" as follows (beware of the long lines!): echo "# image dateobs texp gain size bin optelem aperture lamb1 lamb2 obstype targname ra_targ dec_targ ra_aper dec_aper pa_aper orientat sunangle moonangle sun_alt occdhtav" > _INVENTORY echo "# " >> _INVENTORY echo "# image dateobs t_exp gain size bin optelem aperture lamb1 lamb2 ___obstype___ ________targname________ ra_targ dec_targ ra_aper dec_aper pa_aper orientat sunangle moonangle sun_alt occdhtav" >> _INVENTORY echo "# root_typ.fits[extn] [YYYY-MM-DDThh:mm:ss] [sec][e/adu][pixels] [\AA] [\AA] [deg] [deg] [deg] [deg] [deg] [deg] [deg] [deg] [deg] [deg C]" >> _INVENTORY echo "#========================================================================================================================================================================================================================================================" >> _INVENTORY sed 's:\t:,:g' tmpINVENTORY | sed 's:ORIG/::g' | awk -F, '{printf("%-22s %sT%s %7.2f %3d %4dx%-4d %1dx%1d %-6s %-11s%5d %5d %-13s %-24s %10.6f %+10.6f %10.6f %+10.6f %+11.6f %+8.3f %+11.6f %+11.6f %+11.6f %10.6f\n",tolower($1),$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19,$20,$21,$22,$23,$24,$25)}' | sort -k 2,2 >> _INVENTORY Run (source) the above script: $> source fminventory The result should be a 249-character wide fixed-format ASCII inventory file that lists each of the individual frames in each of the FITS files (datasets), sorted on the actual start date and time. Prepare the lists and scripts for removal of the herringbone pattern-noise in all unbinned (1x1) STIS/CCD data frames in this inventory. First, split the inventory list into lists of the different data types (for which 'autofilet' needs to be run with differently optimized input parameters) and extract the root names and image extension numbers: $> grep '1x1' _INVENTORY | grep '1062x1044' | wc -l XXXX <-- number of unbinned, full-frame STIS CCD frames $> grep '1x1' _INVENTORY | grep '1062x1044' | \ sed 's:_raw.fits\[: :g' | sed 's:_wav.fits\[:_wav :g' | \ awk -F] '{print $1,$2}' | cut -c1-17,65-82,94-132 > _tmp.lis Biases and dark frames... $> egrep -e '(BIAS|DARK)' _tmp.lis | awk '{print $1,$2}' > _bfile.lis Spectroscopic science observations (including flatfields)... $> egrep -ve '(BIAS|DARK|50CCD|F28|F25|WAVE|_wav)' _tmp.lis | \ awk '{print $1,$2}' > _sfile.lis Direct imaging science observations... $> egrep -ve '(BIAS|DARK|CCDF|ANY)' _tmp.lis | \ egrep -e '(50CCD|F28|F25)' | awk '{print $1,$2}' > _ifile.lis $> egrep -ve '(BIAS|DARK|CCDF)' _tmp.lis | grep 'MIRVIS' | grep 'ANY' | \ awk '{print $1,$2}' >> _ifile.lis Wavelength calibration exposures... $> egrep -e '(_wav|WAV)' _tmp.lis | awk '{print $1,$2}' > _wfile.lis # Check that all unbinned, full-size frames are accounted for: $> wc -l _?file.lis TTT _bfile.lis UUU _ifile.lis VVV _sfile.lis WW _wfile.lis XXXX total <-- should match total number of unbinned frames $> rm _tmp.lis Now, per data type, see which frames have been pattern-subtracted already and which need to be processed (for this first batch, *all* 1x1 binned frames will need to be processed, of course): If not present already, create a file named "dtype.lis" with the following content: b i s w If not present already, create a file named "mktodo.tem" with the following content [ NOTE, that for some more verbose implementations of 'ls', one may need to change the "$2" in the "awk '{print $2}'..." statemens to "$4" ] : awk '{print "(\\ls PSUB/"$1"_rwc_"$2".fits >> psub.done ) >>& psub.todo"}' \ _%file.lis | sed 's:+:_:g' > mktodo source mktodo awk '{print $2}' psub.todo | sed 's:_wav:+wav:g' | awk -F_ '{print $1}' | \ sed 's:PSUB/::g' | sed 's:+wav:_wav:g' > _%root.lis awk '{print $2}' psub.todo | awk -F_ '{print $NF}' | \ sed 's/.fits://g' > _%extn.lis \rm mktodo psub.todo psub.done Replicate "mktodo.tem" while substituting every occurence of '%' with the next entry from "dtype.lis" (this, again, is easy with the 'narg' utility): $> narg mktodo.tem dtype.lis '%' > mktodo.csh $> cat mktodo.csh <-- check resulting script $> source mktodo.csh <-- run script Check: $> wc -l _?root.lis $> wc -l _?extn.lis TTT _broot.lis TTT _bextn.lis UUU _iroot.lis UUU _iextn.lis VVV _sroot.lis VVV _sextn.lis WW _wroot.lis WW _wextn.lis XXXX total XXXX total The thus created files "_[bisw]root.lis" and "_[bisw]extn.lis" serve as input for the templates of the IDL batch-mode scripts that perform the actual processing of a single frame within a STIS imset. Processing the CCDFLATs is often unnecessary, since the pattern amplitude is typically only ~0.09-0.10% of the signal level in even the darkest illuminated portions of the flat fields, but this statement may not hold in the flats with the NUV filters inserted nor for the ones taken for specific calibration programs (e.g., the annealing monitor program). 'autofilet' will likely report a spurious pattern in flats with a high overall signal level (as it should). ------------------------------------------------------------------------------ * In ~0.40% of the frames, the IDL spline fitting core-routine will diverge (for the specified order of the fit) and cause IDL to hang (instead of properly trapping this condition) while consuming 100% CPU-time. In order to still be able to batch-process the frames, we need to run three scripts: one is the actual C-shell script that starts up an IDL batch-script for a single STIS/CCD frame and waits a given number of seconds (depending on the order of the spline fit in x and y and on how fast the machine is) before starting up the next batch script; the second is that IDL batch script; and the third is a script that monitors that there is never more than 1 IDL process running at a given time. Not exactly elegant programming!! ------------------------------------------------------------------------------ If not present already, create a template script "doidlbatch.tem" with the following content: !quiet = 2 .run autofilet.pro .run autofilet.pro autofilet, "ORIG/&1_raw.fits", "PSUB/&1_rwc_&2.fits",$ "TABLES/&1_fps_&2.dat", exten=&2,$ xorder=&3, yorder=&4, nrej=5 exit If not present already, create a template script "_idlbatch.tem" with the following content: sed 's/&1/$1/g' doidlbatch.tem | sed 's/wav_raw/wav/g' | \ sed 's/&2/$2/g' | sed 's/&3/$3/g' | sed 's/&4/$4/g' > tmpdoidl idl -quiet < tmpdoidl Create a file named "xyfit.lis" with the following optimized default fit order in x and y (note, this file is not actually read by any script, but is for the user's reference only): 15 4 <-- if a hang occurs, retry with 17 4 , 17 5 , 18 5 6 6 <-- if a hang occurs, retry with 7 7 , 6 8 , 7 8 15 29 <-- if a hang occurs, retry with 15 31 , 17 29 , 16 30 6 15 <-- if a hang occurs, retry with 6 17 , 7 15 , 7 16 Now construct the main controling script that will process the entire batch of STIS/CCD frames, where the above numbers are used in the 'sed' commands: $> sed 's:$3:15:g' _idlbatch.tem | sed 's:$4:4:g' > idlbatch.tem $> narg idlbatch.tem _broot.lis '$1' _bextn.lis '$2' > idlbatch.csh $> sed 's:$3:6:g' _idlbatch.tem | sed 's:$4:6:g' > idlbatch.tem $> narg idlbatch.tem _iroot.lis '$1' _iextn.lis '$2' >> idlbatch.csh $> sed 's:$3:15:g' _idlbatch.tem | sed 's:$4:29:g' > idlbatch.tem $> narg idlbatch.tem _sroot.lis '$1' _sextn.lis '$2' >> idlbatch.csh $> sed 's:$3:6:g' _idlbatch.tem | sed 's:$4:15:g' > idlbatch.tem $> narg idlbatch.tem _wroot.lis '$1' _wextn.lis '$2' >> idlbatch.csh $> echo 'sleep 1 ; \\rm tmpdoidl' >> idlbatch.csh $> rm idlbatch.tem $> chmod 700 idlbatch.csh Start the supervising (monitoring) script and let it run in the background, then start the IDL batch job script (NB: on some computers a single instance of IDL seems to spawn multiple processes, 3 in particular. On recent 64-bit dual-core 'linux' machines I have not seen this problem, but if you encounter this problem, try specifying "-k 3" in the following): $> jobmon.csh -p idl -k 1 -m 10 -d 180 >& jobmon.log & Start the IDL batch job script: $> idlbatch.csh When finished, stop the monitoring script by recalling its process back to the foreground and then hitting ^C (Control-C) in the 'xterm' in which it was started: $> fg ^C Verify that all frames that needed to be cleaned indeed have been cleaned, i.e., that one pattern-cleaned "*_rwc_*.fits" frame exists in the PSUB/ repository for each original frame (see also the log-file of the 'jobmon' script). To do so, re-run "mktodo.csh": $> source mktodo.csh $> wc -l _?root.lis $> wc -l _?extn.lis The 'wc' commands should produce only empty lists (i.e., "nothing more to do"). If one or more of these lists are somehow not empty (because 'autofilet' hung, failed, or was erroneously killed), re-create & run new "idlbatch.csh" script, but with the alternate fit-order values listed above: $> sed 's:$3:17:g' _idlbatch.tem | sed 's:$4:4:g' > idlbatch.tem $> narg idlbatch.tem _broot.lis '$1' _bextn.lis '$2' > idlbatch.csh $> sed 's:$3:7:g' _idlbatch.tem | sed 's:$4:7:g' > idlbatch.tem $> narg idlbatch.tem _iroot.lis '$1' _iextn.lis '$2' >> idlbatch.csh $> sed 's:$3:15:g' _idlbatch.tem | sed 's:$4:31:g' > idlbatch.tem $> narg idlbatch.tem _sroot.lis '$1' _sextn.lis '$2' >> idlbatch.csh $> sed 's:$3:6:g' _idlbatch.tem | sed 's:$4:17:g' > idlbatch.tem $> narg idlbatch.tem _wroot.lis '$1' _wextn.lis '$2' >> idlbatch.csh $> echo 'sleep 1 ; \\rm tmpdoidl' >> idlbatch.csh $> rm idlbatch.tem $> chmod 700 idlbatch.csh $> jobmon.csh -p idl -k 1 -m 10 -d 180 >& jobmon.log & $> idlbatch.csh After this finishes, bring 'jobmon' to the foreground and ^C it, then source "mktodo.csh" again and check that the root and extn lists are now empty. If not, repeat the above commands again with again a slightly different fit-order as indicated in file "xyfit.lis". If the lists *are* empty, save and rename a copy of logfile "autofilet.log": $> \cp -p autofilet.log autofilet.sav and clean-up what we don't need any more: $> rm mktodo.csh ; rm idlbatch.csh ; rm jobmon.log $> rm _?root.lis ; rm _?extn.lis Grep from "autofilet.log" each occurance of "WARNING: anomalous pattern" and check whether it was a flat field or similar exposure: $> echo "imstat.format=no\nimstat.nclip=1\n" > chkflt.cl $> grep "WARNING: anomalous" autofilet.log | awk -F/ '{print $2}' | \ awk '{print "imstat (\"PSUB/"$1 }' | sed 's:_raw.fits\[:_rwc_:g' | \ sed 's/\]/.fits\[431:630,421:620\]", fields="image,mean"/g' | \ sed 's:$:, >> "chkflt.log"):g' >> chkflt.cl $> head chkflt.cl Run the IRAF script just created: cl> cl < chkflt.cl Check in "chkflt.log" which of the images have a mean level >~3000 ADU. Those will be either CCD flat fields or similar exposures. Since removal of the ~few ADU herringbone pattern does not improve such exposures significantly anyway, such frames do not need to be visually verified. Visual verification is needed only for frames with mean levels < 3000 ADU. Now edit the "autofilet.log" file and search for each occurance of "WARNING: anomalous pattern" and see whether the offending frame corresponds to a frame with an image title of "CCDFLAT" or with a mean level larger than ~3000 ADU or so. If so, then it is a flatfield or flatfield-like exposure to monitor slit positions and light baffling, and you can simply change: "VERIFIED=NO GENUINE=UNKNOWN" to "VERIFIED=NO GENUINE=N/A" If it is *not* a flatfield (i.e., mean value <~ 3000 ADU) then for each frame (i.e., root name '#########' or '#########_wav', and extension '*' or '**') that issued a warning: $> sed 's/&/#########/g' chkclean.tem | sed 's/@/*/g' > chkclean.cl (replacing the #-marks and *-sign with the actual characters, of course!). cl> !ds9 (if not already open) cl> task askstr=askstr.cl (if not already defined) cl> cl < chkclean.cl This will attempt to create and load (1) the original 'raw' frame, (2) the pattern-cleaned 'rwc' frame, and (3) a frame with the inferred herring-bone pattern, all loaded with the same stretch around the mean level. Blink between each to assess whether or not the pattern visible in (1) is reduced in (2) and that (3) indeed seems to correspond to that pattern and does not look "wrong" or "corrupted" in some way -- it probably takes looking at many frames before you get a feel for this. Note, that you have to change the stretch and contrast (and possibly even the cut-levels) while doing this until you can see the wavy herring-bone pattern and read-noise as greyscale levels between light- and dark- grey (i.e., spanning ~80% of the full range between white and black; that way, whether or not the pattern was removed properly in (2) will become clear). After changing stretch in frame (1), you have to select "Match Scales" and "Match Colorbars" from the "Frame" menu at the top of the 'ds9' window so frames (2) and (3) will have the same stretch as frame (1). Note that, unless you remove the pattern noise from hundreds if not thousands of frames, you may never have to do this tedious verification step. If the pattern had an anomalous frequency and/or amplitude and/or driftwidth, but appears to have been removed correctly from frame (2) and corresponds to a reasonable pattern in frame (3), then change "VERIFIED=NO GENUINE=UNKNOWN" to: "VERIFIED=YES GENUINE=YES" otherwise to: "VERIFIED=YES GENUINE=NO" (note the fixed format and all-capitals). Next, we must merge the individual cleaned frames into STIS 'imsets' (Multi- Extension FITS files) and restore their original headers, in order to process them further with 'calstis'. Using the image headers and the edited "autofilet.log" file, construct a fixed-format pattern characteristics table, matched in order to the inventory file, and with spurious patterns flagged. Note, that we assume that, for non-flatfield frames in which the initially detected pattern was spurious but which were cleaned successfully after adjusting the fit-parameters, the successful log-entry appears later in the "autofilet.log" file than the spurious entry. It is also assumed that only the successfully cleaned image "*_rwc_*.fits" remains in "PSUB/" (other attempts will have been overwritten). If not present already, create a template script "chkfreq.tem" with the following content (beware of the long lines): echo -n "& : extracting pattern and obsmode info ... " listhead & > /tmp/lhead echo "VERIFIED=* GENUINE=0" >> /tmp/lhead set warn="" set warn=`echo "&" | sed 's:PSUB:ORIG:g' | sed 's:wav_rwc_:wav.fits\\\\[:g' | sed 's:_rwc_:_raw.fits\\\\[:g' | sed 's:.fits$::g'` grep "$warn" autofilet*.log | grep GENUINE | sed 's:NO:1:g' | sed 's:YES:0:g' | sed 's:N/A:*:g' >> /tmp/lhead grep 'FILENAME' /tmp/lhead | awk '{printf("%-27s ",$2)}' | sed 's:'\''::g' >> chkfreq.log grep 'freq=' /tmp/lhead | awk '{print $2,$4,$6}' | sed 's:freq=::g' | sed 's:width=::g' | sed 's:pmax=::g' | awk '{printf("%7.1f %7.4f %8.5f ",$1,$2,$3)}' >> chkfreq.log grep 'GENUINE=' /tmp/lhead | tail -1 | awk -F= '{printf("%s ",$NF)}' >> chkfreq.log grep 'ROOTNAME' /tmp/lhead | head -1 | awk '{printf("%s ",$2)}' | sed 's:'\''::g' >> chkfreq.log grep 'TDATEOBS' /tmp/lhead | awk '{printf("\"%10sT",$2)}' | sed 's:'\''::g' >> chkfreq.log grep 'TTIMEOBS' /tmp/lhead | awk '{printf("%8s\" ",$2)}' | sed 's:'\''::g' >> chkfreq.log grep 'TDATEOBS' /tmp/lhead | awk '{print $2}' | awk -F"-" '{printf("%4s %02d %02d ",$1,$2,$3)}' | sed 's:'\''::g' >> chkfreq.log grep 'TTIMEOBS' /tmp/lhead | awk '{print $2}' | awk -F":" '{printf("%2s %02d %02d ",$1,$2,$3)}' | sed 's:'\''::g' >> chkfreq.log grep 'CCDGAIN' /tmp/lhead | awk '{printf(" %1d ",$3)}' >> chkfreq.log grep 'BINAXIS1' /tmp/lhead | awk '{printf("%1d%1s",$2,"x")}' >> chkfreq.log grep 'BINAXIS2' /tmp/lhead | awk '{printf("%1d ",$2)}' >> chkfreq.log grep 'NAXIS1 ' /tmp/lhead | awk '{printf("%4d ",$3)}' >> chkfreq.log grep 'NAXIS2 ' /tmp/lhead | awk '{printf("%4d ",$3)}' >> chkfreq.log grep 'OPT_ELEM' /tmp/lhead | awk '{printf("%-9s",$2)}' | sed 's:'\''::g' >> chkfreq.log grep 'APERTURE' /tmp/lhead | awk '{printf("%-13s",$2)}' | sed 's:'\''::g' >> chkfreq.log grep 'TARGNAME' /tmp/lhead | awk '{printf("%-25s\n",$2)}' | sed 's:'\''::g' >> chkfreq.log unset warn ; \rm /tmp/lhead ; sleep 1 echo "Done." $> grep -v '#' _INVENTORY | grep '1062x1044' | sed 's:raw.fits:rwc_:g' | \ sed 's:wav.fits:wav_rwc_:g' | awk '{print "PSUB/"$1}' | \ sed 's:\[::g' | sed 's:\]:.fits:g' > chkfreq.lis $> narg chkfreq.tem chkfreq.lis '&' > chkfreq.csh $> source chkfreq.csh This script will take quite a while to finish. When it finishes: $> echo "# imgname freq width ampl qf root dateobs yyyy mm dd HH MM SS gain binning nax1 nax2 optelem aper targname" > tmpPFQ.head $> echo "# " >> tmpPFQ.head $> echo "# dataset freq width pmax fq root date-obs yr mn dy hh mm ss G bin nax1 nax2 optelem aper targname" >> tmpPFQ.head $> echo "# (1.s) (2) (3) (4) (5) (6.s) (7.s) (8) (9 10 11 12 13)(14)(15)(16) (17) (18.s) (19.s) (20.s) " >> tmpPFQ.head $> echo "#========================================================================================================================================================================" >> tmpPFQ.head $> cat tmpPFQ.head chkfreq.log > PATTERNFQ Check the format of the resulting logfile "PATTERNFQ" and if correct: $> rm tmpPFQ.head chkfreq.lis chkfreq.csh chkfreq.log autofilet.sav $> grep -v '^#' PATTERNFQ | awk -F: '{print $2}' | \ sed 's:_wav_rwc:+wav_rwc:g' | sed 's:_: :g' | \ awk '{ if ( $7==0 ) { print "PSUB/"$1"_rwc_"$3","$4","$5","$6 } \ else {print "FAIL/"$1"_raw_"$3","$4","$5","$6}}' | \ sed 's:+:_:g' > allframes.lis $> wc -l allframes.lis XXXXXX allframes.lis $> grep FAIL allframes.lis > failframes.lis $> wc -l failframes.lis 0 or YYYY failframes.lis If the "failframes.lis" file is non-empty, then: $> mkdir FAIL/ $> echo "\!quiet = 2" > fcidlbatch $> echo ".run framecopy.pro" >> fcidlbatch $> echo ".run framecopy.pro\n" >> fcidlbatch $> sed 's:_: :g' failframes.lis | sed 's:,: :g' | \ sed 's:FAIL/::g' | sed 's:.fits::g' | \ awk '{print "%ORIG/"$1"_raw.fits%, "$3", %FAIL/"$1"_raw_"$3".fits%"}' |\ sed 's:%:":g' | sed 's:^:framecopy, :g' >> fcidlbatch $> echo "\nexit" >> fcidlbatch $> idl -quiet < fcidlbatch If not present already, create a template script "mkf2m.tem" with the following content: grep &1 tmpinfo.lis > &1.inp echo "\nMKF2M \t -- &1.inp -- \t "`date +%Y-%m-%dT%H:%M:%S` >> fits2mef.log cat &1.inp >> fits2mef.log echo " " >> fits2mef.log echo "fits2mef CLEAN/&1_raw.fits ORIG/&1_raw.fits &1.inp >> fits2mef.log" > dof2mef sleep 1 ; source dof2mef \mv &1.inp TABLES/f2m/. \rm -f dof2mef If not present already, create a script "dof2mbatch" with the following content (this is a rather long script): #!/bin/tcsh echo "\nStarting fits2mef processing at "`date` echo "Generating lists of frames and unique root names..." echo -n " Total number of frames to merge into MEF files = " tail -n +6 PATTERNFQ | wc -l tail -n +6 PATTERNFQ | sed 's:#::g' | grep -v "_wav_rwc" | sed 's:_: :g' | sed 's:.fits::g' | awk '{if($7==0){print $3",PSUB/"$1"_rwc_"$3".fits,"$4","$5","$6","$7} else {print $3",FAIL/"$1"_raw_"$3".fits,"$4","$5","$6","$7} }' | sed 's:*:1:g' > tmpinfo.lis tail -n +6 PATTERNFQ | sed 's:#::g' | grep "_wav_rwc" | sed 's:_: :g' | sed 's:.fits::g' | awk '{if($8==0){print $4",PSUB/"$1"_wav_rwc_"$4".fits,"$5","$6","$7","$8} else {print $4",FAIL/"$1"_wav_"$4".fits,"$5","$6","$7","$8} }' | sed 's:*:1:g' > tmpinfo_wav.lis echo -n " Total number of frames accounted for = " cat tmpinfo.lis tmpinfo_wav.lis | wc -l awk -F/ '{print $2}' tmpinfo.lis | awk -F_ '{print $1}' | sort -u > _root.lis awk -F/ '{print $2}' tmpinfo_wav.lis | awk -F_ '{print $1}' | sort -u > _root_wav.lis echo -n " Total number of MEF files to write = " cat _root.lis _root_wav.lis | wc -l echo "Generating script to run..." echo "# DOALLF2MEF:\n" > doallf2mef.csh narg mkf2m.tem _root.lis '&1' >> doallf2mef.csh sed 's:tmpinfo:tmpinfo_wav:g' mkf2m.tem | sed 's:.inp:_wav.inp:g' | sed 's:raw:wav:g' > _mkf2m.tem narg _mkf2m.tem _root_wav.lis '&1' >> doallf2mef.csh echo "\n# DOALLF2MEF: Finished." >> doallf2mef.csh chmod 744 doallf2mef.csh echo "Executing script 'doallf2mef.csh'..." ./doallf2mef.csh sleep 60 mv doallf2mef.csh TABLES/f2m/. mv fits2mef.log TABLES/f2m/. \rm -f _mkf2m.tem tmpinfo.lis tmpinfo_wav.lis _root.lis _root_wav.lis sleep 3 echo -n " Number of frames merged into MEF files = " cat TABLES/f2m/*.inp | wc -l echo -n " Number of MEF files written = " \ls -1 CLEAN/*.fits | wc -l echo "Finished processing of batch at "`date` Now merge all individual frames back into STIS imsets: $> mkdir CLEAN $> mkdir TABLES/f2m $> chmod 744 dof2mbatch $> dof2mbatch Check whether everything went right: $> grep failed TABLES/f2m/fits2mef.log (should return none) $> \ls -lh CLEAN/ | awk '{print $5}' | sort -nu (should return only muliples of a single full-frame (i.e., N times ~4.3 Mb)) $> find CLEAN/ -type f -name '*.fits' | wc -l WWWWWW <-- should match the totalnumber of full-frame unbinned datasets $> chmod 644 dof2mbatch (<-- so we won't run it again inadvertently) That's it! ##############################################################################