SDFRED1
Data Reduction Software for Subaru Suprime-Cam

Introduction

Suprime-Cam is a mosaic CCD camera with ten 2048 x 4096-pixels CCDs. SDFRED1, the Suprime-cam Deep Field REDuction package 1, is the software package used for reducing Suprime-Cam data. SDFRED1 allows the immense data output from Suprime-Cam to be reduced semi-automatically, or even fully automatically, using standard parameters. After reduction, the reduced images are ready for scientific analysis.

SDFRED1 is optimized for photometry of deep field or blank field images. Images that contain objects that occupy a significant portion of a chip may not reduce properly with this package. Moreover, we have not performed any quantitative tests for shape measurements to images produced by SDFRED1, such as those for weak lensing analysis.  Special procedures and cautions for reducing such data will be described in each section.

Notices on Data

1. Data taken from November 2000 through December 2000: the CCD linearity has a problem for DET-ID=0,1,7,8,9.
2. Data taken from November 2000 through January 2001: the WCS information in the FITS header is wrong. The flag of CDELT1 is negated, and the CD matrix is wrong. Since SDFRED1 uses WCS information in step (11), registration (makemos.csh) fails with this data.
3. Data taken from October 2000 through March 2001: one CCD (DET-ID=6; w67c1) is dead. However, reduction with SDFRED1 is possible for nine chips. Note that CCDs were replaced in April 2001 and flat pattern and gain is changed, while the basic format of CCDs (e.g., overscan format, chip names) is unchanged.
4. All the data taken on and after July 21, 2008 cannot be reduced using SDFRED1 because Suprime-Cam was largely upgraded in July 2008, including CCD replacement and modification of the file format.

Copyright

The copyright for SDFRED1 software belongs to Masami Ouchi who originally developed the package, and to Masafumi Yagi who supplied the basic code and made improvements. You may freely copy and distribute SDFRED1, but the following two papers should be cited in any scientific paper based on data reduced with SDFRED1:

Yagi et al., 2002, AJ, 123, 66

Ouchi et al., 2004, ApJ, 611, 660

How to Obtain the SDFRED1 package

You can get the latest version of the package at:

We strongly recommend that you visit this web page from time to time to see updated information.

Other Software Requirements

• Basic software packages:
  C compiler, Perl, csh, sh/bash

• SExtractor
  http://terapix.iap.fr/rubrique.php?id_rubrique=91
  (Bertin & Arnouts 1996, A&AS, 117, 393)

• IRAF(this is not necessarily required for SDFRED11, but is pretty handy to have)
  http://iraf.nao.ac.jp/ or_ http://iraf.net/

  The latest SDFRED1 (20080610) has been tested with version 2.12.2

Computer Hardware and Operating System Requirements

SDFRED1 requires a UNIX-based computer equipped with 256MB of memory or more. Requirements for storage space depend on the quantity of data, but, typically you need several GB to several hundred GB. (To reduce the practice dataset, 10GB will besufficient.)

The latest version of SDFRED1 (06/10/2008) has been tested with Linux.

Installation

1. Uncompress the software package in an appropriate directory.

$ tar xvzf sdfred20080610.tar.gz 

or

$ gunzip -c sdfred20080610.tar.gz | tar xvf -

(In this text, % or $ at the beginning of a line represents a shell prompt.)

A directory with a name like sdfred20080610/ will be created in the current directory. The eight-digit number represents the date of the software release.

2. Compilation

Move (cd) into the sdfred20080610 directory and build the software as follows.

$ cd sdfred20080610/
$ ./configure
$ make all 

Then programs are installed into sdfred20080610/bin/

3. Adding directory to the PATH

Use an editor to add the sdfred20080610/bin directory (executables directory) to your PATH environment in your shell configuration file. In this example, /wa03a/subaru20/sdfred20080610/bin is the directory where SDFRED binaries are installed. You should modify to work with your environment.

bash users:

Edit ~/.bashrc *and* ~/.bash_profile as follows;

Example: 

$ emacs ~/.bashrc
$ emacs ~/.bash_profile

PATH=/wa03a/subaru20/sdfred20080610/bin:$PATH
export PATH

~/.bashrc is used when a new terminal is open, and ~/.bash_profile is used when you log into the computer. Therefore you need to modify both files.

After the files are updated, reflect the new setting to the current environment. This procedure is required only when you change the shell configuration files.

$ source ~/.bashrc 

csh/tcsh users:

Edit ~/.cshrc to add the directory where SDFRED binaries are installed (executables directory) as follows;

Example: 

% emacs ~/.cshrc

set path=( /wa03a/subaru20/sdfred20080610/bin $path )

After the files are updated, reflect the new setting to the current environment. This procedure is required only when you change the shell configuration files.

% source ~/.cshrc
% rehash

4. Setting environmental variables

LANG, and LC_ALL should be set as "C", so that shell scripts and Perl scripts work correctly

bash users:

Again, edit ~/.bashrc *and* ~/.bash_profile as follows;

$ emacs ~/.bashrc
$ emacs ~/.bash_profile

Add following two lines.

export LANG=C
export LC_ALL=C

And reflect the setting to the current session.

$ source ~/.bashrc

csh/tcsh users:

Again, edit ~/.cshrc as follows;

Example: 

% emacs ~/.cshrc

Add following two lines.

setenv LANG C
setenv LC_ALL C

And reflect the setting to the current session.

% source ~/.cshrc

5. Other settings

If you prefer to use IRAF,  don't forget to execute mkiraf in your "work directory".

Additional settings may be required, depending on your computer environment.

When you finished with the settings, make sure your environment is as follows

Example: 

$ env
$ which namechange.csh

In env command result, check LANG, and LC_ALL environment.

Make sure the "which" command can find the directory you have installed.

Once you have successfully finished the software preparation, you do not have to take these steps again.

Preparation of Data

Before staring a reduction, you need to sort data files into directories.

SDFRED requires that all input files must be in the "current" directory. If you want to process some files in a different directory, the files should be recognized as if they are in the "current" directory, by making symbolic links, or by other way.

Sample Data

A sample dataset (spcam_training_data.tar.gz; 753MB uncompressed) is available at spcam_training_data.tar.gz.

We have checked that the software works well for this dataset. If you encounter any problem(s) during regular data reduction, we suggest you  diagnose it using this well-tested sample data.

The specific examples given in this manual refer to this sample dataset.

Example:
extract sample data
$ tar xvzf spcam_training_data.tar.gz

After the data extraction, check the images. Which are the object frames? Which can be used as flat flames? Which are the standard star frames?

Moreover, inspect images by eye using saoimage-ds9 or other your favorite FITS viewer. It is a good starting point to check for such issues as "Are there any files (i.e., exposures) showing distorted/elongated stars?", which allows you to eliminate bad data.  

SUPA0020*.fits are the object frames (target object), and SUPA0019512*.fits are the standard frames in the sample dataset.

In this version (Ver.1.4), the data are assumed to be reduced/analyzed in two directories (object/ and standard/) which are in the same directory as spcam_training_data.

---(work directory root) - spcam_training_data/
                         - object/
                         - standard/

$ mkdir object standard

The result of using ls is;

$ ls 
object spcam_training_data spcam_training_data.tar.gz standard

Then link object frames into object/ directory

$ cd object/
$ ln -s ../spcam_training_data/SUPA0020*.fits . 

Copy blank map data

$ cp ../spcam_training_data/blankmap* .

Link standard object frames into standard/ directory

$ cd ../standard/
$ ln -s ../spcam_training_data/SUPA0019512?.fits . 

Retrieval from smoka

All data obtained with the Subaru Telescope can be obtained through the archive system,  SMOKA, after the 18-months of the proprietary period has passed.

After getting the desired data, we suggest moving the target object and the standard object one into separate directories.

Check that data in a directory have observed with the same filter (e.g. V-band), and be the same data type (object/standard). Otherwise, you need to separate them into directories.

Data Reduction Steps 

A typical procedure for reducing target object frames is:

Standard Object frames:

Each of the processes applies to a different subset of data frames. The basic data flow consists of making lists of these various subsets, and then providing these lists to various programs within the SDFRED package. Each step produces a new set of images or data files. The naming convention for the various new files are in parentheses after each step in the list above.

For reference, we provide typical times to reduce the sample dataset (five exposures), and a somewhat larger set (17 exposures) in the left- and right hand of the fourth columns, respectively.  The time is measured using an Intel Xeon 3.0GHz WS with RedHat 5. The time is not linear to the number of the exposures. Although the actual time it will take to reduce any dataset will differ, the times should provide an idea of the relative amount of time one can expect each step to take.

To save disk space, you can delete files after they have been used in the next step (except *.mos and *mflats*.fits, since they are also used in the standard object reduction).　 The users are advised to keep the H*.fits (renamed), fTo_RH*.fits (flat fielded), and AspgfTo_RH*.fits (AG probe masked) files, since you never know when you might need to re-reduce these data sets.

Note that other temporary files (tmp*; *.fits) are also produced in reduction processes. These can safely be removed.

Reduction of Target Object

Step 1: Basic Data Inspection and Renaming of Data Frames

$ namechange.csh [raw fits file list]

  raw fits file list = names of raw data files

Prior to data reduction, it is useful to rename the data files using information associated with sensible parameters e.g., date of observations, exposure, and component CCD.

The filename such as SUPA... is changed as H[Date][type][ID]_[chipname].fits.

where the date, YYMMDD, is one day prior to DATE-OBS, and corresponds to Hawaiian Standard Time (HST) of the first half of the night.

ID is the frame serial number of the observation day of each type(bias,dark,object). It should be noted that both target object(s) and standard star(s) have the same ID of "object".

Example: 

$ cd object/ 

enters the directory of object frames

$ ls -1 SUPA*.fits > namechange.lis

Notice that the option of "ls -1" is "minus one".

$ namechange.csh namechange.lis

The namechange.lis should be like that

$ cat namechange.lis
SUPA00204610.fits
SUPA00204611.fits
SUPA00204612.fits
... 

Result

H030425object025_si001s.fits
H030425object025_si002s.fits
H030425object025_si005s.fits
... 

If you make symbolic links to the files in ../spcam_training_data/, they still points to SUPA***, but their names show up as H030425object025_si001s.fits. That is OK.

Note that each CCD of Suprime-Cam has a name:

-------- AG probe location ----------
w67c1 w6c1 si006s si002s w7c3
w93c2 w9c2 si005s si001s w4c5

Step 2: Subtraction overscan and bias

 $ overscansub.csh [overscansub.lis]

   overscansub.lis = list of raw data files

The script overscansub.csh issues a command that subtracts the median value of the overscan region in each line, and trims the overscan region off from the frame. The Suprime-Cam CCDs typically have an overscan level of about 10000 ADU.

Since the CCDs in Suprime-Cam have very little bias pattern, our experience suggests that subtracting overscan should suffice for many cases.

Example:

$ ls -1 H*.fits > ovserscansub.lis
$ overscansub.csh ovserscansub.lis

The overscansub.lis should be like that

$ cat overscansub.lis
H030425object025_si001s.fits
H030425object025_si002s.fits
H030425object025_si005s.fits
... 

Checkpoints:

• Compare the values of original frame and overscan subtracted image. The latter should be about 10000 ADU smaller
• Checking the file headers with a procedure like IRAF's imhead (cl> imhead H*.fits) should show that the image sizes are smaller after removing the overscan region.

Step 3: Making flat field frames

 $ mask_mkflat_HA.csh [mkflat.lis]  [base name]  [lower value]  [upper value]

   mkflat.lis = list of files to use to make flats
   base name = basename for the flats
   lower value = minimum value to accept (0.5 is recommended)
   upper value = maximum value to accept (1.3 is recommended)

The script mask_mkflat_HA.csh creates a flat from files with objects. The flat file is used to correct the difference in sensitivities between pixels in a frame. Areas vignetted by the AG probe is masked out, normalized, and a median of them is taken.

There are three basic types of flats: sky flats (blank fields), twilight flats, and dome flats. Sky flats usually give the best result. This is because the optical path of the light for the night sky as well as the wavelength dependency are similar to those in your scientific frames. In fact, the target frames can be used to produce flats as long as there are no large objects that extend several hundred pixels in the frames.

Example:

$ ls -1 To_RH*.fits >mkflat.lis
$ mask_mkflat_HA.csh mkflat.lis obj 0.5 1.3

This is an example of making a sky flat by combining the object frames

After running the script, there should be flat files for the 10 CCDs.

obj_mflat_si001s.fits
obj_mflat_si002s.fits
...

The mflat files should have values around unity and should have a smooth pattern without much local structure. U-band and bands redder than z will have more structure than other bands. However, any local variations should be continuous. If there are abrupt changes in the flat values, consider creating another flat after eliminating (possible) bad exposures. Note that the value -32768 is the blank value used in SDFRED, and it is OK.

Note 1:

In principle, a flat can be produced with a minimum of three exposures. However, the smaller the number of frames used, the larger the noise and residual effects of objects in the frame. We recommend using at least six frames, ideally over 20 frames, to produce a (sensible) flat --- especially if you attempt to make a flat from sky frames. In the sample data, the flat is created from five exposures, yielding noticeable (but acceptable for robust reduction) error.

Note 2:

Keep in mind that the users should not mix the different types of flat exposures to make a flat frame because the background illuminations have intrinsically different slopes.   For example, SDFRED may produce flats with discontinuous stripes when applied to frames with different illumination patterns. This is due to the algorithm used in SDFRED.

Note 3:

The SDFRED command uses a parameter file to mask out known bad columns and hot pixels. The default parameter file is optimized for data taken after August 2002 (Messia V). To reduce data taken before this, follow the procedures below.

$ cd sdfred20080610

For data taken before March 2001 (old CCD):

$ cp sdfredSH/mask_mkflat_HA/blankmap_oldCCD/* sdfredSH/mask_mkflat_HA/blankmap/

For data taken between April 2001 and August 2002 (Messia III):

$ cp sdfredSH/mask_mkflat_HA/blankmap_messiaIII/* sdfredSH/mask_mkflat_HA/blankmap/

To return to the default:

$ cp sdfredSH/mask_mkflat_HA/blankmap_messiaV/* sdfredSHmask_mkflat_HA/blankmap/

Note 4:

In the sample data, the flat is created in object/ directory, and will be shared for reducing standard-star data (by copying it into the standard/ directory). In many cases, several targets taken with an identical filter configuration can be mixed to create flats. If this is the case, it would be prudent to create another directory ../flat/. Subsequently, make symbolic links of To_R* files in all targets, and execute any commands in flat/ directory. (Note that the standard star frames that have shorter exposure times must not be included in the list.)

e.g.)
---(work directory root) - object1/
                         - object2/
                         - object3/
                         - flat/
                         - standard/

Step 4: Flat Fielding

 $ ffield.csh [ffiled_mf.lis]  [ffield_im.lis]

   ffield_mf.lis = list of flats to be used
   ffield_im.lis = list of (overscan subtracted) files to be flat fielded

This command corrects the pixel-to-pixel variation in sensitivity, and the effect of vignetting of the telescope optics.

Example:

$ ls -1 obj_mflat*.fits > ffield_mf.lis
$ ls -1 To_*.fits > ffield_im.lis
$ ffield.csh ffield_mf.lis ffield_im.lis

After the flat fielding, the background in each file should be almost flat. The circular illumination pattern seen in the raw data at the edge of the focal plane (w67c1, w93c2, w7c3, w4c5) should be nearly gone at this stage. If the flat is poor, you will see a low-level (several percent of variation) illumination pattern. However, even if you are not happy with the residual pattern, you can safely ignore it and proceed to Step 5 as long as you don't wish to achieve very high accuracy in photometry (i.e., down to several percent). This is because such an illumination pattern will be subtracted at the later, at Step 8.

On the other hand, if your scientific goals require a rather high accuracy in photometry --- down to several percent, consider going back to Step 3 to make another flat. The "ffield.csh" used in Step 4 will help you to see the difference in the patterns appeared in the frames listed in mkflat.lis. We suggest eliminating the frames that have different patterns from those of the object frames from mkflat.lis in the Step 3.
 

Step 5: Distortion correction and atmospheric dispersion correction

 $ distcorr.csh [distcorr.lis]

   distcorr.lis = list of (flat fielded) files to be corrected

The script distcorr.csh corrects the field distortion due to the telescope optics, and the differential atmospheric dispersion. The input frames are assumed to be flat-fielded images. The corrections are based on the airmass and other values recorded in the FITS header.

Example:

$ ls -1 fTo_RH030*.fits >distcorr.lis
$ distcorr.csh distcorr.lis

After the distortion correction, diagonal patterns may show up in the background. This is due to the fact that fractional pixel shifts smooth out the noise while integer pixel shifts leave the original noise characteristics intact.

Step 6: Measurement of PSF size

 $ fwhmpsf_batch.csh [fwhmpsf_batch.lis] [max number of objects] 
   [min peak flux] [max peak flux] [min FWHM] [max FWHM]

   fwhmpsf_batch.lis = list of images to check PSF
   max number of objects = the number of stars to use to measure 
                           the PSF in each image
   min peak flux = minimum peak flux of stars to use
   max peak flux = maximum peak flux of stars to use
   min FWHM = minimum FWHM of stars to use
   max FWHM = maximum FWHM of stars to use

Before coadding, equalization of the PSF is required. The script fwhmpsf_batch.csh is used to determine an appropriate target PSF for a list of images. The script measures the FWHM of the PSF in several images. The script outputs a log and a histogram (exposure by exposure) to the standard output.

Example:

$ ls -1 gfTo_RH03042*.fits > fwhmpsf_batch.lis
$ fwhmpsf_batch.csh fwhmpsf_batch.lis 50 2000 30000 2.0 7.0 \
 > fwhmpsf_batch.log

The log file should contain entries such as:

gfTo_RH030425object025_si001s.fits   3.60   1 6 15 13 0
gfTo_RH030425object025_si002s.fits   3.80   1 1 20 16 0
gfTo_RH030425object025_si005s.fits   3.60   2 13 19 0 0
...
3.3 |
3.4 |
3.5 |**
3.6 |**
3.7 |*
3.8 |**
3.9 |**
4.0 |
4.1 |*

The log format of each line is as follows:

[name of image]

[mean FWHM of PSF]

[number of objects with within 0.1'' of mean PSF-0.2]

[# within 0.1'' of mean PSF-0.1]

[# within 0.1'' of mean PSF]

[# within 0.1'' of mean PSF+0.1]

[# within 0.1'' of mean PSF+0.2]

An ASCII histogram following the log illustrates the distribution of mean PSFs.

Some training is required to determine the target PSF size.

Here, the sample images have FWHM values centered around 3.7 pixels. Adopting Target=4.1 would be the most conservative selection if you are interested in measuring fluxes of objects, but not in their structure. Target=3.9 would be a choice, if you believe the difference between 4.1 and 3.9 is negligible. Alternatively, if you have plenty of data, you may want to adopt a more strict threshold to exclude images having degraded PSFs (e.g, all data whose PSF are larger than 3.7).  The selection depends on your scientific goals.  

Note 1:

 $ fwhmpsf.csh [image file] [max number of objects] 
   [min peak flux] [max peak flux] [min FWHM] [max FWHM]

   image file = the image to check PSF
   max number of objects = the number of stars to use to measure 
                           the PSF in each image
   min peak flux = minimum peak flux of stars to use
   max peak flux = maximum peak flux of stars to use
   min FWHM = minimum FWHM of stars to use
   max FWHM = maximum FWHM of stars to use

Use the script fwhmpsf.csh to find the PSF of a single image. The parameters are the same as for fwhmpsf_batch.csh. Just supply the name of a image rather than a list of images.

Example:
$ fwhmpsf.csh gfTo_RH030425object025_si001s.fits 50 2000 30000 2.0 7.0

This produces output like:

gfTo_RH030425object025_si001s.fits   3.60   1 6 15 13 0

This output indicates that the image gfTo_RH030425object025_si001s.fits has a PSF FWHM of 3.6 pixels.

Note 2:

 $ starselect.csh [image name][max number of objects][min peak flux]
     [max peak flux][min FWHM][max FWHM][output file]

   image name = name of image to check
   max number of objects = the number of stars to use to measure the PSF
   min peak flux = minimum peak flux of stars to use
   max peak flux = maximum peak flux of stars to use
   min FWHM = minimum FWHM of stars to use
   max FWHM = maximum FWHM of stars to use
   output file = name of file with location of selected stars

The script starselect.csh is useful for searching for the appropriate parameters ([max number of objects] [min peak flux] [max peak flux] [min FWHM] and [max FWHM]) for selecting stellar objects in an image.

$ starselect.csh gfTo_RH030425object025_si001s.fits 50 2000 \
  30000 2.0 7.0 output.reg

The script will produce an output file (output.reg) that contains the location of stellar objects satisfying the given criteria. The output is formatted so that the stellar objects are plotted with green circles when you plot using saoimage-ds9. If the majority of the selected objects are "real stellar objects"  (stars for many cases), then the parameters are appropriate for psf_match for a given image. If you realize that the quality of the data varies image by image, determine whether or not a single set of parameters can be applied for whole the data set.  If it cannot, it is better to run psfmatch_batch multiple times using the appropriate criteria for each subset of data.

Using saoimage-ds9 is the easiest way to display an image and overlay the location of the selected stars.

$ ds9 gfTo_RH030425object025_si001s.fits

Select "Region", "Load", and select output.reg. Then green circles will be overlaid on the image. If more than half of the objects selected are stellar objects, the parameters you adopted are appropriate

Note 3:

The scripts fwhmpsf_batch.psf, starselect.csh and psfmatch_batch.csh(next step), may not work in crowded fields. In such fields, it may be necessary to estimate the PSF manually. If this is the case, to obtain the same results as the psfmatch script, each image that has a PSF more than 0.1'' smaller than the target PSF should be Gaussian smoothed with a Gaussian that has a sigma=sqrt(PSF_target^2 - PSF_image^2)/2.35482.  The psfmatch program iterates around the estimated value of the sigma until the results converges to the best matched final PSF.

Step 7: Equalize the PSF size

 $ psfmatch_batch.csh [psfmatch_batch.lis] [max number of objects]
     [min peak flux] [max peak flux] [min FWHM] [max FWHM] [target FWHM]

   psfmatch_batch.lis = the list of images to match to a single PSF
   max number of objects = the number of stars to use to measure the 
                           PSF in each image
   min peak flux = minimum peak flux of stars to use
   max peak flux = maximum peak flux of stars to use
   min FWHM = minimum FWHM of stars to use
   max FWHM = maximum FWHM of stars to use
   target FWHM = FWHM to smooth all the data to

The script psfmatch_batch.csh attempts to match the PSF of all images to be combined to a predetermined target FWHM. Images with PSFs smaller than the target (within a small range) are Gaussian smoothed, other images are simply copied. The target PSF should represent the typical PSF for the exposure, having the worst (i.e., the largest) PSF among the exposures to be combined. 

The command prints a log to the standard output with the following columns:

[name of psf_matched_image]
[FWHM of PSF after matching]
[number of objects with within 0.1'' of target PSF-0.2]
[# within 0.1'' of  target PSF-0.1]
[# within 0.1'' of target PSF]
[# within 0.1'' of  target PSF+0.1]
[# within 0.1'' of  target PSF+0.2]

The log can be used to check whether or not the PSF matching worked properly. To determine whether or not the command has ended successfully, make sure that the number of objects falling into (1) the bin of +/-0.1" of the final FWHM must be the peak of the distribution, and (2) outside bins e.g., PSF+0.2 arcsecond and PSF-0.2 are significantly small.

  *
  *
  *
  *
 **
 **
 ***
 ***
 ***
*****

Example:

$ ls -1 gfTo_RH03042*.fits > psfmatch_batch.lis
$ psfmatch_batch.csh psfmatch_batch.lis 50 2000 30000 2.0 7.0 3.7 > psfmatch_batch.log &

The command produces output like:

pgfTo_RH030429object017_si001s.fits   3.70   0 5 38 6 0

The IRAF task "imexam" is handy for checking PSFs. (Display image; cl> imexam image.fits; place cursor above a star; type "r" or "a" to measure FWHM)

Keep in mind that each software routine may be using different fitting algorithms and may return different FWHM values. SDFRED adopts FWHM values generated by SExtractor, which are different from those produced by IRAF's imexam task. The purpose of checking with IRAF is not to find an exact match in the FWHM values, but to confirm that the output images have comparable PSF sizes after the matching.

The appropriate parameter values for psfmatch will change depending on the quality of the data. Different bandpasses, integration times, and weather conditions will require different parameters.

Step 8: Subtracting the Sky Background

 $ skysb.csh [skysb.lis] [sky-mesh]

   skysb.lis = list of images to sky subtract
   sky-mesh = size of mesh for determining sky values

The script skysb.csh (1) computes a mesh pattern that represents the sky background, (2) interpolates the pattern, and (3) subtracts it from the image. The script creates a grid --- referred to as "sky-mesh size squares" ---  on the image with a grid spacing having the half of the "sky-mesh" size. An appropriate sky-mesh size will be selected for each mesh, and assigned to the pixel located at the center of the mesh. After rejecting the outliers, the sky values for other pixels will be given by interpolating bilineary from the surrounding meshes. Note that the sky-mesh size must be selected at least twice the largest object in interest due to the Nyquist sampling theorem.

Example:
$ ls -1 pgfTo_RH03042*.fits > skysb.lis
$ skysb.csh skysb.lis 64 > skysb.log

Once the sky background level is subtracted, the background in an image should be around zero without a spatial gradient. If there is an extended object(s) spreading over a large fraction of the image, the algorithm will most likely fail. Subtraction of sky background in crowded fields requires special data handling and you will need to estimate the sky background manually.

Step 9: Masking the AG Shade

 $ mask_AGX.csh [mask_AGX.lis]

   mask_AGX.lis = list of files to mask

The script mask_AGX.csh will mask areas vignetted by the AG (Auto-Guider)  probe by the value -32768. The script should only affect the top few hundred rows of the data from chips w671, w6c1, si006s, si002s, and w7c3. Other files are not affected.

Example:
$ ls -1 spgfTo_RH03042*.fits > mask_AGX.lis
$ mask_AGX.csh mask_AGX.lis

Although only half the CCDs are potentially affected by the AG probe, the input file list should include all the object files so that files with the same naming convention exist to make list-making for subsequent steps easier.

Step 10: Masking Bad Pixels

Data in some pixels may be corrupted due to instrument trouble and/or other problems which may have occurred during the observation. Such regions must be common among the exposures (i.e., they are not time variable), and should be masked accordingly. For instance, we suggest masking the background areas where flattening fails and systematically deviates from zero. If plenty of exposures cover the observed region, we suggest not spending much time with this step. This is because outliers will be rejected automatically in Step 12.　

The SDFRED1 package offers three methods --- linear, circular, and rectangular regions --- to specify regions to be masked for eliminating bad pixels. Here, "Linear region" connects the two points (x1,y1) - (x2,y2), extends the line to the edges of the image, and masks the pixels within "width" from the line. The "circular region" masks the pixels in a circle. The "rectangular regions" masks rectangular regions aligned to the pixel coordinate.

Linear region

 $ line_bank [input image] [x1] [y1] [x2] [y2] [width] 
    [blank value] [output image]

   input image = name of image to mask
   x1 = x coordinate of start of line
   y1 = y coordinate of start of line
   x2 = x coordinate of end of line
   y2 = y coordinate of end of line
   width = width of line
   blank value = mask value (usually -32768)
   output image = name of masked image

The script line_bank masks a linear structure such as satellite trails.

Example:

$ line_blank AspgfTo_RH030425object025_si001s.fits \
  88 112 1940 837 30 -32768 lAspgfTo_RH030425object025_si001s.fits

Circular region

 $ circular_blanks [input image] [blanklist] [blank value] [output image]

   input image = name of image to mask
   blank list = a text file describing the x and y coordinates, as well as the radii of the areas to be masked
   blank value = mask value (usually -32768)
   output image = name of masked image

The script circular_blanks masks circular regions.

Example:

$ circular_blanks lAspgfTo_RH030425object025_si001.fits \
  blanklist -32768 clAspgfTo_RH030425object025_si001s.fits

where blanklist looks like:

$ cat blanklist
365 1835 80
1202 3582 100

Rectangular regions

 $ blank.csh [blank list]

   blank list = list of images to be masked 

For each image, xxx.fits, in the blank list the script blank.csh will look for a file named blankmap_xxx in the same directory, and mask rectangular regions specified in the file to -32768. Each line in the file blank_xxx should contain the x and y coordinates of two opposite corners of a rectangular area.

The IRAF routine imexam is useful for getting the coordinates. (cl> imexam; press "b" at two corners to define a rectangle; the coordinates of the corners will be printed to the screen in the order of x1 x2 y1 y2.)

Example:

$ ls -1 AspgfTo_RH03042*.fits > blank.lis
$ blank.csh blank.lis

Mask files have been included for a subset of images,

blankmap_AspgfTo_RH030425object025_si002s

...

These files have entries like:

$ cat blankmap_AspgfTo_RH030425object025_si002s
1974 2034 2356 2634
1528 1804 4024 4070
...

The script masks the regions specified in the corresponding blankmap_xxx file. If the blankmap_xxx file does not exist, the script will simply copy the image file to the output.

Note that the blankmap_* parameter files included in the sample dataset are from the Subaru Deep Field project and are not necessarily suitable for the sample dataset. If all of the sample files are applied to the sample data, areas that shouldn't normally be masked will be masked. These files are strictly for practice use. You may wish to create your own mask files to apply to the sample data.

Step 11: Estimating Alignment and Scaling

 $ makemos.csh [makemos.lis] [starsel nskysigma] [starsel npix] 
               [starsel peakmin] [starsel peakmax] 
               [aperture phot radius in pix] [output mos-file name]

   makemos.lis = list of images to align
   starsel nskysigma = signal to noise ratio of objects per pixel
                       to use for alignment
   starsel npix = number of continuous pixels with [starsel nskysigma] 
                  to identify object
   starsel peakmin = minimum value of peak pixel of alignment stars
   starsel peakmax = maximum value of peak pixel of alignment stars
   aperture phot radius in pix = radius to use for aperture photometry
   output mos-file name = file to record alignment and scaling

Signal-to-noise ratio (S/N) can be improved by combining multiple images (if you have them) to produce a final image. The script makemos.csh determines the shifts, rotations, and flux scales of different images. The script identifies stellar objects in each image and determines the the shifts, rotations, and flux scale from stellar objects common to multiple images. The first image in the list is used as the reference image.

Example:

$ ls -1 bAspgfTo_RH03042*.fits > makemos.lis
$ makemos.csh makemos.lis 5 20 500 10000 10 all.mos > makemos.log

The script will print to the standard output the number of stellar objects selected for alignment and scaling.

     ...
     selected stars = 721
     ...

The script is likely to fail if the number of selected stars per image is either small (< 30) or very large (>1000). Optimizing key parameters such as [starsel nskysigma], [starsel npix], [starsel peakmin], and [starsel peakmax] will help the script to select appropriate stellar objects. 

The best parameters for selecting objects in this step may be different from PSF measurement for many cases. This is because a different underlying algorithm is employed to find a wider range of objects to determine relative positions and flux scaling that works over a range of fluxes.

Example:

$ cat all.mos
bAspgfTo_RH030425object025_si001s.fits 0.000000 0.000000 0.000000 1.000000
bAspgfTo_RH030425object025_si002s.fits 1.601005 4088.551744 -0.000275 0.989406
bAspgfTo_RH030425object025_si005s.fits -2118.787371 1.104977 -0.000282 0.983195
... 

As shown in the above, you will see five parameters (i.e., columns) in the output *.mos file: the name of the image, the x offset, the y offset, the counter clockwise rotation (radian), and flux ratio. If each result has four output parameters followed by the image name, the alignment or/and scaling have successfully finished. If the alignment and scaling have failed, the output file may not be produced at all, miss some parameters, or have unreasonable values. 

We -- the SDFRED support team -- have been making continuous efforts to provide users more sophisticated method(s) that examines all.mos. Realizing the situation, however, we wish to share the following tips:

1. Inspecting the final image created in the next step must be done. However, bear in mind that it is not the ultimate method. If the number of exposure is large, it is difficult to detect some small defects by visual inspection in the final image.
2. It is always a good idea to make a plot of the 2nd vs. 3rd columns stored in all.mos. The result shows the relative position of each shot, and represents the dither pattern as well as the chip positions. If there is a large leap in value, the matching has failed.
3. The distances between CCD chips should be almost constant. (A slight difference may exist due to atmospheric dispersion between chips.) If the distances between any arbitrarily chosen chip pairs for the same exposure (e.g., between si001s and w67c1) has changed significantly exposure by exposure, the data of the corresponding exposure would be incorrect.  
4. The fifth column of *.mos (relative flux) of a chip should be almost proportional to the exposure time, if sky condition is photometric. (It is affected by atmospheric extinction (airmass), however)

In the next step, each image is converted with the data in all.mos as follows;

 
x_mos =   cos(theta) x  -sin(theta) y + x_local
y_mos =   sin(theta) x  +cos(theta) y + y_local

Note 1:

Step 12: Combining

 $ imcio2a [parameters] [mos file] [result image]

 parameters = parameters that define the combining algorithm usually 
  "-dist_clip -nline=20 -dtype=FITSFLOAT -pixignr=-32768"
 mos file = file containing the alignment and scaling values 
            (output from makemos.csh)
 result image= the name of the final image

imcio2a combines the images into a final combined image using the output from makemos.csh (*.mos). Using the parameter -dist_clip will combine the images using a clipped mean algorithm.

Example:

$ imcio2a -dist_clip -nline=20 -dtype=FITSFLOAT -pixignr=-32768 all.mos all.fits

The parameter -dist_clip can be replaced by -dist_med to get a weighted median combined image or -dist_add to use a weighted mean pixel values.

Note that the header of the output image is incomplete. Use the first file listed in makemos.lis in the previous step as a reference header.

Here are the meanings of the typical parameters:

-dist_clip : use a clipped mean algorithm for combining

-nline=20 : set the y direction buffer width to 20

-dtype=FITSFLOAT : make the output data floating point

-pixignr=-32768 : ignore pixels valued -32768

 $ imcio2a -h

Reduction of Standard Object

Steps 1S through 4S describe a typical procedure for reducing standard stars data. Since the physics behind this is the same as for reducing target objects, you can essentially repeat the procedure. Don't forget to work in the standard/ directory. The flat frames must be the same as those used for the objects, therefore be sure to copy them from the object/ directory.

Step 1S Renaming

$ namechange.csh [raw fits file list]

  raw fits file list = names of raw data files

Renaming is done in the standard/ directory

Example: 

$ cd standard/ 

enters into the directory of standard frames

$ ls -1 SUPA*.fits > namechange.lis
$ namechange.csh namechange.lis

The namechange.lis should be like that

$ cat namechange.lis
SUPA00195120.fits
SUPA00195121.fits
SUPA00195122.fits
... 

Result

H030330object044_si001s.fits
H030330object044_si002s.fits
H030330object044_si005s.fits
... 

Step 2S Overscan and bias subtraction

 $ overscansub.csh [overscansub.lis]

   overscansub.lis = list of raw data files

In standard/ directory, overscan is subtracted from all the data as follows,

Example:

$ ls -1 H*.fits > overscansub.lis
$ overscansub.csh overscansub.lis

$ cat overscansub.lis
H030330object044_si001s.fits
H030330object044_si002s.fits
H030330object044_si005s.fits
... 

and, To_RH030330object044_si001s.fits ... are created.

Step 3S Flat fielding

 $ ffield.csh [ffiled_mf.lis]  [ffield_im.lis]

   ffield_mf.lis = list of flats to be used
   ffield_im.lis = list of (overscan subtracted) files to be flat fielded

The flat frames used in this step (ffield_mf.lis) must be identical to those used for the target(s) in order to cancel out the uncertainty in the normalization.

Example:

$ cp ../object/obj_mflat*.fits .
$ ls -1 obj_mflat*.fits > ffield_mf.lis
$ ls -1 To_RH*.fits > ffield_im.lis
$ ffield.csh ffield_mf.lis ffield_im.lis

and fTo_RH030330object044_si001s.fits ... are created.

Step 4S Distortion correction and atmospheric dispersion correction

 $ distcorr.csh [distcorr.lis]

   distcorr.lis = list of (flat fielded) files to be corrected

The distortion correction is required since it slightly changes the sizes of the pixels, yielding slightly different flux value(s).

Example:

$ ls -1 fTo_RH*.fits >distcorr.lis
$ distcorr.csh distcorr.lis

Step 5S Correction of relative flux scale among chips

Recall that the relative flux "scale" between different CCD chips has not yet been corrected even after the Step 4S. For example, the sensitivity of w67c1 is about one half in 2002-2008/06 data. If there is a star that has 10000 ADU in w67c1, it should have ~ 20000 ADU if it was observed with the other chip. The relative flux scale should be corrected according to the *.mos created in step (11) of the target object.

This step is unnecessary if the standard star is only in si001s. Since standard stars are distributed in several chips in the sample data, this process is needed.

In this step, the data is divided by a typical relative value of the chip to the reference chip. Currently (SDFRED ver1.*), the script for this step is not provided. Users should do this process manually.

Acknowledgment