IVM Data Reduction Procedures

This document describes how to reduce raw IVM magnetogram data using the IDL procedures in /solar/IVM/idl. The procedure is the one we use for reducing the morning survey magnetograms in near real-time, and as such it may not include techniques which could improve the results but are very slow. On the other hand, it does

compute the flat-field corrections in the best way we know,
coalign successive images by rigid translation, using the geometry images to determine the shifts,
locate the line center spectral position pixel by pixel, and
compute the magnetic field parameters using a decent implementation of the derivative method.

The software described here is the version of June 1995.

Contents

Data files
Programs
Calling syntax
Procedure
Options
Time and storage requirements
Restrictions

Data files

The magnetogram raw data file is named IMGRyymmdd.hhmm, with the lower-case characters replaced by the date and time of the observation. The data file is typically 62 MB, though sometimes smaller.
There are five ancillary files needed:
- two dark frames, named IDK[01]yymmdd.hhmmss,
- two flat-field data files named IFA[01]yymmdd.hhmmss, and
- a calibration data file named ICARyymmdd.hhmm.
These files total about 20MB.
Output is to a FITS format file, named IMGMyymmdd.hhmm, containing six images: continuum intensity, field line-of-sight component, field transverse component, field transverse azimuth (referenced to celestial north), line center offset in A, and line-of-sight velocity in km/sec.

Programs:

The driver is a function called rivm. It calls a number of other routines, many of which are in the IVM directory, /solar/IVM/idl. Some Yohkoh software routines are used.

Calling syntax

gram = rivm(calfile,datafile[,fitmap])

The names of the calibration and data files are the only required arguments. Default dark and flat files are the most recently observed dark and flat, before the data observation. Their names are recorded in the data file header and extracted by rivm. If different dark and flat files are needed, they can be specified with keyword variables.

Procedure

Rivm executes the following steps in the reduction process:

Open a journal file in the current directory, named 'rivm_yymmdd_hhmm'. The timestamp in the file name is the time of the data reduction. This journal is useful later on, since it contains printouts of parameters in the flat-field fits, the coregistration shifts, and sample results from the magnetic-field derivation.
Read the header (Observation block or Oblk) from the data file. Extract from it the image size, number of spectral steps, number of spectrum repeats, Fabry-Perot settings for each spectral step, wavelength of the line. Determine if 4-frame or 6-frame modulation sequence was used.
Calculate flat-field images for each wavelength, using procedure parfl2. The details are described in the header of parfl2, but the intent is to separate the overall vignetting, and fixed features such as dust, from features which depend on the individual wavelength setting. This must be done without distorting the line profile in the process.
Apply the flat to the calibration data. The continuum wavelength setting is used for all the cal data, so only one of the flat frames is needed. Demodulate the calibration data and construct the calibration matrix, which consists of a 4x4 matrix at each pixel in the image plane. The matrix is smoothed using a 16 x 16 median filter, since it is expected that pixel-to-pixel variations will be only noise. Invert the 4x4 matrix at each spatial pixel and write the result to a file named 'ICAR..._matinv'.
Split the data file into two files, one for data-camera images and one for geometry-camera images--unless there are only data-cam images, as for some early datasets. The output files are named IMG0... and IMG1..., with the remainder of the filenames copied from the data filename.
Flat-field the data-camera images, subtracting a mean dark level. Write the result to a file named 'IMG0..._df'.
If there exist geometry images, flat-field them and then use them to compute the spatial shifts of each image in the set, compared to a reference image (the one at the middle of the set). Use these shifts to coalign the data-camera images. The assumption here is that the majority of the difference between successive frames which is caused by 'seeing' is a rigid translation, the same for the geometry and data cameras. And that the geometry camera sees no changes due to polarization modulation, since it's looking at a wide wavelength range. Any higher-order image distortion is ignored. We might be able to reduce the seeing-noise by 'destretching' the images, but it would be time-consuming and possibly subject to other noise in the data which would make destretching ineffective. The files produced are called 'IMG1..._df' and 'IMG[01]..._df_re'. The [x,y] pixel shifts are written to an IDL save file named 'shiftsyymmdd_hhmm', identified by the time of processing. These data may be useful for debugging if the results seem odd. The values are usually around a tenth of a pixel if the guider was working well; a pixel or two if the guider was off. Many-pixel values indicate pathology: occasionally the guider will jump out of the spot for one frame, then return, but lots of large values probably means the flat data are bad.
If the spectral scan is repeated in the observation, the spectra are averaged. We have often used a setup with 20 spectral points, repeated twice. Write the result to 'IMG0..._df_re_av'.
Optionally, subtract a fixed fraction of the total intensity in each frame from each pixel in the frame. The amount is set by the keyword SCAT. The purpose is to correct for light diffusely scattered in the IVM instrument, as by the retarders, but it is a rather ad hoc correction based on residual sky brightness in limb observations. Use with much caution. If the correction is done, the output is written to 'IMG0..._df_re_av_sc', and the '_sc' persists through successive stages.
Demodulate the data, to obtain uncalibrated Stokes images, using demod or demod2 depending on which modulation sequence is appropriate. Write the result to 'IMG0..._df_re_av_st'.
Apply the inverted calibration matrix to the Stokes data, to remove instrumental offsets and crosstalk, and correct the scaling of the Stokes parameters. Write the result to 'IMG0..._df_re_av_st_ca'.
Optionally, subtract a background corresponding to leakage of the shutter during CCD readout. This was a zero-order correction for the leakage of the Ferroelectric liquid crystal shutters previously used. If applied, the output file is 'IMG0..._df_re_av_st_ca_sh'.
Compute the magnetic field, using the JLS method of comparing the polarizations at a chosen wing position to the derivatives of the intensity profile at the same wavelength position. Multiplies the normalized polarization parameters by intensity to recover Stokes data. Then computes an average intensity spectral profile from a region near the center of the image, and fits it with a Gaussian absorption profile on a continuum which may have slope and curvature. This canonical I fit is used as a starting point in the remainder of the processing. For each spatial pixel, extract the Stokes profile, then call bjeff6, which first fits the I profile. The fit at the previous point is used for starting parameters in the least-squares fit, unless the previous fit was invalid. The intensity derivative at the chosen offset DL is computed from the fit. Smooth the Q,U,V profiles slightly using a convolution with a symmetric kernel, then combine data from several wavelength points near DL on both the red and blue side of the line, fitting points within 40 mA of DL with a quadratic curve. This little dance permits us to use several spectral points to reduce the noise in the polarization data. Optionally return an image of 1/0 points indicating the validity of the I profile fit at each point.
Return the magnetic field data as a Magnetogram structure consistent with Haleakala Polarimeter data and other reduction programs, first clipping any aberrant data values which would invalidate the automatic scaling used in creating the FITS file. Also write the data to a FITS file, with a header intended to preserve much of the data in the original Oblk. The output data includes maps of both line-center position and line-of-sight velocity, and heliographic position (although the latter should not be taken too seriously due to uncertainties in the basic IVM pointing methods).

Options

The options are many, but the default values are usually right so the options are often not required. Fitmap is an optional argument; the remainder are implemented as keywords.

OLD_DATA: Set this keyword if the observations are from the very early days in 1992, before the header sizes were set. It causes RIVM to query the operator for the correct header sizes.
D0FIL, F0FIL: Use these keywords to override the dark and flat filenames in the header. Useful for old data with these entries missing; also if the appropriate dark/flat were obtained just after the data file rather than just before it.
NPTS: Numeric entry to specify the number of spectral positions, for files where this number is not found in the header.
NFUS: Numeric entry to specify the interval in Fabry-Perot units between successive wavelength points. Normally extracted from the header.
FOURSTEP: Set to force demodulation using demod_2, assume the data and calibration were obtained with four-step modulation sequence. This is the default if the header item NFRAMES is 4.
CALSMOOTH: Numeric entry which overrides the default 16-pixel smoothing which is applied to the calibration matrix.
JUMP: Set to one of the strings 'flat', 'shift', 'avg', 'scat', 'demod', 'calib' or 'bgram' to jump into the processing sequence, skipping already executed stages. This may be useful if one wishes to do the magnetic field computation at a different wing position, for example, and the calibrated Stokes file has already been obtained.
REDFLAT: Set if the flat-field data have already been reduced and the inverse flat files are present in the working directory. Skips processing of raw flat data. This is helpful with flare-mode data, i.e. rapid sequences of magnetograms depending on a common flat.
REDCAL: Set if the inverse calibration matrix is present in the working directory. Skips processing of the calibration data file. Again, handy with flare-mode data where one cal serves several magnetogram datasets.
NO_REG: Set to skip coalignment of frames. Usually coalignment is preferred.
SCAT: Set to apply the diffuse scattered light correction. The value specified for scat is the fraction of the average to be subtracted from each frame. This may be useful, but its validity hasn't been carefully explored.
SHUT: Set to apply the shutter leakage correction using shuttersub. Only needed for early data with LC shutter; doesn't work too well for that. Set shut to a string containing the name of the correction file.
NO_BGRAM: Abort processing after computing calibrated Stokes data; do not compute magnetic fields.
DL: Offset, in Angstroms, from line center at which the program is to measure the polarizations and intensity derivative. The default value is 0.100 A, which appears to work the best.
PLOT: If set, the interval in sequential pixels between plots of the Stokes profile fits in bjeff6. These plots are useful for estimating the quality of the data and of the fits, though they increase processing time somewhat. A value of 250 or 500 may be appropriate.
FITMAP Variable into which a yes/no map of the Intensity fit validity is returned.
KEEP_INT: If set, keep all the intermediate files. The default is to delete most of them, retaining the inverse calibration matrix file, the inverse flats, and the final calibrated Stokes file.

Time and storage requirements

The intermediate files are by default deleted, but 130 MB of disk space is required for short-term storage and the Stokes file. If the KEEP_INT option is selected, about 400 MB of free space is required, in addition to the 80 MB needed for the data files. The output FITS file is 660K for a 256 x 256 image size, and can be compressed a little, to about 450K. Processing takes about 35 minutes for a 256 x 256 file with 20 spectral points and two spectral scans, on a Sparc-20/50.

Restrictions

Image is assumed to be square.

Assumes 6.0 mA per unit of F-P setting. This value has been determined from the separation of the 630.15 and 630.25 nm lines. Note that the steps in wavelength do not have to be uniform, but there is some code in xbgram6 (95Apr) that chooses which points in the I-profile to fit and that assumes uniform wavelength steps.

Home

Back

Last updated: 1996 February 15
mickey@ahinahina.ifa.hawaii.edu