Fast Holographic Deconvolution (FHD)

FHD is an imaging algorithm for radio interferometers. It uses the full complex gain for every baseline for gridding, and pre-computes the uv-plane covariance matrix (the Holographic Mapping function). In its current implementation, FHD follows a centroided matched pursuit algorithm for deconvolving point sources. While FHD does not refer back to the visibilities once they have been gridded to a dirty image, it is still properly classified as a visibility-based subtraction algorithm since sources are forward-modeled through the mapping function.

Aside from the basic use of snapshot imaging, there are a number of extensions that expand its capabilities. Most notably, FHD can perform a joint deconvolution of any set of visibilities by regridding each snapshot to a common HEALPix mesh. This allows for long integrations, and even for joint deconvolution of data from different instruments. Additionally, if snapshot observations share precisely the same beam models (these models can change over time and be different for each antenna, the observations must simply have the same change over time etc..), then the mapping function can be re-used between them. This allows for long integrations without an excessively large memory requirement, since the same mapping function can be used for each snapshot.

FHD is typically called by writing a script that is a wrapper for general_obs.pro. This script contains the full file paths to all of the uvfits files that will be used, the path to any ancillary catalog that will be used, and the desired paths where it will put the output. These scripts are contained in the pipeline_scripts repository under FHD_IDL_wrappers.

Useful Resources

• FHD is run by specifying keywords in a wrapper that is passed to the main code. These keywords change the functionality of FHD for fine-tuning and flexibility. Please visit the dictionary.md file for keywords and their definitions.
• Some mathematical assumptions are built into the framework of FHD. Please visit the assumptions.md file (forthcoming!) to explore the parameter space.
• FHD can process data from a multitude of instrumentation. Please visit the examples.md file (forthcoming!) for brief tutorials on specific cases.

Old Readme below

FHD notes – commonly used commands

Contents:

1 Overview

2 Installation instructions

3 Commonly used commands

3a Program control

3b Calibration

3c Gridding

3d Deconvolution

3e Output

4 Example FHD runs - coming soon!

1 Overview FHD is an imaging algorithm for radio interferometers. It uses the full complex gain for every baseline for gridding, and pre-computes the uv-plane covariance matrix (the Holographic Mapping function). In its current implementation, FHD follows a centroided matched pursuit algorithm for deconvolving point sources. While FHD does not refer back to the visibilities once they have been gridded to a dirty image, it is still properly classified as a visibility-based subtraction algorithm since sources are forward-modeled through the mapping function.

Aside from the basic use of snapshot imaging, there are a number of extensions that expand its capabilities. Most notably, FHD can perform a joint deconvolution of any set of visibilities by regridding each snapshot to a common HEALPix mesh. This allows for long integrations, and even for joint deconvolution of data from different instruments. Additionally, if snapshot observations share precisely the same beam models (these models can change over time and be different for each antenna, the observations must simply have the same change over time etc..), then the mapping function can be re-used between them. This allows for long integrations without an excessively large memory requirement, since the same mapping function can be used for each snapshot.

FHD is typically called by writing a script that is a wrapper for general_obs.pro. This script contains the full file paths to all of the uvfits files that will be used, the path to any ancillary catalog that will be used, and the desired paths where it will put the output.

A good example script for the 128T MWA is test_128t.pro

2 Installation instructions

Please see the separate INSTALL.txt file for the latest installation instructions, including all of the needed libraries. If you encounter difficulty getting FHD to run on your platform, please edit and add to these instructions!

3 Commonly used commands

3a Program control options:

beam_recalculate=recalculate_all ; recalculate the beam model

channel=121 ;Use only data with this center frequency channel. Used only in the top-level observation wrappers

cleanup=0 ; Set to delete most of the .sav files when finished

End_fi ; last file index of the supplied list to deconvolve (zero index)

export_images=1 ; If set, generate numerous output images and data files

Force_data; Force visibility data to be read from file, even if all the needed .sav files are already present

Force_no_data; Force visibility data to not be read from a file, even if needed.

grid_recalculate=recalculate_all ; recalculate (grid) the dirty images

mapfn_recalculate=recalculate_all ;recalculate the mapping function

recalculate_all=1 ;All other ‘recalculate’ options default to recalculate_all

Silent ; suppress most status messages

Skip_fi ; If supplied, skip the specified file index (may be a list)

Start_fi ; first file index of the supplied list to deconvolve (zero index)

version=0 ; Add a version number to the output data directory. Used only in the top-level observation wrappers

vis_file_list ; Typically supplied by the observation wrapper. Can be a string array containing the full file paths of the uvfits files to deconvolve OR can be the filepath to a text file (must end .txt), which contains the full filepaths of the visibilities.

3b Calibration options

calibrate_visibilities ; Set to calculate new calibration solutions

calibration_catalog_file_path ; catalog of point sources to use for calibration

max_calibration_sources ; Maximum number of calibration sources to use from catalog

calibration_flux_threshold=0 ; Minimum source flux to use for calibration.

calibration_spectral_index ; Set to a number to use as spectral index and override any catalog value(s).

min_cal_baseline ; Minimum baseline length to use for calculating calibration solution

max_cal_baseline ; Maximum baseline length to use for calculating calibration solution

calibration_model_subtract=0 ; subtract the calibration model from the data before proceeding to deconvolution. Set to fraction of model to subtract (e.g. 0.5 for 50%)

max_cal_iter=10 ; Maximum calibration iterations

Transfer_calibration ; File path of calibration solution to apply. Allowed formats are .sav files with an FHD cal structure, text files, and numpy arrays (not yet supported)

initial_calibration ; Set to string of obsid or filepath of cal.sav file to initiate the calibration to previous solution before calibration loop. In other words, point to an initial guess for the algorithm.

3c Gridding options

dimension=1024. ; Image size, in pixels, used during deconvolution

double_precison_beam=0 ;force the beam model and mapping function to be computed using double precision

flag=0 ; Flag data based on per-tile and per-frequency standard deviation of the visibilities

FoV=90. ; Field of view to be imaged, in degrees. No anti-aliasing filter is currently used, so FoV should be twice the size of the region of interest

Freq_end ; If set, flag all frequencies above freq_end (in MHz)

Freq_start ; If set, flag all frequencies below freq_start (in MHz)

max_baseline=0. ; Maximum baseline length to grid, in wavelengths

min_baseline=0. ; Minimum baseline length to grid, in wavelengths

n_pol=2 ; Grid and deconvolve only n_pol polarizations. Default is to grid all existing. Hard-coded to grid in the order xx, yy, xy, yx

precess=1 ; Set to apply precession corrections to the coordinates. J2000 are used throughout

Rephase_to_zenith=0 ;Set to correct a zenith observation that was re-phased to another location by another program

Tile_flag_list ; Numeric list of tiles to flag (Index starts at 1!)

transfer_mapfn=0 ; use a specific mapping function (specified as a string containing the full file path) for deconvolution, and do not calculate a new one. If set to 1 and a list of uvfits files are given, it will calculate the mapping function for the first file and transfer that mapping function for the rest of the files.

3d Deconvolution options

add_threshold=0.8 ; When fitting source components, fit all sources brighter than this fraction of the brightest pixel

Beam_threshold=0.05 ; Minimum primary beam threshold to use when constructing the source mask

gain_factor=0.15 ; “Clean” gain factor to use

Galaxy_model_fit=0 ; read in and forward-model the Haslam map of diffuse galactic emission. Fit an overall amplitude and subtract from the dirty images prior to deconvolution of point sources.

independent_fit=0 ;set to 1 to fit I, Q, (U, V) seperately. Otherwise, only I (and U) is fit

Local_max_radius=3 ; only fit a source component if it is at least Local_max_radius pixels away from a brighter source

Max_add_souces ; maximum number of sources to be fit in the same deconvolution iteration

max_sources=10000. ; Maximum number of source components to be fit

Reject_pol_sources=0 ; mask bright pixels with high polarization

Scale_gain=0 ; Set to a value 0<scale_gain<1 to use a larger “clean” gain factor for high signal to noise sources

Sigma_cut=2 ; only include source components detected with signal to noise greater than sigma_cut. Also used when condensing components to sources after deconvolution

Smooth_width=7; width of median filter to use to highlight point sources

generate_calibration_catalog.pro ; This is an independent program (not an option to FHD) which will take the output source list from an FHD run and produce a calibration catalog for future calibration.

Generating calibration catalog from FHD output: This is NOT an option to be passed into FHD, but rather an independent program to produce a calibration catalog from and FHD output source list. Simply run:

3e Output options

Combine_healpix; Set to regrid all supplied snapshots to HEALPix after deconvolution, and create combined maps. Also creates a source list structure

image_filter_fn='filter_uv_radial' ;name of a function to call to apply different weights to the uv plane when generating output images. NOT used during deconvolution. Current options are ‘’ (empty string) for natural weighting, ‘filter_uv_uniform’ for radially-averaged uniform weighting, ‘filter_uv_radial’ for radially-averaged weighting

N_avg ; number of fine frequency channels to average together when exporting data cubes for power spectrum estimation

no_ps=1 ;By default, .png, .ps, and .fits copies of the output images are saved. Set no_ps to save only .png and .fits

pad_uv_image=2. ; If set, pad the UV plane with zeroes by this scale factor, to increase the resolution of the output images. Ignored during deconvolution

ps_export=0 ;Set to generate output healpix cubes compatible with Bryna’s Power Spectrum code

Split_ps_export; If set, splits the visibilities into separate cubes for even and odd time samples