A Review on Decomposition/Reconstruction methods for Fusion of Medical Images

An image contains variety of information (in terms of frequency i.e. low and high) and to split that information into sub-parts, Decomposition of image is done so as to extract the desired features present in the image. On decomposing the image, it gets divided into sub-images. Each sub image will contain different information. The coarsest information is fetched in the decomposition phase. The aim of Fusion process is to bring out the best information from the base images. The Fusion technique is applied on the decomposed sub-images obtained from each source image. After successfully fusing the sub-images, single fused image is obtained after applying the reconstruction method on the fused sub-images


Introduction
Fusion of images combines related information from multiple images and produces an image. Combining the images should reduce the redundancy, without distortion. The output image is useful for visual insight and interpretation. According to J. Du (Du, Li, Lu, et al. 2016), Multimodal medical image fusion is the combination of multiple images from single/multiple imaging modalities. Medical image fusion leads to improved image quality while preserving the important features. Fused output images are used in clinical applications and gives better outcome for diagnosis, assessment of disease, etc. The medical images varies highly in their modality like CT ( Computed Tomography) gives anatomical information, MRI ( Magnetic Resonance Imaging) gives soft tissue contrast information and PET( Positron Emission Tomography) is an radionuclide technique which provides functional information.
Fusing the modalities will result in a merged image containing the best information from both the modalities. In this paper, the state of the art decomposition/reconstruction methods are studied. This is followed by summarization of these methods. At last, this review paper concludes that while many decomposition/reconstruction methods have been anticipated, there still exist several future directions to select a method. Rest of the paper is organized as follows, Section 1 contains the introduction of Fusion and the various modalities available, Section 2 contain the work flow of Fusion method including various phases, Section 3 contain Decomposition/Reconstruction methods categorized as Pyramid, wavelet and color based Decomposition/Reconstruction methods., Section 4 contain the summary of the methods available for Decomposition/Reconstruction and Section 5 concludes research work with future directions.

Multi-Modality Medical Image Fusion
Images (Multi-Modality) are acquired and are pre-processed (if needed) to make it suitable for fusion. The acquired images are proceeded to decomposition. The process of decomposition divides the acquired image into sub-images. Various decomposition and reconstruction methods are used in the literature and are also discussed in this paper. The resultant sub-images are then fused using suitable fusion rules. Last step is the reconstruction of a single image from the sub-parts. Reconstruction is done by taking the inverse of the Decomposition method. The purpose of this review paper is to discuss the Decomposition/ Reconstruction methods available in the literature. The following diagram gives a picture of the Fusion process:

Decomposition/Reconstruction Methods
The first step involves the decomposition of multimodality images. Inverse of the first step is the last step i.e. reconstruction of the fused subimages into a final resultant image. Analysis Phase is the other name for Decomposition and Synthesis Phase for the Reconstruction. The Decomposition/Reconstruction methods available in the literature which are mostly used for medical images are as follows:

Multi-resolution Analysis
Medical images contain bone and tissue structures present in human body at many different resolutions and Multi-Resolution approach can assist in the Decomposition/Reconstruction process. Hence, the Multi-Resolution techniques have evolved as a preferable method for Image Fusion.

Pyramid Transform (PT)
The PT fusion involves of three phases; the first step is image decomposition into levels followed by image fusion and the last step being image reconstruction (Fig. 2). PT when applied on input image gives us the multiscale representation subimages. Then, using efficient rules for fusion is applied on these sub-images. The inverse of Pyramid Transform(PT), thus used to obtain the final fused image.

Laplacian Pyramid (LAP)
LAP decomposes images based on the values obtained from the Gaussian Filtered images and smooth Gaussian kernels. The decomposition is applied on the difference value obtained. The simple image resizing routines are used for the successive Gaussian filtered images. J. Du et al. (Du, Li, Xiao, et al. 2016) decomposed the input images using LAP transform into multi-scale image representation. The output has shown better performance for most of the metrics values. After applying LAP, an input image is sub-imaged into one estimated image and a series of residual images. The significance of estimated image is that it provides the approximation of the input image.
The residual images obtained contain high-pass frequency saliency feature information i.e. edge information, texture information which signifies the fused image quality. Medical Image Fusion with Laplacian Pyramids is demonstrated in (Sahu 2014)in which Laplacian Pyramid transform is first applied to the images followed by the Discrete Cosine Transform (DCT). By using Laplacian pyramid, the input image is split into low sub-band images at each pyramidal level. After deleting the maximum of redundant details, the point discontinuities Image acquisition Decomposition Fusion Reconstruction of the images are captured. Use of DCT is preferred instead of DWT as it overcomes the reduced contrast and redundancy problem introduced by latter. Also, DCT works by compressing the source image to half the size by which the spatial density and resolutions are reduced.
Hence making the computation comparatively faster and is also called a Fast Transform. The outcome obtained in the paper is compared with Daubechies Complex wavelet Transform (DCxWT). The proposed method gives the image with high contrast and high information content.

Gradient Pyramid (GRP)
Each level of Gaussian pyramid is treated with gradient operators. Through this the gradient pyramids are generated, which yields diagonal, horizontal, vertical pyramid sets for each source in the Gaussian pyramid. On application of GRP, gradient is used to represent the high-pass frequency part at various levels.

Multi Scale Geometric Analysis/Wavelet Transform
Unlike Sine waves, a waveform of limited duration is called a Wavelet. Local features are well described by Wavelets. Wavelets can be applied to 1 dimension, 2 dimensions or successfully to higher dimensions also. The Wavelet Decomposition is an iterative process and is performed in the form of levels until we reach at the coarsest detail in the image. The Reconstructed image is obtained by inversing Decomposition method.

DWT
Decomposition of the image in different coefficients while preserving the image information is obtained using the Discrete Wavelet Transform (DWT). New coefficients are obtained by appropriately combining the coefficients coming from different images ensuring that the information is collected properly from the source images. The final fused output is the Inverse of DWT.  (Ts and Rangarajan 2017) where the PET image is first decomposed into sub-bands, the low subband is then subtracted from the original PET image followed by application of Lagrange's interpolation to get the enhanced version of the PET image. When fusion is done, inverse DWT is put on to get the vital output. The proposed technique gives high average gradient and very low discrepancy. Therefore we get the output which has high spectral and spatial resolution. The worth of reconstructed output image has improved as the value of PSNR is high and MSE is low.

RDWT (Redundant Discrete Wavelet Transform)
Also called as SWT (Stationery Wavelet Transform), is translation invariant, possesses redundancy. It works by dividing the image into two frequency components i.e. low and high. The low frequency component is the approximation and the high frequency provides detail V.
Bhateja et.al. has used SWT with NSCT in (Bhateja et al. 2015) where the images are first decomposed using SWT. Then fusion rule is applied, images are reconstructed by SWT-1. Then again the images are decomposed using NSCT and again fusion rule is applied, followed by NSCT-1. K. Indira et.al. has compared DWT and SWT on 8 set of images in (Indira, Hemamalini, and Indhumathi 2015) and has results showing better performance of SWT when compared to DWT in terms of RMSE, Entropy, CC, PSNR.

DTCWT (Dual Tree Complex Wavelet Transform)
Dual Tree CWT exhibits the property of Perfect Reconstruction, Shift Invariance. DTCWT also provides Directional selectivity and redundancy is reduced making it preferable over DWT. On applying DTCWT, two fully decimated trees are created. At the first level, one decimated tree is for the representation of odd samples and the second one for the even samples. DTCWT has directional sensitivity by which it means that DT-CWT has the power to give six distinct sub-bands at each decomposition level. This feature allows distinguishing between positive and negative orientations.

DCxWT (Daubechies Complex Wavelet Transform)
The shift invariant property, perfect reconstruction capability and no redundancy of DCxWT makes it a good choice for decomposition and reconstruction. The imaginary coefficients provide the phase information and are symmetric in nature. A new multilevel (DCxWT) based image fusion technique that follows multiresolution principle is proposed by R. Singh et al (Singh and Khare 2014). The decomposition is done from level 2 to level 8. On comparison with state of the art methods DCxWT proved to perform well.

CT (Curvelet Transform)
The curvelet transform is well-suited for multiscale edge enhancement. It is a multi-scale directional transform that gives optimal nonadaptive sparse representation of objects with edges. It is a well-known fact that wavelet has limited directional information whereas Curvelet has the opposite feature. CT being localized in scale, position and orientation can handle discontinuities more precisely. To assist wavelets to perform well at discontinuities, A. Pure et al. (Pure, Gupta, and Shrivastava 2013) has proposed the image to be decomposed by DWT and then by fast discrete curvelet transform. On Simulation, results indicate that higher values of Entropy, PSNR, Standard deviation, Mean along with lower value of RMSE is obtained by the proposed image fusion technique. The fused image has more information content, better PSNR and low RMSE value. Visual inspection also confirmed that the proposed technique preserve both, high spatial resolution and high spectral quality contents.\

ST (Shearlet Transform)
Shearlets work by calculating the shear matrix. Shearlets form a fitted frame at various directions and scales. Theyare optimally sparse in representing images withboundaries. Decomposition of shearlets is similar to Contourlet except that Shearlets work with shear matrix (Miao et al. 2011). Shear matrix works by applying the directional filtering. Shearlet Transform decomposes the image into two parts, decomposition into multi-direction and decomposition into multi-scale. Shear matrix is used for decomposition in multi-direction. And for decomposition in multi-scale, wavelet packets are used.
As there are no limits on the number of directions for the shearing, Use of shearlet transform is advantageous than Contourlet Transform. Q. Miao et al. have shown the fusion results on medical images. From the wavelet family, in particular Haar, Daubechies, on decomposition, reduces the features present in either of the source images. Contourlet offers more sharpness than shearlets. But while analyzing the edges present in the bones and tissues it came out to be distorted greatly. Feature as well as detail information are much richer. The tissues and bone information is also enhanced using Shearlets.   (Bhatnagar, Wu, and Liu 2015) have proposed a contrast based decomposition using NSCT where contrast will measure the split-up between intensity value of a pixel and its neighboring pixel. The authors thus concluded that their algorithm conserves the image details in a better way. Also, the image visual effects are significantly improved.

Color based Method 3.3.1. IHS (Intensity Hue Saturation)
It is a color based decomposition method. The HIS transform is used to convert a multispectral image into three independent components. The multi-spectral input consists of Red, Green and Blue channels (RGB), is converted into image with components as intensity, hue and saturation. Intensity component gives the information about the brightness in a spectrum. The hue represents the spectral wavelength, and saturation is spectrum's purity. For the fusion of multi-sensor images, IHS is used. In particular for medical images, we need to use IHS when we are using colored images like PET, SPECT etc. 2) Decomposing the image into Intensity, Hue and Saturation components; 3) The intensity component of the image is replaced with high resolution image.
4) The component obtained from the previous step is used to reconstruct the image to RGB color space.
There is a considerable increase in the spatial information content after using the proposed method. The color distortion is also reduced as proved by the statistical analysis tools and visual analysis.

Summarization
Decomposition/ Reconstruction methods can be Summarized as follows:

Conclusion
The above review shows that many image Decomposition/Reconstruction techniques/ methods are used in the literature. The selection of Decomposition/Reconstruction method depends on the modality of images and the level up to which information is to be extracted. The study so far has used the same Reconstruction method (i.e. Inverse of Decomposition method) and a very few authors have experimented on different Reconstruction method. Hence the research on Medical Image Fusion can be directed towards this direction also.