Object tracking is a technique used in computer vision and image processing applications. This technique fails to track objects when using only visible sensors in situations with illumination variations, occlusion, camouflage, and adverse climatic circumstances. Thermal sensors are fairly illumination invariant. However, in situations where the temperature is not changing much between the foreground and the background, thermal sensors might not provide the best results, as everything might be single color coded. Visible sensors are known to perform better in these conditions. Thus, the joint use of the visible and thermal sensors would be most beneficial in making a robust object tracking system under such challenging conditions. The proposed method performs object tracking by fusing bimodal information using particle filter with structural similarity index metric (SSIM) as a cue as well as a correlation metric for modality selection. The key idea of this method is to adaptively choose the most discriminating modality with respect to the object being tracked. This is performed by calculation of SSIM between reference and successive frames of both the modalities. The method is evaluated by testing on a variety of extremely challenging video sequences in both imaging modalities and has proven to perform better compared to single modality.