The bilinear mixing model is a more realistic, generalized model that can represent a wide range of real-world hyperspectral images with tolerable accuracy. The use of a spectral library makes the problem more tractable. However, the high mutual coherence of the spectral library creates computational as well as performance issues in library-aided bilinear unmixing. Besides, the high mutual coherence of the spectral library reduces the accuracy of these unmixing methods, and the high cardinality of the spectral library increases the computational complexity. We propose a computationally efficient, two-phase library pruning approach for unmixing hyperspectral image, which also withstands a highly coherent spectral library. In this work, we first segregate the data into pixels generated due to linear and bilinear interaction using the subspace clustering method and subsequent rank estimation strategy. We subsequently reduce the mutual coherence of the spectral library and prune the linear interactions. In the next stage, we create a library corresponding to the bilinear components assuming that only the secondary reflections of the pruned library elements may be prevalent in these pixels. We perform pruning using a novel, low-rank based, sequential approach. Finally, we compute the abundance of the matrix by exploiting sparseness of the abundance matrix and include its low-rankness, and spatial structural similarity as regularization. We validate the overall advantages of our proposed framework on several real and synthetic data experiments.