Markov random field (MRF) provides a useful model for integrating contextual information into remote sensing image classification. However, there are two limitations when using the conventional MRF model in hyperspectral image classification. First, the maximum likelihood classifier used in MRF to estimate the spectral-based probability needs accurate estimation of covariance matrix for each class, which is often hard to obtain with a small number of training samples for hyperspectral imagery. Second, a fixed spatial neighboring impact parameter for all pixels causes overcorrection of spatially high variation areas and makes class boundaries blurred. This paper presents an improved method for integrating a support vector machine (SVM) and Markov random field to classify the hyperspectral imagery. An adaptive spatial neighboring impact parameter is assigned to each pixel according to its spatial contextual correlation. Experimental results of a hyperspectral image show that the classification accuracy from the proposed method has been improved compared to those from the conventional MRF model and pixel-wise classifiers including the maximum likelihood classifier and SVM classifier.