We propose an action recognition method based on Riemannian manifold and adaptive weighted feature fusion. First, we divide the video into several sequence frames, and we propose to use the frame-action relevance to eliminate the slight disturbance of the background, so that the subsequent calculation can focus on the moving region of the target better. On this basis, our proposed framework extracts histogram of oriented gradients (HOG) and histogram of optical flow (HOF) features as original appearance and motion features. Aiming at the problem that the existing action recognition methods do not pay enough attention to the change rate of appearance and motion, an innovative feature names as Riemannian manifold distance feature (RMDF) is proposed, which can represent the topological relationship between features at different times in the same position and capture the change rate of current features. In the feature fusion stage, we use an unsupervised method of solving optimization problems to obtain the weights between different features. The strategy of adaptive weight allocation is adopted to fuse HOG, HOF, and their corresponding RMDFs to obtain the final representation of the video. Finally, it is sent to a back propagation neural network for classification and recognition. Experiments on three benchmark datasets show that our method still has great advantages compared with the most advanced methods.