Improving the training and application of Chinese language models for low-order grammar

Lingjun Kong

doi:10.1117/12.3033811

13 June 2024 Improving the training and application of Chinese language models for low-order grammar

Lingjun Kong

Proceedings Volume 13180, International Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2024); 131802D (2024) https://doi.org/10.1117/12.3033811
Event: International Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2024), 2024, Guangzhou, China

Abstract

This paper presents a method to enhance the training of low-order grammar language models in response to the issue of out-of-vocabulary (OOV) words and incorrect words in acoustic model and low-order grammar language model decoding, which results in a decrease in Chinese speech recognition accuracy. To address the problem of incorrect words caused by polyphonic Chinese character, multiple phonemes are added to the Chinese dictionary to improve the phoneme model training. Prior to training the low-order grammar language model, the one-dimensional word segmentation is converted into a vectorized segmentation structure for processing. This enables the training of language models to obtain different probability distributions for the same word, thereby reducing the occurrence of OOV words. Finally, this method is compared with traditional language model training methods and evaluated with the same acoustic parameters. The experiment demonstrates an 8.5% reduction in word error rate for speech recognition when using low-order grammar language models.

(2024) Published by SPIE. Downloading of the abstract is permitted for personal use only.

Citation Download Citation

Lingjun Kong "Improving the training and application of Chinese language models for low-order grammar", Proc. SPIE 13180, International Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2024), 131802D (13 June 2024); https://doi.org/10.1117/12.3033811

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available