1.

INTRODUCTION

Speech enhancement is currently widely used in mobile communications, speech recognition front-end modules and hearing aids, etc¹. When constructing a deep learning model for speech reconstruction, the short-term Fourier transformation (STFT) is used to convert the waveforms into spectrum as the input features. The spectrum is generally represented as a complex form which usually needs to be decomposed into the amplitude and phase component in the real-value network. Therefore, the amplitude spectrum and phase spectrum of clean speech needs to be estimated in speech reconstruction. However, because the imaginary structure is not obvious and the neural network learning is difficult, the phase estimation is often neglected. This leads to most methods only pay attention to the estimation of the amplitude spectrum, and the phase information is reused with the noisy speech when reconstructing the clean speech². The studies have shown that the clean speech reconstructed with the noise signal phase has an error compared to reconstruction with the actual clean speech phase. The lower the signal-to-noise ratio (SNR), the greater the error. The application of the speech reconstruction is often in a noisy environment, and the SNR is relatively low. Therefore, it is urgent to have an effective way to estimate the phase information of the clean speech and improve the reconstruction accuracy and sound quality of the speech reconstruction. This paper improved the speech reconstruction with the complex-valued full convolution neural network (CFCCN) which can simultaneously estimate the amplitude and phase components of the clean signal in the noisy speech.

2.

RELATED WORK

There have been a large number of research results on speech enhancement based on deep learning in recent years. Pascual published the SEGAN speech enhancement model, and it has been cited more than 80 times³. Rethage proposed end-to-end deep speech enhancement models wavenet in 2019, the noisy speech is employed as the original waveform input without the need to calculate any explicit time-frequency representation, and the performance has made great progress⁴. In 2018, Yoshiki et al. used the ideal time-frequency (T-F) masking and the amplitude spectrum of the target speech as a supervised speech enhancement target, and estimated and synthesized the target speech waveform through a deep neural network. The experiment proved that the separated speech can significantly suppress the noise⁵. Erdogan et al.⁶ further introduced phase information into time-frequency masking in 2019, and proposed the phase sensitive mask (PSM) to reconstruct speech signals. Subsequently, Williamson proposed the complex ratio masking feature, and experimentally verified that complex ratio masking can remove noise more effectively compared to IBM, IRM and PSM. In 2019, an online data-driven noise adaptive updates method for speech enhancement based on variational Bayesian non-negative matrix factorization, which improved the model’s robustness to actual unknown noise. On the basis of this research, the speech signal separation with the combination of variational Bayesian non-negative matrix factorization and deep neural network was proposed in 2020, which can better solve the problem of source confusion⁷. With the continuous improvement of the GAN model, GAN-based speech enhancement methods have also emerged recently and achieved good results^8-9.

3.

PROPOSED METHOD

3.1

Overview of CFCCN

As shown in Figure 1, the architecture of the CFCCN is divided into the following stages: 1) Design phase: The building blocks include the complex convolution filter, complex activation functions, complex batch-norm, complex pooling and the weight initialization strategies; 2) training phase: we use international standard database to build cRM calibration data sets, network training sets and network training sets and testing set. We train the CFCCN model based on cRM training data; 3) Testing stage: The cRM of the input noisy speech spectrum is estimated using the trained CFCCN model. Then we apply the cRM mask matrix to the noisy sample to reconstruct the spectrogram of the speech component..

Figure 1.

The architecture of the CFCCN.

4.

EXPERIMENT

4.3

Result and discussion

Figure 2 shows the subjective comparison of different algorithms. Figure 2a is the clean speech Figure 2b is the clean speech corrupted by white noise at 5dB. Figures 2c-2f illustrates the quasi-clean speech reconstructed of SEGAN, NAAGN, DARGAN^{3, 8-9} and CFCCN respectively.

Figure 2.

Subjective comparisons of different algorithms.

According to waveforms, Figure 2f obtains a cleaner signal than the other comparison, and is closer to the Figure 2a. In terms of the spectrograms, Figures 2c-2e are still noisy compared with Figure 2a. On the contrary, the voice parts of Figure 2f are of less artifacts and distortions, and the noises appear to be suppressed. The results show that Figure 2f obtains a cleaner signal than the other comparison, and is closer to Figure 2a. Furthermore, Tables 1-2 demonstrate the comparison of gSIR, gSDR. It shows that the CFCCN obtains the best performance on NOISEX-92 and DEMAND dataset due to its effectiveness of complex-valued networks for speech reconstruction and phase correction. SEGAN³ and NAAGN⁸ outperform DARGAN⁹ on average, mainly because the deployed dynamic attention mechanism has more speech coherence and fidelity.

Table 1.

gSIR comparison: CFCCN vs. baselines.

Datasets/Model	NOISEX-92	DEMAND
SEGAN3	7.2	7.8
NAAGN8	7.3	7.5
DARGAN9	7.9	8.3
CFCCN	8.4	9.2

Table 2.

gSDR comparison: CFCCN vs. baselines.

Datasets/Model	NOISEX-92	DEMAND
SEGAN3	6.2	6.0
NAAGN8	6.5	5.8
DARGAN9	6.9	6.5
CFCCN	7.4	7.3

5.

CONCLUSION

This work improved the speech reconstruction with CFCCN. Based on the theoretical of complex-value neural network, the complex-valued building blocks are designed for CFCCN to handle complex domain operations and estimate cRM. The experiment results show that in terms of subjective and objective measurements, this work achieves at least improvement of 1.3%-12.5% in contrast to the state-of-the-arts DNN based speech reconstruction methods.