We investigate optimal waveform design using fractional Fourier transform in signal-dependent interference, as well as additive channel noise for stochastic extended target. Within constraints on waveform energy and bandwidth, optimal waveform design in fractional Fourier domain based on the signal-to-interference-plus-noise ratio criterion, probability of detection criterion, and mutual information criterion are modeled, respectively. In addition, the relationship between the optimal waveforms that are designed based on the three criteria is discussed. Simulations are conducted to illustrate that for all of the three criteria, the energy of optimal waveform can be distributed in some narrow bands where the target power is large and the interference power is small in fractional Fourier domain. Finally, the fractional Fourier domain waveform design method is proved more flexible and effective than traditional Fourier domain waveform design method, especially when the spectral density of target response and interference are relatively dispersed and flat.