It is generally assumed that using a retro-reflective surface as the target of an interferometric measurement, an accurate alignment to the illumination beam is not really necessary. On the contrary, we show that even a few degrees of angular misalignment can corrupt the interferometric signal with the onset of a large artifact, which is also misleading because it is similar to the waveform of the high-level feedback regime. We first provide the experimental evidence of the artifact in a self-mixing interferometer, showing that the artifact is canceled after an alignment to <0.2 deg error, and then provide a theoretical evaluation, in good agreement with the observations.