Recently, spectroscopy has emerged as a potential tool for quality evaluation of numerous food and agricultural products because it provides information regarding both spectral distribution and image features of the sample (i.e., hyperspectral imaging). Spectroscopic techniques reveal hidden information regarding the sample and do so in a non-destructive manner. This review describes the various approaches of spectroscopic modalities, especially hyperspectroscopy and vibrational spectroscopies (i.e., Raman spectroscopy and Fourier transform near infrared spectroscopy) combined with chemometrics for the non-destructive assessment of contaminations and defects in agro-food products.