The fluorine doping along with heavy metal oxides remarkably raised the $^3$H$_4$ lifetime and the quantum efficiency in Tm$^{3+}$-doped silicate glasses. 29 mol% of fluorine substitution for oxygen in 70SiO$_2$-15Pbo-12ZnO-3KO$_{1}$2/ glass raised $^3$H$_4$ lifetime to 193 $mutextrm{s}$. Refractive indices were raised by heavy metal oxide substitution, but hardly changed by fluorine substitution. The fluorine doping changed the local structure around Tm$^{3+}$ions, then low energy vibrations related to fluorine are considered to largely reduce the multi-phonon relaxation rates in the oxyfluoride silicate glasses. The $^3$H$_4$ lifetimes and absorption and emission spectra of Tm$^{3+}$doped silicate and oxyfluoride silicate glasses are reported, and Judd-Ofelt calculation results are discussed in this paper.paper.