High Pressure High Temperature (HPHT) treatment can significantly change the color of diamonds. We studied the variation of the optical properties according to the nitrogen arrangement in natural brown diamonds of various types (type IaAB, type IaB, type IaA > B, type IaA < B, IaA = B) after HPHT treatment. The diamonds with different arrangements of nitrogen were annealed at temperatures in the range $1700-1800^{circ}C$ under a stabilizing pressure of 5 GPa. HPHT treated samples were analyzed using UV-Vis-NIR, FT-IR, and PL spectroscopy. The absorption and luminescence spectra were measured to compare the variations of nitrogen arrangement in the natural brown diamonds before and after HPHT treatment. After HPHT treatment, the brown coloration in all types of diamonds was reduced and a decrease in the peaks related to the A-aggregate of nitrogen was more predominant than the B-aggregate. Furthermore, the peaks related to N3 (415.4 nm), H4 (496.4 nm), and platelet decreased and the peaks related to H3 (503.2 nm) and G-band increased after HPHT treatment. In conclusion, spectroscopic analysis of natural brown diamonds after HPHT treatment showed that a yellow color was produced by absorption in the H3 centers and a green color was generated by interaction between absorptions of the H3 and H2 centers.