Oil-based nanofluids were prepared by dispersing spherical and fiber shaped $Al_2O_3$ and AlN nanoparticles in transformer oil. Two hydrophobic surface modification processes using oleic acid (OA) and polyoxyethylene alkyl acid ester (PAAE) were compared in this study. The dispersion stability, viscosity and breakdown voltage of the nanofluids were also characterized. $(Al_2O_3+AlN)$ mixed nanofluid was prepared to take an advantage of the excellent thermal conductivity of AlN and a good convective heat transfer property of fiber shaped $Al_2O_3$. For $(Al_2O_3+AlN)$ particles with 1 % volume fraction in oil, the enhancement of thermal conductivity and convective heat transfer coefficient was nearly 11 % and 30 %, respectively, compared to pure transformer oil. The nanofluid, containing $Al_2O_3+AlN$, successfully lowered the temperature of the heating element and oil itself during a natural convection test using a prototype transformer.