Magnetic field induced structural transition has been systematically investigated for $La_{1-x}Ba_xMnO_3$ with the fine control of carrier doping $(0.15{leq}x{leq}0.20)$. Application of a magnetic field results in the suppression of the rhombohedral-orthorhombic transition temperature $(T_s)$ and the increase of insulator-metal transition temperature $(T_{MI})$. Near x = 0.17, where $T_S$ is similar to $T_{MI}$ at zero magnetic field, we found that the $T_S$ smoothly decreased with magnetic field even though it intersected the $T_{MI}$ near 3 T. Also, the magnetostructural phase diagram obtained from the temperature sweep and from the magnetic field sweep is not significantly modified. By comparing the magnetostructural transition in $La_{1-x}Sr_xMnO_3$, we have suggested that the large disorder originated from ionic size differences between La and Ba may weaken the sensitivity of the kinetic energy of $e_g$ electrons on the degree of lattice distortion in $La_{1-x}Ba_xMnO_3$.