The strain, surface and interfacial energies of III-V ternary systems were calculated for three kinds of structure modes: the Frank-van der Merwe(FM) mode, the Stanski-Krastanov(SK) mode and the Volmer-Weber(VW) mode. The free energy for each mode was estimated as functions of thickness and composition or lattice misfit. Through comparison of the free energy of each mode, it was found that the thickness-composition phase diagrams of III-V ternary systems can be determined only by considering the balance of the free energy and three kinds of structure modes appear in the phase diagrams. The SK mode appears only when the lattice misfit is large and/or the lattice layer is thick. The most stable structure of the SK mode is a cluster with four lattice layers or minimum thickness on a wetting layer of increasing lattice layers. The VW mode appears when the lattice misfit is large and the lattice layer is thin and only in the INPSb/InP and GaPSb/GaP system which have the largest lattice misfit of III-V ternary systems. The stable region of the SK mode in the GaPSb/GaP and InPSb/InP phase diagrams is largest of all because the composition dependence of the strain energy of these systems is stronger than that of the other systems. The critical number of lattice layers below which two-dimensional(2D) layers precede the three-dimensional(3D) nucleation in the SK mode at x=1.0 depends on the lattice misfit.