The positions of vessels in JTIDS where each vessel broadcasts its position, can be found using the relative navigation method. Besides positioning, the relative navigation could be adopted for identification friend or foe, tracking targets, monitoring battle field and etc. In this paper, we have explained the fundamental operation and technical structure for the relative navigation and implemented the simulation platform to evaluate the basic function and performance of the system in arbitrary environment. Using platform, the availability of relative navigation within the group network and the characteristic of the algorithm for position prediction was verified. Based on the simulation result, it was verified that EKF based navigation algorithm could produce great initial error and need quite convergence time. To improve the performance, we proposed a new navigation algorithm which uses the minimum norm estimation algorithm until the EKF converges. The simulation results reveal the relative navigation can be effectively used in the formation flight and collision avoidance system.