Relative navigation system is presented using GPS measurements from a single-channel global positioning system (GPS) simulator. The objective of this study is to provide the real-time inter-satellite relative positions as well as absolute positions for two formation flying satellites in low earth orbit. To improve the navigation performance, the absolute states are estimated using ion-free GRAPHIC (group and phase ionospheric correction) pseudo-ranges and the relative states are determined using double differential carrier-phase data and singled-differential C/A code data based on the extended Kalman filter and the unscented Kalman filter. Furthermore, pseudo-relative dynamic model and modified relative measurement model are developed. This modified EKF method prevents non-linearity of the measurement model from degrading precision by applying linearization about absolute navigation solutions not about the priori estimates. The LAMBDA method also has been used to improve the relative navigation performance by fixing ambiguities to integers for precise relative navigation. The software-based simulation has been performed and the steady state accuracies of 1 m and 6 mm ($1{sigma}$ of 3-dimensional difference errors) are achieved for the absolute and relative navigation using EKF for a short baseline leader/follower formation. In addition, the navigation performances are compared for the EKF and the UKF for 10 hours simulation, and relative position errors are mm-level for the two filters showing the similar trends.