This paper describes attitude determination algorithm for the low earth orbit(LEO) spacecraft using stellar inertial sensors. The cascaded gyro/star tracker extended Kalman filter is constructed to fuse two sensor data. And then the smoothing of the measurement are proposed for an unreasonable jump of star tracker. The smoothing algorithm for the rejection of star tracker error jumps is designed by scalar adaptive filter. The proposed algorithms operate to process the measurement of gyro/star tracker Kalman filter, therefore, it is comparatively simple to apply these methods to other integration systems. Simulations to gyro/star tracker integrated system show that the proposed method is effective.