A background initialization algorithm based on the spatiotemporal similarity measure in a motion tracking system is proposed. From the accumulated difference between the base frame and the other frames in a subinterval, the regions affected by moving objects are located. The median is applied over the subsequence in the subinterval in which co-located regions share the similarity. The outputs from each subinterval are filtered by second stage median filter. The proposed method showed good results even in the busy and crowded sequences where the real background does not exit.