It is very difficult that a previous surveillance system and vehicle detection system find objects on a limited and unstable illumination condition. This paper proposes a robust method of adaptively updating a reference background image for solving problems that are generated by the unstable illumination. The first input image is set up as the reference background image, and is divided into three block categories according to an edge component. Then a block state analysis, which uses a rate of change of the brightness, a stability, a color information, and an edge component on each block, is applied to the input image. On the reference background image, neighbourhood blocks having the same state of a updated block are merged as a block. The proposed method can generate a robust reference background image because it distinguishes a moving object area from an unstable illumination. The proposed method very efficiently updates the reference background image from the point of view of the management and the processing time. In order to demonstrate the superiority of the proposed stable manner in situation that an illumination quickly changes.