The loss due to caching replacement in the wireless mobile proxy caching structure has a significant effect on streaming QoS. This paper proposes a similarity-based caching loss minimization (SCLM) for minimizing the loss caused by the caching replacement. The proposed scheme divides object segments, and then it performs the similarity relation about them. Segments that perform the similarity relation generates similarity relation tree (SRT). The similarity is an important metric for deciding a relevance feedback, and segments that satisfy these requirements in the cache block for caching replacement. Simulation results show that the proposed scheme has better performance than the existing prefix caching scheme, segment-based caching scheme, and bi-directional proxy scheme in terms of QoS, average delayed startup ratio, cache throughput, and cache response ratio.