This paper proposes a robust damper design technique for adjacent structures against model uncertainty. This approach introduces multi-objective optimization based system identification using measurement information which enables reasonable selection of the perturbation range in the robust design. Moreover, in order to improve the numerical efficiency in sampling the structural models required for the robust design of large structures, we define new objective functions which enable us to minimize the number of candidate models suitable to the purpose of the robust design. In addition, the performance index is newly employed to evaluate the robust performance of the sampled structural models, and the robust design has been performed according to the performance index. As a numerical example to demonstrate the efficiency of the proposed method, 5-story and 10-story two adjacent buildings are taken into account, and the existing and newly proposed robust design approaches are compared with each other. The results demonstrate that the proposed approach can guarantee more robust damper system only using small number of samples of the structural models because of using the measurement information which leads to improvement in the numerical efficiency, compared with the existing robust design methods.