In this paper, the energy-based plastic-damage model previously proposed by the authors [International Journal of Solids and Structures, 43(3-4): 583-612] is first simplified with an empirically defined evolution law for the irreversible strains, and then it is extended to its rate-dependent version to account for the strain rate effect. Regarding the energy dissipation by the motion of the structure under dynamic loadings, within the framework of continuum damage mechanics a new damping model is proposed and incorporated into the developed rate-dependent plastic-damage mode, leading to a unified constitutive model which is capable of directly considering the damping on the material scale. Pertinent computational aspects concerning the numerical implementation and the algorithmic consistent modulus for the unified model are also discussed in details, through which the dynamic nonlinear analysis of damping structures can be coped with by the same procedures as those without damping. The proposed unified plastic-damage model is verfied by the simulations of concrete specimens under different quasistatic and high rate straining loading conditions, and is then applied to the Koyna dam under earthquake motions. The numerical predictions agree fairly well with the results obtained from experimental tests and/or reported by other investigators, demonstrating its capability for reproducing most of the typical nonlinear performances of concrete under quasi-static and dynamic loading conditions.