The stability of a class of switched stochastic nonlinear retarded systems with asynchronous switching controller is investigated. By constructing a virtual switching signal and using the average dwell time approach incorporated with Razumikhin-type theorem, the sufficient criteria for pth moment exponential stability and global asymptotic stability in probability are given. It is shown that the stability of the asynchronous stochastic systems can be guaranteed provided that the average dwell time is sufficiently large and the mismatched time between the controller and the systems is sufficiently small. This result is then applied to a class of switched stochastic nonlinear delay systems where the controller is designed with both state and switching delays. A numerical example illustrates the effectiveness of the obtained results.