Colloidal silver nanoparticles (AgNPs) have been commercialized as the typically stabilized form via the addition of a variety of surfactants or polymers. Herein, to examine the effects of stabilizing AgNPs in suspension, we modified the surface of bare AgNPs with four type of surfactants (NaDDBS, SDS, TW80, CTAB) and polymers (PVP, PAA, PAH, CMC). The modified AgNPs was applied to compare suspension stability and nanotoxicity test using Escherichia coli (E. coli) as a model organism. Modification of AgNPs surface using chemical stabilizer may be not related with molecular weight, but chemical structure such as ionic state and functional group of stabilizer. In this study, it is noteworthy that AgNPs modified with a cationic stabilizer (CTAB, PAH) were importantly toxic to E. coli, rather than anionic stabilizers (NaDDBS, SDS). Comparing similar anionic stabilizer, i.e., NaDDBS and SDS, the result showed that lipophilicity of chemical structure can affect on E. coli, because NaDDBS, which contains a lipophilic benzene ring, accelerated the cytotoxicity of AgNPs. Interestingly, none of the stabilizers tested, including biocompatible nonionic stabilizers (i.e., TW80 and cellulose) caused a reduction in AgNP toxicity. This showed that toxicity of AgNPs cannot be reduced using stabilizers.