This paper presents a framework for the self-organization of swarm systems based on coupled nonlinear oscillators (CNOs). In this scheme, multiple agents in a swarm self-organize to flock and arrange themselves as a group using CNOs, which are able to keep a certain distance by the attractive and repulsive forces among different agents. A theoretical approach of flocking behavior by CNOs and a design guideline of CNO parameters are proposed. Finally, the formation scenario for cooperative multi-agent groups is investigated to demonstrate group behaviors such as aggregation, migration, homing and so on. The task for each group in this scenario is to perform a series of processes such as gathering into a whole group or splitting into two groups, and then to return to the base while avoiding collision with agents in different groups and maintaining the formation of each group.