This paper provides a comprehensive study of optimum design of a helicopter tailrotor driveshaft made of the flexible matrix composites (FMCs). Since the driveshaft transmits power while subjected to large bending deformation due to aerodynamic loadings, the FMCs can be ideal for enhancing the drivetrain performance by absorbing the lateral deformation without shaft segmentation. However, the increased lateral flexibility and high internal damping of the FMCs may induce whirling instability at supercritical operating conditions. Thus, the purpose of optimization in this paper is to find a set of tailored FMC parameters that compromise between the lateral flexibility and the whirling stability while satisfying several criteria such as torsional buckling safety and the maximum shaft temperature at steadystate conditions. At first, the drivetrain was modeled based on the finite element method and the classical laminate theory with complex modulus approach. Then, an objective function was defined as a combination of an allowable bending deformation and external damping and a genetic algorithm was applied to search for an optimum set with respect to ply angles and stack sequences. Results show that an optimum laminate consists of two groups of layers: (i) one has ply angles well below 45$^{circ}$ and the other far above 45$^{circ}$ and (ii) the number of layers with low ply angles is much bigger than that with high ply angles. It is also found that a thick FMC shaft is desirable for both lateral flexibility and whirling stability. The genetic algorithm was effective in converging to several local optimums, whose laminates exhibit similar patterns as mentioned above.