Abstract: To reduce the nonlinear flutter amplitude of long-span bridges, a two-degree-of-freedom bending-torsion coupled nonlinear self-excited force model is established. According to the tuned mass damper control method, the motion equation of the wind-bridge-TMD system is derived, and a new TMD parameter optimization method based on the GBO optimization algorithm is constructed. Taking the long-span suspension bridge segment model as an example, the control effect on the nonlinear flutter of the long-span bridge of the optimization method and of the traditional TMD parameter optimization method is compared. The sensitivity of TMD parameters to the nonlinear flutter control effect is studied, and the influence of structural and aerodynamic nonlinear effects on the TMD optimization design under different flutter control objectives is analyzed. The research results show that the two-degree-of-freedom bending-torsion coupled nonlinear self-excited force model can accurately predict the nonlinear flutter amplitude of the main beam. The new TMD parameter optimization method considering structural and aerodynamic nonlinearity has better control effect on the nonlinear flutter amplitude of long-span bridges than that of the traditional TMD parameter optimization method. The TMD optimized by this method not only significantly reduces the nonlinear flutter amplitude of the bridge deck under different wind speeds, but also effectively reduce the minimum mass ratio of TMD under different flutter control target designs. The research conclusions provide a theoretical basis for the application of TMD in nonlinear flutter control of long-span bridges.