Abstract: Buckling-Restrained Braces (BRBs) are widely used in seismic structural systems due to their stable hysteretic performance. The primary reason for their resistance to buckling under compression is the restraining effect on the core flexural deformation. While extensive research has been conducted on the anti-buckling mechanism of flat plate cores, there remains a lack of understanding regarding the sequence of overall buckling and local buckling for tubular cores. The influence of key parameters on the stability of tubular BRBs is also not well understood. Therefore, this study reveals the sequence of overall buckling and local buckling of square hollow section (SHS) cores within engineering parameter ranges, as well as the evolution of contact modes through theoretical derivation. A formula for calculating local contact force is provided. Subsequently, finite element models are established upon existing experimental results to analyze the impacts of the width-to-thickness ratio, of the outer tube stiffness, and of the gap on the load-carrying capacities of BRBs under cyclic loading. Lastly, the sequence of overall contact and local contact is reflected through finite element analysis and the formula proposed is verified. The calculation results show that local contact mode would occur at the state of overall n points contact when core width-to-thickness ratio and overall slenderness are within specific ranges. The overall stability is enhanced as the bending stiffness of the outer tube increases, but its influence on local stability is limited. SHS cores with a width-to-thickness ratio (b/t) of 10 are prone to occur overall buckling, while those with b/t of 13 and 18 are more likely to exhibit interactive buckling and local buckling modes. Reducing gap does not enhance the stability of the specimen which is prone to occur local buckling.