Application of the stability theorems for multilayer quasigeostrophic flows reveals that the three-layer model may be nonlinearly unstable while in linearly subcritical conditions, the instability being then due to explosive resonant interaction of Rossby waves. This contrasts with the Phillips two-layer model for which linear theory suffices to explain any instability and motivates this study of the nonlinear saturation of instability in the three-layer model.
A rigorous bound on the disturbance eddy energy is calculated using Shepherd's method for a wide range of basic shear and channel width. The method is applied using stable basic flows whose stability is established by either Arnol'd's first or second theorem. For flows unstable through explosive interaction only, the bound indicates that the disturbance energy can attain as much as 40% of the basic flow energy, the maximum disturbance energy being obtained for flows close to linear instability.
With regard to linear instability, an important difference between two- and three-layer flows is the disappearing of the short-wave cutoff for certain basic shears in the three-layer model. The significance of this phenomenon in the context of saturation is discussed.