Chaotic Time Series Prediction of Beam-Ring Structured Systems Based on the S-Transformer With Convolutional Attention Mechanism

Abstract

Abstract. Since its introduction, the Transformer has demonstrated robust performance. However, its capacity to predict nonlinear dynamical systems remains constrained. In this study, we enhance the structure and propose a novel network model, S-Transformer (Simplified Transformer), which exhibits high efficiency and accuracy in forecasting the chaos time series. A convolutional attention mechanism is introduced in place of the attention mechanism inherent to the Transformer architecture, which replaces the traditional multihead attention mechanism. To evaluate the model's universality, experiments were conducted on both the deployable antenna beam-ring structure and the classic Lorenz-63 chaotic benchmark. The mean squared error (MSE) is employed as the evaluation metric. Numerical simulations on the beam-ring structure demonstrate that the prediction accuracy of the S-Transformer is 75% higher than the long short-term memory encoder-decoder (LSTM ED). Furthermore, the Conv-AT (Conv-Attention Transformer) model achieves a 96% improvement over the S-Transformer in this application. On the Lorenz-63 benchmark, the S-Transformer exhibits superior long-term stability and significantly extends the valid prediction horizon compared to the LSTM ED. It can be concluded that the enhanced networks exhibit a pronounced capability for forecasting complex nonlinear systems. Some of the research highlights are as follows: (1) A high-precision simplified Transformer model is constructed. (2) Convolutional attention replaces traditional attention for better results. (3) The simplified Transformer shows a 75% accuracy improvement over earlier models. (4) The new attention model further improves accuracy by 96%.