1.1.

Forecasting Filter

In most existing systems, the servo-controller is a simple integrator with a gain that can be adjusted to make the best possible compromise between increasing the rejection bandwidth (high gain) and reducing the propagation of WFS noise into the command signals (low gain).² In this architecture, no attempt is made at predicting the command signals into the future to reduce the lag error (an integrator can be seen as a zero-order predictor). However, it is reasonable to expect a significant level of correlation between realizations of the atmospheric turbulence separated by only 1 ms. Therefore, this correlation can be used to attempt to increase the accuracy of the command signals by forecasting. Several methods have already been proposed to explore this idea with predictive controllers, either based on statistical models of atmospheric turbulence or being purely data-driven. See, e.g., the recent work presented on data-driven predictive control³ and the references therein.

We propose adding a forecasting filter to the existing integral control that takes command signals computed from previous frames and forecasts the command for a future frame. We expect that such a forecasting filter would be easy to retrofit as an extension of conventional AO RTCs, in which the current forecasting filter is implicitly the identity operator. Figure 1 shows where in the control system the filter would be placed. An ideal forecasting filter would itself be computationally minimal to keep the additional time delay small. For example, the computational complexity of the forecasting filter could be compared with that of wavefront reconstruction.

Many previous studies on AO control rely on testing with data obtained by end-to-end AO simulation software implementing a statistical model of the atmospheric turbulence (e.g., Ref. 4) or by laboratory experiments using artificial turbulence (e.g., Ref. 3). Instead, we design and evaluate our forecasting filter using a purely data-driven approach based on on-sky AO telemetry data, as outlined in Sec. 2. AO telemetry data are usually readily available in operating AO systems and have been successfully used in a few previous studies.^5–7 In Sec. 3, we examine the command signal as a time series, and in Sec. 4, we outline the linear forecasting models.

In Sec. 5, we apply these models to the data and approximate parameters such as how many previous frames are required and what levels of improvement are attainable. Finally, in Sec. 6, we explore more complex, nonlinear models for forecasting using neural networks.

4.1.

Image Stabilization

Image stabilization is controlled using TTM. Dynamic image motion is caused by atmospheric turbulence as well as by other effects within the telescope, such as wind shake and vibrations caused by the electromechanical systems.

Our aim is to make forecasts for the tip and tilt channels separately. We define two types for forecasting; models of the first type use a single channel and assume that tip and tilt are independent, and the other allows for linear interactions between both channels.

In the independent type forecasting, a single channel is used as input to forecast the command corresponding to that channel. Models using the dependent type of forecasting have access to both channels and therefore double the number of parameters needed to define them compared with the independent type. For example, a ${\text{Single}}_{\text{tip}}$ model trained with a Lookback of 50 will have 50 parameter coefficients; a ${\text{Double}}_{\text{tip}}$ model with the same Lookback will have 100 parameter coefficients: 50 derived from each of the tip and tilt channels while making a forecast for the tip channel. The model parameters are found with ordinary least squares using a small amount of data from the session.

4.2.

High-Order Turbulence Mitigation

The actuators of the DM are arranged in a square lattice with an annular shape. The uncontrolled actuators are located around the edge with one in the center of the array (caused by telescope central obscuration). For simplicity, we use separate linear models for each of the 136 controllable actuators. These 136 separate forecasts are compiled into a single forecast for the DM.

We also examine the quality of forecasts using modes of the large-scale patterns of deformation in the mirror, such as Zernike modes. Low order modes, such as defocus and astigmatisms, contain most of the variance of the atmospheric turbulence and have a higher signal-to-noise ratio than higher order modes. We see in Sec. 5.2.2 that the amount of Lookback required for a given improvement may decrease with the mode index; however, the method that we present here keeps the amount of Lookback consistent for each mode.

We use the ALTAIR system’s modal basis, which spans the entire wavefront subspace spanned by the 136 controlled DM actuators. This orthonormal modal basis is constructed such that each mode fits as closely as possible the corresponding Zernike polynomial. Converting into modal space is done via a $136\times 136$ conversion matrix, which is available in an ALTAIR configuration file. Again, one model was fitted to each mode, meaning that the forecast requires combining many small models together before being converted back into actuator commands. The benefit of forecasting commands using modes is that a few modes can be used to forecast the majority of the variation while ignoring the noisier high order modes. This is more important where computational limits exist and implementing one forecasting filter per DM actuator is not feasible. Also, the ALTAIR modes are not rigorously statistically independent; however, the low order modes are very close to the Karhunen–Loeve modes for the Kolmogorov turbulence, making the underlying assumption of statistical independence reasonable.

5.1.

Tip and Tilt Forecasting

Lookback is the number of previous frames that we give a model when fitting. This AO system uses a 1-kHz frame rate, so a Lookback of 20 corresponds to 20 ms. Each model was evaluated with a 90/10 train/test split. Figure 4 shows that increasing the Lookback reduces the modeling error. Our baseline of using no forecasting filter (Echo) is shown as a dashed horizontal line for each channel. These values are stationary because the Echo forecast is equivalent to having a Lookback of 1. We see that most of the improvement is achieved for a Lookback as low as 5 and using a Lookback greater than 30 has marginal improvements. Moreover, using both channels for forecasting, a slight reduction in error can be achieved.

Fig. 4

Forecasting error as a function of Lookback.

Given a Lookback of 30, we then test how many samples are required to fit the models. To obtain these results, the test set was held constant as the final 10% of the session, and the samples used for fitting the models immediately precede the test set. Therefore, using fewer samples does not introduce a delay between the two sets of data, so the models can be directly compared. Figure 5 shows the RMS tip-tilt error (RMSE) as a function of training data. Each point shows a model fit with the amount of training samples and evaluated on the test set. This figure shows that 10,000 samples are more than sufficient, representing 10 s of data. ${\text{Double}}_{\text{tip}}$ models require more samples for training as they have more degrees of freedom. Although we use 10,000 samples for training going forward, similar results can be achieved using fewer samples. ${\text{Echo}}_{\text{tip}}$ and ${\text{Echo}}_{\text{tilt}}$ provide a reference RMSE of these channels for this part of the session.

Fig. 5

Forecasting error as function of the number of training samples.

Figure 6 shows the improvement of our models compared with Echo for multiple sessions. We fit models for each channel for each session and evaluated them on a testing set composed of the final 10% of the session. Models were fit with a Lookback of 30, with 10,000 samples. Figure 6(a) shows significant improvements over Echo. Figure 6(b) shows a factor improvement of our method above Echo and more clearly illustrates that the ${\text{Double}}_{\text{tip}}$ models consistently perform better than the ${\text{Single}}_{\text{tip}}$ models.

Fig. 6

Forecasting error for different sessions. (a) Full vertical range and (b) relative improvement over Echo.

Forecasting for delays larger than one frame increases the forecasting error. However, in Fig. 7, we can see that our methodology still improves the lag error compared with the Echo baseline. This trend is observed over multiple sessions, with increased variation observed at larger time delays. The ability to forecast for longer time delays is particularly interesting as it opens the possibility of “slowing down” the AO system to increase the exposure time on the WFS, allowing the observer to use fainter sources and therefore increasing the portion of the sky available for NGS AO observations.⁹

Fig. 7

TTM forecast error as a function of AO time delay, expressed in number of frames for (a) tip and (b) tilt channels. The median values are calculated from 32 sessions, and the filled area is one standard deviation.

5.2.

High-Order Turbulence Forecasting

For the high-order turbulence corrected by the DM, we test our linear forecasting model in actuator space (Sec. 5.2.1) and in modal space (Sec. 5.2.2). These models are the same linear autoregressive models outlined in Sec. 4.

5.2.1.

Actuator space forecasting

In actuator space, we use raw actuator commands to forecast subsequent actuator commands. We test models with a Lookback of up to 30 ms on a single session. Figure 8 shows that the forecast improvement plateaus beyond a Lookback of 20 and that over 80% of the improvement can be captured with a Lookback of 5 for this session. The distribution of actuator errors is a function of where they are located in the mirror, with central and edge actuators accounting for the highest improvements. This makes sense because these actuators are controlled by fewer and noisier WFS subapertures and therefore carry more measurement noise. Because of this variation, we show the best and worst performing actuator forecasts as well as the median. The best and worst actuators are selected by their forecasting quality at $\text{Lookback}=1$. Vertical dashed lines show the improvement of the median relative to its lowest observed RMSE, which occurs at $\text{Lookback}=30$.

Fig. 8

Forecasting error as a function of Lookback for high order turbulence using actuator commands as inputs. Each session has 136 actuators; shown here are the two extremes and the median forecasting quality. Vertical dashed lines show the relative improvement of the median compared with the best observed improvement (i.e., at Lookback = 30).

To illustrate that 11Jan08 is a representative session of the range of turbulence conditions, we show in Fig. 9 the trends for multiple sessions. Figure 9 shows the median session forecast, quality and we plotted the best, worst, and median session qualities. Although there is left-over variability within the RMSE values for each session, the trends across sessions are comparable. To help with visual clarity, we only show the median Echo.

Fig. 9

Forecasting error of multiple sessions as a function of (a) Lookback and (b) number of training samples for high-order turbulence using actuator commands as inputs. Each session is summarized by its median performance, and we plotted the best, worst, and median sessions. Vertical dashed lines show improvement relative to the best observed median improvement. (a) 10,000 samples were used for training and (b) Lookback of 20 was used.

5.2.2.

Modal space forecasting

In modal space, actuator commands are converted into modal commands with a conversion matrix. The models in this section then take modal inputs and forecast the modal representation of commands for the subsequent frame. Unlike actuator data, amplitudes vary between modes. As shown in Fig. 10, defocus (the first high-order mode) largely dominates, with amplitudes of subsequent modes decreasing as the mode number increases, as expected in a Kolmogorov turbulence. For all modes except defocus, we find that the forecasting error plateaus within a Lookback of 5. Due to its much higher amplitude, defocus benefits from a longer Lookback.

Fig. 10

Forecasting error as a function of Lookback in modal space. Four modes are selected for plotting. Defocus is mode 2, astigmatism 1 is mode 3. 10,000 samples were used for training.

Forecasting using either modes or actuators provides an improvement over Echo. We compare each method by looking at the relative improvement over the Echo baseline. First, we convert the modal forecast into actuator space. For each actuator, we derive a single value score representing the relative improvement by dividing the RMSE of the Echo forecast by the RMSE of the respective method. Figure 11(a) shows where on the actuator map these improvements occur. These values are from the modal forecast, but the actuator forecast has a nearly identical distribution. The center pixel is blank as it is obscured by the telescope. In Fig. 11(b), the relative improvement by each method is unraveled into an actuator index. Forecasting in modal space results in a lower forecasting error compared with forecasting actuators directly. Because forecasting using modes can influence the entire array, we find that $\sim 50\%$ of our observed improvement is from the first five modes starting with defocus. Therefore, the computational requirement could be reduced if needed by forecasting only a limited number of modes. It is worth noting that here we have restricted the system to one filter per actuator (or per mode). Grouping neighboring actuators using a spatial filter as explained in Ref. 6 might allow for taking advantage of local spatial correlations in the wavefront and thus improve the actuator space forecasting.

Fig. 11

Comparison of DM forecast by method. (a) Relative improvement of modal forecasts mapped to actuator locations over the Echo forecasts. The central pixel is occluded by the telescope. (b) The relative improvement over an Echo forecast at each actuator by each method.

6.1.

Methodology

We define the residual error between linear forecasts and true actuator commands at $t+\tau$ as a new time series, where $\tau$ represents the forecasted time. The NNs take actuator commands from previous frames and fits them to the residual of the corresponding linear forecast at $t+\tau$. This method extracts any potential nonlinear trends from the actuator commands that the linear model is unable to capture. The time series residual is extracted from the 11Jan08 session. For the tip channel analysis, we use forecasts made by the ${\text{Double}}_{\text{tip}}$ model, which is fit with a Lookback of 30. Similarly, for mode 22, we use forecasts made with a linear model fit with a Lookback of 30. These forecasts are subtracted from the true actuator commands, resulting in a time series with $\sim \mathrm{60,000}$ frames. The NNs are trained using the first 90% of the residual time series.

We generate a corrected actuator forecast by adding the residual forecasted by the NNs to the corresponding linear forecast. If the corrected forecast yields a significantly lower RMSE than the corresponding autoregressive forecast alone, then we can estimate the degree of nonlinearity within the time series. Such information could inform the development of more sophisticated models that could be implemented in real time. For this analysis, the new “forecasting filter” in Fig. 1 is extended to include a “nonlinear residual forecasting filter” as shown in Fig. 12.

Fig. 12

Block diagram of a forecasting filter with linear and residual forecasting filter.

6.2.

Neural Network Architectures

We limit our evaluation of NN architectures to three types, following Lim et al.,:¹¹ (1) recurrent NNs, implemented as a long short-term memory (LSTM) network; (2) attention mechanisms, implemented as an LSTM with an attention layer; and (3) dilated convolutional networks, implemented as the WaveNet architecture.

Both (1) and (2) rely on LSTM units, which are regularly used to model stateful sequences due to their ability to handle long-range dependencies¹² by incorporating a hidden state and a gated memory cell from the previous inputs. Whereas traditional sequence-to-sequence architectures based on recurrent NNs need the whole sequence, ultimately keeping only the last compressed state, the attention mechanism¹³ adds the extracapacity to look (attend) at all previous hidden states of the sequence in the Lookback window, assigning weights to the most relevant ones for the prediction.

The WaveNet architecture was first developed as a generative model for raw audio¹⁴ and is based on the convolutional NN handling the whole input sequence, as well as masking the future inputs to preserve the temporal structure from the input data. This popular network architecture has been shown to perform well not only in audio generation but also in language processing and time series forecasts.¹¹

Each architecture went through a process of hyperparameter optimization by training each model with a range of values, and in each case we used a 90/10 validation split. For (1) and (2), we found a stable loss plateau after 500 epochs using a batch size of 2048, with the Adam optimizer set with a learning rate of 1e-5 for the tip channel analysis and 1e-6 for mode 22. In the case of (3), the same loss plateau is achieved using a batch size of 1024, with the Adam optimizer set to a learning rate of 1e-5 after 25 epochs.

6.3.

Results

In Table 1, we present the results of the NN correction for tip and mode 22 for both $t+1$ and $t+7$, which we derive by adding the forecasted residual to the corresponding linear model forecast. From this, we calculate the RMSE of the corrected forecasts against the true actuator values. Due to the stochastic nature of the machine learning algorithms used, we present the mean value of the RMSE along with the standard deviations derived from 10 different seed values on the same session. For comparison, we also present the RMSE of the baseline linear model without any nonlinear correction. Because the linear baseline is deterministic and we are using the same session to isolate the NN initialization variation, the linear baseline does not have error bars.

Table 1

Summary of NN architectures and performance compared with linear baseline.

In most cases, the corrected forecasts made by the NN models approach do not exceed the performance of the autoregressive models. When the corrected forecasts exceed the performance of the autoregressive model, the improvements are very marginal (percent of a mas or percent of a nanometer) and come at a great computational cost. We find no evidence that additional information can be practically extracted from these models to aid in forecasting.