2.1.

Channelization

The exact steps performed by a channelizer are likely to vary from one instrument to the next. The MeerKAT wide-band channelizers (both the original FPGA design and our new GPU design) perform the following steps. These are essentially the same as those performed by Cobalt.⁵

In the signal processing steps described above, the two polarizations remain independent of each other. However, to maintain compatibility with the MeerKAT packet formats, each output packet contains data from both polarizations, and hence the channelizer needs to operate on both polarizations together. Doing so also allows for new features in the future, such as correcting for polarization leakage. We refer to a pipeline performing all the steps above for two polarizations of a single antenna as an F-engine. A single server may run multiple F-engines as independent processes.

2.2.

Network Protocol

MeerKAT uses the streaming protocol for exchanging astronomical data (SPEAD),¹⁰ deployed over multicast user datagram protocol (UDP). This is a protocol for transmitting multi-dimensional arrays of data with associated metadata (such as timestamps). The basic protocol data unit is the “heap,” which may be fragmented into multiple UDP packets and reassembled by the receiver.

Digitizers send voltage samples in 4096-sample heaps, each comprising a single packet. The 10-bit samples are packed, so the heaps contain 5120 bytes of payload. The two polarizations are sent independently.

To reduce the number of F-engine output heaps, each heap contains data for 256 spectra. The data in a heap are a $c\times 256\times 2$ array of Gaussian integers, where $c$ is the number of channels sent to each consumer, and for MeerKAT can be anywhere from 4 to 2048. Output heaps may comprise multiple UDP packets, as they are typically larger than the largest possible UDP packet.

2.3.

Graphics Processing Units

Our target GPUs are those from NVIDIA, and so we will use the terminology used by Compute Unified Device Architecture (CUDA; NVIDIA’s programming toolkit). GPUs from other vendors are similar but use different terminology. CUDA-capable GPUs have multiple levels of parallelism:

The GPUs we have tested all connect to a host system via 16 lanes of a peripheral component interconnect express (PCIe) 4.0 bus. The CUDA busGrind tool typically shows that NVIDIA GPUs can sustain $26\mathrm{GB}/\mathrm{s}$ for unidirectional traffic and $21\mathrm{GB}/\mathrm{s}$ each way for bidirectional traffic. This is 1 to 2 orders of magnitude less than the bandwidth of the random access memory (RAM) on the GPU and can easily become a bottleneck for data streaming applications.

3.1.

Batching

The programming model in CUDA (and other GPU programming interfaces) is batch-oriented rather than based on a continuous stream of data. Large batches of work (millions of threads) allow for sufficient parallelism to keep the GPU fully utilized. We thus break the input stream into chunks of a few million samples. The output stream is similarly decomposed into chunks, but for reasons we will explain later, they are not in one-to-one correspondence. Because the Python code is involved at the batch level (rather than on individual samples or packets), large chunks also help amortize the overheads of the relatively slow interpreter.

To ensure that all parts of the system are kept busy, we use a pipelined design with components connected by queues of chunks, as shown in Fig. 1. In particular, we need host-to-GPU transfers, GPU-to-host transfers, and GPU computations to happen concurrently to maximize the overall throughput. We use Python’s asyncio library to manage these concurrent operations.

Fig. 1

Processing linked by queues. Where the diagram shows a circular buffer, the implementation uses one queue carrying full buffers forward, and a second queue carrying empty buffers backwards.

The PFB uses overlapping windows, which means that some computations require data spanning an input chunk boundary. To allow the PFB calculation to operate on contiguous data, each chunk is allocated on the GPU with some extra space at the end. The prefix of the following chunk is then copied to this space, and computations are performed on this expanded chunk. Provided that this extra space is at least as large as the PFB window size, every PFB window can now be located inside a single chunk, as shown in Fig. 2.

Fig. 2

Overlapping chunks. The dark areas show the chunks as originally received from the network. Each chunk is then extended (light area) to contain a copy of some data from the following chunk. The braces show the overlapping PFB windows, with every spectrum computed from contiguously-stored data.

3.2.

Networking

We use the spead2 library (a high-performance implementation of the SPEAD protocol) both to receive input heaps from the digitizers and to transmit output heaps to the X-engines. On the receive side, it supports collecting multiple heaps into a chunk, reordering them as necessary based on timestamps,¹¹ before passing the chunk to the Python code for processing. It also allows the Python code to control the allocation of the memory: we allocate it in CUDA pinned memory, which allows it to be efficiently copied to the GPU.

It would have been simplest to treat the two polarizations jointly in the receive code, placing them into a single chunk. Unfortunately, spead2 uses locks in a way that prevents multiple threads from working on a chunk in parallel, and we were not able to achieve the required performance with a single thread. Thus, each input chunk contains only a single polarization, and Python code is used to pair up chunks with the same timestamp. This allows the two threads to receive data in parallel, but it adds complexity because this code needs to handle corner cases where one polarization is lost.

On the transmit side, spead2 allows heaps to be defined in advance with pointers to the data. These may then be transmitted many times with the values pointed to changing each time. It also allows a list of heaps to be submitted for transmission in one step. When allocating the output chunks, we also create the corresponding heap structures, thus minimizing the overhead incurred at transmission time.

3.2.1.

Data transfer

In the default implementation, each input sample is involved in four host memory accesses, and each output sample in two, as shown in Fig. 3(a). The network interface card (NIC) writes packets directly to RAM. The spead2 library then assembles the packet payloads into chunks, again in RAM. The final input step is that the GPU pulls the chunks from RAM. On the output side, the GPU copies chunks to RAM, and the NIC pulls data from RAM. There is no need for the CPU to copy data into individual packets, because the NIC is able to gather the headers and payload for each packet from different addresses.

Fig. 3

System data flow. (a) Default. (b) System RAM usage is reduced by having the CPU write directly to the GPU and the NIC read directly from the GPU.

NVIDIA’s GPUs are able to map GPU memory into the system’s address space. This allows for a data flow that places less load on the system’s RAM, as shown in Fig. 3(b). First, when spead2 assembles chunks, it writes directly to the GPU memory, rather than to a staging area in host memory. Second, the NIC is given pointers to GPU memory, rather than to a copy in host memory.

For the latter optimization, the results are disappointing. We found that having the NIC read directly from the GPU performs well only when the GPU is idle; when its memory system is heavily used, the achieved bandwidth drops to below $120\mathrm{Gbit}/\mathrm{s}$, significantly $<160\mathrm{Gbit}/\mathrm{s}$ which we are able to achieve by staging through host memory. We are able to achieve by staging through host memory. NVIDIA recommends¹² using a motherboard where the GPU and NIC sit behind a PCIe switch, which is not the case for the systems we tested, and so better results may be possible. It should also be noted that we were only able to get this feature working at all on a data center GPU (A10) and not on gaming GPUs.

Having the CPU write directly to GPU memory is more promising. We were able to get the feature working on gaming GPUs but with the limitation that only 256 MiB can be mapped. This significantly limits the maximum chunk size, particularly when running multiple engines per GPU, and caused performance to be lower overall.

3.3.

Coarse Delay

The MeerKAT channelizer implements coarse delay by duplicating or removing samples from the stream. This is easy to do with an FPGA, but less so with a GPU since the samples are not streamed one at a time. In addition, any PFB windows overlapping the insertion or removal have a mix of different coarse delays, potentially leading to artefacts. In practice, the derivative of delay is small (less than $3\times {10}^{-9}$ for sidereal targets), and so these artefacts are rare.

Instead of inserting or removing samples, we handle coarse delay by adjusting indices used to fetch samples. For each output spectrum (with a given timestamp), we identify the appropriate position in the input stream at which to load the data to achieve the necessary delay. This approach allows for absolute delays that are essentially unlimited (even negative), and every PFB window uses a consistent delay. However, large step changes in delay (such as when switching targets) can be problematic if they require access to older data that have already been discarded. We can protect against this by increasing the size of the overlap zone shown in Fig. 2 by a number of samples equal to the largest desired instantaneous delay change. This is not a major issue for a dish array as a big change in delay center usually requires the dishes to be slewed, during which time the data will be discarded anyway.

A similar problem is that the two polarizations may have different delays, although the difference is expected to be very small since the delays are dominated by the geometric component, which is common. Provided the overlap zone is large enough, we can always find a pair of chunks with the same timestamp that holds the data for both polarizations.

3.4.

Polyphase Filter Bank

In this subsection, we describe only the filtering step (Eq. 1). The fast Fourier transform (FFT) step is described in Sec. 3.5.

It is easier to implement the filter efficiently if the coarse delay can be treated as a constant. As previously noted, coarse delay changes are rare, so we handle this by splitting each chunk into regions with fixed coarse delay and using a separate kernel invocation for each region. We will thus ignore it in the following exposition, as it is simply an index offset in the chunk.

The input samples can be viewed as a 2D array with width $2n$, in which the $i$’th column undergoes a FIR filter with weights $w_i$. This maps easily to CUDA, with one thread for each column. The thread holds $w_i$ in its registers, as well as a sliding window of input samples, thus minimizing the number of memory accesses required. However, this does not provide sufficient parallelism to fully occupy the GPU: at least hundreds of thousands of threads are needed. We thus split each column into smaller pieces, with a thread per piece. While the output space is completely partitioned between threads, some inputs are loaded by multiple threads, as shown in Fig. 4. There is thus a trade-off between having too few threads (and not fully utilizing the GPU) and too many (and performing many redundant loads). A heuristic we found worked reasonably well (but which may be need to be tuned to the GPU model) is to ensure that each thread computes at least $8T$ outputs, where $T$ is the number of taps, unless this would lead to fewer than 131,072 ($2^{17}$) threads.

Fig. 4

Relationship of threads to input samples in the PFB. Each black arrow shows the input samples loaded by a single thread, for a four-tap PFB.

3.5.

Fast Fourier Transform

Due to coarse delays, each invocation of the PFB FIR kernel produces a variable amount of data. We keep invoking it until we have enough data to fill an output chunk. The last invocation may need to be truncated to avoid overrunning the output buffer. Once this is done, we use a library to perform a batched one-dimensional (1D) FFT.

We have considered two libraries for the FFT: cuFFT (provided as part of CUDA) and vkFFT.¹³ The latter is highly configurable; we have used the defaults, except that the transformation is out-of-place. Figure 5 shows the performance of these two libraries on R2C and complex-to-complex (C2C) transforms. It is clear that for batched 1D transforms with the sizes of interest, cuFFT has superior performance, and so we do not consider vkFFT further.

Fig. 5

FFT library performance, with batched single-precision 1D transformations. All results use $2^{25}$ output values. Time is given per complex output value, and the horizontal line indicates the theoretical bound given by the memory bandwidth of the GeForce RTX 3070 Ti. Note that for R2C transforms, the input size is twice the output size.

3.5.1.

FFT precision

For MeerKAT, the digitizer samples are 10-bit signed integers, and the F-engine outputs are 8-bit signed integers. Since half-precision floating point (FP16) has a 10-bit mantissa, one might expect that a half-precision FFT would be sufficient, as rounding errors would be smaller than the quantization errors associated with the input and output.

While this may be true for a single instance of the FFT, it ignores the statistical properties of the errors. Provided the signal is suitably dithered,¹⁴ quantization errors will have zero mean and will not affect the expected value of the Fourier transform. In contrast, the rounding errors in the FFT are data-dependent, have spectral features, and have non-zero mean. It is known that fixed-point PFB implementations for radio astronomy benefit from extra internal precision for the FFT¹⁵ (MeerKAT uses 22-bit registers¹), but we are not aware of any studies for low-precision floating point. To test the effect of using an FP16 FFT, we synthesized some data as follows:

• 1. Generate a tone at a fixed frequency.

• 2. Quantize it to 10-bit signed integers, using a uniform dither.

• 3. Perform a R2C transform using cuFFT, in either FP16 or FP32.

• 4. Repeat the above many times and average the results (in double precision).

• 5. Square the absolute values of the averages to convert voltage to power.

An example of the results are shown in Fig. 6. While the noise floor is similar, there are harmonics at around $-75\mathrm{dB}$. The period of the features varies depending on the binary representation of the channel number used for the tone, reflecting the structure of the FFT. This is significantly higher than the noise floor of the PFB (Fig. 7) and in the MeerKAT environment could potentially cause strong sources of narrow-band radio frequency interference (RFI)¹⁶ to contaminate useful parts of the band. We thus chose to stick with FP32 for the FFT.

Fig. 6

FFT simulation using (a) FP32 and (b) FP16, averaged over $2^{24}$ iterations. Only power is shown (not phase). The tone is in channel 192 of 1024.

Fig. 7

Channel response for 1024 and 32,768 channels with 16 taps.

3.6.

Post-Processing

The remaining steps are applying fine delay, bandpass corrections, and quantizing to 8-bit signed Gaussian integers. These are all quite straight-forward to implement, as they can be computed independently for each sample. We use inline parallel thread execution (PTX; CUDA’s intermediate representation) to perform the quantization with rounding and saturation.

In addition, the data are transposed: the input is time-major, channel-minor, but the layout expected by the X-engines is channel-major within each heap. The transposition is done in shared memory to improve the memory access pattern.¹⁷

3.7.

Lost Data

So far, we have assumed a lossless network in which all expected packets actually arrive. While we aim to have enough headroom that data loss does not routinely happen, we still need to handle it gracefully. Within each chunk, spead2 sets flags indicating which heaps were actually received. This information is carried through the pipeline: if the window of input samples for a PFB has any missing data, the output spectrum is flagged as unusable. Any output heap that contains unusable spectra is simply not transmitted. This may also cause usable spectra to be discarded but since this is not expected to occur during normal operation, we have not attempted to optimize it.

4.1.

R2C Transform

We will start by eliminating the extra pass required for the R2C transform. While the cuFFT documentation does not describe how R2C transforms are implemented, the name of the final kernel (postprocessC2C_kernelmem) suggests that it uses a technique that first treats the even and odd elements as real and imaginary components of complex numbers, performs a C2C transform, and then performs post-processing to get the final result.¹⁸

Instead of having cuFFT apply this technique, we can apply it manually, with cuFFT handling just the C2C step. The advantage of doing the post-processing ourselves is that it can be integrated into the post-processing kernel, thus eliminating a round trip to memory.

4.2.

Unzipping the FFT

The two passes used by cuFFT’s C2C transform correspond to the “four-step” FFT,¹⁹ in which a transform of size $ab$ is decomposed into $b$ transforms of size $a$ followed by $a$ transforms of size $b$, with the smaller transforms all computed within a faster level of the memory hierarchy (in this case, on-chip shared memory).

As in the previous subsection, we can improve efficiency by performing this decomposition ourselves and merging some of the steps with existing kernels. Our approach is actually based on the “six-step” FFT:¹⁹

In our implementation, there are no explicit transposition passes; instead, indexing of the surrounding operations is adjusted to take the transposition into account. This makes the transposition “free” in the sense that it does not directly cause extra memory transfers, but it does lead to less efficient memory access patterns as contiguous accesses are replaced with strided access.

We incorporate step 1 into the PFB FIR kernel (adjusting the addresses at which values are written), perform step 2 with cuFFT, and fold the remaining steps into the post-processing kernel.

We refer to $b$ as the “unzipping” factor. While the four-step FFT is normally used with $a$ and $b$ having similar magnitude, we prefer to use a small value, specifically $b=4$, for several reasons:

Figure 5 shows that the cost of step 2 is largely independent of $a$ provided it is at most 8192, so the increase in $a$ that comes from reducing $b$ is not an issue. For simplicity, we have kept $b=4$ for all channel counts (1024 to 32,768).

We also attempted to make the transpositions more explicit using shared memory¹⁷ but found that the synchronization overheads outweighed the benefits. It is possible that a more sophisticated implementation (for example, using recent CUDA asynchronous application programming interfaces) would achieve better results.

5.1.

Hardware

Unless otherwise noted, all results are for a GeForce RTX 3070 Ti GPU. To improve reproducibility, we have locked the graphics and memory clocks to 1575 Hz and 9251 MHz respectively, which gives theoretical performance of 19.35 TFLOP/s (single precision) and $592\mathrm{GB}/\mathrm{s}$. Despite this, we have found that the performance of the post-processing kernel drops by 20% to 25% if it is repeated thousands of times in a tight loop, so the results for that kernel are measured on 1000 iterations at a time. This does not seem to occur when mixed with the other kernels in real-world usage.

The CPU is an Advanced Micro Devices (AMD) EPYC 7313P (Milan) with 16 cores, 3 GHz base clock, and 3.7 GHz boost clock, equipped with 64 GiB of DDR4-3200 on a Supermicro H12SSL-i motherboard. We considered disabling the boost clock to give more consistent results (similar to locking the GPU clocks) but found that doing so made a huge reduction in performance and did not substantially improve consistency. We thus chose to keep the boost clocks enabled so that results correspond more closely to real-world usage. Our tests are relatively short-running, and it is possible that performance will decline in a system that runs continuously due to thermal limitations.

The network card is an NVIDIA ConnectX-6 Dx with dual $100\mathrm{Gbit}/\mathrm{s}$ ports.

5.2.

Channel Response

To measure the channel response, we use a simulated digitizer that generates a common full-scale tone in both polarizations but with independent dithering. We then cross-correlate the F-engine outputs and integrate over 8 s (we use a cross-correlation rather than an auto-correlation so the dithering noise is uncorrelated). By varying the frequency of the tone by small (sub-channel) amounts, we can determine the channel response of the engine. The 8-bit F-engine output does not have enough dynamic range to give a full picture, so we use different gain settings for different tones. Figure 7 shows the result for 1024 and 32,768 channels and 16 taps. It also shows the theoretical ideal computed by taking the Fourier transform of the PFB weights (a Hann-windowed sinc filter). There is extremely good agreement down to a noise floor around $-120\mathrm{dB}$. We believe the noise floor is higher with fewer channels because the ratio of coherent gain (gain for narrow-band signals) to incoherent gain (gain for white noise) depends on the channel count.

5.3.

GPU Throughput

5.3.1.

Polyphase filter-bank

Figure 8 shows a “roofline” plot of the performance of the pre-processing filter kernel, for 32,768 channels (the results for other channel counts are qualitatively similar). All the results are in the left-hand side of the graph, indicating that memory accesses dominate the performance. The configurations with up to 16 taps all use 75% or more of the available memory bandwidth. However, as the number of taps goes up, the number of registers needed increases, the number of threads that can be run concurrently decreases and the GPU’s ability to hide memory latency is reduced. With 32 taps, the theoretical occupancy (fraction of the theoretical maximum number of concurrent threads) is 41.67%.

Fig. 8

Roofline plot for PFB FIR filter. Within each cluster, the points are for 16, 12, 10, 8, and 4 bits per sample from left to right. All results are for 32,768 channels. The line indicates theoretical maximum performance.

While the roofline plot shows that the memory accesses that do occur are performed efficiently, it does not consider that some memory accesses are redundant. The kernel loads many bytes more than once, and if this is not absorbed by the caches, it will harm the throughput. The maximum potential throughput of the kernel can be computed from the total size of the input and output buffers and the theoretical bandwidth of the device. Figure 9 shows the achieved efficiency relative to this ideal, again for 32,768 channels.

Fig. 9

Efficiency of the PFB FIR filter. All results are for 32,768 channels.

The results above all use factor-4 unzipping. This results in uncoalesced memory writes, which reduces the performance by 4% on average over the test scenarios, and 19% in the worst case.

5.3.3.

Post-processing

As with the other kernels, the post-processing is memory-bound, with an arithmetic intensity of 6 to 7 flops per byte. Figure 10 shows the efficiency relative to the ideal of accessing every input and output value once at the theoretical maximum bandwidth. The efficiency declines with increasing channel counts because the memory access pattern causes some cache lines to be loaded multiple times. For 32,768 channels, an extra 7% memory traffic is generated.

Fig. 10

Efficiency of the post-processing kernel.

5.3.4.

Overall GPU throughput

For even a mid-range GPU, the maximum sampling rate that can be handled is limited by PCIe bandwidth rather than the computations on the GPU. For each combination of PFB taps and input bit depth, we have estimated the GPU memory bandwidth required to ensure that it does not become the bottleneck. To make this estimate, we used the following process:

• 1. Assume a PCIe bandwidth of $160\mathrm{Gbit}/\mathrm{s}$ in each direction (achievable on NVIDIA Ampere GPUs with a little headroom), and from this determine a sampling rate.

• 2. Measure the time required to run all the kernels on our test system, and use linear scaling to determine a memory bandwidth of a hypothetical GPU that would run the kernels just fast enough.

• 3. Since the measurement above does not include any PCIe transfers, add the GPU memory bandwidth required for transferring the inputs and outputs over PCIe. Similarly, add time to copy the prefix of each chunk to the suffix of the previous chunk. In both cases, we assume 100% efficiency in the memory accesses.

Figure 11 shows the results for 1024 and 32,768 channels. It may seem counter-intuitive that going from 8 to 16 bits per sample reduces the required bandwidth. This occurs because the PCIe bandwidth is kept constant and hence the sampling rate decreases. The bulk of the on-GPU memory traffic is performed in single-precision FP rather than scaling with the input bit depth, and so decreasing the sample rate decreases the memory bandwidth needed for that traffic.

Fig. 11

Estimated GPU memory bandwidth. This is the minimum bandwidth the GPU will require for the computations to become bottlenecked by a PCIe 4.0 bus. The top of each bar represents 32,768 channels while the horizontal line near the top represents 1024 channels.

To validate this model, we can artificially limit the memory clock on our test hardware to simulate a lower-end GPU. This is not a perfect test since a lower-end GPU would generally have fewer streaming multiprocessors and a narrower memory bus but should give some indication of the accuracy. Our GPU only supports a few fixed memory frequencies, so we fix it to 810 MHz, which gives a theoretical bandwidth of $51.84\mathrm{GB}/\mathrm{s}$. At 32,768 channels, 10-bit samples, and 16 taps, the model indicates a maximum sampling rate of $1077\text{Msample}/\mathrm{s}$. In practice, we found that $930\text{Msample}/\mathrm{s}$ was the highest rate we could run the full engine without falling behind the incoming data (to the nearest $10\text{Msample}/\mathrm{s}$). This shows that there are additional overheads not accounted for by the model, but (at least for this case) they are $<15\%$.

5.4.

System Tuning

We found that we needed to do a substantial amount of system-level tuning to obtain good performance, using a combination of hardware placement, basic input/output system (BIOS) settings, and kernel settings.

To test the throughput of the whole system, we run either one or four instances of the F-engine on the system under test. Input data are provided by digitizer simulators (one per F-engine) running on another machine. The output data are sent into the network (as multicast streams). Using four engines is representative of how the code is expected to be deployed for the MeerKAT extension, where the highest sampling rate will be $1750\text{Msample}/\mathrm{s}$. In this case, each engine is assigned to a single quadrant of the CPU and hence to a single core complex die (CCD). Tests with a single engine are aimed at measuring the maximum bandwidth achievable with the current implementation, and use one thread per CCD (one network receive thread per polarization, one network transmit thread and the main Python thread). The four engines operate entirely independently, just as if they were running on separate hosts.

Since the UDP protocol in use is lossy, it is not possible to measure maximum throughput directly. Rather, we repeat a number of experiments in which a fixed sampling rate is chosen, and we observe the F-engine input over 20 s to check if there are any gaps in the received timestamps (indicating packet loss). The engine is allowed to run for a few seconds before this observation period begins, as it is quite common for some packets to be lost while the process “warms up.” We then use a binary search to determine the highest sampling rate for which no packets are lost, to the nearest $10\text{Msample}/\mathrm{s}$. As the sampling rate approaches the critical rate at which the implementation can keep up, packet loss during the 20 s window becomes a random event, and so we see variation of a few percent even when the configuration is not changed.

These results should be seen as upper bounds, as running for 20 s under ideal conditions (for example, with no changes to delay) does not guarantee stable operation in real-world use.

5.4.3.

Hardware placement

The motherboard has five ×16 PCIe 4.0 slots, but performance-wise they are not all the same. The I/O die of the CPU is split into four quadrants. Each quadrant supports 32 PCIe lanes, but we found that a quadrant is not able to sustain full-bandwidth transfers between these lanes and system memory: the maximum is around 36 GB/s in each direction. We thus found extremely poor performance when placing the GPU and the NIC in a pair of slots connected to the same quadrant.

Even when using slots attached to different quadrants, not all combinations are equal. Table 3 shows the results with various combinations of slots, and Fig. 12 shows the association of the slots to the quadrants of the CPU.²²

Table 3

Effect of PCIe slots on performance. Percentages are relative to the top row, which is the baseline used in other results.

NIC slot	GPU slot	One engine		Four engines
		Msample/s	%	Msample/s	%
3	7	6260	100.0	2080	100.0
1	7	610	9.7	150	7.2
3	6	6170	98.6	2000	96.2
1	6	610	9.7	150	7.2
7	3	6180	98.7	2080	100.0
5	3	3550	56.7	900	43.3

Fig. 12

Mapping of ×16 PCIe slots to CPU quadrants on H12SSL-i motherboard.

5.5.

Power Consumption

We used the four-engine test case to measure power consumption, as it places greater demand on the system (by virtue of having greater total bandwidth). We used a sampling rate of $2000\text{Msample}/\mathrm{s}$, and other parameters are the same as for the system tuning results.

With the GPU clocks locked to the base values, the power consumption for the GPU, as reported by nvidia-smi, is 156 W, and the power for the whole system, as reported by the baseboard management controller, is 407 W. Unlocking the clocks causes power usage to increase by 73 W. On the other hand, the graphics clock can be reduced as low as 660 MHz without causing any loss of data, but doing so saves only 8 W compared to using the base clocks.

When using a sampling rate of $1712\text{Msample}/\mathrm{s}$, 4096 channels (a common configuration for MeerKAT), and base GPU clocks, the system power usage is 388 W, or 97 W per antenna. Within the margins of error, this is the same per-antenna power consumption as the current Square Kilometre Array Reconfigurable Application Board (SKARAB) FPGA platform used in MeerKAT. It should be noted that the SKARAB platform is almost a decade old, and hence is not representative of the power consumption of more modern FPGAs.