Challenges of real-time processing in HPC environments: the ASKAP experience

Eric Bastholm; Juan C. Guzman; Maxim Voronkov; Paulus Lahur

doi:10.1117/12.2313137

6 July 2018 Challenges of real-time processing in HPC environments: the ASKAP experience

Eric Bastholm, Juan C. Guzman, Maxim Voronkov, Paulus Lahur

Proceedings Volume 10707, Software and Cyberinfrastructure for Astronomy V; 107070J (2018) https://doi.org/10.1117/12.2313137
Event: SPIE Astronomical Telescopes + Instrumentation, 2018, Austin, Texas, United States

Abstract

The Commonwealth Scientific and Industrial Research Organization (CSIRO) Astronomy and Space Science division operates the Australian Square Kilometre Array Pathfinder (ASKAP), a state-of-the-art radio telescope located at the Murchison Radio Observatory (MRO) in Western Australia. The data rates produced are significant enough that it is necessary to process in real-time in high performance computing (HPC) environment. This presents a number of challenges with respect to resource availability and the performance variability resulting from interactions of many parallel software and hardware components. These issues include: disk performance and capacity, inter-node and intranode parallel communication overheads, and process isolation. Typically, supercomputing facilities are used for batch processing of large established data sources or simulations. Small intermittent processing delays resulting from temporary resource contention issues do not affect these processes as the total compute time is the dominating factor and small wait times are insignificant. However, for real-time processing of high data ingest rates these small and unpredictable contentions for resources can be problematic. In our case, even delays of seconds can have a negative effect on our ability to reliably ingest the data. We have learned much from addressing these challenges and our experience and solutions will provide valuable input to the radio astronomy community, especially for larger projects, like the Square Kilometer Array (SKA), where the same challenges will present at even larger scales.

Conference Presentation

Citation Download Citation

Eric Bastholm, Juan C. Guzman, Maxim Voronkov, and Paulus Lahur "Challenges of real-time processing in HPC environments: the ASKAP experience", Proc. SPIE 10707, Software and Cyberinfrastructure for Astronomy V, 107070J (6 July 2018); https://doi.org/10.1117/12.2313137

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available