The ALMA Observatory experienced a cyberattack in late October 2022, disrupting regular operations and critical computing services for seven weeks. Although no scientific data was lost, operations data essential for computing services were affected. The incident highlighted the organization's limited knowledge of recovery assets and the absence of a specific cyber-attack disaster recovery policy. The attack raised cybersecurity awareness within the organization, shifting it from a secondary role to a cultural necessity. The paper discusses post-attack actions, strategies, and efforts to integrate cybersecurity into the organization's culture for enhanced cyber safety.
The Atacama Large Millimetre/submillimetre Array (ALMA) is the world’s largest ground-based facility for observations at millimeter/submillimeter wavelengths. Inaugurated in March 2013, ALMA has already accomplished ten years of continued steady-state operations. It comprises 66 antennas located approximately 5000 meters at the Chajnantor Plateau in the Atacama Desert in Northern Chile. The ALMA partnership established the ALMA 2030 development program to improve ALMA’s capability to avoid obsolescence for the next decade. The Wideband Sensitivity Upgrade (WSU) project, the first initiative of the ALMA 2030 development program, will replace the entire digital processing system, which includes the wideband digitizers, data transmission system, and data correlation system. A working group was charged to develop a WSU Deployment Concept based on a parallel deployment approach to minimize scientific downtime during the upgrade period, which could last up to five years. In this paper, the authors present the relevant aspects of this analysis and conclusions, which will pave the road to address the definition of the AIVC concept and the corresponding AIVC plan of the WSU project.
The COVID-19 pandemic forced some ALMA Observatory’s teams to change their working models from observatory on-site or office-based to fully remote. The performance results obtained by the groups during this emergency evidenced that a hybrid working model would be suitable to be implemented in the long term, especially for the teams that concentrate their activities out of the observatory site or Santiago’s offices. Science and computing groups were the most suitable teams for adopting a different working model. There were many lessons learned from this experience which contributed to establishing a permanent hybrid model. The ALMA Software group, consisting of 18 engineers, transitioned in this direction taking into consideration all the knowledge learned during the pandemic and developing a smooth and successful experience by maintaining productivity levels and cohesive team spirit despite the physical location of the group members. This paper provides an overview of key considerations, challenges, and benefits associated with the shift towards a hybrid working model. Factors/challenges such as technological infrastructure, communication and collaboration, collaborators' well-being and performance metrics are analyzed from the manager/supervisors' point of view. The paper also describes the challenges that the group will face shortly, and the actions developed to mitigate the risks and disadvantages of the new working environment.
The Atacama Large Millimeter/submillimeter Array (ALMA) has been working in the operations regime since 2013. After almost 10 years of successful operation, obsolescence of hardware and software emerged. On the other hand, the ALMA 2030 plan will add new disrupting capabilities to the ALMA telescope. Both efforts will require an increased amount of technical time for testing in order to minimize the risks to introduce instability in the operation when new equipment and software are integrated into the telescope. Therefore, a process to design and implement a new simulation environment, which must be comparable to the production environment, was started in 2017 and passed the Critical Design and Manufacturing Review (CDMR) in 2020. In this paper, the current status of the project was reviewed focusing on the assembling and integration period, and use cases that are started to be built on top of this testing facility.
ALMA has been operating since 2013 and it keeps on adding an ever-growing new set of capabilities. Every new feature implies among other things a new software release that has to be implemented, tested and deployed around the world. In this paper we present the new deployment process that allowed ALMA to deliver faster releases to reliable testing and production environments. This was achieved through the use of container-based services, both for applications and data. This implied that tasks that in the past were done manually, are now fully automated, in order to avoid human errors and maintain consistency between what was tested and what is finally installed in production. All this, under a unique and complex operation environment that includes the two main operation facilities in Chile at ALMA Operation Site Facilities (OSF) and Santiago Central Office (SCO), and the different executive headquarters located across the ALMA global network. We also explain how we managed to address the issue of ever-growing observational data, which made it difficult to replicate the production environment data into our testing infrastructure. Our solution consisted in using a container-based database that allowed us to create a full copy of the production database in a very short time. All those changes enabled JAO to improve its software testing process allowing a monthly release cycle.
KEYWORDS: Antennas, Optical correlators, Observatories, Computing systems, Signal processing, Hardware testing, Current controlled current source, Adaptive optics, Electronics, Control systems
The Atacama Large Millimeter /sub-millimeter Array (ALMA) has been working in operations phase regime since 2013. The transition to the operations phase has changed the priorities within the observatory, in which, most of the available time will be dedicated to science observations at the expense of technical time required for testing newer version of ALMA software. Therefore, a process to design and implement a new simulation environment, which must be comparable - or at least- be representative of the production environment was started in 2017. Concepts of model in the loop and hardware in the loop were explored. In this paper we review and present the experiences gained and lessons learned during the design and implementation of the new simulation environment.
The ALMA software is a large collection of modules for implementing all the functionality needed by the observatory's day-to-day operations from proposal preparation to the scientific data delivery. ALMA software subsystems include among many others: array/antenna control, correlator, telescope calibration, submission and processing of science proposals and data archiving.
The implementation of new features and improvements for each software subsystem must be in close coordination with observatory milestones, the need to rapidly respond to operational issues, regular maintenance activities and testing resources available to verify and validate new and improved software capabilities. This paper describes the main issues detected managing all these factors together and the different approaches used by the observatory in the search of an optimal solution.
In this paper, we describe the software delivery process adopted by ALMA during the construction phase and its further evolution in early operations. We also present the acceptance process implemented by the observatory for the validation of the software before it can be used for science observations. We provide details of the main roles and responsibilities during software verification and validation as well as their participation in the process for reviewing and approving changes into the accepted software versions.
Finally, we present ideas on how these processes should evolve in the near future, considering the operational reality of the ALMA observatory as it moves into full operations, and summarize the progress implementing some of these ideas and lessons learnt.
After its inauguration and the formal completion of the construction phase, the software development effort at the Atacama Large Millimeter/submillimeter Array (ALMA) continues at roughly the same level as during construction – gradually adding capabilities as required by and offered to the scientific community. In the run-up to a new yearly Observing Cycle several software releases have to be prepared, incorporating this new functionality. However, the ALMA observatory is used on a daily basis to produce scientific data for the approved projects within the current Observing Cycle, and also by engineering teams to extend existing capabilities or to diagnose and fix problems – so the preparation of new software releases up to their deployment competes for resources with all other activities. Testing a new release and ensuring its quality is of course fundamental, but can on the other hand not monopolize the observatory's resources or jeopardize its commitments to the scientific community.
The Atacama Large Millimeter /submillimeter Array (ALMA) has entered into operation phase since 2013. This transition changed the priorities within the observatory, in which, most of the available time will be dedicated to science observations at the expense of technical time. Therefore, it was planned to design and implement a new simulation environment, which must be comparable - or at least- be representative of the production environment. Concepts of model in the loop and hardware in the loop were explored. In this paper we review experiences gained and lessons learnt during the design and implementation of the new simulation environment.
The Atacama Large Millimeter /submillimeter Array (ALMA) will be a unique research instrument composed of at least 66 reconfigurable high-precision antennas, located at the Chajnantor plain in the Chilean Andes at an elevation of 5000 m. The observatory has another office located in Santiago of Chile, 1600 km from the Chajnantor plain. In the Atacama desert, the wonderful observing conditions imply precarious living conditions and extremely high operation costs: i.e: flight tickets, hospitality, infrastructure, water, electricity, etc. It is clear that a purely remote operational model is impossible, but we believe that a mixture of remote and local operation scheme would be beneficial to the observatory, not only in reducing the cost but also in increasing the observatory overall efficiency. This paper describes the challenges and experience gained in such experimental proof of the concept. The experiment was performed over the existing 100 Mbps bandwidth, which connects both sites through a third party telecommunication infrastructure. During the experiment, all of the existent capacities of the observing software were validated successfully, although room for improvement was clearly detected. Network virtualization, MPLS configuration, L2TPv3 tunneling, NFS adjustment, operational workstations design are part of the experiment.
KEYWORDS: Databases, Data acquisition, Antennas, Data storage, Control systems, Observatories, Data analysis, Data modeling, Computer architecture, Instrument modeling
The Atacama Large Millimeter /submillimeter Array (ALMA) will be a unique research instrument composed of at least
66 reconfigurable high-precision antennas, located at the Chajnantor plain in the Chilean Andes at an elevation of 5000
m. This paper describes the experience gained after several years working with the monitoring system, which has a
strong requirement of collecting and storing up to 150K variables with a highest sampling rate of 20.8 kHz. The original
design was built on top of a cluster of relational database server and network attached storage with fiber channel
interface. As the number of monitoring points increases with the number of antennas included in the array, the current
monitoring system demonstrated to be able to handle the increased data rate in the collection and storage area (only one
month of data), but the data query interface showed serious performance degradation. A solution based on no-SQL
platform was explored as an alternative to the current long-term storage system. Among several alternatives, mongoDB
has been selected. In the data flow, intermediate cache servers based on Redis were introduced to allow faster streaming
of the most recently acquired data to web based charts and applications for online data analysis.
The software for the Atacama Large Millimeter/submillimeter Array (ALMA) that has been developed in a collaboration of ESO, NRAO, NAOJ and the Joint ALMA Observatory for well over a decade is an integrated end-to-end software system of about six million lines of source code. As we enter the third cycle of science observations, we reflect on some of the decisions taken and call out ten topics where we could have taken a different approach at the time, or would take a different approach in today’s environment. We believe that these lessons learned should be helpful as the next generation of large telescope projects move into their construction phases.
After the inauguration of the Atacama Large Millimeter/submillimeter Array (ALMA), the Software Operations Group in Chile has refocused its objectives to: (1) providing software support to tasks related to System Integration, Scientific Commissioning and Verification, as well as Early Science observations; (2) testing the remaining software features, still under development by the Integrated Computing Team across the world; and (3) designing and developing processes to optimize and increase the level of automation of operational tasks. Due to their different stakeholders, each of these tasks presents a wide diversity of importances, lifespans and complexities. Aiming to provide the proper priority and traceability for every task without stressing our engineers, we introduced the Kanban methodology in our processes in order to balance the demand on the team against the throughput of the delivered work.
The aim of this paper is to share experiences gained during the implementation of Kanban in our processes, describing the difficulties we have found, solutions and adaptations that led us to our current but still evolving implementation, which has greatly improved our throughput, prioritization and problem traceability.
The ALMA Test Interferometer appeared as an infrastructure solution to increase both ALMA time availability for science activities and time availability for Software testing and Engineering activities at a reduced cost (<30000K USD) and a low setup time of less than 1 hour. The Test Interferometer could include up to 16 Antennas when used with only AOS resources and a possible maximum of 4 Antennas when configured using Correlator resources at OSF. A joined effort between ADC and ADE-IG took the challenge of generate the Test Interferometer from an already defined design for operations which imposed a lot of complex restrictions on how to implement it. Through and intensive design and evaluation work it was determined that is possible to make an initial implementation using the ACA Correlator and now it is also being tested the feasibility to implement the Testing Interferometer connecting the Test Array at AOS with Correlator equipment installed at the OSF, separated by 30 km. app. Lastly, efforts will be done to get interferometry between AOS and OSF Antennas with a baseline of approximately 24 km.
The Atacama Large Millimeter /submillimeter Array (ALMA) will be a unique research instrument composed of at least
66 reconfigurable high-precision antennas, located at the Chajnantor plain in the Chilean Andes at an elevation of 5000
m. Each antenna contains instruments capable of receiving radio signals from 31.3 GHz up to 950 GHz. These signals
are correlated inside a Correlator and the spectral data are finally saved into the Archive system together with the
observation metadata. This paper describes the progress in the development of the ALMA operation support software,
which aims to increase the efficiency of the testing, distribution, deployment and operation of the core observing
software. This infrastructure has become critical as the main array software evolves during the construction phase. In
order to support and maintain the core observing software, it is essential to have a mechanism to align and distribute the
same version of software packages across all systems. This is achieved rigorously with weekly based regression tests and
strict configuration control. A build farm to provide continuous integration and testing in simulation has been established
as well. Given the large amount of antennas, it is imperative to have also a monitoring system to allow trend analysis of
each component in order to trigger preventive maintenance activities. A challenge for which we are preparing this year
consists in testing the whole ALMA software performing complete end-to-end operation, from proposal submission to
data distribution to the ALMA Regional Centers. The experience gained during deployment, testing and operation
support will be presented.
KEYWORDS: Solar thermal energy, Antennas, Switches, Observatories, Chemical elements, Astronomy, Adaptive optics, Computer networks, Dynamical systems, Local area networks
ALMA is the first astronomical project being constructed and operated under industrial approach due to the huge amount
of elements involved. In order to achieve the maximum through put during the engineering and scientific commissioning
phase, several production lines have been established to work in parallel. This decision required modification in the
original system architecture in which all the elements are controlled and operated within a unique Standard Test
Environment (STE). The advance in the network industry and together with the maturity of virtualization paradigm
allows us to provide a solution which can replicate the STE infrastructure without changing their network address
definition. This is only possible with Virtual Routing and Forwarding (VRF) and Virtual LAN (VLAN) concepts. The
solution allows dynamic reconfiguration of antennas and other hardware across the production lines with minimum time
and zero human intervention in the cabling. We also push the virtualization even further, classical rack mount servers are
being replaced and consolidated by blade servers. On top of them virtualized server are centrally administrated with
VMWare ESX. Hardware costs and system administration effort will be reduced considerably. This mechanism has been
established and operated successfully during the last two years. This experience gave us confident to propose a solution
to divide the main operation array into subarrays using the same concept which will introduce huge flexibility and
efficiency for ALMA operation and eventually may simplify the complexity of ALMA core observing software since
there will be no need to deal with subarrays complexity at software level.
The main telescope of the UC Observatory Santa Martina is a 50cm optical telescope donated by ESO to Pontificia
Universidad Catolica de Chile. During the past years the telescope has been refurbished and used as the main facility for
testing and validating new instruments under construction by the center of Astro-Engineering UC. As part of this work,
the need to develop a more efficient and flexible control system arises. The new distributed control system has been
developed on top of Internet Communication Engine (ICE), a framework developed by Zeroc Inc. This framework
features a lightweight but powerful and flexible inter-process communication infrastructure and provides binding to
classic and modern programming languages, such as, C/C++, java, c#, ruby-rail, objective c, etc. The result of this work
shows ICE as a real alternative for CORBA and other de-facto distribute programming framework. Classical control
software architecture has been chosen and comprises an observation control system (OCS), the orchestrator of the
observation, which controls the telescope control system (TCS), and detector control system (DCS). The real-time
control and monitoring system is deployed and running over ARM based single board computers. Other features such as
logging and configuration services have been developed as well. Inter-operation with other main astronomical control
frameworks are foreseen in order achieve a smooth integration of instruments when they will be integrated in the main
observatories in the north of Chile
The ALMA software is a large collection of modules, which implements the functionality needed for the observatory day-to-day operations, including among others Array/Antenna Control, Correlator, Telescope Calibration
and Data Archiving. Many software patches must periodically be applied to fix problems detected during operations or to introduce enhancements after a release has been deployed and used under regular operational
conditions. Under this scenery, it has been imperative to establish, besides a strict conguration control system,
a weekly regression test to ensure that modications applied do not impact system stability and functionality.
A test suite has been developed for this purpose, which reflects the operations performed by the commissioning
and operations groups, and that aims to detect problems associated to the changes introduced at different versions
of ALMA software releases. This paper presents the evolution of the regression test suite, which started at the
ALMA Test Facility, and that has been adapted to be executed in the current operational conditions. Topics
about the selection of the tests to be executed, the validation of the obtained data and the automation of the
test suite are also presented.
KEYWORDS: Antennas, Software development, Observatories, Optical correlators, Astronomy, Software engineering, Prototyping, Information technology, Solar thermal energy, Control systems
Starting 2009, the ALMA project initiated one of its most exciting phases within construction: the first antenna
from one of the vendors was delivered to the Assembly, Integration and Verification team. With this milestone and
the closure of the ALMA Test Facility in New Mexico, the JAO Computing Group in Chile found itself in the front
line of the project's software deployment and integration effort. Among the group's main responsibilities are the
deployment, configuration and support of the observation systems, in addition to infrastructure administration,
all of which needs to be done in close coordination with the development groups in Europe, North America
and Japan. Software support has been the primary interaction key with the current users (mainly scientists,
operators and hardware engineers), as the software is normally the most visible part of the system.
During this first year of work with the production hardware, three consecutive software releases have been
deployed and commissioned. Also, the first three antennas have been moved to the Array Operations Site, at
5.000 meters elevation, and the complete end-to-end system has been successfully tested. This paper shares the
experience of this 15-people group as part of the construction team at the ALMA site, and working together
with Computing IPT, on the achievements and problems overcomed during this period. It explores the excellent
results of teamwork, and also some of the troubles that such a complex and geographically distributed project
can run into. Finally, it approaches the challenges still to come, with the transition to the ALMA operations
plan.
The Atacama Large Millimeter Array (ALMA) is a joint project between astronomical organizations in Europe, North
America, and Japan. ALMA will consist of at least 50 twelve meter antennas operating in the millimeter and submillimeter
wavelength range. It will be located at an altitude above 5000m in the Chilean Atacama desert. The ALMA
Test Facility (ATF), located in New Mexico, USA, is a proving ground for development and testing of hardware,
software, commissioning and operational procedure.
At the ATF emphasis has shifted from hardware testing to software and operational functionality. The support of the
varied goals of the ATF requires stable control software and at the same time flexibility for integrating newly developed
features. For this purpose regression testing has been introduced in the form of a semi-automated procedure. This
supplements the established offline testing and focuses on operational functionality as well as verifying that previously
fixed faults did not re-emerge.
The regression tests are carried out on a weekly basis as a compromise between the developers' response- and the
available technical time. The frequent feedback allows the validation of submitted fixes and the prompt detection of sideeffects
and reappearing issues. Results of nine months are presented that show the evolution of test outcomes, supporting
the conclusion that the regression testing helped to improve the speed of convergence towards stable releases at the ATF.
They also provided an opportunity to validate newly developed or re-factored software at an early stage at the test
facility, supporting its eventual integration. Hopefully this regression test procedure will be adapted to commissioning
operations in Chile.
HARPS is a new high resolution fibre-fed spectrograph dedicated to the
extremely precise measurement of stellar radial velocities. After being used for about one year including the commissioning runs we report a very successful implementation of the measures taken to maximise stability, efficiency and spectral performance. Using the Simultaneous ThAr Reference Method a short term precision of 0.2 m/s during one night and a long term precision of the order of 1 m/s have been achieved. Equipped with a fully automated data reduction pipeline that produces solar system barycentric radial velocities in near real-time, HARPS promises to deliver data of unequalled quality. HARPS will primarily be used for the search for exoplanets and in the field of asteroseismology. First exciting scientific results confirm these expectations.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.