Proceedings Article | 6 July 2018
KEYWORDS: Telescopes, Data storage, Databases, Systems engineering, Distributed computing, Control systems, Device simulation, Astronomical telescopes, Observatories, Computing systems
The Mayall 4-meter telescope will soon host the Dark Energy Spectroscopic Instrument (DESI). To integrate DESI into Mayall’s software ecosystem we built an engineering data system called MED (Mayall Engineering Database) that meets the following requirements. (1) MED consists of proven, open-source components to minimize its development and deployment efforts. (2) MED collects data from various sources (log-rotated flat files, existing SQL databases, and our inhouse messaging protocol) to provide a single source for communication between the DESI software and Mayall’s existing subsystems (TCS, dome, primary mirror, weather stations) or future ones (e.g. new calibration lamps). (3) MED facilitates monitoring and troubleshooting by providing an easy way to graph time series in real time or graph time series collected over the past year. (4) MED facilitates generating audio-visual alarms in the control room (e.g. when the dome vignettes) as well as automated email alerts (e.g. when the computer room temperature exceeds 30ºC). (5) MED facilitates semiautomated generation of end-of-night metrics (actual time spent on science, instrument overhead, time lost to technical problems or weather, etc.) that go into a night-log report. (6) MED has persistent, long-term storage for metrics that help with performance analysis and strategic planning. (7) The part of MED essential for science operations at night has builtin capability for high-availability. (8) Finally, for maximum flexibility MED is schema-less. We describe MED’s design and open-source technologies. Our requirements involve 3 timescales, each with its own database: SQL (long) is for metrics that we need to keep forever. InfluxDB (medium) is a time series database that stores variables for one year and, along with Grafana as its visualization front-end, is for monitoring and troubleshooting. Redis (short) is an in-memory data structure store that lacks history, as it keeps the latest value for each variable, and is for system operations, acting as the message broker between subsystems. We outline our Redis data structure and its high-resolution time stamp. We also describe MED’s implementation techniques and show how we tie the open-source components together with scripts and config files, and take advantage of Fluentd, another open-source tool, to convert and pipe telemetry from one source to another with ease
