The NOAO Data Lab virtual storage system

Matthew J. Graham; Michael J. Fitzpatrick; Patrick Norris; Kenneth J. Mighell; Knut Olsen; Elizabeth B. Stobie; Stephen T. Ridgway; Adam S. Bolton; Abhijit Saha; Lijuan Wendy Huang

doi:10.1117/12.2233306

26 July 2016 The NOAO Data Lab virtual storage system

Matthew J. Graham, Michael J. Fitzpatrick, Patrick Norris, Kenneth J. Mighell, Knut Olsen, Elizabeth B. Stobie, Stephen T. Ridgway, Adam S. Bolton, Abhijit Saha, Lijuan Wendy Huang

Author Affiliations +

Proceedings Volume 9913, Software and Cyberinfrastructure for Astronomy IV; 99131I (2016) https://doi.org/10.1117/12.2233306
Event: SPIE Astronomical Telescopes + Instrumentation, 2016, Edinburgh, United Kingdom

Abstract

Collaborative research/computing environments are essential for working with the next generations of large astronomical data sets. A key component of them is a distributed storage system to enable data hosting, sharing, and publication. VOSpace¹ is a lightweight interface providing network access to arbitrary backend storage solutions and endorsed by the International Virtual Observatory Alliance (IVOA). Although similar APIs exist, such as Amazon S3, WebDav, and Dropbox, VOSpace is designed to be protocol agnostic, focusing on data control operations, and supports asynchronous and third-party data transfers, thereby minimizing unnecessary data transfers. It also allows arbitrary computations to be triggered as a result of a transfer operation: for example, a file can be automatically ingested into a database when put into an active directory or a data reduction task, such as Sextractor, can be run on it. In this paper, we shall describe the VOSpace implementations that we have developed for the NOAO Data Lab. These offer both dedicated remote storage, accessible as a local file system via FUSE, and a local VOSpace service to easily enable data synchronization.

Citation Download Citation

Matthew J. Graham, Michael J. Fitzpatrick, Patrick Norris, Kenneth J. Mighell, Knut Olsen, Elizabeth B. Stobie, Stephen T. Ridgway, Adam S. Bolton, Abhijit Saha, and Lijuan Wendy Huang "The NOAO Data Lab virtual storage system", Proc. SPIE 9913, Software and Cyberinfrastructure for Astronomy IV, 99131I (26 July 2016); https://doi.org/10.1117/12.2233306

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available