Data Intensive Infrastructure

Abstract

Summary form only given, as follows. The complete presentation was not made available for publication as part of the conference proceedings. Modern research intensive organisations face challenges storing and preserving the increasing amounts of data generated by scientific instruments and high performance computers. Data must be delivered in a variety of modes depending on the end use, ranging from Web portals through to supercomputers. Building infrastructure to meet this need is complex and expensive. There is a need for mechanisms that support both managed and unmanaged data in a coherent and scalable way, often over a physically distributed multi-campus environment. In this talk I will discuss the ways we are delivering such infrastructure at the University of Queensland. Long term hierarchical storage, and many of the computing systems, are housed in a commercial Tier 3 data centre 20 kms from the main campus in St Lucia. Some high performance machines and desktops, and all scientific instruments, are housed on campus. University researchers work with local, national and international collaborators, requiring the need to share data securely and efficiently across a variety of scales. Our COTS based "MeDiCI data fabric" provides seamless access to data in such an environment. In order to improve standards of management, curation and preservation of data, a locally developed meta-data management service called RDM provides a single point of access for storage requests. Recent work on the CAMERA environment links unmanaged collections to managed repositories in a flexible and efficient manner. Finally, the fabric delivers data to a range of commodity and novel computing platforms such as the FlashLite data intensive cluster and the Wiener GPU supercomputer.