- Data management at these extreme scales is challenging and covers not only pre-processing,​ data production, and data analysis workflows. While there are many research approaches and science databases that aim to manage data and improved ​their limits over time, practitioners still struggle to manage their data in the Petabyte era. For instance, achieving high performance and providing means to easily localize data upon request. With billions of files, the scalability of the manual and fine-grained data management in HPC environment reaches its limitations. + Data management at these extreme scales is challenging and covers not only pre-processing,​ data production, and data analysis workflows. While there are many research approaches and science databases that aim to manage data and improve ​their limits over time, practitioners still struggle to manage their data in the Petabyte era. For instance, achieving high performance and providing means to easily localize data upon request. With billions of files, the scalability of the manual and fine-grained data management in HPC environment reaches its limitations.
Various domain-specific solutions have been developed that mitigate performance and management issues enabling data management in the Petabyte era. However, due to new storage technologies and heterogeneous environments,​ the challenges increase and so does the development effort for individual solutions. Various domain-specific solutions have been developed that mitigate performance and management issues enabling data management in the Petabyte era. However, due to new storage technologies and heterogeneous environments,​ the challenges increase and so does the development effort for individual solutions.
- * 14:30 **The Met Office Cold Storage Future: Tape or Cloud?** -- //Richard Lawrence// \\ //// + * 14:30 **The Met Office Cold Storage Future: Tape or Cloud?** -- //Richard Lawrence// \\ //The Met Office hosts one of the largest environmental science archives in the world using tape as the primary storage mechanism. The archive has over 275 petabytes of data stored today and a growth that is expected to exceed 5 exabytes during the next decade. When combined with data ingress and egress rates that exceed 200 terabytes a day each, the Met Office needs to ensure that the archive does not become the bottleneck to the production of our operational Weather forecasts and research needs. We will examine the current archive system, look at its current pain points and how the Met Office expects the needs of the archive to change in the short term. We then outline the UK government principle of ‘cloud first’ for digital designs and see if this can be applied to large scale Science IT. The talk will then progress onto how our current approach measures up to public cloud offerings looking at the capability, risk, benefits and costs of a cloud based archive.