Technology Integration

SSD Lifetime Analysis under Checkpointing and Memory Extension Workload

Future extreme-scale compute systems will take advantage of non-volatile memory technology for both memory extension and checkpointing. While SSD devices offer desirable properties such as lower cost ($/GB) and power consumption than DRAM, they suffer from poor random write performance and write endurance problems. Specifically the SSD lifetime can be highly influenced by the access pattern, the level of over-provisioning, and the choices of garbage collection reclaiming policy and wear-leveling techniques on the SSDs. In this project, we explore the SSD lifetime and performance issues for various scientific applications on extreme-scale systems that employ SSDs for both memory extension as well as checkpointing. The work involves several areas.

• Characterizing memory accesses and checkpoint I/O of HPC applications on the SSDs.
• SSD lifetime analysis for different scientific applications.
• Developing techniques to improve the SSD lifetime in both checkpoint and memory extension libraries.

Contributors: Youngjae Kim, Chao Wang, and Sudharshan Vazhkudai

  Top

NVMalloc

DRAM is a precious resource in extreme-scale machines and is increasingly becoming scarce, mainly due to the growing number of cores per node. On future multi-petaflop and exaflop machines, the memory pressure is likely to be so severe that we need to rethink our memory usage models. Fortunately, the advent of non-volatile memory (NVM) offers a unique opportunity in this space. Current NVM offerings possess several desirable properties, such as low cost and power efficiency, but also suffer from high latency and lifetime issues. We need rich techniques to be able to use them alongside DRAM. NVMalloc is an approach for exploiting NVM as a secondary memory partition so that applications can explicitly allocate and manipulate memory regions therein. More specifically, it is a middleware library with a suite of services that enables applications to access node-local or distributed NVM storage systems in a seamless fashion. The work inovolves several areas.

• Exploring NVM as a secondary memory partition for HPC systems
• Developing the NVMalloc library with a suite of services that enables applications to access NVM systems
• Studying the performance of NVMalloc with various HPC benchmark suites
• Developing data management techniques to efficiently use the NVM partition and DRAM partition

Contributors: Youngjae Kim, Chao Wang, and Sudharshan Vazhkudai

Status: On-going

Selected publications

• Chao Wang, Sudharshan Vazhkudai, Xiaosong Ma, Fei Meng, Youngjae Kim, Christian Engelmann, "NVMalloc: Exposing an Aggregate SSD Store as a Memory Partition in Extreme-Scale Machines", Proceedings of the 26th IEEE Int'l Parallel and Distributed Processing Symposium (IPDPS 2012), Shanghai, China, May 21-25, 2012. (118/569 = 20.7%) pdf

  Top

Active Processing

Active Flash

Modern scientific discovery is increasingly driven by large-scale supercomputing simulations, followed by data analysis tasks. These data analyses are either performed offline, on smaller-scale clusters, or on the supercomputer itself. Unfortunately, these techniques suffer from performance and energy inefficiencies due to increased data movement between the compute and storage subsystems. We propose Active Flash, an in-situ scientific data analysis approach, wherein data analysis is conducted on the solid-state device (SSD), where the data already resides. Active Flash has the potential to enable true out-of-core data analytics by freeing up both the compute core and the associated main memory. The work involves several areas.

• Developing analytical models for the performance and energy feasibility of Active Flash, and comparing those models against traditional in-situ and offline data analysis schemes
• Building an Active Flash prototype to demonstrate its viability
• Studying co-scheduling of analyses and internal flash management activities on the SSD controller
• Defining efficient representations of data object interfaces for Active Flash

Contributors: Sudharshan Vazhkudai and Youngjae Kim

Status: On-going

Selected publications

• Devesh Tiwari, Simona Bobila, Sudharshan Vazhkudai, Youngjae Kim, Xiaosong Ma, Peter Desnoyers, Yan Solin, "Active Flash: Towards Energy-Efficient, In-Situ Data Analytics on Extreme-Scale Machines,", (To appear) Proceedings of the USENIX Conference on File and Storage Technologies (FAST 2013), February, 2013. (24/127 = 18.9%)
• Simona Boboila, Youngjae Kim, Sudharshan Vazhkudai, Peter Desnoyers, Galen M. Shipman, "Active Flash: Out-of-core Data Analytics on Flash Storage", (Proceedings of the 28th IEEE Symposium on Massive Storage Systems and Technologies (MSST 2012), Monterey, CA, April 16-20, 2012. (14/57 = 24.5%)

AnalyzeThis

The need for novel data analysis is urgent in the face of a data deluge from modern applications. Traditional approaches to data analysis incur significant data movement costs, moving data back and forth between the storage system and the processor. Emerging Active Flash devices enable processing on the flash, where the data already resides. An array of such Active Flash devices allows us to revisit how analysis workflows interact with storage systems. By seamlessly blending together the flash storage and data analysis, we create an analysis workflow-aware storage system, AnalyzeThis. Our guiding principle is that analysis-awareness be deeply ingrained in each and every layer of the storage, elevating data analyses as first-class citizens, and transforming AnalyzeThis into a potent analytics-aware appliance. We implement the AnalyzeThis storage system atop an emulation platform of the Active Flash array. Our results indicate that AnalyzeThis is viable, expediting workflow execution and minimizing data movement.

TechInt contributors: Sudharshan Vazhkudai, Hyogi Sim

Selected publications:

• Hyogi Sim, Youngjae Kim, Sudharshan S. Vazhkudai, Devesh Tiwari, Ali Anwar, Ali R. Butt, Lavanya Ramakrishnan, "AnalyzeThis: An Analysis Workflow-Aware Storage System," Proceedings of Supercomputing 2015 (SC15): 28th Int'l Conference on High Performance Computing, Networking, Storage and Analysis, Austin, TX, November 2015. pdf

AnalyzeThat

AnalyzeThat is a programmable shared-memory system for parallel data processing with PIM (processing-in-memory) devices.

Through a rich PIM-Aware Data Structure (PADS), AnalyzeThat bundles data, analysis tasks, and the runtime needed to interface with the PIM device array. The PADS abstraction provides (i) a sophisticated key-value data container that allows programmers to easily store data on multiple PIMs, (ii) a suite of parallel operations with which users can easily implement data analysis applications, and (iii) an embedded runtime (i.e., hidden from the programmer), which provides the mechanisms needed to overlay both the data and the tasks on the PIM device array in an intelligent fashion, based on PIM-specific information collected from the hardware.

Contributors: Hyogi Sim, Sudharshan Vazhkudai

  Top

Emerging Storage Technologies

Hard disk drives (HDDs) have been the preferred media for data storage in high-performance computing. A center-wide file system at ORNL has deployed 13,440 HDDs for back-end storage systems, running the Lustre parallel file system. However, there are shortcomings inherent to HDDs. Alongside improvements in HDD technology, significant advances have also been made in various forms of solid-state memory such as NAND flash memory, phase-change memories (PRAM), and spin-transfer torque memories (STT-RAM). These emerging storage technologies close a huge gap between DRAM and disk based storage systems. In this project, we evaluate these emerging memory technologies, study how they are to coexist with magnetic disks, and address technology challenges towards employing these memory devices in HPC storage systems. More specifically we measure the technology's impact on I/O intensive scientific applications. The work includes several areas.

• Benchmarking emerging memory technologies and studying their performance and behavior
• Detailed performance modeling of these memory devices for different scientific applications
• Identifying technology challenges towards employing these devices in HPC storage systems
• Understanding internal memory management algorithms at the device controller

Contributors: Youngjae Kim, Sarp Oral, and Sudharshan Vazhkudai

Status: On-going

Selected publications:

• Youngjae Kim, Junghee Lee, Sarp Oral, David Dillow, Feiyi Wang, Galen M. Shipman, "Coordinating Garbage Collection for Arrays of Solid-state Drives", (To appear) IEEE Transactions on Computers (IEEE TC), 2013.
• Ramya Prabhakar, Sudharshan Vazhkudai, Youngjae Kim, Ali Butt, Min Li, Mahmut Kandemir, "Provisioning a Multi-Tiered Data Staging Area for Extreme-Scale Machines", Proceedings of the 31th Int'l Conference on Distributed Computing Systems (ICDCS 2011), Minneapolis, Minnesota, June 20-24, 2011. pdf

  Top