Interactivity in High Performance Computing
Louis Jenkins, University of Rochester
Increasingly, data sets have required more and more compute and memory to process, which can only be met by larger and more powerful machines. This type of data, often referred to as ’Big Data’, requires enough memory and compute to hold and process that data. Traditionally, such large machines are reserved for scientific computing applications, referred to as traditional High Performance Computing (tHPC), in which such applications are optimized to leverage all available resources to complete their tasks as quickly as possible. Allocation of machine resources are monolithic in nature, utilizing batch processing systems that provide access to the requested compute and memory for a fixed length of time after waiting in a job queue. Data science applications require more interactive workflows, which typically consists of bursts of compute followed by indeterminate lengths of human interaction, referred to as think time. Such interactive jobs, referred to as interactive High Performance Computing (iHPC), such as Exploratory Data Analysis (EDA), have computations that may adapt to results obtained from previous computations, such as in the case of applications utilizing Jupyter Notebooks. However, batch processing provides the incentive to over-provision and underutilize resources given the lengthy queue times, which can range from hours to days, and data scientists require turn-around times that are ‘fast enough’ by human perspective, in the order of seconds to minutes. My work is focused on devising a model for such a system in which tHPC and iHPC applications can coexist on a HPC computing system. In such a system, the performance impact on tHPC applications must be minimal, and iHPC turn-around times must be ‘fast enough’. This can be achieved by utilization of existing techniques such as co-allocation, co-scheduling,
Abstract Author(s): Louis Jenkins, Sreepathi Pai, Michael Scott, Bradford Chamberlain