Access to parallel and distributed computation has enabled researchers and developers to improve algorithms and performance in many applications. Recent research has focused on next generation special purpose systems with multiple kinds of coprocessors, known as heterogeneous system-on-chips (SoC). In this paper, we introduce a method to intelligently schedule--and learn to schedule--a stream of tasks to available processing elements in such a system. We use deep reinforcement learning enabling complex sequential decision making and empirically show that our reinforcement learning system provides for a viable, better alternative to conventional scheduling heuristics with respect to minimizing execution time.
@cite_14: For problem solving, making reactive decisions based on problem description is fast but inaccurate, while search-based planning using heuristics gives better solutions but could be exponentially slow. In this paper, we propose a new approach that improves an existing solution by iteratively picking and rewriting its local components until convergence. The rewriting policy employs a neural network trained with reinforcement learning. We evaluate our approach in two domains: job scheduling and expression simplification. Compared to common effective heuristics, baseline deep models and search algorithms, our approach efficiently gives solutions with higher quality.
@cite_4: Minimizing job scheduling time is a fundamental issue in data center networks that has been extensively studied in recent years. The incoming jobs require different CPU and memory units, and span different number of time slots. The traditional solution is to design efficient heuristic algorithms with performance guarantee under certain assumptions. In this paper, we improve a recently proposed job scheduling algorithm using deep reinforcement learning and extend it to multiple server clusters. Our study reveals that deep reinforcement learning method has the potential to outperform traditional resource allocation algorithms in a variety of complicated environments.
@cite_9: The design of DNNs has increasingly focused on reducing the computational complexity in addition to improving accuracy. While emerging DNNs tend to have fewer weights and operations, they also reduce the amount of data reuse with more widely varying layer shapes and sizes. This leads to a diverse set of DNNs, ranging from large ones with high reuse (e.g., AlexNet) to compact ones with high bandwidth requirements (e.g., MobileNet). However, many existing DNN processors depend on certain DNN properties, e.g., a large number of channels, to achieve high performance and energy efficiency and do not have sufficient flexibility to efficiently process a diverse set of DNNs. In this work, we present Eyexam, a performance analysis framework that quantitatively identifies the sources of performance loss in DNN processors. It highlights two architectural bottlenecks in many existing designs. First, their dataflows are not flexible enough to adapt to the varying layer shapes and sizes of different DNNs. Second, their network-on-chip (NoC) can't adapt to support both high data reuse and high bandwidth scenarios. Based on this analysis, we present Eyeriss v2, a high-performance DNN accelerator that adapts to a wide range of DNNs. Eyeriss v2 has a new dataflow, called Row-Stationary Plus (RS+), that enables the spatial tiling of data from all dimensions to fully utilize the parallelism for high performance. To support RS+, it has a low-cost and scalable NoC design, called hierarchical mesh, that connects the high-bandwidth global buffer to the array of processing elements (PEs) in a two-level hierarchy. This enables high-bandwidth data delivery while still being able to harness any available data reuse. Compared with Eyeriss, Eyeriss v2 has a performance increase of 10.4x-17.9x for 256 PEs, 37.7x-71.5x for 1024 PEs, and 448.8x-1086.7x for 16384 PEs on DNNs with widely varying amounts of data reuse.
@cite_1: Heterogeneous multiprocessor system-on-chips (SoCs) provide a wide range of parameters that can be managed dynamically. For example, one can control the type (big little), number and frequency of active cores in state-of-the-art mobile processors at runtime. These runtime choices lead to more than 10× range in execution time, 5× range in power consumption, and 50× range in performance per watt. Therefore, it is crucial to make optimum power management decisions as a function of dynamically varying workloads at runtime. This paper presents a reinforcement learning approach for dynamically controlling the number and frequency of active big and little cores in mobile processors. We propose an efficient deep Q-learning methodology to optimize the performance per watt (PPW). Experiments using Odroid XU3 mobile platform show that the PPW achieved by the proposed approach is within 1 percent of the optimal value obtained by an oracle.
@cite_6: Resource management problems in systems and networking often manifest as difficult online decision making tasks where appropriate solutions depend on understanding the workload and environment. Inspired by recent advances in deep reinforcement learning for AI problems, we consider building systems that learn to manage resources directly from experience. We present DeepRM, an example solution that translates the problem of packing tasks with multiple resource demands into a learning problem. Our initial results show that DeepRM performs comparably to state-of-the-art heuristics, adapts to different conditions, converges quickly, and learns strategies that are sensible in hindsight.
@cite_15: Client-side video players employ adaptive bitrate (ABR) algorithms to optimize user quality of experience (QoE). Despite the abundance of recently proposed schemes, state-of-the-art ABR algorithms suffer from a key limitation: they use fixed control rules based on simplified or inaccurate models of the deployment environment. As a result, existing schemes inevitably fail to achieve optimal performance across a broad set of network conditions and QoE objectives. We propose Pensieve, a system that generates ABR algorithms using reinforcement learning (RL). Pensieve trains a neural network model that selects bitrates for future video chunks based on observations collected by client video players. Pensieve does not rely on pre-programmed models or assumptions about the environment. Instead, it learns to make ABR decisions solely through observations of the resulting performance of past decisions. As a result, Pensieve automatically learns ABR algorithms that adapt to a wide range of environments and QoE metrics. We compare Pensieve to state-of-the-art ABR algorithms using trace-driven and real world experiments spanning a wide variety of network conditions, QoE metrics, and video properties. In all considered scenarios, Pensieve outperforms the best state-of-the-art scheme, with improvements in average QoE of 12 --25 . Pensieve also generalizes well, outperforming existing schemes even on networks for which it was not explicitly trained.
@cite_16: Today's integrated circuit technologies allow the design of complete systems on a single chip which execute complex applications specified as a set of tasks. The tasks are managed by an Operating System whose main role consists in defining the resource allocation and the temporal scheduling. One of the main characteristics of these architectures is the heterogeneity of their execution resources which makes this scheduling complex. In this paper, we propose a neural network based model for the design of heterogeneous multiprocessor architectures scheduler. Previous works have shown that neural networks using the Hopfield model can be defined to schedule tasks on an homogeneous architecture. This approach was extended to take platform heterogeneity into account. The work presented in this paper is based on a new neural network structure using inhibitor neurons. These neurons allow to limit the number of additional neurons and the number of network re-initialisations to reach convergence. We compare our network to the classic solutions based on the Hopfield neural network. These comparisons show that the number of neurons is reduced by a factor of more than two, which reduces the time convergence. We also compare our approach in terms of number of migrations with the PFair algorithm which is known as an optimal solution in the context of homogeneous architecture. The results show that our solution significantly limit the number of task migrations. Finally, we present results in the context of heterogeneous multiprocessor architectures, which is representative of complex system-on-chip.
Resource management has been actively researched in many communities. Several works have been applying deep RL to optimally allocate resources, distribute tasks, and optimize power management decisions @cite_1 . DeepRM uses standard deep Q-learning algorithm to formalize resource management as a Tetris game, however, it only work with homogeneous settings and not consider task dependency @cite_6 . A variant of DeepRM leverages convolution neural networks as a backbone network to improve performance in scheduling @cite_4 . Subsequent work in DeepRM, Pensieve applies a resource managing algorithm to video streaming to optimally control the bitrate and successfully reduced buffering @cite_15 . Moreover, Hopfield neural network has been applied to design heterogeneous multiprocessor architecture scheduler @cite_16 . More recent work combines heuristic and learning algorithms, starting from an existing plan and iteratively improving it and successfully applying it in heterogeneous job scheduling task @cite_14 . However, their work follows the general MDP setting where, again, the agent chooses action at every timestep. From the perspective of hardware, recent work has proposed new accelerator architectures which have potential advances @cite_9 .