Title: An Orchestrated Empirical Study on Deep Learning Frameworks and Platforms,
Abstract: Deep learning (DL) has recently achieved tremendous success in a variety of
cutting-edge applications, e.g., image recognition, speech and natural language
processing, and autonomous driving. Besides the available big data and hardware
evolution, DL frameworks and platforms play a key role to catalyze the
research, development, and deployment of DL intelligent solutions. However, the
difference in computation paradigm, architecture design and implementation of
existing DL frameworks and platforms brings challenges for DL software
development, deployment, maintenance, and migration. Up to the present, it
still lacks a comprehensive study on how current diverse DL frameworks and
platforms influence the DL software development process.
In this paper, we initiate the first step towards the investigation on how
existing state-of-the-art DL frameworks (i.e., TensorFlow, Theano, and Torch)
and platforms (i.e., server/desktop, web, and mobile) support the DL software
development activities. We perform an in-depth and comparative evaluation on
metrics such as learning accuracy, DL model size, robustness, and performance,
on state-of-the-art DL frameworks across platforms using two popular datasets
MNIST and CIFAR-10. Our study reveals that existing DL frameworks still suffer
from compatibility issues, which becomes even more severe when it comes to
different platforms. We pinpoint the current challenges and opportunities
towards developing high quality and compatible DL systems. To ignite further
investigation along this direction to address urgent industrial demands of
intelligent solutions, we make all of our assembled feasible toolchain and
dataset publicly available.