query: Natural Policy Gradient Methods with Parameter-based Exploration for Control Tasks

positive: Policy Gradient Methods for Robotics The acquisition and improvement of motor skills and control policies for robotics from trial and error is of essential importance if robots should ever leave precisely pre-structured environments. However, to date only few existing reinforcement learning methods have been scaled into the domains of high-dimensional robots such as manipulator, legged or humanoid robots. Policy gradient methods remain one of the few exceptions and have found a variety of applications. Nevertheless, the application of such methods is not without peril if done in an uninformed manner. In this paper, we give an overview on learning with policy gradient methods for robotics with a strong focus on recent advances in the field. We outline previous applications to robotics and show how the most recently developed methods can significantly improve learning performance. Finally, we evaluate our most promising algorithm in the application of hitting a baseball with an anthropomorphic arm

positive: PEGASUS: A Policy Search Method for Large MDPs and POMDPs We propose a new approach to the problem of searching a space of policies for a Markov decision process (MDP) or a partially observable Markov decision process (POMDP), given a model. Our approach is based on the following observation: Any (PO)MDP can be transformed into an “equivalent” POMDP in which all state transitions (given the current state and action) are deterministic. This reduces the general problem of policy search to one in which we need only consider POMDPs with deterministic transitions. We give a natural way of estimating the value of all policies in these transformed POMDPs. Policy search is then simply performed by searching for a policy with high estimated value. We also establish conditions under which our value estimates will be good, recovering theoretical results similar to those of Kearns, Mansour and Ng [7], but with “sample complexity” bounds that have only a polynomial rather than exponential dependence on the horizon time. Our method applies to arbitrary POMDPs, including ones with infinite state and action spaces. We also present empirical results for our approach on a small discrete problem, and on a complex continuous state/continuous action problem involving learning to ride a bicycle.

positive: Simulation-based optimization of Markov reward processes We propose a simulation based algorithm for optimizing the average reward in a Markov Reward Process that depends on a set of parameters As a special case the method applies to Markov Decision Processes where optimization takes place within a parametrized set of policies The algorithm involves the simulation of a single sample path and can be implemented on line A convergence result with probability is provided This research was supported by contracts with Siemens AG Munich Germany and Alcatel Bell Belgium and by contract DMI with the National Science Foundation Introduction Markov Decision Processes and the associated dynamic programming DP methodology Ber a Put provide a general framework for posing and analyzing problems of se quential decision making under uncertainty DP methods rely on a suitably de ned value function that has to be computed for every state in the state space However many inter esting problems involve very large state spaces curse of dimensionality In addition DP assumes the availability of an exact model in the form of transition probabilities In many practical situations such a model is not available and one must resort to simulation or experimentation with an actual system For all of these reasons dynamic programming in its pure form may be inapplicable The e orts to overcome the aforementioned di culties involve two main ideas The use of simulation to estimate quantities of interest thus avoiding model based computations The use of parametric representations to overcome the curse of dimensionality Parametric representations and the associated algorithms can be broadly classi ed into three main categories a Parametrized value functions Instead of associating a value V i with each state i one uses a parametric form V i r where r is a vector of tunable parameters weights and V is a so called approximation architecture For example V i r could be the output of a multilayer perceptron with weights r when the input is i Other representations are possible e g involving polynomials linear combina tions of feature vectors state aggregation etc When the main ideas from DP are combined with such parametric representations one obtains methods that go un der the names of reinforcement learning or neuro dynamic programming see BT SB for textbook expositions as well as the references therein A key char acteristic is that policy optimization is carried out in an indirect fashion one tries to obtain a good approximation of the optimal value function of dynamic programming and uses it to construct policies that are close to optimal Such methods are reason ably well though not fully understood and there have been some notable practical successes see BT SB for an overview including the world class backgammon player by Tesauro Tes b Parametrized policies In an alternative approach which is the one considered in this paper the tuning of a parametrized value function is bypassed Instead one considers a class of policies described in terms of a parameter vector Simulation is employed to estimate the gradient of the performance metric with respect to and the policy is improved by updating in a gradient direction In some cases the re quired gradient can be estimated using IPA in nitesimal perturbation analysis see e g HC Gla CR and the references therein For general Markov processes and in the absence of special structure IPA is inapplicable but gradient estimation is still possible using likelihood ratio methods Gly Gly GG LEc GI c Actor critic methods A third approach which is a combination of the rst two includes parametrizations of the policy actor and of the value function critic BSA While such methods seem particularly promising theoretical understand ing has been limited to the impractical case of lookup representations one parameter per state KB This paper concentrates on methods based on policy parametrization and approx imate gradient improvement in the spirit of item b above While we are primarily interested in the case of Markov Decision Processes almost everything applies to the more general case of Markov Reward Processes that depend on a parameter vector and we proceed within this broader context We start with a formula for the gradient of the performance metric that has been presented in di erent forms and for various contexts in Gly CC FH JSJ TH CW We then suggest a method for estimating the terms that appear in that formula This leads to a simulation based method that updates the parameter vector at every regeneration time in an approximate gradient direction Furthermore we show how to construct an on line method that updates the parameter vector at each time step The resulting method has some conceptual similarities with those described in CR that reference assumes however the availability of an IPA estimator with certain guaranteed properties that are absent in our context and in JSJ which however does not contain convergence results The method that we propose only keeps in memory and updates K numbers where K is the dimension of Other than itself this includes a vector similar to the eligibility trace in Sutton s temporal di erence methods and as in JSJ an estimate of the average reward under the current value of If that estimate was accurate our method would be a standard stochastic gradient algorithm However as keeps changing is generally a biased estimate of the true average reward and the mathematical structure of our method is more complex than that of stochastic gradient algorithms For reasons that will become clearer later standard approaches e g martingale arguments or the ODE approach do not seem to su ce for establishing convergence and a more elaborate proof is necessary Our gradient estimator can also be derived or interpreted in terms of likelihood ratios Gly GG It takes the same form as the one presented in p of Gly but it is used di erently The development in Gly leads to a consistent estimator of the gradient assuming that a very large number of regenerative cycles are estimated while keeping the policy parameter at a xed value Presumably would be then updated after such a long simulation In contrast our method updates much more frequently and retains the desired convergence properties despite the fact that any single cycle results in a biased gradient estimate An alternative simulation based stochastic gradient method again based on a likeli hood ratio formula has been provided in Gly and uses the simulation of two regen erative cycles to construct an unbiased estimate of the gradient We note some of the di erences with the latter work First the methods in Gly involve a larger number of auxiliary quantities that are propagated in the course of a regenerative cycle Second our method admits a modi cation see Sections that can make it applicable even if the time until the next regeneration is excessive in which case likelihood ratio based methods su er from excessive variance Third our estimate  of the average reward is obtained as a weighted average of all past rewards not just over the last regenerative cycle In contrast an approach such as the one in Gly would construct an independent estimate of  during each regenerative cycle which should result in higher variance Finally our method brings forth and makes crucial use of the value di erential reward function of dy namic programming This is important because it paves the way for actor critic methods in which the variance associated with the estimates of the di erential rewards is poten tially reduced by means of learning value function approximation Indeed subsequent to the rst writing of this paper this latter approach has been pursued in KT SMS In summary the main contributions of this paper are as follows We introduce a new algorithm for updating the parameters of a Markov Reward Process on the basis of a single sample path The parameter updates can take place either during visits to a certain recurrent state or at every time step We also specialize the method to Markov Decision Processes with parametrically represented policies In this case the method does not require the transition probabilities to be known We establish that the gradient with respect to the parameter vector of the perfor mance metric converges to zero with probability which is the strongest possible result for gradient related stochastic approximation algorithms The method admits approximate variants with reduced variance such as the one described in Section or various types of actor critic methods The remainder of this paper is organized as follows In Section we introduce our framework and assumptions and state some background results including a formula for the gradient of the performance metric In Section we present an algorithm that per forms updates during visits to a certain recurrent state present our main convergence result and provide a heuristic argument Sections and deal with variants of the algo rithm that perform updates at every time step In Section we specialize our methods to the case of Markov Decision Processes that are optimized within a possibly restricted set of parametrically represented randomized policies We present some numerical results in Section and conclude in Section The lengthy proof of our main results is developed in the appendices Markov Reward Processes Depending on a Parameter In this section we present our general framework make a few assumptions and state some basic results that will be needed later We consider a discrete time nite state Markov chain fing with state space S f Ng whose transition probabilities depend on a parameter vector K and are denoted by pij P in j j in i Whenever the state is equal to i we receive a one stage reward that also depends on and is denoted by gi For every K let P be the stochastic matrix with entries pij Let P fP j Kg be the set of all such matrices and let P be its closure Note that every element of P is also a stochastic matrix and therefore de nes a Markov chain on the same s

positive: Using EM for Reinforcement Learning We discsus Hinton’s (1989) relative payoff procedure (RPP), a static reinforcement learning algorithm whose foundation is not stochastic gradient ascent. We show circumstances under which applying the RPP is guaranteed to increase the mean return, even though it can make large changes in the values of the parameters. The proof is based on a mapping between the RPP and a form of the expectation-maximisation procedure of Dempster, Laird & Rubin (1976).

negative: Learning to update Auto-associative Memory in Recurrent Neural Networks for Improving Sequence Memorization Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain memory differentiability, or by discounting the RNN representation learning towards encoding shorter local contexts than encouraging long sequence encoding. Associative memory, which studies the compression of multiple patterns in a fixed size memory, were rarely considered in recent years. Although some recent work tries to introduce associative memory in RNN and mimic the energy decay process in Hopfield nets, it inherits the shortcoming of rule-based memory updates, and the memory capacity is limited. This paper proposes a method to learn the memory update rule jointly with task objective to improve memory capacity for remembering long sequences. Also, we propose an architecture that uses multiple such associative memory for more complex input encoding. We observed some interesting facts when compared to other RNN architectures on some well-studied sequence learning tasks.

negative: Single-Channel and Multi-Channel Sinusoidal Audio Coding Using Compressed Sensing Compressed sensing (CS) samples signals at a much lower rate than the Nyquist rate if they are sparse in some basis. In this paper, the CS methodology is applied to sinusoidally modeled audio signals. As this model is sparse by definition in the frequency domain (being equal to the sum of a small number of sinusoids), we investigate whether CS can be used to encode audio signals at low bitrates. In contrast to encoding the sinusoidal parameters (amplitude, frequency, phase) as current state-of-the-art methods do, we propose encoding few randomly selected samples of the time-domain description of the sinusoidal component (per signal segment). The potential of applying compressed sensing both to single-channel and multi-channel audio coding is examined. The listening test results are encouraging, indicating that the proposed approach can achieve comparable performance to that of state-of-the-art methods. Given that CS can lead to novel coding systems where the sampling and compression operations are combined into one low-complexity step, the proposed methodology can be considered as an important step towards applying the CS framework to audio coding applications.

negative: Image Retargeting Quality Assessment: A Study of Subjective Scores and Objective Metrics This paper presents the result of a recent large-scale subjective study of image retargeting quality on a collection of images generated by several representative image retargeting methods. Owning to many approaches to image retargeting that have been developed, there is a need for a diverse independent public database of the retargeted images and the corresponding subjective scores to be freely available. We build an image retargeting quality database, in which 171 retargeted images (obtained from 57 natural source images of different contents) were created by several representative image retargeting methods. And the perceptual quality of each image is subjectively rated by at least 30 viewers, meanwhile the mean opinion scores (MOS) were obtained. It is revealed that the subject viewers have arrived at a reasonable agreement on the perceptual quality of the retargeted image. Therefore, the MOS values obtained can be regarded as the ground truth for evaluating the quality metric performances. The database is made publicly available (Image Retargeting Subjective Database, [Online]. Available: http://ivp.ee.cuhk.edu.hk/projects/demo/retargeting/index.html) to the research community in order to further research on the perceptual quality assessment of the retargeted images. Moreover, the built image retargeting database is analyzed from the perspectives of the retargeting scale, the retargeting method, and the source image content. We discuss how to retarget the images according to the scale requirement and the source image attribute information. Furthermore, several publicly available quality metrics for the retargeted images are evaluated on the built database. How to develop an effective quality metric for retargeted images is discussed through a specifically designed subjective testing process. It is demonstrated that the metric performance can be further improved, by fusing the descriptors of shape distortion and content information loss.

negative: The ecology of human performance: a framework for considering the effect of context. In theory and in practice, context (as an area of concern to occupational therapists) has not received the same attention as performance components and performance areas. The Ecology of Human Performance serves as a framework for considering the effect of context. Context is described as a lens from which persons view their world. The interrelationship of person and context determines which tasks fall within the person's performance range. The Ecology of Human Performance framework provides guidelines for encompassing context in occupational therapy theory, practice, and research.

negative: Particle swarm optimization for hyper-parameter selection in deep neural networks Deep neural networks (DNNs) have achieved unprecedented success in a wide array of tasks. However, the performance of these systems depends directly on their hyper-parameters which often must be selected by an expert. Optimizing the hyper-parameters remains a substantial obstacle in designing DNNs in practice. In this work, we propose to select them using particle swarm optimization (PSO). Such biologically-inspired approaches have not been extensively exploited for this task. We demonstrate that PSO efficiently explores the solution space, allowing DNNs of a minimal topology to obtain competitive classification performance over the MNIST dataset. We showed that very small DNNs optimized by PSO retrieve promising classification accuracy for CIFAR-10. Also, PSO improves the performance of existing architectures. Extensive experimental study, backed-up with the statistical tests, revealed that PSO is an effective technique for automating hyper-parameter selection and efficiently exploits computational resources.

negative: Learning user-specific parameters in a multibiometric system Biometric systems that use a single biometric trait have to contend with noisy data, restricted degrees of freedom, failureto-enroll problems, spoof attacks, and unacceptable error rates. Multibiometric systems that use multiple traits of an individual for authentication, alleviate some of these problems while improving verification performance. We demonstrate that the performance of multibiometric systems can be further improved by learning user-specific parameters. Two types of parameters are considered here. (i) Thresholds that are used to decide if a matching score indicates a genuine user or an impostor, and (ii) weights that are used to indicate the importance of matching scores output by each biometric trait. User-specific thresholds are computed using the cumulative histogram of impostor matching scores corresponding to each user. The user-specific weights associated with each biometric are estimated by searching for that set of weights which minimizes the total verification error. The tests were conducted on a database of 50 users who provided fingerprint, face and hand geometry data, with 10 of these users providing data over a period of two months. We observed that user-specific thresholds improved system performance by ∼ 2%, while user-specific weights improved performance by ∼ 3%.

negative: What we know about leadership. Effectiveness and personality. Although psychologists know a great deal about leadership, persons who make decisions about real leaders seem largely to ignore their accumulated wisdom. In an effort to make past research more accessible, interpretable, and relevant to decision makers, this article defines leadership and then answers nine questions that routinely come up when practical decisions are made about leadership (e.g., whom to appoint, how to evaluate them, when to terminate them.

negative: Effects of level of meditation experience on attentional focus: is the efficiency of executive or orientation networks improved? The present investigation examined the contributions of specific attentional networks to long-term trait effects of meditation. It was hypothesized that meditation could improve the efficiency of executive processing (inhibits prepotent/incorrect responses) or orientational processing (orients to specific objects in the attentional field). Participants (50 meditators and 10 controls) were given the Stroop (measures executive attention) and Global-Local Letters (measures orientational attention) tasks. Results showed that meditation experience was associated with reduced interference on the Stroop task (p < 0.03), in contrast with a lack of effect on interference in the Global-Local Letters task. This suggests that meditation produces long-term increases in the efficiency of the executive attentional network (anterior cingulate/prefrontal cortex) but no effect on the orientation network (parietal systems). The amount of time participants spent meditating each day, rather than the total number of hours of meditative practice over their lifetime, was negatively correlated with interference on the Stroop task (r = -0.31, p < 0.005).

negative: Sarcasm detection of tweets: A comparative study Sarcasm is a nuanced form of communication where the individual states opposite of what is implied. One of the major challenges of sarcasm detection is its ambiguous nature. There is no prescribed definition of sarcasm. Another major challenge is the growing size of the languages. Every day hundreds of new slang words are being created and used on these sites. Hence, the existing corpus of positive and negative sentiments may not prove to be accurate in detecting sarcasm. Also, the recent developments in online social networks allow its users to use varied kind of emoticons with the text. These emoticons may change the polarity of the text and make it sarcastic. Due to these difficulties and the inherently tricky nature of sarcasm it is generally ignored during social network analysis. As a result the results of such analysis are affected adversely. Thus, sarcasm detection poses to be one of the most critical problems which we need to overcome. Detection of sarcastic content is vital to various NLP based systems such as text summarization and sentiment analysis. In this paper we address the problem of sarcasm detection by leveraging the most common expression of sarcasm — “positive sentiment attached to a negative situation”. Our work uses two ensemble based approaches — voted ensemble classifier and random forest classifier. Unlike current approaches to sarcasm detection which rely on existing corpus of positive and negative sentiments for training the classifiers, we use a seeding algorithm to generate training corpus. The proposed model also uses a pragmatic classifier to detect emoticon based sarcasm.

negative: Components for parametric urban design in Grasshopper from street network to building geometry The main contribution of our work is in combining the methods for parametric urban design of highly specialized software such as CityEngine and general-purpose parametric modeling platform such as Grasshopper. Our work facilitates and prompts the use of parametric tools by architects and planners for urban design. In this paper we present a custom grasshopper component for street network generation and block subdivision. The component was developed in C# using the RhinoCommon SDK. We used Grasshopper for the development of an urban design proposal at a teaching exercise. To meet the requirements of the urban design project, additional functionalities had to be added to the range of existing Grasshopper components. In particular, we needed components for street network generation and block subdivision. To develop the component we implemented the street expansion strategies described in (Weber et al., 2009) and the methods for block subdivision described in (Vanegas et al., 2009). Additionally, we adapted and enhanced the strategies to meet the NURBS modeling capabilities of Rhinoceros.

negative: Elicitation of Security requirements for E-Health system by applying Model Oriented Security Requirements Engineering (MOSRE) Framework E-health is a health care system which is supported by electronic process and communication. The information that is kept in the system must be accurate. In case of false information, it may cause harm to human life. So this system needs more security to protect the credential information. E-Health system is the most security sensitive process handled electronically. The highest achievable security is never too much for an E-Health system. So when system is being built, tasks such as Security Requirements Elicitation, Specification and Validation are essential to assure the Quality of the resulting secure E-Health system. By considering the Security requirements as functional requirements in the Requirement phase, the completeness of Security Requirements for E-Health system can be developed. In this paper we propose Model Oriented Security Requirements Engineering (MOSRE) Framework in the early phases of E-Health system development, to identify assets, threats and vulnerabilities. This helps in standardizing the Security Requirements for secure E-Health system without any security issues.

negative:
Delta-9-tetrahydrocannabinol may palliate altered chemosensory perception in cancer patients: results of a randomized, double-blind, placebo-controlled pilot trial. BACKGROUND
A pilot study (NCT00316563) to determine if delta-9-tetrahydrocannabinol (THC) can improve taste and smell (chemosensory) perception as well as appetite, caloric intake, and quality of life (QOL) for cancer patients with chemosensory alterations.

PATIENTS AND METHODS
Adult advanced cancer patients, with poor appetite and chemosensory alterations, were recruited from two sites and randomized in a double-blinded manner to receive either THC (2.5 mg, Marinol(®); Solvay Pharma Inc., n = 24) or placebo oral capsules (n = 22) twice daily for 18 days. Twenty-one patients completed the trial. At baseline and posttreatment, patients completed a panel of patient-reported outcomes: Taste and Smell Survey, 3-day food record, appetite and macronutrient preference assessments, QOL questionnaire, and an interview.

RESULTS
THC and placebo groups were comparable at baseline. Compared with placebo, THC-treated patients reported improved (P = 0.026) and enhanced (P < 0.001) chemosensory perception and food 'tasted better' (P = 0.04). Premeal appetite (P = 0.05) and proportion of calories consumed as protein increased compared with placebo (P = 0.008). THC-treated patients reported increased quality of sleep (P = 0.025) and relaxation (P = 0.045). QOL scores and total caloric intake were improved in both THC and placebo groups.

CONCLUSIONS
THC may be useful in the palliation of chemosensory alterations and to improve food enjoyment for cancer patients.

negative: Completely Derandomized Self-Adaptation in Evolution Strategies This paper puts forward two useful methods for self-adaptation of the mutation distribution - the concepts of derandomization and cumulation. Principle shortcomings of the concept of mutative strategy parameter control and two levels of derandomization are reviewed. Basic demands on the self-adaptation of arbitrary (normal) mutation distributions are developed. Applying arbitrary, normal mutation distributions is equiv-alent to applying a general, linear problem encoding. The underlying objective of mutative strategy parameter control is roughly to favor previously selected mutation steps in the future. If this objective is pursued rigor-ously, a completely derandomized self-adaptation scheme results, which adapts arbitrary normal mutation distributions. This scheme, called covariance matrix adaptation (CMA), meets the previously stated demands. It can still be considerably improved by cumulation - utilizing an evolution path rather than single search steps. Simulations on various test functions reveal local and global search properties of the evolution strategy with and without covariance matrix adaptation. Their performances are comparable only on perfectly scaled functions. On badly scaled, non-separable functions usually a speed up factor of several orders of magnitude is ob-served. On moderately mis-scaled functions a speed up factor of three to ten can be expected.

negative: Handbook of Multibiometrics (International Series on Biometrics) Will reading habit influence your life? Many say yes. Reading handbook of multibiometrics 6 international series on biometrics is a good habit; you can develop this habit to be such interesting way. Yeah, reading habit will not only make you have any favourite activity. It will be one of guidance of your life. When reading has become a habit, you will not make it as disturbing activities or as boring activity. You can gain many benefits and importances of reading.

negative: Photonic ADC: overcoming the bottleneck of electronic jitter. Accurate conversion of wideband multi-GHz analog signals into the digital domain has long been a target of analog-to-digital converter (ADC) developers, driven by applications in radar systems, software radio, medical imaging, and communication systems. Aperture jitter has been a major bottleneck on the way towards higher speeds and better accuracy. Photonic ADCs, which perform sampling using ultra-stable optical pulse trains generated by mode-locked lasers, have been investigated for many years as a promising approach to overcome the jitter problem and bring ADC performance to new levels. This work demonstrates that the photonic approach can deliver on its promise by digitizing a 41 GHz signal with 7.0 effective bits using a photonic ADC built from discrete components. This accuracy corresponds to a timing jitter of 15 fs - a 4-5 times improvement over the performance of the best electronic ADCs which exist today. On the way towards an integrated photonic ADC, a silicon photonic chip with core photonic components was fabricated and used to digitize a 10 GHz signal with 3.5 effective bits. In these experiments, two wavelength channels were implemented, providing the overall sampling rate of 2.1 GSa/s. To show that photonic ADCs with larger channel counts are possible, a dual 20-channel silicon filter bank has been demonstrated.

negative: Full-body illusions and minimal phenomenal selfhood We highlight the latest research on body perception and self-consciousness, but argue that despite these achievements, central aspects have remained unexplored, namely, global aspects of bodily self-consciousness. Researchers investigated central representations of body parts and actions involving these, but neglected the global and unitary character of self-consciousness, the 'I' of experience and behaviour. We ask, what are the minimally sufficient conditions for the appearance of a phenomenal self, that is, the fundamental conscious experience of being someone? What are necessary conditions for self-consciousness in any type of system? We offer conceptual clarifications, discuss recent empirical evidence from neurology and cognitive science and argue that these findings offer a new entry point for the systematic study of global and more fundamental aspects of self-consciousness.

negative: A Fuzzy-based approach to programming language independent source-code plagiarism detection Source-code plagiarism detection in programming, concerns the identification of source-code files that contain similar and/or identical source-code fragments. Fuzzy clustering approaches are a suitable solution to detecting source-code plagiarism due to their capability to capture the qualitative and semantic elements of similarity. This paper proposes a novel Fuzzy-based approach to source-code plagiarism detection, based on Fuzzy C-Means and the Adaptive-Neuro Fuzzy Inference System (ANFIS). In addition, performance of the proposed approach is compared to the Self- Organising Map (SOM) and the state-of-the-art plagiarism detection Running Karp-Rabin Greedy-String-Tiling (RKR-GST) algorithms. The advantages of the proposed approach are that it is programming language independent, and hence there is no need to develop any parsers or compilers in order for the fuzzy-based predictor to provide detection in different programming languages. The results demonstrate that the performance of the proposed fuzzy-based approach overcomes all other approaches on well-known source code datasets, and reveals promising results as an efficient and reliable approach to source-code plagiarism detection.

negative: Dynamic Online Pricing with Incomplete Information Consider the pricing decision for a manager at large online retailer, such as Amazon.com, that sells millions of products. The pricing manager must decide on real-time prices for each of these product. Due to the large number of products, the manager must set retail prices without complete demand information. A manager can run price experiments to learn about demand and maximize long run profits. There are two aspects that make the online retail pricing different from traditional brick and mortar settings. First, due to the number of products the manager must be able to automate pricing. Second, an online retailer can make frequent price changes. Pricing differs from other areas of online marketing where experimentation is common, such as online advertising or website design, as firms do not randomize prices to different customers at the same time. In this paper we propose a dynamic price experimentation policy where the firm has incomplete demand information. For this general setting, we derive a pricing algorithm that balances earning profit immediately and learning for future profits. The proposed approach marries statistical machine learning and economic theory. In particular we combine multi-armed bandit (MAB) algorithms with partial identification of consumer demand into a unique pricing policy. Our automated policy solves this problem using a scalable distribution-free algorithm. We show that our method converges to the optimal price faster than standard machine learning MAB solutions to the problem. In a series of Monte Carlo simulations, we show that the proposed approach perform favorably compared to methods in computer science and revenue management. ∗Ross School of Business, University of Michigan. Contact: ericmsch@umich.edu †Rady School of Management, University of California, San Diego. Contact: kamisra@ucsd.edu ‡Department of Electrical Engineering and Computer Science, University of Michigan. Contact: jabernet@umich.edu

negative: Unsupervised local deep feature for image recognition Unsupervised feature learning is an important problem in computer vision and machine learning. Various unsupervised learning methods, e.g., PCA, KMeans, autoencoder, have been applied for visual recognition tasks. Especially, autoencoder has superior image recognition performance, which is usually used as a global feature extractor. To make better use of autoencoder for image recognition, we propose unsupervised local deep feature (ULDF). We collect patches from training images and use autoencoder to transform vectorized patches into short codes. Then these codes are encoded using Fisher Vector with spatial pyramid. Image recognition with ULDF only requires an efficient linear SVM classifier. Finally, we perform multi-scale voting to achieve more accurate classification performance. In the experiments, we evaluate ULDF on the handwritten digit recognition task and the shape classification task, and show that it achieves highly competitive performance comparing to the state-of-the-art image recognition methods. The results suggest that the proposed local deep feature is superior than traditional global autoencoder. © 2016 Elsevier Inc. All rights reserved.

negative: Transanal endoscopic total mesorectal excision: technical aspects of approaching the mesorectal plane from below—a preliminary report Laparoscopic total mesorectal excision (TME) for low rectal cancer can be technically challenging. This report describes our initial experience with a hybrid laparoscopic and transanal endoscopic technique for TME in low rectal cancer. Between December 2012 and October 2013, we identified patients with rectal cancer < 5 cm from the anorectal junction (ARJ) who underwent laparoscopic-assisted TME with a transanal minimally invasive surgery (TAMIS) technique. A standardized stepwise approach was used in all patients. Resection specimens were examined for completeness and measurement of margins. Preoperative magnetic resonance imaging (MRI) characteristics and short-term postoperative outcomes were examined. All values are mean ± standard deviation. Ten patients (8 males; median age: 60.5 (range 36–70) years) were included. On initial MRI, all tumors were T2 or T3, mean tumor height from the ARJ was 28.9 ± 12.2 mm, mean circumferential resection margin was 5.3 ± 3.1 mm , and the mean angle between the anal canal and the levator ani was 83.9° ± 9.7°. All patients had had preoperative chemoradiotherapy, TME via TAMIS, and distal anastomosis. There were no intraoperative complications, anastomotic leaks, or 30-day mortality. The pathologic quality of all mesorectal specimens was excellent. The distal resection margin was 19.4 ± 10.4 mm, the mean circumferential resection margin was 13.8 ± 5.1 mm, and the median lymph node harvest was 10.5 (range 5–15) nodes. A combined laparoscopic and transanal approach can achieve a safe and oncologically complete TME dissection for low rectal tumors. This approach may improve clinical outcomes in these technically difficult cases, but larger prospective studies are needed.

negative: TreeSegNet: Adaptive Tree CNNs for Subdecimeter Aerial Image Segmentation. For the task of subdecimeter aerial imagery segmentation, fine-grained semantic segmentation results are usually difficult to obtain because of complex remote sensing content and optical conditions. Recently, convolutional neural networks (CNNs) have shown outstanding performance on this task. Although many deep neural network structures and techniques have been applied to improve the accuracy, few have paid attention to better differentiating the easily confused classes. In this paper, we propose TreeSegNet which adopts an adaptive network to increase the classification rate at the pixelwise level. Specifically, based on the infrastructure of DeepUNet, a Tree-CNN block in which each node represents a ResNeXt unit is constructed adaptively according to the confusion matrix and the proposed TreeCutting algorithm. By transporting feature maps through concatenating connections, the Tree-CNN block fuses multiscale features and learns best weights for the model. In experiments on the ISPRS 2D semantic labeling Potsdam dataset, the results obtained by TreeSegNet are better than those of other  Corresponding author. Tel.: +86 18911938368. E-mail address: ilydouble@gmail.com TreeSegNet: Adaptive Tree CNNs for Subdecimeter Aerial Image Segmentation 2 published state-of-the-art methods. Detailed comparison and analysis show that the improvement brought by the adaptive Tree-CNN block is significant.

negative: A sub-µW, 1.0V CMOS temperature sensor circuit insensitive to device parameters In this paper, we firstly propose the CMOS temperature sensor circuit operating in the subthreshold region. The circuit has advantages of 1.0 V operation and sub-µW power consumption. Furthermore, it is insensitive to the device parameter of the fabrication process. Next, we design the high sensitivity temperature sensor circuit that can be realized by cascade-connection of the proposed temperature sensor circuits. The proposed circuits were evaluated through HSPICE with a standard 0.35 µm CMOS device parameter. The simulation demonstrate the maximum power consumption is 0.27 µW under the condition that the temperature range is from 0 °C to 100 °C, the sensitivity is 1.01 mV/ °C and the maximum differential error is 0.87 %.

negative: Fresh apps: an empirical study of frequently-updated mobile apps in the Google play store Mobile app stores provide a unique platform for developers to rapidly deploy new updates of their apps. We studied the frequency of updates of 10,713 mobile apps (the top free 400 apps at the start of 2014 in each of the 30 categories in the Google Play store). We find that a small subset of these apps (98 apps representing ˜1 % of the studied apps) are updated at a very frequent rate — more than one update per week and 14 % of the studied apps are updated on a bi-weekly basis (or more frequently). We observed that 45 % of the frequently-updated apps do not provide the users with any information about the rationale for the new updates and updates exhibit a median growth in size of 6 %. This paper provides information regarding the update strategies employed by the top mobile apps. The results of our study show that 1) developers should not shy away from updating their apps very frequently, however the frequency varies across store categories. 2) Developers do not need to be too concerned about detailing the content of new updates. It appears that users are not too concerned about such information. 3) Users highly rank frequently-updated apps instead of being annoyed about the high update frequency.

negative: Energy Efficient Clustering Algorithms for Wireless Sensor Networks Energy efficiency is a major concern in wireless sensor networks (WSNs). Many clustering algorithms have been proposed for such a purpose. This paper investigates the existing clustering algorithms. The algorithms have been classified and some representatives are described in each category. After analyzing the strengths and the weaknesses of each category, an important characteristic of WSNs is pointed out for further improvement of energy efficiency for WSNs.

negative: Graded Entailment for Compositional Distributional Semantics The categorical compositional distributional model of natural language provides a conceptually motivated procedure to compute the meaning of sentences, given grammatical structure and the meanings of its words. This approach has outperformed other models in mainstream empirical language processing tasks. However, until now it has lacked the crucial feature of lexical entailment – as do other distributional models of meaning. In this paper we solve the problem of entailment for categorical compositional distributional semantics. Taking advantage of the abstract categorical framework allows us to vary our choice of model. This enables the introduction of a notion of entailment, exploiting ideas from the categorical semantics of partial knowledge in quantum computation. The new model of language uses density matrices, on which we introduce a novel robust graded order capturing the entailment strength between concepts. This graded measure emerges from a general framework for approximate entailment, induced by any commutative monoid. Quantum logic embeds in our graded order. Our main theorem shows that entailment strength lifts compositionally to the sentence level, giving a lower bound on sentence entailment. We describe the essential properties of graded entailment such as continuity, and provide a procedure for calculating entailment strength.

negative: Autonomous exploration of motor skills by skill babbling Autonomous exploration of motor skills is a key capability of learning robotic systems. Learning motor skills can be formulated as inverse modeling problem, which targets at finding an inverse model that maps desired outcomes in some task space, e.g., via points of a motion, to appropriate actions, e.g., motion control policy parameters. In this paper, autonomous exploration of motor skills is achieved by incrementally learning inverse models starting from an initial demonstration. The algorithm is referred to as skill babbling, features sample-efficient learning, and scales to high-dimensional action spaces. Skill babbling extends ideas of goal-directed exploration, which organizes exploration in the space of goals. The proposed approach provides a modular framework for autonomous skill exploration by separating the learning of the inverse model from the exploration mechanism and a model of achievable targets, i.e. the workspace. The effectiveness of skill babbling is demonstrated for a range of motor tasks comprising the autonomous bootstrapping of inverse kinematics and parameterized motion primitives.

negative: Stochastic Gradient Hamiltonian Monte Carlo Hamiltonian Monte Carlo (HMC) sampling methods provide a mechanism for defining distant proposals with high acceptance probabilities in a Metropolis-Hastings framework, enabling more efficient exploration of the state space than standard random-walk proposals. The popularity of such methods has grown significantly in recent years. However, a limitation of HMC methods is the required gradient computation for simulation of the Hamiltonian dynamical system—such computation is infeasible in problems involving a large sample size or streaming data. Instead, we must rely on a noisy gradient estimate computed from a subset of the data. In this paper, we explore the properties of such a stochastic gradient HMC approach. Surprisingly, the natural implementation of the stochastic approximation can be arbitrarily bad. To address this problem we introduce a variant that uses second-order Langevin dynamics with a friction term that counteracts the effects of the noisy gradient, maintaining the desired target distribution as the invariant distribution. Results on simulated data validate our theory. We also provide an application of our methods to a classification task using neural networks and to online Bayesian matrix factorization.

negative: Convergent Temporal-Difference Learning with Arbitrary Smooth Function Approximation We introduce the first temporal-difference learning algorithms that converge with smooth value function approximators, such as neural networks. Conventional temporal-difference (TD) methods, such as TD(λ), Q-learning and Sarsa have been used successfully with function approximation in many applications. However, it is well known that off-policy sampling, as well as nonlinear function approximation, can cause these algorithms to become unstable (i.e., the parameters of the approximator may diverge). Sutton et al. (2009a, 2009b) solved the problem of off-policy learning with linear TD algorithms by introducing a new objective function, related to the Bellman error, and algorithms that perform stochastic gradient-descent on this function. These methods can be viewed as natural generalizations to previous TD methods, as they converge to the same limit points when used with linear function approximation methods. We generalize this work to nonlinear function approximation. We present a Bellman error objective function and two gradient-descent TD algorithms that optimize it. We prove the asymptotic almost-sure convergence of both algorithms, for any finite Markov decision process and any smooth value function approximator, to a locally optimal solution. The algorithms are incremental and the computational complexity per time step scales linearly with the number of parameters of the approximator. Empirical results obtained in the game of Go demonstrate the algorithms’ effectiveness.

negative: Exploration of Large Networks with Covariates via Fast and Universal Latent Space Model Fitting Latent space models are effective tools for statistical modeling and exploration of network data. These models can effectively model real world network characteristics such as degree heterogeneity, transitivity, homophily, etc. Due to their close connection to generalized linear models, it is also natural to incorporate covariate information in them. The current paper presents two universal fitting algorithms for networks with edge covariates: one based on nuclear norm penalization and the other based on projected gradient descent. Both algorithms are motivated by maximizing likelihood for a special class of inner-product models while working simultaneously for a wide range of different latent space models, such as distance models, which allow latent vectors to affect edge formation in flexible ways. These fitting methods, especially the one based on projected gradient descent, are fast and scalable to large networks. We obtain their rates of convergence for both inner-product models and beyond. The effectiveness of the modeling approach and fitting algorithms is demonstrated on five real world network datasets for different statistical tasks, including community detection with and without edge covariates, and network assisted learning.

negative: Curriculum Learning Based on Reward Sparseness for Deep Reinforcement Learning of Task Completion Dialogue Management Learning from sparse and delayed reward is a central issue in reinforcement learning. In this paper, to tackle reward sparseness problem of task oriented dialogue management, we propose a curriculum based approach on the number of slots of user goals. This curriculum makes it possible to learn dialogue management for sets of user goals with large number of slots. We also propose a dialogue policy based on progressive neural networks whose modules with parameters are appended with previous parameters fixed as the curriculum proceeds, and this policy improves performances over the one with single set of parameters.

negative: Deep Transfer in Reinforcement Learning by Language Grounding In this paper, we explore the utilization of natural language to drive transfer for reinforcement learning (RL). Despite the wide-spread application of deep RL techniques, learning generalized policy representations that work across domains remains a challenging problem. We demonstrate that textual descriptions of environments provide a compact intermediate channel to facilitate effective policy transfer. We employ a model-based RL approach consisting of a differentiable planning module, a model-free component and a factorized representation to effectively utilize entity descriptions. Our model outperforms prior work on both transfer and multi-task scenarios in a variety of different environments.

negative: Learning Structured Representation for Text Classification via Reinforcement Learning Representation learning is a fundamental problem in natural language processing. This paper studies how to learn a structured representation for text classification. Unlike most existing representation models that either use no structure or rely on pre-specified structures, we propose a reinforcement learning (RL) method to learn sentence representation by discovering optimized structures automatically. We demonstrate two attempts to build structured representation: Information Distilled LSTM (ID-LSTM) and Hierarchically Structured LSTM (HS-LSTM). ID-LSTM selects only important, taskrelevant words, and HS-LSTM discovers phrase structures in a sentence. Structure discovery in the two representation models is formulated as a sequential decision problem: current decision of structure discovery affects following decisions, which can be addressed by policy gradient RL. Results show that our method can learn task-friendly representations by identifying important words or task-relevant structures without explicit structure annotations, and thus yields competitive performance.