Question: Represent the article for finding a single-sentence TLDR

Question: Sentiment classification is an active research area with several applications including analysis of political opinions, classifying comments, movie reviews, news reviews and product reviews. To employ rule based sentiment classification, we require sentiment lexicons. However, manual construction of sentiment lexicon is time consuming and costly for resource-limited languages. To bypass manual development time and costs, we tried to build Amharic Sentiment Lexicons relying on corpus based approach. The intention of this approach is to handle sentiment terms specific to Amharic language from Amharic Corpus. Small set of seed terms are manually prepared from three parts of speech such as noun, adjective and verb. We developed algorithms for constructing Amharic sentiment lexicons automatically from Amharic news corpus. Corpus based approach is proposed relying on the word co-occurrence distributional embedding including frequency based embedding (i.e. Positive Point-wise Mutual Information PPMI). First we build word-context unigram frequency count matrix and transform it to point-wise mutual Information matrix. Using this matrix, we computed the cosine distance of mean vector of seed lists and each word in the corpus vocabulary. Based on the threshold value, the top closest words to the mean vector of seed list are added to the lexicon. Then the mean vector of the new sentiment seed list is updated and process is repeated until we get sufficient terms in the lexicon. Using PPMI with threshold value of 100 and 200, we got corpus based Amharic Sentiment lexicons of size 1811 and 3794 respectively by expanding 519 seeds. Finally, the lexicon generated in corpus based approach is evaluated. Most of sentiment mining research papers are associated to English languages. Linguistic computational resources in languages other than English are limited. Amharic is one of resource limited languages. Due to the advancement of World Wide Web, Amharic opinionated texts is increasing in size. To manage prediction of sentiment orientation towards a particular object or service is crucial for business intelligence, government intelligence, market intelligence, or support decision making. For carrying out Amharic sentiment classification, the availability of sentiment lexicons is crucial. To-date, there are two generated Amharic sentiment lexicons. These are manually generated lexicon and dictionary based Amharic SWN and SOCAL lexicons . However, dictionary based generated lexicons has short-comings in that it has difficulty in capturing cultural connotation and language specific features of the language. For example, Amharic words which are spoken culturally and used to express opinions will not be obtained from dictionary based sentiment lexicons. The word ጉርሻ/"feed in other people with hands which expresses love and live in harmony with others"/ in the Amharic text: "እንደ ጉርሻ ግን የሚያግባባን የለም... ጉርሻ እኮ አንዱ ለሌላው የማጉረስ ተግባር ብቻ አይደለም፤ በተጠቀለለው እንጀራ ውስጥ ፍቅር አለ፣ መተሳሰብ አለ፣ አክብሮት አለ።" has positive connotation or positive sentiment. But the dictionary meaning of the word ጉርሻ is "bonus". This is far away from the cultural connotation that it is intended to represent and express. We assumed that such kind of culture (or language specific) words are found in a collection of Amharic texts. However, dictionary based lexicons has short comings to capture sentiment terms which has strong ties to language and culture specific connotations of Amharic. Thus, this work builds corpus based algorithm to handle language and culture specific words in the lexicons. However, it could probably be impossible to handle all the words in the language as the corpus is a limited resource in almost all less resourced languages like Amharic. But still it is possible to build sentiment lexicons in particular domain where large amount of Amharic corpus is available. Due to this reason, the lexicon built using this approach is usually used for lexicon based sentiment analysis in the same domain from which it is built. The research questions to be addressed utilizing this approach are: How can we build an approach to generate Amharic Sentiment Lexicon from corpus?How do we evaluate the validity and quality of the generated lexicon? In this work, we set this approach to build Amharic polarity lexicons in automatic way relying on Amharic corpora which is mentioned shortly. The corpora are collected from different local news media organizations and also from facebook news' comments and you tube video comments to extend and enhance corpus size to capture sentiment terms into the generated PPMI based lexicon. In this part, we will present the key papers addressing corpus-based Sentiment Lexicon generation. In , large polarity lexicon is developed semiautomatically from the web by applying graph propagation method. A set of positive and negative sentences are prepared from the web for providing clue to expansion of lexicon. The method assigns a higher positive value if a given seed phrase contains multiple positive seed words, otherwise it is assigned negative value. The polarity p of seed phrase i is given by: where β is the factor that is responsible for preserving the overall semantic orientations between positive and negative flow over the graph. Both quantitatively and qualitatively, the performance of the web generated lexicon is outperforming the other lexicons generated from other manually annotate lexical resources like WordNet. The authors in developed two domain specific sentiment lexicons (historical and online community specific) from historical corpus of 150 years and online community data using word embedding with label propagation algorithm to expand small list of seed terms. It achieves competitive performance with approaches relying on hand curated lexicons. This revealed that there is sentiment change of words either positively to negatively or vice-versa through time. Lexical graph is constructed using PPMI matrix computed from word embedding. To fill the edges of two nodes (w i, w j), cosine similarity is computed. To propagate sentiment from seeds in lexical graph, random walk algorithm is adapted. That says, the polarity score of a seed set is proportional to probability of random walk from the seed set hitting that word. The generated lexicon from domain specific embedding outperforms very well when compared with the baseline and other variants. Our work is closely associated to the work of. generated emotion based lexicon by bootstrapping corpus using word distributional semantics (i.e. using PPMI). Our approach is different from their work in that we generated sentiment lexicon rather than emotion lexicon. The other thing is that the approach of propagating sentiment to expand the seeds is also different. We used cosine similarity of the mean vector of seed words to the corresponding word vectors in the vocabulary of the PPMI matrix. Besides, the threshold selection, the seed words part of speech are different from language to language. For example, Amharic has few adverb classes unlike Italian. Thus, our seed words do not contain adverbs. There are variety of corpus based strategies that include count based(e.g. PPMI) and predictive based(e.g. word embedding) approaches. In this part, we present the proposed count based approach to generate Amharic Sentiment lexicon from a corpus. In Figure 1, we present the proposed framework of corpus based approach to generate Amharic Sentiment lexicon. The framework has four components: (Amharic News) Corpus Collections, Preprocessing Module, PPMI Matix of Word-Context, Algorithm to generate (Amharic) Sentiment Lexicon ing in the Generated (Amharic) Sentiment Lexicon. The algorithm and the seeds in figure 1 are briefly described as follows. To generate Amharic Sentiment lexicon, we follow four major steps: 1. Prepare a set of seed lists which are strongly negatively and positively polarized Adjectives, Nouns and Verbs (Note: Amharic language contains few adverbs , adverbs are not taken as seed word). We will select at least seven most polarized seed words for each of aforementioned part-of-speech classes . Selection of seed words is the most critical that affects the performance of bootstrapping algorithm . Most authors choose the most frequently occurring words in the corpus as seed list. This is assumed to ensure the greatest amount of contextual information to learn from, however, we are not sure about the quality of the contexts. We adapt and follow seed selection guidelines of. After we tried seed selection based on , we update the original seed words. Sample summary of seeds are presented in Table 1. 2. Build semantic space word-context matrix using the number of occurrences(frequency) of target word with its context words of window size±2. Word-Context matrix is selected as it is dense and good for building word rich representations (i.e. similarity of words) unlike word-document design which is sparse and computationally expensive . Initially, let F be word-context raw frequency matrix with n r rows and n c columns formed from Amharic text corpora. Next, we apply weighting functions to select word semantic similarity descriminant features. There are variety of weighting functions to get meaningful semantic similarity between a word and its context. The most popular one is Point-wise Mutual Information (PMI) . In our case we use positive PMI by assigning 0 if it is less than 0 . Then, let X be new PMI based matrix that will be obtained by applying Positive PMI(PPMI) to matrix F. Matrix X will have the same number of rows and columns matrix F. The value of an element f ij is the number of times that word w i occurs in the context c j in matrix F. Then, the corresponding element x ij in the new matrix X would be defined as follows: Where, P M I(w i, c j) is the Point-wise Mutual Information that measures the estimated co-occurrence of word w i and its context c j and is given as: Where, P (w i, c j) is the estimated probability that the word w i occurs in the context of c j, P (w i) is the estimated probability of w i and P (c j) is the estimated probability of c i are defined in terms of frequency f ij. 3. Compute the cosine distance between target term and centroid of seed lists (e.g. centroid for positive adjective seeds, − − → µ + adj). To find the cosine distance of a new word from seed list, first we compute the centroids of seed lists of respective POS classes; for example, centroids for positive seeds S+ and negative seeds S-, for adjective class is given by: Similarly, centroids of the other seed classes will be found. Then, the cosine distances of target word from positive and negative adjective seeds of centroids, As word-context matrix x is vector space model, the cosine of the angle between two words vectors is the same as the inner product of the normalized unit word vectors. After we have cosine distances between word w i and seed with centroid w i, µ + adj, the similarity measure can be found using either: Similarly, the similarity score, Sim(w i, − − → µ − adj) can also be computed. This similarity score for each target word is mapped and scaled to appropriate real number. A target word whose sentiment score is below or above a particular threshold can be added to that sentiment dictionary in ranked order based on PMI based cosine distances. We choose positive PMI with cosine measure as it is performed consistently better than the other combination features with similarity metrics: Hellinger, Kullback-Leibler, City Block, Bhattacharya and Euclidean . 4. Repeat from step 3 for the next target term in the matrix to expand lexicon dictionary. Stop after a number of iterations defined by a threshold acquired experimental testing. The detail algorithm for generating Amharic sentiment lexicon from PPMI is presented in algorithm 1 Algorithm description: The algorithm 1 reads the seed words and generates the merge of expanded seed words using PPMI. Line 1 loads the seed words and assigns to their corresponding category of seed words. Similarly, from line 2 to 6 loads the necessary lexical resources such as PPMI matrix, vocabulary list, Amharic-English, AmharicAmharic, Amharic-Sentiment SWN and in line 7, the output Amharic Sentiment Lex. by PPMI is initialized to Null. From line 8 to 22, it iterates for each seed words polarity and categories. That is, line 9 to 11 checks that each seed term is found in the corpus vocabulary. Line 12 initializes the threshold by a selected trial number(in our case 100,200,1000, etc.). From line 13 to 22, iterates from i=0 to threshold in order to perform a set of operations. That is, line 16 computes the mean of the seed lexicon based on equation 3 specified in the previous section. Line 17 computes the similarity between the mean vector and the PPMI word-word co occurrence matrix and returns the top i most closest terms to the mean vector based on equation 5. Lines 18-19, it removes top closest items which has different part-of-speech to the seed words. Line 20-21 check the top i closest terms are has different polarity to the seed lexicon. Remove the term from top_ten_closest_terms list Update seed_lexicon by inserting top_ten_closest_terms list AM_Lexicon_by_PPMI ←AM_Lexicon_by_PPMI + seed_lexicon; algorithm 1: Amharic Sentiment Lexicon Generation Algorithm Using PPMI finding domain dependent opinions which might not be possible by sentiment lexicon generated using dictionary based approach. The quality of this lexicon will be evaluated using similar techniques used in dictionary based approaches . However, this approach may not probably produce sentiment lexicon with large coverage as the corpus size may be insufficient to include all polarity words in Amharic language. To reduce the level of this issue, we combine the lexicons generated in both dictionary based and corpus based approaches for Amharic Sentiment classification. Using corpus based approach, Amharic sentiment lexicon is built where it allows finding domain dependent opinions which might not be possible by sentiment lexicon generated using dictionary based approach. In this section, we have attempted to develop new approaches to bootstrapping relying on word-context semantic space representation of large Amharic corpora. We have manually prepared Amharic opinion words with highest sentimental strength either positively or negatively from three parts-of -speech categories: Adjectives, Nouns and Verbs. We expanded each seed category from which the snapshot seed words are presented in Table 1 Table 2. The corpus and data sets used in this research are presented as follows: i. Amharic Corpus: The size of this corpus is 20 milion tokens (teams from Addis Ababa University et al.). This corpus is used to build PPMI matrix and also to evaluate the coverage of PPMI based lexicon. ii. Facebook Users' Comment This data set is used to build PPMI matrix and also to evaluate subjectivity detection, lexicon coverage and lexicon based sentiment classification of the generated Amharic Sentiment lexicon. The data set is manually annotated by Government Office Affairs Communication(GOAC) professional and it is labeled either positive and negative. iii. Amharic Sentiment Lexicons: The Amharic sentiment lexicons includes manual , Amharic SentiWordNet(SWN) and Amharic Semantic Orientation Calculator(SOCAL) . These lexicons are used as benchmark to compare the performance of PPMI based lexicons. Using the aforementioned corpus, Amharic News Corpus with 11433 documents and 2800 Facebook News post and Users Comments are used to build word-context PPMI. First, we tried to clean the data set. After cleansing, we tokenized and word-Context count dictionary is built. Relying on this dictionary and in turn, it is transformed into PPMI Sparse matrices. This matrix is saved in file for further tasks of Amharic Sentiment lexicon generation. The total vocabulary size is 953,406. After stop words are removed and stemming is applied, the vocabulary size is reduced to 231,270. Thus, the size of this matrix is. Based on this word-word information in this matrix, the point-wise mutual information (PMI) of word-word is computed as in equation 2. The PMI is input to further computation of our corpus based lexicon expansion algorithm 1. Finally, we generated Amharic sentiment lexicons by expanding the seeds relying on the PPMI matrix of the corpus by implementing this algorithm 1 at two threshold iteration values of 100 and 200. With these iterations, we got corpus based Amharic Sentiment lexicons of size 1811 and 3794 respectively. We think, iterations >200 is better. Thus, our further discussion on evaluation of the approach is based on the lexicon generated at 200 iteration(i.e. Lexicon size of 3794). This lexicon saved with entries containing stemmed word, part of speech, polarity strength and polarity sign. Sample of this lexicon is presented in Table 2. ['ሀሰተኛ/lair/, ሀሰት/fake/, የሀሰት/fake/, ከሀሰተኛ/from fake maker/, ሀሰተኛና/lair and/, የሀሰተኛ/for fake maker/, ለሀሰተኛ/to lair/, ሀሴት/pleasure/, ታህሳት//, ሀሰትነት/fakeness/, ሀሰትን/fake/, ከሀሰተኛው/from the lair/, ሀሰተኛው/lair/, ሀሰቱን/lair/, ከሀሰት/from lair/, ሀሰትና/fake and/, ከሀሰቱ/from the fake/, ሀሴትና/pleasure and/, ሀሴትን/the pleasure/, ሀሳት/fake/, ለሀሰትና/to fake and/, ሁሰት//, የሀሰትና/for fake and/, ለሀሰት/to fake/, ሀሰቶች/alot of fake/,etc. We will evaluate in three ways: external to lexicon and internal to lexicon. External to lexicon is to test the usefulness and the correctness of each of the lexicon to find sentiment score of sentiment labeled Amharic comments corpus. Internal evaluation is compute the degree to which each of the generated lexicons are overlapped( or agreed) with manual, SOCAL and SWN(Amharic) Sentiment lexicons. In this part, we evaluate the accuracy of the subjectivity detection rate of generated PPMI based Amharic lexicon on 2800 facebook annotated comments. The procedures for aggregating sentiment score is done by summing up the positive and negative sentiment values of opinion words found in the comment if those opinion words are also found in the sentiment lexicon. If the sum of positive score greater or less than sum of negative score, then the comment is said to be a subjective comment, otherwise the comment is either neutral or mixed. Based on this technique, the subjectivity detection rate is presented in Table 3. Discussion: As subjectivity detection rate of the PPMI lexicon and others are depicted in Table 3, the detection rate of PPMI lexicon performs better than the baseline(manual lexicon). Where as Lexicon from SWN outperforms the PPMI based Lexicon with 2% accuracy. This is to evaluate the coverage of the generated lexicon externally by using the aforementioned Amharic corpus(both facebook comments and general corpus). That is, the coverage of PPMI based Amharic Sentiment Lexicon on facebook comments and a general Amharic corpus is computed by counting the occurrence tokens of the corpus in the generated sentiment lexicons and both positive and negative counts are computed in percent and it presented in Table 4. Table 4 depicted that the coverage of PPMI based Amharic sentiment lexicon is better than the manual lexicon and SOCAL. However, it has less coverage than SWN. Unlike SWN, PPMI based lexicon is generated from corpus. Due to this reason its coverage to work on a general domain is limited. It also demonstrated that the positive and negative count in almost all lexicons seems to have balanced and uniform distribution of sentiment polarity terms in the corpus. In this part, we will evaluate to what extent the generated PPMI based Lexicon agreed or overlapped with the other lexicons. This type of evaluation (internal) which validates by stating the percentage of entries in the lexicons are available in common. The more percentage means the better agreement or overlap of lexicons. The agreement of PPMI based in percentage is presented in Table 5. Discussion: Table 5 presents the extent to which the PPMI based lexicon is agreed with other lexicons. PPMI based lexicon has got the highest agreement rate (overlap) with SWN lexicon than the other lexicons. In this part, Table 6 presents the lexicon based sentiment classification performance of generated PPMI based Amharic lexicon on 2821 annotated Amharic Facebook Comments. The classification accuracy of generated PPMI based Lexicon and other lexicons are compared. Discussion: Besides the other evaluations of the generated PPMI based lexicon, the usefulness of this lexicon is tested on actual lexicon based Amharic sentiment classification. As depicted in Table 6 The accuracy of PPMI based lexicon for lexicon based sentiment classification is better than the manual benchmark lexicon. As discussed on dictionary based lexicons for lexicon based sentiment classification in earlier section, using stemming and negation handling are far improving the performance lexicon based classification. Besides combination of lexicons outperforms better than the individual lexicon. Creating a sentiment lexicon generation is not an objective process. The generated lexicon is dependent on the task it is applied. Thus, in this work we have seen that it is possible to create Sentiment lexicon for low resourced languages from corpus. This captures the language specific features and connotations related to the culture where the language is spoken. This can not be handled using dictionary based approach that propagates labels from resource rich languages. To the best of our knowledge, the the PPMI based approach to generate Amharic Sentiment lexicon form corpus is performed for first time for Amharic language with almost minimal costs and time. Thus, the generated lexicons can be used in combination with other sentiment lexicons to enhance the performance of sentiment classifications in Amharic language. The approach is a generic approach which can be adapted to other resource limited languages to reduce cost of human annotation and the time it takes to annotated sentiment lexicons. Though the PPMI based Amharic Sentiment lexicon outperforms the manual lexicon, prediction (word embedding) based approach is recommended to generate sentiment lexicon for Amharic language to handle context sensitive terms.

represent this text to find a much much longer article about it with say hundreds-thousands of words

Corpus based Algorithm is developed generate Amharic Sentiment lexicon relying on corpus

Represent the TLDR for finding the corresponding scientific article

This work aims to provide quantitative answers to the relative importance of concepts of interest via concept activation vectors (CAV). In particular, this framework enables non-machine learning experts to express concepts of interest and test hypotheses using examples (e.g., a set of pictures that illustrate  the concept). We show that CAV can be learned given a relatively small set of examples. Hypothesis testing with CAV can answer whether a particular concept (e.g., gender) is more important in predicting a given class (e.g., doctor) than other sets of concepts. Interpreting networks with CAV does not require any retraining or modification of the network.


Question: Represent the text to find a very short summary (1-2 sentences max)

Question: Optimistic initialisation is an effective strategy for efficient exploration in reinforcement learning (RL). In the tabular case, all provably efficient model-free algorithms rely on it. However, model-free deep RL algorithms do not use optimistic initialisation despite taking inspiration from these provably efficient tabular algorithms. In particular, in scenarios with only positive rewards, Q-values are initialised at their lowest possible values due to commonly used network initialisation schemes, a pessimistic initialisation. Merely initialising the network to output optimistic Q-values is not enough, since we cannot ensure that they remain optimistic for novel state-action pairs, which is crucial for exploration. We propose a simple count-based augmentation to pessimistically initialised Q-values that separates the source of optimism from the neural network. We show that this scheme is provably efficient in the tabular setting and extend it to the deep RL setting. Our algorithm, Optimistic Pessimistically Initialised Q-Learning (OPIQ), augments the Q-value estimates of a DQN-based agent with count-derived bonuses to ensure optimism during both action selection and bootstrapping. We show that OPIQ outperforms non-optimistic DQN variants that utilise a pseudocount-based intrinsic motivation in hard exploration tasks, and that it predicts optimistic estimates for novel state-action pairs. In reinforcement learning (RL), exploration is crucial for gathering sufficient data to infer a good control policy. As environment complexity grows, exploration becomes more challenging and simple randomisation strategies become inefficient. While most provably efficient methods for tabular RL are model-based (; ;), in deep RL, learning models that are useful for planning is notoriously difficult and often more complex than modelfree methods. Consequently, model-free approaches have shown the best final performance on large complex tasks (; 2016;), especially those requiring hard exploration . Therefore, in this paper, we focus on how to devise model-free RL algorithms for efficient exploration that scale to large complex state spaces and have strong theoretical underpinnings. Despite taking inspiration from tabular algorithms, current model-free approaches to exploration in deep RL do not employ optimistic initialisation, which is crucial to provably efficient exploration in all model-free tabular algorithms. This is because deep RL algorithms do not pay special attention to the initialisation of the neural networks and instead use common initialisation schemes that yield initial Q-values around zero. In the common case of non-negative rewards, this means Q-values are initialised to their lowest possible values, i.e., a pessimistic initialisation. While initialising a neural network optimistically would be trivial, e.g., by setting the bias of the final layer of the network, the uncontrolled generalisation in neural networks changes this initialisation quickly. Instead, to benefit exploration, we require the Q-values for novel state-action pairs must remain high until they are explored. An empirically successful approach to exploration in deep RL, especially when reward is sparse, is intrinsic motivation . A popular variant is based on pseudocounts , which derive an intrinsic bonus from approximate visitation counts over states and is inspired by the tabular MBIE-EB algorithm . However, adding a positive intrinsic bonus to the reward yields optimistic Q-values only for state-action pairs that have already been chosen sufficiently often. Incentives to explore unvisited states rely therefore on the generalisation of the neural network. Exactly how the network generalises to those novel state-action pairs is unknown, and thus it is unclear whether those estimates are optimistic when compared to nearby visited state-action pairs. Figure 1 Consider the simple example with a single state and two actions shown in Figure 1. The left action yields +0.1 reward and the right action yields +1 reward. An agent whose Q-value estimates have been zero-initialised must at the first time step select an action randomly. As both actions are underestimated, this will increase the estimate of the chosen action. Greedy agents always pick the action with the largest Q-value estimate and will select the same action forever, failing to explore the alternative. Whether the agent learns the optimal policy or not is thus decided purely at random based on the initial Q-value estimates. This effect will only be amplified by intrinsic reward. To ensure optimism in unvisited, novel state-action pairs, we introduce Optimistic Pessimistically Initialised Q-Learning (OPIQ). OPIQ does not rely on an optimistic initialisation to ensure efficient exploration, but instead augments the Q-value estimates with count-based bonuses in the following manner: where N (s, a) is the number of times a state-action pair has been visited and M, C > 0 are hyperparameters. These Q + -values are then used for both action selection and during bootstrapping, unlike the above methods which only utilise Q-values during these steps. This allows OPIQ to maintain optimism when selecting actions and bootstrapping, since the Q + -values can be optimistic even when the Q-values are not. In the tabular domain, we base OPIQ on UCB-H , a simple online Q-learning algorithm that uses count-based intrinsic rewards and optimistic initialisation. Instead of optimistically initialising the Q-values, we pessimistically initialise them and use Q + -values during action selection and bootstrapping. Pessimistic initialisation is used to enable a worst case analysis where all of our Q-value estimates underestimate Q * and is not a requirement for OPIQ. We show that these modifications retain the theoretical guarantees of UCB-H. Furthermore, our algorithm easily extends to the Deep RL setting. The primary difficulty lies in obtaining appropriate state-action counts in high-dimensional and/or continuous state spaces, which has been tackled by a variety of approaches (; ; ; a) and is orthogonal to our contributions. We demonstrate clear performance improvements in sparse reward tasks over 1) a baseline DQN that just uses intrinsic motivation derived from the approximate counts, 2) simpler schemes that aim for an optimistic initialisation when using neural networks, and 3) strong exploration baselines. We show the importance of optimism during action selection for ensuring efficient exploration. Visualising the predicted Q + -values shows that they are indeed optimistic for novel state-action pairs. We consider a Markov Decision Process (MDP) defined as a tuple (S, A, P, R), where S is the state space, A is the discrete action space, P (·|s, a) is the state-transition distribution, R(·|s, a) is the distribution over rewards and γ ∈ is the discount factor. The goal of the agent is then to maximise the expected discounted sum of rewards: E[∞ t=0 γ t r t |r t ∼ R(·|s t, a t)], in the discounted episodic setting. A policy π(·|s) is a mapping from states to actions such that it is a valid probability distribution. Deep Q-Network (DQN) uses a nonlinear function approximator (a deep neural network) to estimate the action-value function, Q(s, a; θ) ≈ Q * (s, a), where θ are the parameters of the network. Exploration based on intrinsic rewards (e.g.,), which uses a DQN agent, additionally augments the observed rewards r t with a bonus β/ N (s t, a t) based on pseudo-visitation-counts N (s t, a t). The DQN parameters θ are trained by gradient descent on the mean squared regression loss L with bootstrapped'target' y t: Figure 2: A simple regression task to illustrate the effect of an optimistic initialisation in neural networks. Left: 10 different networks whose final layer biases are initialised at 3 (shown in green), and the same networks after training on the blue data points (shown in red). Right: One of the trained networks whose output has been augmented with an optimistic bias as in equation 1. The counts were obtained by computing a histogram over the input space [−2, 2] with 50 bins. The expectation is estimated with uniform samples from a replay buffer D . D stores past transitions (s t, a t, r t, s t+1), where the state s t+1 is observed after taking the action a t in state s t and receiving reward r t. To improve stability, DQN uses a target network, parameterised by θ −, which is periodically copied from the regular network and kept fixed for a number of iterations. Our method Optimistic Pessimistically Initialised Q-Learning (OPIQ) ensures optimism in the Qvalue estimates of unvisited, novel state-action pairs in order to drive exploration. This is achieved by augmenting the Q-value estimates in the following manner: and using these Q + -values during action selection and bootstrapping. In this section, we motivate OPIQ, analyse it in the tabular setting, and describe a deep RL implementation. Optimistic initialisation does not work with neural networks. For an optimistic initialisation to benefit exploration, the Q-values must start sufficiently high. More importantly, the values for unseen state-action pairs must remain high, until they are updated. When using a deep neural network to approximate the Q-values, we can initialise the network to output optimistic values, for example, by adjusting the final bias. However, after a small amount of training, the values for novel state-action pairs may not remain high. Furthermore, due to the generalisation of neural networks we cannot know how the values for these unseen state-action pairs compare to the trained state-action pairs. Figure 2 (left), which illustrates this effect for a simple regression task, shows that different initialisations can lead to dramatically different generalisations. It is therefore prohibitively difficult to use optimistic initialisation of a deep neural network to drive exploration. Instead, we augment our Q-value estimates with an optimistic bonus. Our motivation for the form of the bonus in equation 1, C (N (s,a)+1) M, stems from UCB-H , where all tabular Q-values are initialised with H and the first update for a state-action pair completely overwrites that value because the learning rate for the update (η 1) is 1. One can alternatively view these Q-values as zero-initialised with the additional term Q(s, a) + H · 1{N(s, a) < 1}, where N (s, a) is the visitation count for the state-action pair (s, a). Our approach approximates the discrete indicator function 1 as (N (s, a) + 1) −M for sufficiently large M. However, since gradient descent cannot completely overwrite the Q-value estimate for a state-action pair after a single update, it is beneficial to have a smaller hyperparameter M that governs how quickly the optimism decays. for each timestep t = 1,..., H do Take action a t ← arg max a Q + t (s t, a). Receive r(s t, a t, t) and s t+1. Increment N (s t, a t, t). For a worst case analysis we assume all Q-value estimates are pessimistic. In the common scenario where all rewards are nonnegative, the lowest possible return for an episode is zero. If we then zero-initialise our Q-value estimates, as is common for neural networks, we are starting with a pessimistic initialisation. As shown in Figure 2 (left), we cannot predict how a neural network will generalise, and thus we cannot predict if the Q-value estimates for unvisited state-action pairs will be optimistic or pessimistic. We thus assume they are pessimistic in order to perform a worst case analysis. However, this is not a requirement: our method works with any initialisation and rewards. In order to then approximate an optimistic initialisation, the scaling parameter C in equation 1 can be chosen to guarantee unseen Q + -values are overestimated, for example, C:= H in the undiscounted finite-horizon tabular setting and C:= 1/(1 − γ) in the discounted episodic setting (assuming 1 is the maximum reward obtainable at each timestep). However, in some environments it may be beneficial to use a smaller parameter C for faster convergence. These Q + -values are then used both during action selection and during bootstrapping. Note that in the finite horizon setting the counts N, and thus Q +, would depend on the timestep t. Hence, we split the optimistic Q + -values into two parts: a pessimistic Q-value component and an optimistic component based solely on the counts for a state-action pair. This separates our source of optimism from the neural network function approximator, yielding Q + -values that remain high for unvisited state-action pairs, assuming a suitable counting scheme. Figure 2 (right) shows the effects of adding this optimistic component to a network's outputs. + -values provide an increased incentive to explore. By using optimistic Q + estimates, especially during action selection and bootstrapping, the agent is incentivised to try and visit novel state-action pairs. Being optimistic during action selection in particular encourages the agent to try novel actions that have not yet been visitied. Without an optimistic estimate for novel state-action pairs the agent would have no incentive to try an action it has never taken before at a given state. Being optimistic during bootstrapping ensures the agent is incentivised to return to states in which it has not yet tried every action. This is because the maximum Q + -value will be large due to the optimism bonus. Both of these effects lead to a strong incentive to explore novel state-action pairs. In order to ensure that OPIQ has a strong theoretical foundation, we must ensure it is provably efficient in the tabular domain. We restrict our analysis to the finite horizon tabular setting and only consider building upon UCB-H for simplicity. Achieving a better regret bound using UCB-B and extending the analysis to the infinite horizon discounted setting are steps for future work. Our algorithm removes the optimistic initialisation of UCB-H, instead using a pessimistic initialisation (all Q-values start at 0). We then use our Q + -values during action selection and bootstrapping. Pseudocode is presented in Algorithm 1. Theorem 1. For any p ∈, with probability at least 1 − p the total regret of Q + is at most The proof is based on that of Theorem 1 from . Our Q + -values are always greater than or equal to the Q-values that UCB-H would estimate, thus ensuring that our estimates are also greater than or equal to Q *. Our overestimation relative to UCB-H is then governed by the quantity H/(N (s, a) + 1) M, which when summed over all timesteps does not depend on T for M > 1. As M → ∞ we exactly recover UCB-H, and match the asymptotic performance of UCB-H for M ≥ 1. Smaller values of M in our optimism decaying more slowly, which in more exploration. The full proof is included in Appendix I. We also show that OPIQ without optimistic action selection or the count-based intrinsic motivation term b T N is not provably efficient by showing it can incur linear regret with high probability on simple MDPs (see Appendices G and H). Our primary motivation for considering a tabular algorithm that pessimistically initialises its Q-values, is to provide a firm theoretical foundation on which to base a deep RL algorithm, which we describe in the next section. For deep RL, we base OPIQ on DQN , which uses a deep neural network with parameters θ as a function approximator Q θ. During action selection, we use our Q + -values to determine the greedy action: where C action is a hyperparameter governing the scale of the optimistic bias during action selection. In practice, we use an -greedy policy. After every timestep, we sample a batch of experiences from our experience replay buffer, and use n-step Q-learning . We recompute the counts for each relevant state-action pair, to avoid using stale pseudo-rewards. The network is trained by gradient decent on the loss in equation 2 with the target: where C bootstrap is a hyperparameter that governs the scale of the optimistic bias during bootstrapping. For our final experiments on Montezuma's Revenge we additionally use the Mixed Monte Carlo (MMC) target , which mixes the target with the environmental monte carlo return for that episode. Further details are included in Appendix D.4. We use the method of static hashing to obtain our pseudocounts on the first 2 of 3 environments we test on. For our experiments on Montezuma's Revenge we count over a downsampled image of the current game frame. More details can be found in Appendix B. A DQN with pseudocount derived intrinsic reward (DQN + PC) can be seen as a naive extension of UCB-H to the deep RL setting. However, it does not attempt to ensure optimism in the Q-values used during action selection and bootstrapping, which is a crucial component of UCB-H. Furthermore, even if the Q-values were initialised optimistically at the start of training they would not remain optimistic long enough to drive exploration, due to the use of neural networks. OPIQ, on the other hand, is designed with these limitations of neural networks in mind. By augmenting the neural network's Q-value estimates with optimistic bonuses of the form C (N (s,a)+1) M, OPIQ ensures that the Q + -values used during action selection and bootstrapping are optimistic. We can thus consider OPIQ as a deep version of UCB-H. Our show that optimism during action selection and bootstrapping is extremely important for ensuring efficient exploration. Tabular Domain: There is a wealth of literature related to provably efficient exploration in the tabular domain. Popular model-based algorithms such as R-MAX , MBIE (and MBIE-EB) , UCRL2 and UCBVI are all based on the principle of optimism in the face of uncertainty. adopt a Bayesian viewpoint and argue that posterior sampling (PSRL) is more practically efficient than approaches that are optimistic in the face of uncertainty, and prove that in Bayesian expectation PSRL matches the performance of any optimistic algorithm up to constant factors. prove that an optimistic variant of PSRL is provably efficient under a frequentist regret bound. The only provably efficient model-free algorithms to date are delayed Q-learning and UCB-H (and UCB-B) . Delayed Q-learning optimistically initialises the Q-values that are carefully controlled when they are updated. UCB-H and UCB-B also optimistically initialise the Q-values, but also utilise a count-based intrinsic motivation term and a special learning rate to achieve a favourable regret bound compared to model-based algorithms. In contrast, OPIQ pessimistically initialises the Q-values. Whilst we base our current analysis on UCB-H, the idea of augmenting pessimistically initialised Q-values can be applied to any model-free algorithm. Deep RL Setting: A popular approach to improving exploration in deep RL is to utilise intrinsic motivation , which computes a quantity to add to the environmental reward. Most relevant to our work is that of , which takes inspiration from MBIE-EB . utilise the number of times a state has been visited to compute the intrinsic reward. They outline a framework for obtaining approximate counts, dubbed pseudocounts, through a learned density model over the state space. show that RLSVI achieves provably efficient Bayesian expected regret, which requires a prior distribution over MDPs, whereas OPIQ achieves provably efficient worse case regret. Bootstrapped DQN with a prior is thus a model-free algorithm that has strong theoretical support in the tabular setting. Empirically, however, its performance on sparse reward tasks is worse than DQN with pseudocounts. shift and scale the rewards so that a zero-initialisation is optimistic. When applied to neural networks this approach does not in optimistic Q-values due to the generalisation of the networks. empirically show that using a pseudocount intrinsic motivation term performs much better empirically on hard exploration tasks. attempt to generalise the notion of a count to include information about the counts of future state-actions pairs in a trajectory, which they use to provide bonuses during action selection. extend delayed Q-learning to utilise these generalised counts and prove the scheme is PAC-MDP. The generalised counts are obtained through E-values which are learnt using SARSA with a constant 0 reward and E-value estimates initialised at 1. When scaling to the deep RL setting, these E-values are estimated using neural networks that cannot maintain their initialisation for unvisited state-action pairs, which is crucial for providing an incentive to explore. By contrast, OPIQ uses a separate source to generate the optimism necessary to explore the environment. We compare OPIQ against baselines and ablations on three sparse reward environments. The first is a randomized version of the Chain environment proposed by and used in with a chain of length 100, which we call Randomised Chain. The second is a two-dimensional maze in which the agent starts in the top left corner (white dot) and is only rewarded upon finding the goal (light grey dot). We use an image of the maze as input and randomise the actions similarly to the chain. The third is Montezuma's Revenge from Arcade Learning environment , a notoriously difficult sparse reward environment commonly used as a benchmark to evaluate the performance and scaling of Deep RL exploration algorithms. See Appendix D for further details on the environments, baselines and hyperparameters used. We compare OPIQ against a variety of DQN-based approaches that use pseudocount intrinsic rewards, the DORA agent (which generates count-like optimism bonuses using a neural network), and two strong exploration baselines: -greedy DQN: a standard DQN that uses an -greedy policy to encourage exploration. We anneal linearly over a fixed number of timesteps from 1 to 0.01. DQN + PC: we add an intrinsic reward of β/ N (s, a) to the environmental reward based on . DQN R-Subtract (+PC): we subtract a constant from all environmental rewards received when training, so that a zero-initialisation is optimistic, as described for a DQN in and based on. DQN Bias (+PC): we initialise the bias of the final layer of the DQN to a positive value at the start of training as a simple method for optimistic initialisation with neural networks. DQN + DORA: we use the generalised counts from as an intrinsic reward. DQN + DORA OA: we additionally use the generalised counts to provide an optimistic bonus during action selection. DQN + RND: we add the RND bonus from as an intrinsic reward. BSP: we use Bootstrapped DQN with randomised prior functions . In order to better understand the importance of each component of our method, we also evaluate the following ablations: Optimistic Action Selection (OPIQ w/o OB): we only use our Q + -values during action selection, and use Q during bootstrapping (without Optimistic Bootstrapping). The intrinsic motivation term remains. Optimistic Action Selection and Bootstrapping (OPIQ w/o PC): we use our Q + -values during action selection and bootstrapping, but do not include an intrinsic motivation term (without Pseudo Counts). We first consider the visually simple domain of the randomised chain and compare the count-based methods. Figure 3 shows the performance of OPIQ compared to the baselines and ablations. OPIQ significantly outperforms the baselines, which do not have any explicit mechanism for optimism during action selection. A DQN with pseudocount derived intrinsic rewards is unable to reliably find the goal state, but setting the final layer's bias to one produces much better performance. For the DQN variant in which a constant is subtracted from all rewards, all of the configurations (including those with pseudocount derived intrinsic bonuses) were unable to find the goal on the right and thus the agents learn quickly to latch on the inferior reward of moving left. Compared to its ablations, OPIQ is more stable in this task. OPIQ without pseudocounts performs similarly to OPIQ but is more varied across seeds, whereas the lack of optimistic bootstrapping in worse performance and significantly more variance across seeds. We next consider the harder and more visually complex task of the Maze and compare against all baselines. Figure 4 shows that only OPIQ is able to find the goal in the sparse reward maze. This indicates that explicitly ensuring optimism during action selection and bootstrapping can have a significant positive impact in sparse reward tasks, and that a naive extension of UCB-H to the deep RL setting (DQN + PC) in insufficient exploration. Figure 4 (right) shows that attempting to ensure optimistic Q-values by adjusting the bias of the final layer (DQN Bias + PC), or by subtracting a constant from the reward (DQN R-Subtract + PC) has very little effect. As expected DQN + RND performs poorly on this domain compared to the pseudocount based methods. The visual input does not vary much across the state space, ing in the RND bonus failing to provide enough intrinsic motivation to ensure efficient exploration. Additionally it does not feature any explicit mechanism for optimism during action selection, and thus Figure 4 (right) shows it explores the environment relatively slowly. Both DQN+DORA and DQN+DORA OA also perform poorly in this domain since their source of intrinsic motivation disappears quickly. As noted in Figure 2, neural networks do not maintain their starting initialisations after training. Thus, the intrinsic reward DORA produces goes to 0 quickly since the network producing its bonuses learns to generalise quickly. BSP is the only exploration baseline we test that does not add an intrinsic reward to the environmental reward, and thus it performs poorly compared to the other baselines on this environment. Figure 5 shows that OPIQ and all its ablations manage to find the goal in the maze. OPIQ also explores slightly faster than its ablations (right), which shows the benefits of optimism during both action selection and bootstrapping. In addition, the episodic reward for the the ablation without optimistic bootstrapping is noticeably more unstable (Figure 5, left). Interestingly, OPIQ without pseudocounts performs significantly worse than the other ablations. This is surprising since the theory suggests that the count-based intrinsic motivation is only required when the reward or transitions of the MDP are stochastic , which is not the case here. We hypothesise that adding PC-derived intrinsic bonuses to the reward provides an easier learning problem, especially when using n-step Q-Learning, which yields the performance gap. However, our show that the PC-derived intrinsic bonuses are not enough on their own to ensure sufficient exploration. The large difference in performance between DQN + PC and OPIQ w/o OB is important, since they only differ in the use of optimistic action selection. The in Figures 4 and 5 show that optimism during action selection is extremely important in exploring the environment efficiently. Intuitively, this makes sense, since this provides an incentive for the agent to try actions it has never tried before, which is crucial in exploration. Finally, we consider Montezuma's Revenge, one of the hardest sparse reward games from the ALE . Note that we only train up to 12.5mil timesteps (50mil frames), a 1/4 of the usual training time (50mil timesteps, 200mil frames). Figure 7 shows that OPIQ significantly outperforms the baselines in terms of the episodic reward and the maximum episodic reward achieved during training. The higher episode reward and much higher maximum episode reward of OPIQ compared to DQN + PC once again demonstrates the importance of optimism during action selection and bootstrapping. In this environment BSP performs much better than in the Maze, but achieves significantly lower episodic rewards than OPIQ. Figure 8 shows the distinct number of rooms visited across the training period. We can see that OPIQ manages to reliably explore 12 rooms during the 12.5mil timesteps, significantly more than the other methods, thus demonstrating its improved exploration in this complex environment. Our on this challenging environment show that OPIQ can scale to high dimensional complex environments and continue to provide significant performance improvements over an agent only using pseudocount based intrinsic rewards. This paper presented OPIQ, a model-free algorithm that does not rely on an optimistic initialisation to ensure efficient exploration. Instead, OPIQ augments the Q-values estimates with a count-based optimism bonus. We showed that this is provably efficient in the tabular setting by modifying UCB-H to use a pessimistic initialisation and our augmented Q + -values for action selection and bootstrapping. Since our method does not rely on a specific initialisation scheme, it easily scales to deep RL when paired with an appropriate counting scheme. Our showed the benefits of maintaining optimism both during action selection and bootstrapping for exploration on a number of hard sparse reward environments including Montezuma's Revenge. In future work, we aim to extend OPIQ by integrating it with more expressive counting schemes. For the tabular setting, we consider a discrete finite-horizon Markov Decision Process (MDP), which can be defined as a tuple (S, A, {P t}, {R t}, H, ρ), where S is the finite state space, A is the finite action space, P t (·|s, a) is the state-transition distribution for timestep t = 1,..., H, R t (·|s, a) is the distribution over rewards after taking action a in state s, H is the horizon, and ρ is the distribution over starting states. Without loss of generality we assume that R t (·|s, a) ∈. We use S and A to denote the number of states and the number of actions, respectively, and N (s, a, t) as the number of times a state-action pair (s, a) has been visited at timestep t. Our goal is to find a set of policies π t: S → A, π:= {π t}, that chooses the agent's actions at time t such that the expected sum of future rewards is maximised. To this end we define the Q-value at time t of a given policy π as Q π t (s, a):= E r + Q π t+1 (s, π t+1 (s)) | r∼Rt(·|s,a), s ∼Pt(·|s,a), where Q π t (s, a) = 0, ∀t > H. The agent interacts with the environment for K episodes, T:= KH, yielding a total regret: refers to the starting state and π k to the policy at the beginning of episode k. We are interested in bounding the worst case total regret with probability 1 − p, 0 < p < 1. ) is an online Q-learning algorithm for the finite-horizon setting outlined above where the worse case total regret is bounded with a probability of 1−p by O(H 4 SAT log(SAT /p). All Q-values for timesteps t ≤ H are optimistically initialised at H. The learning rate is defined as η N = H+1 H+N, where N:= N (s t, a t, t) is the number of times state-action pair (s t, a t) has been observed at step t and η 1 = 1 at the first encounter of any state-action pair. The update rule for a transition at step t from state s t to s t+1, after executing action a t and receiving reward r t, is: where b, is the count-based intrinsic motivation term. In deep RL, the primary difficulty for exploration based on count-based intrinsic rewards is obtaining appropriate state-action counts. In this paper we utilise approximate counting schemes (; ;) in order to cope with continuous and/or highdimensional state spaces. In particular, for the chain and maze environments we use static hashing , which projects a state s to a low-dimensional feature vector φ(s) = sign(Af (s)), where f flattens the state s into a single dimension of length D; A is a k × D matrix whose entries are initialised i.i.d. from a unit Gaussian: N; and k is a hyperparameter controling the granularity of counting: higher k leads to more distinguishable states at the expense of generalisation. Given the vector φ(s), we use a counting bloom filter to update and retrieve its counts efficiently. To obtain counts N (s, a) for state-action pairs, we maintain a separate data structure of counts for each action (the same vector φ(s) is used for all actions). This counting scheme is tabular and hence the counts for sufficiently different states do not interfere with one another. This ensures Q + -values for unseen state-action pairs in equation 1 are large. For our experiments on Montezuma's Revenge we use the same method of downsampling as in , in which the greyscale state representation is resized from (42x42) to (11x8) and then binned from {0, ..., 255} into 8 categories. We then maintain tabular counts over the new representation. The granularity of the counting scheme is an important modelling consideration. If it is too granular, then it will assign an optimistic bias in regions of the state space where the network should be trusted A 2-dimensional gridworld maze with a sparse reward in which the agent can move Up, Down, Left or Right. The agent starts each episode at a fixed location and must traverse through the maze in order to find the goal which provides +10 reward and terminates the episode, all other rewards are 0. The agent interacts with the maze for 250 timesteps before being reset. Empty space is represented by a 0, walls are 1, the goal is 2 and the player is 3. The state representation is a greyscaled image of the entire grid where each entry is divided by 3 to lie in. The shape of the state representation is:. Once again the effect of each action is randomised at each state at the beginning of training. Figure 11 shows the structure of the maze environment. We follow the suggestions in (b) and use the same environmental setup as used in . Specifically, we use stick actions with a probability of p = 0.25, a frame skip of 4 and do not show a terminal state on the loss of life. In all experiments we set γ = 0.99, use RMSProp with a learning rate of 0.0005 and scale the gradient norms during training to be at most 5. The network used is a MLP with 2 hidden layers of 256 units and ReLU non-linearities. We use 1 step Q-Learning. Training lasts for 100k timesteps. is fixed at 0.01 for all methods except for -greedy DQN in which it is linearly decayed from 1 to 0.01 over {100, 50k, 100k} timesteps. We train on a batch size of 64 after every timestep with a replay buffer of size 10k. The target network is updated every 200 timesteps. The embedding size used for the counts is 32. We set β = 0.1 for the scale of the count-based intrinsic motivation. For reward subtraction we consider subtracting {0.1, 1, 10} from the reward. For an optimistic initialisation bias, we consider setting the final layer's bias to {0.1, 1, 10}. We consider both of the methods with and without count-based intrinsic motivation. For OPIQ and its ablations we consider: M ∈ {0.1, 0.5, 2, 10}, C action ∈ {0.1, 1, 10}, C bootstrap ∈ {0.01, 0.1, 1, 10}. For all methods we run 20 independent runs across the cross-product of all relevant parameters considered. We then sort them by the median test reward (largest area underneath the line) and report the median, lower and upper quartiles. The best hyperparameters we found were: DQN -greedy: Decay rate: 100 timesteps. Optimistic Initialisation Bias: Bias: 1, Pseudocount intrinsic motivation: True. Reward Subtraction: Constant to subtract: 1, Pseudocount intrinsic motivation: False. OPIQ: M: 0.5, C action: 1, C bootstrap: 1. OPIQ without Optimistic Bootstrapping: M: 2, C action: 10. OPIQ without Pseudocounts: M: 2, C action: 10, C bootstrap: 10. For We use 3 step Q-Learning. Training lasts for 1mil timesteps. is decayed linearly from 1 to 0.01 over 50k timesteps for all methods except for -greedy DQN in which it is linearly decayed from 1 to 0.01 over {100, 50k, 100k} timesteps. We train on a batch of 64 after every timestep with a replay buffer of size 250k. The target network is updated every 1000 timesteps. The embedding dimension for the counts is 128. For DQN + PC we consider β ∈ {0.01, 0.1, 1, 10, 100}. For all other methods we set β = 0.1 as it performed best. For reward subtraction we consider subtracting {0.1, 1, 10} from the reward. For an optimistic initialisation bias, we consider setting the final layer's bias to {0.1, 1, 10}. Both methods utilise a count-based intrinsic motivation. For OPIQ and its ablations we set M = 2 since it worked best in preliminary experiments. We consider: C action ∈ {0.1, 1, 10, 100}, C bootstrap ∈ {0.01, 0.1, 1, 10}. For the RND bonus we use the same architecture as the DQN for both the target and predictor networks, except the output is of size 128 instead of |A|. We scale the squared error by β rnd ∈ {0.001, 0.01, 0.1, 1, 10, 100}: For DQN + DORA we use the same architecture for the E-network as the DQN. We add a sigmoid non-linearity to the output and initialise the final layer's weights and bias to 0 as described in . We sweep across the scale of the intrinsic reward β dora ∈ {}. For DQN + DORA OA we use β dora = and sweep across β dora_action ∈ {}. For BSP we use the following architecture: We use K = 10 different bootstrapped DQN heads, and sweep over β bsp ∈ {0.1, 1, 3, 10, 30, 100}. For all methods we run 8 independent runs across the cross-product of all relevant parameters considered. We then sort them by the median episodic reward (largest area underneath the line) and report the median, lower and upper quartiles. The best hyperparameters we found were: OPIQ without Optimistic Bootstrapping: M: 2, C action: 100. OPIQ without Pseudocounts: M: 2, C action: 100, C bootstrap: 0.1. The network used is the standard DQN used for Atari . We use 3 step Q-Learning. Training lasts for 12.5mil timesteps (50mil frames in Atari). is decayed linearly from 1 to 0.01 over 1mil timesteps. We train on a batch of 32 after every 4th timestep with a replay buffer of size 1mil. The target network is updated every 8000 timesteps. For all methods we consider β mmc ∈ {0.005, 0.01, 0.025}. For DQN + PC we consider β ∈ {0.01, 0.1, 1}. For OPIQ and its ablations we set M = 2. We consider: C action ∈ {0.1, 1}, C bootstrap ∈ {0.01, 0.1}, β ∈ {0.01, 0.1}. For the RND bonus we use the same architectures as in (target network is smaller than the learned predictor network) except we use ReLU non-linearity. The output is the same of size 512. We scale the squared error by β rnd ∈ {0.001, 0.01, 0.1, 1}: For BSP we use the same architecture as in . We use K = 10 different bootstrapped DQN heads, and sweep over β bsp ∈ {0.1, 1, 10, 100}. For all methods we run 4 independent runs across the cross-product of all relevant parameters considered. We then sort them by the median maximum episodic reward (largest area underneath the line) and report the median, lower and upper quartiles. The best hyperparameters we found were: OPIQ without Optimistic Bootstrapping: M= 2, C action = 0.1, β mmc = 0.005. We do a single gradient descent step on a minibatch of the 32 most recently visited states. We also recompute the intrinsic rewards when sampling minibatches to train the DQN. The intrinsic reward used for a state s, is the squared error between the predictor network and the target network β rnd ||predictor(s) − target(s)|| 2 2. DQN + DORA: We train the E-values network using n-step SARSA (same n as the DQN) with γ E = 0.99. We maintain a replay buffer of size (batch size * 4) and sample batch size elements to train every timestep. The intrinsic reward we use is s,a). DQN + DORA OA: We train the DQN + DORA agent described above and additionally augment the Q-values used for action selection with. We train each Bootstrapped DQN head on all of the data from the replay buffer (as is done in (; 2018). We normalise the gradients of the shared part of the network by 1/K, where K is the number of heads. The output of each head is Q k + β bsp p k, where p k is a randomly initialised network (of the same architecture as Q k) which is kept fixed throughout training. β bsp is a hyperparameter governing the scale of the prior regularisation. For our experiments on Montezuma's Revenge we additionally mixed the 3 step Q-Learning target with the environmental rewards monte carlo return for the episode. That is, the 3 step targets y t become: If the episode hasn't finished yet, we used 0 for the monte carlo return. Our implementation differs from in that we do not use the intrinsic rewards as part of the monte carlo return. This is because we recompute the intrinsic rewards whenever we are using them as part of the targets for training, and recomputing all the intrinsic rewards for an entire episode (which can be over 1000 timesteps) is computationally prohibitive. E FURTHER E.1 RANDOMISED CHAIN Figure 12: The number of distinct states visited over training for the chain environment. The median across 20 seeds is plotted and the 25%-75% quartile is shown shaded. We can see that OPIQ and ablations explore the environment much more quickly than the count-based baselines. The ablation without optimistic bootstrapping exhibits significantly more variance than the other ablations, showing the importance of optimism during bootstrapping. On this simple task the ablation without count-based intrinsic motivation performs on par with the full OPIQ. This is most likely due to the simpler nature of the environment that makes propagating rewards much easier than the Maze. The importance of directed exploration is made abundantly clear by the -greedy baseline that fails to explore much of the environment. Figure 13 compares OPIQ with differing values of M. We can clearly see that a small value of 0.1 in insufficient exploration, due to the over-exploration of already visited state-action pairs. Additionally if M is too large then the rate of exploration suffers due to the decreased optimism. On this task we found that M = 0.5 performed best, but on the harder Maze environment we found that M = 2 was better in preliminary experiments. E.2 MAZE Figure 14: The Q + -values OPIQ used during bootstrapping with C bootstrap = 0.01. Figure 14 shows the values used during bootstrapping for OPIQ. These Q-values show optimism near the novel state-action pairs which provides an incentive for the agent to return to this area of the state space. E.3 MONTEZUMA'S REVENGE To emphasise the necessity of optimistic Q-value estimates during exploration, we analyse the simple failure case for pessimistically initialised greedy Q-learning provided in the introduction. We use Algorithm 1, but use Q instead of Q + for action selection. We will assume the agent will act greedily with respect to its Q-value estimates and break ties uniformly: Consider the single state MDP in Figure 17 with H = 1. We use this MDP to show that with 0.5 probability pessimistically initialised greedy Q-learning never finds the optimal policy. The agent receives a reward of +1 for selecting the right action and 0.1 otherwise. Therefore the optimal policy is to select the right action. Now consider the first episode: ∀a, Q 1 (s, a) = 0. Thus, the agent selects an action at random with uniform probability. If it selects the left action, it updates: Thus, in the second episode it selects the left action again, since Q 1 (s, L) > 0 = Q 1 (s, R). Our estimate of Q 1 (s, L) never drops below 0.1, and so the right action is never taken. Thus, with probability of 1 2 it never selects the correct action (also a linear regret of 0.9T). This counterexample applies for any non-negative intrinsic motivation (including no intrinsic motivation), and is unaffected if we utilise optimistic bootstrapping or not. Despite introducing an extra optimism term with a tunable hyperparameter M, OPIQ still requires the intrinsic motivation term b T i to ensure it does not under-explore in stochastic environments. We will prove that OPIQ without the intrinsic motivation term b T i does not satisfy Theorem 1. Specifically we will show that there exists a 1 state, 2 action MDP with stochastic reward function such that for all M > 0 the probability of incurring linear regret is greater than the allowed failure probability p. We choose to use stochastic rewards as opposed to stochastic transitions for a simpler proof. Figure 18: The parametrised MDP. The MDP we will use is shown in Figure 18, where λ > 1 and a ∈ s.t p < 1 − a. H = 1, S = 1 and A = 2. The reward function for the left action is stochastic, and will return +1 reward with probability a and 0 otherwise. The reward for the right action is always a/λ. Let p > 0, the probability with which we are allowed to incur a total regret not bounded by the theorem. OPIQ cannot depend on the value of λ or a as they are unknown. ). OPIQ will recover the sub-optimal policy of taking the right action if every time we take the left action we receive a 0 reward. This will happen since our Q + -value estimate for the left action will eventually drop below the Q-value estimate for the right action which is a/λ > 0. The sup-optimal policy will incur a linear regret, which is not bounded by the theorem. Our probability of failure is at least (1 − a) R, where R is the number of times we select the left action, which decreases as R increases. This corresponds to receiving a 0 reward for every one of the R left transitions we take. Note that (1 − a) R is an underestimate of the probability of failure. For the first 2 episodes we will select both actions, and with probability (1 − a) the left action will return 0 reward. Our Q-values will then be: It is possible to take the left action as long as since the optimistic bonus for the right action decays to 0. This then provides a very loose upper bound for R as (λ a) 1/M, which then leads to a further underestimation of the probability of failure. Assume for a contradiction that (1 − a) R < p: This provides our contradiction as we choose λ s.t M > log(λ/a)/ log(log(p)/ log(1 − a)). We can always pick such a λ because log(λ/a) can get arbitrarily close to 0. So our probability of failure (of which (1 − a) R is a severe underestimate) is greater than the allowed probability of failure p. Theorem 1. For any p ∈, with probability at least 1 − p the total regret of Q + is at most OPIQ is heavily based on UCB-H , and as such the proof very closely mirrors its proof except for a few minor differences. For completeness, we reproduce the entirety of the proof with the minor adjustments required for our scheme. The proof is concerned with bounding the regret of the algorithm after K episodes. The algorithm we follow takes as input the value of K, and changes the magnitudes of b T N based on it. We will make use of a corollary to Azuma's inequality multiple times during the proof. Theorem 2. (Azuma's Inequality). Let Z 0,..., Z n be a martingale sequence of random variables such that ∀i∃c i: |Z i − Z i−1 | < c i almost surely, then: Corollary 1. Let Z 0,..., Z n be a martingale sequence of random variables such that ∀i∃c i: |Z i − Z i−1 | < c i almost surely, then with probability at least 1 − δ: (1 − η j) The following properties hold for η i N: Lemma 2. Adapted slightly from Define V For any (s, a, t) ∈ S × A × [H], episode k ≤ K and N = N (s, a, t). Suppose (s, a) was previously taken at step t of episodes k 1,..., k N < k. Then: Proof. We have the following recursive formula for Q + at episode k and timestep t: We can produce a similar formula for Q *: From the Bellman Optimality Equation By definition of P and, with probability at least 1 − δ, the following holds simultaneously for all (s, a, t, k) where N = N (s, a, t) and k 1,..., k N < k are the episodes where (s, a) was taken at step t. Proof. For each fixed (s, a, t) ∈ S × A × [H], let k 0 = 0 and k i = min({k ∈ [K]|k > k i−1, (s k t, a k t) = (s, a)} ∪ {K + 1}) k i is then the episode at which (s, a) was taken at step t for the ith time, or k i = K + 1 if it has been taken fewer than i times. Then the random variable k i is a stopping time. Let F i be the σ-field generated by all the random variables until episode k i step t. Then by Azuma's Inequality we have that with probability at least 1 − 2δ/(SAHK) Then by a Union bound over all τ ∈ [K], we have that with probability at least 1 − 2δ/(SAH): Since From Lemma 1 we have that Since inequality equation 9 holds for all fixed τ ∈ [K] uniformly, it also holds for a random variable τ τ = N = N k (s, a, t) ≤ K. Also note that 1[k i ≤ K] = 1 for all i ≤ N. We can then additionally apply a union bound over all s ∈ S, a ∈ A, t ∈ [H] to give: which holds with probability 1 − 2δ for all (s, a, t, k) ∈ S × A × [H] × [K]. We then rescale δ to δ/2. By Lemma 1 we have that b Note on stochastic rewards: We have assumed so far that the reward function is deterministic for a simpler presentation of the proof. If we allowed for a stochastic reward function, the previous lemmas can be easily adapted to allow for it. Lemma 2 would give us: We will now prove that our algorithm is provably efficient. We will do this by showing it has sub-linear regret, following closely the proof of Theorem 1 from . Theorem 1. For any p ∈, with probability at least 1 − p the total regret of Q + is at most O(H 4 SAT log(SAT /p)) for M ≥ 1 and at most O(H 1+M SAT 1−M + H 4 SAT log(SAT /p)) for 0 < M < 1.

Represent this text to find a much much longer article about it with say hundreds-thousands of words

We augment the Q-value estimates with a count-based bonus that ensures optimism during action selection and bootstrapping, even if the Q-value estimates are pessimistic.

represent this text to find a much much longer article about it with say hundreds-thousands of words

A backward model of previous (state, action) given the next state, i.e. P(s_t, a_t | s_{t+1}), can be used to simulate additional trajectories terminating at states of interest! Improves RL learning efficiency.