title: AutoFS: Automated Feature Selection via Diversity-aware Interactive Reinforcement Learning
authors: Wei Fan Kunpeng Liu Hao Liu Pengyang Wang Yong Ge Yanjie Fu
text: 
  
  
  
  
  
  
  
  
  AutoFS: Automated Feature Selection via Diversity-aware Interactive Reinforcement Learning
  
  
  
  
      Wei Fan1, Kunpeng Liu1, Hao Liu2,  Pengyang Wang1,  Yong Ge3, Yanjie Fu1
  1 Department of Computer Science, University of Central Florida, Orlando 
  
  2 Business Intelligence Lab, Baidu Research, Beijing 
  
  3 Eller College of Management, University of Arizona, Tucson
  
  1{weifan, kunpengliu, pengyang.wang}@knights.ucf.edu, 
   
  2{liuhao30}@baidu.com, 
  3{yongge}@arizona.edu,
  1{Yanjie.Fu}@ucf.edu
  
  •  Yanjie Fu is the contact author.
  
  
  
      Received December 30, 2023; accepted December 30, 2023
  =================================================================================================================================================================================================================================================================================================================================================================================================================================================
  
  
  
  
  
  
  
  In this paper, we study the problem of balancing effectiveness and efficiency in automated feature selection.
  Feature selection is to find the optimal feature subset from large-scale feature space, and is a fundamental intelligence for machine learning and predictive analysis.
  After exploring many feature selection methods, we observe a computational dilemma: 1) traditional feature selection methods (e.g., K-Best, decision tree based ranking, mRMR) are mostly efficient, but difficult to identify the best subset; 
  2) the emerging reinforced feature selection methods automatically navigate feature space to explore the best subset, but are usually inefficient. 
  Are automation and efficiency always apart from each other? Can we bridge the gap between effectiveness and efficiency under automation?
  Motivated by such a computational dilemma, this study is to develop a novel feature space navigation method. 
  To that end, we propose an Interactive Reinforced Feature Selection (IRFS) framework that guides agents by not just self-exploration experience, but also diverse external skilled trainers to accelerate learning for feature exploration.
  Specifically, we formulate the feature selection problem into an interactive reinforcement learning framework. 
  In this framework, we first model two trainers skilled at different searching strategies: (1) KBest based trainer; (2) Decision Tree based trainer. 
  We then develop two strategies: (1) to identify assertive and hesitant agents to diversify agent training, and (2) to enable the two trainers to take the teaching role in different stages to fuse the experiences of the trainers and diversify teaching process.
  Such a hybrid teaching strategy can help agents to learn broader knowledge, and  thereafter be more effective.
  
  Finally, we present extensive experiments on real-world datasets to demonstrate the improved performances of our method: more efficient than existing reinforced selection and more effective than classic selection.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  edition2
  
  
  In this paper, we study the integration of feature selection and interactive reinforcement learning.
  Feature selection is important in data mining and machine learning, which aims to find most relevant features for downstream tasks.
  While traditional feature selection has proven to be effective, an emerging reinforced feature selection method automates the feature subspace exploration and performs better.
  However, the searching space of reinforced feature selection could be very large, making exploration inefficient as it is difficult to find optimal feature subset in short-term; hence, we aim  to  address  the  problem  with  Interactive  Reinforcement Learning (IRL), which guides agents to learn with the help of teacher-like trainers and thus could speed up convergence in the apprenticeship of reinforcement learning. 
  Specifically, in this paper, we first formulate the feature selection problem into an interactive reinforcement learning framework, in which we propose a KBest algorithm based teacher-like trainer, which could guide multiple agents to act more wisely.
  Then, to further improve the IRL algorithm, we propose another trainer with a different style, namely Decision Tree based teacher-like trainer, and we develop a Hybrid Teaching strategy which makes teaching more diversified and effective, by integrating the experience from two proposed trainers. Taught alternatively by different trainers, agents could learn a broader range of knowledge.
  Also, we devise an efficient reward scheme based on the history of agent actions. Instead of sharing reward equally, our scheme weights and assigns reward according to the importance of each feature. Finally, we present extensive experiments which illustrate improved performances of our methods.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  In this paper, we study the problem of integrating feature selection and interactive reinforcement learning. 
  Feature selection is an important preprocessing step in data mining and machine learning, which explores the feature space and finds the most relevant features. 
  Traditional feature selection methods have proven to be effective, which bring lots of advantages to the downstream task (e.g. the improvement of predictive accuracy).
  Recently, an emerging reinforced feature selection method has shown reinforcement learning could be used to automate feature subspace exploration process.
  
  However, though reinforced feature selection performs better than traditional feature selection, its searching space could be very large, making the exploration via reinforcement learning comparatively inefficient. 
  As a result, we aim to address the problem with interactive reinforcement learning,  which could improve the efficiency of reinforced feature selection while still being effective in finding the optimal feature subset. 
  
  Interactive reinforcement learning guides agents to learn with the help of teacher-like trainers. 
  
  
  In this regard, we first formulate the feature selection problem into an interactive reinforcement learning framework, in which we propose a KBest algorithm based teacher-like trainer, which could guide multiple agents to act more wisely. 
  
  Then, to further improve the IRL algorithm, we propose another trainer with a different style, namely Decision Tree based teacher-like trainer, and we develop a Hybrid Teaching strategy which makes teaching more diversified and effective, by integrating the experience from two proposed trainers. Taught alternatively by different trainers, agents could learn a broader range of knowledge. 
  Also, we devise an efficient reward scheme based on the history of agent actions. Instead of sharing reward equally, our scheme weights and assigns reward according to the importance of each feature. Finally, we present extensive experiments which illustrate improved performances of our methods.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § INTRODUCTION
  
  
  
  
  
  In this paper, we aim to develop a new method to balance the effectiveness and efficiency in automated feature selection. 
  Feature selection is to find the optimal feature subset from large feature space, which is a fundamental component for a wide range of data mining and machine learning tasks. 
  
  
  
  
  
  
  
  Classic feature selection methods include: filter methods (e.g., univariate feature selection <cit.>), wrapper methods (e.g., branch and bound algorithms <cit.>), and  embedded methods (e.g., LASSO <cit.>). 
  Recently, the emerging reinforced feature selection methods <cit.> formulate feature selection  into a reinforcement learning task, in order to automate the selection process. 
  Through preliminary exploration, we have observed an interesting computational dilemma in feature selection:
  (1)  classic feature selection methods are mostly efficient, but difficult to identify the best subset; 
  2) the emerging reinforced feature selection methods automatically navigate feature space to explore the best subset, but are usually inefficient. 
  Are automation and efficiency always apart from each other? Can we bridge the gap between effectiveness and efficiency under automation to strive for a balance?
  Motivated by the above dilemma, we focus on developing a new insight for addressing the effectiveness-efficiency balancing problem.
  In this paper, our key insight is to integrate both self-exploration experience and external skilled trainers to simultaneously achieve both effectiveness and efficiency in feature exploration. 
  
  
  To this end, two questions arise toward the goal.
  
  
  First, how can we introduce external skilled trainers to improve feature selection? 
  Recently, Interactive Reinforcement Learning (IRL) <cit.>, as an emerging technology, has shown its superiority on accelerating the exploration of reinforcement learning through learning from external knowledge. 
  Such interactive learning paradigm provides great potential for us to analogously introduce feature selection trainers to guide agents to explore and learn more efficiently.
  
  To this end, we reformulate feature selection problem into an interactive reinforcement learning framework, which we call Interactive Reinforced Feature Selection (IRFS), as illustrated in Figure <ref>.
  
  We propose a multi-agent reinforcement learning component for feature selection, where an agent is a feature selector which issues select/deselect action toward its corresponding feature;
  
  we integrate the interactive learning concept into multi-agent reinforcement learning, in order to bring in prior knowledge from external trainers (e.g., K-Best, decision tree). 
  These trainers can teach agents their selection experiences by recommending actions. The agents take recommended actions to improve the efficiency of their policy functions. 
  
  
  
  
  
  
  
  
  After introducing external trainers, how can we develop a new teaching strategy to enable external trainers to effectively guide agents? 
  Through preliminary exploration, we have identified an interesting property of agent learning; that is, diversity matters. 
  Our initial insight is to diversify the set of external trainers.
  We propose a KBest based trainer and a Decision Tree based trainer, since these algorithms can provide more awareness in measuring feature-feature correlations, and, thus, can teach agents to learn more about feature space. 
  
  However, this initial idea has a weakness: given a static feature set, these trainers (e.g.,  K-Best algorithm) always identify similar feature subsets; that is,  the trainers only provide mostly similar or the same advice to agents every time, which jeopardize agent learning. 
  
  To improve this initial insight, we propose the second insight: to diversify the advice by selecting
  set of agents that will receive advice. Specifically, we randomly select a group of agents each time, which we call participated agents (features), to advise from trainers. In this way, we add randomness to improve robustness. 
  
  Moreover, we propose the third insight:  different agents will receive personalized and tailored advice. 
  Specifically, we categorize the participated agents into: assertive agents (features) and hesitant agents (features). 
  The assertive agents are more confident to select their corresponding features, and, thus, don't need advice from trainers. 
  The hesitant agents are less confident about their decisions, and, thus, need advice from trainers. 
  Finally, we propose the fourth insight: to diversify the teaching process. 
  Specifically, we develop Hybrid Teaching strategy to iteratively let various trainers take the teacher role at different stages, to allow agents to learn a wide range of knowledge.
  
  
  
  
  
  
  
  After introducing external trainers, how can we develop a new teaching strategy to enable external trainers to effectively guide agents? 
  Intuitively, we directly consider a classical feature selection method as a trainer (e.g., K-Best algorithm). 
  However, this initial idea has a weakness: given a static feature set, single trainer always identifies similar feature subsets; that is, the trainer only provides mostly similar or the same advice to agents every time, which jeopardizes agent learning. 
  Through preliminary exploration, we have identified an interesting property of agent learning; that is, diversity matters.
  
  Our initial insight is to diversify the set of external trainers. So, we propose a KBest based trainer and a Decision Tree based trainer, and thus both of they can teach agents to learn more about feature space. 
  
  To improve this initial insight, we propose the second insight: to diversify the advice by selecting set of agents that will receive advice. Specifically, we randomly select a group of agents each step, which we call participated agents (features), to advise from trainers. In this way, we add randomness to improve robustness. 
  
  Moreover, we propose the third insight:  different agents will receive personalized and tailored advice. 
  Specifically, we categorize the participated agents into: assertive agents (features) and hesitant agents (features). 
  The assertive agents are more confident to select their corresponding features, and  thus don't need advice from trainers. 
  The hesitant agents are less confident about their decisions, and thus   need advice from trainers. 
  
  Finally, we propose the fourth insight: to diversify the teaching process. 
  Specifically, we develop Hybrid Teaching strategy to iteratively let various trainers take the teacher role at different stages, to allow agents to learn a wide range of knowledge. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  In summary, in this paper, we develop a diversity-aware interactive reinforcement learning paradigm for feature selection. 
  Our contributions are as follows: 
  (1) We formulate the  feature selection problem into an interactive reinforcement learning framework, which we name Interactive Reinforced Feature Selection (IRFS). 
  (2) We propose a teaching diversification strategy, including (a) diversifying the external trainers; 
  (b) diversifying the advice that agents will receive; 
  (c) diversifying the set of agents that will receive advice; 
  (d) diversifying the teaching style.
  
  (3) We conduct extensive experiments on real-world datasets to demonstrate the advantage of our methods.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  To improve this initial insight, we propose the second insight: to diversify the advice by selecting set of agents that will receive advice. 
  Specifically, we randomly select a group of agents each time, which we call participated agents (features), to advise from external trainers. In this way, we add randomness to improve robustness. 
  
   
  
  Specifically, we categorize the participated selected agents into assertive agents (features) and hesitant agents (features). 
  The assertive agents are more confident to select their corresponding features, and, thus, don't need advice from external trainers. 
  Moreover, we devise a new personalized reward scheme based on the selected frequency ratio of each agent in historical action records.
  Intuitively, if a feature is frequently chosen by an agent, this feature is more likely to play an important role in the ultimate feature subset. Therefore, we assign a higher reward to actions selecting such frequently-chosen features.
  The hesitant agents are less confident about their decisions, and, thus, need are with a higher probability to accept the advice from  trainers. 
  Finally, we propose the fourth insight: to diversify the teaching process. 
  Specifically, we develop Hybrid Teaching strategy to iteratively let various trainers take the teacher role at different stages, to allow agents to learn a wide range of knowledge.
  Third, we develop a hybrid teaching strategy to diversify the teaching process, where different external trainers take the teacher role at different learning stages.
  
  
  
  Also, we devise a new personalized reward scheme based on the selected frequency ratio of each agent in historical action records.
  Intuitively, if a feature is frequently chosen by an agent, this feature is more likely to play an important role in the ultimate feature subset. Therefore, an action that selects such frequently-chosen feature should receive higher reward.
  It doesn't belong to any above challenges, so I tried to merge it with personalized advice part.
  
  In summary, in this paper, we develop a diversity-aware interactive reinforcement learning paradigm for feature selection. 
  Our contributions are as follows: 
  (1) We formulate the  feature selection problem into an interactive reinforcement learning framework, which we call Interactive Reinforced Feature Selection (IRFS).
  
  (2) We propose a teaching diversification strategy, including a, diversifying external trainers; b, diversifying the set of agents that will receive advice; c, diversifying the advice that agents will receive; d, diversifying teaching style.
  We propose a teaching diversification module, which including, a set of diversified external trainers, diversified agent subset selection strategy, personalized reward scheme with diversified agent acceptance confidence, and hybrid teaching strategy.
  (3) We design a new reward scheme that personalizes assigned reward to multiple agents. 
  (4) We conduct extensive experiments on real-world datasets, the results demonstrate the advantage of our methods.
  
  
  
  
  
  
  § PRELIMINARIES
  
  
  
  
  
  We introduce some definitions of the reinforced feature selection; 
  
  we present the problem formulation 
  and the overview of our framework. Table <ref> shows commonly used notations.
  
  
  
  
  
  
  
   §.§ Definitions and Problem Statement
  
  
  
  
  
  Definition 2.1. Agents. Each feature f_i is associated with an agent agt_i. After observing a state of the environment, agents use their policy networks to make decisions on the selection of their corresponding features.
  
  Definition 2.2. Actions. Multiple agents corporately make decisions to select a feature subset. For a single agent, its action space a_i contains two actions, i.e., select and deselect.
  
  
  
  
  
  
  
  
  
  
  Definition 2.3. State. The state s is defined as the representation of the environment, which is the selected feature subset in each step. 
  
  Definition 2.4. Reward. The reward is to inspire the feature subspace exploration.  We measure overall reward based on the predictive accuracy in the downstream task, and equally assign the reward to the agents that select features at current step.
  
  
  
  
  
  
  
  
  
  
  
  
  Definition 2.5. Trainer. In the apprenticeship of reinforcement learning, the actions of agents are immature, and thus it is important to give some advice to the agents. we define the source of the advice as trainer.
  
  Definition 2.6. Advice.  Trainers give agents advice on their actions.
  Multiple agents will follow the advice to take actions.
  
  Definition 2.7. Problem Statement. In this paper, we study the feature selection problem with interactive reinforcement learning. Formally, given a set of features F =  {f_1, f_2,  ..., f_N } where N is the number of features, our aim is to find an optimal feature subset F' ⊆ F which is most appropriate for the downstream task. 
  
  In this work, considering the existence of N features, we create N agents { agt_1, agt_2, ..., agt_N } correspondingly for feature {f_1, f_2,  ..., f_N }. 
  All the agents use their own policy networks {π_1, π_2, ..., π_N } to make decisions to select or deselect their corresponding features, where the actions are denoted by { a_1, a_2, ..., a_N   }. 
  For i ∈ [1,N],  a_i = 1 means agent agt_i decides to select feature f_i; a_i = 0 means agent agt_i decides to deselect feature f_i. Whenever actions are issued in a step, the selected feature subset changes, and then we can derive the state representation s. Finally, the reward is assigned to the certain agents based on their actions. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Framework Overview
  
  
  We first introduce some important components we will use in Interactive Reinforced Feature Selection (IRFS).
  
  
  
    §.§.§  Participated/Non-participated Features (Agents)
   we propose to use classical feature selection methods as trainers. However, 
  the trainers only provide mostly similar or the same advice to agents every time.
  To solve the problem and diversify the advice, we dynamically change the input to the trainer by selecting a set of features, which we call participated features.   We define the participated features as those features that were selected by agents in last step, e.g., if at step t-1 agents select f_2, f_3,f_5, the participated features at step t are f_2, f_3, f_5. The corresponding agents of participated features are participated agents; other agents are non-participated agents which select/deselect their corresponding non-participated features.
  
  
  
    §.§.§  Assertive/Hesitant Features (Agents)
   we dynamically divide the participated features into assertive features and hesitant features, whose corresponding  agents  are  accordingly  called assertive  agents and hesitant  agents. Specifically, at step t, the participated features are divided into: the features decided to be selected by the policies are defined as assertive features, while the features decided to be deselected by the policies are defined as hesitant features. For example, at step t participated features are f_2,f_3,f_5, and policy networks π_2,π_3,π_5 decide to select f_2 and deselect f_3, f_5. Then, assertive features are f_2 and assertive agents are agt_2; hesitant features are f_3,f_5 and hesitant agents are agt_3,agt_5.
  
  
  
  
  
  
  
  
    §.§.§  Initial/Advised Actions
   In each step, agents use their policy networks to firstly make action decisions, which we call initial actions. Then, agents take advice from external trainers and update their actions; we call the actions advised by trainers as advised actions. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Figure <ref> shows an overview of our framework.  There are multiple agents, each of which has its own Deep Q-Network (DQN) <cit.> as policy. At the beginning of each step,  each agent makes initial action decision, from which we can divide all the agents into assertive agents, hesitant agents and non-participated agents. Then trainers provide action advice to agents and agents take the advised actions.
  
  
  After agents take actions, we derive a selected feature subset, whose representation is the state. 
  Following <cit.>, we measure overall reward based on the predictive accuracy in the downstream task, and equally assign them to agents that select features at current step.
  
  Then, a tuple of four components is stored into the memory unit, including the last state, the current state, the actions and the reward. 
  In the training process, for every agent, we select mini-batches from their memory units to train their policy networks, in order to maximize the long-term reward. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § METHOD
  
  
  
  
  
  We present details of Interactive Reinforced Feature Selection and diversification strategies to improve this framework. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Interactive Reinforced Feature Selection with KBest Based Trainer
  
  
  
  
  We propose to formulate the feature selection problem into an interactive reinforcement learning framework called Interactive Reinforced Feature Selection (IRFS). Figure <ref> illustrates the general process of how agents are advised by the trainer. In this formulation, we propose an advanced trainer based on a filter feature selection method, namely KBest based trainer. In our framework, KBest based trainer advises hesitant agents by comparing hesitant features with assertive features. We show how the trainer gives advice as follows:
  
  
  
  
  
  
  
  
  
  
  
  
  
  1) Identifying Assertive/Hesitant Agents
  
  
  
  Given the policy networks {π_1^t, π_2^t, ..., π_N^t  } of multiple agents at step t, each agent makes an initial decision to select or deselect its corresponding feature (not actual actions to take); thus, we get an initial action list at step t, denoted by {a_1^t', a_2^t', ..., a_N^t'}. We record actions that agents has already taken at step t-1, denoted by {a_1^t-1, a_2^t-1, ..., a_N^t-1}. 
  Then, we can find participated features at step t by F_p = { f_i | i∈ [1,N], a_i^t-1 = 1 }. 
  Also, we can identify assertive features as well as assertive agents. Assertive features are F_a = { f_i | i∈ [1,N], f_i ∈ F_p  & a_i^t' = 1 }; assertive agents are H_a = { agt_i | i∈ [1,N], f_i ∈ F_p  & a_i^t' = 1 }. 
  Similarly, we can identify hesitant features and hesitant agents. i.e., hesitant features are F_h = { f_i | i∈ [1,N], f_i ∈ F_p  & a_i^t' = 0 }; hesitant agents are H_h = { agt_i | i∈ [1,N], f_i ∈ F_p  & a_i^t' = 0 }.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  2) Acquiring Advice from KBest Based Trainer
  
  After identifying assertive/hesitant agents, we propose a KBest based trainer, which can advise hesitant agents to update their initial actions. 
  Our perspective is: if the trainer thinks a hesitant feature is even better than half of the assertive features, its corresponding agent should change the action from deselection to selection. 
  
  
  
  
  
  
  
  
  
   Step1: (Warm-up) we obtain the number of assertive features by m = |F_a|, and the number of hesitant features by n = |F_h|. 
  We set the integer k = [m/2 + n]. Then, we use K-Best algorithm to select top k features in F_p; we denote these k features by F_KBest. 
  
   Step2: (Advise) The indices of agents which need to change actions are selected by I_advised = {  i   |  i ∈ [1,N],   f_i ∈ F_h    and    f_i ∈  F_kbest}. 
  
  Finally, we can get the advised action that agt_i will finally take at step t, denoted by a_i^t. Formally:
    
  
      a_i^t={a_i^t', i ∈ I_advised
      a_i^t', i ∉ I_advised
      .
   
  
  
  
  
  
  
  
  
  
   §.§ Interactive Reinforced Feature Selection with Decision Tree Based Trainer 
  
  
  
  In our IRFS framework, we propose another trainer based on a wrapper feature selection method, namely Decision Tree based trainer. The pipeline of IRFS with Decision Tree based trainer guiding agents could also be demonstrated in a two-phase process: (1) Identifying Assertive/Hesitant Agents; (2) Acquiring Advice from Decision Tree Based Trainer. 
  The first phase is the same as phase one in Section <ref>, and the second phase is detailed as follows:
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  Step1: (Warm-up) We denote the participated feature set as F_p = {f_p_1, f_p_2, ..., f_p_t}. We train a decision tree on F_p, and then get the feature importance for each feature, denoted by { imp_f_p_1, imp_f_p_2, ..., imp_f_p_t}. For hesitant features F_h, their importance IMP_h = {imp_f_p_j |  f_p_j∈ F_h  }; for assertive features F_a, their importance IMP_a = {imp_f_p_j |  f_p_j∈ F_a  }. We denote the median of IMP_a by g.
  
  Step2: (Advise) The indices of agents which need to change actions are selected by I_advised = {  p_j  |  f_p_j∈ F_h    and    imp_f_p_j > g  }. Finally, similar to Section <ref>, we can get the advised action based on the I_advised.
  
      a_i^t={a_i^t', i ∈ I_advised
      
      a_i^t', i ∉ I_advised
      .
   
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Interactive Reinforced Feature Selection with Hybrid Teaching Strategy
  
  
  
  In the scenario of human being learning, teaching is commonly divided into several stages, such as elementary school, middle school and university. 
  For human students, in different stages, they are always taught by different teachers who have different teaching styles and different expert knowledge. 
  
  Inspired by the human's learning process, we propose a Hybrid Teaching strategy, which makes agents learn from different trainers in different periods.
  
  
  Specifically, from step 0 to step T, agents are  guided by one trainer.
  
  
  From step T to step 2T, agents are offered advice with the help of another trainer.
  
  Finally, after step 2T, agents explore and learn all by themselves without the trainer. 
  
  
  
  
  
  The Hybrid Teaching strategy diversifies the teaching process and improves the exploration.
  
  In the exploration, the randomness can lead to the similar participated features. Thus, for a single trainer, it gives similar advice when taking similar participated features as input. 
  In this regard, our strategy encourages more diversified exploration by introducing different trainers with different advice though the input is similar. 
  Also, advice given by only one trainer is likely to underperform, making agents suffer from unwise guidance. This strategy can provide a trade-off between advice from two trainers and thus decrease the influence of bad advice. 
  Moreover, agents will lose self-study ability if they always depend on trainers' advice. Considering this, agents finally explore and learn without trainers.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Measuring Reward with Historical Action Records
  
  
  
  
  
  
  In reinforcement learning, precise reward measurement is important; hence, we aim to improve the way to calculate reward. In our IRFS, agent actions advised by trainers are advantageous for long-term reward.
  Considering this, we propose a new personalized reward scheme based on the records of agent actions. Our scheme also combines the predictive accuracy Acc of downstream task and the feature correlation R of the feature subspace for better reward measurement.
  
  
  
  
  
  
  
  
  Action Records In reinforcement learning, agents are more likely to choose advantageous actions for the purpose of maximizing the long-term reward. Intuitively, if a feature is always selected by the agent, this feature could be considered important to the optimal feature subset. Thus, the selection of important features should receive higher reward. In this regard, we propose to personalize the reward towards different agents based on the importance of their corresponding features, which can be quantified by  their selected  frequency  ratio.
  
  Predictive Accuracy We aim to find an optimal feature subset which gets good performance in the downstream task. Naturally, we propose to utilize the performance Acc as part of reward. The higher the performance is, the higher reward agents should receive.
  
  Feature Correlation We also propose to use another characteristic of the selected feature subset to measure the reward: feature correlation. Specifically, a qualified feature subset is usually of low average correlation; high average correlation of features is unfavourable for an optimal feature subset.
  
  
  
  
  Figure <ref> shows the general process of how we measure reward. We introduce the steps of reward scheme as follows:
  
  
  
  
  
  Step1: Given agent actions {a_1^t, a_2^t, ..., a_N^t} at step t, we get a selected feature subset F_s = { f_i | a_i^t = 1 }. Then, we can calculate predictive accuracy Acc of the downstream task on the selected features.
  
  
  
  Step2: We use Pearson correlation coefficient <cit.> to quantify the correlation of selected feature subset. Formally, Pearson correlation coefficient ρ is given by: 
  
  
  
  
  
      ρ_X,Y = cov(X,Y)/σ_Xσ_Y
  
  where X, Y are two features, σ is standard deviation, cov denotes the covariance. Then, the feature correlation of the selected feature subset, denoted by R, is computed as the sum of pairwise Pearson correlation coefficient. Formally:
  
      R = 1/ |F_s|^2∑_ f_u, f_v ∈ F_sρ_f_u,f_v
  
  
  
  
  Step3: We record actions and denote historical action records by {h_1^t, h_2^t, ...,h_N^t}, where h_i^t = (a_i^0, a_i^1, ..., a_i^t) for agt_i at step t.
  
  
  
      
      
  
  Next, we can use action record to calculate the importance of every feature, which is used to weight reward towards agents. For agent agt_i at step t, its reward weight is denoted by W_i^t:
  
      W_i^t = ∑ h_i^t/∑_i=1^N∑ h_i^t
  
  
  Finally, we measure the reward by combining the predictive accuracy, the feature correlation and the action records. Formally, for agent agt_i at step t, its reward is measured by:
  
      r_i^t={
       W_i^t(Acc - p R), a_i^t = 1  
      
      0, a_i^t = 0 
      .
  
  where p is a tuning parameter, Acc is predictive accuracy, R is feature correlation.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § EXPERIMENTAL RESULTS
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Data Description
  
  
  
  
  
    §.§.§ Forest Cover (FC) Dataset
   This dataset is from Kaggle[https://www.kaggle.com/c/forest-cover-type-prediction/data] and includes 15120 samples and 54 different features.
  
  
  
  
  
    §.§.§ Spambase Dataset
   
  This dataset includes 4601 samples and 57 different features, indicating whether a particular word or character was frequently occurring in the email <cit.>. 
  
  
  
    §.§.§ Insurance Company Benchmark (ICB) Dataset
   This dataset contains information about customers, which consists of 86 variables and 5000 samples 
  <cit.>. 
  
  
  
    §.§.§ Musk Dataset
   
  It has 6598 samples and 168 features, aiming to classify the label `musk' and `non-musk' <cit.>.
  
  
  
  
  
  
  
   §.§ Evaluation Metrics
  
  
  
  
  
  Best Acc: Accuracy is the ratio of the number of correct predictions to the number of all predictions. 
  In feature selection, an important task is to find the optimal feature subset, which has good performance in the downstream task. Considering this, we use Best Acc (BA) to stand for the performance of feature selection, which is given by BA_l = max(Acc_i, Acc_i+1, ..., Acc_
  i+l), where i is the beginning step and l is the number of exploration steps. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Baseline Algorithm
  
  
  
  
  
  
  
  (1) K-Best Feature Selection. K-Best algorithm <cit.> ranks features by their ranking scores with the label vector. Then, it selects the top k highest scoring features. In the experiments, we set k equals to half of the number of input features.
  
  (2) Decision Tree Recursive Feature Elimination (DT-RFE). 
  DT-RFE <cit.> first trains a decision tree on all the features and get the feature importance; then it recursively deselects the least important features. In the experiments, we set the selected feature number half of the feature space.
  
  
  
  (3) mRMR. The mRMR <cit.> ranks features by minimizing feature’s redundancy and maximizing their relevance and selects the top-k ranked features as the  result. In experiments, we set k equals to half of the number of input features.
  
  (4) LASSO. LASSO <cit.> conducts feature selection and shrinkage via l1 penalty. In this method, features whose coefficients are 0 will be dropped. All the parameter settings are the same as <cit.>.
  
  (5) Multi-agent Reinforcement Learning Feature Selection (MARLFS). MARLFS <cit.> is a basic multi-agent reinforced feature selection method, which could be seen as a variant of our proposed IRFS without any trainer. 
  
  
  In the experiments, our KBest based trainer uses mutual information to select features, supported by scikit-learn. 
  Our Decision Tree based teacher uses decision tree classifier with default parameters in scikit-learn. 
  In state representation, following previous work <cit.>, we utilize graph convolutional network (GCN) <cit.> to update features. 
  In the experience replay <cit.>, each agent has its memory unit. For agent agt_i at step t, we store a tuple {s_i^t, r_i^t, s_i^t+1, a_i^t} to the memory unit. 
  The policy networks are set as two linear layers of 128 middle states with ReLU as activation function. In the exploration process, the discount factor γ is set to 0.9, and we use ϵ-greedy exploration with ϵ equals to 0.9.
  To train the policy networks, we select mini-batches with 16 as batch size and use Adam Optimizer with a learning rate of 0.01. 
  
  To compare fairly with baseline algorithms, we fix the downstream task as a decision tree classifier with default parameters in scikit-learn. We randomly split the data into train data (90%) and test data (10%). 
  All the evaluations are performed on Intel E5-1680 3.40GHz CPU in a x64 machine with 128GB RAM.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Overall Performance
  
  
  
  
  We compare our proposed method with baseline methods in terms of overall Best Acc on different real-world datasets. In general, Figure <ref> shows our proposed Interactive Reinforced Feature Selection (IRFS) method achieves the best overall performance on four datasets. The basic reinforced feature selection method (MARLFS) has better performance than traditional feature selection methods in most cases, while our IRFS shows further improvement compared to MARLFS.
  
  
  
  
  
   §.§ Study of Interactive Reinforced Feature Selection
  
  
  
  We aim to study the impacts of our interactive reinforced feature selection methods on the efficiency of exploration. Our main study subjects include KBest based trainer, Decision Tree based trainer and Hybrid Teaching strategy. 
  Accordingly, we consider four different methods: 
  (i) MARLFS: The basic reinforced feature selection method, which could be seen as a variant of IRFS without any trainer.
  (ii) IRFS with KBT (KBest based Trainer): a variant of IRFS with only KBT as the trainer. 
  (iii) IRFS with DTT (Decision Tree based Trainer): a variant IRFS with only DTT as the trainer. 
  (iv) IRFS with HT (Hybrid Teaching strategy): a variant of IRFS using Hybrid Teaching strategy with two proposed trainers.
  Figure <ref> shows the comparisons of best accuracy over exploration steps on four datasets. We can observe that both KBT and DTT could reach higher accuracy than MARLFS in few steps (1500 steps), which signifies the trainer's guidance improves the exploration efficiency by speeding up the process of finding optimal subsets. Also, IRFS with HT could find better subsets in short-term compared to MARLFS, KBT and DTT. A potential interpretation is Hybrid Teaching strategy integrates experience from two trainers and takes advantage of broader range of knowledge, leading to the further improvement.
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Study of Personalized Reward Scheme
  
  
  
  We aim to study the impacts of our Personalized Reward Scheme (PRS) in our framework. So, we consider two variant methods: (i) PRS: a variant of IRFS without trainers which utilizes our proposed PRS. (ii) HT+PRS: a variant of IRFS which uses Hybrid Teaching (HT) strategy and measures reward with PRS. 
  Figure <ref> demonstrates comparisons of performance in best accuracy over steps. An observation stands out is that the variant methods exceed original reinforced feature selection methods, as methods with PRS always find better feature subsets faster than MARLFS. Moreover, with PRS, HT method is further improved and achieves better performance in short-termed feature subspace exploration. These observations demonstrate our proposed reward scheme improves our the reinforced feature selection.
  
  
  
  
  
  
  
  
  
   §.§ Study of Variant Methods
  
  
  
  
  
  
  In this section, we present the performance comparison of variant methods in terms of Best Acc and Ave Acc. Figure <ref> shows the Best Acc comparison of different methods. In all cases we observe an improvement with respect to the MARLFS baseline, yet HT+PRS outperforms all the other methods. The improved performance reveals the combination of Hybrid Teaching and Personalized Reward Scheme  yields better results, while both the teaching process and precise reward measurement improve the effectiveness in finding the optimal feature subset.
  
  
  
  We evaluate our methods in terms of Ave Acc on four different datasets. 
  Figure <ref> shows the result of average performance, where we observe all the proposed methods exceed the basic MARLFS. 
  We observe the improvement from trainers and PRS, which reveals after trainers guide agents to act wisely, the advised actions are beneficial for higher average exploration quality, and a precise reward scheme is also  helpful for feature subspace exploration. 
   
  
  
  
  
  
  
   §.§ Study of the action-changed agents
  
  
  
  In each exploration step, we record the number of hesitant agents that take advice and change their actions from deselection to selection. Figure <ref> shows the percentage of action-changed agents in the exploration, where 0∼9 denotes the number of action-changed agents. 
  For example, in Figure <ref>, in 28% of the exploration steps, there are 2 agents that change their actions.
  We observe in most steps the number of agents that change actions is non-zero, which signifies the improved performance is due to these advised actions that actually work for better exploration.
  Another observation is Musk Dataset is darker than other datasets, demonstrating there are more action-changed agents. This is because the input features of Musk Dataset far are more than other datasets, and our trainers would give more advice to agents. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
   §.§ Study of Personalised Reward Scheme
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  § RELATED WORK
  
  
  
  
  
  
  
  
  
  
  
  Traditional feature selection can be grouped into: 
  (1) Filter methods calculate relevance scores, rank features and then select top-ranking ones (e.g. univariate feature selection <cit.>). (2) Wrapper methods make use of predictors, considering the prediction performance as objective function <cit.>. (e.g. branch and bound algorithms <cit.>). (3) Embedded methods directly incorporate feature selection as part of predictors.(e.g. LASSO <cit.>).  
  
  
  Recently, reinforcement learning <cit.> has been applied into different domains, such as robotics <cit.>, urban computing <cit.>, and feature selection. 
  
  Some reinforced feature selection studies create a single agent to make decisions <cit.> but its action space is too large. Others create multi-agents to make decisions <cit.>. 
  
  Reinforcement Learning <cit.> is a good method to address optimal decision-making problem. 
  An intuitive idea to speed up the learning process is to include external advice in the apprenticeship <cit.>. In Interactive Reinforcement Learning (IRL), an action is interactively encouraged by a trainer with prior knowledge <cit.>. 
  Using a trainer to directly advise on future actions is known as policy shaping <cit.>. 
  For advice, they could come from humans and robots <cit.> and also from previously trained artificial agent <cit.>. 
  
  
  
  
  
  
  
  
  
  
  
  
  
  § CONCLUSION
  
  
  
  
  In this paper, we studied the problem of balancing the effectiveness and efficiency of automated feature selection. We formulate feature selection into an interactive reinforcement learning framework, where we propose KBest based trainer and Decision Tree based trainer. To diversify the teaching process, we identify assertive and hesitant agents to give advice, and we develop a Hybrid Teaching strategy to integrate experience from two trainers. 
  
  Finally, we present extensive experiments which illustrate the improved performance.
  
  
  
  § ACKNOWLEDGMENT
  
  
  
  This research was partially supported by the National Science Foundation (NSF) via the grant numbers: 1755946, I2040950, 2006889.
  
  
  IEEEtran
   
  
  
  
  
  
  
  § EASE OF USE
  
  
  
  
   §.§ Maintaining the Integrity of the Specifications
  
  
  The IEEEtran class file is used to format your paper and style the text. All margins, 
  column widths, line spaces, and text fonts are prescribed; please do not 
  alter them. You may note peculiarities. For example, the head margin
  measures proportionately more than is customary. This measurement 
  and others are deliberate, using specifications that anticipate your paper 
  as one part of the entire proceedings, and not as an independent document. 
  Please do not revise any of the current designations.
  
  
  
  § PREPARE YOUR PAPER BEFORE STYLING
  
  Before you begin to format your paper, first write and save the content as a 
  separate text file. Complete all content and organizational editing before 
  formatting. Please note sections <ref>–<ref> below for more information on 
  proofreading, spelling and grammar.
  
  Keep your text and graphic files separate until after the text has been 
  formatted and styled. Do not number text heads— will do that 
  for you.
  
  
  
   §.§ Abbreviations and Acronyms
  
  Define abbreviations and acronyms the first time they are used in the text, 
  even after they have been defined in the abstract. Abbreviations such as 
  IEEE, SI, MKS, CGS, ac, dc, and rms do not have to be defined. Do not use 
  abbreviations in the title or heads unless they are unavoidable.
  
  
  
   §.§ Units
  
  
  
    * Use either SI (MKS) or CGS as primary units. (SI units are encouraged.) English units may be used as secondary units (in parentheses). An exception would be the use of English units as identifiers in trade, such as “3.5-inch disk drive”.
  
    * Avoid combining SI and CGS units, such as current in amperes and magnetic field in oersteds. This often leads to confusion because equations do not balance dimensionally. If you must use mixed units, clearly state the units for each quantity that you use in an equation.
  
    * Do not mix complete spellings and abbreviations of units: “Wb/m2” or “webers per square meter”, not “webers/m2”. Spell out units when they appear in text: “. . . a few henries”, not “. . . a few H”.
  
    * Use a zero before decimal points: “0.25”, not “.25”. Use “cm3”, not “cc”.)
  
  
  
  
   §.§ Equations
  
  Number equations consecutively. To make your 
  equations more compact, you may use the solidus ( / ), the exp function, or 
  appropriate exponents. Italicize Roman symbols for quantities and variables, 
  but not Greek symbols. Use a long dash rather than a hyphen for a minus 
  sign. Punctuate equations with commas or periods when they are part of a 
  sentence, as in:
  
      a+b=γ
  
  
  Be sure that the 
  symbols in your equation have been defined before or immediately following 
  the equation. Use “(<ref>)”, not “Eq. (<ref>)” or “equation (<ref>)”, except at 
  the beginning of a sentence: “Equation (<ref>) is . . .”
  
  
  
   §.§ -Specific Advice
  
  
  Please use “soft” (e.g., ) cross references instead
  of “hard” references (e.g., ). That will make it possible
  to combine sections, add equations, or change the order of figures or
  citations without having to go through the file line by line.
  
  Please don't use the  equation environment. Use
   or  instead. The 
  environment leaves unsightly spaces around relation symbols.
  
  Please note that the  environment in 
  will increment the main equation counter even when there are no
  equation numbers displayed. If you forget that, you might write an
  article in which the equation numbers skip from (17) to (20), causing
  the copy editors to wonder if you've discovered a new method of
  counting.
  
   does not work by magic. It doesn't get the bibliographic
  data from thin air but from .bib files. If you use  to produce a
  bibliography you must send the .bib files. 
  
   can't read your mind. If you assign the same label to a
  subsubsection and a table, you might find that Table I has been cross
  referenced as Table IV-B3. 
  
   does not have precognitive abilities. If you put a
   command before the command that updates the counter it's
  supposed to be using, the label will pick up the last counter to be
  cross referenced instead. In particular, a  command
  should not go before the caption of a figure or a table.
  
  Do not use  inside the  environment. It
  will not stop equation numbers inside  (there won't be
  any anyway) and it might stop a wanted equation number in the
  surrounding equation.
  
  
  
   §.§ Some Common Mistakes
  
  
  
    * The word “data” is plural, not singular.
  
    * The subscript for the permeability of vacuum μ_0, and other common scientific constants, is zero with subscript formatting, not a lowercase letter “o”.
  
    * In American English, commas, semicolons, periods, question and exclamation marks are located within quotation marks only when a complete thought or name is cited, such as a title or full quotation. When quotation marks are used, instead of a bold or italic typeface, to highlight a word or phrase, punctuation should appear outside of the quotation marks. A parenthetical phrase or statement at the end of a sentence is punctuated outside of the closing parenthesis (like this). (A parenthetical sentence is punctuated within the parentheses.)
  
    * A graph within a graph is an “inset”, not an “insert”. The word alternatively is preferred to the word “alternately” (unless you really mean something that alternates).
  
    * Do not use the word “essentially” to mean “approximately” or “effectively”.
  
    * In your paper title, if the words “that uses” can accurately replace the word “using”, capitalize the “u”; if not, keep using lower-cased.
  
    * Be aware of the different meanings of the homophones “affect” and “effect”, “complement” and “compliment”, “discreet” and “discrete”, “principal” and “principle”.
  
    * Do not confuse “imply” and “infer”.
  
    * The prefix “non” is not a word; it should be joined to the word it modifies, usually without a hyphen.
  
    * There is no period after the “et” in the Latin abbreviation “et al.”.
  
    * The abbreviation “i.e.” means “that is”, and the abbreviation “e.g.” means “for example”.
  
  An excellent style manual for science writers is <cit.>.
  
  
  
   §.§ Authors and Affiliations
  
  The class file is designed for, but not limited to, six authors. A 
  minimum of one author is required for all conference articles. Author names 
  should be listed starting from left to right and then moving down to the 
  next line. This is the author sequence that will be used in future citations 
  and by indexing services. Names should not be listed in columns nor group by 
  affiliation. Please keep your affiliations as succinct as possible (for 
  example, do not differentiate among departments of the same organization).
  
  
  
   §.§ Identify the Headings
  
  Headings, or heads, are organizational devices that guide the reader through 
  your paper. There are two types: component heads and text heads.
  
  Component heads identify the different components of your paper and are not 
  topically subordinate to each other. Examples include Acknowledgments and 
  References and, for these, the correct style to use is “Heading 5”. Use 
  “figure caption” for your Figure captions, and “table head” for your 
  table title. Run-in heads, such as “Abstract”, will require you to apply a 
  style (in this case, italic) in addition to the style provided by the drop 
  down menu to differentiate the head from the text.
  
  Text heads organize the topics on a relational, hierarchical basis. For 
  example, the paper title is the primary text head because all subsequent 
  material relates and elaborates on this one topic. If there are two or more 
  sub-topics, the next level head (uppercase Roman numerals) should be used 
  and, conversely, if there are not at least two sub-topics, then no subheads 
  should be introduced.
  
  
  
   §.§ Figures and Tables
  
  
  
    
  Positioning Figures and Tables Place figures and tables at the top and 
  bottom of columns. Avoid placing them in the middle of columns. Large 
  figures and tables may span across both columns. Figure captions should be 
  below the figures; table heads should appear above the tables. Insert 
  figures and tables after they are cited in the text. Use the abbreviation 
  “Fig. <ref>”, even at the beginning of a sentence.
  
  
  
  
  
  Figure Labels: Use 8 point Times New Roman for Figure labels. Use words 
  rather than symbols or abbreviations when writing Figure axis labels to 
  avoid confusing the reader. As an example, write the quantity 
  “Magnetization”, or “Magnetization, M”, not just “M”. If including 
  units in the label, present them within parentheses. Do not label axes only 
  with units. In the example, write “Magnetization (A/m)” or “Magnetization 
  {A[m(1)]}”, not just “A/m”. Do not label axes with a ratio of 
  quantities and units. For example, write “Temperature (K)”, not 
  “Temperature/K”.
  
  
  
  § ACKNOWLEDGMENT
  
  
  The preferred spelling of the word “acknowledgment” in America is without 
  an “e” after the “g”. Avoid the stilted expression “one of us (R. B. 
  G.) thanks …”. Instead, try “R. B. G. thanks…”. Put sponsor 
  acknowledgments in the unnumbered footnote on the first page.
  
  
  
  § REFERENCES
  
  
  Please number citations consecutively within brackets <cit.>. The 
  sentence punctuation follows the bracket <cit.>. Refer simply to the reference 
  number, as in <cit.>—do not use “Ref. <cit.>” or “reference <cit.>” except at 
  the beginning of a sentence: “Reference <cit.> was the first …”
  
  Number footnotes separately in superscripts. Place the actual footnote at 
  the bottom of the column in which it was cited. Do not put footnotes in the 
  abstract or reference list. Use letters for table footnotes.
  
  Unless there are six authors or more give all authors' names; do not use 
  “et al.”. Papers that have not been published, even if they have been 
  submitted for publication, should be cited as “unpublished” <cit.>. Papers 
  that have been accepted for publication should be cited as “in press” <cit.>. 
  Capitalize only the first word in a paper title, except for proper nouns and 
  element symbols.
  
  For papers published in translation journals, please give the English 
  citation first, followed by the original foreign-language citation <cit.>.
  
  00
  b1 G. Eason, B. Noble, and I. N. Sneddon, “On certain integrals of Lipschitz-Hankel type involving products of Bessel functions,” Phil. Trans. Roy. Soc. London, vol. A247, pp. 529–551, April 1955.
  b2 J. Clerk Maxwell, A Treatise on Electricity and Magnetism, 3rd ed., vol. 2. Oxford: Clarendon, 1892, pp.68–73.
  b3 I. S. Jacobs and C. P. Bean, “Fine particles, thin films and exchange anisotropy,” in Magnetism, vol. III, G. T. Rado and H. Suhl, Eds. New York: Academic, 1963, pp. 271–350.
  b4 K. Elissa, “Title of paper if known,” unpublished.
  b5 R. Nicole, “Title of paper with only first word capitalized,” J. Name Stand. Abbrev., in press.
  b6 Y. Yorozu, M. Hirano, K. Oka, and Y. Tagawa, “Electron spectroscopy studies on magneto-optical media and plastic substrate interface,” IEEE Transl. J. Magn. Japan, vol. 2, pp. 740–741, August 1987 [Digests 9th Annual Conf. Magnetics Japan, p. 301, 1982].
  b7 M. Young, The Technical Writer's Handbook. Mill Valley, CA: University Science, 1989.
  
  
  
  
  IEEE conference templates contain guidance text for composing and formatting conference papers. Please ensure that all template text is removed from your conference paper prior to submission to the conference. Failure to remove the template text from your paper may result in your paper not being published.
  
  
  
  
  
  