Due to delay and energy constraints, a cognitive radio may not be able to perform spectrum sensing in all available channels. Therefore, a sensing policy is needed to decide which channels to sense. The channel selection problem is the problem of designing such a sensing policy to maximize throughput while avoiding interference to primary users. The channel selection problem can be formulated as a reinforcement learning problem. Channel selection schemes that employ reinforcement machine learning algorithms are vulnerable to belief manipulation attacks that contaminate the knowledge base of the learning algorithms. In this paper, we analyze the security of channel selection algorithms that are based on reinforcement learning and propose mitigation techniques that make these algorithms more robust against belief manipulation attacks.
@cite_4: Opportunistic spectrum access (OSA) that allows secondary users to independently search for and exploit instantaneous spectrum availability is considered. The design objective is to maximize the throughput of a secondary user while limiting the probability of colliding with primary users. Integrated in the joint design are three basic components: a spectrum sensor that identifies spectrum opportunities, a sensing strategy that determines which channels in the spectrum to sense, and an access strategy that decides whether to access based on potentially erroneous sensing outcomes. This joint design is formulated as a constrained partially observable Markov decision process (POMDP), and a separation principle is established. The separation principle reveals the optimality of myopic policies for the design of the spectrum sensor and the access strategy, leading to closed-form optimal solutions. Furthermore, it decouples the design of the sensing strategy from that of the spectrum sensor and the access strategy, and reduces the constrained POMDP to an unconstrained one. Numerical examples are provided to study the tradeoff between sensing time and transmission time, the interaction between the physical layer spectrum sensor and the MAC layer sensing and access strategies, and the robustness of the ensuing design to model mismatch.
@cite_8: This paper considers the design of efficient strategies that allow cognitive users to choose frequency bands to sense and access among multiple bands with unknown parameters. First, the scenario in which a single cognitive user wishes to opportunistically exploit the availability of frequency bands is considered. By adopting tools from the classical bandit problem, optimal as well as low complexity asymptotically optimal solutions are developed. Next, the multiple cognitive user scenario is considered. The situation in which the availability probability of each channel is known is first considered. An optimal symmetric strategy that maximizes the total throughput of the cognitive users is developed. To avoid the possible selfish behavior of the cognitive users, a game-theoretic model is then developed. The performance of both models is characterized analytically. Then, the situation in which the availability probability of each channel is unknown a priori is considered. Low-complexity medium access protocols, which strike an optimal balance between exploration and exploitation in such competitive environments, are developed. The operating points of these low-complexity protocols are shown to converge to those of the scenario in which the availability probabilities are known. Finally, numerical results are provided to illustrate the impact of sensing errors and other practical considerations.
@cite_6: This paper considers the following stochastic control problem that arises in opportunistic spectrum access: a system consists of n channels where the state (“good” or “bad”) of each channel evolves as independent and identically distributed Markov processes. A user can select exactly k channels to sense and access (based on the sensing result) in each time slot. A reward is obtained whenever the user senses and accesses a “good” channel. The objective is to design a channel selection policy that maximizes the expected discounted total reward accrued over a finite or infinite horizon. In our previous work we established the optimality of a greedy policy for the special case of k=1 (i.e., single channel access) under the condition that the channel state transitions are positively correlated over time. In this paper we show under the same condition the greedy policy is optimal for the general case of k≥1; the methodology introduced here is thus more general. This problem may be viewed as a special case of the restless bandit problem, with multiple plays. We discuss connections between the current problem and existing literature on this class of problems.
@cite_0: We focus on an opportunistic communication system consisting of multiple independent channels with time-varying states. With limited sensing, a user can only sense and access a subset of channels and accrue rewards determined by the state of the sensed channels. We formulate the problem of optimal sequential channel probing as a restless multi-armed bandit process, for which a powerful index policy-Whittle's index policy-can be implemented based on the indexability of the system. Exploiting the underlying structure of the multi-channel opportunistic access problem, we establish the indexability and obtain the Whittle's index in closed-form, which leads to a direct implementation of Whittle's index policy with little complexity. Furthermore, we show that Whittle's index policy is equivalent to the myopic policy when channels are statistically identical.
@cite_11: We consider a multi-channel opportunistic communication system where the states of these channels evolve as independent and statistically identical Markov chains (the Gilbert- Elliot channel model). A user chooses one channel to sense and access in each slot and collects a reward determined by the state of the chosen channel. The problem is to design a sensing policy for channel selection to maximize the average reward, which can be formulated as a multi-arm restless bandit process. In this paper, we study the structure, optimality, and performance of the myopic sensing policy. We show that the myopic sensing policy has a simple robust structure that reduces channel selection to a round-robin procedure and obviates the need for knowing the channel transition probabilities. The optimality of this simple policy is established for the two-channel case and conjectured for the general case based on numerical results. The performance of the myopic sensing policy is analyzed, which, based on the optimality of myopic sensing, characterizes the maximum throughput of a multi-channel opportunistic communication system and its scaling behavior with respect to the number of channels. These results apply to cognitive radio networks, opportunistic transmission in fading environments, downlink scheduling in centralized networks, and resource-constrained jamming and anti-jamming.
The formulation of spectrum learning problem as a restless multi-armed bandit process is investigated in @cite_4 -- @cite_8 . It is proved that when channels are identical and independent the myopic policy is the optimal policy @cite_11 -- @cite_6 , and that this policy is a special case of Whittle's index policy for the restless bandit problem which can be computed for non-identical channels as well @cite_0 . An asymptotically optimal policy is proposed in @cite_8 for a more realistic case where the policy does not require any prior statistical knowledge about the traffic pattern and the channels are different. Despite all these work that assume a non-adversarial environment, this paper investigates the spectrum learning problem in an adversarial environment where active attackers aim to reduce the throughput of the cognitive radio network.