ervation Wireless environment Learning Action Decision Figure 10.4 The reinforcement learning cycle. 10.2 ML-Based Methodologies for SS Two Q-learning-based spectrum assignment methods were developed in [239]. The inde- pendent Q-learning-based scheme was designed for a case in which the SUs could not exchange information, while the collaborative Q-learning-based scheme was designed for a case in which information could be exchanged among SUs. It was shown that the col- laborative Q-learning-based assignment method performed better than the independent Q-learning-based method. Most cases fell short of the perfect CSI, and imperfect CSI can degrade the performance of system spectrum selection. To overcome this challenge, an RL-based robust decentral- ized multi-agent resource allocation scheme was proposed in [95]. They introduced cloud computing to enlarge storage space, reduce operating expenditures, and enhance the flexi- bility of cooperation. A cooperative framework in a multi-agent system was then developed to improve the performance of their proposed scheme in terms of network capacity, outage probability, and convergence speed. To better manage users in a CRN, the clustering operation organizes SUs into logical groups based on their common features. Clustering can provide network scalability, spectrum stability, and fulfill cooperative tasks. In each cluster, one SU can either act as a leader (or cluster head) that manages essential CR operations, such as channel sensing and routing, or as a member node that associates itself with the cluster head. Cluster size represents the number of nodes in a cluster and affects various performance metrics. In the CRN, the cluster size adjustment and cluster head selection can significantly impact system performance. In [89], the authors proposed a first-of-its-kind cluster size adjustment scheme based on RL. The proposed scheme adapts the cluster size according to the number of white spaces to improve network scalability and cluster stability. It was shown that their 