Generative Diffusion Model (GDM) for Optimization of Wi-Fi Networks
Tie Liu Xuming Fang Rong He






Generative Diffusion Model (GDM) for Optimization of Wi-Fi Networks
    
Tie Liu, Xuming Fang, Rong He
Key Laboratory of Information Coding and Transmission, Southwest Jiaotong University, Chengdu, China

tliu@my.swjtu.edu.cn, xmfang@swjtu.edu.cn, rhe@swjtu.edu.cn

The work was supported in part by NSFC under Grant 62071393, and Fundamental Research Funds for the Central Universities under Grant 2682023ZTPY058. (Corresponding author: Xuming Fang)

    April 29, 2024
===============================================================================================================================================================================================================================================================================================================================================================================================




Generative Diffusion Models (GDMs), have made significant strides in modeling complex data distributions across diverse domains. Meanwhile, Deep Reinforcement Learning (DRL) has demonstrated substantial improvements in optimizing Wi-Fi network performance. Wi-Fi optimization problems are highly challenging to model mathematically, and DRL methods can bypass complex mathematical modeling, while GDMs excel in handling complex data modeling. Therefore, combining DRL with GDMs can mutually enhance their capabilities. The current MAC layer access mechanism in Wi-Fi networks is the Distributed Coordination Function (DCF), which dramatically declines in performance with a high number of terminals. In this paper, we apply diffusion models to deep deterministic policy gradient, namely the Deep Diffusion Deterministic Policy (D3PG) algorithm to optimize the Wi-Fi performance. Although such integrations have been explored previously, we are the first to apply it to Wi-Fi network performance optimization. We propose an access mechanism that jointly adjusts the contention window and frame length based on the D3PG algorithm. Through simulations, we have demonstrated that this mechanism significantly outperforms existing Wi-Fi standards in dense Wi-Fi scenarios, maintaining performance even as the number of users sharply increases.



  Wi-Fi, generative diffusion models (GDMs),  deep reinforcement learning, networks optimization.




§ INTRODUCTION

As Wi-Fi technology evolves, the rapid development of advanced network applications such as 8K video streaming, augmented/virtual reality (AR/VR), and remote surgery poses unprecedented challenges. These demands have significantly surpassed the capabilities of existing standards such as 802.11ax, driving the development of the next generation of Wi-Fi technologies, particularly IEEE 802.11be (Wi-Fi 7). These new technologies aim to deliver extremely high throughput (EHT) and reduced latency to meet the complex requirements of future applications.

Meanwhile, traditional mathematical modeling approaches struggle with the stochastic and variable nature of wireless channels and the complex interdependences among parameters. In this paper, we explore the optimization of the MAC layer access mechanism by jointly adjusting the contention window and frame length, which are interdependent parameters. Longer data frames occupy the channel for extended periods, increasing the probability of collisions with other devices. For instance, in environments with high collision probabilities, using longer frames could lead to more retransmissions and reduce network efficiency. This issue can be mitigated either by increasing the contention window or by reducing the frame length. Recent advancements in machine learning, especially reinforcement learning algorithms, offer effective solutions by bypassing complex mathematical modeling steps through their adaptive learning capabilities and data processing ability. These algorithms are increasingly pivotal in optimizing Wi-Fi networks to meet the evolving demands.

In recent years, Generative Artificial Intelligence (GAI) has made remarkable strides in computer vision<cit.>. GAI technologies excel at creating innovative content by analyzing extensive datasets that include text, images, and music. Particularly in the field of data augmentation<cit.>, generative models synthesize additional data to bolster machine learning models, providing critical support in scenarios where data is scarce or privacy concerns are paramount. For network optimization tasks, such as the joint optimization of interrelated parameters, GAI is invaluable for managing highly complex data distributions. Nevertheless, the challenge lies in determining the optimal combination of parameter adjustments for various environments.

Du et al. <cit.> introduced how generative diffusion models can be applied to network optimization problem. They provide an easy example of power optimization, which shows promising performance in sum rate compared to DRL methods. In another work of Du et al. <cit.>, they address the increasing demand for efficient content generation capabilities in the next generation of Internet with the emergence of the Metaverse. They introduce the Artificial Intelligence Generative Optimal Decision algorithm (AGOD), making it possible to apply generative diffusion theory to network performance optimization.

The current Wi-Fi standards utilize the Medium Access Control (MAC) layer mechanism known as DCF, which is based on Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA) and Binary Exponential Backoff (BEB). The key point of DCF is listen before talk. Before sending data, the device first checks whether the wireless channel is idle. If the channel is busy, the device will wait for a random backoff period selected within [0,CW_max], which doubles with each subsequent collision, before checking the channel again. If the channel is free, the device will then send the data. 

In the backoff period, the Contention Window (CW) is adjustsed according to BEB mechanism. However, the performance of BEB degrades significantly when faced with a high number of terminals. Bianchi <cit.> highlighted the detrimental impact of BEB in dense stations (STAs) scenarios, illustrating that the concurrent presence of numerous terminals contending for channel access can strain the probability of collisions and packet loss, resulting in a decrease in total throughput. This issue becomes increasingly problematic in complex future environments, where numerous STAs within a Basic Service Set (BSS) could precipitate a sharp decline in performance.

In Wi-Fi transmission, frame aggregation is an effective technique used to enhance network performance. The length of aggregated frames can impact latency, channel utilization, and throughput, etc. Therefore, it is necessary to study how to optimize the length of aggregated frames under different channel conditions and varying Quality of Service (QoS) requirements. Camps-Mur et al. <cit.> discuss how to utilize the frame aggregation mechanism in 802.11 standard to improve the QoS in Wi-Fi networks. Coronado et al. <cit.> proposed an adaptive machine learning-based approach for optimizing frame size selection by considering channel conditions, demonstrating an average goodput improvement of 18.36% over standard aggregation mechanisms.

This paper proposes a joint optimization approach of adjusting the CW and frame length to enhance throughput in Wi-Fi networks with densely distributed terminals. The relationship between the optimal CW and frame length for each user is extremely complex, so simultaneous adjustments for each terminal necessitate a model or algorithm capable of effectively modeling these complex relationships. This represents a significant challenge in optimizing network performance while accommodating the high density of terminals. We apply a method named D3PG which combines GDMs and DDPG. D3PG melds the strengths of the DDPG with the sophisticated modeling capabilities of GDMs.

The rest of the paper is organized as follows. Section <ref> presents the system model. Section <ref> introduces how to apply diffusion models into DDPG. Section <ref> gives the performance of D3PG. The simulation and analysis are based on NS3 platform<cit.>. In the end
, we summarize this paper in Section <ref>.



§ SYSTEM MODEL

Our system model is illustrated in Fig. <ref>, which includes one Access Point (AP) and N_s Stations (STAs), which are randomly distributed within a circle of radius r around the AP. The D3PG agent locates at the AP, which gathers all necessary data for training the model, either directly or through data reported by the STAs. Our objective is to train the model to optimize the total throughput in dense Wi-Fi scenarios.



The main difference between DDPG and D3PG is the modeling of policy network, while the optimization process can also be regarded as a Markov Decision Process (MDP)<cit.> that requires the model's agent to collect state, action, and reward information. During training, the agent at the AP collects the state information s_t, and s_t={ITP_t, PLT_t,i|i∈ N_s} is consisted of channel Idle Time Proportion (ITP) and Packet Loss Rate (PLR), and these two variables reflect the overall system status. ITP is the percentage that the AP detects the channel is idle in a certain period, which is calculated as below:

    ITP_t=t_Idle/t_period

PLR indicates the reliability of data transmission for a STA, which is calculated by the number of transmitted frames and the number of received ACK frames n_ACK, i:

    PLT_t,i=1-n_TX, i/n_ACK, i

where n_TX, i and n_ACK, i represent separately the number of transmitted frames and received ACK frames for the ith STA. Then the D3PG agent, starting from the acquired state s_t, processes through a conditioned diffusion model along with randomly generated noise matching the action space size to get the optimal action. After multiple timesteps and reverse processes, the model outputs an action a_t={CW_t, i, L_t, i|i∈ N_s}, which consists of contention window CW_t, i and frame length L_t, i for each STA. The model directly outputs continuous numbers between 0 and 1, and we discretely map CW_t, i to values between CW_min and CW_max, and L_t, i to values between L_min and L_max.
The AP then issues the action to STAs, executing it for a certain period, results in a reward r_t and the next state s_t+1. This completes one trajectory τ_t=(s_t, a_t, r_t, s_t+1) which is stored in an experience pool D for future model optimization.

The reward function is designed to optimize these adjustments based on the improvement in throughput.

    r_t=2× (sigmoid(Thr_t/λ)-0.5)

where Thr_t represents the throughput of the whole system at time t, λ is a hyperparameter that controls the sensitivity of the reward to throughput. Considering practicality, and λ can be set to a desired throughput or the maximum Shannon capacity. This allows us to normalize any positive real number representing throughput to a range between 0 and 1, accurately reflecting reward information and aiding in model convergence.

In this approach, D3PG enables the AP to learn optimal policies for complex network environments, thereby enhancing overall network performance in scenarios that are increasingly prevalent in modern Wi-Fi applications.




§ DESIGN OF OPTIMIZATION SCHEME

The D3PG algorithm is based on the deep deterministic policy gradients algorithm, with the primary distinction being in replacing the decision network by a conditioned diffusion model. The generative diffusion model can be divided into two parts: the forward diffusion process and the reverse diffusion process.



 §.§ Forward process


In computer vision, x_t represents an image, whereas in our network optimization context, x_t denotes a solution which is CW or frame length setting for each STA at time t. The following equation shows how noise diffuses in a solution.


    x_t+1 = √(β)×ϵ + √(1-β)× x_t

where ϵ is Gaussian noise with the same dimensions as x_t, and β is a noise diffusion coefficient, which is a scalar between 0 and 1. x_t+1 is the result of x_t plus the noise. The value of β starts near zero at the beginning and gradually increases close to 1. This process represents the diffusion of noise into the original data, with the gradual increase in β indicating that the speed of diffusion is accelerating.

Following a series of derivations, we get the following equation:

    x_t = √(1-α̅)×ϵ + √(α̅)× x_0

where x_0 is an optimal solution, α̅=α_tα_t-1…α_1, and α_t=1-β_t. So we get the relationship between x_0 and x_t. 
This completes the derivation of the forward diffusion process for the generative diffusion model.



 §.§ Reverse process

The reverse diffusion process involves deriving the relationship from a known x_t back to x_0. Similar to the forward process, we first derive the relationship for one step of reverse diffusion, which is P(x_t-1 | x_t). According to Bayes' theorem, we have:

    P(x_t-1|x_t) = P(x_t|x_t-1)P(x_t-1)/P(x)

Using (<ref>) and (<ref>) derived from the forward diffusion process and substituting and rearranging terms in (<ref>), we find that P(x_t-1 | x_t) follows the following normal distribution:


    P(x_t-1|x_t)∼ 
        N(μ, σ^2)



    μ=√(α_t)(1-α̅_t-1)/1-α̅_̅t̅x_t
        +√(α̅_t-1)(1-α_t)/1-α_tx_t-√(1-α̅)×ϵ/√(α̅)



    σ=√(√(1-α_t)√(1-α̅_t-1)/√(1-α̅_̅t̅))


In (<ref>), ϵ represents the noise added directly to an optimal solution x_0 at any given moment t, suggests that knowing the noise ϵ added from x_0 to x_t allows us to determine the probability distribution of the previous moment x_t-1. This is the essence of the reverse diffusion process.

A neural network can be trained, with inputs including the noised solution x_t, to predict the noise ϵ that has been added to the optimal solution x_0. This predicted noise allows us to obtain the probability distribution of the solution at the previous moment. Sampling randomly from this distribution produces a solution of the previous moment, which can then be fed back into the network to predict its noise, and this process is repeated until the desired solution x_0 is obtained.

By training a neural network to continuously perform conditioned reverse diffusion, where environmental information is input together with the noised solution x_t, final x_0 obtained is the result of optimizing using the generative diffusion model.






§ PERFORMANCE EVALUATION

To demonstrate the effectiveness of the D3PG algorithm, we used PyTorch to assist us in implementing the D3PG algorithm architecture and NS3 platform, an open-source, discrete-event network simulation tool to create the Wi-Fi scenario. To apply machine learning algorithms to our designed Wi-Fi scenario and to adjust multiple network parameters, we utilized the NS3-AI module<cit.>. NS3-AI is a module provided for NS3 platform, which is the interface between NS3 and PyTorch, enabling users to leverage artificial intelligence technologies within the NS3 simulation environment. 

The topology of our simulated Wi-Fi network (basically based on Wi-Fi 6) is shown in Fig. <ref>. The parameters of our NS3 simulation environment are presented in the table <ref>.



D3PG hyperparameter setting is below in table <ref>:



In our study, we first compared the throughput performance of our algorithm in a scenario with 64 STAs. We selected three different algorithms as baseline algorithms for performance comparison. They are:


    
  * Baseline 1: Utilizes the existing 802.11 Wi-Fi standard's MAC layer contention access mechanism, which is based on BEB and CSMA/CA, with fixed frame aggregation length.
    
  * Baseline 2: Proximal Policy Optimization (PPO) algorithm proposed by Schulman et al. <cit.>, which is an on-policy reinforcement learning algorithm.
    
  * Baseline 3: DDPG algorithm that does not utilize generative diffusion models, with the same hyperparameter settings as our D3PG algorithm.


It should be noted that due to the randomness of Wi-Fi networks, we conducted five repeated simulations for each algorithm and took the average to minimize the impact of randomness on the comparative results. As can be seen from Fig. <ref>, the D3PG algorithm demonstrated an enhancement of 74.6% over baseline 1, it further showed additional increase of 13.5% in throughput compared to baseline 2, and an additional increase of 10.5% over baseline 3. Specifically, compared to the baseline 3, the primary reason for the greater enhancement is the aid of generative diffusion model, the D3PG algorithm can more accurately map the relationships between environmental information and action combinations, resulting in more precise parameter control in complex scenarios.



Fig. <ref> and Fig. <ref> respectively show the changes in total throughput and cumulative rewards during the training process of the D3PG and baseline 3 algorithms. It is observed that the training process of the D3PG algorithm is more stable, whereas the baseline 3 algorithm  exhibits greater fluctuations. This is because D3PG has a stronger exploratory ability, which allows the model to reach optimum performance more quickly and shortens the training time. This feature gives D3PG another advantage: in rapidly changing Wi-Fi environments, if the model needs to restart training due to environmental changes, the D3PG algorithm can converge more quickly, minimizing the impact on user experience.







Finally, we compared the throughput performance of our algorithm with baselines across networks of different scales. As shown in Fig. <ref>, in the networks with different numbers of STAs, all the STAs are still uniformly distributed within a circle centered on the AP with a radius of 7.5 meters. It is shown that our algorithm consistently enhances performance across all network sizes. For baseline 1, as the number of users initially increases, there is a rise in throughput due to the increase of users, so the system throughput increases with the number of users. However, as the number of users continues to grow, internal competition within the BSS intensifies. The BEB mechanism for changing contention windows leads to increased collisions, ultimately resulting in a continuous decline in total throughput. In the case of our algorithm, as the users increase, the total throughput also increases until it reaches a point where it stabilizes. In the network of small number of users, the network is not saturated, once users get more and more, the network becomes saturated, and the total throughput stabilizes. With the adjustments made by our algorithm, the total throughput does not undergo a rapid decline, even as internal competition significantly intensifies.








§ CONCLUSION

In this paper, we introduce a MAC layer access mechanism based on the D3PG algorithm, which enhances network throughput by jointly adjusting the contention window and frame aggregation for each STA. This mechanism addresses the sharp decline in throughput performance caused by increased competition within BSS in dense Wi-Fi scenarios under the existing 802.11 Wi-Fi standard. The throughput improvements with D3PG optimization are significant, showing a 76.4% increase over the baseline 1 and surpassing two other reinforcement learning algorithms baseline 2 and baseline 3 by 13.5% and 10.5%, respectively. Besides, supported by generative diffusion models, which excel at modeling complex data distributions, the D3PG algorithm offers more stable training, faster convergence, and greater flexibility and adaptability. We initially applied GAI to optimize the MAC layer performance of Wi-Fi networks. The results indicate that this approach effectively enhances performance, but the scenarios are relatively simple, and the effectiveness of further applications requires more in-depth study.



