CN113316154A - Authorized and unauthorized D2D communication resource joint intelligent distribution method - Google Patents

Authorized and unauthorized D2D communication resource joint intelligent distribution method Download PDF

Info

Publication number
CN113316154A
CN113316154A CN202110581716.5A CN202110581716A CN113316154A CN 113316154 A CN113316154 A CN 113316154A CN 202110581716 A CN202110581716 A CN 202110581716A CN 113316154 A CN113316154 A CN 113316154A
Authority
CN
China
Prior art keywords
agent
action
user
reward
state
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202110581716.5A
Other languages
Chinese (zh)
Other versions
CN113316154B (en
Inventor
裴二荣
徐成义
陶凯
宋珈锐
黄一格
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chongqing University of Post and Telecommunications
Original Assignee
Chongqing University of Post and Telecommunications
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chongqing University of Post and Telecommunications filed Critical Chongqing University of Post and Telecommunications
Priority to CN202110581716.5A priority Critical patent/CN113316154B/en
Publication of CN113316154A publication Critical patent/CN113316154A/en
Application granted granted Critical
Publication of CN113316154B publication Critical patent/CN113316154B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W24/00—Supervisory, monitoring or testing arrangements
    • H04W24/02—Arrangements for optimising operational condition
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W16/00—Network planning, e.g. coverage or traffic planning tools; Network deployment, e.g. resource partitioning or cells structures
    • H04W16/02—Resource partitioning among network components, e.g. reuse partitioning
    • H04W16/10—Dynamic resource partitioning
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W24/00—Supervisory, monitoring or testing arrangements
    • H04W24/06—Testing, supervising or monitoring using simulated traffic
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W28/00—Network traffic management; Network resource management
    • H04W28/02—Traffic management, e.g. flow control or congestion control
    • H04W28/0215—Traffic management, e.g. flow control or congestion control based on user or device properties, e.g. MTC-capable devices
    • H04W28/0221—Traffic management, e.g. flow control or congestion control based on user or device properties, e.g. MTC-capable devices power availability or consumption

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Mobile Radio Communication Systems (AREA)

Abstract

The invention relates to a joint intelligent distribution method of authorized and unauthorized D2D communication resources, belonging to the field of D2D communication. The invention comprises the following steps: s1: establishing a D2D user communication model; s2: establishing an objective function to be optimized; s3: establishing a multi-agent deep reinforcement learning D2D communication model; s4: setting an action set, a state set and a reward function of the multi-agent; s5: the intelligent agent takes action according to the Actor network of the intelligent agent to obtain the state, the reward and the next state; s6: calculating a TDerror of the Critic network, updating parameters of the Critic network, calculating a counterfactual baseline of each agent by the Critic network, updating parameters of the Actor network through the counterfactual baseline, and updating the state; s7: steps S5-S6 are repeated until the target state is reached. According to the multi-agent deep reinforcement learning framework, the agents continuously interact with the environment, continuously train and learn, find the optimal strategy, and the action corresponding to the adopted strategy obtains the optimal throughput, and meanwhile, the communication quality of WiFi and cellular users is guaranteed.

Description

Authorized and unauthorized D2D communication resource joint intelligent distribution method
Technical Field
The invention belongs to the field of D2D communication, and relates to a joint intelligent allocation method for authorized and unauthorized D2D communication resources.
Background
With the rapid popularization of intelligent terminals, the number of connection devices in this year, including smart phones, tablets, car networking devices, internet of things devices and the like, is expected to reach 500 hundred million, and data traffic is increased by 1000 times, so that the requirement for the evolution of a wireless communication technology is more urgent. On one hand, wireless communication performance indexes such as network capacity, spectrum efficiency and the like need to be obviously improved to deal with the explosively increased data traffic; on the other hand, the expansion of cellular communication applications needs to be realized, and the end user experience is improved.
In the face of the situation that the data traffic and the authorized spectrum which are explosively increased are almost distributed, the new spectrum is expanded to improve the system capacity so as to adapt to the rapid increase of the mobile traffic and the diversification of the services, and the method becomes the current primary target. The frequency spectrum working frequency band of the unlicensed frequency band mainly comprises a 2.4GHz frequency band, a 5GHz frequency band and a 60GHz millimeter wave frequency band. Since the free space loss increases as the frequency band increases, only the frequency band of 6GHz or less is considered to be able to better resist path fading. In the frequency band below 6GHz, the 2.4GHz frequency band is already densely occupied by WiFi, Bluetooth and other wireless technologies, the interference is complex, the 5GHz frequency band has an available space close to 500MHz, only one part is occupied by WiFi, and the utilization rate is low. Therefore, the D2D technology can be realized in the 5GHz band as long as the interference between the D2D technology and the WiFi is controlled within an acceptable range. The 5GHz unlicensed frequency band channel has large fading and is suitable for short-distance communication, D2D is used as a proximity service, the property of the D2D is very consistent when the D2D is placed in the unlicensed frequency band, and compared with other short-distance communication technologies (WiFi direct connection, Bluetooth, Zigbee protocol and the like) which work in the unlicensed frequency band, the D2D communication has great advantages, pairing of users and resource allocation of channels and power are controlled by a base station, and access is more efficient and safer. D2D communication is deployed in the unlicensed frequency band, so that the use rate of the unlicensed spectrum can be improved, and seamless fusion with the existing cellular system can be realized. Similar to the LTE-U technology, the D2D communication is initially operated only in the licensed spectrum, and there is no coexistence mechanism with the WiFi system, and if the unlicensed frequency band is directly accessed, the performance of the WiFi system is seriously affected, so that it is also ensured that the D2D user and the original WiFi user coexist harmoniously when the D2D communication is implemented in the unlicensed frequency band.
The existing D2D communication is mainly deployed in an authorized frequency band, joint deployment of the D2D in an unlicensed frequency band and an authorized frequency band is rarely considered, and authorization and unlicensed selection of a D2D user is considered on the premise of ensuring the minimum communication quality of a WiFi user, and the problem of NP-hard is solved due to the fact that the problem of D2D communication resource allocation is solved, and the traditional algorithm is difficult to solve. Therefore, the current very popular machine learning method is utilized to solve the problem which is difficult to solve in the traditional algorithm, and has very important research significance.
Disclosure of Invention
In view of this, the invention provides a joint intelligent allocation method for authorized and unlicensed D2D communication resources, which solves the problem of joint authorized and unlicensed spectrum selection D2D communication resource allocation that is difficult to solve by the conventional algorithm.
In order to achieve the purpose, the invention provides the following technical scheme:
a joint intelligent allocation method for authorized and unauthorized D2D communication resources comprises the following steps:
s1: establishing a D2D user communication model;
s2: establishing an objective function to be optimized;
s3: establishing a multi-agent deep reinforcement learning D2D communication model;
s4: setting an action set, a state set and a reward function of the multi-agent;
s5: the intelligent agent takes action according to the Actor network of the intelligent agent to obtain the state, the reward and the next state;
s6: and calculating TD error of the Critic network, updating parameters of the Critic network, calculating counterfactual baselines of each agent by the Critic network, updating parameters of the Actor network through the counterfactual baselines, and updating the state.
S7: steps S5-S6 are repeated until the target state is reached.
Further, in step S1, the number of D2D accessing the WiFi band is calculated, but which D2D is selected to avoid the licensed band, and the power and channel selection of the remaining D2D still have a serious impact on the cellular user.
Multiplexing the authorized frequency band: in this mode, two D2D may multiplex the uplink of the same existing cellular user for direct communication. The spectral efficiency of D2D versus k for the channel multiplexing cellular user m is:
Figure BDA0003085521030000021
in the formula pk,mIs the transmit power of the kth D2D pair,
Figure BDA0003085521030000022
is the transmit power of the cellular user m,
Figure BDA0003085521030000023
is the channel gain, B, of D2D k to cellular user mCIs the granted bandwidth of the channel and,
Figure BDA0003085521030000024
is the noise power density, hk,mIs the interference power gain between cellular user m and the receiver of D2D pair k. The spectral efficiency of cellular user m multiplexed by D2D for k is:
Figure BDA0003085521030000031
wherein
Figure BDA0003085521030000032
Is the channel power gain, h, of the cellular user m and the base stationk,BD2D channel gain between the transmitting terminal k and the base station.
The existence of D2D communication has a great influence on cellular and WiFi users, so it is proposed to perform mode selection and resource allocation on the total D2D users after determining the maximum number of D2D capable of accessing the WiFi licensed band under the condition of meeting the WiFi user minimum throughput, so as to reduce the degradation of cellular and WiFi users caused by D2D communication to the maximum extent.
When x isiD2D multiplexes i with the channel of the uplink cellular user; x is the number ofiIf 0, D2D will access the WiFi unlicensed band for i.
When theta isi,m1, D2D multiplexes i with the channel of the uplink cellular user m; thetai,m0, indicates that D2D multiplexes the channel of the uplink cellular user m to i.
One channel can be multiplexed by a plurality of D2D users, and only one channel can be selected by one D2D for data transmission.
Further, in step S2, in order to obtain the maximum cellular user and the system throughput of the licensed band D2D user, there is
Figure BDA0003085521030000033
s.t.xk∈{0,1}，θk,m∈{0,1}
0≤pk,m≤pmax
Figure BDA0003085521030000034
Figure BDA0003085521030000035
The first term above represents the selection of access authorization and authorization exemption for the D2D user, the second term represents the power limit for the D2D user, the third term represents the satisfaction of minimum WiFi throughput requirements, and the fourth term represents the assurance that the D2D user and the cellular user meet minimum signal-to-noise ratio requirements.
Further, in step S3, in order to solve the NP-hard problem in D2D communication resource allocation, a Multi-Agent reinforcement learning method, COMA (computational Multi-Agent), is adopted to model the Multi-Agent environment as markov game to optimize the strategy, and meanwhile, consider the behavior strategies of other agents, the method is to make a single Agent perform the optimization of the strategyThe effect of an agent on the reward is marginalized by comparing the action taken by the agent at a certain point in time t with all other actions it may take at t, which may be achieved by a centralized Critic, so that the cost function of all agents is the same, but each agent will receive a customized error term based on its counterfactual action. In a collaborative agent system, when evaluating how little the contribution of an agent's actions is, the agent's actions can be changed to default actions, and the current actions can be seen to increase or decrease the overall score compared to the default actions, if increasing, the agent's current actions are better than the default actions, and if decreasing, the agent's current actions are worse than the default actions. And this default action is referred to as the baseline. However, the following problem is how to determine the default action, for example, by confirming the default action in some way, and the quality of the default action needs additional simulation for evaluation, which increases the complexity of the calculation. Instead of using default actions, COMA calculates this baseline by solving the edge distribution for the current agent's policy using the current behavior value function, with no additional simulation to calculate this baseline. In this way, the COMA may avoid designing additional default actions and additional simulation calculations. Therefore, strategies requiring multi-agent coordination are better learned for each agent. COMA provides an efficient algorithm to perform credit allocation for reward functions, and the deep learning training process results in a large amount of computational overhead. The training process is completed at the BS, the historical information collected by the D2D user in the execution process is uploaded to the BS, the centralized training is completed at the BS, and the Critic obtains the strategy of the intelligent agent at the BS to evaluate the quality of the action. In the distributed execution process, the D2D user obtains A from the base stationj(s, u) updating own Actor network, wherein the Actor selects behaviors based on the state observed by the agent from the environment, the agent continuously interacts with the environment, the agent performs enough training times, and finally converges on an action with the maximum reward value, thereby obtaining the optimal strategy.
Further, in step S4, in the RL model of the D2D underlying communication, the agent D2D takes corresponding actions for interacting with the environment and according to the policy. At each time t, agent D2D observes a state S from state space StAnd taking corresponding actions (selecting mode, selecting RB, selecting power level) from the action space A according to the strategy pi. After performing this action, the environment enters a new state st+1Agent receives the reward.
State space S: at any time t, the system state is represented by the joint SINR value of all D2D at that time t:
Figure BDA0003085521030000051
an action space A:
Figure BDA0003085521030000052
mode selection, power level selection, and RB selection, respectively
Wherein, the number of authorized and unauthorized selections: 2, power level: 10, RB selection number: 20. so the action space number of each agent: α × β × η is 400.
The reward function R: the reward function in the RL drives the whole learning process, so a reasonable reward function is the key, and the reward function is designed into three parts: the selection mode of the D2D, the speeds of the D2D and the cellular users and the signal-to-noise ratios of the two, if the agent enters the unlicensed band in the selected mode, the reward obtained by the agent is set to a positive value, but a larger negative value is obtained when the number of the D2D exceeds the satisfied maximum access number, if the action taken by the agent causes the signal-to-noise ratios of the cellular users and the D2D users to be larger than the set threshold value, the reward function is a negative value as the sum of the corresponding speed and the reward of the selected cellular users of the same multiplexing spectrum is used as the reward, and conversely, if the action taken by the agent causes the signal-to-noise ratio of the D2D or the cellular users to be smaller than the set threshold value, because the signal-to-noise ratio is smaller than the signal-to-noise ratio, the signal cannot be decoded.
The limitation of the number of the access to the authorization-free mode designs a function
Figure BDA0003085521030000053
Limiting SINR of D2D and CU
Figure BDA0003085521030000054
Wherein
Figure BDA0003085521030000055
Figure BDA0003085521030000056
Further, in step S5, the hyper-parameter γ, α in the network is initialized firstθ,αλ,αφBeta, state s0And the parameters λ, φ in the Actor, Critic network0，φ1，...，φJ,θ0,θ1,...,θJ. Each agent takes the action with the highest probability according to the policy network of the agent as the action taken in the current state, so that the action states taken by all agents are combined to obtain the action state from the environment state stCombined action oftD2D SINR reward
Figure BDA0003085521030000065
And the next state st+1。
Further, in step S6, the monte carlo method provides a basis for model-free learning, but they still have the limitation of discontinuous, offline learning. The TD error method makes up the difference between the Monte Carlo method and the dynamic programming, and is the core idea of RL. The TD method can also learn in a model-free environment, and can iteratively learn from value estimates (online learning), allowing training in a continuous environment. Calculating TD error from Critic network:
Figure BDA0003085521030000061
strategy parameter updating is carried out based on strategy gradient, and TD error adopts a gradient ascending method:
Figure BDA0003085521030000062
to solve the problem of confidence allocation in a multi-agent. The COMA algorithm solves the confidence assignment problem with a counterfactual baseline by marginalizing the effect of a single agent on the reward and comparing the action taken by the agent at a certain time t with all other actions that may be taken at t, by centralized Critic, so that the value function is the same for all agents, but each agent gets a specific error term based on its counterfactual action. The jth agent counterfactual baseline is defined as:
Figure BDA0003085521030000063
the jth agent updates the Actor network parameters of the jth agent through the counterfactual base line according to the formula:
Figure BDA0003085521030000064
and the intelligent agent updates the network according to the merit function obtained by the Critic network.
Further, in step S7, the training process is completed by the BS, the history information collected during the execution process of the D2D user is uploaded to the BS, the BS completes centralized training, and Critic obtains the policy of the agent at the BS to evaluate how well to take action. In the distributed execution process, the D2D user obtains A from the base stationj(s, u) updating own Actor network, wherein the Actor selects behaviors based on the state observed by the agent from the environment, the agent continuously interacts with the environment, the agent performs enough training times, and finally converges on an action with the maximum reward value, thereby obtaining the optimal strategy.
The invention has the following effective effects: in the D2D resource allocation problem, the D2D user is allocated with the unlicensed frequency band, the frequency spectrum and the power jointly, the number of D2D accessible to the unlicensed frequency band can be determined on the premise of ensuring the minimum communication quality of the WiFi user, then the D2D entering the unlicensed frequency band is determined, and the power and the frequency spectrum are allocated to the D2D still remaining in the licensed frequency band, so that the throughput of the cellular user and the D2D user in the licensed frequency band is maximized, an effective multi-agent deep reinforcement learning algorithm is provided, and the problem of NP-hard which is difficult to solve in the traditional algorithm is solved.
Drawings
In order to make the object, technical scheme and beneficial effect of the invention better clear, the invention provides the following drawings for illustration:
FIG. 1 is a schematic flow chart of an embodiment of the present invention;
FIG. 2 is a diagram of a network model for D2D communication;
FIG. 3 is a diagram of an AC network framework model according to an embodiment of the present invention;
FIG. 4 is a COMA model diagram of a multi-agent deep reinforcement learning algorithm according to an embodiment of the present invention.
Detailed Description
Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
The invention provides a joint intelligent allocation method of authorized and unauthorized D2D communication resources, aiming at the problem of uplink data transmission in a cellular network and a D2D network. In order to obtain the number of accessible unlicensed bands D2D, the number of accessible unlicensed bands is determined by utilizing an established two-dimensional time Markov model on the premise of ensuring the WiFi communication quality, after the number of accessible unlicensed bands is obtained, for selecting a part of D2D users to remove the unlicensed bands and performing power and spectrum allocation on the rest D2D in the licensed bands, in order to maximize the throughput of D2D users and cellular users in the licensed bands, a multi-agent deep reinforcement learning method COMA algorithm is provided, a multi-agent environment is modeled as a Markov game to optimize strategies, and meanwhile, the action strategies of other agents are considered, the method is to marginalize the influence of a single agent on rewards, the action taken by the agent at a certain time point t is compared with all other actions which the agent may take at t, and the action can be realized by a centralized criticc, so the cost function of all agents is the same, but each agent will receive a customized error term based on its counter-fact actions, in this way the COMA can avoid designing additional default actions and additional simulation calculations. Therefore, strategies requiring multi-agent coordination are better learned for each agent. COMA provides an efficient algorithm to perform credit allocation for reward functions, and the deep learning training process results in a large amount of computational overhead. Therefore, the training process is completed by the BS, the historical information collected by the D2D user during the execution process is uploaded to the BS, centralized training is completed at the BS, and Critic obtains the strategy of the agent at the base station to evaluate the quality of the action taken. In the distributed execution process, the D2D obtains counterfactual baselines from the BS so as to update own Actor network, the Actor selects behaviors based on states observed by the agent from the environment, the agent continuously interacts with the environment, the agent performs enough training times, and finally converges on an action with the maximum reward value, so that the optimal strategy is obtained. A flow chart of a multi-intelligent deep reinforcement learning method for joint authorized and unauthorized D2D communication resource allocation in D2D communication is shown in fig. 1.
A diagram of a network model based on D2D communication is shown in fig. 2. The D2D user transmits in the authorized frequency band, it will multiplex the channel with the existing cellular user, thus cause the communication between D2D user and cellular user to interfere, the D2D user chooses to work in the unlicensed frequency band, then will influence the user communication quality of WiFi frequency band, consider that the access way of deploying D2D user in the unlicensed frequency band is LBT, therefore can model D2D user and WiFi user as two-dimensional Markov model, and then can confirm the quantity that can access to the unlicensed frequency band on the premise of guaranteeing the communication quality of WiFi user. Considering that the communication link between devices is uplink resource sharing, since it is much easier to handle channel interference in the uplink than in the downlink, in order to maximize the system capacity deployed in the licensed band, the same channel can be used by multiple pairs of D2D, but only one channel can be selected for multiplexing per pair of D2D. Therefore, authorization and authorization-free selection and power and spectrum allocation need to be performed on D2D users, and the problem is an NP-hard problem, so a machine learning method is used to solve the problem, in which D2D users are regarded as agents, actions are authorization and authorization-free selection, power and spectrum selection, a combined state is SINR of all D2D users, a reasonable reward function is set for multiple agents, the agents continuously interact with the environment to select actions, update states and update network parameters, the agents continuously learn in the environment, and select corresponding actions with maximum reward.
As shown in fig. 1, a joint intelligent allocation method for authorized and unauthorized D2D communication resources includes the following steps:
s1: establishing a D2D user communication model;
s2: establishing an objective function to be optimized;
s3: establishing a multi-agent deep reinforcement learning D2D communication model;
s4: setting an action set, a state set and a reward function of the multi-agent;
s5: the intelligent agent takes action according to the Actor network of the intelligent agent to obtain the state, the reward and the next state;
s6: and calculating TD error of the Critic network, updating parameters of the Critic network, calculating counterfactual baselines of each agent by the Critic network, updating parameters of the Actor network through the counterfactual baselines, and updating the state.
S7: steps S5-S6 are repeated until the target state is reached.
To improve spectral efficiency, D2D users reuse cellular user resources on the licensed band, thereby causing interference to cellular users. Multiplexing the authorized frequency band: in this mode, two D2D may multiplex the uplink of the same existing cellular user for direct communication. The spectral efficiency of D2D versus k for the channel multiplexing cellular user m is:
Figure BDA0003085521030000091
in the formula pk,mIs the transmit power of the kth D2D pair,
Figure BDA0003085521030000092
is the transmit power of the cellular user m,
Figure BDA0003085521030000093
is the channel gain, B, of D2D k to cellular user mCIs the granted bandwidth of the channel and,
Figure BDA0003085521030000094
is the noise power density, hk,mIs the interference power gain between cellular user m and the receiver of D2D pair k. The spectral efficiency of cellular user m multiplexed by D2D for k is:
Figure BDA0003085521030000095
wherein
Figure BDA0003085521030000096
Is the channel power gain, h, of the cellular user m and the base stationk,BIs the channel gain between D2D transmitter k and the base station.
The existence of D2D communication has a great influence on cellular and WiFi users, so it is proposed to perform mode selection and resource allocation on D2D users after determining the maximum number of D2D capable of accessing the WiFi licensed band under the condition of meeting the minimum throughput of WiFi users, so as to reduce the degradation of cellular and WiFi users caused by D2D communication to the maximum extent.
When x isiD2D multiplexes i with the channel of the uplink cellular user, xi＝0D2D will access the WiFi unlicensed band for i.
When theta isi,m1, D2D multiplexes i the channel of the uplink cellular user m, θi,m0, indicates that D2D multiplexes the channel of the uplink cellular user m to i.
In order to obtain the maximum cellular user and system throughput for D2D, there is
Figure BDA0003085521030000097
Figure BDA0003085521030000101
s.t.xk∈{0,1}，θk,m∈{0,1}
0≤pk,m≤pmax
Figure BDA0003085521030000102
Figure BDA0003085521030000103
The first term above represents the selection of access authorization and authorization exemption for the D2D user, the second term represents the power limit for the D2D user, the third term represents the satisfaction of minimum WiFi throughput requirements, and the fourth term represents the assurance that the D2D user and the cellular user meet minimum signal-to-noise ratio requirements.
In order to solve the NP-hard problem in D2D communication resource allocation, a multi-agent deep reinforcement learning COMA method is used herein, which models the multi-agent environment as markov game to optimize the strategy, and considers the behavior strategy of other agents, the method is to marginalize the influence of a single agent on the reward, compare the behavior taken by the agent at a certain time point t with all other behaviors it may take at t, which can be realized by a centralized Critic, so that the cost functions of all agents are the same, but the cost functions are the sameEach agent will receive a customized error item based on its counterfactual action. In a collaborative agent system, when evaluating how little the contribution of an agent's actions is, the agent's actions can be changed to default actions, and the current actions can be seen to increase or decrease the overall score compared to the default actions, if increasing, the agent's current actions are better than the default actions, and if decreasing, the agent's current actions are worse than the default actions. And this default action is referred to as the baseline. However, the following problem is how to determine the default action, for example, by confirming the default action in some way, and the quality of the default action needs additional simulation for evaluation, which increases the complexity of the calculation. Instead of using default actions, COMA calculates this baseline by solving the edge distribution for the current agent's policy using the current behavior value function, with no additional simulation to calculate this baseline. In this way, the COMA may avoid designing additional default actions and additional simulation calculations. Therefore, strategies requiring multi-agent coordination are better learned for each agent. COMA provides an efficient algorithm to perform credit allocation for reward functions, and the deep learning training process results in a large amount of computational overhead. The training process is completed by the BS, historical information collected by the D2D user in the execution process is uploaded to the BS, centralized training is completed at the BS, and the Critic obtains the strategy of the intelligent agent at the BS to evaluate the quality of action. In the distributed execution process, the D2D user obtains A from the base stationj(s, u) updating own Actor network, wherein the Actor selects behaviors based on the state observed by the agent from the environment, the agent continuously interacts with the environment, the agent performs enough training times, and finally converges on an action with the maximum reward value, thereby obtaining the optimal strategy. The AC network model is shown in fig. three.
In the RL model of D2D underlying communication, agent D2D is a framework based on AC network model for interacting with the environment and taking corresponding actions according to policies. At each time t, agent D2D pairs with a slave stateObserving a state S in space StAnd taking corresponding actions (selecting mode, selecting RB, selecting power level) from the action space A according to the strategy pi. After performing this action, the environment enters a new state st+1Agent receives the reward.
State space S: at any time t, the system state is represented by the joint SINR value of all D2D at that time t
Figure BDA0003085521030000111
An action space A:
Figure BDA0003085521030000112
mode selection, power level selection, and RB selection, respectively
Wherein, the number of authorized and unauthorized selections: 2, power level: 10, RB selection number: 20. so the action space number of each agent: α × β × η is 400.
The reward function R: the reward function in the RL drives the whole learning process, so a reasonable reward function is the key, and the reward function is designed into three parts: the selection mode of the D2D, the speeds of the D2D and the cellular users and the signal-to-noise ratios of the two, if the agent enters the unlicensed band in the selected mode, the reward obtained by the agent is set to a positive value, but a larger negative value is obtained when the number of the D2D exceeds the satisfied maximum access number, if the action taken by the agent causes the signal-to-noise ratios of the cellular users and the D2D users to be larger than the set threshold value, the reward function is a negative value as the sum of the corresponding speed and the reward of the selected cellular users of the same multiplexing spectrum is used as the reward, and conversely, if the action taken by the agent causes the signal-to-noise ratio of the D2D or the cellular users to be smaller than the set threshold value, because the signal-to-noise ratio is smaller than the signal-to-noise ratio, the signal cannot be decoded.
The limitation of the number of the access to the authorization-free mode designs a function
Figure BDA0003085521030000113
Limiting SINR of D2D and CU
Figure BDA0003085521030000121
Wherein
Figure BDA0003085521030000122
Figure BDA0003085521030000123
First, a hyper-parameter gamma, alpha in the network is initializedθ,αλ,
Figure BDA0003085521030000124
Beta, state s0And Actor, the parameter lambda in the critical network,
Figure BDA0003085521030000125
θ0,θ1,...,θJ. Each agent takes the action with the highest probability according to the policy network of the agent as the action taken in the current state, so that the action states taken by all agents are combined to obtain the action state from the environment state stCombined action oftD2D SINR reward
Figure BDA0003085521030000126
And the next state st+1。
The Monte Carlo method provides a basis for model-free learning, but the Monte Carlo method still has the limitation of discontinuous and offline learning. The TD error method makes up the difference between the Monte Carlo method and the dynamic programming, and is the core idea of RL. The TD method can also learn in a model-free environment, and can iteratively learn from value estimates (online learning), allowing training in a continuous environment. Calculating TD error from Critic network:
Figure BDA0003085521030000127
strategy parameter updating is carried out based on strategy gradient, and TD error adopts a gradient ascending method:
to solve the problem of confidence allocation in a multi-agent. The COMA algorithm solves the confidence assignment problem with a counterfactual baseline by marginalizing the effect of a single agent on the reward and comparing the action taken by the agent at a certain time t with all other actions that may be taken at t, by centralized Critic, so that the value function is the same for all agents, but each agent gets a specific error term based on its counterfactual action. The jth agent counterfactual baseline is defined as:
Figure BDA0003085521030000128
the jth agent updates the Actor network parameters of the jth agent through the counterfactual base line according to the formula:
Figure BDA0003085521030000129
a schematic diagram of a multi-agent deep reinforcement learning COMA algorithm is shown in FIG. 4.
The training process is completed at the BS, the historical information collected by the D2D user in the execution process is uploaded to the BS, the centralized training is completed at the BS, and the Critic obtains the strategy of the intelligent agent at the BS to evaluate the quality of the action. In the distributed execution process, the D2D user obtains A from the base stationj(s, u) updating own Actor network, wherein the Actor selects behaviors based on the state observed by the agent from the environment, the agent continuously interacts with the environment, the agent performs enough training times, and finally converges on an action with the maximum reward value, thereby obtaining the optimal strategy.
Finally, it is noted that the above-mentioned preferred embodiments illustrate rather than limit the invention, and that, although the invention has been described in detail with reference to the above-mentioned preferred embodiments, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the scope of the invention as defined by the appended claims.

Claims (8)

1. A joint intelligent allocation method for authorized and unauthorized D2D communication resources comprises the following steps:
s1: establishing a D2D user communication model;
s2: establishing an objective function to be optimized;
s3: establishing a multi-agent deep reinforcement learning D2D communication model;
s4: setting an action set, a state set and a reward function of the multi-agent;
s5: the intelligent agent takes action according to the Actor network of the intelligent agent to obtain the state, the reward and the next state;
s6: and calculating TD error of the Critic network, updating parameters of the Critic network, calculating counterfactual baselines of each agent by the Critic network, updating parameters of the Actor network through the counterfactual baselines, and updating the state.
S7: steps S5-S6 are repeated until the target state is reached.
2. The method of claim 1 for joint intelligent allocation of granted and denied D2D communication resources, wherein: in step S1. The number of D2D access to WiFi bands is calculated, but selecting which D2D to go to WiFi bands is a mode selection problem, and the remaining power and channel selection of D2D still have a serious impact on CU users.
Multiplexing the allowed frequency band: in this mode, two D2D may multiplex the uplink of the same existing cellular user for direct communication. The spectral efficiency of D2D versus k for the channel multiplexing cellular user m is:
Figure FDA0003085521020000011
in the formula pk,mIs the transmit power of the kth D2D pair,
Figure FDA0003085521020000012
is the transmit power of the cellular user m,
Figure FDA0003085521020000013
is the channel gain, B, of D2D k to cellular user mCIs the allowed channel bandwidth and is,
Figure FDA0003085521020000014
is the noise power density, hk,mIs the interference power gain between cellular user m and the receiver of D2D pair k. The spectral efficiency of cellular user m multiplexed by D2D for k is:
Figure FDA0003085521020000015
wherein
Figure FDA0003085521020000016
Is the channel power gain, h, of the cellular user m and the base stationk,BIs the channel gain between D2D transmitting terminal k and the base station.
The existence of D2D communication has a great influence on cellular and WiFi users, so it is proposed to perform mode selection and resource allocation on the total D2D users after determining the maximum number of D2D capable of accessing the WiFi licensed band under the condition of meeting the WiFi user minimum throughput, so as to reduce the degradation of cellular and WiFi users caused by D2D communication to the maximum extent.
When x isiD2D multiplexes i with the channel of the uplink cellular user, xiIf 0, D2D will access the WiFi unlicensed band for i.
When theta isi,m1, D2D multiplexes i the channel of the uplink cellular user m, θi,mTable (0)D2D is shown multiplexing the channels of the uplink cellular user m for i.
3. The method of claim 2 for joint intelligent allocation of granted and denied D2D communication resources, further comprising: in step S2, in order to obtain the maximum cellular user and the system throughput of the D2D user in the authorized frequency band, the method comprises
Figure FDA0003085521020000021
s.t.xk∈{0,1}，θk,m∈{0,1}
0≤pk,m≤pmax
Figure FDA0003085521020000022
Figure FDA0003085521020000023
The first term above represents the selection of access authorization and authorization exemption for the D2D user, the second term represents the power limit for the D2D user, the third term represents the satisfaction of minimum WiFi throughput requirements, and the fourth term represents the assurance that the D2D user and the cellular user meet minimum signal-to-noise ratio requirements.
4. The method of claim 3 for joint intelligent allocation of granted and denied D2D communication resources, further comprising: in step S3, to solve the NP-hard problem in D2D communication resource allocation, a Multi-Agent reinforcement learning method, COMA (statistical artificial Multi-Agent), is adopted to model the Multi-Agent environment as markov game to optimize the strategy, and simultaneously consider the behavior strategies of other agents, by marginalizing the effect of a single Agent on the reward, and comparing the behavior of the Agent at a certain time point t with all other behaviors it may take at t, which can be done byIt is implemented with a centralized Critic, so the cost function of all agents is the same, but each agent will receive a custom error term based on its counterfactual action. In a collaborative agent system, when evaluating how little the contribution of an agent's actions is, the agent's actions can be changed to default actions, and the current actions can be seen to increase or decrease the overall score compared to the default actions, if increasing, the agent's current actions are better than the default actions, and if decreasing, the agent's current actions are worse than the default actions. And this default action is referred to as the baseline. However, the following problem is how to determine the default action, for example, by confirming the default action in some way, and the quality of the default action needs additional simulation for evaluation, which increases the complexity of the calculation. Instead of using default actions, COMA calculates this baseline by solving the edge distribution for the current agent's policy using the current behavior value function, with no additional simulation to calculate this baseline. In this way, the COMA may avoid designing additional default actions and additional simulation calculations. Therefore, strategies requiring multi-agent coordination are better learned for each agent. COMA provides an efficient algorithm to perform credit allocation for reward functions, and the deep learning training process results in a large amount of computational overhead. The training process is completed at the BS, the historical information collected by the D2D user in the execution process is uploaded to the BS, the centralized training is completed at the BS, and the Critic obtains the strategy of the intelligent agent at the BS to evaluate the quality of the action. In the distributed execution process, the D2D user obtains A from the base stationj(s, u) updating own Actor network, wherein the Actor selects behaviors based on the state observed by the agent from the environment, the agent continuously interacts with the environment, the agent performs enough training times, and finally converges on an action with the maximum reward value, thereby obtaining the optimal strategy.
5. A joint intelligence of granted and denied D2D communication resources as claimed in claim 4The assignable method is characterized in that further, in step S4, in the RL model of D2D underlying communication, the agent D2D is paired to interact with the environment and take corresponding actions according to the policy. At each time t, agent D2D observes a state S from state space StAnd taking corresponding actions (selecting mode, selecting RB, selecting power level) from the action space A according to the strategy pi. After performing this action, the environment enters a new state st+1Agent receives the reward. State space S: at any time t, the system state is represented by the joint SINR value of all D2D at that time t
Figure FDA0003085521020000031
An action space A:
Figure FDA0003085521020000032
mode selection, power level selection, and RB selection, respectively
Wherein, the mode selection: 2, power level: 10, RB selection: 20. so the action space number of each agent: α × β × η is 400.
The reward function R: the reward function in the RL drives the whole learning process, so a reasonable reward function is the key, and the reward function is designed into three parts: the selection mode of the D2D, the speeds of the D2D and the cellular users and the signal-to-noise ratios of the two, if the agent enters the unlicensed band in the selected mode, the reward obtained by the agent is set to a positive value, but a larger negative value is obtained when the number of the D2D exceeds the satisfied maximum access number, if the action taken by the agent causes the signal-to-noise ratios of the cellular users and the D2D users to be larger than the set threshold value, the reward function is a negative value as the sum of the corresponding speed and the reward of the selected cellular users of the same multiplexing spectrum is used as the reward, and conversely, if the action taken by the agent causes the signal-to-noise ratio of the D2D or the cellular users to be smaller than the set threshold value, because the signal-to-noise ratio is smaller than the signal-to-noise ratio, the signal cannot be decoded.
The limitation of the number of the access to the authorization-free mode designs a function
Figure FDA0003085521020000041
Limiting SINR of D2D and CU
Figure FDA0003085521020000042
Wherein
Figure FDA0003085521020000043
Figure FDA0003085521020000044
6. The method of claim 5 for joint intelligent allocation of granted and denied D2D communication resources, further comprising: in step S5, the hyper-parameter γ, α in the network is first initializedθ,αλ,
Figure FDA0003085521020000045
Beta, state s0And parameters in the Actor, Critic network
Figure FDA0003085521020000046
Each agent takes the action with the highest probability according to the policy network of the agent as the action taken in the current state, so that the action states taken by all agents are combined to obtain the action state from the environment state stCombined action oftD2D SINR reward
Figure FDA0003085521020000047
And the next state st+1。
7. The method of claim 6, further characterized in that in step S6, the monte carlo method provides a basis for model-free learning, but they still have the limitation of discontinuous off-line learning. The TD error method makes up the difference between the Monte Carlo method and the dynamic programming, and is the core idea of RL. The TD method can also learn in a model-free environment, and can iteratively learn from value estimates (online learning), allowing training in a continuous environment. Calculating TD error from Critic network:
Figure FDA0003085521020000051
strategy parameter updating is carried out based on strategy gradient, and TD error adopts a gradient ascending method:
λt+1＝λt+αλ▽λQλ(st,ut)δt
to solve the problem of confidence allocation in a multi-agent. The COMA algorithm solves the confidence assignment problem with a counterfactual baseline by marginalizing the effect of a single agent on the reward and comparing the action taken by the agent at a certain time t with all other actions that may be taken at t, by centralized Critic, so that the value function is the same for all agents, but each agent gets a specific error term based on its counterfactual action. The jth agent counterfactual baseline is defined as:
Figure FDA0003085521020000052
the jth agent updates the Actor network parameters of the jth agent through the counterfactual base line according to the formula:
Figure FDA0003085521020000053
and the intelligent agent updates the network according to the merit function obtained by the Critic network.
8. The method of claim 7, wherein the training process is performed by the BS in step S7, history information collected during the performance process of the D2D user is uploaded to the BS, the BS performs centralized training, and Critic obtains the policy of the agent at the BS to evaluate how well to take the action. The training process is completed at the BS, the historical information collected by the D2D user in the execution process is uploaded to the BS, the centralized training is completed at the BS, and the Critic obtains the strategy of the intelligent agent at the BS to evaluate the quality of the action. In the distributed execution process, the D2D user obtains A from the base stationj(s, u) updating own Actor network, wherein the Actor selects behaviors based on the state observed by the agent from the environment, the agent continuously interacts with the environment, the agent performs enough training times, and finally converges on an action with the maximum reward value, thereby obtaining the optimal strategy.
CN202110581716.5A 2021-05-26 2021-05-26 Authorized and unauthorized D2D communication resource joint intelligent distribution method Active CN113316154B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110581716.5A CN113316154B (en) 2021-05-26 2021-05-26 Authorized and unauthorized D2D communication resource joint intelligent distribution method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110581716.5A CN113316154B (en) 2021-05-26 2021-05-26 Authorized and unauthorized D2D communication resource joint intelligent distribution method

Publications (2)

Publication Number Publication Date
CN113316154A true CN113316154A (en) 2021-08-27
CN113316154B CN113316154B (en) 2022-06-21

Family

ID=77375597

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110581716.5A Active CN113316154B (en) 2021-05-26 2021-05-26 Authorized and unauthorized D2D communication resource joint intelligent distribution method

Country Status (1)

Country Link
CN (1) CN113316154B (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114363938A (en) * 2021-12-21 2022-04-15 重庆邮电大学 Cellular network flow unloading method
CN114466386A (en) * 2022-01-13 2022-05-10 重庆邮电大学 Direct access method for D2D communication
CN114928549A (en) * 2022-04-20 2022-08-19 清华大学 Communication resource allocation method and device of unauthorized frequency band based on reinforcement learning
CN115278593A (en) * 2022-06-20 2022-11-01 重庆邮电大学 Transmission method of unmanned aerial vehicle-non-orthogonal multiple access communication system based on semi-authorization-free protocol
CN115515101A (en) * 2022-09-23 2022-12-23 西北工业大学 Decoupling Q learning intelligent codebook selection method for SCMA-V2X system
CN116367332A (en) * 2023-05-31 2023-06-30 华信咨询设计研究院有限公司 Hierarchical control-based D2D resource allocation method under 5G system
WO2024032228A1 (en) * 2022-08-12 2024-02-15 华为技术有限公司 Reinforcement learning training method and related device

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160066337A1 (en) * 2014-09-03 2016-03-03 Futurewei Technologies, Inc. System and Method for D2D Resource Allocation
US20190124667A1 (en) * 2017-10-23 2019-04-25 Commissariat A L'energie Atomique Et Aux Energies Alternatives Method for allocating transmission resources using reinforcement learning
CN110267338A (en) * 2019-07-08 2019-09-20 西安电子科技大学 Federated resource distribution and Poewr control method in a kind of D2D communication
CN110493826A (en) * 2019-08-28 2019-11-22 重庆邮电大学 A kind of isomery cloud radio access network resources distribution method based on deeply study
WO2019231289A1 (en) * 2018-06-01 2019-12-05 Samsung Electronics Co., Ltd. Method and apparatus for machine learning based wide beam optimization in cellular network
CN110769514A (en) * 2019-11-08 2020-02-07 山东师范大学 Heterogeneous cellular network D2D communication resource allocation method and system
CN111556572A (en) * 2020-04-21 2020-08-18 北京邮电大学 Spectrum resource and computing resource joint allocation method based on reinforcement learning
US20210136785A1 (en) * 2019-10-30 2021-05-06 University Of Ottawa System and method for joint power and resource allocation using reinforcement learning
CN112822781A (en) * 2021-01-20 2021-05-18 重庆邮电大学 Resource allocation method based on Q learning

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160066337A1 (en) * 2014-09-03 2016-03-03 Futurewei Technologies, Inc. System and Method for D2D Resource Allocation
US20190124667A1 (en) * 2017-10-23 2019-04-25 Commissariat A L'energie Atomique Et Aux Energies Alternatives Method for allocating transmission resources using reinforcement learning
WO2019231289A1 (en) * 2018-06-01 2019-12-05 Samsung Electronics Co., Ltd. Method and apparatus for machine learning based wide beam optimization in cellular network
CN110267338A (en) * 2019-07-08 2019-09-20 西安电子科技大学 Federated resource distribution and Poewr control method in a kind of D2D communication
CN110493826A (en) * 2019-08-28 2019-11-22 重庆邮电大学 A kind of isomery cloud radio access network resources distribution method based on deeply study
US20210136785A1 (en) * 2019-10-30 2021-05-06 University Of Ottawa System and method for joint power and resource allocation using reinforcement learning
CN110769514A (en) * 2019-11-08 2020-02-07 山东师范大学 Heterogeneous cellular network D2D communication resource allocation method and system
CN111556572A (en) * 2020-04-21 2020-08-18 北京邮电大学 Spectrum resource and computing resource joint allocation method based on reinforcement learning
CN112822781A (en) * 2021-01-20 2021-05-18 重庆邮电大学 Resource allocation method based on Q learning

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Y.LUO: "Dynamic resource allocations based on Q-learning for D2D communication in cellular networks", 《014 11TH INTERNATIONAL COMPUTER CONFERENCE ON WAVELET ACTIEV MEDIA TECHNOLOGY AND INFORMATION PROCESSING》 *
滑思忠: "D2D 通信资源复用分配奖惩加权算法研究", 《计算机应用研究》 *

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114363938A (en) * 2021-12-21 2022-04-15 重庆邮电大学 Cellular network flow unloading method
CN114363938B (en) * 2021-12-21 2024-01-26 深圳千通科技有限公司 Cellular network flow unloading method
CN114466386A (en) * 2022-01-13 2022-05-10 重庆邮电大学 Direct access method for D2D communication
CN114466386B (en) * 2022-01-13 2023-09-29 深圳市晨讯达科技有限公司 Direct access method for D2D communication
CN114928549A (en) * 2022-04-20 2022-08-19 清华大学 Communication resource allocation method and device of unauthorized frequency band based on reinforcement learning
CN115278593A (en) * 2022-06-20 2022-11-01 重庆邮电大学 Transmission method of unmanned aerial vehicle-non-orthogonal multiple access communication system based on semi-authorization-free protocol
WO2024032228A1 (en) * 2022-08-12 2024-02-15 华为技术有限公司 Reinforcement learning training method and related device
CN115515101A (en) * 2022-09-23 2022-12-23 西北工业大学 Decoupling Q learning intelligent codebook selection method for SCMA-V2X system
CN116367332A (en) * 2023-05-31 2023-06-30 华信咨询设计研究院有限公司 Hierarchical control-based D2D resource allocation method under 5G system
CN116367332B (en) * 2023-05-31 2023-09-15 华信咨询设计研究院有限公司 Hierarchical control-based D2D resource allocation method under 5G system

Also Published As

Publication number Publication date
CN113316154B (en) 2022-06-21

Similar Documents

Publication Publication Date Title
CN113316154B (en) Authorized and unauthorized D2D communication resource joint intelligent distribution method
CN109474980B (en) Wireless network resource allocation method based on deep reinforcement learning
Azari et al. Risk-aware resource allocation for URLLC: Challenges and strategies with machine learning
Wu et al. Secrecy-based energy-efficient data offloading via dual connectivity over unlicensed spectrums
US20210326695A1 (en) Method and apparatus employing distributed sensing and deep learning for dynamic spectrum access and spectrum sharing
CN105230070B (en) A kind of wireless resource allocation methods and radio resource allocation apparatus
Wu et al. QoE-based distributed multichannel allocation in 5G heterogeneous cellular networks: A matching-coalitional game solution
Gu et al. Dynamic path to stability in LTE-unlicensed with user mobility: A matching framework
Amichi et al. Spreading factor allocation strategy for LoRa networks under imperfect orthogonality
CN112351433A (en) Heterogeneous network resource allocation method based on reinforcement learning
Zhang et al. Resource optimization-based interference management for hybrid self-organized small-cell network
Bi et al. Deep reinforcement learning based power allocation for D2D network
Yuan et al. Deep reinforcement learning for resource allocation with network slicing in cognitive radio network
Jiang Reinforcement learning-based spectrum sharing for cognitive radio
Kaur et al. Intelligent spectrum management based on reinforcement learning schemes in cooperative cognitive radio networks
Lall et al. Multi-agent reinfocement learning for stochastic power management in cognitive radio network
CN110049436B (en) Distributed channel allocation and sharing method and system based on heterogeneous spectrum
CN115811788B (en) D2D network distributed resource allocation method combining deep reinforcement learning and unsupervised learning
Banitalebi et al. Distributed Learning-Based Resource Allocation for Self-Organizing C-V2X Communication in Cellular Networks
CN113286374A (en) Scheduling method, training method of scheduling algorithm, related system and storage medium
Khuntia et al. An actor-critic reinforcement learning for device-to-device communication underlaying cellular network
CN115915454A (en) SWIPT-assisted downlink resource allocation method and device
Chen et al. Power allocation based on deep reinforcement learning in HetNets with varying user activity
Kim Femtocell network power control scheme based on the weighted voting game
Ginde et al. A game-theoretic analysis of link adaptation in cellular radio networks

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant