WO2023077597A1 - Cell selection method and device - Google Patents

Abstract

A cell selection method and device, the method comprising: determining at least one reward condition for cell selection and a reward value corresponding to the at least one reward condition; and training a reinforcement learning model for cell selection according to the reward value corresponding to the at least one reward condition.

Description

Method and device for cell selection

This application claims the priority of the PCT patent application with the application number PCT/CN2021/128217 and the title of the invention "Method and device for cell selection" filed with the China Patent Office on November 02, 2021, the entire contents of which are incorporated herein by reference In this application.

technical field

The embodiments of the present application relate to the communication field, and in particular to a method and device for cell selection.

Background technique

In the New Radio (NR) system, when a terminal device using network services moves from one cell to another, or due to wireless transmission business load adjustment, activation operation and maintenance, equipment failure, etc., in order to ensure To ensure the continuity of communication and the quality of service, the system needs to transfer the communication link between the terminal equipment and the source cell to a new cell, that is, to perform a handover process.

The traditional handover relies on the measurement report of the terminal equipment to select the target cell, which may trigger unnecessary handover procedures, such as ping-pong handover. The signaling interaction caused by measurement configuration and measurement reporting may cause a large transmission delay. In this way, the timeliness of the measurement results cannot be guaranteed when the handover command is issued, resulting in handover failure. Therefore, how to perform cell selection to improve the handover success rate is an urgent problem to be solved.

Contents of the invention

The present application provides a method and device for cell selection, which uses a reinforcement learning model to select a cell, which is beneficial for a terminal device to select a suitable cell.

In a first aspect, a method for cell selection is provided, including: determining at least one reward condition for cell selection and a reward value corresponding to the at least one reward condition; according to the at least one reward condition and the at least one reward The reward value corresponding to the condition trains the reinforcement learning model for cell selection.

In a second aspect, a method for cell selection is provided, including: using a reinforcement learning model to determine a selected target cell according to state information of a terminal device in multiple cells.

In a third aspect, a device for cell selection is provided, configured to perform the method in the above first aspect or various implementations thereof.

Specifically, the terminal device includes a functional module for executing the method in the above first aspect or its various implementation manners.

In a fourth aspect, a device for cell selection is provided, configured to execute the method in the above second aspect or various implementations thereof.

Specifically, the network device includes a functional module for executing the method in the above second aspect or each implementation manner thereof.

In a fifth aspect, a communication device is provided, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the above first aspect or its various implementations.

In a sixth aspect, a communication device is provided, including a processor and a memory. The memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the above second aspect or its various implementations.

In a seventh aspect, a chip is provided for implementing any one of the above first aspect to the second aspect or the method in each implementation manner thereof.

Specifically, the chip includes: a processor, configured to call and run a computer program from the memory, so that the device installed with the device executes any one of the above-mentioned first to second aspects or any of the implementations thereof. method.

In an eighth aspect, there is provided a computer-readable storage medium for storing a computer program, and the computer program causes a computer to execute any one of the above-mentioned first to second aspects or the method in each implementation manner thereof.

A ninth aspect provides a computer program product, including computer program instructions, the computer program instructions cause a computer to execute any one of the above first to second aspects or the method in each implementation manner.

In a tenth aspect, a computer program is provided, which, when running on a computer, causes the computer to execute any one of the above-mentioned first to second aspects or the method in each implementation manner.

Through the above technical solution, by setting at least one reward condition and corresponding reward value for cell selection, and further training the reinforcement learning model for cell selection based on the at least one reward condition and its corresponding reward value, it is beneficial to select to a suitable cell, avoiding the problems in traditional handover.

Description of drawings

FIG. 1 is a schematic diagram of a communication system architecture provided by an embodiment of the present application.

Fig. 2 is a schematic diagram of a decision flow of cell handover in the related art.

Fig. 3 is a schematic flowchart of reinforcement learning.

Fig. 4 is a schematic flowchart of a method for cell selection according to an embodiment of the present application.

Fig. 5 is a schematic diagram of cell deployment according to an embodiment of the present application.

Fig. 6 is a schematic diagram of the maximum optional position of sprinkle points according to an embodiment of the present application.

Fig. 7 is a schematic diagram of cell deployment according to another embodiment of the present application.

Fig. 8 is a schematic diagram of overlapping sectors according to an embodiment of the present application.

Fig. 9 is a schematic flowchart of another method for cell selection according to an embodiment of the present application.

Fig. 10 is a schematic block diagram of a device for cell selection according to an embodiment of the present application.

Fig. 11 is a schematic block diagram of another device for cell selection according to an embodiment of the present application.

Fig. 12 is a schematic block diagram of a communication device provided according to an embodiment of the present application.

Fig. 13 is a schematic block diagram of a chip provided according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. With regard to the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the scope of protection of this application.

The technical solution of the embodiment of the present application can be applied to various communication systems, such as: Global System of Mobile communication (Global System of Mobile communication, GSM) system, code division multiple access (Code Division Multiple Access, CDMA) system, broadband code division multiple access (Wideband Code Division Multiple Access, WCDMA) system, General Packet Radio Service (GPRS), Long Term Evolution (LTE) system, Advanced long term evolution (LTE-A) system , New Radio (NR) system, evolution system of NR system, LTE (LTE-based access to unlicensed spectrum, LTE-U) system on unlicensed spectrum, NR (NR-based access to unlicensed spectrum) on unlicensed spectrum unlicensed spectrum (NR-U) system, Non-Terrestrial Networks (NTN) system, Universal Mobile Telecommunications System (UMTS), Wireless Local Area Networks (WLAN), Wireless Fidelity (Wireless Fidelity, WiFi), fifth-generation communication (5th-Generation, 5G) system or other communication systems, etc.

Generally speaking, the number of connections supported by traditional communication systems is limited and easy to implement. However, with the development of communication technology, mobile communication systems will not only support traditional communication, but also support, for example, Device to Device (Device to Device, D2D) communication, Machine to Machine (M2M) communication, Machine Type Communication (MTC), Vehicle to Vehicle (V2V) communication, or Vehicle to everything (V2X) communication, etc. , the embodiments of the present application may also be applied to these communication systems.

Optionally, the communication system in the embodiment of the present application may be applied to a carrier aggregation (Carrier Aggregation, CA) scenario, may also be applied to a dual connectivity (Dual Connectivity, DC) scenario, and may also be applied to an independent (Standalone, SA) deployment Web scene.

Optionally, the communication system in the embodiment of the present application may be applied to an unlicensed spectrum, where the unlicensed spectrum may also be considered as a shared spectrum; or, the communication system in the embodiment of the present application may also be applied to a licensed spectrum, where, Licensed spectrum can also be considered as non-shared spectrum.

The embodiments of the present application describe various embodiments in conjunction with network equipment and terminal equipment, wherein the terminal equipment may also be referred to as user equipment (User Equipment, UE), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication device, user agent or user device, etc.

The terminal device can be a station (STATION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (Session Initiation Protocol, SIP) phone, a wireless local loop (Wireless Local Loop, WLL) station, a personal digital assistant (Personal Digital Assistant, PDA) devices, handheld devices with wireless communication functions, computing devices or other processing devices connected to wireless modems, vehicle-mounted devices, wearable devices, next-generation communication systems such as terminal devices in NR networks, or future Terminal equipment in the evolved public land mobile network (Public Land Mobile Network, PLMN) network, etc.

In the embodiment of this application, the terminal device can be deployed on land, including indoor or outdoor, handheld, wearable or vehicle-mounted; it can also be deployed on water (such as ships, etc.); it can also be deployed in the air (such as aircraft, balloons and satellites) superior).

In this embodiment of the application, the terminal device may be a mobile phone (Mobile Phone), a tablet computer (Pad), a computer with a wireless transceiver function, a virtual reality (Virtual Reality, VR) terminal device, an augmented reality (Augmented Reality, AR) terminal Equipment, wireless terminal equipment in industrial control, wireless terminal equipment in self driving, wireless terminal equipment in remote medical, wireless terminal equipment in smart grid , wireless terminal equipment in transportation safety, wireless terminal equipment in smart city, or wireless terminal equipment in smart home.

As an example but not a limitation, in this embodiment of the present application, the terminal device may also be a wearable device. Wearable devices can also be called wearable smart devices, which is a general term for the application of wearable technology to intelligently design daily wear and develop wearable devices, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not only a hardware device, but also achieve powerful functions through software support, data interaction, and cloud interaction. Generalized wearable smart devices include full-featured, large-sized, complete or partial functions without relying on smart phones, such as smart watches or smart glasses, etc., and only focus on a certain type of application functions, and need to cooperate with other devices such as smart phones Use, such as various smart bracelets and smart jewelry for physical sign monitoring.

In the embodiment of the present application, the network device may be a device for communicating with the mobile device, and the network device may be an access point (Access Point, AP) in WLAN, a base station (Base Transceiver Station, BTS) in GSM or CDMA , or a base station (NodeB, NB) in WCDMA, or an evolved base station (Evolutional Node B, eNB or eNodeB) in LTE, or a relay station or access point, or a vehicle-mounted device, a wearable device, and an NR network The network equipment (gNB) in the network or the network equipment in the future evolved PLMN network or the network equipment in the NTN network, etc.

As an example but not a limitation, in this embodiment of the present application, the network device may have a mobile feature, for example, the network device may be a mobile device. Optionally, the network equipment may be a satellite or a balloon station. For example, the satellite can be a low earth orbit (low earth orbit, LEO) satellite, a medium earth orbit (medium earth orbit, MEO) satellite, a geosynchronous earth orbit (geosynchronous earth orbit, GEO) satellite, a high elliptical orbit (High Elliptical Orbit, HEO) satellite. ) Satellite etc. Optionally, the network device may also be a base station installed on land, water, and other locations.

In this embodiment of the present application, the network device may provide services for a cell, and the terminal device communicates with the network device through the transmission resources (for example, frequency domain resources, or spectrum resources) used by the cell, and the cell may be a network device ( For example, a cell corresponding to a base station), the cell may belong to a macro base station, or may belong to a base station corresponding to a small cell (Small cell), and the small cell here may include: a metro cell (Metro cell), a micro cell (Micro cell), a pico cell ( Pico cell), Femto cell, etc. These small cells have the characteristics of small coverage and low transmission power, and are suitable for providing high-speed data transmission services.

Exemplarily, a communication system 100 applied in this embodiment of the application is shown in FIG. 1 . The communication system 100 may include a network device 110, and the network device 110 may be a device for communicating with a terminal device 120 (or called a communication terminal, terminal). The network device 110 can provide communication coverage for a specific geographical area, and can communicate with terminal devices located in the coverage area.

FIG. 1 exemplarily shows one network device and two terminal devices. Optionally, the communication system 100 may include multiple network devices and each network device may include other numbers of terminal devices within the coverage area. This application The embodiment does not limit this.

Optionally, the communication system 100 may further include other network entities such as a network controller and a mobility management entity, which is not limited in this embodiment of the present application.

It should be understood that a device with a communication function in the network/system in the embodiment of the present application may be referred to as a communication device. Taking the communication system 100 shown in FIG. 1 as an example, the communication equipment may include a network equipment 110 and a terminal equipment 120 with communication functions. The network equipment 110 and the terminal equipment 120 may be the specific equipment described above, and will not be repeated here. The communication device may also include other devices in the communication system 100, such as network controllers, mobility management entities and other network entities, which are not limited in this embodiment of the present application.

It should be understood that the terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations. In addition, the character "/" in this article generally indicates that the contextual objects are an "or" relationship.

It should be understood that the "indication" mentioned in the embodiments of the present application may be a direct indication, may also be an indirect indication, and may also mean that there is an association relationship. For example, A indicates B, which can mean that A directly indicates B, for example, B can be obtained through A; it can also indicate that A indirectly indicates B, for example, A indicates C, and B can be obtained through C; it can also indicate that there is an association between A and B relation.

In the description of the embodiments of the present application, the term "corresponding" may indicate that there is a direct or indirect correspondence between the two, or that there is an association between the two, or that it indicates and is indicated, configuration and is configuration etc.

In the embodiment of this application, "predefinition" can be realized by pre-saving corresponding codes, tables or other methods that can be used to indicate related information in devices (for example, including terminal devices and network devices). The implementation method is not limited. For example, pre-defined may refer to defined in the protocol.

In the embodiment of the present application, the "protocol" may refer to a standard protocol in the communication field, for example, it may include the LTE protocol, the NR protocol, and related protocols applied in future communication systems, which is not limited in the present application.

In order to facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific examples. The following related technologies may be optionally combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them belong to the protection scope of the embodiments of the present application. The embodiment of the present application includes at least part of the following content.

To facilitate a better understanding of the embodiments of the present application, the switching related to the present application will be described.

Similar to the LTE system, the NR system supports the handover process of the UE in the connected state. When a user using network services moves from one cell to another, or due to reasons such as wireless transmission business load adjustment, activation operation maintenance, equipment failure, etc., in order to ensure the continuity of communication and the quality of service, the system will The communication link between the user and the source cell is transferred to the new cell, that is, the handover process is performed.

Taking the switching process of the Xn interface as an example, the whole switching process is divided into the following three stages:

(1) Handover preparation: including measurement control and reporting, handover request and confirmation. The handover confirmation message includes the handover command generated by the target cell, and the source cell does not allow any modification to the handover command generated by the target cell, and directly forwards the handover command to the UE.

(2) Handover execution: UE immediately executes the handover process after receiving the handover command, that is, the UE disconnects the source cell and connects with the target cell (such as performing random access, sending a radio resource control (Radio Resource Control, RRC) handover completion message to Target base station, etc.); sequence number (sequence number, SN) state transfer, data forwarding.

(3) Handover is completed: the target cell, the Access and Mobility Management Function (AMF) entity and the User Plane Function (UPF) entity execute the path switch (Path Switch), releasing the source base station UE context.

FIG. 2 is a schematic interaction diagram of the handover decision process.

S1. The AMF entity provides mobile control information;

S2. Measurement and reporting;

S3. The source gNB decides to switch;

S4. The source gNB sends a handover request to the target gNB;

S5. The target gNB performs admission control;

S6. The target gNB sends a handover request acknowledgment (Acknowledgment, ACK) to the source gNB;

S7. The UE and the source gNB perform radio access network (Radio Access Network, RAN) handover and start.

That is, the UE receives the measurement configuration of the source base station, and performs measurement and reporting according to the measurement configuration. The source base station decides whether to perform handover based on the UE's measurement report.

In order to facilitate the understanding of the technical solutions of the embodiments of the present application, the reinforcement learning related to the present application will be described.

Reinforcement learning is that the agent (Agent) learns in a "trial and error" way, and the reward-guided behavior obtained by interacting with the environment, the goal is to enable the agent to obtain the maximum reward. Reinforcement learning is different from supervised learning in connectionist learning. , mainly manifested in the reinforcement signal. The reinforcement signal provided by the environment in reinforcement learning is an evaluation of the quality of the generated action (usually a scalar signal), rather than telling the reinforcement learning system (reinforcement learning system, RLS) how to do it. produce the correct action. With little information from the external environment, RLS must learn by experience. In this way, RLS acquires knowledge in an action-evaluation environment and improves the action plan to suit the environment.

Reinforcement learning is mostly used in scenarios that need to interact with the environment. Given an environment state (State), the agent selects a corresponding action (Action) according to a certain policy (Policy), and the environment will happen again after executing this Action. Change, that is, the state will be converted to a new state S', and the program will get a reward value (Reward) after each Action is executed, and the agent adjusts its strategy according to the size of the reward value obtained, so that after all steps are executed After that, that is, when the state reaches the terminal state (Terminal), the sum of the obtained Rewards is the largest.

Fig. 3 is a schematic diagram of reinforcement learning execution flow. Among them, the agent can be understood as a program, the agent can observe the environment and get the state, take action on the state according to the policy, and get a reward, and the environment has changed, so the agent will get a new state and continue to execute.

The improvement of the deep reinforcement learning model (DQN) over reinforcement learning (Q-learning) lies in: one is to use the neural network to approximate the value function, the other is to use the target Q-network to update the target, and the third is to use experience playback. In the deep reinforcement learning model, it mainly includes three parts: state, behavior and reward. The goal is to maximize the reward observed during the interaction between the agent and the environment. Specifically, in the iterative process, the agent observes a set of states from the state space, and selects an action to execute based on the learning strategy from the action space. The decision-making strategy is determined by DQN, and the strategy principle is to maximize the reward for the model.

Traditional handover relies on UE's measurement report to select a target cell, which may trigger unnecessary handover procedures, such as ping-pong handover.

The signaling interaction caused by measurement configuration and measurement reporting may cause a large transmission delay, and the timeliness of the measurement results cannot be guaranteed when the handover command is issued, resulting in handover failure. Therefore, how to perform cell handover to improve the handover success rate is an urgent problem to be solved.

FIG. 4 is a schematic flowchart of a method 200 for cell selection according to an embodiment of the present application. As shown in FIG. 4, the method 200 includes at least part of the following:

S210. Determine at least one reward condition for cell selection and a reward value corresponding to the at least one reward condition;

S220. Train the reinforcement learning model for cell selection according to the reward value corresponding to at least one reward condition.

In some embodiments of the present application, the method 200 may be executed by an agent (agent), and the agent may be set on a terminal device or a network device. That is, a terminal device or a network device can use a reinforcement learning model to select a cell, which is conducive to selecting a suitable cell and improving user experience. In the following, the cell selection by the terminal device using the reinforcement learning model is taken as an example for illustration, and the implementation manner on the network device side is similar, so details are not repeated here.

It should be understood that the present application does not limit a specific reinforcement learning model, for example, it may include but not limited to a deep Q network (DQN) model.

In some embodiments of the present application, the reward condition corresponding to the reinforcement learning model is related to the first information, where the first information includes but is not limited to at least one of the following:

The signal quality information of the cell, the coverage area of the cell, the load information of the cell, and the residence time of the terminal equipment in the cell.

By designing reward conditions related to the first information, and further designing the reward value corresponding to the reward conditions, it is beneficial to assist the terminal equipment to select a suitable cell and reduce common problems in cell selection, such as ping-pong handover, premature handover, handover To the wrong cell and other issues.

In some embodiments of the present application, the signal quality information of a cell may be characterized by at least one of the following indicators:

Reference Signal Receiving Power (RSRP), Reference Signal Receiving Quality (Reference Signal Receiving Quality, RSRQ), Signal to Interference plus Noise Ratio (SINR).

Hereinafter, the RSRP is used as an example to describe the signal quality information of a cell, but the present application is not limited thereto.

In some embodiments of the present application, the at least one reward condition includes a target reward condition that the target cell needs to meet.

That is, a cell that meets the target reward condition can be considered as a target cell, and a cell that does not meet the target reward condition is not considered a target cell. Alternatively, when the cell selected by the terminal device satisfies the target reward condition, it is considered that the handover is successful, and when the cell selected by the terminal device does not meet the target reward condition, it is considered that the handover fails.

Further, a corresponding reward value is assigned to the at least one reward condition.

For example, when the cell selected by the terminal device meets the target reward condition, configure a larger reward value, such as a positive reward value, and configure a smaller reward value, such as a negative reward value, when the cell selected by the terminal device does not meet the target reward condition. reward value.

In some embodiments of the present application, the target reward conditions include at least one of the following:

The signal quality information of the cell is greater than or equal to the signal quality threshold;

The cell has the largest signal quality information among multiple candidate cells;

The terminal device is located within the coverage of the cell;

The load of the cell meets the load threshold;

The dwell time of the terminal device in the cell is greater than the time threshold.

In some embodiments, if the channel quality information of the cell is greater than or equal to the signal quality threshold, it may be considered that the signal quality of the cell is good and can meet the transmission requirements of the terminal device.

In some embodiments, if the load of the cell meets the load threshold, it may be considered that the load capacity of the cell is good and can satisfy the access of the terminal equipment.

In some embodiments, the dwell time of the terminal device in the cell is longer than the duration threshold, and the handover of the terminal device can be considered as a successful handover.

In some embodiments, the load of the cell meets a load threshold, comprising:

The available load of the cell is greater than or equal to the first load threshold, that is, the cell has sufficient available load; and/or

The used load of the cell is less than or equal to the second load threshold, that is, the existing load of the cell is less, in other words, the cell still has enough available load.

In some embodiments, the signal quality threshold may be preconfigured. For example, the signal quality threshold may be preconfigured to the terminal device as environmental data (specifically, a reinforcement learning model in the terminal device).

In some embodiments, the load threshold, such as the first load threshold or the second load threshold, may be pre-configured. For example, the first load threshold or the second load threshold may be preconfigured to the terminal device (specifically, a reinforcement learning model in the terminal device) as environment data.

In some embodiments, the duration threshold may be preconfigured. For example, the duration threshold may be preconfigured to the terminal device (specifically, a reinforcement learning model in the terminal device) as environment data.

In some embodiments of the present application, corresponding target reward conditions may be set according to different optimization targets.

In some embodiments, the target bonus condition includes a first bonus condition, wherein the first bonus condition includes:

The signal quality information of the cell is greater than or equal to the signal quality threshold, and the signal quality information of the cell is the largest among multiple candidate cells.

For example, among the candidate cells whose signal quality satisfies the signal quality threshold, the cell with the best signal quality is selected.

Therefore, performing model training based on the first reward condition is conducive to selecting a cell with the best signal quality.

In some other embodiments, the target reward condition includes a second reward condition, wherein the second reward condition includes:

The signal quality information of the cell is greater than or equal to the signal quality threshold, the terminal device is located within the coverage of the cell, and the signal quality information of the cell is the largest among multiple candidate cells.

For example, among the candidate cells whose signal quality satisfies the signal quality threshold and where the terminal device is located within the coverage of the cell, the cell with the best signal quality is selected.

Therefore, performing model training based on the second reward condition is conducive to selecting a cell with the best signal quality and the terminal location meeting the cell coverage.

In yet other embodiments, the target reward condition includes a third reward condition, wherein the third reward condition includes:

The signal quality information of the cell is greater than or equal to the signal quality threshold, the load of the cell meets the load threshold, and the signal quality information of the cell is the largest among multiple candidate cells.

For example, among the candidate cells whose signal quality satisfies the signal quality threshold and whose load satisfies the load threshold, the cell with the best signal quality is selected.

Therefore, performing model training based on the third reward condition is conducive to selecting a cell with the best signal quality and the cell load meeting the load threshold.

In still other embodiments, the target reward condition includes a fourth reward condition, wherein the fourth reward condition includes:

The signal quality information of the cell is greater than or equal to the signal quality threshold, the load of the cell meets the load threshold, the terminal device is located within the coverage of the cell, and the signal quality information of the cell is the largest among multiple candidate cells.

For example, among the candidate cells whose signal quality meets the signal quality threshold, where the terminal is located within the coverage of the cell, and whose load meets the load threshold, the cell with the best signal quality is selected.

Therefore, performing model training based on the fourth reward condition is conducive to selecting a cell with the best signal quality, the terminal location meets the cell coverage, and the cell load meets the load threshold.

In some embodiments, the target bonus condition includes at least one of the first bonus condition, the second bonus condition, the third bonus condition, and the fourth bonus condition.

In some embodiments, the plurality of candidate cells may include all neighboring cells around the current location of the terminal device.

In some embodiments of the present application, the terminal device may sequentially train the reinforcement learning model based on different target reward conditions.

For example, the training is performed on the basis of each target reward condition sequentially according to the sequence of constraint conditions corresponding to the reward condition from less to more.

As an example, firstly, the reinforcement learning model is trained based on the first reward condition, and then the reinforcement learning model is trained based on the second reward condition when the reinforcement learning model converges, and then the reinforcement learning model is trained based on the second reward condition The reinforcement learning model is trained on the three reward conditions, and the reinforcement learning model is trained based on the fourth reward condition when the reinforcement learning model converges.

In some embodiments of the present application, the reward value when the selected cell satisfies different reward conditions may be defined.

For example, when the selected cell satisfies the target reward condition, a first reward value is given.

For another example, when the selected cell does not satisfy the target reward condition, a second reward value is given.

Wherein, the first reward value is greater than the second reward value.

In some embodiments, it can be set that when the terminal device selects a cell that meets the aforementioned first reward condition, second reward condition, third reward condition and fourth reward condition, it corresponds to the same reward value, or it can also correspond to different reward values value.

For example, when the terminal device selects a cell that satisfies the first reward condition, the second reward condition, the third reward condition and the fourth reward condition, it will give the first reward value.

For another example, when the terminal device selects a cell that satisfies the first reward condition, reward value X is given; when the terminal device selects a cell that meets the second reward condition, reward value Y is given; when the terminal device selects a cell that meets the third reward condition, reward value Y is given. For the reward value Z, when the terminal device selects a cell satisfying the fourth reward condition, the reward value P is given, where X<Y<Z<P.

In some embodiments of the present application, the reinforcement learning model used for cell selection may be trained according to the state space of the terminal device, the aforementioned at least one reward condition, and the corresponding reward value. Alternatively, the reinforcement learning model may be trained according to the state space of the terminal device, the behavior space of the terminal device, the aforementioned at least one reward condition, and the corresponding reward value.

Optionally, the state space of the terminal device may be used to describe state information of the terminal device in multiple cells.

In some embodiments, the state space of the terminal device includes state information of the terminal device at multiple moments, for example, includes state information of the terminal device at a first moment. Optionally, the status information of the terminal device at the first moment includes but is not limited to at least one of the following:

Information about the cell to which the terminal device belongs at the first moment, such as cell identification information (Cell ID);

Signal quality information of the cell to which the terminal device belongs at the first moment, such as RSRP size;

The switching status information of the terminal device at the first moment is used to indicate whether the terminal device is switched at the first moment;

The location information of the terminal device at the first moment, such as three-dimensional coordinates.

For example, taking signal quality information represented by RSRP as an example, the state information of terminal device n at time t can be expressed as

表示时刻t终端设备n所属(或者，所选择的，所驻留的)的小区(或者说，扇区)。 in,

Indicates the cell (or sector) to which the terminal device n belongs (or, selected, camped) at time t.

RSRP _t ⁿ represents the RSRP size of the cell to which terminal device n belongs at time t.

表示切换的状态，例如，取值为0表示没有切换，取值为1表示切换。是否切换的判断依据为：终端设备在时刻t选择的小区和上一时刻选择的小区是否发生了变化。

Indicates the state of switching, for example, a value of 0 means no switching, and a value of 1 means switching. The basis for judging whether to switch is: whether the cell selected by the terminal device at time t has changed from the cell selected at the previous time.

P _t ⁿ represents the location information of terminal device n.

It should be understood that the above status information of the terminal device is only an example, and in other embodiments, the status information of the terminal device may also include other information, such as cell load information, or other information used to assist in cell selection. There is no limit to this.

In some embodiments, the behavior space of the terminal device may include behavior information of the terminal device at multiple moments, for example, including behavior information of the terminal device at the first moment, and the behavior information of the terminal device at the first moment is used to indicate that the terminal The device selects a certain cell at the first moment.

表示终端设备n在时刻t选择的小区。 For example, the behavior information of terminal device n at time t can be expressed as

Indicates the cell selected by terminal device n at time t.

表示发生了小区切换，否则，表示未发生切换。 Among them, if

Indicates that a cell handover has occurred; otherwise, it indicates that a handover has not occurred.

In some embodiments of the present application, the terminal device may determine the corresponding reward value according to the state information at each time in the multiple moments in combination with the aforementioned at least one reward condition.

For example, according to the status information at the first moment, it is determined whether the cell to which the terminal device belongs at the first moment is the target cell, and the corresponding reward value is further determined.

As an example, if at the first moment, the RSRP of the cell to which the terminal device belongs satisfies the RSRP threshold, and the RSRP of the cell is the largest among the candidate cells, then the cell is determined to be the target cell, and the first reward value is given.

As an example, if at the first moment, the RSRP of the cell to which the terminal device belongs satisfies the RSRP threshold, the location of the terminal device is within the coverage of the cell, and the RSRP of the cell is the largest among the candidate cells, then the cell is determined to be the target cell, give the first reward value.

As an example, if the RSRP of the cell to which the terminal device belongs does not meet the RSRP threshold at the first moment, it is determined that the cell is not the target cell, and a second reward value is given.

As an example, the aforementioned first reward condition and corresponding reward value can be defined as:

表示终端设备n在时刻t执行动作

得到的奖励值。 in,

Indicates that terminal device n executes an action at time t

The reward value obtained.

As an example, the aforementioned second reward condition and corresponding reward value can be defined as:

表示终端设备n在时刻t执行动作

得到的奖励值。 in,

Indicates that terminal device n executes an action at time t

The reward value obtained.

As an example, the aforementioned third reward condition and corresponding reward value can be defined as:

表示终端设备n在时刻t执行动作

得到的奖励值。 in,

Indicates that terminal device n executes an action at time t

The reward value obtained.

As an example, the aforementioned fourth reward condition and corresponding reward value can be defined as:

表示终端设备n在时刻t执行动作

得到的奖励值。 in,

Indicates that terminal device n executes an action at time t

The reward value obtained.

In some embodiments of the present application, the state space and behavior space of the terminal device may be acquired according to a simulated action trajectory of the terminal device.

For example, first, the terminal device randomly selects a cell within the range of selectable cells as the starting point of the trajectory, and determines the cell to which the terminal device belongs, the signal quality information of the cell, the handover status information, etc. at the current time and at the current coordinates, that is, the current Time status information. It is further determined whether the cell is a target cell based on the aforementioned target reward condition, and then the corresponding reward value is obtained.

Furthermore, the terminal device starts to move. For example, the terminal device uses the probability hyperparameter and the Q strategy to select the cell to switch to. After switching to this cell, it can obtain the state information at the coordinates after the next move, that is, the cell information after switching , signal quality information of the cell after handover, handover state information, and the like. Based on the aforementioned target reward conditions, it is determined whether the cell is a target cell, and then the corresponding reward value can be obtained. Based on two state information adjacent in time, state switching can be established, and state switching samples can be obtained, including state information at the current moment, action (that is, which cell to switch to), reward value, and state information at the next moment. The state switching samples are then stored in the experience pool for training the reinforcement learning model.

In some embodiments of the present application, the selection of a cell by the terminal device may be regarded as corresponding to one handover, that is, the selection of a cell by the terminal device may be equivalent to switching to the cell by the terminal device.

In some embodiments of the present application, the terminal device may determine the immediate reward value corresponding to cell selection (or cell switching) according to the state space of the terminal device. In other embodiments, the terminal device may also consider that the terminal device switches to a certain The state of each cell after a period of time determines the delay reward value corresponding to cell selection (or cell switching).

For example, the terminal device can switch the corresponding event type according to the dwell time in the cell, for example, whether it is a ping-pong handover, whether it is a premature handover event, whether it is a handover to a wrong cell event, etc., and give the corresponding delay reward value. It is beneficial to avoid handover events such as ping-pong handover, premature handover, and handover to a wrong cell.

For example, when the cell selected by the terminal device satisfies the aforementioned target reward condition, and the selected cell is different from the cell selected at the previous moment, in this case, it may be considered as a successful handover, and a third reward value is given.

For another example, when the cell selected by the terminal device satisfies the target reward condition, but the selected cell is the same as the cell selected at the previous moment, in this case, it can be considered as a ping-pong handover, and a fourth reward value is given, wherein, The fourth bonus value is smaller than the third bonus value.

For another example, when the cell selected by the terminal device does not meet the target reward condition, but it is successfully switched to the cell again within a certain period of time, or in other words, after a certain period of time, the cell meets the target reward condition, this situation can be regarded as is a premature switching event, the fifth reward value is given. Wherein, the fifth reward value is smaller than the third reward value.

For another example, when the cell selected by the terminal device does not satisfy the target reward condition and the handover event is handover to a wrong cell, a sixth reward value is given. Wherein, the sixth reward value is smaller than the third reward.

For another example, when the cell selected by the terminal device satisfies the target reward condition, the cumulative reward value for this handover is determined according to the dwell time of the terminal device in the cell. For example, if the cell selected by the terminal device at the previous moment is the same as the cell selected at the current moment, the reward value will be accumulated, and the longer the dwell time, the higher the accumulated reward value.

In some embodiments of the present application, the terminal device also needs to acquire environment data for model training, for example, location coordinates of the terminal device, location coordinates of the base station, signal quality information of the cell, and the like.

It should be understood that the environment data may be collected in any network environment, which is not limited in this application. The following describes the collection of environmental data in two typical network scenarios.

Scenario 1: Cells do not overlap

Step 1: Set up the network environment.

For example, the network environment may be an urban microcell (UMI) scenario, and the base station is lower than surrounding buildings.

As an example, the scene layout is a hexagonal network. Considering 19 micro base stations, and each micro base station has 3 sectors, there are 57 cells in total. Fig. 5 is a schematic diagram of cell deployment.

Step 2: Determine the trajectory of the end device.

First, determine the starting point of terminal equipment movement. For example, use the sprinkle point positioning method. For example, the starting position of the terminal device can be randomly selected within the range of the sprinkle point.

For example, in [-200,200], sprinkle points at intervals of ten meters horizontally and vertically, covering base stations 1-7 and cells 1-21. Fig. 6 is a schematic diagram of scattered spots of optional positions of terminal equipment.

Then, when the terminal device starts to move, each step can choose four directions: up, down, left, and right, and the direction is selected randomly.

Step 3: Determine the environment data, such as the base station location in the terminal equipment movement track, the terminal equipment location, the signal quality of the cell and other information.

a. Base station location coordinates: The data dimension is 3x19, including the three-dimensional coordinate points of 19 base stations.

B. The position coordinates of the terminal equipment: the data dimension is 3x1681, including 1681 three-dimensional coordinate points of the optional trajectory points of the terminal

C. RSRP: The data dimension is 1681x57, including all RSRP values of 57 cells corresponding to 1681 terminal points.

Step 4: Threshold Setting

For example, setting the RSRP threshold: consider that the selected cell must meet the RSRP threshold range, for example, the set value is -114dB.

For another example, set the cell load threshold: randomly assign loads to all cells, for example, randomly distribute between 0-20, consider that the selected cell must meet the cell load threshold range, and set the value to 15.

Scenario 2: Cells do not overlap

Step 1: Set up the network environment.

For example, the network environment may be an urban microcell (UMI) scenario, and the base station is lower than surrounding buildings.

As an example, the scene layout is a hexagonal network. Considering 38 micro base stations, and each micro base station has 3 sectors, there are 114 cells in total. FIG. 7 is a schematic diagram of cell deployment, and FIG. 8 is a schematic diagram of overlapping sectors.

Step 2: Determine the trajectory of the end device.

First, determine the starting point of terminal equipment movement. For example, use the sprinkle point positioning method. For example, the starting position of the terminal device can be randomly selected within the scattered point range,

For example, in [-200,200], sprinkle points at intervals of ten meters horizontally and vertically, covering base stations 1-7 and cells 1-21. Fig. 5 is a schematic diagram of scattered spots of optional positions of the terminal equipment.

Then, when the terminal device starts to move, each step can choose four directions: up, down, left, and right, and the direction is selected randomly.

Step 3: Determine the environment data, such as the base station location in the terminal equipment movement track, the terminal equipment location, the signal quality of the cell and other information.

a. Base station location coordinates: The data dimension is 3x38, including the three-dimensional coordinate points of 38 base stations.

B. The position coordinates of the terminal equipment: the data dimension is 3x1681, including 1681 three-dimensional coordinate points of the optional trajectory points of the terminal

C. RSRP: The data dimension is 1681x114, including all RSRP values of 114 cells corresponding to 1681 terminal points.

Step 4: Threshold Setting

For example, setting the RSRP threshold: consider that the selected cell must meet the RSRP threshold range, for example, the set value is -114dB.

For another example, set the cell load threshold: randomly assign loads to all cells, for example, randomly distribute between 0-20, consider that the selected cell must meet the cell load threshold range, and set the value to 15.

It should be understood that the above-mentioned scene layout and environment data are only examples, which may be adjusted according to a specific optimization goal, which is not limited in the present application.

In the following, the training process of the model will be described by taking the reinforcement learning model as the DQN model as an example.

Step 1. Initialize the DQN model, for example, set the number of training rounds of the DQN model, take 1000 rounds as an example, and the batch size, for example, set it to 64, or 128, etc.

Step 2, reset the state space of the terminal device.

Further, a location is randomly selected as the starting point of the trajectory of the terminal device. Taking the scenario shown in FIG. 6 as an example, a certain location in cell 1 (denoted as location 1) is selected as the starting point, and the current moment is recorded as the first moment.

Determine the status information of the terminal device at the first moment, that is, under the coordinates of the current location (namely, location 1), the cell to which the terminal device belongs, such as cell 1, the RSRP size of the corresponding cell (for example, the RSRP size of cell 1), and the corresponding Switching state (for the initial moment, the switching state is 0, indicating no switching).

Then, according to the state information at the first moment, it is determined whether the cell to which the terminal equipment belongs is a target cell. The fourth reward condition determines whether the cell to which the terminal device belongs is the target cell. Further, the reward value corresponding to the current state of the terminal device is determined, or it can also be regarded as the reward value corresponding to the terminal device selecting cell 1, or the reward value corresponding to the terminal device switching to cell 1.

Step 3: The terminal device starts to move. For example, the terminal device uses the probability hyperparameter and the Q strategy to select a cell to switch to. After switching to this cell, the coordinates (recorded as position 2) under the status information. That is, the cell information after the handover, the signal quality information of the cell after the handover, the handover state information, and the like.

Taking the scenario shown in FIG. 6 as an example, assuming that the terminal device switches to cell 6 at the second moment, the status information at the second moment may include identification information of cell 6, signal quality information of cell 6, location information of the terminal device, and corresponding The switching status of , for example, a value of 1 indicates that switching has occurred.

Then determine whether the sub-district to which the terminal equipment belongs (such as sub-district 6) is a target sub-district according to the state information at the second moment, and the specific judgment conditions are related to the optimization target, for example, according to the aforementioned first reward condition, the second reward condition, the second reward condition A third bonus condition or a fourth bonus condition is determined. Further, the reward value corresponding to the current state of the terminal device is determined, or it can also be regarded as the reward value corresponding to the terminal device selecting cell 6, or the reward value corresponding to the terminal device switching to cell 6.

Furthermore, based on two temporally adjacent state information, state switching can be established, and state switching samples can be obtained, including state information at the current moment, action (that is, which cell to switch to), reward value, and state information at the next moment. The state switching samples are then stored in the experience pool for training the reinforcement learning model.

For example, for the aforementioned example, the state information at the current moment in the state switching sample can be the state information at the first moment, the action can be switching to cell 6, the state information at the next moment can be the state information at the second moment, and the reward value is the reward value for switching to cell 6.

Step 4. When the number of samples in the experience pool is greater than the batch size, select (for example, randomly select) a sample of the batch size from the resource pool, and use the samples to train the DQN model.

For example, in step four, the Q value of the DQN model can be calculated using the formula Q(s,a)=E[R _s +γmaxQ(s',a)|s,a], where R _s is Next, the reward value obtained by taking action A, γ is the discount factor, γ∈(0,1], for example, set to 0.9, the discount coefficient reflects the impact of the old action on the Q value.

Step 5: Return to step 3 until the trajectory exploration of 1000 steps is completed.

When the 1000-step trajectory exploration is completed, return to step 2 and enter the next round of training until the training is completed, and output the DQN training model.

Step 6: Output the reward training map.

In some embodiments of the present application, the DQN model may be trained according to the aforementioned first reward condition, second reward condition, third reward condition and fourth reward condition in sequence.

To sum up, in the embodiment of the present application, by setting at least one reward condition for cell selection and the corresponding reward value, the reward condition takes into account various factors for cell selection (for example, the signal quality of the cell, the load, the coverage of the cell, etc.), further, according to the at least one reward condition and the corresponding reward value, using the historical trajectory of the terminal device to select the cell as experience to train the reinforcement learning model, which is conducive to selecting a suitable cell, Avoid problems such as ping-pong switching and premature switching in traditional cell selection.

The method for cell selection according to the embodiment of the present application is described above in conjunction with FIG. 4 to FIG. 8 from the perspective of model training. The method for cell selection according to another embodiment of the present application is described in detail below in conjunction with FIG. Method of cell selection.

FIG. 9 is a schematic flowchart of a method 300 for cell selection according to another embodiment of the present application. As shown in FIG. 9, the method 300 includes at least part of the following content:

S310, using a reinforcement learning model to determine a selected target cell (or in other words, a handover target cell) according to state information of the terminal device in multiple cells.

It should be understood that, in the embodiment of the application, the method 300 may be executed by a terminal device, or may also be executed by a network device.

In some embodiments of the present application, the reinforcement learning model may be obtained through training using the method described in method 200 .

It should be understood that the present application does not limit a specific reinforcement learning model, for example, it may include but not limited to a deep Q network (DQN) model.

In the method 300, the state information of the terminal device in multiple cells may correspond to the state space of the terminal device in the method 200.

In some embodiments of the present application, the method 300 may be a testing method after reinforcement learning model training.

In this case, the method 300 may also include:

Whether the handover is successful is determined according to whether the target cell satisfies a preset condition. That is, determine the handover success rate of cell handover based on the reinforcement learning model.

In some embodiments of the present application, the preset condition is related to at least one of the following:

The signal quality information of the cell, the coverage area of the cell, the load information of the cell, and the residence time of the terminal equipment in the cell.

As an example but not a limitation, the preset conditions include at least one of the following:

The signal quality information of the target cell is greater than or equal to the signal quality threshold;

The signal quality information of the target cell among multiple candidate cells is the largest;

The terminal device is located within the coverage of the target cell;

The load of the target cell meets the load threshold;

The dwell time of the terminal device in the target cell is longer than the time threshold.

In some embodiments, the load of the target cell meets a load threshold, including:

The available load of the target cell is greater than or equal to the first load threshold; and/or

The used load of the target cell is less than or equal to the second load threshold.

In some embodiments, the preset condition may correspond to the target reward condition in the aforementioned method 200 .

In some embodiments, for example, if the reinforcement learning model is trained based on the first reward condition in the foregoing method 200, the preset condition may be the first reward condition.

For example, the preset condition is that the signal quality information of the target cell is greater than or equal to the signal quality threshold, and the signal quality information of the target cell is the largest among multiple candidate cells, which is recorded as the first preset condition.

In some embodiments, for example, if the reinforcement learning model is trained based on the second reward condition in the aforementioned method 200, the preset condition may be the second reward condition.

For example, the preset condition is that the signal quality information of the target cell is greater than or equal to the signal quality threshold, the terminal device is located within the coverage of the target cell, and the signal quality information of the target cell is the largest among multiple candidate cells, which is recorded as the second preset conditions.

In some embodiments, for example, if the reinforcement learning model is trained based on the third reward condition in the foregoing method 200, the preset condition may be the third reward condition.

For example, the preset condition is that the signal quality information of the target cell is greater than or equal to the signal quality threshold, the load of the target cell meets the load threshold, and the signal quality information of the target cell is the largest among multiple candidate cells, which is recorded as the third preset condition.

In some embodiments, for example, if the reinforcement learning model is trained based on the fourth reward condition in the aforementioned method 200, the preset condition may be the fourth reward condition.

For example, the preset condition is that the signal quality information of the target cell is greater than or equal to the signal quality threshold, the load of the target cell meets the load threshold, the terminal device is located within the coverage of the target cell, and the signals of the target cell in multiple candidate cells The maximum quality information is recorded as the fourth preset condition.

In some embodiments of the present application, the state information of the terminal device in multiple cells includes the state information of the terminal device in the first cell. As an example but not limitation, the state of the terminal device in the first cell Information includes at least one of the following:

Signal quality information of the first cell, load information of the first cell, whether the terminal device is within the coverage of the first cell, location information of the terminal device, location information of the cell (or coverage information of the cell), and handover status.

Wherein, the handover state is used to indicate whether the first cell is the same as the cell selected at the last moment, for example, if they are the same, it indicates that handover has not occurred; otherwise, it indicates that handover has occurred.

In some embodiments of the present application, the S310 includes:

A reinforcement learning model is used to determine a handover target cell according to at least one reward condition and its corresponding reward value, as well as state information of the terminal equipment in multiple cells.

For example, determining a reward value for the terminal device to switch to each cell according to the state information of the terminal device in each of the multiple cells and the at least one reward condition;

A target cell among the plurality of cells is determined according to a reward value for the terminal device to switch to each cell.

As an example, the cell with the largest reward value is selected as the target cell.

The test method of the reinforcement learning model will be described below in combination with specific embodiments.

Step 1. Reset the state space of the terminal device.

Randomly select the starting point of the trajectory of the terminal device, and determine the cell to which the terminal device belongs, the RSRP size of the cell, and the corresponding handover status under the current location coordinates.

Step 2: The terminal device starts to move, and the terminal device uses the trained reinforcement learning model to determine which cell to switch to, assuming that the reinforcement learning model determines to switch to cell X.

Step 3: Determine whether the handover is successful according to whether the cell X satisfies a preset condition.

Step 4: outputting a simulation diagram of the result of the handover success rate.

In some embodiments, the reinforcement learning model can be tested separately according to different preset conditions.

For example, first test the reinforcement learning model according to the first preset condition, and then test the reinforcement learning model according to the second preset condition when the switching success rate meets the requirements, and if the switching success rate meets the requirements, Then test the reinforcement learning model according to the third preset condition, and then test the reinforcement learning model according to the fourth preset condition when the switching success rate meets the requirements.

In other embodiments of the present application, the method 300 may be a method for using a reinforcement learning model.

For example, a terminal device may report its status information in multiple cells to the network device, and the network device may use a reinforcement learning model to determine a target cell for the terminal device to switch to according to the status information of the terminal device in multiple cells. Further instruct the terminal device to switch to the target cell.

For another example, the terminal device may use a reinforcement learning model to determine a target cell for the terminal device to switch to according to status information of the terminal device in multiple cells. Further, the terminal device initiates handover to the target cell.

To sum up, in the embodiment of this application, by using the reinforcement learning model trained according to at least one reward condition and the corresponding reward value for cell selection, it is beneficial to select a suitable cell and avoid ping-pong switching in traditional cell selection. premature and other issues.

The method embodiment of the present application is described in detail above in conjunction with FIG. 4 to FIG. 9 , and the device embodiment of the present application is described in detail below in conjunction with FIG. 10 to FIG. 13 . It should be understood that the device embodiment and the method embodiment correspond to each other, similar to The description can refer to the method embodiment.

Fig. 10 shows a schematic block diagram of a device 400 for cell selection according to an embodiment of the present application. As shown in Figure 10, the device 400 includes:

A processing unit 410, configured to determine at least one reward condition for cell selection and a reward value corresponding to the at least one reward condition; and pair the at least one reward condition and the reward value corresponding to the at least one reward condition for The reinforcement learning model for cell selection is trained.

In some embodiments of the present application, the at least one reward condition includes a target reward condition that the target cell needs to meet, and the target reward condition includes at least one of the following:

The signal quality information of the cell is greater than or equal to the signal quality threshold;

The cell has the largest signal quality information among multiple candidate cells;

The terminal device is located within the coverage of the cell;

The load of the cell meets the load threshold;

The dwell time of the terminal device in the cell is longer than the time threshold.

In some embodiments of the present application, the load of the cell meets the load threshold, including:

The available load of the cell is greater than or equal to the first load threshold; and/or

The used load of the cell is less than or equal to the second load threshold.

In some embodiments of the present application, the target reward condition includes a first reward condition, wherein the first reward condition includes:

The signal quality information of the cell is greater than or equal to the signal quality threshold, and the signal quality information of the cell is the largest among multiple candidate cells.

In some embodiments of the present application, the target reward condition includes a second reward condition, wherein the second reward condition includes:

The signal quality information of the cell is greater than or equal to the signal quality threshold, the terminal device is located within the coverage of the cell, and the signal quality information of the cell is the largest among multiple candidate cells.

In some embodiments of the present application, the target reward condition includes a third reward condition, wherein the third reward condition includes:

The signal quality information of the cell is greater than or equal to the signal quality threshold, the load of the cell meets the load threshold, and the signal quality information of the cell is the largest among multiple candidate cells.

In some embodiments of the present application, the target reward condition includes a fourth reward condition, wherein the fourth reward condition includes:

The signal quality information of the cell is greater than or equal to the signal quality threshold, the load of the cell meets the load threshold, the terminal device is located within the coverage of the cell, and the signal quality information of the cell is the largest among multiple candidate cells.

In some embodiments of the present application, the determining the reward value corresponding to the at least one reward condition includes:

Giving a first reward value when the selected cell satisfies the target reward condition; or

When the selected cell does not meet the target reward condition, give a second reward value;

Wherein, the first reward value is greater than the second reward value.

In some embodiments of the present application, the processing unit 410 is further configured to:

If the selected cell satisfies the target reward condition and the selected cell is different from the cell selected at the previous moment, giving a third reward value; or

When the selected cell satisfies the target reward condition, but the selected cell is the same as the cell selected at the previous moment, giving a fourth reward value; or

In the case that the selected cell does not meet the target reward condition, and the handover event is too early handover, giving a fifth reward value; or

If the selected cell does not meet the target reward condition, and the handover event is a handover to a wrong cell, giving a sixth reward value; or

In the case that the selected cell satisfies the target reward condition, giving a reward value according to the residence time of the terminal device in the cell;

Wherein, the third reward value is greater than the fourth reward value, the third reward value is greater than the fifth reward value, and the third reward value is greater than the sixth reward value.

In some embodiments of the present application, the processing unit 410 is further configured to:

Obtaining the state space and behavior space of the terminal device, wherein the state space of the terminal device includes state information of the terminal device at multiple moments, and the behavior space of the terminal device includes behaviors of the terminal device at multiple moments information;

The reinforcement learning model is trained according to the state space and behavior space of the terminal device, and the at least one reward condition and the reward value corresponding to the at least one reward condition.

In some embodiments of the present application, the state information of the terminal device at multiple moments includes state information at a first moment, and the state information at the first moment includes at least one of the following:

Information about the cell to which the terminal device belongs at the first moment;

Signal quality information of the cell to which the terminal device belongs at the first moment;

The switching status information of the terminal device at the first moment is used to indicate whether the terminal device is switched at the first moment;

The location information of the terminal device at the first moment.

In some embodiments of the present application, the handover state information of the terminal device at the first moment is based on whether the information of the cell to which the terminal device belongs at the first moment and the cell to which the terminal device belongs at the second moment The same determination, wherein, the second moment is the previous moment of the first moment.

In some embodiments of the present application, the behavior information of the terminal device at multiple moments includes behavior information of the terminal device at a first moment, and the behavior information of the terminal device at the first moment is used to indicate the The terminal device selects the first cell at the first moment.

In some embodiments of the present application, the processing unit 410 is further configured to:

Determine the reward value corresponding to the behavior information at the first moment according to the status information at the first moment and the at least one reward condition;

The state information at the second moment, the behavior information at the first moment, the reward value corresponding to the behavior at the first moment and the state information at the first moment are stored in the experience pool, wherein the second moment is a moment immediately preceding said first moment;

The reinforcement learning model is trained using the experience pool.

In some embodiments of the present application, the reinforcement learning model includes a deep Q-network model.

Optionally, in some embodiments, the above-mentioned communication unit may be a communication interface or a transceiver, or an input-output interface of a communication chip or a system-on-chip. The aforementioned processing unit may be one or more processors.

It should be understood that the device 400 according to the embodiment of the present application may correspond to the terminal device or the network device in the method embodiment of the present application, and the above-mentioned and other operations and/or functions of each unit in the device 400 are to realize the The corresponding processes of the terminal device or the network device in the method 200 shown in 8 are not repeated here for the sake of brevity.

Fig. 11 shows a schematic block diagram of a device 500 for cell selection according to an embodiment of the present application. As shown in Figure 11, the device 500 includes:

The processing unit 510 is configured to use a reinforcement learning model to determine a selected target cell according to state information of the terminal device in multiple cells.

In some embodiments of the present application, the processing unit 510 is further configured to:

Whether the handover is successful is determined according to whether the target cell satisfies a preset condition.

In some embodiments of the present application, the preset conditions include at least one of the following:

The signal quality information of the target cell is greater than or equal to a signal quality threshold;

The target cell has the largest signal quality information among multiple candidate cells;

The terminal device is located within the coverage of the target cell;

The load of the target cell meets a load threshold;

The dwell time of the terminal device in the target cell is greater than a time threshold.

In some embodiments of the present application, the state information of the terminal device in multiple cells includes state information of the terminal device in the first cell, wherein the state information of the terminal device in the first cell includes the following At least one of:

Signal quality information of the first cell, load information of the first cell, and whether the terminal device is within the coverage of the first cell.

In some embodiments of the present application, the processing unit 510 is further configured to:

determining a reward value for the terminal device to switch to each cell according to the state information of the terminal device in each of the multiple cells and at least one reward condition;

A target cell among the plurality of cells is determined according to a reward value for the terminal device to switch to each cell.

Optionally, in some embodiments, the above-mentioned communication unit may be a communication interface or a transceiver, or an input-output interface of a communication chip or a system-on-chip. The aforementioned processing unit may be one or more processors.

It should be understood that the device 500 according to the embodiment of the present application may correspond to the terminal device or the network device in the method embodiment of the present application, and the above-mentioned and other operations and/or functions of each unit in the device 500 are respectively in order to realize the For the sake of brevity, the corresponding processes of the terminal device or the network device in method 300 are not repeated here.

Fig. 12 is a schematic structural diagram of a communication device 600 provided by an embodiment of the present application. The communication device 600 shown in FIG. 12 includes a processor 610, and the processor 610 can call and run a computer program from a memory, so as to implement the method in the embodiment of the present application.

Optionally, as shown in FIG. 12 , the communication device 600 may further include a memory 620 . Wherein, the processor 610 can invoke and run a computer program from the memory 620, so as to implement the method in the embodiment of the present application.

Wherein, the memory 620 may be an independent device independent of the processor 610 , or may be integrated in the processor 610 .

Optionally, as shown in FIG. 12, the communication device 600 may further include a transceiver 630, and the processor 610 may control the transceiver 630 to communicate with other devices, specifically, to send information or data to other devices, or receive other Information or data sent by the device.

Wherein, the transceiver 630 may include a transmitter and a receiver. The transceiver 630 may further include antennas, and the number of antennas may be one or more.

Optionally, the communication device 600 may specifically be the network device of the embodiment of the present application, and the communication device 600 may implement the corresponding processes implemented by the network device in each method of the embodiment of the present application. For the sake of brevity, details are not repeated here. .

Optionally, the communication device 600 may specifically be the mobile terminal/terminal device of the embodiment of the present application, and the communication device 600 may implement the corresponding processes implemented by the mobile terminal/terminal device in each method of the embodiment of the present application, for the sake of brevity , which will not be repeated here.

FIG. 13 is a schematic structural diagram of a chip according to an embodiment of the present application. The chip 700 shown in FIG. 13 includes a processor 710, and the processor 710 can call and run a computer program from a memory, so as to implement the method in the embodiment of the present application.

Optionally, as shown in FIG. 13 , the chip 700 may further include a memory 720 . Wherein, the processor 710 can invoke and run a computer program from the memory 720, so as to implement the method in the embodiment of the present application.

Wherein, the memory 720 may be an independent device independent of the processor 710 , or may be integrated in the processor 710 .

Optionally, the chip 700 may also include an input interface 730 . Wherein, the processor 710 can control the input interface 730 to communicate with other devices or chips, specifically, can obtain information or data sent by other devices or chips.

Optionally, the chip 700 may also include an output interface 740 . Wherein, the processor 710 can control the output interface 740 to communicate with other devices or chips, specifically, can output information or data to other devices or chips.

Optionally, the chip can be applied to the network device in the embodiment of the present application, and the chip can implement the corresponding processes implemented by the network device in the methods of the embodiment of the present application. For the sake of brevity, details are not repeated here.

Optionally, the chip can be applied to the mobile terminal/terminal device in the embodiments of the present application, and the chip can implement the corresponding processes implemented by the mobile terminal/terminal device in the various methods of the embodiments of the present application. For the sake of brevity, here No longer.

It should be understood that the chip mentioned in the embodiment of the present application may also be called a system-on-chip, a system-on-chip, a system-on-a-chip, or a system-on-a-chip.

It should be understood that the processor in the embodiment of the present application may be an integrated circuit chip, which has a signal processing capability. In the implementation process, each step of the above-mentioned method embodiments may be completed by an integrated logic circuit of hardware in a processor or instructions in the form of software. The above-mentioned processor can be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), an off-the-shelf programmable gate array (Field Programmable Gate Array, FPGA) or other available Program logic devices, discrete gate or transistor logic devices, discrete hardware components. Various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like. The steps of the method disclosed in connection with the embodiments of the present application may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory, and the processor reads the information in the memory, and completes the steps of the above method in combination with its hardware.

It can be understood that the memory in the embodiments of the present application may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the non-volatile memory can be read-only memory (Read-Only Memory, ROM), programmable read-only memory (Programmable ROM, PROM), erasable programmable read-only memory (Erasable PROM, EPROM), electronically programmable Erase Programmable Read-Only Memory (Electrically EPROM, EEPROM) or Flash. The volatile memory can be Random Access Memory (RAM), which acts as external cache memory. By way of illustration and not limitation, many forms of RAM are available, such as Static Random Access Memory (Static RAM, SRAM), Dynamic Random Access Memory (Dynamic RAM, DRAM), Synchronous Dynamic Random Access Memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (Synchlink DRAM, SLDRAM ) and Direct Memory Bus Random Access Memory (Direct Rambus RAM, DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but not be limited to, these and any other suitable types of memory.

It should be understood that the above-mentioned memory is illustrative but not restrictive. For example, the memory in the embodiment of the present application may also be a static random access memory (static RAM, SRAM), a dynamic random access memory (dynamic RAM, DRAM), Synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection Dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM), etc. That is, the memory in the embodiments of the present application is intended to include, but not be limited to, these and any other suitable types of memory.

The embodiment of the present application also provides a computer-readable storage medium for storing computer programs.

Optionally, the computer-readable storage medium can be applied to the network device in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the network device in the methods of the embodiments of the present application. For brevity, here No longer.

Optionally, the computer-readable storage medium can be applied to the mobile terminal/terminal device in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the mobile terminal/terminal device in the various methods of the embodiments of the present application , for the sake of brevity, it is not repeated here.

The embodiment of the present application also provides a computer program product, including computer program instructions.

Optionally, the computer program product may be applied to the network device in the embodiment of the present application, and the computer program instructions cause the computer to execute the corresponding process implemented by the network device in each method of the embodiment of the present application. For the sake of brevity, the Let me repeat.

Optionally, the computer program product can be applied to the mobile terminal/terminal device in the embodiments of the present application, and the computer program instructions cause the computer to execute the corresponding processes implemented by the mobile terminal/terminal device in the methods of the embodiments of the present application, For the sake of brevity, details are not repeated here.

The embodiment of the present application also provides a computer program.

Optionally, the computer program can be applied to the network device in the embodiment of the present application. When the computer program is run on the computer, the computer executes the corresponding process implemented by the network device in each method of the embodiment of the present application. For the sake of brevity , which will not be repeated here.

Optionally, the computer program can be applied to the mobile terminal/terminal device in the embodiment of the present application. When the computer program is run on the computer, the computer executes each method in the embodiment of the present application to be implemented by the mobile terminal/terminal device For the sake of brevity, the corresponding process will not be repeated here.

Those skilled in the art can appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions described above are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The above is only a specific implementation of the application, but the scope of protection of the application is not limited thereto. Anyone familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the application. Should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (50)

A method for cell selection, characterized by comprising: determining at least one reward condition for cell selection and a reward value corresponding to the at least one reward condition; A reinforcement learning model for cell selection is trained according to the at least one reward condition and the reward value corresponding to the at least one reward condition. The method according to claim 1, wherein the at least one reward condition includes a target reward condition that the target cell needs to meet, and the target reward condition includes at least one of the following: The signal quality information of the cell is greater than or equal to the signal quality threshold; The cell has the largest signal quality information among multiple candidate cells; The terminal equipment is located within the coverage area of the cell; The load of the cell meets the load threshold; The dwell time of the terminal device in the cell is greater than the time threshold. The method according to claim 2, wherein the load of the cell meets a load threshold, comprising: The available load of the cell is greater than or equal to the first load threshold; and/or The used load of the cell is less than or equal to the second load threshold. The method according to claim 2 or 3, wherein the target reward condition comprises a first reward condition, wherein the first reward condition comprises: The signal quality information of the cell is greater than or equal to the signal quality threshold, and the signal quality information of the cell is the largest among multiple candidate cells. The method according to any one of claims 2-4, wherein the target reward condition includes a second reward condition, wherein the second reward condition includes: The signal quality information of the cell is greater than or equal to the signal quality threshold, the terminal device is located within the coverage of the cell, and the signal quality information of the cell is the largest among multiple candidate cells. The method according to any one of claims 2-5, wherein the target reward condition includes a third reward condition, wherein the third reward condition includes: The signal quality information of the cell is greater than or equal to the signal quality threshold, the load of the cell meets the load threshold, and the signal quality information of the cell is the largest among multiple candidate cells. The method according to any one of claims 2-6, wherein the target reward condition includes a fourth reward condition, wherein the fourth reward condition includes: The signal quality information of the cell is greater than or equal to the signal quality threshold, the load of the cell meets the load threshold, the terminal device is located within the coverage of the cell, and the signal quality information of the cell is the largest among multiple candidate cells. The method according to any one of claims 2-7, wherein the determining the reward value corresponding to the at least one reward condition comprises: Giving a first reward value when the selected cell satisfies the target reward condition; or When the selected cell does not meet the target reward condition, give a second reward value; Wherein, the first reward value is greater than the second reward value. The method according to any one of claims 2-8, wherein the determining the reward value corresponding to the at least one reward condition comprises: If the selected cell satisfies the target reward condition and the selected cell is different from the cell selected at the previous moment, giving a third reward value; or When the selected cell satisfies the target reward condition, but the selected cell is the same as the cell selected at the previous moment, giving a fourth reward value; or In the case that the selected cell does not meet the target reward condition, and the handover event is too early handover, giving a fifth reward value; or If the selected cell does not meet the target reward condition, and the handover event is a handover to a wrong cell, giving a sixth reward value; or In the case that the selected cell satisfies the target reward condition, giving a reward value according to the residence time of the terminal device in the cell; Wherein, the third reward value is greater than the fourth reward value, the third reward value is greater than the fifth reward value, and the third reward value is greater than the sixth reward value. The method according to any one of claims 1-9, wherein the reinforcement learning model for cell selection is trained according to the at least one reward condition and the reward value corresponding to the at least one reward condition ,include: Obtaining the state space and behavior space of the terminal device, wherein the state space of the terminal device includes state information of the terminal device at multiple moments, and the behavior space of the terminal device includes behaviors of the terminal device at multiple moments information; The reinforcement learning model is trained according to the state space and behavior space of the terminal device, and the at least one reward condition and the reward value corresponding to the at least one reward condition. The method according to claim 10, wherein the state information of the terminal device at multiple moments includes state information at a first moment, and the state information at the first moment includes at least one of the following: Information about the cell to which the terminal device belongs at the first moment; Signal quality information of the cell to which the terminal device belongs at the first moment; The switching status information of the terminal device at the first moment is used to indicate whether the terminal device is switched at the first moment; The location information of the terminal device at the first moment. The method according to claim 11, wherein the handover state information of the terminal equipment at the first moment is based on the information of the cell to which the terminal equipment belongs at the first moment and the cell information of the terminal equipment at the second It is determined whether the cell to which the time belongs is the same, wherein the second time is a previous time of the first time. The method according to any one of claims 10-12, wherein the behavior information of the terminal device at multiple moments includes the behavior information of the terminal device at a first moment, and the terminal device at the The behavior information at the first moment is used to indicate that the terminal device has selected the first cell at the first moment. The method according to any one of claims 10-13, characterized in that, according to the state space and behavior space of the terminal device, and the at least one reward condition and the reward corresponding to the at least one reward condition value, the reinforcement learning model is trained, including: Determine the reward value corresponding to the behavior information at the first moment according to the status information at the first moment and the at least one reward condition; The state information at the second moment, the behavior information at the first moment, the reward value corresponding to the behavior at the first moment and the state information at the first moment are stored in the experience pool, wherein the second moment is a moment immediately preceding said first moment; The reinforcement learning model is trained using the experience pool. The method according to any one of claims 1-14, wherein the reinforcement learning model comprises a deep Q-network model. A method for cell selection, characterized by comprising: A reinforcement learning model is used to determine the selected target cell according to the status information of the terminal equipment in multiple cells. The method according to claim 16, further comprising: Whether the handover is successful is determined according to whether the target cell satisfies a preset condition. The method according to claim 17, wherein the preset conditions include at least one of the following: The signal quality information of the target cell is greater than or equal to a signal quality threshold; The target cell has the largest signal quality information among multiple candidate cells; The terminal device is located within the coverage of the target cell; The load of the target cell meets a load threshold; The dwell time of the terminal device in the target cell is greater than a time threshold. The method according to any one of claims 16-18, wherein the status information of the terminal device in multiple cells includes the status information of the terminal device in the first cell, wherein the terminal device is in The status information of the first cell includes at least one of the following: Signal quality information of the first cell, load information of the first cell, and whether the terminal device is within the coverage of the first cell. The method according to any one of claims 16-19, characterized in that, using a reinforcement learning model to determine the selected target cell according to the state information of the terminal device in multiple cells includes: determining a reward value for the terminal device to switch to each cell according to the state information of the terminal device in each of the multiple cells and at least one reward condition; A target cell among the plurality of cells is determined according to a reward value for the terminal device to switch to each cell. A device for cell selection, characterized by comprising: A processing unit, configured to determine at least one reward condition for cell selection and a reward value corresponding to the at least one reward condition; and A reinforcement learning model for cell selection is trained according to the at least one reward condition and the reward value corresponding to the at least one reward condition. The device according to claim 21, wherein the at least one reward condition includes a target reward condition that the target cell needs to satisfy, and the target reward condition includes at least one of the following: The signal quality information of the cell is greater than or equal to the signal quality threshold; The cell has the largest signal quality information among multiple candidate cells; The terminal equipment is located within the coverage area of the cell; The load of the cell meets the load threshold; The dwell time of the terminal device in the cell is greater than the time threshold. The device according to claim 22, wherein the load of the cell meets a load threshold, comprising: The available load of the cell is greater than or equal to the first load threshold; and/or The used load of the cell is less than or equal to the second load threshold. The device according to claim 22 or 23, wherein the target reward condition comprises a first reward condition, wherein the first reward condition comprises: The signal quality information of the cell is greater than or equal to the signal quality threshold, and the signal quality information of the cell is the largest among multiple candidate cells. The device according to any one of claims 22-24, wherein the target reward condition comprises a second reward condition, wherein the second reward condition comprises: The signal quality information of the cell is greater than or equal to the signal quality threshold, the terminal device is located within the coverage of the cell, and the signal quality information of the cell is the largest among multiple candidate cells. The device according to any one of claims 22-25, wherein the target reward condition comprises a third reward condition, wherein the third reward condition comprises: The signal quality information of the cell is greater than or equal to the signal quality threshold, the load of the cell meets the load threshold, and the signal quality information of the cell is the largest among multiple candidate cells. The device according to any one of claims 22-26, wherein the target reward condition comprises a fourth reward condition, wherein the fourth reward condition comprises: The signal quality information of the cell is greater than or equal to the signal quality threshold, the load of the cell meets the load threshold, the terminal device is located within the coverage of the cell, and the signal quality information of the cell is the largest among multiple candidate cells. The device according to any one of claims 22-27, wherein the determining the reward value corresponding to the at least one reward condition comprises: Giving a first reward value when the selected cell satisfies the target reward condition; or When the selected cell does not meet the target reward condition, give a second reward value; Wherein, the first reward value is greater than the second reward value. The device according to any one of claims 22-28, wherein the processing unit is further configured to: If the selected cell satisfies the target reward condition and the selected cell is different from the cell selected at the previous moment, giving a third reward value; or When the selected cell satisfies the target reward condition, but the selected cell is the same as the cell selected at the previous moment, giving a fourth reward value; or In the case that the selected cell does not meet the target reward condition, and the handover event is too early handover, giving a fifth reward value; or If the selected cell does not meet the target reward condition, and the handover event is a handover to a wrong cell, giving a sixth reward value; or In the case that the selected cell satisfies the target reward condition, giving a reward value according to the residence time of the terminal device in the cell; Wherein, the third reward value is greater than the fourth reward value, the third reward value is greater than the fifth reward value, and the third reward value is greater than the sixth reward value. The device according to any one of claims 21-29, wherein the processing unit is further configured to: Obtaining the state space and behavior space of the terminal device, wherein the state space of the terminal device includes state information of the terminal device at multiple moments, and the behavior space of the terminal device includes behaviors of the terminal device at multiple moments information; The reinforcement learning model is trained according to the state space and behavior space of the terminal device, and the at least one reward condition and the reward value corresponding to the at least one reward condition. The device according to claim 30, wherein the state information of the terminal device at multiple moments includes state information at a first moment, and the state information at the first moment includes at least one of the following: Information about the cell to which the terminal device belongs at the first moment; Signal quality information of the cell to which the terminal device belongs at the first moment; The switching status information of the terminal device at the first moment is used to indicate whether the terminal device is switched at the first moment; The location information of the terminal device at the first moment. The device according to claim 31, wherein the handover state information of the terminal device at the first moment is based on the information of the cell to which the terminal device belongs at the first moment and the cell information of the terminal device at the second It is determined whether the cell to which the time belongs is the same, wherein the second time is a previous time of the first time. The device according to any one of claims 30-32, wherein the behavior information of the terminal device at multiple moments includes the behavior information of the terminal device at a first moment, and the terminal device at the The behavior information at the first moment is used to indicate that the terminal device has selected the first cell at the first moment. The device according to any one of claims 30-33, wherein the processing unit is further configured to: Determine the reward value corresponding to the behavior information at the first moment according to the status information at the first moment and the at least one reward condition; The state information at the second moment, the behavior information at the first moment, the reward value corresponding to the behavior at the first moment and the state information at the first moment are stored in the experience pool, wherein the second moment is a moment immediately preceding said first moment; The reinforcement learning model is trained using the experience pool. The device according to any one of claims 21-34, wherein the reinforcement learning model comprises a deep Q-network model. A device for cell selection, characterized by comprising: The processing unit is configured to use a reinforcement learning model to determine a selected target cell according to state information of the terminal device in multiple cells. The device according to claim 36, wherein the processing unit is further configured to: Whether the handover is successful is determined according to whether the target cell satisfies a preset condition. The device according to claim 37, wherein the preset conditions include at least one of the following: The signal quality information of the target cell is greater than or equal to a signal quality threshold; The target cell has the largest signal quality information among multiple candidate cells; The terminal device is located within the coverage of the target cell; The load of the target cell meets a load threshold; The dwell time of the terminal device in the target cell is greater than a time threshold. The device according to any one of claims 36-38, wherein the status information of the terminal device in multiple cells includes the status information of the terminal device in the first cell, wherein the terminal device is in The status information of the first cell includes at least one of the following: Signal quality information of the first cell, load information of the first cell, and whether the terminal device is within the coverage of the first cell. The device according to any one of claims 36-39, wherein the processing unit is further configured to: determining a reward value for the terminal device to switch to each cell according to the state information of the terminal device in each of the multiple cells and at least one reward condition; A target cell among the plurality of cells is determined according to a reward value for the terminal device to switch to each cell. A communication device, characterized by comprising: a processor and a memory, the memory is used to store a computer program, the processor is used to invoke and run the computer program stored in the memory, and execute any of the following claims 1 to 15 one of the methods described. A chip, characterized by comprising: a processor, configured to call and run a computer program from a memory, so that a device installed with the chip executes the method according to any one of claims 1 to 15. A computer-readable storage medium, characterized by being used for storing a computer program, the computer program causes a computer to execute the method according to any one of claims 1 to 15. A computer program product, characterized by comprising computer program instructions, the computer program instructions cause a computer to execute the method according to any one of claims 1 to 15. A computer program, characterized in that the computer program causes a computer to execute the method according to any one of claims 1 to 15. A communication device, characterized by comprising: a processor and a memory, the memory is used to store a computer program, the processor is used to invoke and run the computer program stored in the memory, and execute any of the following claims 16-20 one of the methods described. A chip, characterized by comprising: a processor, configured to invoke and run a computer program from a memory, so that a device installed with the chip executes the method according to any one of claims 16 to 20. A computer-readable storage medium, characterized in that it is used to store a computer program, the computer program causes a computer to perform the method according to any one of claims 16 to 20. A computer program product, characterized by comprising computer program instructions, the computer program instructions causing a computer to execute the method as claimed in any one of claims 16 to 20. A computer program, characterized in that the computer program causes a computer to execute the method according to any one of claims 16-20.

Images