WO2025179566A1

WO2025179566A1 - Devices and methods for model monitoring for ai/ml based mobility

Info

Publication number: WO2025179566A1
Application number: PCT/CN2024/079473
Authority: WO
Inventors: Boyuan ZHANG; Gang Wang
Original assignee: NEC Corp
Current assignee: NEC Corp
Priority date: 2024-02-29
Filing date: 2024-02-29
Publication date: 2025-09-04
Anticipated expiration: 2026-08-29

Abstract

Embodiments of the present disclosure provide a solution for model monitoring for artificial intelligence/machine learning (AI/ML) based mobility. In a solution, a communication device obtains respective accuracy indications of a set of prediction results for a measurement event, wherein the set of prediction results are generated using an AI/ML model, a predication result is corresponding to a measurement result triggering reporting of the measurement event, and an accuracy indication of the prediction result is associated with the corresponding measurement result. The communication device determines a performance monitoring result of the AI/ML model based on the respective accuracy indications.

Description

DEVICES AND METHODS FOR MODEL MONITORING FOR AI/ML BASED MOBILITY

FIELDS

Example embodiments of the present disclosure generally relate to the field of communication techniques and in particular, to devices and methods for model monitoring for artificial intelligence/machine learning (AI/ML) based mobility.

BACKGROUND

With the development of AI and ML techniques, various measurements and events in the network may be predicted. By analyzing and forecasting resource management, measurement results, handover (HO) failures, and measurement events, mobility performance and optimize network efficiency may be enhanced.

SUMMARY

In a first aspect, there is provided a communication device comprising: a processor configured to cause the communication device to: obtain respective accuracy indications of a set of prediction results for a measurement event, wherein the set of prediction results are generated using an Artificial Intelligence/Machine learning (AI/ML) model, a predication result is corresponding to a measurement result triggering reporting of the measurement event, and an accuracy indication of the prediction result is associated with the corresponding measurement result; and determine a performance monitoring result of the AI/ML model based on the respective accuracy indications.

In a second aspect, there is provided a communication method performed by a communication device. The method comprises: obtaining respective accuracy indications of a set of prediction results for a measurement event, wherein the set of prediction results are generated using an Artificial Intelligence/Machine learning (AI/ML) model, a predication result is corresponding to a measurement result triggering reporting of the measurement event, and an accuracy indication of the prediction result is associated with the corresponding measurement result; and determining a performance monitoring result of the AI/ML model based on the respective accuracy indications.

In a third aspect, there is provided a computer readable medium having instructions stored thereon, the instructions, when executed on at least one processor, causing the at least one processor to carry out the method according to the second aspect.

Other features of the present disclosure will become easily comprehensible through the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

Through the more detailed description of some example embodiments of the present disclosure in the accompanying drawings, the above and other objects, features and advantages of the present disclosure will become more apparent, wherein:

FIG. 1 illustrates an example communication environment in which example embodiments of the present disclosure can be implemented;

FIG. 2 illustrates a signaling flow of model monitoring in accordance with some embodiments of the present disclosure;

FIG. 3 illustrates a schematic diagram of monitoring based on inference accuracy in accordance with some embodiments of the present disclosure;

FIG. 4 illustrates an example flowchart of a model monitoring procedure in accordance with some embodiments of the present disclosure;

FIG. 5 illustrates a schematic diagram of accuracy indication determination in accordance with some embodiments of the present disclosure;

FIG. 6 illustrates a flowchart of a method implemented at a terminal device according to some example embodiments of the present disclosure;

FIG. 7 illustrates a simplified block diagram of an apparatus that is suitable for implementing example embodiments of the present disclosure.

Throughout the drawings, the same or similar reference numerals represent the same or similar element.

DETAILED DESCRIPTION

Principle of the present disclosure will now be described with reference to some example embodiments. It is to be understood that these embodiments are described only for the purpose of illustration and help those skilled in the art to understand and implement the present disclosure, without suggesting any limitation as to the scope of the disclosure. Embodiments described herein can be implemented in various manners other than the ones described below.

In the following description and claims, unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skills in the art to which this disclosure belongs.

As used herein, the term ‘terminal device’ refers to any device having wireless or wired communication capabilities. Examples of the terminal device include, but not limited to, user equipment (UE) , personal computers, desktops, mobile phones, cellular phones, smart phones, personal digital assistants (PDAs) , portable computers, tablets, wearable devices, internet of things (IoT) devices, Ultra-reliable and Low Latency Communications (URLLC) devices, Internet of Everything (IoE) devices, machine type communication (MTC) devices, devices on vehicle for V2X communication where X means pedestrian, vehicle, or infrastructure/network, devices for Integrated Access and Backhaul (IAB) , Space borne vehicles or Air borne vehicles in Non-terrestrial networks (NTN) including Satellites and High Altitude Platforms (HAPs) encompassing Unmanned Aircraft Systems (UAS) , eXtended Reality (XR) devices including different types of realities such as Augmented Reality (AR) , Mixed Reality (MR) and Virtual Reality (VR) , the unmanned aerial vehicle (UAV) commonly known as a drone which is an aircraft without any human pilot, devices on high speed train (HST) , or image capture devices such as digital cameras, sensors, gaming devices, music storage and playback appliances, or Internet appliances enabling wireless or wired Internet access and browsing and the like. The ‘terminal device’ can further has ‘multicast/broadcast’ feature, to support public safety and mission critical, V2X applications, transparent IPv4/IPv6 multicast delivery, IPTV, smart TV, radio services, software delivery over wireless, group communications and IoT applications. It may also incorporate one or multiple Subscriber Identity Module (SIM) as known as Multi-SIM. The term “terminal device” can be used interchangeably with a UE, a mobile station, a subscriber station, a mobile terminal, a user terminal or a wireless device.

The term “network device” refers to a device which is capable of providing or hosting a cell or coverage where terminal devices can communicate. Examples of a network device include, but not limited to, a Node B (NodeB or NB) , an evolved NodeB (eNodeB or eNB) , a next generation NodeB (gNB) , a transmission reception point (TRP) , a remote radio unit (RRU) , a radio head (RH) , a remote radio head (RRH) , an IAB node, a low power node such as a femto node, a pico node, a reconfigurable intelligent surface (RIS) , and the like.

The terminal device or the network device may have Artificial intelligence (AI) or Machine learning capability. It generally includes a model which has been trained from numerous collected data for a specific function, and can be used to predict some information.

The terminal or the network device may work on several frequency ranges, e.g., FR1 (e.g., 450 MHz to 6000 MHz) , FR2 (e.g., 24.25GHz to 52.6GHz) , frequency band larger than 100 GHz as well as Tera Hertz (THz) . It can further work on licensed/unlicensed/shared spectrum. The terminal device may have more than one connection with the network devices under Multi-Radio Dual Connectivity (MR-DC) application scenario. The terminal device or the network device can work on full duplex, flexible duplex and cross division duplex modes.

The embodiments of the present disclosure may be performed in test equipment, e.g., signal generator, signal analyzer, spectrum analyzer, network analyzer, test terminal device, test network device, channel emulator. In some embodiments, the terminal device may be connected with a first network device and a second network device. One of the first network device and the second network device may be a master node and the other one may be a secondary node. The first network device and the second network device may use different radio access technologies (RATs) . In some embodiments, the first network device may be a first RAT device and the second network device may be a second RAT device. In some embodiments, the first RAT device is eNB and the second RAT device is gNB. Information related with different RATs may be transmitted to the terminal device from at least one of the first network device or the second network device. In some embodiments, first information may be transmitted to the terminal device from the first network device and second information may be transmitted to the terminal device from the second network device directly or via the first network device. In some embodiments, information related with configuration for the terminal device configured by the second network device may be transmitted from the second network device via the first network device. Information related with reconfiguration for the terminal device configured by the second network device may be transmitted to the terminal device from the second network device directly or via the first network device.

As used herein, the singular forms ‘a’ , ‘an’ and ‘the’ are intended to include the plural forms as well, unless the context clearly indicates otherwise. The term ‘includes’ and its variants are to be read as open terms that mean ‘includes, but is not limited to. ’ The term ‘based on’ is to be read as ‘at least in part based on. ’ The term ‘one embodiment’ and ‘an embodiment’ are to be read as ‘at least one embodiment. ’ The term ‘another embodiment’ is to be read as ‘at least one other embodiment. ’ The terms ‘first, ’ ‘second, ’ and the like may refer to different or same objects. Other definitions, explicit and implicit, may be included below.

In some examples, values, procedures, or apparatus are referred to as ‘best, ’ ‘lowest, ’ ‘highest, ’ ‘minimum, ’ ‘maximum, ’ or the like. It will be appreciated that such descriptions are intended to indicate that a selection among many used functional alternatives can be made, and such selections need not be better, smaller, higher, or otherwise preferable to other selections.

As used herein, the term “resource, ” “transmission resource, ” “uplink resource, ” or “downlink resource” may refer to any resource for performing a communication, such as a resource in time domain, a resource in frequency domain, a resource in space domain, a resource in code domain, or any other resource enabling a communication, and the like. In the following, unless explicitly stated, a resource in both frequency domain and time domain will be used as an example of a transmission resource for describing some example embodiments of the present disclosure. It is noted that example embodiments of the present disclosure are equally applicable to other resources in other domains.

As used herein, a model may be equivalent to at least one of the following: an AI/ML model, a ML model, an AI model, a data-driven, a data processing model, an algorithm, a functionality, a procedure, a process, an entity, a function, a feature, a feature group, a model identifier (ID) , an ID, a functionality ID, a configuration ID, a scenario ID, a site ID, or a dataset ID. As a result, the above terms may be used interchangeably.

In some embodiments, the model may be represented by or associated with a channel, a resource, a resource set, a reference signal (RS) resource, a RS resource set, a RS port, a set of RS ports, a RS port ID, or a set of RS port IDs.

In some embodiments, the model may comprise a set of weights values that may be learned during training, for example for a specific architecture or configuration, where a set of weights values may also be called a parameter set.

In some embodiments, the model may be used to predict a target cell, or measurements of a set of beams of a set of candidate cells in future based on at least historical measurements (e.g., L1-RSRP, L1-SINR) of a set of beams of a set of candidate cells.

In some embodiments, an input of the ML model (i.e., AI input) may refer to the input of a model and indicate data inputted into the model, which may be equivalent to data.

In some embodiments, an output of ML model (i.e., AI output) may refers to the output of a model and indicate result (s) outputted by the model, which is equivalent to label/data.

As used herein, the term “AI/ML model” may refer to a data driven algorithm that applies AI/ML techniques to generate a set of outputs based on a set of inputs. In the context of the present disclosure, the term “AI/ML model” may be interchangeably with the terms “model” , “AI model” and “ML model” .

The term “UE-side (AI/ML) model” used herein may refer to an AI/ML Model of which inference is performed entirely at the UE. The term “network-side (AI/ML) model” used herein may refer to an AI/ML Model of which inference is performed entirely at the network. The term “one-sided (AI/ML) model” used herein may refer to a UE-side (AI/ML) model or a network-side (AI/ML) model. The term “two-sided (AI/ML) model” used herein may refer to a paired AI/ML Model (s) over which joint inference is performed, where joint inference comprises AI/ML Inference whose inference is performed jointly across the UE and the network, i. e, the first part of inference is firstly performed by UE and then the remaining part is performed by gNB, or vice versa.

The term “AI/ML model transfer” used herein may refer to a delivery of an AI/ML model over the air interface, either parameters of a model structure known at the receiving end or a new model with parameters. Delivery may contain a full model or a partial model. The term “model download” used herein may refer to model transfer from the network to UE. The term “model upload” used herein may refer to model transfer from UE to the network. The term “federated learning /federated training” used herein may refer to a machine learning technique that trains an AI/ML model across multiple decentralized edge nodes (e.g., UEs, gNBs) each performing local model training using local data samples. The technique requires multiple interactions of the model, but no exchange of local data samples.

The term “model activation” used herein may refer to enabling an AI/ML model for a specific function. The term “model deactivation” used herein may refer to disabling an AI/ML model for a specific function. The term “model switching” used herein may refer to deactivating a currently active AI/ML model and activating a different AI/ML model for a specific function.

As used herein, the term “AI/ML model delivery” may refer to a generic term referring to delivery of an AI/ML model from one entity to another entity in any manner. It is to be noted that an entity could mean a network node/function (e.g., gNB, LMF, etc. ) , UE, proprietary server, etc. The term “model registration” used herein may refer to a process of informing the existence of an AI/ML model to the network or to the UE with an identification, along with model description information of the AI/ML model for the network to enable life cycle management (LCM) .

The term “model update” as used herein may refer to a process of updating the model parameters and/or model structure of a model. The term “model parameter update” as used herein may refer to a process of updating the model parameters of a model.

Principles and implementations of the present disclosure will be described in detail below with reference to the figures.

FIG. 1 illustrates a schematic diagram of an example communication environment 100 in which example embodiments of the present disclosure can be implemented. In the communication environment 100, a plurality of communication devices, including a terminal device 110 and a network device 120, can communicate with each other.

In the example of FIG. 1, the terminal device 110 may be a UE and the network device 120 may be a base station serving the UE. The serving area of the network device 120 may be called a cell 102.

It is to be understood that the number of devices and their connections shown in FIG. 1 are only for the purpose of illustration without suggesting any limitation. The communication environment 100 may include any suitable number of devices configured to implementing example embodiments of the present disclosure. Although not shown, it would be appreciated that one or more additional devices may be located in the cell 102, and one or more additional cells may be deployed in the communication environment 100. It is noted that although illustrated as a network device, the network device 120 may be another device than a network device. Although illustrated as a terminal device, the terminal device 110 may be other device than a terminal device.

In the following, for the purpose of illustration, some example embodiments are described with the terminal device 110 operating as a UE and the network device 120 operating as a base station. However, in some example embodiments, operations described in connection with a terminal device may be implemented at a network device or other device, and operations described in connection with a network device may be implemented at a terminal device or other device.

In some example embodiments, a link from the network device 120 to the terminal device 110 is referred to as a downlink (DL) , while a link from the terminal device 110 to the network device 120 is referred to as an uplink (UL) . In DL, the network device 120 is a transmitting (TX) device (or a transmitter) and the terminal device 110 is a receiving (RX) device (or a receiver) . In UL, the terminal device 110 is a TX device (or a transmitter) and the network device 120 is a RX device (or a receiver) .

The communications in the communication environment 100 may conform to any suitable standards including, but not limited to, Global System for Mobile Communications (GSM) , Long Term Evolution (LTE) , LTE-Evolution, LTE-Advanced (LTE-A) , New Radio (NR) , Wideband Code Division Multiple Access (WCDMA) , Code Division Multiple Access (CDMA) , GSM EDGE Radio Access Network (GERAN) , Machine Type Communication (MTC) and the like. The embodiments of the present disclosure may be performed according to any generation communication protocols either currently known or to be developed in the future. Examples of the communication protocols include, but not limited to, the first generation (1G) , the second generation (2G) , 2.5G, 2.75G, the third generation (3G) , the fourth generation (4G) , 4.5G, the fifth generation (5G) communication protocols, 5.5G, 5G-Advanced networks, or the sixth generation (6G) networks.

In the communication environment 100, an AI/ML model can be applied to different scenarios to achieve better performances.

In some embodiments, the AI/ML model may be a two-sided model, which comprises a first part for using by the terminal device 110 and a second part for using the network device 120. The first part may be used to generate an intermediate result from an initial result and the second part may be used to generate a reconstructed result from the intermediate result. In the following, the first part may be referred to as a terminal part, UE side part, or UE part, which can be used interchangeably. The second part may be referred to as a network part, network (NW) side part, or NW part, which can be used interchangeably.

In some embodiments, the terminal device 110 can perform AI/ML based RRM measurement and event prediction. For example, cell-level measurement prediction including intra and inter-frequency may be performed using two-sided AI model (i.e., UE side and NW side model) . Inter-cell Beam-level measurement prediction for L3 Mobility may be performed using two-sided AI model. For another example, HO failure/radio link failure (RLF) prediction and measurement events prediction may be performed using UE side model.

A measurement event refers to an event that occurs in a wireless communication network and is related to measurements. These events are typically associated with aspects such as wireless signal quality, channel conditions, and network load. There are several common measurement events, such as Event A1, Event A2, …, Event A6. The following will describe in detail.

Event A1, as shown in Table 1, is used to determine whether the signal quality of the current serving cell exceeds a predefined threshold. The entering and leaving conditions may be determined by comparing the measurement result with the threshold value plus or minus the hysteresis value.

Table 1

The variables in the formula of Table 1 are defined as follows. Ms is the measurement result of the serving cell, not taking into account any offsets. Hys is the hysteresis parameter for this event (i.e., hysteresis as defined within reportConfigNR for this event) . Thresh is the threshold parameter for this event (i.e., a1-Threshold as defined within reportConfigNR for this event) . Ms is expressed in dBm in case of RSRP, or in dB in case of RSRQ and RS-SINR. Hys is expressed in dB. Thresh is expressed in the same unit as Ms.

Event A2, as shown in Table 2, is used to determine whether the signal quality of the current serving cell falls below a pre-set threshold. The entering and leaving conditions may be determined by comparing the measurement result with the threshold value plus or minus the hysteresis value.

Table 2

The variables in the formula of Table 2 are defined as follows. Ms is the measurement result of the serving cell, not taking into account any offsets. Hys is the hysteresis parameter for this event (i.e. hysteresis as defined within reportConfigNR for this event) . Thresh is the threshold parameter for this event (i.e., a2-Threshold as defined within reportConfigNR for this event) . Ms is expressed in dBm in case of RSRP, or in dB in case of RSRQ and RS-SINR. Hys is expressed in dB. Thresh is expressed in the same unit as Ms.

Event A3, as shown in Table 3, is used to determine the offset quality of the neighboring cell signals compared to the serving cell. The entering and leaving conditions may be determined by comparing the combination of the neighboring cell signal measurement results with the neighboring cell frequency offset, neighboring cell clock offset, and hysteresis value, with the combination of the serving cell signal measurement results, serving cell frequency offset, serving cell clock offset, and offset value. This comparison helps to determine the entering and leaving conditions.

Table 3

The variables in the formula of Table 3 are defined as follows. Mn is the measurement result of the neighbouring cell, not taking into account any offsets. Ofn is the measurement object specific offset of the reference signal of the neighbour cell (i.e., offsetMO as defined within measObjectNR corresponding to the neighbour cell) . Ocn is the cell specific offset of the neighbour cell (i.e., cellIndividualOffset as defined within measObjectNR corresponding to the frequency of the neighbour cell) and set to zero if not configured for the neighbour cell. Mp is the measurement result of the SpCell, not taking into account any offsets. Ofp is the measurement object specific offset of the SpCell (i.e., offsetMO as defined within measObjectNR corresponding to the SpCell) . Ocp is the cell specific offset of the SpCell (i.e., cellIndividualOffset as defined within measObjectNR corresponding to the SpCell) and is set to zero if not configured for the SpCell. Hys is the hysteresis parameter for this event (i.e., hysteresis as defined within reportConfigNR for this event) . Off is the offset parameter for this event (i.e., a3-Offset as defined within reportConfigNR for this event) . Mn, Mp are expressed in dBm in case of RSRP, or in dB in case of RSRQ and RS-SINR. Ofn, Ocn, Ofp, Ocp, Hys, Off are expressed in dB.

It is noted that the definition of Event A3 also applies to CondEvent A3.

Event A4, as shown in Table 4, is used to determine the superiority or inferiority of the neighboring cell signals relative to a threshold. The entering and leaving conditions may be determined by comparing the combination of the neighboring cell signal measurement results with the neighboring cell frequency offset, neighboring cell clock offset, and hysteresis value with the threshold. This comparison helps to determine the entering and leaving conditions.

Table 4

The variables in the formula of Table 4 are defined as follows. Mn is the measurement result of the neighbouring cell, not taking into account any offsets. Ofn is the measurement object specific offset of the neighbour cell (i.e., offsetMO as defined within measObjectNR corresponding to the neighbour cell) . Ocn is the measurement object specific offset of the neighbour cell (i.e., cellIndividualOffset as defined within measObjectNR corresponding to the neighbour cell) and set to zero if not configured for the neighbour cell. Hys is the hysteresis parameter for this event (i.e., hysteresis as defined within reportConfigNR for this event) . Thresh is the threshold parameter for this event (i.e., a4-Threshold as defined within reportConfigNR for this event) . Mn is expressed in dBm in case of RSRP, or in dB in case of RSRQ and RS-SINR. Ofn, Ocn, Hys are expressed in dB. Thresh is expressed in the same unit as Mn. It is noted that the definition of Event A4 also applies to CondEvent A4.

Event A5, as shown in Table 5, is used to determine the quality of the primary serving cell signal relative to threshold 1, as well as the quality of the neighbor cell signal relative to threshold 2.

Table 5

The variables in the formula of Table 5 are defined as follows. Mp is the measurement result of the NR SpCell, not taking into account any offsets. Mn is the measurement result of the neighbouring cell, not taking into account any offsets. Ofn is the measurement object specific offset of the neighbour cell (i.e., offsetMO as defined within measObjectNR corresponding to the neighbour cell) . Ocn is the cell specific offset of the neighbour cell (i.e., cellIndividualOffset as defined within measObjectNR corresponding to the neighbour cell) and set to zero if not configured for the neighbour cell. Hys is the hysteresis parameter for this event (i.e., hysteresis as defined within reportConfigNR for this event) . Thresh1 is the threshold parameter for this event (i.e., a5-Threshold1 as defined within reportConfigNR for this event) . Thresh2 is the threshold parameter for this event (i.e., a5-Threshold2 as defined within reportConfigNR for this event) . Mn, Mp are expressed in dBm in case of RSRP, or in dB in case of RSRQ and RS-SINR. Ofn, Ocn, Hys are expressed in dB. Thresh1 is expressed in the same unit as Mp. Thresh2 is expressed in the same unit as Mn. It is noted that the definition of Event A5 also applies to CondEvent A5.

Event A6, as shown in Table 6, is used to determine when the neighbor cell's signal becomes offset better than the serving cell's signal.

Table 6

The variables in the formula of Table 6 are defined as follows. Mn is the measurement result of the neighbouring cell, not taking into account any offsets. Ocn is the cell specific offset of the neighbour cell (i.e., cellIndividualOffset as defined within the associated measObjectNR) , and set to zero if not configured for the neighbour cell. Ms is the measurement result of the serving cell, not taking into account any offsets. Ocs is the cell specific offset of the serving cell (i.e., cellIndividualOffset as defined within the associated measObjectNR) , and is set to zero if not configured for the serving cell. Hys is the hysteresis parameter for this event (i.e., hysteresis as defined within reportConfigNR for this event) . Off is the offset parameter for this event (i.e., a6-Offset as defined within reportConfigNR for this event) . Mn, Ms are expressed in dBm in case of RSRP, or in dB in case of RSRQ and RS-SINR. Ocn, Ocs, Hys, Off are expressed in dB.

AI/ML models can be used to predict whether these measurement events will be triggered in advance. Given that, there is a need to monitor performance of an AI/ML model such that a proper action can be taken to in case of a performance degradation.

Embodiments of the present disclosure provide a solution for model monitoring for AI/ML based mobility. A set of prediction results may be generated using an AI/ML model, and a predication result may be corresponding to a measurement result triggering reporting of the measurement event. Then, respective accuracy indications of the set of prediction results may be obtained and an accuracy indication of the prediction result may be associated with the corresponding measurement result. A performance monitoring result of the AI/ML model may be determined based on the respective accuracy indications. The determination of the performance monitoring result may be performed at a terminal device and/or a network device.

In this way, a performance of the AI/ML model can be monitored with respect to the actual measurement result. As such, an appropriate decision regarding the AI/ML model can be made in particular in the case of performance degradation.

Some embodiments regarding model inference are first described. The terminal device 110 generates a set of prediction results for a measurement event by using an AI/ML model. A prediction result corresponds to a time (which is also referred to as a “prior time” ) and comprises a predicted possibility that the measurement event is to be triggered at the corresponding time.

For the output metric of AI/ML based measurement event, the prediction result may include some parameters for each measurement event. The following will describe by taking the Event A1 to Event A6 as examples.

In some embodiments, the prediction result further comprises at least one of: an indication of the corresponding time, wherein the corresponding time comprises a time instant or a time period; one or more predicted measurement results corresponding to the measurement event; respective predicted parameter values of one or more parameters for defining a condition for the measurement event; an identification of a predicted cell to which the measurement event is to be applied, respective confidence levels of the one or more predicted measurement results, or respective confidence levels of the predicted parameter values.

For example, for the Event A1, the prediction result includes expected prior time, e.g., a time point or time duration, which indicates the time or time duration to trigger measurement event A1. The prediction result further includes predicted Ms, predicted Hys, predicted Thresh and predicted possibility to trigger measurement event A1. Ms is the measurement result of the serving cell, not taking into account any offsets. Hys is the hysteresis parameter for the Event A1. Thresh is the threshold parameter for the Event A1. Alternatively, the possibility may be related to each parameter. One expected prior time may correspond to a set of prediction parameters.

For the Event A2, the prediction result includes expected prior time, e.g., a time point or time duration, which indicates the time or time duration to trigger measurement event A2. The prediction result further includes predicted Ms, predicted Hys, predicted Thresh and predicted possibility to trigger measurement event A2. Ms is the measurement result of the serving cell, not taking into account any offsets. Hys is the hysteresis parameter for the Event A2. Thresh is the threshold parameter for the Event A2. Alternatively, the possibility may be related to each parameter. One expected prior time may correspond to a set of prediction parameters.

For the Event A3, the prediction result includes expected prior time, e.g., a time point or time duration, which indicates the time or time duration to trigger measurement event A3. The prediction result further includes predicted Mn, predicted Ofn, predicted Ocn, predicted Mp, predicted Ofp, predicted Ocp, predicted Hys and predicted Off. Moreover, potential neighbor cell ID of which AI/ML based measurement event can be applied. Mn is the measurement result of the neighbouring cell, not taking into account any offsets. Ofn is the measurement object specific offset of the reference signal of the neighbour cell. Ocn is the cell specific offset of the neighbour cell and set to zero if not configured for the neighbour cell. Mp is the measurement result of the SpCell, not taking into account any offsets. Ofp is the measurement object specific offset of the SpCell. Ocp is the cell specific offset of the SpCell and is set to zero if not configured for the SpCell. Hys is the hysteresis parameter for this event. Off is the offset parameter for this event. Alternatively, the possibility may be related to each parameter. One expected prior time may correspond to a set of prediction parameters.

For the Event A4, the prediction result includes expected prior time, e.g., a time point or time duration, which indicates the time or time duration to trigger measurement event A4. The prediction result further includes predicted Mn, predicted Ofn, predicted Ocn, predicted Hys and predicted Off. Mn is the measurement result of the neighbouring cell, not taking into account any offsets. Ofn is the measurement object specific offset of the neighbour cell. Ocn is the measurement object specific offset of the neighbour cell and set to zero if not configured for the neighbour cell. Hys is the hysteresis parameter for this event. Thresh is the threshold parameter for this event. Moreover, potential cell ID of which AI/ML based measurement event can be applied. Alternatively, the possibility may be related to each parameter. One expected prior time may correspond to a set of prediction parameters.

For the Event A5, the prediction result includes expected prior time, e.g., a time point or time duration, which indicates the time or time duration to trigger measurement event A5. The prediction result further includes predicted Mp, predicted Mn, predicted Ofn, predicted Ocn, predicted Hys, predicted Thresh1 and predicted Thresh2. Mp is the measurement result of the NR SpCell, not taking into account any offsets. Mn is the measurement result of the neighbouring cell, not taking into account any offsets. Ofn is the measurement object specific offset of the neighbour cell. Ocn is the cell specific offset of the neighbour cell and set to zero if not configured for the neighbour cell. Hys is the hysteresis parameter for this event. Thresh1 is the threshold parameter for this event. Thresh2 is the threshold parameter for this event. Moreover, potential cell ID of which AI/ML based measurement event can be applied. Alternatively, the possibility may be related to each parameter. One expected prior time may correspond to a set of prediction parameters.

For the Event A6, the prediction result includes expected prior time, e.g., a time point or time duration, which indicates the time or time duration to trigger measurement event A6. The prediction result further includes predicted Mn, predicted Ms, predicted Ocn, predicted Ocs, predicted Hys, and predicted Off. Mn is the measurement result of the neighbouring cell, not taking into account any offsets. Ocn is the cell specific offset of the neighbour cell and set to zero if not configured for the neighbour cell. Ms is the measurement result of the serving cell, not taking into account any offsets. Ocs is the cell specific offset of the serving cell and is set to zero if not configured for the serving cell. Hys is the hysteresis parameter for this event. Off is the offset parameter for this event. Moreover, potential cell ID of which AI/ML based measurement event can be applied. Alternatively, the possibility may be related to each parameter. One expected prior time may correspond to a set of prediction parameters.

For each measurement event, the set of prediction results may be constructed as a table. A prediction result in the set of prediction results may constructed as an entry in the table, for example, a row in the table.

Reference is made to FIG. 2, which illustrates a signaling flow 200 of model monitoring in accordance with some embodiments of the present disclosure. For the purposes of discussion, the signaling flow 200 will be discussed with reference to FIG. 1, for example, by using the terminal device 110 and the network device 120. In some embodiments, the signaling flow 200 may further involve another network device, that is, a network device 201 as shown in FIG. 2.

In some embodiments, the network device 120 may determine (204) configuration information for performance monitoring of the AL/ML model. The configuration information may include at least one of a first configuration associated with measurement results for the measurement event, a second configuration associated with one or more parameters of the measurement event, or a third configuration associated with determination of the performance monitoring result. In some embodiments, the network device 120 may transmit (206) the configuration information to the terminal device 110. Correspondingly, the terminal device 110 may receive (208) the configuration information from the network device 120.

The first configuration is related to data collection to get ground truths, and is also referred to as configuration set-A. In some embodiments, the first configuration may include one or more configuration parameters, such as an indication of whether to perform reporting of the measurement event based on actual measurements, an indication of a resource for measurement event reporting, or a first timer (for example, timer A1) for obtaining a measurement result corresponding to a prediction result.

In an example, the configuration set-A may include a Boolean indication on whether the terminal device 110 can perform traditional measurement reporting in addition to the AI/ML based measurement reporting. Alternatively, or in addition, the configuration set-A may include a scheduling request (SR) configuration for medium access control (MAC) control element (CE) . For example, a specific SR resource can be included within the SR configuration for traditional measurement reporting and/or the AI/ML based measurement reporting. Alternatively, or in addition, the configuration set-A may include a timer-A1 configuration. For example, when the terminal device 110 triggers the AI/ML based measurement reporting, it may activate the timer A1. Within the duration of the timer A1, if the terminal device 110 can successfully trigger traditional measurement reporting, it will stop the timer. Otherwise, the terminal device 110 may regard the AI/ML based measurement reporting as a failure indication of this monitoring cycle, as will be described below.

The second configuration is related to how to perform data fuzzification in determining whether a prediction result is accurate, and is also referred to as configuration set-B. In some embodiments, the second configuration may include one or more configuration parameters, such as respective tolerance metrics for one or more parameters of the measurement event. The one or more parameters may include the prior time, or the parameters described above in Tables 1-6. FIG. 3 shows a schematic diagram of monitoring based on inference accuracy. As shown in FIG. 3, tolerance between the prediction and actual measurement may be applied to determine inference accuracy.

In some embodiments, the tolerance metric may include at least one of: an offset value, a percentage value, a ratio value, or a value range. Some examples are given by taking the measurement event A1 as an example. Ms is the measurement result, and a configured range (which is an example tolerance metric) regarding to the predicted Ms may be determined. For example, the configured range may be applied as one of the following configurations. In an example, an offset Os can be configured towards the terminal device 110. If the predicted Ms falls into [Ms-Os, Ms+Os] , it means the predicted Ms is accurate enough. Alternatively, or in addition, a percentage value Ps may be configured towards the terminal device 110 regarding to Ms. If the predicted Ms can be falling into [ (1-Ps) *Ms, (1+Ps) *Ms] , it means the predicted Ms is accurate enough. Alternatively, or in addition, a ratio threshold Rs can be configured towards the terminal device 110. If the predicted Ms (marked as PMs) and Ms can satisfy the following condition: | (PMs-Ms) /Ms|<=Rs, it means the predicted Ms is accurate enough. Alternatively, or in addition, a range configuration with an upper bound and lower bound may be configured to the terminal device 110.

For the prior time, the terminal device 110 may receive a configured range regarding to determine whether the predicted prior time and the time difference between actual measurement time and AI/ML triggered time is accurate enough. The range configuration can apply at least one of the following manners. In an example, an offset Os can be configured towards the terminal device 110. The prior time t may be considered as accurate, if the prior time t can be falling into [Tm –Ta –Os, Tm-Ta+Os] , where Tm is the time at which the traditional measurement event is triggered to report, Ta is the time at which the AI/ML based measurement event is triggered.

Alternatively, or in addition, a percentage value Ps can be configured towards the terminal device. If the prior time t can be falling in to [ (Tm –Ta) * (1 –Ps) , (Tm –Ta) * (1 + Ps) ] , it means the predicted prior time is accurate enough, otherwise, there is quite large error. Alternatively, or in addition, a ratio threshold Th can be configured towards the terminal device 110. If the| (Tm -Ta) /t -1|<= Th, it means the predicted prior time is accurate enough, otherwise, there is quite large error. Alternatively, or in addition, a range configuration with an upper bound and lower bound may be configured to the terminal device 110.

Similar configuration may be applied for other measurement events A2 to A6.

In some embodiments, the second configuration may indicate at least one anchor parameter of the one or more parameters. That is, the network device 120 may configure the anchor parameter reference for the terminal device 110. For each measurement event, some examples for the anchor parameter are given. For example, if Ms is configured as an anchor parameter, when the terminal device 110 performs fuzzification, it may mainly determine the accuracy based on whether the predicted Ms is accurate. If the prior time t is configured as an anchor parameter, when the terminal device 110 performs fuzzification, it will mainly determine the accuracy based on whether the predicted prior time t is accurate. If Mn is configured as an anchor parameter, when the terminal device 110 performs fuzzification, it may mainly determine the accuracy based on whether predicted Mn is accurate. If Mp is configured as an anchor parameter, when the terminal device 110 performs fuzzification, it may mainly determine the accuracy based on whether predicted Mp is accurate.

The third configuration is related to how to perform repetitive data collection to obtain performance results, and is also referred to as configuration set-C. In some embodiments, the third configuration may include one or more configuration parameters. The third configuration may include a first timer for obtaining a measurement result corresponding to a prediction result. For example, timer-A may be configured. When UE triggers AI/ML based event report, the UE will trigger the timer-A. When UE performs measurement report with respect to the AI/ML based event report, the UE will stop the timer-A. If the timer-A expires while the corresponding measurement report is still not triggered, the UE will regard this cycle as one failure cycle.

Alternatively, or in addition, the third configuration may include a second timer for obtaining a third threshold number of result pairs, a result pair comprising a prediction result and a corresponding measurement result, or in other words a result pair corresponding to a monitoring cycle. For example, a timer-B may be configured. When UE triggers the model monitoring procedure, it will trigger the timer-B. When the configured number of monitoring cycles is achieved, UE will stop the timer B. Otherwise, if the timer-B is expired while the model monitoring is not complete, the UE will report model monitoring failure indication towards the network. Such indication may be carried within at least one of the following: radio resource control (RRC) messages, UL MAC CE, or uplink control information (UCI) .

Alternatively, or in addition, the third configuration may include a first threshold number for counting accurate indications. For example, a threshold-A may be configured. After UE finishes all monitoring cycles for AI/ML model monitoring, of which the times is counter-A, the UE will decide how many cycles can be regard as accurate cycle based on the configuration set-B. If the number of accurate cycles is larger than the threshold A, it means that the current model performance is good enough, otherwise, model update procedure should be triggered.

Alternatively, or in addition, the third configuration may include a second threshold number for counting failure indications and a third timer for obtaining two adjacent failure indications. For example, a failure counter-B and timer-D may be configured. If one failure indication or failure indication report is triggered, the timer-D will be triggered. Within the duration of timer-D, if new failure indication or failure indication report is received, the counter B would be incremented, and timer-D would be reset. If the counter B reaches to a threshold, it will trigger model failure. Otherwise, UE will continue model monitoring until the counter-A reaches.

Alternatively, or in addition, the third configuration may include the third threshold number for counting the result pairs. For example, the counter-A as described above may be configured. When the UE performs one AI/ML based event report and one measurement report, it takes as one cycle for AI/ML model monitoring. When the times of cycle are less than the configured counter-A, the UE will continue to perform model monitoring until the cycle number is equal to counter-A.

Continuing with the flow 200, based on the configuration information, a monitoring procedure 210 may be performed. During the monitoring procedure 210, a set of prediction results may be generated using the AI/ML model, and a predication result may be corresponding to a measurement result triggering reporting of the measurement event. Then, respective accuracy indications of the set of prediction results may be obtained and an accuracy indication of the prediction result may be associated with the corresponding measurement result. A performance monitoring result of the AI/ML model may be determined based on the respective accuracy indications. The monitoring procedure 210 may be performed at the terminal device 110, or at the network device 120, or partially at the terminal device 110 and partially at the network device 120.

Reference is now made to FIG. 4 to illustrate an example flowchart of the monitoring procedure 400. It is to be noted that the monitoring procedure 400 is an example of the monitoring procedure 210. For purpose of illustration, the monitoring procedure 400 is described with respect to the terminal device 110. However, it is to be understood that some of the blocks may be performed at the network device 120, for example, based on the reporting from the terminal device 110.

In an example, after receiving the configuration information from the network device 120, the terminal device 110 may trigger the monitoring procedure 400 and start the configured second timer, for example, the configured timer-B.

At block 405, a monitoring cycle may be started. At block 410, a prediction result for the current monitoring cycle may be obtained. For example, the terminal device 110 may generate the AI/ML based event report, and the network device 120 may receive the AI/ML based event report from the terminal device 110. Upon triggering of the AI/ML based event reporting, the termina device 110 may start the first timer, for example the timer A or A1 mentioned above.

In an example, for each model monitoring cycle, when the terminal device 110 finishes last round of AI/ML model indication towards the network device 120, the terminal device 110 may perform the event reporting and trigger the configured timer-A. For example, the event reporting can be reported via: RRC message, UL MAC CE or UCI.

The event report may include at least one of the following information: an event ID, a model ID, a predicted Ms, a predicted Mn, a predicted Mp, an expected prior time, an accuracy possibility, or a time stamp of the time at which the AI/ML event reporting is triggered. When the AI/ML based event report is triggered, the terminal device 110 may trigger the timer-A1.

At block 420, whether a measurement report corresponding to the prediction result is triggered may be determined. In other words, whether a measurement result triggering reporting of the measurement event to the network device 120 is obtained may be determined. For example, if measurement event reporting is triggered within the duration of timer-A and timer-A1 is determined.

If a measurement report corresponding to the prediction result is not triggered within the duration of the first timer, the procedure 400 may proceed to block 440. At block 440, an accuracy indication for the current monitoring cycle may be determined as a failure indication, which indicates a failed prediction of the measurement event. In other words, the current monitoring cycle may be considered as failed. For example, when the timer-A expired while the traditional measurement reporting is not triggered, the UE may regard this monitoring cycle as one failure cycle.

In some embodiments, at block 450, the number of failure indications may be determined with respect to a second threshold number (such as the failure counter B) , a time difference between obtaining of two adjacent failure indications below a threshold time difference. If the number of failure indications exceeds the second threshold number, the procedure may proceed to block 460. At block 460, model failure is declared, in other words, the performance monitoring result may be determined as a negative result. Such an example is given in Table 7.

Table 7

If at block 450, it is determined that the number of failure indications does not exceed the second threshold number, the procedure may proceed to block 470. At block 470, it is determined whether the number of monitoring cycles reaches a threshold number, for example, the counter A. If the number of monitoring cycles does not reach the threshold number, the monitoring will continue and the procedure 400 may return back to block 405 to start a next monitoring cycle.

Reference is made back to block 420. In some embodiments, a measurement report corresponding to the prediction result may be triggered within the duration of the first timer. In other words, the terminal device 110 may transmit the corresponding measurement result to the network device 120. For example, the terminal device 110 may perform traditional measurement event reporting. The event reporting can be reported via: RRC message, UL MAC CE, or UCI. The event reporting content may include at least one of the following information: an event ID, a measured Ms, a measured Mn, a measured Mp, or a time stamp of the time at which the traditional measurement reporting is triggered.

If at block 420 a measurement report corresponding to the prediction result is triggered within the duration of the first timer, the procedure 400 may proceed to block 430. At block 430, an accuracy indication for the monitoring cycle may be determined based on the prediction result and the corresponding measurement result.

In some embodiments, for each parameter of the one or more parameters of the measurement event, an intermediate result of whether prediction of the parameter is accurate may be determined based on a predicted value for the parameter, a measured value for the parameter and a tolerance metric. The tolerance metric may be that described above, such as an offset value, a percentage value, a ratio value, or a value range. Then, the accuracy indication for the prediction result may be determined based on respective intermediate results for the one or more parameter.

In some embodiments, if at least one anchor parameter is configured, the accuracy indication may be determined in account for the at least one anchor parameter. For example, if prediction of the at least one anchor parameter is accurate, an accurate indication for the prediction result may be determined to indicate an accurate prediction of the measurement event. If prediction of an anchor parameter of the at least one anchor parameter is inaccurate, an inaccurate indication for the first prediction result may be determined to indicate an inaccurate prediction of the measurement event. An example is shown in FIG. 5. In this example, the prior time and Ms are configured as anchor parameters. Since the prediction of the prior time and Ms is accurate, the current monitoring cycle is determined as accurate.

In an example, after UE triggers both AI/ML based measurement event and traditional measurement event, the UE may determine whether each predicted parameter is accurate enough based on the configured parameter as described above with reference to FIG. 2. For each parameter, the UE may determine whether the predicted value is within the configured range. If the predicted value is within the configured range, the UE may regard the corresponding predicted parameter as accurate, otherwise, the UE may regard the corresponding predicted parameter as inaccurate.

In addition, the UE may check if the parameter is configured as an anchor parameter. If all parameters which are configured as anchor parameters are accurate enough, it means the corresponding monitoring cycle can be indicated as accurate. Otherwise, if at least one of the parameters which are configured as anchor parameters is not accurate, it means the corresponding monitoring cycle cannot be indicated as accurate. Optionally, the UE may report the indication of each monitoring cycle towards the network, via at least one of the following: RRC message, UL MAC CE or UCI.

Continuing with the procedure 400, after block 430, the procedure may proceed to block 470. At block 470, it is determined whether the number of monitoring cycles reaches a threshold number, for example, the counter A. If the number of monitoring cycles does not reach the threshold number, the monitoring will continue and the procedure 400 may return back to block 405 to start a next monitoring cycle.

If the number of monitoring cycles reaches the threshold number, the procedure 400 may proceed to block 480. At block 480, a performance monitoring result of the AI/ML model may be determined based on the respective accuracy indications of the monitoring cycles.

In some embodiments, a first number of accurate indications indicating an accurate prediction of the measurement event may be determined. Then, the performance monitoring result may be determined at least based on the first number. In an example, the performance monitoring result may be determined based on a ratio of the first number to a number of the set of prediction results. In another example, if the first number exceeds a first threshold number, the performance monitoring result may be determined as a positive result. If the first number is below the first threshold number, the performance monitoring result may be determined as a negative result.

As an example, if the indication for the current monitoring cycle is accurate enough, then the UE may increment the counter-C. After the UE finishing one monitoring cycle, the UE may increment the counter-D. If the counter-D equals to the counter-A, the UE will check if counter-C can exceed threshold-A. If counter-C can exceed threshold-A, it means that the model performance of the corresponding AI/ML model is good enough. Otherwise, the model performance is not good enough and model updating procedure would be triggered.

As another example, if the UE counts that the number of monitoring cycles is equal to the configured value of max-count, the UE may autonomously calculate the model performance result based on the following equation: performance result = the number of accurate indications /max-count [or the number of monitoring cycles] .

In some embodiments, the terminal device 110 may report the performance monitoring result to the network device 120, for example, if the performance monitoring result is below a configured threshold. The performance monitoring result may be reported via at least one of RRC message, UL MAC CE or UCI.

In some embodiments, at block 470, it may be determined whether a third threshold number of result pairs are obtained within in a time duration of a second timer, a result pair comprising a prediction result and a corresponding measurement result. In response to a failure in obtaining the third threshold number of result pairs within in the time duration of the second timer, a model failure may be determined for the AI/ML model.

As an example, a timer-B may be configured. When UE triggers the model monitoring procedure, it will trigger the timer-B. When the configured number of monitoring cycles are achieved, the UE will stop the timer B. Otherwise, if the timer-B is expired while the model monitoring is not complete, the UE will report model monitoring failure indication towards the network.

An example monitoring procedure 400 is described above. In some embodiments, during the monitoring procedure 400, the terminal device 110 may transmit, to the network device 120, at least one of the set of prediction results and corresponding measurement results, the respective accuracy indications of the set of prediction results, or the performance monitoring result. In some embodiments, the monitoring may be performed by the network device 120. In such embodiments, the network device 120 may receive, from the terminal device 110, the set of prediction results and corresponding measurement results, or the respective accuracy indications of the set of prediction results.

In some embodiments, to report the predicted results and measurement results for the measurement event, dedicated resources may be configured to the terminal device 110.

In some embodiments, a configured grant (CG) may be used for transmission of the set of prediction results and corresponding measurement results from the terminal device 110 to the terminal device 120. For example, when the UE is activated to perform model monitoring for the AI/ML based measurement event, the UE may be configured with a configured grant used for the AI/ML based event reporting and/or traditional based event reporting. The configured grant may be either configured grant type 1 or configured grant type 2, which will be associated with the specific logical channel identification (s) (LCID (s) ) used for traditional event reporting and/or AI/ML based event reporting.

In some embodiments, a third timer may be configured to be started upon use of a resource within the configured grant and to be restarted upon use of a further resource within the configured grant, and the configured grant is deactivated upon expiration of the third timer. For example, the timer-C may be configured towards the UE. Once the UE use the resources within the configured grant, the timer-C may be triggered. Within the duration of the timer-C, if the UE uses the subsequent resources of the configured grant, the UE will stop the timer-C and re-trigger the timer-C. Otherwise, if the timer-C is expired, which means the UE does not use the configured grant for plenty of time, then the configured grant would be de-activated autonomously.

In some embodiments, the configured grant is deactivated upon a fourth threshold number of consecutive resources in time domain within the configured grant are unused for the transmission. For example, the counter-C and max-count value may be configured towards the UE. If the UE uses one resource within the configured grant, the counter-C would be reset to 0. For each resource slot within the configured grant, if the UE cannot use the subsequent resource, then the counter-C would be incremented. If the value of counter-C equals to the max-count value, then the configured grant may be de-activated autonomously.

In some embodiments, an SR may be transmitted from the terminal device 110 to the network device 120. For example, the prediction result and/or the measurement result may be included in UL MAC CE. In other words, the UL MAC CE may be adopted to perform AI/ML based measurement reporting. For example, if there is not enough UL resource to accommodate the UL MAC CE and its sub MAC header, then the UE may transmit the SR according to the specific SR configuration with the specific SR resource.

In some embodiments, in response to the SR from the terminal device 110, the network device 120 may allocate respective resources for transmission of a prediction result and a corresponding measurement result from the terminal device 110 to the network device 120. In other words, in response to a single SR, two dedicated resources may be allocated for the prediction result and the measurement result respectively.

In some embodiments, a first resource for transmission of a prediction result from the terminal device 110 to the network device 120 is allocated in response to a scheduling request, and a second resource for transmission of a corresponding measurement result is allocated based on the received prediction result. For example, based on the prior time in the received prediction result, the network device 120 may be aware the time when a traditional measurement report is to be triggered. As such, the network device 120 may allocate a dedicated resource over the prior time.

In an example, when the UE performs the UL MAC CE transmission, the purpose is to allow the network to get information about the content of the AI/ML prediction, also to allow the network to get the information about the prior time that traditional measurement report would be triggered. The network may provide uplink shared channel (UL-SCH) resource for the traditional measurement report accordingly.

In such embodiments, in order to avoid signaling overhead or repetitive resource request for both AI/ML based event reporting and traditional measurement reporting, the network would configure two resources at the same time for the UE.

In some embodiments, related information exchange may occur between NG-RANs. For example, if handover of the terminal device 110 from the network device 120 to another network device (for example, the network device 201) occurs, the network device 120 may exchange related information with the network device 201.

In some embodiments, in case of handover from the network device 120 to the network device 201, the network device 120 may transmit (212) model monitoring information associated with performance monitoring of the AI/ML model to the network device 201. Correspondingly, the network device 201 may receive (214) the model monitoring information from the network device 120. For example, when the UE is triggered to perform the AI/ML model monitoring procedure, yet, a handover is triggered. If the source-gNB transmits HANDOVER COMMAND message towards the UE, the source-gNB may transmit AI/ML model monitoring related parameters towards the target-gNB.

In some embodiments, the model monitoring information may include configuration information for performance monitoring of the AL/ML model. For example, the model monitoring information may include at least one of the configuration set-A, configuration set-B, or configuration set-C.

Alternatively, or in addition, the model monitoring information may include resource information of a configured grant for transmission of the set of prediction results and corresponding measurement results. For example, the model monitoring information may include information of configured grant used for the AI/ML based reporting and measurement-based reporting.

Alternatively, or in addition, the model monitoring information may include information about at least a portion of the respective accuracy indications. For example, the model monitoring information may include the number of indications received after triggering model monitoring, as well as the number of accurate indications, inaccurate indications, or failure indications.

Alternatively, or in addition, the model monitoring information may include respective durations of one or more timers triggered during performance monitoring of the AI/ML model. For example, the model monitoring information may include a remaining duration of triggered timer-A, timer-B, or timer C.

FIG. 6 illustrates a flowchart of a communication method 600 implemented at a communication device in accordance with some embodiments of the present disclosure. In some embodiments, the method 600 may be implemented at the terminal device 110 or the network device 120, which is also referred to as a first network device.

At block 610, the communication device obtains respective accuracy indications of a set of prediction results for a measurement event. The set of prediction results are generated using an AI/ML model, a predication result is corresponding to a measurement result triggering reporting of the measurement event, and an accuracy indication of the prediction result is associated with the corresponding measurement result.

At block 620, the communication device determines a performance monitoring result of the AI/ML model based on the respective accuracy indications.

In some embodiments, the communication device may obtain a first prediction result of the set of prediction results, the first prediction result comprising respective predicted values for one or more parameters of the measurement event; obtain a first measurement result corresponding to the first prediction result, the first measurement result comprising respective measured values for the one or more parameters; and for each parameter of the one or more parameters, determine an intermediate result of whether prediction of the parameter is accurate based on a predicted value for the parameter, a measured value for the parameter and a tolerance metric; and determine the accuracy indication for the first prediction result based on respective intermediate results for the one or more parameter.

In some embodiments, the tolerance metric comprises at least one of: an offset value, a percentage value, a ratio value, or a value range.

In some embodiments, the one or more parameters comprise at least one anchor parameter, and the communication device may in accordance with a determination that prediction of the at least one anchor parameter is accurate, determine an accurate indication for the first prediction result to indicate an accurate prediction of the measurement event; and in accordance with determination that prediction of an anchor parameter of the at least one anchor parameter is inaccurate, determine an inaccurate indication for the first prediction result to indicate an inaccurate prediction of the measurement event.

In some embodiments, the communication device may obtain a second prediction result of the set of prediction results; and in response to a failure in obtaining, within a time duration of a first timer, a second measurement result corresponding to the second prediction result, determine a failure indication for the second prediction result to indicate a failed prediction of the measurement event.

In some embodiments, the communication device may determine a first number of accurate indications indicating an accurate prediction of the measurement event; and determine the performance monitoring result at least based on the first number.

In some embodiments, the communication device may determine the performance monitoring result based on a ratio of the first number to a number of the set of prediction results.

In some embodiments, the communication device may in accordance with a determination that the first number exceeds a first threshold number, determine the performance monitoring result as a positive result; and in accordance with a determination that the first number is below the first threshold number, determine the performance monitoring result as a negative result.

In some embodiments, the communication device may determine a second number of failure indications indicating a failed prediction of the measurement event, a time difference between obtaining of two adjacent failure indications below a threshold time difference; and in accordance with a determination that the second number exceeds a second threshold number, determine the performance monitoring result as a negative result.

In some embodiments, the communication device may determine whether a third threshold number of result pairs are obtained within in a time duration of a second timer, a result pair comprising a prediction result and a corresponding measurement result; and in response to a failure in obtaining the third threshold number of result pairs within in the time duration of the second timer, determine a model failure for the AI/ML model.

In some embodiments, the communication device comprises a terminal device, and the terminal device may receive, from a network device, configuration information for performance monitoring of the AL/ML model, the configuration information comprising at least one of: a first configuration associated with measurement results for the measurement event, a second configuration associated with one or more parameters of the measurement event, or a third configuration associated with determination of the performance monitoring result.

In some embodiments, the first configuration comprises at least one of: an indication of whether to perform reporting of the measurement event based on actual measurements, an indication of a resource for measurement event reporting, or a first timer for obtaining a measurement result corresponding to a prediction result.

In some embodiments, the second configuration comprises at least one of: respective tolerance metrics for one or more parameters of the measurement event, or at least one anchor parameter of the one or more parameters.

In some embodiments, the second configuration comprises at least one of: a first timer for obtaining a measurement result corresponding to a prediction result, a second timer for obtaining a third threshold number of result pairs, a result pair comprising a prediction result and a corresponding measurement result, a first threshold number for counting accurate indications, a second threshold number for counting failure indications and a third timer for obtaining two adjacent failure indications, or the third threshold number for counting the result pairs.

In some embodiments, the terminal device may transmit, to the first network device, at least one of: the set of prediction results and corresponding measurement results, the respective accuracy indications of the set of prediction results, or the performance monitoring result.

In some embodiments, the communication device comprises a first network device and the first network device may receive from a terminal device at least one of: the set of prediction results and corresponding measurement results, or the respective accuracy indications of the set of prediction results.

In some embodiments, the communication device comprises a terminal device which generates the set prediction results or a first terminal device.

In some embodiments, a configured grant is used for transmission of the set of prediction results and corresponding measurement results from the terminal device to the first terminal device.

In some embodiments, a third timer is configured to be started upon use of a resource within the configured grant and to be restarted upon use of a further resource within the configured grant, and the configured grant is deactivated upon expiration of the third timer.

In some embodiments, the configured grant is deactivated upon a fourth threshold number of consecutive resources in time domain within the configured grant are unused for the transmission.

In some embodiments, respective resources for transmission of a prediction result and a corresponding measurement result from the terminal device to the first network device are allocated in response to a scheduling request, or a first resource for transmission of a prediction result from the terminal device to the first network device is allocated in response to a scheduling request, and a second resource for transmission of a corresponding measurement result is allocated based on the received prediction result.

In some embodiments, model monitoring information associated with performance monitoring of the AI/ML model is transmitted from the first network device to a second network device in response to a handover of the terminal device from the first network device to a second network device.

In some embodiments, the model monitoring information comprises at least one of:configuration information for performance monitoring of the AL/ML model, resource information of a configured grant for transmission of the set of prediction results and corresponding measurement results, information about at least a portion of the respective accuracy indications, or respective durations of one or more timers triggered during performance monitoring of the AI/ML model.

FIG. 7 is a simplified block diagram of a device 700 that is suitable for implementing embodiments of the present disclosure. The device 700 can be considered as a further example implementation of any of the devices as shown in FIG. 1. Accordingly, the device 700 can be implemented at or as at least a part of the terminal device 110 or the network device 120, or the network device 201.

As shown, the device 700 includes a processor 710, a memory 720 coupled to the processor 710, a suitable transceiver 740 coupled to the processor 710, and a communication interface coupled to the transceiver 740. The memory 720 stores at least a part of a program 730. The transceiver 740 may be for bidirectional communications or a unidirectional communication based on requirements. The transceiver 740 may include at least one of a transmitter 742 and a receiver 744. The transmitter 742 and the receiver 744 may be functional modules or physical entities. The transceiver 740 has at least one antenna to facilitate communication, though in practice an Access Node mentioned in this application may have several ones. The communication interface may represent any interface that is necessary for communication with other network elements, such as X2/Xn interface for bidirectional communications between eNBs/gNBs, S1/NG interface for communication between a Mobility Management Entity (MME) /Access and Mobility Management Function (AMF) /SGW/UPF and the eNB/gNB, Un interface for communication between the eNB/gNB and a relay node (RN) , or Uu interface for communication between the eNB/gNB and a terminal device.

The program 730 is assumed to include program instructions that, when executed by the associated processor 710, enable the device 700 to operate in accordance with the embodiments of the present disclosure, as discussed herein with reference to FIGS. 1 to 6. The embodiments herein may be implemented by computer software executable by the processor 710 of the device 700, or by hardware, or by a combination of software and hardware. The processor 710 may be configured to implement various embodiments of the present disclosure. Furthermore, a combination of the processor 710 and memory 720 may form processing means 750 adapted to implement various embodiments of the present disclosure.

The memory 720 may be of any type suitable to the local technical network and may be implemented using any suitable data storage technology, such as a non-transitory computer readable storage medium, semiconductor based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory, as non-limiting examples. While only one memory 720 is shown in the device 700, there may be several physically distinct memory modules in the device 700. The processor 710 may be of any type suitable to the local technical network, and may include one or more of general purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on multicore processor architecture, as non-limiting examples. The device 700 may have multiple processors, such as an application specific integrated circuit chip that is slaved in time to a clock which synchronizes the main processor.

According to embodiments of the present disclosure, a communication device comprising a circuitry is provided. The circuitry is configured to: obtain respective accuracy indications of a set of prediction results for a measurement event, wherein the set of prediction results are generated using an Artificial Intelligence/Machine learning (AI/ML) model, a predication result is corresponding to a measurement result triggering reporting of the measurement event, and an accuracy indication of the prediction result is associated with the corresponding measurement result; and determine a performance monitoring result of the AI/ML model based on the respective accuracy indications. According to embodiments of the present disclosure, the circuitry may be configured to perform any method implemented by the communication device as discussed above.

The term “circuitry” used herein may refer to hardware circuits and/or combinations of hardware circuits and software. For example, the circuitry may be a combination of analog and/or digital hardware circuits with software/firmware. As a further example, the circuitry may be any portions of hardware processors with software including digital signal processor (s) , software, and memory (ies) that work together to cause an apparatus, such as a terminal device or a network device, to perform various functions. In a still further example, the circuitry may be hardware circuits and or processors, such as a microprocessor or a portion of a microprocessor, that requires software/firmware for operation, but the software may not be present when it is not needed for operation. As used herein, the term circuitry also covers an implementation of merely a hardware circuit or processor (s) or a portion of a hardware circuit or processor (s) and its (or their) accompanying software and/or firmware.

According to embodiments of the present disclosure, an apparatus is provided. The apparatus comprises means for obtaining respective accuracy indications of a set of prediction results for a measurement event, wherein the set of prediction results are generated using an Artificial Intelligence/Machine learning (AI/ML) model, a predication result is corresponding to a measurement result triggering reporting of the measurement event, and an accuracy indication of the prediction result is associated with the corresponding measurement result; and means for determining a performance monitoring result of the AI/ML model based on the respective accuracy indications. In some embodiments, the apparatus may comprise means for performing the respective operations of the method 600. In some example embodiments, the apparatus may further comprise means for performing other operations in some example embodiments of the method 600. The means may be implemented in any suitable form. For example, the means may be implemented in a circuitry or software module.

In summary, embodiments of the present disclosure provide the following aspects.

In an aspect, it is proposed a communication device comprising: a processor configured to cause the communication device to: obtain respective accuracy indications of a set of prediction results for a measurement event, wherein the set of prediction results are generated using an Artificial Intelligence/Machine learning (AI/ML) model, a predication result is corresponding to a measurement result triggering reporting of the measurement event, and an accuracy indication of the prediction result is associated with the corresponding measurement result; and determine a performance monitoring result of the AI/ML model based on the respective accuracy indications.

In some embodiments, the communication device is caused to: obtain a first prediction result of the set of prediction results, the first prediction result comprising respective predicted values for one or more parameters of the measurement event; obtain a first measurement result corresponding to the first prediction result, the first measurement result comprising respective measured values for the one or more parameters; and for each parameter of the one or more parameters, determine an intermediate result of whether prediction of the parameter is accurate based on a predicted value for the parameter, a measured value for the parameter and a tolerance metric; and determine the accuracy indication for the first prediction result based on respective intermediate results for the one or more parameter.

In some embodiments, the one or more parameters comprise at least one anchor parameter, and the communication device is caused to: in accordance with a determination that prediction of the at least one anchor parameter is accurate, determine an accurate indication for the first prediction result to indicate an accurate prediction of the measurement event; and in accordance with determination that prediction of an anchor parameter of the at least one anchor parameter is inaccurate, determine an inaccurate indication for the first prediction result to indicate an inaccurate prediction of the measurement event.

In some embodiments, the communication device is caused to: obtain a second prediction result of the set of prediction results; and in response to a failure in obtaining, within a time duration of a first timer, a second measurement result corresponding to the second prediction result, determine a failure indication for the second prediction result to indicate a failed prediction of the measurement event.

In some embodiments, the communication device is caused to: determine a first number of accurate indications indicating an accurate prediction of the measurement event; and determine the performance monitoring result at least based on the first number.

In some embodiments, the communication device is caused to: determine the performance monitoring result based on a ratio of the first number to a number of the set of prediction results.

In some embodiments, the communication device is caused to: in accordance with a determination that the first number exceeds a first threshold number, determine the performance monitoring result as a positive result; and in accordance with a determination that the first number is below the first threshold number, determine the performance monitoring result as a negative result.

In some embodiments, the communication device is caused to: determine a second number of failure indications indicating a failed prediction of the measurement event, a time difference between obtaining of two adjacent failure indications below a threshold time difference; and in accordance with a determination that the second number exceeds a second threshold number, determine the performance monitoring result as a negative result.

In some embodiments, the communication device is caused to: determine whether a third threshold number of result pairs are obtained within in a time duration of a second timer, a result pair comprising a prediction result and a corresponding measurement result; and in response to a failure in obtaining the third threshold number of result pairs within in the time duration of the second timer, determine a model failure for the AI/ML model.

In some embodiments, the communication device comprises a terminal device, and the terminal device is further caused to: receive, from a network device, configuration information for performance monitoring of the AL/ML model, the configuration information comprising at least one of: a first configuration associated with measurement results for the measurement event, a second configuration associated with one or more parameters of the measurement event, or a third configuration associated with determination of the performance monitoring result.

In some embodiments, the terminal device is further caused to: transmit, to the first network device, at least one of: the set of prediction results and corresponding measurement results, the respective accuracy indications of the set of prediction results, or the performance monitoring result.

In some embodiments, the communication device comprises a first network device and the first network device is further caused to: receive from a terminal device at least one of: the set of prediction results and corresponding measurement results, or the respective accuracy indications of the set of prediction results.

In some embodiments, the model monitoring information comprises at least one of: configuration information for performance monitoring of the AL/ML model, resource information of a configured grant for transmission of the set of prediction results and corresponding measurement results, information about at least a portion of the respective accuracy indications, or respective durations of one or more timers triggered during performance monitoring of the AI/ML model.

In an aspect, a communication device comprises: at least one processor; and at least one memory coupled to the at least one processor and storing instructions thereon, the instructions, when executed by the at least one processor, causing the device to perform the method implemented by the communication device discussed above.

In an aspect, a computer readable medium having instructions stored thereon, the instructions, when executed on at least one processor, causing the at least one processor to perform the method implemented by the communication device discussed above.

In an aspect, a computer program comprising instructions, the instructions, when executed on at least one processor, causing the at least one processor to perform the method implemented by the communication device discussed above.

Generally, various embodiments of the present disclosure may be implemented in hardware or special purpose circuits, software, logic or any combination thereof. Some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software which may be executed by a controller, microprocessor or other computing device. While various aspects of embodiments of the present disclosure are illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it will be appreciated that the blocks, apparatus, systems, techniques or methods described herein may be implemented in, as non-limiting examples, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing devices, or some combination thereof.

The present disclosure also provides at least one computer program product tangibly stored on a non-transitory computer readable storage medium. The computer program product includes computer-executable instructions, such as those included in program modules, being executed in a device on a target real or virtual processor, to carry out the process or method as described above with reference to FIGS. 1 to 7. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, or the like that perform particular tasks or implement particular abstract data types. The functionality of the program modules may be combined or split between program modules as desired in various embodiments. Machine-executable instructions for program modules may be executed within a local or distributed device. In a distributed device, program modules may be located in both local and remote storage media.

Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowcharts and/or block diagrams to be implemented. The program code may execute entirely on a machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.

The above program code may be embodied on a machine readable medium, which may be any tangible medium that may contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine readable medium may be a machine readable signal medium or a machine readable storage medium. A machine readable medium may include but not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine readable storage medium would include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM) , a read-only memory (ROM) , an erasable programmable read-only memory (EPROM or Flash memory) , an optical fiber, a portable compact disc read-only memory (CD-ROM) , an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are contained in the above discussions, these should not be construed as limitations on the scope of the present disclosure, but rather as descriptions of features that may be specific to particular embodiments. Certain features that are described in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable sub-combination.

Although the present disclosure has been described in language specific to structural features and/or methodological acts, it is to be understood that the present disclosure defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

A communication device comprising:

a processor configured to cause the communication device to:

obtain respective accuracy indications of a set of prediction results for a measurement event, wherein the set of prediction results are generated using an Artificial Intelligence/Machine learning (AI/ML) model, a predication result is corresponding to a measurement result triggering reporting of the measurement event, and an accuracy indication of the prediction result is associated with the corresponding measurement result; and

determine a performance monitoring result of the AI/ML model based on the respective accuracy indications.
The device of claim 1, wherein the communication device is caused to:

obtain a first prediction result of the set of prediction results, the first prediction result comprising respective predicted values for one or more parameters of the measurement event;

obtain a first measurement result corresponding to the first prediction result, the first measurement result comprising respective measured values for the one or more parameters; and

for each parameter of the one or more parameters, determine an intermediate result of whether prediction of the parameter is accurate based on a predicted value for the parameter, a measured value for the parameter and a tolerance metric; and

determine the accuracy indication for the first prediction result based on respective intermediate results for the one or more parameter.
The device of claim 2, wherein the tolerance metric comprises at least one of:

an offset value,

a percentage value,

a ratio value, or

a value range.
The device of claim 2, wherein the one or more parameters comprise at least one anchor parameter, and the communication device is caused to:

in accordance with a determination that prediction of the at least one anchor parameter is accurate, determine an accurate indication for the first prediction result to indicate an accurate prediction of the measurement event; and

in accordance with determination that prediction of an anchor parameter of the at least one anchor parameter is inaccurate, determine an inaccurate indication for the first prediction result to indicate an inaccurate prediction of the measurement event.
The device of claim 1, wherein the communication device is caused to:

obtain a second prediction result of the set of prediction results; and

in response to a failure in obtaining, within a time duration of a first timer, a second measurement result corresponding to the second prediction result, determine a failure indication for the second prediction result to indicate a failed prediction of the measurement event.
The device of claim 1, wherein the communication device is caused to:

determine a second number of failure indications indicating a failed prediction of the measurement event, a time difference between obtaining of two adjacent failure indications below a threshold time difference; and

in accordance with a determination that the second number exceeds a second threshold number, determine the performance monitoring result as a negative result.
The device of claim 1, wherein the communication device is caused to:

determine whether a third threshold number of result pairs are obtained within in a time duration of a second timer, a result pair comprising a prediction result and a corresponding measurement result; and

in response to a failure in obtaining the third threshold number of result pairs within in the time duration of the second timer, determine a model failure for the AI/ML model.
The device of claim 1, wherein the communication device comprises a terminal device, and the terminal device is further caused to:

receive, from a network device, configuration information for performance monitoring of the AL/ML model, the configuration information comprising at least one of:

a first configuration associated with measurement results for the measurement event,

a second configuration associated with one or more parameters of the measurement event, or

a third configuration associated with determination of the performance monitoring result.
The device of claim 8, wherein the second configuration comprises at least one of:

a first timer for obtaining a measurement result corresponding to a prediction result,

a second timer for obtaining a third threshold number of result pairs, a result pair comprising a prediction result and a corresponding measurement result,

a first threshold number for counting accurate indications,

a second threshold number for counting failure indications and a third timer for obtaining two adjacent failure indications, or

the third threshold number for counting the result pairs.
The device of claim 8, wherein the terminal device is further caused to:

transmit, to the first network device, at least one of:

the set of prediction results and corresponding measurement results,

the respective accuracy indications of the set of prediction results, or

the performance monitoring result.
The device of claim 1, wherein the communication device comprises a first network device and the first network device is further caused to:

receive from a terminal device at least one of:

the set of prediction results and corresponding measurement results, or

the respective accuracy indications of the set of prediction results.
The device of claim 1, wherein the communication device comprises a terminal device which generates the set prediction results or a first terminal device.
The device of claim 12, wherein a configured grant is used for transmission of the set of prediction results and corresponding measurement results from the terminal device to the first terminal device.
The device of claim 13, wherein a third timer is configured to be started upon use of a resource within the configured grant and to be restarted upon use of a further resource within the configured grant, and the configured grant is deactivated upon expiration of the third timer.
The device of claim 13, wherein the configured grant is deactivated upon a fourth threshold number of consecutive resources in time domain within the configured grant are unused for the transmission.
The device of claim 12, wherein respective resources for transmission of a prediction result and a corresponding measurement result from the terminal device to the first network device are allocated in response to a scheduling request, or

a first resource for transmission of a prediction result from the terminal device to the first network device is allocated in response to a scheduling request, and a second resource for transmission of a corresponding measurement result is allocated based on the received prediction result.
The device of claim 12, wherein model monitoring information associated with performance monitoring of the AI/ML model is transmitted from the first network device to a second network device in response to a handover of the terminal device from the first network device to a second network device.
The device of claim 17, wherein the model monitoring information comprises at least one of:

configuration information for performance monitoring of the AL/ML model,

resource information of a configured grant for transmission of the set of prediction results and corresponding measurement results,

information about at least a portion of the respective accuracy indications, or

respective durations of one or more timers triggered during performance monitoring of the AI/ML model.
A communication method implemented at a communication device, comprising:

obtaining respective accuracy indications of a set of prediction results for a measurement event, wherein the set of prediction results are generated using an Artificial Intelligence/Machine learning (AI/ML) model, a predication result is corresponding to a measurement result triggering reporting of the measurement event, and an accuracy indication of the prediction result is associated with the corresponding measurement result; and

determining a performance monitoring result of the AI/ML model based on the respective accuracy indications.
A computer readable medium having instructions stored thereon, the instructions, when executed on at least one processor, causing the at least one processor to perform the method according to claim 19.