WO2025092999A1 - Model performance supervision method and apparatus, and device - Google Patents

Abstract

The present application belongs to the field of communications. Disclosed are a model performance supervision method and apparatus, and a device, which can solve the problem of performance supervision for AI models. The model performance supervision method in the embodiments of the present application comprises: a first device acquiring first information, wherein the first information is used for representing the uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature; and the first device determining third information on the basis of the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used for determining the validity of the first AI model.

Description

Model performance supervision method, device and equipment

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the priority of the Chinese patent application filed with the China Patent Office on November 3, 2023, with application number 202311467210.7 and invention name “Model performance supervision method, device and equipment”, the entire contents of which are incorporated by reference in this application.

Technical Field

The present application relates to the field of communications, and more specifically, to a model performance supervision method, apparatus and device.

Background Art

In the New Radio (NR) system, artificial intelligence (AI) models are introduced to improve system performance. For example, AI models are introduced for positioning, beam management, channel state information (CSI) prediction, mobility management, CSI compression, etc. However, when the wireless propagation environment changes, the performance of the AI model may be difficult to guarantee. How to supervise the performance of the AI model is a problem that needs to be solved.

Summary of the invention

The embodiments of the present application provide a model performance supervision method, device and equipment, which can solve the performance supervision problem of AI models.

In the first aspect, a model performance supervision method is provided, comprising:

The first device acquires first information, wherein the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;

The first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

Secondly, a model performance supervision method is provided, including:

The second device receives third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is the first feature, and the second information is a label corresponding to the first feature;

The second device determines the validity of the first AI model based on the third information.

Thirdly, a model performance supervision method is provided, including:

The first device obtains fourth information, where the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;

The first device determines the validity of the second AI model based on the fourth information; or, the first device sends the fourth information to the second device.

Fourthly, a model performance supervision method is provided, including:

The second device receives fourth information from the first device, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;

The second device determines the validity of the second AI model based on the fourth information.

In a fifth aspect, a model performance monitoring device is provided, comprising:

an acquisition unit, configured to acquire first information, wherein the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;

A processing unit, used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

In a sixth aspect, a model performance monitoring device is provided, comprising:

a transceiver unit, configured to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature;

A processing unit is used to determine the validity of the first AI model based on the third information.

In a seventh aspect, a model performance monitoring device is provided, comprising:

an acquisition unit, configured to acquire fourth information, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;

A processing unit, used to determine the validity of the second AI model according to the fourth information; or a transceiver unit, used to send the fourth information to the second device.

In an eighth aspect, a model performance monitoring device is provided, comprising:

a transceiver unit, configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second AI model, and the input of the second AI model is the third feature;

A processing unit, configured to determine the validity of the second AI model based on the fourth information.

In the ninth aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

In the tenth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface or the processor is used to obtain first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first AI model, and the input of the first AI model is a first feature; the processor is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

In the eleventh aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the second aspect are implemented.

In the twelfth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface is used to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; the processor is used to determine the validity of the first AI model based on the third information.

In the thirteenth aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the third aspect are implemented.

In the fourteenth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface or the processor is used to obtain fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of a second AI model, and the input of the second AI model is a third feature; the processor is used to determine the validity of the second AI model based on the fourth information; or, the communication interface is used to send the fourth information to a second device.

In the fifteenth aspect, a model performance monitoring device is provided, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the fourth aspect are implemented.

In the sixteenth aspect, a model performance supervision device is provided, comprising a processor and a communication interface; wherein the communication interface is used to receive fourth information from a first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of a second AI model, and the input of the second AI model is a third feature; the processor is used to determine the validity of the second AI model based on the fourth information.

In a seventeenth aspect, a readable storage medium is provided, wherein a program or instruction is stored on the readable storage medium. When the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented, or the steps of the method described in the third aspect are implemented, or the steps of the method described in the fourth aspect are implemented.

In the eighteenth aspect, a wireless communication system is provided, including: a first device and a second device, wherein the first device can be used to execute the steps of the method described in the first aspect or the third aspect, and the second device can be used to execute the steps of the method described in the second aspect or the fourth aspect.

In the nineteenth aspect, a chip is provided, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instructions to implement the method as described in the first aspect, or the method as described in the second aspect, or the method as described in the third aspect, or the method as described in the fourth aspect.

In the twentieth aspect, a computer program/program product is provided, wherein the computer program/program product is stored in a storage medium, and the program/program product is executed by at least one processor to implement the steps of the model performance supervision method as described in at least one of the first to fourth aspects.

In the embodiments of the first aspect or the second aspect of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, so as to achieve performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.

In the embodiments of the third aspect or the fourth aspect of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

FIG1 is a schematic diagram of a communication system architecture provided in an embodiment of the present application.

FIG2 is a schematic diagram of a neural network provided in the present application.

FIG. 3 is a schematic diagram of a neuron provided in the present application.

FIG4 is a schematic flowchart of a model performance supervision method provided according to an embodiment of the present application.

FIG5 is a schematic diagram of improving positioning accuracy based on soft information according to an embodiment of the present application.

FIG6 is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.

FIG7 is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.

FIG8 is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.

FIG9 is a schematic block diagram of a model performance monitoring device provided according to an embodiment of the present application.

FIG10 is a schematic block diagram of another model performance monitoring device provided according to an embodiment of the present application.

FIG11 is a schematic block diagram of another model performance monitoring device provided according to an embodiment of the present application.

FIG12 is a schematic block diagram of yet another model performance monitoring device provided according to an embodiment of the present application.

FIG13 is a schematic block diagram of a communication device provided according to an embodiment of the present application.

FIG14 is a schematic diagram of the hardware structure of a terminal provided according to an embodiment of the present application.

FIG15 is a schematic block diagram of a network-side device provided according to an embodiment of the present application.

FIG16 is a schematic block diagram of another network-side device provided according to an embodiment of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

The terms "first", "second", etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "or" in the present application represents at least one of the connected objects. For example, "A or B" covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B. The character "/" generally indicates that the objects associated with each other are in an "or" relationship.

The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.

It is worth noting that the technology described in the embodiments of the present application is not limited to the Ambient Internet of Things (IoT) system, but can also be used in other wireless communication systems, such as Long Term Evolution (LTE)/LTE-Advanced (LTE-A) system, Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Orthogonal Frequency Division Multiple Access (OFDMA), Single-carrier Frequency-Division Multiple Access (SC-FDMA), Wireless Local Area Networks (WLAN), Wireless Fidelity (WiFi), Bluetooth system, or other systems. The terms "system" and "network" in the embodiments of the present application are often used interchangeably, and the described technology can be used for the above-mentioned systems and radio technologies as well as other systems and radio technologies. The following description describes a New Radio (NR) system for example purposes, and NR terminology is used in most of the following description, but these technologies can also be applied to systems other than NR systems, such as ^6th Generation (6G) communication systems.

Fig. 1 shows a block diagram of a wireless communication system applicable to the embodiments of the present application. The wireless communication system includes a terminal 11 and a network side device 12, wherein the terminal 11 can communicate with the network side device 12 directly or through other network elements.

Among them, the terminal 11 can be a mobile phone, a tablet computer (Tablet Personal Computer), a laptop computer (Laptop Computer), a notebook computer, a personal digital assistant (Personal Digital Assistant, PDA), a handheld computer, a netbook, an ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR), a virtual reality (Virtual Reality, VR) device, a robot, a wearable device (Wearable Device), an aircraft (flight vehicle), a vehicle-mounted device (Vehicle User Equipment, VUE), a ship-mounted device, a pedestrian terminal (Pedestrian User Equipment, PUE), a smart home (home appliances with wireless communication functions, such as refrigerators, televisions, washing machines or furniture, etc.), a game console, a personal computer (Personal Computer, PC), a teller machine or a self-service machine and other terminal side devices. Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart Bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc. Among them, the vehicle-mounted device can also be called a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip or a vehicle-mounted unit, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiment of the present application.

The network side device 12 may include an access network device or a core network device.

Among them, the access network equipment can also be called radio access network (Radio Access Network, RAN) equipment, radio access network function or radio access network unit. The access network equipment may include base stations, wireless local area network (Wireless Local Area Network, WLAN) access points (Access Point, AS) or wireless fidelity (Wireless Fidelity, WiFi) nodes, etc. Among them, the base station can be called node B (Node B, NB), evolved node B (Evolved Node B, eNB), next generation node B (the next generation Node B, gNB), new radio node B (New Radio Node B, NR Node B), access point, relay station (Relay Base Station, RBS), serving base station (Serving Base Station, SBS), base transceiver station (Base Transceiver Station, BTS), radio base station, radio transceiver, base The base station is not limited to specific technical terms as long as the same technical effect is achieved. It should be noted that in the embodiments of the present application, only the base station in the NR system is taken as an example for introduction, and the specific type of the base station is not limited.

Among them, the core network equipment may include but is not limited to at least one of the following: core network nodes, core network functions, mobility management entity (Mobility Management Entity, MME), access mobility management function (Access and Mobility Management Function, AMF), session management function (Session Management Function, SMF), user plane function (User Plane Function, UPF), policy control function (Policy Control Function, PCF), policy and charging rules function unit (Policy and Charging Rules Function, PCRF), edge application service discovery function (Edge Application Server Discovery Function, EASDF), unified data management (U nified Data Management, UDM), Unified Data Repository (UDR), Home Subscriber Server (HSS), Centralized network configuration (CNC), Network Repository Function (NRF), Network Exposure Function (NEF), Local NEF (L-NEF), Binding Support Function (BSF), Application Function (AF), Location Management Function (LMF), etc. It should be noted that, in the embodiments of the present application, only the core network device in the NR system is taken as an example for introduction, and the specific type of the core network device is not limited.

In order to facilitate a better understanding of the embodiments of the present application, the technologies related to the present application are explained.

Artificial intelligence (AI) has been widely used in various fields. Integrating artificial intelligence into wireless communication networks and significantly improving technical indicators such as throughput, latency, and user capacity are important tasks for future wireless communication networks. There are many ways to implement AI modules, such as neural networks, decision trees, support vector machines, Bayesian classifiers, etc. This application takes neural networks as an example for illustration, but does not limit the specific type of AI modules.

An exemplary neural network can be shown in FIG2 , where the neural network is composed of neurons, and the neurons can be shown in FIG3 , where α ₁ , α ₂ , … α _K are inputs, w is a weight (multiplicative coefficient), b is a bias (additive coefficient), and σ(.) is an activation function. Common activation functions include Sigmoid, tanh, Rectified Linear Unit (ReLU), and the like.

The parameters of the neural network are optimized using a gradient optimization algorithm. A gradient optimization algorithm is a type of algorithm that minimizes or maximizes an objective function (sometimes called a loss function), and the objective function is often a mathematical combination of model parameters and data. For example, given data X and its corresponding label Y, we build a neural network model f(.). With the model, we can get the predicted output f(x) based on the input x, and we can calculate the difference between the predicted value and the true value (f(x)-Y), which is the loss function. The purpose of model training is to find the appropriate w, b to minimize the value of the above loss function. The smaller the loss value, the closer the constructed neural network model is to the actual situation.

For example, in the training process of neural network models, common optimization algorithms are basically based on the error back propagation (BP) algorithm. The basic idea of BP algorithm is that the learning process is a forward propagation of the signal. The error back propagation process consists of two processes: forward propagation and error back propagation. During forward propagation, the input sample is transmitted from the input layer, processed by each hidden layer layer by layer, and then transmitted to the output layer. If the actual output of the output layer does not match the expected output, the error back propagation stage is entered. Error back propagation is to propagate the output error layer by layer through the hidden layer in some form, and distribute the error to all units in each layer, so as to obtain the error signal of each layer unit. This error signal is used as the basis for correcting the weights of each unit. This process of adjusting the weights of each layer of the signal forward propagation and error back propagation is repeated. The process of continuous adjustment of weights is also the learning and training process of the network. This process continues until the error of the network output is reduced to an acceptable level, or until the pre-set number of learning times is reached.

Common optimization algorithms include Gradient Descent, Stochastic Gradient Descent (SGD), mini-batch gradient descent, Momentum, Nesterov (specifically stochastic gradient descent with momentum), Adaptive Gradient Descent (Adagrad), Adadelta, root mean square prop (RMSprop), Adaptive Momentum Estimation (Adam), etc.

When these optimization algorithms are backpropagating errors, they can calculate the derivative/partial derivative of neurons based on the error/loss obtained from the loss function, add the influence of the learning rate, the previous gradient/derivative/partial derivative, etc., get the gradient, and pass the gradient to the previous layer.

To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

FIG4 is a schematic flow chart of a model performance supervision method 200 according to an embodiment of the present application. As shown in FIG4 , the model performance supervision method 200 may include at least part of the following contents:

S210: The first device obtains first information, where the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;

S220, the first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

It should be understood that FIG4 shows the steps or operations of the model performance supervision method 200, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG4.

In an embodiment of the present application, third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.

Exemplarily, other types of information may be as follows: motion state information (such as speed, acceleration, etc.), signal quality measurement information (such as reference signal received power (Reference Signal Received Power, RSRP), signal to interference plus noise ratio (Signal to Interference plus Noise Ratio, SINR), reference signal received quality (Reference Signal Received Quality, RSRQ), etc.).

Exemplarily, the judgment result of the effectiveness of the first AI model obtained based on the embodiment of the present application can also be combined with the judgment result of the effectiveness of the first AI model obtained by other model supervision methods to obtain the final conclusion on the effectiveness of the first AI model.

It should be noted that in the embodiment of the present application, the higher the uncertainty of the output of the first AI model, the lower the accuracy of the output of the first AI model.

The tag described in the embodiment of the present application is obtained in some way and is associated with the target task. This tag can be obtained through measurement or other prior information; for example, based on the positioning of the AI model, the input of the AI model is the channel state information, and the output of the AI model is the uncertainty or probability distribution of the position. Then this tag is the position information corresponding to the channel state information. The position information can be obtained by GPS and other positioning methods, or by a positioning reference unit with a known position. arrive.

In an embodiment of the present application, the first information may also be referred to as soft information (such as probability distribution, confidence level, confidence interval, etc.), wherein the soft information gives the probability distribution or confidence level of the possible results. Optionally, the first AI model may be a soft information AI model or may not be a soft information AI model. Among them, the soft information AI model refers to a type of AI model that outputs soft information (such as probability distribution, confidence level, confidence interval, etc.), including both classic probability models and AI models based on neural networks; the soft information AI model measures the possibility of different prediction results and gives the probability distribution or confidence level of each possible result.

It should be noted that compared with the hard information (hard value) obtained by AI model reasoning (such as Time of Arrival (TOA), Reference Signal Time Difference (RSTD), Angle of Arrival (AoA), Angle of Departure (AoD), Reference Signal Received Power (RSRP), Line of Sight (LOS) indication, Non Line of Sight (NLOS) indication, etc.), the soft information obtained by AI model reasoning can significantly improve the reasoning accuracy and robustness. Specifically, soft information can better describe the uncertainty of the world, improve the robustness of model reasoning, and provide better security for some businesses that require relatively high reasoning reliability.

In some embodiments, the dimension of the first feature is D1 dimension, and correspondingly, the first information is D2 group of soft information of the second feature, wherein the D2 group of soft information describes the range of the output of the first AI model. If there is only one group of soft information (ie, D2=1), then this group of soft information describes the uncertainty of the output of the entire first AI model; if there are at least two groups of soft information (ie, D2≥2), then each group of soft information describes the uncertainty of some features of the output of the first AI model.

For example, for prediction tasks: each prediction moment corresponds to a set of soft information, or each feature at each prediction moment corresponds to a set of soft information; for other tasks, each feature corresponds to a set of soft information, or all features correspond to a set of soft information. For example, if the output of the first AI model is 2-dimensional, then it corresponds to 2 sets of soft information.

For example, for a positioning task, the first AI model output is a 2-dimensional horizontal position coordinate, the 2-dimensional horizontal position coordinate corresponds to a set of soft information, or each dimension of the 2-dimensional horizontal position coordinate corresponds to a set of soft information.

In some embodiments, the AI model described in this application may also be referred to as an AI unit, an AI model/AI unit, a machine learning (ML) model, an ML unit, an AI structure, an AI function, an AI feature, a neural network, a neural network function, a neural network function, etc., or the AI model described in this application may also refer to a processing unit capable of implementing specific algorithms, formulas, processing procedures, capabilities, etc. related to AI, or the AI model described in this application may be a processing method, algorithm, function, module or unit for a specific data set, or the AI model described in this application may be a processing method, algorithm, function, module or unit running on AI/ML related hardware such as a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), etc., and this application does not specifically limit this. Optionally, the specific data set includes the input or output of the AI model.

In some embodiments, the identifier of the AI model described in this application may be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, or an identifier of a specific data set associated with the AI model described in this application, or an identifier of a specific scenario, environment, channel feature, or device related to the AI model described in this application, or an identifier of a function, feature, capability, or module related to the AI model described in this application. This application does not make any specific limitations on this.

It should be noted that the generalization ability of AI models is limited. A model trained based on data from one scenario may fail when applied to another scenario. Even a model trained based on data from the same scenario will fail over time. Failure refers to the reduction in reasoning accuracy of the AI model to the point where it cannot meet the target requirements. Therefore, the performance of the AI model needs to be supervised.

In some embodiments, the first AI model may be an activated model, or the first AI model may be an inactivated model. Specifically, when the first AI model is an inactivated model, after determining the validity of the first AI model, when performing AI model selection, an effective AI model may be preferentially selected from at least two AI models, thereby facilitating the selection of an AI model.

In some embodiments, the first information includes but is not limited to at least one of the following: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence. The embodiment of the present application clarifies the content of the first information, which is beneficial to realizing the performance supervision of the AI model.

Exemplarily, the amount of information contained in the first information may be one or at least two, that is, the first information may include but is not limited to at least one of the following: parameters of the probability density distribution of one or at least two second features, confidence intervals of one or at least two second features, values of one or at least two second features, values of one or at least two second features and their probabilities, and values of one or at least two second features and their confidence levels.

In some embodiments, the parameters of the probability density distribution include, but are not limited to, at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

For example, the type of probability density distribution may include but is not limited to at least one of the following: Gaussian distribution, Poisson distribution.

For example, for a Gaussian distribution, there is a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, such as a typical Gaussian distribution with a mean of μ and a standard deviation of σ.

For example, the 90% confidence interval is: [μ-1.645σ, μ+1.645σ], which means that there is a 90% probability that the predicted target value is within the interval [μ-1.645σ, μ+1.645σ].

For example, the 95% confidence interval is: [μ-1.96σ, μ+1.96σ], which means that there is a 95% probability that the predicted target value is within the interval [μ-1.96σ, μ+1.96σ].

Exemplarily, there may be a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, where the mean is μ, the standard deviation is σ, the coefficient z may be obtained by the probability density distribution function or the type of probability density distribution, and the confidence interval with a confidence level of p% may be as follows:
[μ-z _p% σ, μ+z _p% σ].

In some embodiments, the first information is output information of the first AI model, or the first information is information determined based on the output information of the first AI model.

Exemplarily, the output information of the first AI model is at least one of the following: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence. That is, the first information is the output information of the first AI model.

Exemplarily, the output information of the first AI model is the value of the second feature, and the first information can be determined based on the value of the second feature. Optionally, the first information is determined based on the value of the second feature and other information (such as motion state information (speed, acceleration, etc.), signal quality measurement information (such as RSRP, SINR, RSRQ, etc.)). For example, the worse the signal quality, the higher the uncertainty of the second feature obtained based on the value of the second feature and other information; for another example, the faster the movement speed, the higher the uncertainty of the second feature obtained based on the value of the second feature and other information.

For example, if the output information of the first AI model is the value of the second feature, at least one of the following can be determined based on the value of the second feature: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a probability of the value of the second feature, and a confidence level of the value of the second feature. That is, the first information is information determined based on the output information of the first AI model.

In some embodiments, the first AI model can be used to implement one of the following functions: positioning, beam management, channel state information (CSI) prediction, mobility management, and CSI compression. Of course, the first AI model can also be used to implement other functions, which is not limited in this application.

Specifically, the model performance supervision method 200 described in this embodiment can be used to implement at least two different functions, that is, the model performance supervision method 200 described in this embodiment can be applicable to a public label-based AI model supervision framework, thereby avoiding the need to design performance supervision solutions for different functions.

In some embodiments, when the first AI model is used to implement the positioning function, the first feature may include but is not limited to at least one of the following: time domain channel impulse response, RSRP (such as layer 1 RSRP or layer 3 RSRP), frequency domain channel impulse response, time domain waveform of the received signal. Optionally, the time domain channel impulse response includes at least one of the following: time information, power information, phase information. Optionally, the frequency domain channel impulse response includes at least one of the following: frequency information (subcarrier number and interval), power information, phase information.

In some embodiments, when the first AI model is used to implement the positioning function, the second feature may include but is not limited to at least one of the following: line-of-sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication.

For example, when the first AI model is used to realize the positioning function, the input of the first AI model (i.e., the first feature) is the time domain channel impulse response, and the output of the first AI model is the soft information of the intermediate feature quantity (i.e., the second feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.), and the intermediate feature quantity includes at least one of the following: line of sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc. The position coordinates can be further determined based on the soft information of the intermediate feature quantity; the output of the first AI model may also be the soft information of the position coordinates.

In some embodiments, when the first AI model is used to implement the beam management function, the first feature may include beam information at T1 historical moments, such as sequence number, angle, L1-RSRP, etc., and the second feature includes beam information at T2 future moments.

For example, when the first AI model is used to implement the beam management function, the input of the first AI model (i.e., the first feature) is the beam information at T1 historical moments, such as serial number, angle, L1-RSRP, etc., and the output of the first AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the beam information (i.e., the second feature) at T2 future moments. For example, the vertical beam and the horizontal beam at each moment correspond to a set of soft information, such as the confidence interval and probability density distribution of L1-RSRP, etc., and the beam information at each moment corresponds to a set of soft information.

In some embodiments, when the first AI model is used to implement the CSI prediction function, the first feature may include the CSI at T1 historical moments, and the second feature may include the CSI at T2 future moments.

For example, when the first AI model is used to implement the CSI prediction function, the input of the first AI model (i.e., the first feature) is the CSI at T1 historical moments, and the output of the first AI model is the soft information (such as parameters of the probability density distribution, confidence interval, confidence level, etc.) of the CSI at T2 moments in the future (i.e., the second feature), such as the CSI at each moment corresponds to a set of soft information, such as each dimension of the CSI at each moment corresponds to a set of soft information, or the CSI at each moment corresponds to a set of soft information.

In some embodiments, when the first AI model is used to implement the mobility management function, the first feature may include the layer 1 RSRP (L1-RSRP) or layer 3 RSRP (L3-RSRP) at the historical T1 moment, and the second feature may include the RSRP at the future T2 moments or the decision of whether a cell switching occurs at the future T2 moments. It should be noted that L3-RSRP is obtained by filtering L1-RSRP.

For example, when the first AI model is used to implement the mobility management function, the input of the first AI model (i.e., the first feature) is the L1-RSRP or L3-RSRP at T1 historical moments, and the output of the first AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the RSRP (i.e., the second feature) at T2 moments in the future; or the output of the first AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the decision on whether a cell switching will occur (i.e., the second feature) at T2 moments in the future, such as 1 for switching and 0 for no switching, and the soft information is a value between 0 and 1.

In some embodiments, when the first AI model is used to implement the CSI compression function, the first feature may include uncompressed CSI, and the second feature may include compressed CSI.

For example, when the first AI model is used to implement the CSI compression function, the input of the first AI model (ie, the first feature) is the uncompressed CSI, and the output of the first AI model is the soft information of the compressed CSI (ie, the second feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.).

In some embodiments, the label corresponding to the first feature (ie, the second information) may be real, or may be measured or estimated, and this is not limited in the embodiments of the present application.

It should be noted that the type of label corresponding to the first feature is consistent with the type of output of the first AI model. For example, both are location information, both are TOA, both are CSI, both are RSRP, etc.; this depends on the specific task type.

In some embodiments, the label corresponding to the first feature (ie, the second information) may be measured, estimated, or stored by the first device, or the label corresponding to the first feature (ie, the second information) may be obtained by the first device from the second device or other devices.

In some embodiments, in the above S210, the first device obtains the first information, including one of the following:

The first device receives first information from the second device;

The first device obtains first information through output information of the first AI model;

The first device receives output information of the first AI model from the second device, and obtains first information according to the output information of the first AI model.

In some embodiments, when the first device obtains the first information through output information of the first AI model, the first AI model can be deployed on the first device side.

In some embodiments, when the first device receives the first information from the second device, the first AI model can be deployed on the second device side. For example, the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device.

In some embodiments, when the first device receives output information of the first AI model from the second device and obtains the first information based on the output information of the first AI model, the first AI model can be deployed on the second device side.

In some embodiments, the first device determines the validity of the first AI model based on the third information. Specifically, after the first device determines the third information based on the first information and the second information, the first device determines the validity of the first AI model based on the third information. Optionally, the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model. For example, the first indication information occupies 1 bit; wherein a value of 0 indicates that the first AI model is valid, and a value of 1 indicates that the first AI model is invalid; or, a value of 1 indicates that the first AI model is valid, and a value of 0 indicates that the first AI model is invalid. Optionally, the first indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.

In some embodiments, the first device sends third information to the second device. Further, the second device may determine the validity of the first AI model based on the third information. Optionally, the first device receives second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model. For example, the second indication information occupies 1 bit; wherein a value of 0 indicates that the first AI model is valid, and a value of 1 indicates that the first AI model is invalid; or, a value of 1 indicates that the first AI model is valid, and a value of 0 indicates that the first AI model is invalid. Optionally, the second indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.

Exemplarily, when the first AI model is deployed on the first device side, the first device obtains first information through the output information of the first AI model, then the first device obtains second information from a local or other device (such as a second device or a device other than the first device and the second device), and the first device determines third information based on the first information and the second information, and then, the first device determines the validity of the first AI model based on the third information, and finally, the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.

Exemplarily, when the first AI model is deployed on the first device side, the first device obtains the first information through the output information of the first AI model, then the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information, and then the first device sends the third information to the second device, after which the second device determines the validity of the first AI model based on the third information, and finally, the second device sends the second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

Exemplarily, when the first AI model is deployed on the second device side, the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device. Then, the first device obtains the second information from the local or other devices (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device determines the validity of the first AI model based on the third information. Finally, the first device sends the first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.

Exemplarily, in the case where the first AI model is deployed on the second device side, the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device, then the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information, and then the first device sends the third information to the second device. Three information, then, the second device determines the validity of the first AI model according to the third information, and finally, the second device sends second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

Exemplarily, when the first AI model is deployed on the second device side, the first device receives the output information of the first AI model from the second device, and the first device obtains the first information based on the output information of the first AI model. Then, the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device determines the validity of the first AI model based on the third information. Finally, the first device sends the first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.

Exemplarily, when the first AI model is deployed on the second device side, the first device receives the output information of the first AI model from the second device, and the first device obtains the first information based on the output information of the first AI model. Then, the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device sends the third information to the second device, and then, the second device determines the validity of the first AI model based on the third information. Finally, the second device sends the second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

In some embodiments, the first device may be a terminal, a network side device, or a third-party server, wherein the network side device includes an access network device or a core network device.

In some embodiments, the second device may be a terminal, a network side device, or a third-party server, wherein the network side device includes an access network device or a core network device.

In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize the mean of the first probability of N samples of the first feature, or the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature; wherein the first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer. Optionally, the N samples can also be replaced by N inference processes of the first AI model.

Exemplarily, when the first information includes parameters of the probability density distribution of the second feature, the third information may be a log-likelihood value.

In some embodiments, when the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula 1:

Wherein, _L1 represents the third information, i represents the i-th sample among the N samples, Represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean (such as the statistical mean) of the probability density distribution of the second feature corresponding to the i-th sample, _yi represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

In some embodiments, when the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula 2:

Wherein, L ₂ represents the third information, i represents the i-th sample among the N samples, represents the mean (such as the statistical mean) of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, _yi represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

Optionally, s is agreed upon by a protocol, or s is determined by the first device, or s is configured or indicated by the second device.

In some embodiments, in the above formula 1 or formula 2, g(·) is related to the type of probability density distribution, for example, g(·) can be determined based on the type of probability density distribution.

For example, taking the Gaussian distribution as the type of probability density distribution, the above formula 1 can be transformed into the following formula 3.

For example, taking the Gaussian distribution as the type of probability density distribution, the above formula 2 can be transformed into the following formula 4.

In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to determine the validity of the first AI model, including:

When the third information is greater than or equal to the first threshold, the first AI model is valid; or,

When the third information is less than the first threshold, the first AI model fails.

Optionally, the first threshold is agreed upon by a protocol, or the first threshold is determined by the first device, or the first threshold is configured or indicated by the second device.

In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer. Optionally, the N samples can also be replaced by N inference processes of the first AI model.

In some embodiments, when the first information includes a confidence interval of the second feature, the third information is determined based on the following formula 5:

Wherein, L ₃ represents the third information, i represents the i-th sample in the N samples, x _i represents the input information corresponding to the i-th sample, y _i represents the label corresponding to the i-th sample, when y _i is within the confidence interval f(x _i ) of the second feature, otherwise

In some embodiments, when the first information includes a confidence interval of the second feature, the third information is used to determine the validity of the first AI model, including:

When the third information is greater than or equal to the second threshold, the first AI model is valid; or,

When the third information is less than the second threshold, the first AI model fails.

Optionally, the second threshold is agreed upon by a protocol, or the second threshold is determined by the first device, or the second threshold is configured or indicated by the second device.

In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the average of the weighted distances between the labels corresponding to the N samples of the first feature and the value of the second feature, where the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer. Optionally, the N samples can also be replaced by N inference processes of the first AI model.

In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is determined based on the following formula 6:

Wherein, L ₄ represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, and _yi represents the label corresponding to the i-th sample.

Exemplarily, in the above formula 6, the diagonal elements of Σ ^-1 are P powers of probability or confidence, where P is greater than or equal to 0 and P is an integer.

In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to determine the validity of the first AI model, including:

When the third information is less than or equal to the third threshold, the first AI model is valid; or,

When the third information is greater than a third threshold, the first AI model fails.

Optionally, the third threshold is agreed upon by a protocol, or the third threshold is determined by the first device, or the third threshold is configured or indicated by the second device.

It should be noted that in the above formula, _yi , It can be a specific numerical value, or a vector or a matrix, which is not limited in the embodiments of the present application.

In some embodiments, the N samples are acquired within M time units, where M is a positive integer.

Optionally, the time unit may include at least one of: orthogonal frequency-division multiplexing (OFDM) symbol, time slot, subframe, frame, microsecond, millisecond, second, minute, hour, day, week, month.

Optionally, M is configured or indicated by the second device, or M is agreed upon by a protocol.

In some embodiments, the scene identifiers or data set identifiers associated with the N samples are the same.

In some embodiments, the first device may obtain at least one of the following parameters from the second device:

The number of samples N used for AI model performance supervision;

The minimum number of samples N used for AI model performance supervision;

Get the time range T of N samples;

The type and parameters of the probability density distribution; for example, in the case of a Gaussian mixture model, the number of Gaussian distributions involved needs to be indicated;

Parameter P;

Parameter M.

In some embodiments, the first device reports at least one of the following parameters to the second device:

The number of samples N used for AI model performance supervision;

The minimum number of samples N used for AI model performance supervision;

Get the time range T of N samples;

The type and parameters of the probability density distribution; for example, in the case of a Gaussian mixture model, the number of Gaussian distributions involved needs to be indicated;

Parameter P;

Parameter M.

In some embodiments, the third information may be positive incentive information or negative incentive information; for example, when the negative incentive information is greater than a certain threshold or the positive incentive information is less than or equal to a certain threshold, the first AI model is considered to be invalid.

Exemplarily, the positive incentive information may include but is not limited to at least one of the following:

Among the N samples of the first feature, or the sample size or proportion;

The number or proportion of samples whose labels are within the confidence interval among the N samples of the first feature;

The number or proportion of samples for which the weighted distance between the label and the value of the second feature is less than or equal to t ₃ among the N samples of the first feature.

Exemplarily, the negative incentive information may include but is not limited to at least one of the following:

Among the N samples of the first feature, or the sample size or proportion;

The number or proportion of samples whose labels are outside the confidence interval among the N samples of the first feature;

Among the N samples of the first feature, the number or proportion of samples whose weighted distance between the label and the value of the second feature is greater than t ₃ .

Specifically, _t1 may be the first threshold, _t2 may be the second threshold, and _t3 may be the third threshold.

Therefore, in the embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model (i.e., the first information) and the label corresponding to the input feature of the first AI model (i.e., the second information), and the The validity of the first AI model is determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding labels can improve the accuracy of the performance supervision of the first AI model.

The following describes a solution for AI model positioning based on soft information through a specific example. Soft information can improve the robustness of the reasoning results of the AI model because the reasoning results cover multiple possible results and their probability distribution. It is conducive to further processing and utilization of the reasoning results of the AI model, such as combining soft information with other types of information (such as information output by other AI models used to implement positioning functions) to obtain more accurate target results, and it can also provide better security for some businesses that require relatively high reasoning reliability.

Example 1

The input of the ith AI model is the time-domain channel impulse response (CIR) of the ith transmission reception point (TRP), including the time, power, and phase information of the multipath.

The output of the ith AI model is the mean μ _i and standard deviation σ _i of the line-of-sight TOA between the ith TRP and the terminal;

The line-of-sight TOA estimate _xi between the ith TRP and the terminal is modeled as a Gaussian distribution:

For N TRPs, the likelihood function is modeled as:

Where x = [x ₁ , ..., x _N ] ^T ;

The maximum likelihood estimation problem can be transformed into a weighted least squares problem:

Where μ = [μ ₁ , ..., μ _N ] ^T , Σ is an N*N matrix, and its i-th diagonal element is Its goal is to find a location The N TOA estimates obtained by combining this position with the coordinates of the N TRPs are The weighted distance to the N TOA mean μ estimated by the AI model is the smallest.

because and The relationship between is nonlinear, so it can be solved by linear approximation and greedy algorithm. Taking the particle swarm optimization algorithm as an example, the specific implementation scheme and simulation results are given as shown in Figure 5.

Among them, different optimization algorithms, TRP numbers, and corresponding positioning accuracy can be shown in Table 1 below.

Table 1

In addition, the framework can support mixed positioning of multiple types of soft information. In the above example, only the soft information μ, σ of TOA of N TRPs is given. In addition, soft information of angles can also be included. Its likelihood function can be written as follows:

x, α, and z refer to different types of information, such as TOA, AOD, and AOA, respectively.

Secondly, the number of TRPs for different types of information can also be different. For example, the number of TRPs for x is N ₁ , the number of TRPs for α is N ₂ , and the number of TRPs for z is N ₃ . Their likelihood functions can be written as follows:

The above, in combination with Figures 4 to 5, describes in detail the first device side embodiment of the present application. The following, in combination with Figure 6, describes in detail the second device side embodiment of the present application. It should be understood that the second device side embodiment corresponds to the first device side embodiment, and similar descriptions can refer to the first device side embodiment.

FIG6 is a schematic flow chart of a model performance supervision method 300 according to an embodiment of the present application. As shown in FIG6 , the model performance supervision method 300 may include at least part of the following contents:

S310, the second device receives third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is the first feature, and the second information is a label corresponding to the first feature;

S320: The second device determines the validity of the first AI model based on the third information.

It should be understood that FIG6 shows the steps or operations of the model performance supervision method 300, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG6.

In some embodiments, the first information includes at least one of the following:

Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.

In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

In some embodiments, when the first information includes a parameter of a probability density distribution of the second feature, the third information is used to characterize a mean of the first probabilities of N samples of the first feature, or the third information is used to characterize a mean of the logarithms of the first probabilities of N samples of the first feature;

The first probability is the probability of the label corresponding to each of the N samples under the probability density distribution of the second feature, and N is a positive integer.

In some embodiments, when the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:

Wherein, _L1 represents the third information, i represents the i-th sample among the N samples, Represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, _yi represents the label corresponding to the i-th sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

In some embodiments, when the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:

Wherein, L ₂ represents the third information, i represents the i-th sample among the N samples, Indicates that the i-th sample The corresponding mean of the probability density distribution of the second feature, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, _yi represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the above S320 may specifically include:

When the third information is greater than or equal to the first threshold, the second device determines that the first AI model is valid; or,

When the third information is less than the first threshold, the second device determines that the first AI model is invalid.

In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.

In some embodiments, when the first information includes a confidence interval of the second feature, the third information is determined based on the following formula:

Wherein, L ₃ represents the third information, i represents the i-th sample in the N samples, x _i represents the input information corresponding to the i-th sample, y _i represents the label corresponding to the i-th sample, when y _i is within the confidence interval f(x _i ) of the second feature, otherwise

In some embodiments, when the first information includes the confidence interval of the second feature, the above S320 may specifically include:

When the third information is greater than or equal to the second threshold, the second device determines that the first AI model is valid; or,

When the third information is less than the second threshold, the second device determines that the first AI model is invalid.

In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is determined based on the following formula:

Wherein, L ₄ represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, and _yi represents the label corresponding to the i-th sample.

In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the above S320 may specifically include:

When the third information is less than or equal to the third threshold, the second device determines that the first AI model is valid; or,

When the third information is greater than the third threshold, the second device determines that the first AI model is invalid.

In some embodiments, before the second device receives the third information from the first device, the second device sends the first information to the first device.

In some embodiments, the second device sends second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model. Optionally, the second indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.

Therefore, in the embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model (i.e., the first information) and the label corresponding to the input feature of the first AI model (i.e., the second information), and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning result of the first AI model, by The reasoning results of the first AI model cover multiple possible results and their probability distribution, which is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain more accurate target results. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding labels can improve the accuracy of the performance supervision of the first AI model.

FIG. 7 is a schematic flow chart of a model performance supervision method 400 according to an embodiment of the present application. As shown in FIG. 7 , the model performance supervision method 400 may include at least part of the following contents:

S410: The first device obtains fourth information, where the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;

S420, the first device determines the validity of the second AI model according to the fourth information; or, the first device sends the fourth information to the second device.

It should be understood that FIG7 shows the steps or operations of the model performance supervision method 400, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG7.

In an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

Exemplarily, the judgment result of the effectiveness of the second AI model obtained based on the embodiment of the present application can also be combined with the judgment result of the effectiveness of the second AI model obtained by other model supervision methods to obtain the final conclusion on the effectiveness of the second AI model.

It should be noted that in the embodiment of the present application, the higher the uncertainty of the output of the second AI model, the lower the accuracy of the output of the second AI model.

Illustratively, the second AI model described in the embodiment of the present application may be the same as or different from the first AI model described above, and the present application is not limited to this.

In an embodiment of the present application, the fourth information may also be referred to as soft information (such as probability distribution, confidence level, confidence interval, etc.), wherein the soft information gives the probability distribution or confidence level of the possible results. Optionally, the second AI model may be a soft information AI model or may not be a soft information AI model. Among them, the soft information AI model refers to a type of AI model that outputs soft information (such as probability distribution, confidence level, confidence interval, etc.), including both classic probability models and AI models based on neural networks; the soft information AI model measures the possibility of different prediction results and gives the possibility, probability distribution or confidence level of each possible result.

It should be noted that compared with the hard information (hard value) (such as TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc.) obtained by AI model reasoning, the soft information obtained by AI model reasoning can significantly improve the reasoning accuracy and robustness. Specifically, soft information can better describe the uncertainty of the world, improve the robustness of model reasoning, and provide better security for some businesses that require relatively high reasoning reliability.

In some embodiments, the dimension of the third feature is D3, and correspondingly, the fourth information is D4 group of soft information of the fourth feature, wherein D4 group of soft information describes the range of the output of the second AI model. If there is only one group of soft information (ie, D4=1), then this group of soft information describes the uncertainty of the output of the entire second AI model; if there are at least two groups of soft information (ie, D4≥2), then each group of soft information describes the uncertainty of some features of the output of the second AI model.

For example, for prediction tasks: each prediction moment corresponds to a set of soft information, or each feature at each prediction moment corresponds to a set of soft information; for other tasks, each feature corresponds to a set of soft information, or all features correspond to a set of soft information.

For example, each feature corresponds to a set of soft information, and the output of the second AI model is 2-dimensional, corresponding to 2 sets of soft information.

For example, for the positioning task, the second AI model output is a 2-dimensional horizontal position coordinate, the 2-dimensional horizontal position coordinate corresponds to a set of soft information, or each dimension of the 2-dimensional horizontal position coordinate corresponds to a set of soft information.

In some embodiments, the AI model described in this application may also be referred to as an AI unit, an AI model/AI unit, an ML model, an ML unit, an AI structure, an AI function, an AI feature, a neural network, a neural network function, a neural network function, etc., or the AI model described in this application may also refer to a processing unit capable of implementing specific algorithms, formulas, processing procedures, capabilities, etc. related to AI, or the AI model described in this application may be a processing method, algorithm, function, module or unit for a specific data set, or the AI model described in this application may be a processing method, algorithm, function, module or unit running on AI/ML related hardware such as a GPU, NPU, TPU, ASIC, etc., which is not specifically limited in this application. Optionally, the specific data set includes the input or output of the AI model.

In some embodiments, the identifier of the AI model described in this application may be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, or an identifier of a specific data set associated with the AI model described in this application, or an identifier of a specific scenario, environment, channel feature, or device related to the AI model described in this application, or an identifier of a function, feature, capability, or module related to the AI model described in this application. This application does not make any specific limitations on this.

It should be noted that the generalization ability of AI models is limited. A model trained based on data from one scenario may fail when applied to another scenario. Even a model trained based on data from the same scenario will fail over time. Failure refers to the reduction in reasoning accuracy of the AI model to the point where it cannot meet the target requirements. Therefore, the performance of the AI model needs to be supervised.

In some embodiments, the second AI model may be an activated model, or the second AI model may be an inactivated model. Specifically, when the second AI model is an inactivated model, after determining the validity of the second AI model, when performing AI model selection, a valid AI model may be preferentially selected from at least two AI models, thereby facilitating the selection of an AI model.

In some embodiments, the fourth information includes but is not limited to at least one of the following: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence. The embodiment of the present application clarifies the content of the fourth information, which is beneficial to realizing the performance supervision of the AI model.

Exemplarily, the amount of information contained in the fourth information may be one or at least two, that is, the fourth information may include but is not limited to at least one of the following: parameters of the probability density distribution of one or at least two fourth features, confidence intervals of one or at least two fourth features, values of one or at least two fourth features, values of one or at least two fourth features and their probabilities, and values of one or at least two fourth features and their confidence levels.

In some embodiments, the parameters of the probability density distribution include, but are not limited to, at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

For example, the type of probability density distribution may include but is not limited to at least one of the following: Gaussian distribution, Poisson distribution.

For example, for a Gaussian distribution, there is a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, such as a typical Gaussian distribution with a mean of μ and a standard deviation of σ.

For example, the 90% confidence interval is: [μ-1.645σ, μ+1.645σ], which means that there is a 90% probability that the predicted target value is within the interval [μ-1.645σ, μ+1.645σ].

For example, the 95% confidence interval is: [μ-1.96σ, μ+1.96σ], which means that there is a 95% probability that the predicted target value is within the interval [μ-1.96σ, μ+1.96σ].

Exemplarily, there may be a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, where the mean is μ, the standard deviation is σ, the coefficient z may be obtained by the probability density distribution function or the type of probability density distribution, and the confidence interval with a confidence level of p% may be as follows:
[μ-z _p% σ, μ+z _p% σ].

In some embodiments, the fourth information is output information of the second AI model, or the fourth information is based on Information determined by the output information of the second AI model.

Exemplarily, the output information of the second AI model is at least one of the following: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence. That is, the fourth information is the output information of the second AI model.

Exemplarily, the output information of the second AI model is the value of the fourth feature, and the fourth information can be determined based on the value of the fourth feature, such as based on the value of the fourth feature and other information (such as motion state information (speed, acceleration, etc.), signal quality measurement information (such as RSRP, SINR, RSRQ, etc.)). For example, the worse the signal quality, the higher the uncertainty of the fourth feature obtained based on the value of the fourth feature and other information; for another example, the faster the movement speed, the higher the uncertainty of the fourth feature obtained based on the value of the fourth feature and other information.

For example, if the output information of the second AI model is the value of the fourth feature, at least one of the following can be determined based on the value of the fourth feature: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a probability of the value of the fourth feature, and a confidence level of the value of the fourth feature. That is, the fourth information is information determined based on the output information of the second AI model.

In some embodiments, the first AI model can be used to implement one of the following functions: positioning, beam management, CSI prediction, mobility management, CSI compression. Of course, the first AI model can also be used to implement other functions, which is not limited in this application.

Specifically, the model performance supervision method 400 described in this embodiment can be used to realize at least two different functions. That is, the model performance supervision method 400 described in this embodiment can be applicable to a public AI model supervision framework based on Ground truth label, thereby avoiding the need to design performance supervision solutions for different functions.

In some embodiments, when the second AI model is used to implement the positioning function, the third feature may include but is not limited to at least one of the following: time domain channel impulse response, RSRP (such as layer 1 RSRP or layer 3 RSRP), frequency domain channel impulse response, time domain waveform of the received signal. Optionally, the time domain channel impulse response includes at least one of the following: time information, power information, phase information. Optionally, the frequency domain channel impulse response includes at least one of the following: frequency information (subcarrier sequence number and interval), power information, phase information.

In some embodiments, when the second AI model is used to implement the positioning function, the fourth feature may include but is not limited to at least one of the following: line-of-sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication.

For example, when the second AI model is used to realize the positioning function, the input of the second AI model (i.e., the third feature) is the time domain channel impulse response, and the output of the second AI model is the soft information of the intermediate feature quantity (i.e., the fourth feature) (such as parameters of probability density distribution, confidence interval, confidence level, etc.), and the intermediate feature quantity includes at least one of the following: line of sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc. The position coordinates can be further determined based on the soft information of the intermediate feature quantity; the output of the second AI model may also be the soft information of the position coordinates.

In some embodiments, when the second AI model is used to implement the beam management function, the third feature may include beam information at T1 historical moments, such as sequence number, angle, L1-RSRP, etc., and the fourth feature includes beam information at T2 future moments.

For example, when the second AI model is used to implement the beam management function, the input of the second AI model (i.e., the third feature) is the beam information at T1 historical moments, such as serial number, angle, L1-RSRP, etc., and the output of the second AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the beam information (i.e., the fourth feature) at T2 future moments. For example, the vertical beam and the horizontal beam at each moment correspond to a set of soft information, such as the confidence interval and probability density distribution of L1-RSRP, etc.; the beam information at each moment corresponds to a set of soft information.

In some embodiments, when the second AI model is used to implement the CSI prediction function, the third feature may include the CSI at T1 historical moments, and the fourth feature may include the CSI at T2 future moments.

For example, when the second AI model is used to implement the CSI prediction function, the input of the second AI model (i.e., the third feature) is the CSI at T1 historical moments, and the output of the second AI model is the soft information (such as parameters of the probability density distribution, confidence interval, confidence level, etc.) of the CSI at T2 moments in the future (i.e., the fourth feature), such as the CSI at each moment corresponds to a set of soft information, such as each dimension of the CSI at each moment corresponds to a set of soft information, or the CSI at each moment corresponds to a set of soft information.

In some embodiments, when the second AI model is used to implement the mobility management function, the third feature may include the layer 1 RSRP (L1-RSRP) or layer 3 RSRP (L3-RSRP) at the historical T1 moment, and the fourth feature may include the RSRP at the future T2 moments or the decision of whether a cell switching occurs at the future T2 moments. It should be noted that L3-RSRP is obtained by filtering L1-RSRP.

For example, when the second AI model is used to implement the mobility management function, the input of the second AI model (i.e., the third feature) is the L1-RSRP or L3-RSRP at T1 historical moments, and the output of the second AI model is the soft information of the RSRP (i.e., the fourth feature) at T2 moments in the future (such as parameters of the probability density distribution, confidence interval, confidence level, etc.); or the output of the second AI model is the soft information of the decision on whether a cell switching will occur (i.e., the fourth feature) at T2 moments in the future (such as parameters of the probability density distribution, confidence interval, confidence level, etc.), such as 1 for switching and 0 for no switching, and the soft information is a value between 0 and 1.

In some embodiments, when the second AI model is used to implement the CSI compression function, the third feature may include uncompressed CSI, and the fourth feature may include compressed CSI.

For example, when the second AI model is used to implement the CSI compression function, the input of the second AI model (ie, the third feature) is the uncompressed CSI, and the output of the second AI model is the soft information of the compressed CSI (ie, the fourth feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.).

In some embodiments, in the above S410, the first device obtains the fourth information, including one of the following:

The first device obtains the fourth information from the second device;

The first device obtains the fourth information through the output information of the second AI model;

The first device receives the output information of the second AI model from the second device, and obtains the fourth information according to the output information of the second AI model.

In some embodiments, when the first device obtains the fourth information through the output information of the second AI model, the second AI model can be deployed on the first device side.

In some embodiments, when the first device obtains the fourth information from another device, the second AI model can be deployed on the other device side. The other device can be the second device or a device other than the first device and the second device.

In some embodiments, when the first device receives output information of the second AI model from the second device and obtains fourth information based on the output information of the second AI model, the second AI model can be deployed on the second device side.

In some embodiments, when the first device determines the validity of the second AI model according to the fourth information, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model. For example, the third indication information occupies 1 bit; wherein a value of 0 indicates that the second AI model is valid, and a value of 1 indicates that the second AI model is invalid; or, a value of 1 indicates that the second AI model is valid, and a value of 0 indicates that the second AI model is invalid.

Optionally, the third indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.

In some embodiments, when the first device sends the fourth information to the second device, the first device receives the fourth indication information from the second device, wherein the fourth indication information is used to indicate the validity of the second AI model. Specifically, after receiving the fourth information, the second device can determine the validity of the second AI model according to the fourth information. For example, the fourth indication information occupies 1 bit; wherein a value of 0 indicates that the second AI model is valid, and a value of 1 indicates that the second AI model is invalid; or, a value of 1 indicates that the second AI model is valid, and a value of 0 indicates that the second AI model is invalid.

Optionally, the fourth indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.

Exemplarily, when the second AI model is deployed on the first device side, the first device obtains the fourth information through the output information of the second AI model. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model.

Exemplarily, when the second AI model is deployed on the first device side, the first device obtains the fourth information through the output information of the second AI model, then the first device sends the fourth information to the second device, and then the second device receives the fourth information based on the output information of the second AI model. The validity of the second AI model is determined according to the fourth information. Finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

Exemplarily, when the second AI model is deployed on the second device side, the second device obtains the fourth information through the output information of the second AI model, and the second device sends the fourth information to the first device. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model.

Exemplarily, when the second AI model is deployed on the third device side, the third device obtains the fourth information through the output information of the second AI model, and the third device sends the fourth information to the first device, then the first device sends the fourth information to the second device, and then the second device determines the validity of the second AI model based on the fourth information, and finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

Exemplarily, when the second AI model is deployed on the second device side, the first device receives the output information of the second AI model from the second device, and the first device obtains fourth information based on the output information of the second AI model. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model.

Exemplarily, when the second AI model is deployed on the second device side, the first device receives the output information of the second AI model from the second device, and the first device obtains fourth information based on the output information of the second AI model. Then, the first device sends the fourth information to the second device. Thereafter, the second device determines the validity of the second AI model based on the fourth information. Finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

In some embodiments, the first device may be a terminal, a network-side device, or a third-party server.

In some embodiments, the second device may be a terminal, a network-side device, or a third-party server.

In some embodiments, when the fourth information includes a parameter of a probability density distribution of the fourth feature, the first device determines the validity of the second AI model according to the fourth information, including:

When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the first device determines that the second AI model fails; or,

When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the first device determines that the second AI model is invalid; or,

When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, the first device determines that the second AI model has failed;

Wherein, N is a positive integer.

Optionally, the N samples may also be replaced by N reasoning processes of the second AI model.

In some embodiments, the first threshold is agreed upon by a protocol, or the first threshold is determined by the first device, or the first threshold is configured or indicated by the second device.

In some embodiments, the second threshold is agreed upon by a protocol, or the second threshold is determined by the first device, or the second threshold is configured or indicated by the second device.

In some embodiments, the third threshold is agreed upon by a protocol, or the third threshold is determined by the first device, or the third threshold is configured or indicated by the second device.

In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the first device determines the validity of the second AI model according to the fourth information, including:

When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, the first device determines that the second AI model is invalid; or,

When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the fifth threshold, the first device determines that the second AI model is invalid; or,

Among the N samples of the third feature, the width of the corresponding confidence interval of the fourth feature is greater than or equal to the fourth When the ratio of the number of samples of the threshold to the total number of samples N is greater than or equal to a sixth threshold, the first device determines that the second AI model is invalid;

Wherein, N is a positive integer.

In some embodiments, the fourth threshold is agreed upon by a protocol, or the fourth threshold is determined by the first device, or the fourth threshold is configured or indicated by the second device.

In some embodiments, the fifth threshold is agreed upon by a protocol, or the fifth threshold is determined by the first device, or the fifth threshold is configured or indicated by the second device.

In some embodiments, the sixth threshold is agreed upon by a protocol, or the sixth threshold is determined by the first device, or the sixth threshold is configured or indicated by the second device.

In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the first device determines the validity of the second AI model according to the fourth information, including:

When the probability or confidence of the fourth feature is less than or equal to the seventh threshold, the first device determines that the second AI model fails; or,

When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to the eighth threshold, the first device determines that the second AI model fails; or,

When, among the N samples of the third feature, the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold accounts for a proportion of the total number of samples N that is greater than or equal to the ninth threshold, the first device determines that the second AI model has failed;

Wherein, N is a positive integer.

In some embodiments, the seventh threshold is agreed upon by a protocol, or the seventh threshold is determined by the first device, or the seventh threshold is configured or indicated by the second device.

In some embodiments, the eighth threshold is agreed upon by a protocol, or the eighth threshold is determined by the first device, or the eighth threshold is configured or indicated by the second device.

In some embodiments, the ninth threshold is agreed upon by a protocol, or the ninth threshold is determined by the first device, or the ninth threshold is configured or indicated by the second device.

Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

The above, in conjunction with Figure 7, describes in detail the first device side embodiment of the present application. The following, in conjunction with Figure 8, describes in detail the second device side embodiment of the present application. It should be understood that the second device side embodiment corresponds to the first device side embodiment, and similar descriptions can refer to the first device side embodiment.

FIG8 is a schematic flow chart of a model performance supervision method 500 according to an embodiment of the present application. As shown in FIG8 , the model performance supervision method 500 may include at least part of the following contents:

S510: The second device receives fourth information from the first device, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;

S520: The second device determines the validity of the second AI model according to the fourth information.

It should be understood that FIG8 shows the steps or operations of the model performance supervision method 500, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG8.

In some embodiments, the fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

In some embodiments, when the fourth information includes a parameter of a probability density distribution of the fourth feature, the second device determines the validity of the second AI model according to the fourth information, including:

When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the second device determines that the second AI model fails; or,

When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the second device determines that the second AI model is invalid; or,

When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, the second device determines that the second AI model has failed;

Wherein, N is a positive integer.

In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the second device determines the validity of the second AI model according to the fourth information, including:

When the width of the confidence interval of the fourth feature is greater than or equal to the fourth threshold, the second device determines that the second AI model is invalid; or,

When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the fifth threshold, the second device determines that the second AI model is invalid; or,

When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to a sixth threshold, the second device determines that the second AI model is invalid;

Wherein, N is a positive integer.

In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the second device determines the validity of the second AI model according to the fourth information, including:

When the probability or confidence of the fourth feature is less than or equal to the seventh threshold, the second device determines that the second AI model fails; or,

When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to the eighth threshold, the second device determines that the second AI model fails; or,

When, among the N samples of the third feature, the proportion of the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold to the total number of samples N is greater than or equal to the ninth threshold, the second device determines that the second AI model is invalid;

Wherein, N is a positive integer.

In some embodiments, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

Optionally, the fourth indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.

Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

The model performance supervision method provided in the embodiment of the present application can be executed by a model performance supervision device, or a processing unit in the model performance supervision device for executing the model performance supervision method. Taking the execution of a model performance supervision method by a supervision device as an example, the model performance supervision device provided in an embodiment of the present application is explained.

Fig. 9 shows a schematic block diagram of a model performance monitoring device 600 according to an embodiment of the present application. As shown in Fig. 9, the model performance monitoring device 600 includes:

An acquisition unit 610 is used to acquire first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first artificial intelligence AI model, and the input of the first AI model is a first feature;

Processing unit 620 is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

In some embodiments, the acquiring unit 610 acquires the first information, including one of the following:

receiving the first information from a second device;

Obtaining the first information through output information of the first AI model;

Receive output information of the first AI model from a second device, and obtain the first information according to the output information of the first AI model.

In some embodiments, the model performance monitoring device 600 further includes: a transceiver unit 630;

The processing unit 620 is further configured to determine the validity of the first AI model according to the third information; or,

The transceiver unit 630 is used to send the third information to the second device.

In some embodiments, when the model performance monitoring device 600 determines the validity of the first AI model according to the third information, the transceiver unit 630 is further used to send first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model; or,

When the model performance monitoring device 600 sends the third information to the second device, the transceiver unit 630 is also used to receive second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model.

In some embodiments, the first information includes at least one of the following:

Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.

In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize the mean of the first probabilities of N samples of the first feature, or the third information is used to characterize the mean of the logarithms of the first probabilities of N samples of the first feature;

The first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer.

In some embodiments, when the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:

Wherein, _L1 represents the third information, i represents the i-th sample among the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the ith sample, _yi represents the label corresponding to the ith sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

In some embodiments, when the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:

Wherein, L ₂ represents the third information, i represents the i-th sample among the N samples, Indicates the i-th The mean of the probability density distribution of the second feature corresponding to samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, _yi represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

In some embodiments, the third information is used to determine the validity of the first AI model, including:

When the third information is greater than or equal to the first threshold, the first AI model is valid; or,

When the third information is less than the first threshold, the first AI model fails.

In some embodiments, when the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.

In some embodiments, the third information is determined based on the following formula:

Wherein, L ₃ represents the third information, i represents the i-th sample among the N samples, x _i represents the input information corresponding to the i-th sample, y _i represents the label corresponding to the i-th sample, and when y _i is within the confidence interval f(x _i ) of the second feature, otherwise

In some embodiments, the third information is used to determine the validity of the first AI model, including:

When the third information is greater than or equal to the second threshold, the first AI model is valid; or,

When the third information is less than the second threshold, the first AI model fails.

In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

In some embodiments, the third information is determined based on the following formula:

Wherein, L ₄ represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, and _yi represents the label corresponding to the i-th sample.

In some embodiments, the third information is used to determine the validity of the first AI model, including:

When the third information is less than or equal to a third threshold, the first AI model is valid; or,

When the third information is greater than a third threshold, the first AI model fails.

In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input/output interface of a communication chip or a system on chip.

It should be understood that the model performance monitoring device 600 according to the embodiment of the present application may correspond to the first device in the method embodiment of the present application, and the above-mentioned and other operations or functions of each unit in the model performance monitoring device 600 are respectively for realizing the corresponding process of the first device in the method 200 shown in Figure 4. For the sake of brevity, they will not be repeated here.

Therefore, in an embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, so as to achieve performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.

Fig. 10 shows a schematic block diagram of a model performance monitoring device 700 according to an embodiment of the present application. As shown in Fig. 10, the model performance monitoring device 700 includes:

The transceiver unit 710 is configured to receive third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence AI model, the input of the first AI model is the first feature, and the second information is the label corresponding to the first feature;

The processing unit 720 is used to determine the validity of the first AI model according to the third information.

In some embodiments, the processing unit 720 is specifically configured to:

When the third information is less than or equal to a third threshold, determining that the first AI model is valid; or,

When the third information is greater than a third threshold, it is determined that the first AI model is invalid.

In some embodiments, before the model performance monitoring apparatus 700 receives the third information from the first device, the transceiver unit 710 is further configured to send the first information to the first device.

In some embodiments, the transceiver unit 710 is further used to send second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.

In some embodiments, the first information includes at least one of the following:

Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.

In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

In some embodiments, when the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize the mean of the first probabilities of N samples of the first feature, or the third information is used to characterize the mean of the logarithms of the first probabilities of N samples of the first feature;

The first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer.

In some embodiments, when the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:

Wherein, _L1 represents the third information, i represents the i-th sample among the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the ith sample, _yi represents the label corresponding to the ith sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

In some embodiments, when the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:

Wherein, L ₂ represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, _yi represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model.

In some embodiments, the processing unit 720 is specifically configured to:

When the third information is greater than or equal to the first threshold, determining that the first AI model is valid; or,

When the third information is less than the first threshold, it is determined that the first AI model is invalid.

In some embodiments, when the first information includes a confidence interval of the second feature, the first The third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.

In some embodiments, the third information is determined based on the following formula:

Wherein, L ₃ represents the third information, i represents the i-th sample among the N samples, x _i represents the input information corresponding to the i-th sample, y _i represents the label corresponding to the i-th sample, and when y _i is within the confidence interval f(x _i ) of the second feature, otherwise

In some embodiments, the processing unit 720 is specifically configured to:

When the third information is greater than or equal to the second threshold, determining that the first AI model is valid; or,

When the third information is less than a second threshold, it is determined that the first AI model is invalid.

In some embodiments, when the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.

In some embodiments, the third information is determined based on the following formula:

Wherein, L ₄ represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, and _yi represents the label corresponding to the i-th sample.

In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input/output interface of a communication chip or a system on chip.

It should be understood that the model performance monitoring device 700 according to the embodiment of the present application may correspond to the second device in the method embodiment of the present application, and the above-mentioned and other operations or functions of each unit in the model performance monitoring device 700 are respectively for implementing the corresponding processes of the second device in the method 300 shown in Figure 6. For the sake of brevity, they will not be repeated here.

Therefore, in an embodiment of the present application, the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, so as to achieve performance supervision of the first AI model. Specifically, the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the first AI model, referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.

Fig. 11 shows a schematic block diagram of a model performance monitoring device 800 according to an embodiment of the present application. As shown in Fig. 11, the model performance monitoring device 800 includes:

An acquisition unit 810 is used to acquire fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature;

The processing unit 820 is used to determine the validity of the second AI model according to the fourth information; or the transceiver unit 830 is used to send the fourth information to the second device.

In some embodiments, the acquiring unit 810 acquires the fourth information, including one of the following:

Acquire the fourth information from other devices;

Obtaining the fourth information through the output information of the second AI model;

receiving output information of the second AI model from the second device, and information, and obtain the fourth information.

In some embodiments, when the model performance monitoring device 800 determines the validity of the second AI model according to the fourth information, the transceiver unit 830 is further used to send third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model; or,

When the model performance monitoring device 800 sends the fourth information to the second device, the transceiver unit 830 is also used to receive fourth indication information from the second device, wherein the fourth indication information is used to indicate the validity of the second AI model.

In some embodiments, the fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

In some embodiments, when the fourth information includes parameters of the probability density distribution of the fourth feature, the processing unit 820 is specifically configured to:

When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, determining that the second AI model is invalid; or,

When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, determining that the second AI model is invalid; or,

When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, it is determined that the second AI model is invalid;

Wherein, N is a positive integer.

In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the processing unit 820 is specifically configured to:

When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, determining that the second AI model is invalid; or,

When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, determining that the second AI model is invalid; or,

When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to the sixth threshold, it is determined that the second AI model is invalid;

Wherein, N is a positive integer.

In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the processing unit 820 is specifically configured to:

When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, determining that the second AI model fails; or,

When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, determining that the second AI model is invalid; or,

When, among the N samples of the third feature, the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold accounts for a proportion of the total number of samples N that is greater than or equal to the ninth threshold, it is determined that the second AI model has failed;

Wherein, N is a positive integer.

In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input/output interface of a communication chip or a system on chip.

It should be understood that the model performance monitoring device 800 according to the embodiment of the present application may correspond to the first device in the method embodiment of the present application, and the above and other operations or functions of each unit in the model performance monitoring device 800 are respectively In order to implement the corresponding process of the first device in the method 400 shown in FIG. 7 , it will not be repeated here for the sake of brevity.

Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

Fig. 12 shows a schematic block diagram of a model performance monitoring device 900 according to an embodiment of the present application. As shown in Fig. 12, the model performance monitoring device 900 includes:

The transceiver unit 910 is configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature;

The processing unit 920 is used to determine the validity of the second AI model according to the fourth information.

In some embodiments, the transceiver unit 910 is further used to send fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.

In some embodiments, the fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.

In some embodiments, the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.

In some embodiments, when the fourth information includes parameters of the probability density distribution of the fourth feature, the processing unit 920 is specifically configured to:

When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, determining that the second AI model is invalid; or,

When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, determining that the second AI model is invalid; or,

When, among the N samples of the third feature, the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold accounts for a proportion of the total number of samples N that is greater than or equal to the third threshold, it is determined that the second AI model is invalid;

Wherein, N is a positive integer.

In some embodiments, when the fourth information includes a confidence interval of the fourth feature, the processing unit 920 is specifically configured to:

When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, determining that the second AI model is invalid; or,

When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, determining that the second AI model is invalid; or,

When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to the sixth threshold, it is determined that the second AI model is invalid;

Wherein, N is a positive integer.

In some embodiments, when the fourth information includes the value of the fourth feature and its probability or confidence, the processing unit 920 is specifically configured to:

When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, determining that the second AI model fails; or,

The mean of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to In the case of the eighth threshold, it is determined that the second AI model fails; or,

When, among the N samples of the third feature, the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold accounts for a proportion of the total number of samples N that is greater than or equal to the ninth threshold, it is determined that the second AI model has failed;

Wherein, N is a positive integer.

In some embodiments, the transceiver unit may be a communication interface or a transceiver, or an input/output interface of a communication chip or a system on chip.

It should be understood that the model performance monitoring device 900 according to the embodiment of the present application may correspond to the second device in the method embodiment of the present application, and the above-mentioned and other operations or functions of each unit in the model performance monitoring device 900 are respectively for implementing the corresponding processes of the second device in the method 500 shown in Figure 8. For the sake of brevity, they will not be repeated here.

Therefore, in an embodiment of the present application, the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model. Specifically, the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result. In the performance supervision of the second AI model, the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.

The model performance monitoring device in the embodiment of the present application can be an electronic device, such as an electronic device with an operating system, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or a network-side device, or can be a device other than a terminal or a network-side device. Exemplarily, the terminal can include but is not limited to the types of terminals 11 listed above, the network-side device can include but is not limited to the types of network-side devices 12 listed above, and other devices can be servers, network attached storage (NAS), etc., which are not specifically limited in the embodiment of the present application.

The model performance monitoring device provided in the embodiment of the present application can implement the various processes implemented by the method embodiments of Figures 4 to 8 and achieve the same technical effect. To avoid repetition, it will not be repeated here.

As shown in FIG13, the embodiment of the present application further provides a communication device 1000, including a processor 1001 and a memory 1002, and the memory 1002 stores a program or instruction that can be run on the processor 1001. For example, when the communication device 1000 is a first device, the program or instruction is executed by the processor 1001 to implement the various steps of the above-mentioned model performance supervision method 200 or model performance supervision method 400 embodiment, and can achieve the same technical effect. When the communication device 1000 is a second device, the program or instruction is executed by the processor 1001 to implement the various steps of the above-mentioned model performance supervision method 300 or model performance supervision method 500 embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

The embodiment of the present application also provides a terminal, including a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps performed by the first device or the second device in the method embodiment shown in Figures 4 to 8. This terminal embodiment corresponds to the above-mentioned first device or second device side method embodiment, and each implementation process and implementation method of the above-mentioned method embodiment can be applied to the terminal embodiment and can achieve the same technical effect. Specifically, Figure 14 is a schematic diagram of the hardware structure of a terminal implementing an embodiment of the present application.

The terminal 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109 and at least some of the components of a processor 1110.

Those skilled in the art will appreciate that the terminal 1100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1110 through a power management system, so as to implement functions such as managing charging, discharging, and power consumption management through the power management system. The terminal structure shown in FIG14 does not constitute a limitation on the terminal, and the terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently, which will not be described in detail here.

It should be understood that in the embodiment of the present application, the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072. The touch panel 11071 is also called a touch screen. The touch panel 11071 may include two parts: a touch detection device and a touch controller. Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

In the embodiment of the present application, after receiving downlink data from the network side device, the RF unit 1101 can transmit the data to the processor 1110 for processing; in addition, the RF unit 1101 can send uplink data to the network side device. Generally, the RF unit 1101 includes but is not limited to an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.

The memory 1109 can be used to store software programs or instructions and various data. The memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1109 may include a volatile memory or a non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

The processor 1110 may include at least one processing unit; optionally, the processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1110.

Exemplarily, the RF unit 1101 is used to obtain first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first AI model, and the input of the first AI model is a first feature; the processor 1110 is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.

Exemplarily, the radio frequency unit 1101 is used to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; the processor 1110 is used to determine the validity of the first AI model based on the third information.

Exemplarily, the RF unit 1101 is used to obtain fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; the processor 1110 is used to determine the validity of the first AI model based on the fourth information; or, the RF unit 1101 is used to send the fourth information to the second device.

Exemplarily, the radio frequency unit 1101 is used to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; the processor 1110 is used to determine the validity of the first AI model based on the fourth information.

It can be understood that the implementation process of each implementation method mentioned in this embodiment can refer to the relevant description of the method embodiment and achieve the same or corresponding technical effect. To avoid repetition, it will not be repeated here.

The embodiment of the present application also provides a network side device, including a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method embodiment shown in Figures 4 to 8. The network side device embodiment corresponds to the first device or second device method embodiment described above, and each implementation process and implementation method of the method embodiment described above can be applied to the network side device embodiment, and can achieve the same technical effect.

Specifically, the embodiment of the present application also provides a network side device. As shown in Figure 15, the network side device 1200 includes: an antenna 121, a radio frequency device 122, a baseband device 123, a processor 124 and a memory 125. The antenna 121 is connected to the radio frequency device 122. In the uplink direction, the radio frequency device 122 receives information through the antenna 121 and sends the received information to the baseband device 123 for processing. In the downlink direction, the baseband device 123 processes the information to be sent and sends it to the radio frequency device 122. The radio frequency device 122 processes the received information and sends it out through the antenna 121.

The method executed by the network-side device in the above embodiment may be implemented in the baseband device 123, which includes a baseband processor.

The baseband device 123 may include, for example, at least one baseband board, on which at least two chips are arranged, as shown in Figure 15, one of the chips is, for example, a baseband processor, which is connected to the memory 125 through a bus interface to call the program in the memory 125 to execute the network device operations shown in the above method embodiment.

The network side device may also include a network interface 126, which is, for example, a Common Public Radio Interface (CPRI).

Specifically, the network side device 1200 of the embodiment of the present application also includes: instructions or programs stored in the memory 125 and executable on the processor 124. The processor 124 calls the instructions or programs in the memory 125 to execute the method executed by each unit shown in any one of Figures 9 to 12, and achieves the same technical effect. To avoid repetition, it will not be repeated here.

Specifically, the embodiment of the present application further provides a network side device. As shown in FIG16 , the network side device 1300 includes: a processor 1301, a network interface 1302, and a memory 1303. Among them, the network interface 1302 is, for example, a common public radio interface (CPRI).

Specifically, the network side device 1300 of the embodiment of the present application also includes: instructions or programs stored in the memory 1303 and executable on the processor 1301. The processor 1301 calls the instructions or programs in the memory 1303 to execute the method executed by each unit shown in any one of Figures 9 to 12, and achieves the same technical effect. To avoid repetition, it will not be repeated here.

An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned model performance supervision method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

The processor is the processor in the terminal described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk. In some examples, the readable storage medium may be a non-transient readable storage medium.

An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model performance supervision method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

An embodiment of the present application further provides a computer program/program product, which is stored in a storage medium. The computer program/program product is executed by at least one processor to implement the various processes of the above-mentioned model performance supervision method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

An embodiment of the present application also provides a communication system, including: a first device and a second device, wherein the first device can be used to execute the steps performed by the first device in the model performance supervision method as described above, and the second device can be used to execute the steps performed by the second device in the model performance supervision method as described above.

It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted or combined. In addition, the features described with reference to certain examples may be combined in other examples.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of a computer software product plus a necessary general hardware platform, and of course, can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, disk, CD, etc.), including several instructions to enable a terminal or a network-side device to execute the methods described in each embodiment of the present application.

The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms of implementation methods without departing from the purpose of the present application and the scope of protection of the claims, and these implementation methods are all within the protection of the present application.

Claims (53)

A model performance supervision method, comprising: The first device acquires first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first artificial intelligence (AI) model, and the input of the first AI model is a first feature; The first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model. The method according to claim 1, wherein The first device obtains first information, including one of the following: The first device receives the first information from the second device; The first device obtains the first information through output information of the first AI model; The first device receives output information of the first AI model from the second device, and obtains the first information according to the output information of the first AI model. The method according to claim 1 or 2, further comprising: The first device determines the validity of the first AI model according to the third information; or, The first device sends the third information to the second device. The method according to claim 3, wherein the method further comprises: When the first device determines the validity of the first AI model according to the third information, the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model; or When the first device sends the third information to the second device, the first device receives second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model. The method according to any one of claims 1 to 4, wherein the first information includes at least one of the following: Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level. The method according to claim 5, wherein The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution. The method according to claim 5 or 6, wherein: In the case where the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize a mean of the first probabilities of N samples of the first feature, or the third information is used to characterize a mean of the logarithms of the first probabilities of the N samples of the first feature; The first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer. The method according to claim 7, wherein: In the case where the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:
Wherein, _L1 represents the third information, i represents the i-th sample among the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the ith sample, _yi represents the label corresponding to the ith sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model. The method according to claim 7, wherein: In the case where the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:
Wherein, L ₂ represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, _yi represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model. The method according to any one of claims 7 to 9, wherein The third information is used to determine the validity of the first AI model, including: When the third information is greater than or equal to the first threshold, the first AI model is valid; or, When the third information is less than the first threshold, the first AI model fails. The method according to claim 5 or 6, wherein: In the case where the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer. The method according to claim 11, wherein The third information is determined based on the following formula:
Wherein, L ₃ represents the third information, i represents the i-th sample among the N samples, x _i represents the input information corresponding to the i-th sample, y _i represents the label corresponding to the i-th sample, and when y _i is within the confidence interval f(x _i ) of the second feature, otherwise The method according to claim 11 or 12, wherein The third information is used to determine the validity of the first AI model, including: When the third information is greater than or equal to the second threshold, the first AI model is valid; or, When the third information is less than the second threshold, the first AI model fails. The method according to claim 5 or 6, wherein: When the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer. The method according to claim 14, wherein The third information is determined based on the following formula:
Wherein, L ₄ represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, and _yi represents the label corresponding to the i-th sample. The method according to claim 14 or 15, wherein The third information is used to determine the validity of the first AI model, including: When the third information is less than or equal to a third threshold, the first AI model is valid; or, When the third information is greater than a third threshold, the first AI model fails. A model performance supervision method, comprising: The second device receives third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; The second device determines the validity of the first AI model based on the third information. The method according to claim 17, wherein before the second device receives the third information from the first device, the method further comprises: The second device sends the first information to the first device. The method according to claim 17 or 18, wherein the method further comprises: The second device sends second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model. The method according to any one of claims 17 to 19, wherein the first information includes at least one of the following: Parameters of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level. The method according to claim 20, wherein The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution. The method according to claim 20 or 21, wherein In the case where the first information includes parameters of the probability density distribution of the second feature, the third information is used to characterize a mean of the first probabilities of N samples of the first feature, or the third information is used to characterize a mean of the logarithms of the first probabilities of the N samples of the first feature; The first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer. The method according to claim 22, wherein In the case where the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:
Wherein, _L1 represents the third information, i represents the i-th sample among the N samples, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, represents the mean of the probability density distribution of the second feature corresponding to the ith sample, _yi represents the label corresponding to the ith sample, and g(·) represents the probability density distribution function obeyed by the output of the first AI model. The method according to claim 22, wherein In the case where the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:
Wherein, L ₂ represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample, _yi represents the label corresponding to the i-th sample, s is a positive integer, and g(·) represents the probability density distribution function obeyed by the output of the first AI model. The method according to any one of claims 22 to 24, wherein The second device determines the validity of the first AI model according to the third information, including: When the third information is greater than or equal to the first threshold, the second device determines that the first AI model is valid; or, When the third information is less than the first threshold, the second device determines that the first AI model is invalid. The method according to claim 20 or 21, wherein In the case where the first information includes the confidence interval of the second feature, the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer. The method according to claim 26, wherein The third information is determined based on the following formula:
Wherein, L ₃ represents the third information, i represents the i-th sample among the N samples, x _i represents the input information corresponding to the i-th sample, y _i represents the label corresponding to the i-th sample, and when y _i is within the confidence interval f(x _i ) of the second feature, otherwise The method according to claim 26 or 27, wherein The second device determines the validity of the first AI model according to the third information, including: When the third information is greater than or equal to a second threshold, the second device determines that the first AI model is valid; or, When the third information is less than a second threshold, the second device determines that the first AI model is invalid. The method according to claim 20 or 21, wherein When the first information includes the value of the second feature and its probability or confidence, the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer. The method according to claim 29, wherein The third information is determined based on the following formula:
Wherein, L ₄ represents the third information, i represents the i-th sample among the N samples, represents the mean of the probability density distribution of the second feature corresponding to the i-th sample, and _yi represents the label corresponding to the i-th sample. The method according to claim 29 or 30, wherein The second device determines the validity of the first AI model according to the third information, including: When the third information is less than or equal to a third threshold, the second device determines that the first AI model is valid; or, When the third information is greater than a third threshold, the second device determines that the first AI model is invalid. A model performance supervision method, comprising: The first device obtains fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; The first device determines the validity of the second AI model based on the fourth information; or, the first device sends the fourth information to the second device. The method according to claim 32, wherein The first device obtains fourth information, including one of the following: The first device obtains the fourth information from other devices; The first device obtains the fourth information through the output information of the second AI model; The first device receives the output information of the second AI model from the second device, and obtains the fourth information according to the output information of the second AI model. The method according to claim 32 or 33, further comprising: When the first device determines the validity of the second AI model according to the fourth information, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model; or When the first device sends the fourth information to the second device, the first device receives fourth indication information from the second device, wherein the fourth indication information is used to indicate the validity of the second AI model. The method according to any one of claims 32 to 34, wherein The fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence. The method according to claim 35, wherein The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution. The method according to claim 35 or 36, wherein In a case where the fourth information includes a parameter of a probability density distribution of the fourth feature, the first device determines the validity of the second AI model according to the fourth information, including: When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the first device determines that the second AI model fails; or, When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the first device determines that the second AI model fails; or, When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, the first device determines that the second AI model has failed; Wherein, N is a positive integer. The method according to claim 35 or 36, wherein In a case where the fourth information includes a confidence interval of the fourth feature, the first device determines the validity of the second AI model according to the fourth information, including: When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, the first device determines that the second AI model is invalid; or, When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, the first device determines that the second AI model is invalid; or, When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to a sixth threshold, the first device determines that the second AI model has failed; Wherein, N is a positive integer. The method according to claim 35 or 36, wherein In a case where the fourth information includes a value of the fourth feature and a probability or confidence level thereof, the first device determines the validity of the second AI model according to the fourth information, including: When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, the first device determines that the second AI model fails; or, When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, the first device determines that the second AI model fails; or, Among the N samples of the third feature, the probability or confidence of the corresponding fourth feature is less than or equal to When the ratio of the number of samples of the seventh threshold to the total number of samples N is greater than or equal to the ninth threshold, the first device determines that the second AI model is invalid; Wherein, N is a positive integer. A model performance supervision method, comprising: The second device receives fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence (AI) model, and the input of the second AI model is the third feature; The second device determines the validity of the second AI model based on the fourth information. The method according to claim 40, wherein the method further comprises: The second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model. The method according to claim 40 or 41, wherein The fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence. The method according to claim 42, wherein The parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution. The method according to claim 42 or 43, wherein In a case where the fourth information includes a parameter of a probability density distribution of the fourth feature, the second device determining the validity of the second AI model according to the fourth information includes: When the variance or standard deviation of the probability density distribution of the fourth feature is greater than or equal to the first threshold, the second device determines that the second AI model fails; or, When the mean of the variance or standard deviation of the probability density distribution of the fourth feature corresponding to the N samples of the third feature is greater than or equal to the second threshold, the second device determines that the second AI model is invalid; or, When, among the N samples of the third feature, the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, the second device determines that the second AI model has failed; Wherein, N is a positive integer. The method according to claim 42 or 43, wherein In a case where the fourth information includes a confidence interval of the fourth feature, the second device determines the validity of the second AI model according to the fourth information, including: When the width of the confidence interval of the fourth feature is greater than or equal to a fourth threshold, the second device determines that the second AI model is invalid; or, When the average width of the confidence interval of the fourth feature corresponding to the N samples of the third feature is greater than or equal to a fifth threshold, the second device determines that the second AI model is invalid; or, When, among the N samples of the third feature, the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to a sixth threshold, the second device determines that the second AI model has failed; Wherein, N is a positive integer. The method according to claim 42 or 43, wherein In a case where the fourth information includes the value of the fourth feature and the probability or confidence thereof, the second device determines the validity of the second AI model according to the fourth information, including: When the probability or confidence of the fourth feature is less than or equal to a seventh threshold, the second device determines that the second AI model fails; or, When the average of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to an eighth threshold, the second device determines that the second AI model fails; or, Among the N samples of the third feature, the probability or confidence of the corresponding fourth feature is less than or equal to When the ratio of the number of samples of the seventh threshold to the total number of samples N is greater than or equal to the ninth threshold, the second device determines that the second AI model is invalid; Wherein, N is a positive integer. A model performance monitoring device, comprising: An acquisition unit, configured to acquire first information, wherein the first information is used to characterize the uncertainty or probability distribution of an output of a first artificial intelligence (AI) model, and an input of the first AI model is a first feature; A processing unit, used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model. A model performance monitoring device, comprising: a transceiver unit, configured to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence (AI) model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; A processing unit is used to determine the validity of the first AI model based on the third information. A model performance monitoring device, comprising: an acquisition unit, configured to acquire fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence (AI) model, the input of the second AI model being the third feature; A processing unit, used to determine the validity of the second AI model according to the fourth information; or a transceiver unit, used to send the fourth information to the second device. A model performance monitoring device, comprising: A transceiver unit, configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence (AI) model, and the input of the second AI model is the third feature; A processing unit, configured to determine the validity of the second AI model based on the fourth information. A model performance supervision device, the model performance supervision device is a first device, the model performance supervision device includes a processor and a memory, the memory stores a program or instruction that can be run on the processor, the program or instruction when executed by the processor implements the steps of the model performance supervision method as described in any one of claims 1 to 16, or the program or instruction when executed by the processor implements the steps of the model performance supervision method as described in any one of claims 32 to 39. A model performance supervision device, the model performance supervision device is a second device, the model performance supervision device includes a processor and a memory, the memory stores a program or instruction that can be run on the processor, and the program or instruction when executed by the processor implements the steps of the model performance supervision method as described in any one of claims 17 to 31, or the program or instruction when executed by the processor implements the steps of the model performance supervision method as described in any one of claims 40 to 46. A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the method of the model performance supervision method as described in any one of claims 1 to 16, or implements the steps of the model performance supervision method as described in any one of claims 17 to 31, or implements the steps of the model performance supervision method as described in any one of claims 32 to 39, or implements the steps of the model performance supervision method as described in any one of claims 40 to 46.