[go: up one dir, main page]

WO2025092999A1 - Procédé et appareil de supervision de performance de modèle, et dispositif - Google Patents

Procédé et appareil de supervision de performance de modèle, et dispositif Download PDF

Info

Publication number
WO2025092999A1
WO2025092999A1 PCT/CN2024/129460 CN2024129460W WO2025092999A1 WO 2025092999 A1 WO2025092999 A1 WO 2025092999A1 CN 2024129460 W CN2024129460 W CN 2024129460W WO 2025092999 A1 WO2025092999 A1 WO 2025092999A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
model
feature
samples
probability
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/CN2024/129460
Other languages
English (en)
Chinese (zh)
Inventor
贾承璐
邬华明
王园园
吴昊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Vivo Mobile Communication Co Ltd
Original Assignee
Vivo Mobile Communication Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Vivo Mobile Communication Co Ltd filed Critical Vivo Mobile Communication Co Ltd
Publication of WO2025092999A1 publication Critical patent/WO2025092999A1/fr
Pending legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N7/00—Computing arrangements based on specific mathematical models
    • G06N7/01—Probabilistic graphical models, e.g. probabilistic networks

Definitions

  • the present application relates to the field of communications, and more specifically, to a model performance supervision method, apparatus and device.
  • AI models are introduced to improve system performance.
  • AI models are introduced for positioning, beam management, channel state information (CSI) prediction, mobility management, CSI compression, etc.
  • CSI channel state information
  • the performance of the AI model may be difficult to guarantee. How to supervise the performance of the AI model is a problem that needs to be solved.
  • the embodiments of the present application provide a model performance supervision method, device and equipment, which can solve the performance supervision problem of AI models.
  • a model performance supervision method comprising:
  • the first device acquires first information, wherein the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;
  • the first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.
  • a model performance supervision method including:
  • the second device receives third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is the first feature, and the second information is a label corresponding to the first feature;
  • the second device determines the validity of the first AI model based on the third information.
  • a model performance supervision method including:
  • the first device obtains fourth information, where the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;
  • the first device determines the validity of the second AI model based on the fourth information; or, the first device sends the fourth information to the second device.
  • a model performance supervision method including:
  • the second device receives fourth information from the first device, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;
  • the second device determines the validity of the second AI model based on the fourth information.
  • a model performance monitoring device comprising:
  • an acquisition unit configured to acquire first information, wherein the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;
  • a processing unit used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.
  • a model performance monitoring device comprising:
  • a transceiver unit configured to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature;
  • a processing unit is used to determine the validity of the first AI model based on the third information.
  • a model performance monitoring device comprising:
  • an acquisition unit configured to acquire fourth information, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;
  • a processing unit used to determine the validity of the second AI model according to the fourth information
  • a transceiver unit used to send the fourth information to the second device.
  • a model performance monitoring device comprising:
  • a transceiver unit configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second AI model, and the input of the second AI model is the third feature;
  • a processing unit configured to determine the validity of the second AI model based on the fourth information.
  • a model performance monitoring device which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.
  • a model performance supervision device comprising a processor and a communication interface; wherein the communication interface or the processor is used to obtain first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first AI model, and the input of the first AI model is a first feature; the processor is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.
  • a model performance monitoring device which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the second aspect are implemented.
  • a model performance supervision device comprising a processor and a communication interface; wherein the communication interface is used to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; the processor is used to determine the validity of the first AI model based on the third information.
  • a model performance monitoring device which includes a processor and a memory, wherein the memory stores programs or instructions that can be executed on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the third aspect are implemented.
  • a model performance supervision device comprising a processor and a communication interface; wherein the communication interface or the processor is used to obtain fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of a second AI model, and the input of the second AI model is a third feature; the processor is used to determine the validity of the second AI model based on the fourth information; or, the communication interface is used to send the fourth information to a second device.
  • a model performance monitoring device which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the fourth aspect are implemented.
  • a model performance supervision device comprising a processor and a communication interface; wherein the communication interface is used to receive fourth information from a first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of a second AI model, and the input of the second AI model is a third feature; the processor is used to determine the validity of the second AI model based on the fourth information.
  • a readable storage medium wherein a program or instruction is stored on the readable storage medium.
  • the steps of the method described in the first aspect are implemented, or the steps of the method described in the second aspect are implemented, or the steps of the method described in the third aspect are implemented, or the steps of the method described in the fourth aspect are implemented.
  • a wireless communication system including: a first device and a second device, wherein the first device can be used to execute the steps of the method described in the first aspect or the third aspect, and the second device can be used to execute the steps of the method described in the second aspect or the fourth aspect.
  • a chip comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instructions to implement the method as described in the first aspect, or the method as described in the second aspect, or the method as described in the third aspect, or the method as described in the fourth aspect.
  • a computer program/program product is provided, wherein the computer program/program product is stored in a storage medium, and the program/program product is executed by at least one processor to implement the steps of the model performance supervision method as described in at least one of the first to fourth aspects.
  • the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, so as to achieve performance supervision of the first AI model.
  • the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result.
  • the performance supervision of the first AI model referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.
  • the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model.
  • the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result.
  • the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.
  • FIG1 is a schematic diagram of a communication system architecture provided in an embodiment of the present application.
  • FIG2 is a schematic diagram of a neural network provided in the present application.
  • FIG. 3 is a schematic diagram of a neuron provided in the present application.
  • FIG4 is a schematic flowchart of a model performance supervision method provided according to an embodiment of the present application.
  • FIG5 is a schematic diagram of improving positioning accuracy based on soft information according to an embodiment of the present application.
  • FIG6 is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.
  • FIG7 is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.
  • FIG8 is a schematic flowchart of another model performance supervision method provided according to an embodiment of the present application.
  • FIG9 is a schematic block diagram of a model performance monitoring device provided according to an embodiment of the present application.
  • FIG10 is a schematic block diagram of another model performance monitoring device provided according to an embodiment of the present application.
  • FIG11 is a schematic block diagram of another model performance monitoring device provided according to an embodiment of the present application.
  • FIG12 is a schematic block diagram of yet another model performance monitoring device provided according to an embodiment of the present application.
  • FIG13 is a schematic block diagram of a communication device provided according to an embodiment of the present application.
  • FIG14 is a schematic diagram of the hardware structure of a terminal provided according to an embodiment of the present application.
  • FIG15 is a schematic block diagram of a network-side device provided according to an embodiment of the present application.
  • FIG16 is a schematic block diagram of another network-side device provided according to an embodiment of the present application.
  • first, second, etc. of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by “first” and “second” are generally of one type, and the number of objects is not limited, for example, the first object can be one or more.
  • “or” in the present application represents at least one of the connected objects.
  • “A or B” covers three schemes, namely, Scheme 1: including A but not including B; Scheme 2: including B but not including A; Scheme 3: including both A and B.
  • the character "/" generally indicates that the objects associated with each other are in an "or” relationship.
  • indication in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication).
  • a direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication;
  • an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.
  • LTE Long Term Evolution
  • LTE-A Long Term Evolution
  • CDMA Code Division Multiple Access
  • TDMA Time Division Multiple Access
  • FDMA Frequency Division Multiple Access
  • OFDMA Orthogonal Frequency Division Multiple Access
  • SC-FDMA Single-carrier Frequency-Division Multiple Access
  • WLAN Wireless Local Area Networks
  • WiFi Wireless Fidelity
  • system and “network” in the embodiments of the present application are often used interchangeably, and the described technology can be used for the above-mentioned systems and radio technologies as well as other systems and radio technologies.
  • NR New Radio
  • 6G 6th Generation
  • Fig. 1 shows a block diagram of a wireless communication system applicable to the embodiments of the present application.
  • the wireless communication system includes a terminal 11 and a network side device 12, wherein the terminal 11 can communicate with the network side device 12 directly or through other network elements.
  • the terminal 11 can be a mobile phone, a tablet computer (Tablet Personal Computer), a laptop computer (Laptop Computer), a notebook computer, a personal digital assistant (Personal Digital Assistant, PDA), a handheld computer, a netbook, an ultra-mobile personal computer (Ultra-mobile Personal Computer, UMPC), a mobile Internet device (Mobile Internet Device, MID), an augmented reality (Augmented Reality, AR), a virtual reality (Virtual Reality, VR) device, a robot, a wearable device (Wearable Device), an aircraft (flight vehicle), a vehicle-mounted device (Vehicle User Equipment, VUE), a ship-mounted device, a pedestrian terminal (Pedestrian User Equipment, PUE), a smart home (home appliances with wireless communication functions, such as refrigerators, televisions, washing machines or furniture, etc.), a game console, a personal computer (Personal Computer, PC), a teller machine or a self-service machine and other terminal side devices.
  • a mobile Internet device Mobile Internet Device,
  • Wearable devices include: smart watches, smart bracelets, smart headphones, smart glasses, smart jewelry (smart bracelets, smart Bracelets, smart rings, smart necklaces, smart anklets, smart anklets, etc.), smart wristbands, smart clothing, etc.
  • the vehicle-mounted device can also be called a vehicle-mounted terminal, a vehicle-mounted controller, a vehicle-mounted module, a vehicle-mounted component, a vehicle-mounted chip or a vehicle-mounted unit, etc. It should be noted that the specific type of the terminal 11 is not limited in the embodiment of the present application.
  • the network side device 12 may include an access network device or a core network device.
  • the access network equipment can also be called radio access network (Radio Access Network, RAN) equipment, radio access network function or radio access network unit.
  • the access network equipment may include base stations, wireless local area network (Wireless Local Area Network, WLAN) access points (Access Point, AS) or wireless fidelity (Wireless Fidelity, WiFi) nodes, etc.
  • WLAN wireless Local Area Network
  • AS Access Point
  • WiFi Wireless Fidelity
  • the base station can be called node B (Node B, NB), evolved node B (Evolved Node B, eNB), next generation node B (the next generation Node B, gNB), new radio node B (New Radio Node B, NR Node B), access point, relay station (Relay Base Station, RBS), serving base station (Serving Base Station, SBS), base transceiver station (Base Transceiver Station, BTS), radio base station, radio transceiver, base
  • the base station is not limited to specific technical terms as long as the same technical effect is achieved. It should be noted that in the embodiments of the present application, only the base station in the NR system is taken as an example for introduction, and the specific type of the base station is not limited.
  • the core network equipment may include but is not limited to at least one of the following: core network nodes, core network functions, mobility management entity (Mobility Management Entity, MME), access mobility management function (Access and Mobility Management Function, AMF), session management function (Session Management Function, SMF), user plane function (User Plane Function, UPF), policy control function (Policy Control Function, PCF), policy and charging rules function unit (Policy and Charging Rules Function, PCRF), edge application service discovery function (Edge Application Server Discovery Function, EASDF), unified data management (U nified Data Management, UDM), Unified Data Repository (UDR), Home Subscriber Server (HSS), Centralized network configuration (CNC), Network Repository Function (NRF), Network Exposure Function (NEF), Local NEF (L-NEF), Binding Support Function (BSF), Application Function (AF), Location Management Function (LMF), etc.
  • MME mobility management entity
  • AMF Access Mobility Management Function
  • SMF Session Management Function
  • UPF User Plane Function
  • AI Artificial intelligence
  • neural networks decision trees, support vector machines, Bayesian classifiers, etc. This application takes neural networks as an example for illustration, but does not limit the specific type of AI modules.
  • FIG2 An exemplary neural network can be shown in FIG2 , where the neural network is composed of neurons, and the neurons can be shown in FIG3 , where ⁇ 1 , ⁇ 2 , ... ⁇ K are inputs, w is a weight (multiplicative coefficient), b is a bias (additive coefficient), and ⁇ (.) is an activation function.
  • Common activation functions include Sigmoid, tanh, Rectified Linear Unit (ReLU), and the like.
  • the parameters of the neural network are optimized using a gradient optimization algorithm.
  • a gradient optimization algorithm is a type of algorithm that minimizes or maximizes an objective function (sometimes called a loss function), and the objective function is often a mathematical combination of model parameters and data.
  • an objective function sometimes called a loss function
  • the objective function is often a mathematical combination of model parameters and data.
  • f(.) Given data X and its corresponding label Y, we build a neural network model f(.). With the model, we can get the predicted output f(x) based on the input x, and we can calculate the difference between the predicted value and the true value (f(x)-Y), which is the loss function.
  • the purpose of model training is to find the appropriate w, b to minimize the value of the above loss function. The smaller the loss value, the closer the constructed neural network model is to the actual situation.
  • BP error back propagation
  • the basic idea of BP algorithm is that the learning process is a forward propagation of the signal.
  • the error back propagation process consists of two processes: forward propagation and error back propagation.
  • forward propagation the input sample is transmitted from the input layer, processed by each hidden layer layer by layer, and then transmitted to the output layer. If the actual output of the output layer does not match the expected output, the error back propagation stage is entered.
  • Error back propagation is to propagate the output error layer by layer through the hidden layer in some form, and distribute the error to all units in each layer, so as to obtain the error signal of each layer unit.
  • This error signal is used as the basis for correcting the weights of each unit.
  • This process of adjusting the weights of each layer of the signal forward propagation and error back propagation is repeated.
  • the process of continuous adjustment of weights is also the learning and training process of the network. This process continues until the error of the network output is reduced to an acceptable level, or until the pre-set number of learning times is reached.
  • optimization algorithms When these optimization algorithms are backpropagating errors, they can calculate the derivative/partial derivative of neurons based on the error/loss obtained from the loss function, add the influence of the learning rate, the previous gradient/derivative/partial derivative, etc., get the gradient, and pass the gradient to the previous layer.
  • FIG4 is a schematic flow chart of a model performance supervision method 200 according to an embodiment of the present application.
  • the model performance supervision method 200 may include at least part of the following contents:
  • the first device obtains first information, where the first information is used to characterize uncertainty or probability distribution of an output of a first AI model, and an input of the first AI model is a first feature;
  • the first device determines third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.
  • FIG4 shows the steps or operations of the model performance supervision method 200, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG4.
  • third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model.
  • the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result.
  • referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.
  • motion state information such as speed, acceleration, etc.
  • signal quality measurement information such as reference signal received power (Reference Signal Received Power, RSRP), signal to interference plus noise ratio (Signal to Interference plus Noise Ratio, SINR), reference signal received quality (Reference Signal Received Quality, RSRQ), etc.
  • the judgment result of the effectiveness of the first AI model obtained based on the embodiment of the present application can also be combined with the judgment result of the effectiveness of the first AI model obtained by other model supervision methods to obtain the final conclusion on the effectiveness of the first AI model.
  • the tag described in the embodiment of the present application is obtained in some way and is associated with the target task.
  • This tag can be obtained through measurement or other prior information; for example, based on the positioning of the AI model, the input of the AI model is the channel state information, and the output of the AI model is the uncertainty or probability distribution of the position. Then this tag is the position information corresponding to the channel state information.
  • the position information can be obtained by GPS and other positioning methods, or by a positioning reference unit with a known position. arrive.
  • the first information may also be referred to as soft information (such as probability distribution, confidence level, confidence interval, etc.), wherein the soft information gives the probability distribution or confidence level of the possible results.
  • the first AI model may be a soft information AI model or may not be a soft information AI model.
  • the soft information AI model refers to a type of AI model that outputs soft information (such as probability distribution, confidence level, confidence interval, etc.), including both classic probability models and AI models based on neural networks; the soft information AI model measures the possibility of different prediction results and gives the probability distribution or confidence level of each possible result.
  • the soft information obtained by AI model reasoning can significantly improve the reasoning accuracy and robustness. Specifically, soft information can better describe the uncertainty of the world, improve the robustness of model reasoning, and provide better security for some businesses that require relatively high reasoning reliability.
  • the dimension of the first feature is D1 dimension
  • each prediction moment corresponds to a set of soft information, or each feature at each prediction moment corresponds to a set of soft information; for other tasks, each feature corresponds to a set of soft information, or all features correspond to a set of soft information.
  • each feature corresponds to a set of soft information, or all features correspond to a set of soft information. For example, if the output of the first AI model is 2-dimensional, then it corresponds to 2 sets of soft information.
  • the first AI model output is a 2-dimensional horizontal position coordinate
  • the 2-dimensional horizontal position coordinate corresponds to a set of soft information
  • each dimension of the 2-dimensional horizontal position coordinate corresponds to a set of soft information.
  • the AI model described in this application may also be referred to as an AI unit, an AI model/AI unit, a machine learning (ML) model, an ML unit, an AI structure, an AI function, an AI feature, a neural network, a neural network function, a neural network function, etc., or the AI model described in this application may also refer to a processing unit capable of implementing specific algorithms, formulas, processing procedures, capabilities, etc.
  • the AI model described in this application may be a processing method, algorithm, function, module or unit for a specific data set
  • the AI model described in this application may be a processing method, algorithm, function, module or unit running on AI/ML related hardware such as a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), etc., and this application does not specifically limit this.
  • the specific data set includes the input or output of the AI model.
  • the identifier of the AI model described in this application may be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, or an identifier of a specific data set associated with the AI model described in this application, or an identifier of a specific scenario, environment, channel feature, or device related to the AI model described in this application, or an identifier of a function, feature, capability, or module related to the AI model described in this application.
  • This application does not make any specific limitations on this.
  • the first AI model may be an activated model, or the first AI model may be an inactivated model. Specifically, when the first AI model is an inactivated model, after determining the validity of the first AI model, when performing AI model selection, an effective AI model may be preferentially selected from at least two AI models, thereby facilitating the selection of an AI model.
  • the first information includes but is not limited to at least one of the following: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence.
  • the embodiment of the present application clarifies the content of the first information, which is beneficial to realizing the performance supervision of the AI model.
  • the amount of information contained in the first information may be one or at least two, that is, the first information may include but is not limited to at least one of the following: parameters of the probability density distribution of one or at least two second features, confidence intervals of one or at least two second features, values of one or at least two second features, values of one or at least two second features and their probabilities, and values of one or at least two second features and their confidence levels.
  • the parameters of the probability density distribution include, but are not limited to, at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.
  • the type of probability density distribution may include but is not limited to at least one of the following: Gaussian distribution, Poisson distribution.
  • a Gaussian distribution there is a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, such as a typical Gaussian distribution with a mean of ⁇ and a standard deviation of ⁇ .
  • the 90% confidence interval is: [ ⁇ -1.645 ⁇ , ⁇ +1.645 ⁇ ], which means that there is a 90% probability that the predicted target value is within the interval [ ⁇ -1.645 ⁇ , ⁇ +1.645 ⁇ ].
  • the 95% confidence interval is: [ ⁇ -1.96 ⁇ , ⁇ +1.96 ⁇ ], which means that there is a 95% probability that the predicted target value is within the interval [ ⁇ -1.96 ⁇ , ⁇ +1.96 ⁇ ].
  • the confidence interval there may be a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, where the mean is ⁇ , the standard deviation is ⁇ , the coefficient z may be obtained by the probability density distribution function or the type of probability density distribution, and the confidence interval with a confidence level of p% may be as follows: [ ⁇ -z p% ⁇ , ⁇ +z p% ⁇ ].
  • the first information is output information of the first AI model, or the first information is information determined based on the output information of the first AI model.
  • the output information of the first AI model is at least one of the following: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence. That is, the first information is the output information of the first AI model.
  • the output information of the first AI model is the value of the second feature, and the first information can be determined based on the value of the second feature.
  • the first information is determined based on the value of the second feature and other information (such as motion state information (speed, acceleration, etc.), signal quality measurement information (such as RSRP, SINR, RSRQ, etc.)).
  • motion state information speed, acceleration, etc.
  • signal quality measurement information such as RSRP, SINR, RSRQ, etc.
  • the output information of the first AI model is the value of the second feature
  • at least one of the following can be determined based on the value of the second feature: a parameter of the probability density distribution of the second feature, a confidence interval of the second feature, a probability of the value of the second feature, and a confidence level of the value of the second feature. That is, the first information is information determined based on the output information of the first AI model.
  • the first AI model can be used to implement one of the following functions: positioning, beam management, channel state information (CSI) prediction, mobility management, and CSI compression.
  • CSI channel state information
  • the first AI model can also be used to implement other functions, which is not limited in this application.
  • model performance supervision method 200 described in this embodiment can be used to implement at least two different functions, that is, the model performance supervision method 200 described in this embodiment can be applicable to a public label-based AI model supervision framework, thereby avoiding the need to design performance supervision solutions for different functions.
  • the first feature may include but is not limited to at least one of the following: time domain channel impulse response, RSRP (such as layer 1 RSRP or layer 3 RSRP), frequency domain channel impulse response, time domain waveform of the received signal.
  • RSRP such as layer 1 RSRP or layer 3 RSRP
  • the time domain channel impulse response includes at least one of the following: time information, power information, phase information.
  • the frequency domain channel impulse response includes at least one of the following: frequency information (subcarrier number and interval), power information, phase information.
  • the second feature may include but is not limited to at least one of the following: line-of-sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication.
  • the input of the first AI model i.e., the first feature
  • the output of the first AI model is the soft information of the intermediate feature quantity (i.e., the second feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.)
  • the intermediate feature quantity includes at least one of the following: line of sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc.
  • the position coordinates can be further determined based on the soft information of the intermediate feature quantity; the output of the first AI model may also be the soft information of the position coordinates.
  • the first feature when used to implement the beam management function, may include beam information at T1 historical moments, such as sequence number, angle, L1-RSRP, etc., and the second feature includes beam information at T2 future moments.
  • the input of the first AI model i.e., the first feature
  • the output of the first AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the beam information (i.e., the second feature) at T2 future moments.
  • the vertical beam and the horizontal beam at each moment correspond to a set of soft information, such as the confidence interval and probability density distribution of L1-RSRP, etc.
  • the beam information at each moment corresponds to a set of soft information.
  • the first feature when the first AI model is used to implement the CSI prediction function, the first feature may include the CSI at T1 historical moments, and the second feature may include the CSI at T2 future moments.
  • the input of the first AI model i.e., the first feature
  • the output of the first AI model is the soft information (such as parameters of the probability density distribution, confidence interval, confidence level, etc.) of the CSI at T2 moments in the future (i.e., the second feature)
  • the CSI at each moment corresponds to a set of soft information, such as each dimension of the CSI at each moment corresponds to a set of soft information, or the CSI at each moment corresponds to a set of soft information.
  • the first feature when the first AI model is used to implement the mobility management function, the first feature may include the layer 1 RSRP (L1-RSRP) or layer 3 RSRP (L3-RSRP) at the historical T1 moment, and the second feature may include the RSRP at the future T2 moments or the decision of whether a cell switching occurs at the future T2 moments.
  • L3-RSRP is obtained by filtering L1-RSRP.
  • the input of the first AI model i.e., the first feature
  • the output of the first AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the RSRP (i.e., the second feature) at T2 moments in the future
  • the output of the first AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the decision on whether a cell switching will occur (i.e., the second feature) at T2 moments in the future, such as 1 for switching and 0 for no switching, and the soft information is a value between 0 and 1.
  • the first feature when the first AI model is used to implement the CSI compression function, the first feature may include uncompressed CSI, and the second feature may include compressed CSI.
  • the input of the first AI model ie, the first feature
  • the output of the first AI model is the soft information of the compressed CSI (ie, the second feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.).
  • the label corresponding to the first feature may be real, or may be measured or estimated, and this is not limited in the embodiments of the present application.
  • the type of label corresponding to the first feature is consistent with the type of output of the first AI model. For example, both are location information, both are TOA, both are CSI, both are RSRP, etc.; this depends on the specific task type.
  • the label corresponding to the first feature may be measured, estimated, or stored by the first device, or the label corresponding to the first feature (ie, the second information) may be obtained by the first device from the second device or other devices.
  • the first device obtains the first information, including one of the following:
  • the first device receives first information from the second device
  • the first device obtains first information through output information of the first AI model
  • the first device receives output information of the first AI model from the second device, and obtains first information according to the output information of the first AI model.
  • the first AI model when the first device obtains the first information through output information of the first AI model, the first AI model can be deployed on the first device side.
  • the first AI model when the first device receives the first information from the second device, the first AI model can be deployed on the second device side.
  • the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device.
  • the first AI model when the first device receives output information of the first AI model from the second device and obtains the first information based on the output information of the first AI model, the first AI model can be deployed on the second device side.
  • the first device determines the validity of the first AI model based on the third information. Specifically, after the first device determines the third information based on the first information and the second information, the first device determines the validity of the first AI model based on the third information.
  • the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.
  • the first indication information occupies 1 bit; wherein a value of 0 indicates that the first AI model is valid, and a value of 1 indicates that the first AI model is invalid; or, a value of 1 indicates that the first AI model is valid, and a value of 0 indicates that the first AI model is invalid.
  • the first indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.
  • the first device sends third information to the second device. Further, the second device may determine the validity of the first AI model based on the third information.
  • the first device receives second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model.
  • the second indication information occupies 1 bit; wherein a value of 0 indicates that the first AI model is valid, and a value of 1 indicates that the first AI model is invalid; or, a value of 1 indicates that the first AI model is valid, and a value of 0 indicates that the first AI model is invalid.
  • the second indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.
  • the first device when the first AI model is deployed on the first device side, the first device obtains first information through the output information of the first AI model, then the first device obtains second information from a local or other device (such as a second device or a device other than the first device and the second device), and the first device determines third information based on the first information and the second information, and then, the first device determines the validity of the first AI model based on the third information, and finally, the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.
  • a local or other device such as a second device or a device other than the first device and the second device
  • the first device determines third information based on the first information and the second information
  • the first device determines the validity of the first AI model based on the third information
  • the first device sends first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.
  • the first device when the first AI model is deployed on the first device side, the first device obtains the first information through the output information of the first AI model, then the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information, and then the first device sends the third information to the second device, after which the second device determines the validity of the first AI model based on the third information, and finally, the second device sends the second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.
  • the local or other device such as the second device or a device other than the first device and the second device
  • the first device determines the third information based on the first information and the second information
  • the first device sends the third information to the second device, after which the second device determines the validity of the first AI model based on the third information
  • the second device sends the second indication information to the first
  • the second device when the first AI model is deployed on the second device side, the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device. Then, the first device obtains the second information from the local or other devices (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device determines the validity of the first AI model based on the third information. Finally, the first device sends the first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.
  • the second device obtains the first information through the output information of the first AI model, and the second device sends the first information to the first device, then the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information, and then the first device sends the third information to the second device.
  • the second device determines the validity of the first AI model according to the third information
  • the second device sends second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.
  • the first device when the first AI model is deployed on the second device side, the first device receives the output information of the first AI model from the second device, and the first device obtains the first information based on the output information of the first AI model. Then, the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device determines the validity of the first AI model based on the third information. Finally, the first device sends the first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model.
  • the first indication information is used to indicate the validity of the first AI model.
  • the first device when the first AI model is deployed on the second device side, receives the output information of the first AI model from the second device, and the first device obtains the first information based on the output information of the first AI model. Then, the first device obtains the second information from the local or other device (such as the second device or a device other than the first device and the second device), and the first device determines the third information based on the first information and the second information. Then, the first device sends the third information to the second device, and then, the second device determines the validity of the first AI model based on the third information. Finally, the second device sends the second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.
  • the local or other device such as the second device or a device other than the first device and the second device
  • the first device may be a terminal, a network side device, or a third-party server, wherein the network side device includes an access network device or a core network device.
  • the second device may be a terminal, a network side device, or a third-party server, wherein the network side device includes an access network device or a core network device.
  • the third information is used to characterize the mean of the first probability of N samples of the first feature, or the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature; wherein the first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer.
  • the N samples can also be replaced by N inference processes of the first AI model.
  • the third information may be a log-likelihood value.
  • the third information when the third information is used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula 1:
  • L1 represents the third information
  • i represents the i-th sample among the N samples
  • yi represents the label corresponding to the i-th sample
  • g( ⁇ ) represents the probability density distribution function obeyed by the output of the first AI model.
  • the third information when the third information is used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula 2:
  • L 2 represents the third information
  • i represents the i-th sample among the N samples
  • yi represents the label corresponding to the i-th sample
  • s is a positive integer
  • g( ⁇ ) represents the probability density distribution function obeyed by the output of the first AI model.
  • s is agreed upon by a protocol, or s is determined by the first device, or s is configured or indicated by the second device.
  • g( ⁇ ) is related to the type of probability density distribution, for example, g( ⁇ ) can be determined based on the type of probability density distribution.
  • the third information is used to determine the validity of the first AI model, including:
  • the first AI model is valid
  • the first AI model fails.
  • the first threshold is agreed upon by a protocol, or the first threshold is determined by the first device, or the first threshold is configured or indicated by the second device.
  • the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.
  • the N samples can also be replaced by N inference processes of the first AI model.
  • the third information is determined based on the following formula 5:
  • L 3 represents the third information
  • i represents the i-th sample in the N samples
  • x i represents the input information corresponding to the i-th sample
  • y i represents the label corresponding to the i-th sample, when y i is within the confidence interval f(x i ) of the second feature, otherwise
  • the third information is used to determine the validity of the first AI model, including:
  • the first AI model is valid
  • the first AI model fails.
  • the second threshold is agreed upon by a protocol, or the second threshold is determined by the first device, or the second threshold is configured or indicated by the second device.
  • the third information is used to characterize the average of the weighted distances between the labels corresponding to the N samples of the first feature and the value of the second feature, where the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.
  • the N samples can also be replaced by N inference processes of the first AI model.
  • the third information is determined based on the following formula 6:
  • L 4 represents the third information
  • i represents the i-th sample among the N samples
  • yi represents the label corresponding to the i-th sample.
  • the diagonal elements of ⁇ -1 are P powers of probability or confidence, where P is greater than or equal to 0 and P is an integer.
  • the third information is used to determine the validity of the first AI model, including:
  • the first AI model is valid
  • the first AI model fails.
  • the third threshold is agreed upon by a protocol, or the third threshold is determined by the first device, or the third threshold is configured or indicated by the second device.
  • yi It can be a specific numerical value, or a vector or a matrix, which is not limited in the embodiments of the present application.
  • the N samples are acquired within M time units, where M is a positive integer.
  • the time unit may include at least one of: orthogonal frequency-division multiplexing (OFDM) symbol, time slot, subframe, frame, microsecond, millisecond, second, minute, hour, day, week, month.
  • OFDM orthogonal frequency-division multiplexing
  • M is configured or indicated by the second device, or M is agreed upon by a protocol.
  • the scene identifiers or data set identifiers associated with the N samples are the same.
  • the first device may obtain at least one of the following parameters from the second device:
  • the type and parameters of the probability density distribution for example, in the case of a Gaussian mixture model, the number of Gaussian distributions involved needs to be indicated;
  • the first device reports at least one of the following parameters to the second device:
  • the type and parameters of the probability density distribution for example, in the case of a Gaussian mixture model, the number of Gaussian distributions involved needs to be indicated;
  • the third information may be positive incentive information or negative incentive information; for example, when the negative incentive information is greater than a certain threshold or the positive incentive information is less than or equal to a certain threshold, the first AI model is considered to be invalid.
  • the positive incentive information may include but is not limited to at least one of the following:
  • the negative incentive information may include but is not limited to at least one of the following:
  • the number or proportion of samples whose weighted distance between the label and the value of the second feature is greater than t 3 .
  • t1 may be the first threshold
  • t2 may be the second threshold
  • t3 may be the third threshold
  • the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model (i.e., the first information) and the label corresponding to the input feature of the first AI model (i.e., the second information), and the The validity of the first AI model is determined based on the third information, thereby achieving performance supervision of the first AI model.
  • the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results.
  • the reasoning results of the first AI model are combined with other types of information to obtain a more accurate target result.
  • referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding labels can improve the accuracy of the performance supervision of the first AI model.
  • Soft information can improve the robustness of the reasoning results of the AI model because the reasoning results cover multiple possible results and their probability distribution. It is conducive to further processing and utilization of the reasoning results of the AI model, such as combining soft information with other types of information (such as information output by other AI models used to implement positioning functions) to obtain more accurate target results, and it can also provide better security for some businesses that require relatively high reasoning reliability.
  • the input of the ith AI model is the time-domain channel impulse response (CIR) of the ith transmission reception point (TRP), including the time, power, and phase information of the multipath.
  • CIR channel impulse response
  • the output of the ith AI model is the mean ⁇ i and standard deviation ⁇ i of the line-of-sight TOA between the ith TRP and the terminal;
  • the line-of-sight TOA estimate xi between the ith TRP and the terminal is modeled as a Gaussian distribution:
  • the maximum likelihood estimation problem can be transformed into a weighted least squares problem:
  • ⁇ [ ⁇ 1 , ..., ⁇ N ] T
  • ⁇ is an N*N matrix
  • its i-th diagonal element is Its goal is to find a location
  • the N TOA estimates obtained by combining this position with the coordinates of the N TRPs are
  • the weighted distance to the N TOA mean ⁇ estimated by the AI model is the smallest.
  • the framework can support mixed positioning of multiple types of soft information.
  • soft information ⁇ , ⁇ of TOA of N TRPs is given.
  • soft information of angles can also be included. Its likelihood function can be written as follows:
  • x, ⁇ , and z refer to different types of information, such as TOA, AOD, and AOA, respectively.
  • the number of TRPs for different types of information can also be different.
  • the number of TRPs for x is N 1
  • the number of TRPs for ⁇ is N 2
  • the number of TRPs for z is N 3 .
  • Their likelihood functions can be written as follows:
  • FIG6 is a schematic flow chart of a model performance supervision method 300 according to an embodiment of the present application.
  • the model performance supervision method 300 may include at least part of the following contents:
  • the second device receives third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first AI model, the input of the first AI model is the first feature, and the second information is a label corresponding to the first feature;
  • S320 The second device determines the validity of the first AI model based on the third information.
  • FIG6 shows the steps or operations of the model performance supervision method 300, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG6.
  • the first information includes at least one of the following:
  • Parameters of the probability density distribution of the second feature a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.
  • the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.
  • the third information is used to characterize a mean of the first probabilities of N samples of the first feature, or the third information is used to characterize a mean of the logarithms of the first probabilities of N samples of the first feature;
  • the first probability is the probability of the label corresponding to each of the N samples under the probability density distribution of the second feature, and N is a positive integer.
  • the third information when used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:
  • L1 represents the third information
  • i represents the i-th sample among the N samples
  • yi represents the label corresponding to the i-th sample
  • g( ⁇ ) represents the probability density distribution function obeyed by the output of the first AI model.
  • the third information when used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:
  • L 2 represents the third information
  • i represents the i-th sample among the N samples, Indicates that the i-th sample
  • the corresponding mean of the probability density distribution of the second feature represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample
  • yi represents the label corresponding to the i-th sample
  • s is a positive integer
  • g( ⁇ ) represents the probability density distribution function obeyed by the output of the first AI model.
  • the above S320 may specifically include:
  • the second device determines that the first AI model is valid; or,
  • the second device determines that the first AI model is invalid.
  • the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.
  • the third information is determined based on the following formula:
  • L 3 represents the third information
  • i represents the i-th sample in the N samples
  • x i represents the input information corresponding to the i-th sample
  • y i represents the label corresponding to the i-th sample, when y i is within the confidence interval f(x i ) of the second feature, otherwise
  • the above S320 may specifically include:
  • the second device determines that the first AI model is valid; or,
  • the second device determines that the first AI model is invalid.
  • the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.
  • the third information is determined based on the following formula:
  • L 4 represents the third information
  • i represents the i-th sample among the N samples
  • yi represents the label corresponding to the i-th sample.
  • the above S320 may specifically include:
  • the second device determines that the first AI model is valid; or,
  • the second device determines that the first AI model is invalid.
  • the second device before the second device receives the third information from the first device, the second device sends the first information to the first device.
  • the second device sends second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.
  • the second indication information includes an identifier of the first AI model or an identifier of a function associated with the first AI model.
  • the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model (i.e., the first information) and the label corresponding to the input feature of the first AI model (i.e., the second information), and the validity of the first AI model can be determined based on the third information, thereby achieving performance supervision of the first AI model.
  • the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning result of the first AI model, by
  • the reasoning results of the first AI model cover multiple possible results and their probability distribution, which is conducive to further processing and utilization of the reasoning results.
  • the reasoning results of the first AI model are combined with other types of information to obtain more accurate target results.
  • referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding labels can improve the accuracy of the performance supervision of the first AI model.
  • FIG. 7 is a schematic flow chart of a model performance supervision method 400 according to an embodiment of the present application.
  • the model performance supervision method 400 may include at least part of the following contents:
  • the first device obtains fourth information, where the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;
  • the first device determines the validity of the second AI model according to the fourth information; or, the first device sends the fourth information to the second device.
  • FIG7 shows the steps or operations of the model performance supervision method 400, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG7.
  • the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model.
  • the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result.
  • the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.
  • the judgment result of the effectiveness of the second AI model obtained based on the embodiment of the present application can also be combined with the judgment result of the effectiveness of the second AI model obtained by other model supervision methods to obtain the final conclusion on the effectiveness of the second AI model.
  • the second AI model described in the embodiment of the present application may be the same as or different from the first AI model described above, and the present application is not limited to this.
  • the fourth information may also be referred to as soft information (such as probability distribution, confidence level, confidence interval, etc.), wherein the soft information gives the probability distribution or confidence level of the possible results.
  • the second AI model may be a soft information AI model or may not be a soft information AI model.
  • the soft information AI model refers to a type of AI model that outputs soft information (such as probability distribution, confidence level, confidence interval, etc.), including both classic probability models and AI models based on neural networks; the soft information AI model measures the possibility of different prediction results and gives the possibility, probability distribution or confidence level of each possible result.
  • the soft information obtained by AI model reasoning can significantly improve the reasoning accuracy and robustness. Specifically, soft information can better describe the uncertainty of the world, improve the robustness of model reasoning, and provide better security for some businesses that require relatively high reasoning reliability.
  • each prediction moment corresponds to a set of soft information, or each feature at each prediction moment corresponds to a set of soft information; for other tasks, each feature corresponds to a set of soft information, or all features correspond to a set of soft information.
  • each feature corresponds to a set of soft information
  • the output of the second AI model is 2-dimensional, corresponding to 2 sets of soft information.
  • the second AI model output is a 2-dimensional horizontal position coordinate
  • the 2-dimensional horizontal position coordinate corresponds to a set of soft information
  • each dimension of the 2-dimensional horizontal position coordinate corresponds to a set of soft information.
  • the AI model described in this application may be a processing method, algorithm, function, module or unit for a specific data set, or the AI model described in this application may be a processing method, algorithm, function, module or unit running on AI/ML related hardware such as a GPU, NPU, TPU, ASIC, etc., which is not specifically limited in this application.
  • the specific data set includes the input or output of the AI model.
  • the identifier of the AI model described in this application may be an AI unit identifier, an AI structure identifier, an AI algorithm identifier, or an identifier of a specific data set associated with the AI model described in this application, or an identifier of a specific scenario, environment, channel feature, or device related to the AI model described in this application, or an identifier of a function, feature, capability, or module related to the AI model described in this application.
  • This application does not make any specific limitations on this.
  • the second AI model may be an activated model, or the second AI model may be an inactivated model. Specifically, when the second AI model is an inactivated model, after determining the validity of the second AI model, when performing AI model selection, a valid AI model may be preferentially selected from at least two AI models, thereby facilitating the selection of an AI model.
  • the fourth information includes but is not limited to at least one of the following: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.
  • the embodiment of the present application clarifies the content of the fourth information, which is beneficial to realizing the performance supervision of the AI model.
  • the amount of information contained in the fourth information may be one or at least two, that is, the fourth information may include but is not limited to at least one of the following: parameters of the probability density distribution of one or at least two fourth features, confidence intervals of one or at least two fourth features, values of one or at least two fourth features, values of one or at least two fourth features and their probabilities, and values of one or at least two fourth features and their confidence levels.
  • the parameters of the probability density distribution include, but are not limited to, at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.
  • the type of probability density distribution may include but is not limited to at least one of the following: Gaussian distribution, Poisson distribution.
  • a Gaussian distribution there is a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, such as a typical Gaussian distribution with a mean of ⁇ and a standard deviation of ⁇ .
  • the 90% confidence interval is: [ ⁇ -1.645 ⁇ , ⁇ +1.645 ⁇ ], which means that there is a 90% probability that the predicted target value is within the interval [ ⁇ -1.645 ⁇ , ⁇ +1.645 ⁇ ].
  • the 95% confidence interval is: [ ⁇ -1.96 ⁇ , ⁇ +1.96 ⁇ ], which means that there is a 95% probability that the predicted target value is within the interval [ ⁇ -1.96 ⁇ , ⁇ +1.96 ⁇ ].
  • the confidence interval there may be a conversion relationship between the confidence interval, the confidence level, the mean, and the standard deviation, where the mean is ⁇ , the standard deviation is ⁇ , the coefficient z may be obtained by the probability density distribution function or the type of probability density distribution, and the confidence interval with a confidence level of p% may be as follows: [ ⁇ -z p% ⁇ , ⁇ +z p% ⁇ ].
  • the fourth information is output information of the second AI model, or the fourth information is based on Information determined by the output information of the second AI model.
  • the output information of the second AI model is at least one of the following: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence. That is, the fourth information is the output information of the second AI model.
  • the output information of the second AI model is the value of the fourth feature
  • the fourth information can be determined based on the value of the fourth feature, such as based on the value of the fourth feature and other information (such as motion state information (speed, acceleration, etc.), signal quality measurement information (such as RSRP, SINR, RSRQ, etc.)).
  • the worse the signal quality the higher the uncertainty of the fourth feature obtained based on the value of the fourth feature and other information; for another example, the faster the movement speed, the higher the uncertainty of the fourth feature obtained based on the value of the fourth feature and other information.
  • the output information of the second AI model is the value of the fourth feature
  • at least one of the following can be determined based on the value of the fourth feature: a parameter of the probability density distribution of the fourth feature, a confidence interval of the fourth feature, a probability of the value of the fourth feature, and a confidence level of the value of the fourth feature. That is, the fourth information is information determined based on the output information of the second AI model.
  • the first AI model can be used to implement one of the following functions: positioning, beam management, CSI prediction, mobility management, CSI compression.
  • the first AI model can also be used to implement other functions, which is not limited in this application.
  • model performance supervision method 400 described in this embodiment can be used to realize at least two different functions. That is, the model performance supervision method 400 described in this embodiment can be applicable to a public AI model supervision framework based on Ground truth label, thereby avoiding the need to design performance supervision solutions for different functions.
  • the third feature may include but is not limited to at least one of the following: time domain channel impulse response, RSRP (such as layer 1 RSRP or layer 3 RSRP), frequency domain channel impulse response, time domain waveform of the received signal.
  • RSRP such as layer 1 RSRP or layer 3 RSRP
  • the time domain channel impulse response includes at least one of the following: time information, power information, phase information.
  • the frequency domain channel impulse response includes at least one of the following: frequency information (subcarrier sequence number and interval), power information, phase information.
  • the input of the second AI model i.e., the third feature
  • the output of the second AI model is the soft information of the intermediate feature quantity (i.e., the fourth feature) (such as parameters of probability density distribution, confidence interval, confidence level, etc.)
  • the intermediate feature quantity includes at least one of the following: line of sight TOA, RSTD, AoA, AoD, RSRP, LOS indication, NLOS indication, etc.
  • the position coordinates can be further determined based on the soft information of the intermediate feature quantity; the output of the second AI model may also be the soft information of the position coordinates.
  • the third feature may include beam information at T1 historical moments, such as sequence number, angle, L1-RSRP, etc.
  • the fourth feature includes beam information at T2 future moments.
  • the input of the second AI model i.e., the third feature
  • the output of the second AI model is the soft information (such as parameters of probability density distribution, confidence interval, confidence level, etc.) of the beam information (i.e., the fourth feature) at T2 future moments.
  • the vertical beam and the horizontal beam at each moment correspond to a set of soft information, such as the confidence interval and probability density distribution of L1-RSRP, etc.; the beam information at each moment corresponds to a set of soft information.
  • the third feature may include the CSI at T1 historical moments
  • the fourth feature may include the CSI at T2 future moments.
  • the input of the second AI model i.e., the third feature
  • the output of the second AI model is the soft information (such as parameters of the probability density distribution, confidence interval, confidence level, etc.) of the CSI at T2 moments in the future (i.e., the fourth feature)
  • the CSI at each moment corresponds to a set of soft information, such as each dimension of the CSI at each moment corresponds to a set of soft information, or the CSI at each moment corresponds to a set of soft information.
  • the third feature may include the layer 1 RSRP (L1-RSRP) or layer 3 RSRP (L3-RSRP) at the historical T1 moment
  • the fourth feature may include the RSRP at the future T2 moments or the decision of whether a cell switching occurs at the future T2 moments.
  • L3-RSRP is obtained by filtering L1-RSRP.
  • the input of the second AI model i.e., the third feature
  • the output of the second AI model is the soft information of the RSRP (i.e., the fourth feature) at T2 moments in the future (such as parameters of the probability density distribution, confidence interval, confidence level, etc.)
  • the output of the second AI model is the soft information of the decision on whether a cell switching will occur (i.e., the fourth feature) at T2 moments in the future (such as parameters of the probability density distribution, confidence interval, confidence level, etc.), such as 1 for switching and 0 for no switching, and the soft information is a value between 0 and 1.
  • the third feature when the second AI model is used to implement the CSI compression function, the third feature may include uncompressed CSI, and the fourth feature may include compressed CSI.
  • the input of the second AI model ie, the third feature
  • the output of the second AI model is the soft information of the compressed CSI (ie, the fourth feature) (such as parameters of the probability density distribution, confidence interval, confidence level, etc.).
  • the first device obtains the fourth information, including one of the following:
  • the first device obtains the fourth information from the second device
  • the first device obtains the fourth information through the output information of the second AI model
  • the first device receives the output information of the second AI model from the second device, and obtains the fourth information according to the output information of the second AI model.
  • the second AI model when the first device obtains the fourth information through the output information of the second AI model, the second AI model can be deployed on the first device side.
  • the second AI model when the first device obtains the fourth information from another device, can be deployed on the other device side.
  • the other device can be the second device or a device other than the first device and the second device.
  • the second AI model when the first device receives output information of the second AI model from the second device and obtains fourth information based on the output information of the second AI model, the second AI model can be deployed on the second device side.
  • the third indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.
  • the first device when the first device sends the fourth information to the second device, the first device receives the fourth indication information from the second device, wherein the fourth indication information is used to indicate the validity of the second AI model.
  • the second device can determine the validity of the second AI model according to the fourth information. For example, the fourth indication information occupies 1 bit; wherein a value of 0 indicates that the second AI model is valid, and a value of 1 indicates that the second AI model is invalid; or, a value of 1 indicates that the second AI model is valid, and a value of 0 indicates that the second AI model is invalid.
  • the fourth indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.
  • the first device when the second AI model is deployed on the first device side, the first device obtains the fourth information through the output information of the second AI model. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model.
  • the first device obtains the fourth information through the output information of the second AI model, then the first device sends the fourth information to the second device, and then the second device receives the fourth information based on the output information of the second AI model.
  • the validity of the second AI model is determined according to the fourth information.
  • the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.
  • the second device when the second AI model is deployed on the second device side, the second device obtains the fourth information through the output information of the second AI model, and the second device sends the fourth information to the first device. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model.
  • the third device obtains the fourth information through the output information of the second AI model, and the third device sends the fourth information to the first device, then the first device sends the fourth information to the second device, and then the second device determines the validity of the second AI model based on the fourth information, and finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.
  • the first device when the second AI model is deployed on the second device side, the first device receives the output information of the second AI model from the second device, and the first device obtains fourth information based on the output information of the second AI model. Then, the first device determines the validity of the second AI model based on the fourth information. Finally, the first device sends third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model.
  • the first device when the second AI model is deployed on the second device side, the first device receives the output information of the second AI model from the second device, and the first device obtains fourth information based on the output information of the second AI model. Then, the first device sends the fourth information to the second device. Thereafter, the second device determines the validity of the second AI model based on the fourth information. Finally, the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.
  • the first device may be a terminal, a network-side device, or a third-party server.
  • the second device may be a terminal, a network-side device, or a third-party server.
  • the first device determines the validity of the second AI model according to the fourth information, including:
  • the first device determines that the second AI model fails; or,
  • the first device determines that the second AI model is invalid; or,
  • the first device determines that the second AI model has failed;
  • N is a positive integer.
  • the N samples may also be replaced by N reasoning processes of the second AI model.
  • the first threshold is agreed upon by a protocol, or the first threshold is determined by the first device, or the first threshold is configured or indicated by the second device.
  • the second threshold is agreed upon by a protocol, or the second threshold is determined by the first device, or the second threshold is configured or indicated by the second device.
  • the third threshold is agreed upon by a protocol, or the third threshold is determined by the first device, or the third threshold is configured or indicated by the second device.
  • the first device determines the validity of the second AI model according to the fourth information, including:
  • the first device determines that the second AI model is invalid; or,
  • the first device determines that the second AI model is invalid; or,
  • the width of the corresponding confidence interval of the fourth feature is greater than or equal to the fourth
  • the first device determines that the second AI model is invalid
  • N is a positive integer.
  • the fourth threshold is agreed upon by a protocol, or the fourth threshold is determined by the first device, or the fourth threshold is configured or indicated by the second device.
  • the fifth threshold is agreed upon by a protocol, or the fifth threshold is determined by the first device, or the fifth threshold is configured or indicated by the second device.
  • the sixth threshold is agreed upon by a protocol, or the sixth threshold is determined by the first device, or the sixth threshold is configured or indicated by the second device.
  • the first device determines the validity of the second AI model according to the fourth information, including:
  • the first device determines that the second AI model fails; or,
  • the first device determines that the second AI model fails; or,
  • the first device determines that the second AI model has failed;
  • N is a positive integer.
  • the seventh threshold is agreed upon by a protocol, or the seventh threshold is determined by the first device, or the seventh threshold is configured or indicated by the second device.
  • the eighth threshold is agreed upon by a protocol, or the eighth threshold is determined by the first device, or the eighth threshold is configured or indicated by the second device.
  • the ninth threshold is agreed upon by a protocol, or the ninth threshold is determined by the first device, or the ninth threshold is configured or indicated by the second device.
  • the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model.
  • the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results.
  • the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result.
  • the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.
  • FIG8 is a schematic flow chart of a model performance supervision method 500 according to an embodiment of the present application.
  • the model performance supervision method 500 may include at least part of the following contents:
  • the second device receives fourth information from the first device, wherein the fourth information is used to characterize uncertainty or probability distribution of an output of a second AI model, and an input of the second AI model is a third feature;
  • S520 The second device determines the validity of the second AI model according to the fourth information.
  • FIG8 shows the steps or operations of the model performance supervision method 500, but these steps or operations are merely examples, and the embodiments of the present application may also perform other operations or variations of the operations in FIG8.
  • the fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.
  • the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.
  • the second device determines the validity of the second AI model according to the fourth information, including:
  • the second device determines that the second AI model fails; or,
  • the second device determines that the second AI model is invalid; or,
  • the second device determines that the second AI model has failed;
  • N is a positive integer.
  • the second device determines the validity of the second AI model according to the fourth information, including:
  • the second device determines that the second AI model is invalid; or,
  • the second device determines that the second AI model is invalid; or,
  • the second device determines that the second AI model is invalid;
  • N is a positive integer.
  • the second device determines the validity of the second AI model according to the fourth information, including:
  • the second device determines that the second AI model fails; or,
  • the second device determines that the second AI model fails; or,
  • the second device determines that the second AI model is invalid;
  • N is a positive integer.
  • the second device sends fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.
  • the fourth indication information includes an identifier of the second AI model or an identifier of a function associated with the second AI model.
  • the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model.
  • the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results.
  • the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result.
  • the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.
  • the model performance supervision method provided in the embodiment of the present application can be executed by a model performance supervision device, or a processing unit in the model performance supervision device for executing the model performance supervision method. Taking the execution of a model performance supervision method by a supervision device as an example, the model performance supervision device provided in an embodiment of the present application is explained.
  • Fig. 9 shows a schematic block diagram of a model performance monitoring device 600 according to an embodiment of the present application.
  • the model performance monitoring device 600 includes:
  • An acquisition unit 610 is used to acquire first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first artificial intelligence AI model, and the input of the first AI model is a first feature;
  • Processing unit 620 is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.
  • the acquiring unit 610 acquires the first information, including one of the following:
  • the model performance monitoring device 600 further includes: a transceiver unit 630;
  • the processing unit 620 is further configured to determine the validity of the first AI model according to the third information; or,
  • the transceiver unit 630 is used to send the third information to the second device.
  • the transceiver unit 630 is further used to send first indication information to the second device, wherein the first indication information is used to indicate the validity of the first AI model;
  • the transceiver unit 630 is also used to receive second indication information from the second device, wherein the second indication information is used to indicate the validity of the first AI model.
  • the first information includes at least one of the following:
  • Parameters of the probability density distribution of the second feature a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.
  • the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.
  • the third information is used to characterize the mean of the first probabilities of N samples of the first feature, or the third information is used to characterize the mean of the logarithms of the first probabilities of N samples of the first feature;
  • the third information when used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:
  • L1 represents the third information
  • i represents the i-th sample among the N samples
  • yi represents the label corresponding to the ith sample
  • g( ⁇ ) represents the probability density distribution function obeyed by the output of the first AI model.
  • the third information when used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:
  • L 2 represents the third information
  • i represents the i-th sample among the N samples, Indicates the i-th
  • the mean of the probability density distribution of the second feature corresponding to samples represents the standard deviation or variance of the probability density distribution of the second feature corresponding to the i-th sample
  • yi represents the label corresponding to the i-th sample
  • s is a positive integer
  • g( ⁇ ) represents the probability density distribution function obeyed by the output of the first AI model.
  • the third information is used to determine the validity of the first AI model, including:
  • the first AI model is valid
  • the first AI model fails.
  • the third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.
  • the third information is determined based on the following formula:
  • L 3 represents the third information
  • i represents the i-th sample among the N samples
  • x i represents the input information corresponding to the i-th sample
  • y i represents the label corresponding to the i-th sample
  • the third information is used to determine the validity of the first AI model, including:
  • the first AI model is valid
  • the first AI model fails.
  • the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.
  • the third information is determined based on the following formula:
  • L 4 represents the third information
  • i represents the i-th sample among the N samples
  • yi represents the label corresponding to the i-th sample.
  • the third information is used to determine the validity of the first AI model, including:
  • the first AI model is valid
  • the first AI model fails.
  • the transceiver unit may be a communication interface or a transceiver, or an input/output interface of a communication chip or a system on chip.
  • model performance monitoring device 600 may correspond to the first device in the method embodiment of the present application, and the above-mentioned and other operations or functions of each unit in the model performance monitoring device 600 are respectively for realizing the corresponding process of the first device in the method 200 shown in Figure 4. For the sake of brevity, they will not be repeated here.
  • the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, so as to achieve performance supervision of the first AI model.
  • the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result.
  • the performance supervision of the first AI model referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.
  • Fig. 10 shows a schematic block diagram of a model performance monitoring device 700 according to an embodiment of the present application.
  • the model performance monitoring device 700 includes:
  • the transceiver unit 710 is configured to receive third information from the first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence AI model, the input of the first AI model is the first feature, and the second information is the label corresponding to the first feature;
  • the processing unit 720 is used to determine the validity of the first AI model according to the third information.
  • the processing unit 720 is specifically configured to:
  • the transceiver unit 710 is further configured to send the first information to the first device.
  • the transceiver unit 710 is further used to send second indication information to the first device, wherein the second indication information is used to indicate the validity of the first AI model.
  • the first information includes at least one of the following:
  • Parameters of the probability density distribution of the second feature a confidence interval of the second feature, a value of the second feature, a value of the second feature and its probability, and a value of the second feature and its confidence level.
  • the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.
  • the third information is used to characterize the mean of the first probabilities of N samples of the first feature, or the third information is used to characterize the mean of the logarithms of the first probabilities of N samples of the first feature;
  • the first probability is the probability of the label corresponding to each sample in the N samples under the probability density distribution of the second feature, and N is a positive integer.
  • the third information when used to characterize the mean of the first probabilities of N samples of the first feature, the third information is determined based on the following formula:
  • L1 represents the third information
  • i represents the i-th sample among the N samples
  • yi represents the label corresponding to the ith sample
  • g( ⁇ ) represents the probability density distribution function obeyed by the output of the first AI model.
  • the third information when used to characterize the mean of the logarithm of the first probability of N samples of the first feature, the third information is determined based on the following formula:
  • L 2 represents the third information
  • i represents the i-th sample among the N samples
  • yi represents the label corresponding to the i-th sample
  • s is a positive integer
  • g( ⁇ ) represents the probability density distribution function obeyed by the output of the first AI model.
  • the first information when the first information includes a confidence interval of the second feature, the first The third information is used to characterize the ratio of the number of samples whose corresponding labels are within the confidence interval of the second feature among the N samples of the first feature to the total number of samples N, where N is a positive integer.
  • the third information is determined based on the following formula:
  • L 3 represents the third information
  • i represents the i-th sample among the N samples
  • x i represents the input information corresponding to the i-th sample
  • y i represents the label corresponding to the i-th sample
  • the processing unit 720 is specifically configured to:
  • the third information is used to characterize the mean of the weighted distances between the labels corresponding to N samples of the first feature and the value of the second feature, wherein the weighted weight is determined based on the probability or confidence of the second feature, and N is a positive integer.
  • the third information is determined based on the following formula:
  • L 4 represents the third information
  • i represents the i-th sample among the N samples
  • yi represents the label corresponding to the i-th sample.
  • the transceiver unit may be a communication interface or a transceiver, or an input/output interface of a communication chip or a system on chip.
  • model performance monitoring device 700 may correspond to the second device in the method embodiment of the present application, and the above-mentioned and other operations or functions of each unit in the model performance monitoring device 700 are respectively for implementing the corresponding processes of the second device in the method 300 shown in Figure 6. For the sake of brevity, they will not be repeated here.
  • the third information can be determined based on the uncertainty or probability distribution of the output of the first AI model and the label corresponding to the input feature of the first AI model, and the validity of the first AI model can be determined based on the third information, so as to achieve performance supervision of the first AI model.
  • the uncertainty or probability distribution of the output of the first AI model can improve the robustness of the reasoning results of the first AI model. Since the reasoning results of the first AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results. For example, the reasoning results of the first AI model (including relevant information on uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result.
  • the performance supervision of the first AI model referring to the uncertainty or probability distribution of the output of the first AI model and the corresponding label can improve the accuracy of the performance supervision of the first AI model.
  • Fig. 11 shows a schematic block diagram of a model performance monitoring device 800 according to an embodiment of the present application.
  • the model performance monitoring device 800 includes:
  • An acquisition unit 810 is used to acquire fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature;
  • the processing unit 820 is used to determine the validity of the second AI model according to the fourth information; or the transceiver unit 830 is used to send the fourth information to the second device.
  • the acquiring unit 810 acquires the fourth information, including one of the following:
  • the transceiver unit 830 is further used to send third indication information to the second device, wherein the third indication information is used to indicate the validity of the second AI model; or,
  • the transceiver unit 830 is also used to receive fourth indication information from the second device, wherein the fourth indication information is used to indicate the validity of the second AI model.
  • the fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.
  • the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.
  • the processing unit 820 when the fourth information includes parameters of the probability density distribution of the fourth feature, the processing unit 820 is specifically configured to:
  • the proportion of the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold to the total number of samples N is greater than or equal to the third threshold, it is determined that the second AI model is invalid;
  • N is a positive integer.
  • the processing unit 820 when the fourth information includes a confidence interval of the fourth feature, is specifically configured to:
  • the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to the sixth threshold, it is determined that the second AI model is invalid;
  • N is a positive integer.
  • the processing unit 820 when the fourth information includes the value of the fourth feature and its probability or confidence, the processing unit 820 is specifically configured to:
  • the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold accounts for a proportion of the total number of samples N that is greater than or equal to the ninth threshold, it is determined that the second AI model has failed;
  • N is a positive integer.
  • the transceiver unit may be a communication interface or a transceiver, or an input/output interface of a communication chip or a system on chip.
  • model performance monitoring device 800 may correspond to the first device in the method embodiment of the present application, and the above and other operations or functions of each unit in the model performance monitoring device 800 are respectively In order to implement the corresponding process of the first device in the method 400 shown in FIG. 7 , it will not be repeated here for the sake of brevity.
  • the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model.
  • the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results.
  • the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result.
  • the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.
  • Fig. 12 shows a schematic block diagram of a model performance monitoring device 900 according to an embodiment of the present application.
  • the model performance monitoring device 900 includes:
  • the transceiver unit 910 is configured to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature;
  • the processing unit 920 is used to determine the validity of the second AI model according to the fourth information.
  • the transceiver unit 910 is further used to send fourth indication information to the first device, wherein the fourth indication information is used to indicate the validity of the second AI model.
  • the fourth information includes at least one of the following: a parameter of a probability density distribution of the fourth feature, a confidence interval of the fourth feature, a value of the fourth feature, a value of the fourth feature and its probability, and a value of the fourth feature and its confidence.
  • the parameters of the probability density distribution include at least one of the following: a mean or expectation of the probability density distribution, a variance or standard deviation of the probability density distribution, and an indication of the type of the probability density distribution.
  • the processing unit 920 when the fourth information includes parameters of the probability density distribution of the fourth feature, the processing unit 920 is specifically configured to:
  • the number of samples whose variance or standard deviation of the probability density distribution of the corresponding fourth feature is greater than or equal to the first threshold accounts for a proportion of the total number of samples N that is greater than or equal to the third threshold, it is determined that the second AI model is invalid;
  • N is a positive integer.
  • the processing unit 920 when the fourth information includes a confidence interval of the fourth feature, is specifically configured to:
  • the number of samples whose width of the confidence interval of the corresponding fourth feature is greater than or equal to the fourth threshold accounts for a proportion of the total number of samples N that is greater than or equal to the sixth threshold, it is determined that the second AI model is invalid;
  • N is a positive integer.
  • the processing unit 920 when the fourth information includes the value of the fourth feature and its probability or confidence, is specifically configured to:
  • the mean of the probability or confidence of the fourth feature corresponding to the N samples of the third feature is less than or equal to In the case of the eighth threshold, it is determined that the second AI model fails; or,
  • the number of samples whose probability or confidence of the corresponding fourth feature is less than or equal to the seventh threshold accounts for a proportion of the total number of samples N that is greater than or equal to the ninth threshold, it is determined that the second AI model has failed;
  • N is a positive integer.
  • the transceiver unit may be a communication interface or a transceiver, or an input/output interface of a communication chip or a system on chip.
  • model performance monitoring device 900 may correspond to the second device in the method embodiment of the present application, and the above-mentioned and other operations or functions of each unit in the model performance monitoring device 900 are respectively for implementing the corresponding processes of the second device in the method 500 shown in Figure 8. For the sake of brevity, they will not be repeated here.
  • the validity of the second AI model can be determined based on the uncertainty or probability distribution of the output of the second AI model, thereby achieving performance supervision of the second AI model.
  • the uncertainty or probability distribution of the output of the second AI model can improve the robustness of the reasoning results of the second AI model. Since the reasoning results of the second AI model cover multiple possible results and their probability distributions, it is conducive to further processing and utilization of the reasoning results.
  • the reasoning results of the second AI model (including relevant information about uncertainty or probability distribution) are combined with other types of information to obtain a more accurate target result.
  • the uncertainty or probability distribution of the output of the second AI model is referred to, and the validity of the second AI model can be judged. Since no external information is required, it is also easier to implement.
  • the model performance monitoring device provided in the embodiment of the present application can implement the various processes implemented by the method embodiments of Figures 4 to 8 and achieve the same technical effect. To avoid repetition, it will not be repeated here.
  • the embodiment of the present application further provides a communication device 1000, including a processor 1001 and a memory 1002, and the memory 1002 stores a program or instruction that can be run on the processor 1001.
  • the communication device 1000 is a first device
  • the program or instruction is executed by the processor 1001 to implement the various steps of the above-mentioned model performance supervision method 200 or model performance supervision method 400 embodiment, and can achieve the same technical effect.
  • the communication device 1000 is a second device
  • the program or instruction is executed by the processor 1001 to implement the various steps of the above-mentioned model performance supervision method 300 or model performance supervision method 500 embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
  • the embodiment of the present application also provides a terminal, including a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps performed by the first device or the second device in the method embodiment shown in Figures 4 to 8.
  • This terminal embodiment corresponds to the above-mentioned first device or second device side method embodiment, and each implementation process and implementation method of the above-mentioned method embodiment can be applied to the terminal embodiment and can achieve the same technical effect.
  • Figure 14 is a schematic diagram of the hardware structure of a terminal implementing an embodiment of the present application.
  • the terminal 1100 includes but is not limited to: a radio frequency unit 1101, a network module 1102, an audio output unit 1103, an input unit 1104, a sensor 1105, a display unit 1106, a user input unit 1107, an interface unit 1108, a memory 1109 and at least some of the components of a processor 1110.
  • the terminal 1100 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1110 through a power management system, so as to implement functions such as managing charging, discharging, and power consumption management through the power management system.
  • a power source such as a battery
  • the terminal structure shown in FIG14 does not constitute a limitation on the terminal, and the terminal may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently, which will not be described in detail here.
  • the input unit 1104 may include a graphics processing unit (GPU) 11041 and a microphone 11042, and the graphics processor 11041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode.
  • the display unit 1106 may include a display panel 11061, and the display panel 11061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc.
  • the user input unit 1107 includes a touch panel 11071 and at least one of other input devices 11072.
  • the touch panel 11071 is also called a touch screen.
  • the touch panel 11071 may include two parts: a touch detection device and a touch controller.
  • Other input devices 11072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.
  • the RF unit 1101 can transmit the data to the processor 1110 for processing; in addition, the RF unit 1101 can send uplink data to the network side device.
  • the RF unit 1101 includes but is not limited to an antenna, an amplifier, a transceiver, a coupler, a low noise amplifier, a duplexer, etc.
  • the memory 1109 can be used to store software programs or instructions and various data.
  • the memory 1109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instruction required for at least one function (such as a sound playback function, an image playback function, etc.), etc.
  • the memory 1109 may include a volatile memory or a non-volatile memory.
  • the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.
  • the volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM).
  • RAM random access memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • SDRAM synchronous dynamic random access memory
  • DDRSDRAM double data rate synchronous dynamic random access memory
  • ESDRAM enhanced synchronous dynamic random access memory
  • SLDRAM synchronous link dynamic random access memory
  • DRRAM direct memory bus random access memory
  • the processor 1110 may include at least one processing unit; optionally, the processor 1110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1110.
  • the RF unit 1101 is used to obtain first information, wherein the first information is used to characterize the uncertainty or probability distribution of the output of a first AI model, and the input of the first AI model is a first feature; the processor 1110 is used to determine third information based on the first information and the second information, wherein the second information is a label corresponding to the first feature, and the third information is used to determine the validity of the first AI model.
  • the radio frequency unit 1101 is used to receive third information from a first device, wherein the third information is determined based on the first information and the second information, the first information is used to characterize the uncertainty or probability distribution of the output of the first artificial intelligence AI model, the input of the first AI model is a first feature, and the second information is a label corresponding to the first feature; the processor 1110 is used to determine the validity of the first AI model based on the third information.
  • the RF unit 1101 is used to obtain fourth information, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; the processor 1110 is used to determine the validity of the first AI model based on the fourth information; or, the RF unit 1101 is used to send the fourth information to the second device.
  • the radio frequency unit 1101 is used to receive fourth information from the first device, wherein the fourth information is used to characterize the uncertainty or probability distribution of the output of the second artificial intelligence AI model, and the input of the second AI model is the third feature; the processor 1110 is used to determine the validity of the first AI model based on the fourth information.
  • the embodiment of the present application also provides a network side device, including a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the steps of the method embodiment shown in Figures 4 to 8.
  • the network side device embodiment corresponds to the first device or second device method embodiment described above, and each implementation process and implementation method of the method embodiment described above can be applied to the network side device embodiment, and can achieve the same technical effect.
  • the embodiment of the present application also provides a network side device.
  • the network side device 1200 includes: an antenna 121, a radio frequency device 122, a baseband device 123, a processor 124 and a memory 125.
  • the antenna 121 is connected to the radio frequency device 122.
  • the radio frequency device 122 receives information through the antenna 121 and sends the received information to the baseband device 123 for processing.
  • the baseband device 123 processes the information to be sent and sends it to the radio frequency device 122.
  • the radio frequency device 122 processes the received information and sends it out through the antenna 121.
  • the method executed by the network-side device in the above embodiment may be implemented in the baseband device 123, which includes a baseband processor.
  • the baseband device 123 may include, for example, at least one baseband board, on which at least two chips are arranged, as shown in Figure 15, one of the chips is, for example, a baseband processor, which is connected to the memory 125 through a bus interface to call the program in the memory 125 to execute the network device operations shown in the above method embodiment.
  • the network side device may also include a network interface 126, which is, for example, a Common Public Radio Interface (CPRI).
  • CPRI Common Public Radio Interface
  • the network side device 1200 of the embodiment of the present application also includes: instructions or programs stored in the memory 125 and executable on the processor 124.
  • the processor 124 calls the instructions or programs in the memory 125 to execute the method executed by each unit shown in any one of Figures 9 to 12, and achieves the same technical effect. To avoid repetition, it will not be repeated here.
  • the embodiment of the present application further provides a network side device.
  • the network side device 1300 includes: a processor 1301, a network interface 1302, and a memory 1303.
  • the network interface 1302 is, for example, a common public radio interface (CPRI).
  • CPRI common public radio interface
  • the network side device 1300 of the embodiment of the present application also includes: instructions or programs stored in the memory 1303 and executable on the processor 1301.
  • the processor 1301 calls the instructions or programs in the memory 1303 to execute the method executed by each unit shown in any one of Figures 9 to 12, and achieves the same technical effect. To avoid repetition, it will not be repeated here.
  • An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored.
  • a program or instruction is stored.
  • the various processes of the above-mentioned model performance supervision method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
  • the processor is the processor in the terminal described in the above embodiment.
  • the readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
  • the readable storage medium may be a non-transient readable storage medium.
  • An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model performance supervision method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
  • the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
  • An embodiment of the present application further provides a computer program/program product, which is stored in a storage medium.
  • the computer program/program product is executed by at least one processor to implement the various processes of the above-mentioned model performance supervision method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
  • An embodiment of the present application also provides a communication system, including: a first device and a second device, wherein the first device can be used to execute the steps performed by the first device in the model performance supervision method as described above, and the second device can be used to execute the steps performed by the second device in the model performance supervision method as described above.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Computational Mathematics (AREA)
  • Algebra (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Quality & Reliability (AREA)
  • Computer Hardware Design (AREA)
  • Computational Linguistics (AREA)
  • Mobile Radio Communication Systems (AREA)

Abstract

La présente invention concerne le domaine des communications. Sont divulgués un procédé et un appareil de supervision de performance de modèle, et un dispositif, qui peuvent résoudre le problème de supervision de performance pour des modèles d'IA. Le procédé de supervision de performance de modèle dans les modes de réalisation de la présente demande comprend les étapes suivantes : un premier dispositif acquiert des premières informations, les premières informations étant utilisées pour représenter l'incertitude ou la distribution de probabilité d'une sortie d'un premier modèle d'IA, et une entrée du premier modèle d'IA étant une première caractéristique ; et le premier dispositif détermine des troisièmes informations sur la base des premières informations et des deuxièmes informations, les deuxièmes informations étant une étiquette correspondant à la première caractéristique, et les troisièmes informations étant utilisées pour déterminer la validité du premier modèle d'IA.
PCT/CN2024/129460 2023-11-03 2024-11-01 Procédé et appareil de supervision de performance de modèle, et dispositif Pending WO2025092999A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202311467210.7A CN119938459A (zh) 2023-11-03 2023-11-03 模型性能监督方法、装置及设备
CN202311467210.7 2023-11-03

Publications (1)

Publication Number Publication Date
WO2025092999A1 true WO2025092999A1 (fr) 2025-05-08

Family

ID=95532015

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/129460 Pending WO2025092999A1 (fr) 2023-11-03 2024-11-01 Procédé et appareil de supervision de performance de modèle, et dispositif

Country Status (2)

Country Link
CN (1) CN119938459A (fr)
WO (1) WO2025092999A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180307680A1 (en) * 2015-12-29 2018-10-25 Guangzhou Shenma Mobile Information Technology Co., Ltd. Keyword recommendation method and system based on latent dirichlet allocation model
CN111368997A (zh) * 2020-03-04 2020-07-03 支付宝(杭州)信息技术有限公司 神经网络模型的训练方法及装置
CN113627621A (zh) * 2021-08-13 2021-11-09 北京邮电大学 一种面向光网络传输质量回归估计的主动学习方法
CN115796281A (zh) * 2022-11-18 2023-03-14 南方科技大学 一种基于表征波动的不确定性可靠性推理方法及相关设备

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180307680A1 (en) * 2015-12-29 2018-10-25 Guangzhou Shenma Mobile Information Technology Co., Ltd. Keyword recommendation method and system based on latent dirichlet allocation model
CN111368997A (zh) * 2020-03-04 2020-07-03 支付宝(杭州)信息技术有限公司 神经网络模型的训练方法及装置
CN113627621A (zh) * 2021-08-13 2021-11-09 北京邮电大学 一种面向光网络传输质量回归估计的主动学习方法
CN115796281A (zh) * 2022-11-18 2023-03-14 南方科技大学 一种基于表征波动的不确定性可靠性推理方法及相关设备

Also Published As

Publication number Publication date
CN119938459A (zh) 2025-05-06

Similar Documents

Publication Publication Date Title
EP4422248A1 (fr) Procédé de demande de modèle, procédé de traitement de demande de modèle et dispositif associé
CN116567806A (zh) 基于人工智能ai模型的定位方法及通信设备
WO2023098661A1 (fr) Procédé de positionnement et dispositif de communication
US20240323741A1 (en) Measurement method and apparatus, device, and storage medium
WO2023125951A1 (fr) Procédé et appareil de configuration de modèle de communication, et dispositif de communication
US20250227507A1 (en) Ai model processing method and apparatus, and communication device
US20250193833A1 (en) Data collection method and apparatus
US20240354659A1 (en) Method for Updating Model and Communication Device
WO2023040888A1 (fr) Procédé et appareil de transmission de données
WO2024208260A1 (fr) Procédé d'acquisition de données pour prédiction de csi basée sur l'ia, et appareil
WO2024083004A1 (fr) Procédé de configuration de modèle d'ia, terminal et dispositif côté réseau
WO2024125525A1 (fr) Procédé de rapport de puissance de calcul ia, terminal et dispositif côté réseau
WO2025092999A1 (fr) Procédé et appareil de supervision de performance de modèle, et dispositif
WO2024208211A1 (fr) Procédé de surveillance de performances de prédiction de csi, appareil, terminal et dispositif côté réseau
WO2024120447A1 (fr) Procédé et appareil de déclenchement de supervision de modèle, et ue, dispositif côté réseau, support de stockage lisible et système de communication
WO2023155839A1 (fr) Procédé et appareil d'apprentissage en ligne pour un modèle d'ia, ainsi que dispositif de communication et support de stockage lisible
WO2025092998A1 (fr) Procédé de transmission d'informations, appareil et dispositif
WO2025036223A1 (fr) Procédé de rapport d'informations, procédé de réception d'informations et dispositif
WO2025140432A1 (fr) Procédé et appareil de création de compte-rendu d'informations
WO2025108195A1 (fr) Procédé et appareil de détermination de modèle, et dispositif de communication
WO2025185549A1 (fr) Procédé de transmission d'informations, appareil, et dispositif associé
WO2024208167A1 (fr) Procédé de traitement d'informations, appareil de traitement d'informations, terminal et dispositif côté réseau
WO2024235043A1 (fr) Procédé et appareil d'acquisition d'informations, et dispositif de communication
WO2024067280A1 (fr) Procédé et appareil de mise à jour de paramètre de modèle d'ia, et dispositif de communication
WO2024235140A1 (fr) Procédé et appareil de traitement de positionnement, et dispositif

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24884996

Country of ref document: EP

Kind code of ref document: A1