WO2025240861A1

WO2025240861A1 - Analytics and recommendation enhancements for quality of service (qos) policies

Info

Publication number: WO2025240861A1
Application number: PCT/US2025/029765
Authority: WO
Inventors: Achref METHENNI; Ulises Olvera-Hernandez
Original assignee: InterDigital Patent Holdings Inc
Current assignee: InterDigital Patent Holdings Inc
Priority date: 2024-05-16
Filing date: 2025-05-16
Publication date: 2025-11-20
Anticipated expiration: 2026-11-16

Abstract

A first network device may comprise a processor configured to detect a trigger to request a quality of service (QoS) recommendation, send, in response to the trigger, a request to a second network device for the QoS recommendation, receive, from the second network device, a message comprising the QoS recommendation and information related to the QoS recommendation. The information may include, for example, performance measurements for the QoS recommendation. The performance measurements may indicate, for example, at least one of an accuracy of analytics associated with the QoS recommendation, an efficiency of the QoS recommendation, a validity period associated with the QoS recommendation, and/or a sensitivity of the QoS recommendation. The processor may be configured to determine, based on the received QoS recommendation from the second network device, one or more QoS policies.

Description

Analytics and Recommendation Enhancements for Quality of Service (QoS) Policies

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of United States Provisional Application No. 63/648,397 filed on May 16, 2024, the entire contents of which are incorporated herein by reference.

BACKGROUND

[0002] Certain forms of artificial intelligence (Al) and machine learning (ML) are proposed for providing recommendations related to telecommunications network technologies, such as technologies related to LTE and 5G systems (5GS). There are currently challenges with assessing the performance of such recommendations and how the systems provide certain guarantees in terms of performance. The recommendations provided by these systems may rely on different analytics, as well as different logic, to provide them.

SUMMARY

[0003] A first network device may comprise a processor. The processor may be configured to detect a trigger to request a quality of service (QoS) recommendation. The processor may be configured to send, in response to the trigger, a request to a second network device for the QoS recommendation. The request may include, for example, at least one of a recommendation identifier (ID), an analytics ID configured to be used to determine the QoS recommendation, and/or a trigger ID. The processor may be configured to receive, from the second network device, a message comprising the QoS recommendation and information related to the QoS recommendation. The information may include, for example, performance measurements for the QoS recommendation. The performance measurements may indicate, for example, at least one of an accuracy of analytics associated with the QoS recommendation, an efficiency of the QoS recommendation, a validity period associated with the QoS recommendation, and/or a sensitivity of the QoS recommendation. The processor may be configured to determine, based on the received QoS recommendation from the second network device, one or more QoS policies. The first network device may include, for example, a policy control function (PCF).

[0004] The processor may be configured to determine, after setting up the one or more QoS policies, to assess performance of the QoS recommendation and the one or more QoS policies used.

[0005] The processor may be configured to send, to the second network device, performance assessment parameters. The performance assessment parameters may include, for example, accuracy threshold and/or preferred sensitivity level.

[0006] The processor may be configured to determine, after setting up the one or more QoS policies, to assess the sensitivity of the QoS recommendation and the one or more QoS policies used. [0007] The processor may be configured to determine, after using the one or more QoS policies, to assess a QoS sustainability metric of the one or more QoS policies used.

[0008] The QoS sustainability metric may include, for example, a measure of network analytics sensitivity. The trigger to request the QoS recommendation may include, for example, a request from a third network device. [0009] A first network device may be configured to perform a method that includes one or more of the following steps. The method may include detecting a trigger to request a quality of service (QoS) recommendation. The method may include sending, in response to the trigger, a request to a second network device for the QoS recommendation. The request may include, for example, at least one of a recommendation identifier (ID), an analytics ID configured to be used to determine the QoS recommendation, and/or a trigger ID. The method may include receiving, from the second network device, a message comprising the QoS recommendation and information related to the QoS recommendation. The information may include, for example, performance measurements for the QoS recommendation. The performance measurements may indicate, for example, at least one of an accuracy of analytics associated with the QoS recommendation, an efficiency of the QoS recommendation, a validity period associated with the QoS recommendation, and/or a sensitivity of the QoS recommendation. The method may include determining, based on the received QoS recommendation from the second network device, one or more QoS policies. The first network device may include, for example, a policy control function (PCF).

[0010] The method may include determining, after setting up the one or more QoS policies, to assess performance of the QoS recommendation and the one or more QoS policies used.

[0011] The method may include sending, to the second network device, performance assessment parameters. The performance assessment parameters may include, for example, accuracy threshold and/or preferred sensitivity level. [0012] The method may include determining, after setting up the one or more QoS policies, to assess the sensitivity of the QoS recommendation and the one or more QoS policies used.

[0013] The method may include determining, after using the one or more QoS policies, to assess a QoS sustainability metric of the one or more QoS policies used.

[0014] The QoS sustainability metric may include, for example, a measure of network analytics sensitivity. The trigger to request the QoS recommendation may include, for example, a request from a third network device.

BRIEF DESCRIPTION OF THE DRAWINGS

[0015] FIG. 1A is a system diagram illustrating an example communications system in which one or more disclosed embodiments may be implemented.

[0016] FIG. 1 B is a system diagram illustrating an example wireless transmit/receive unit (WTRU) that may be used within the communications system illustrated in FIG. 1A according to an embodiment.

[0017] FIG. 1C is a system diagram illustrating an example radio access network (RAN) and an example core network (CN) that may be used within the communications system illustrated in FIG. 1A according to an embodiment. [0018] FIG. 1D is a system diagram illustrating a further example RAN and a further example ON that may be used within the communications system illustrated in FIG. 1A according to an embodiment.

[0019] FIGs. 2A and 2B are diagrams of an example procedure where the PCF (an example of analytics consumer) is triggered to request or subscribe to analytics that are related to co-existing QoS flows according to an embodiment. [0020] FIGs. 3A and 3B are diagrams of an example procedure of PCF being triggered to request QoS recommendation from a NWDAF according to an embodiment.

DETAILED DESCRIPTION

[0021] FIG. 1A is a diagram illustrating an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDMA), single-carrier FDMA (SC-FDMA), zero-tail uniqueword DFT-Spread OFDM (ZT UW DTS-s OFDM), unique word OFDM (UW-OFDM), resource block-filtered OFDM, filter bank multicarrier (FBMC), and the like.

[0022] As shown in FIG. 1A, the communications system 100 may include wireless transmit/receive units (WTRUs) 102a, 102b, 102c, 102d, a RAN 104/113, a ON 106/115, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and/or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and/or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d, any of which may be referred to as a "station” and/or a "STA”, may be configured to transmit and/or receive wireless signals and may include a user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a subscription-based unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, a hotspot or Mi-Pi device, an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and/or other wireless devices operating in an industrial and/or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and/or industrial wireless networks, and the like. Any of the WTRUs 102a, 102b, 102c and 102d may be interchangeably referred to as a WTRU. Further, any description herein that is described with reference to a UE may be equally applicable to a WTRU (or vice versa . For example, a WTRU may be configured to perform any of the processes or procedures described herein as being performed by a UE (or vice versa). [0023] The communications systems 100 may also include a base station 114a and/or a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the CN 106/115, the Internet 110, and/or the other networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a gNB, a NR NodeB, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and/or network elements.

[0024] The base station 114a may be part of the RAN 104/113, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and/or the base station 114b may be configured to transmit and/or receive wireless signals on one or more carrier frequencies, which may be referred to as a cell (not shown). These frequencies may be in licensed spectrum, unlicensed spectrum, or a combination of licensed and unlicensed spectrum. A cell may provide coverage for a wireless service to a specific geographical area that may be relatively fixed or that may change over time. The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In an embodiment, the base station 114a may employ multiple-input multiple output (MIMO) technology and may utilize multiple transceivers for each sector of the cell. For example, beamforming may be used to transmit and/or receive signals in desired spatial directions.

[0025] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, centimeter wave, micrometer wave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[0026] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104/113 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 115/116/117 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High- Speed Downlink (DL) Packet Access (HSDPA) and/or High-Speed UL Packet Access (HSUPA).

[0027] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and/or LTE-Advanced (LTE-A) and/or LTE-Advanced Pro (LTE-A Pro). [0028] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement a radio technology such as NR Radio Access , which may establish the air interface 116 using New Radio (NR).

[0029] In an embodiment, the base station 114a and the WTRUs 102a, 102b, 102c may implement multiple radio access technologies. For example, the base station 114a and the WTRUs 102a, 102b, 102c may implement LTE radio access and NR radio access together, for instance using dual connectivity (DC) principles. Thus, the air interface utilized by WTRUs 102a, 102b, 102c may be characterized by multiple types of radio access technologies and/or transmissions sent to/from multiple types of base stations (e.g., a eNB and a gNB).

[0030] In other embodiments, the base station 114a and the WTRUs 102a, 102b, 102c may implement radio technologies such as IEEE 802.11 (i.e., Wireless Fidelity (WiFi), IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 1X, CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[0031] The base station 114b in FIG. 1 A may be a wireless router, Home Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, an industrial facility, an air corridor (e.g., for use by drones), a roadway, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In an embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as I EEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, LTE-A Pro, NR etc.) to establish a picocell or femtocell. As shown in FIG. 1 A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the CN 106/115. [0032] The RAN 104/113 may be in communication with the CN 106/115, which may be any type of network configured to provide voice, data, applications, and/or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. The data may have varying quality of service (QoS) requirements, such as differing throughput requirements, latency requirements, error tolerance requirements, reliability requirements, data throughput requirements, mobility requirements, and the like. The CN 106/115 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and/or perform high-level security functions, such as user authentication. Although not shown in FIG. 1 A, it will be appreciated that the RAN 104/113 and/or the CN 106/115 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104/113 or a different RAT. For example, in addition to being connected to the RAN 104/113, which may be utilizing a NR radio technology, the CN 106/115 may also be in communication with another RAN (not shown) employing a GSM, UMTS, CDMA 2000, WiMAX, E-UTRA, or WiFi radio technology. [0033] The CN 106/115 may also serve as a gateway for the WTRUs 102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and/or the other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and/or the internet protocol (IP) in the TCP/IP internet protocol suite. The networks 112 may include wired and/or wireless communications networks owned and/or operated by other service providers. For example, the networks 112 may include another CN connected to one or more RANs, which may employ the same RAT as the RAN 104/113 or a different RAT.

[0034] Some or all of the WTRUs 102a, 102b, 102c, 102d in the communications system 100 may include multimode capabilities (e.g., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links). For example, the WTRU 102c shown in FIG. 1A may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[0035] FIG. 1B is a system diagram illustrating an example WTRU 102. As shown in FIG. 1 B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keypad 126, a display/touchpad 128, non-removable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and/or other peripherals 138, among others. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[0036] The processor 118 may be a general-purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit/receive element 122. While FIG. 1 B depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[0037] The transmit/receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g, the base station 114a) over the air interface 116. For example, in one embodiment, the transmit/receive element 122 may be an antenna configured to transmit and/or receive RF signals. In an embodiment, the transmit/receive element 122 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit/receive element 122 may be configured to transmit and/or receive both RF and light signals. It will be appreciated that the transmit/receive element 122 may be configured to transmit and/or receive any combination of wireless signals

[0038] Although the transmit/receive element 122 is depicted in FIG. 1B as a single element, the WTRU 102 may include any number of transmit/receive elements 122. More specifically, the WTRU 102 may employ Ml MO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit/receive elements 122 (e.g, multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[0039] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 122 and to demodulate the signals that are received by the transmit/receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as NR and IEEE 802.11 , for example.

[0040] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker/microphone 124, the keypad 126, and/or the display /touchpad 128 (e.g, a liquid crystal display (LCD) display unit or organic light-emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker/microphone 124, the keypad 126, and/or the display /touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and/or the removable memory 132. The non-removable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[0041] The processor 118 may receive power from the power source 134, and may be configured to distribute and/or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g, nickel-cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[0042] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g, longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station (e.g, base stations 114a, 114b) and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment. [0043] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs and/or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, a Virtual Reality and/or Augmented Reality (VR/AR) device, an activity tracker, and the like. The peripherals 138 may include one or more sensors, the sensors may be one or more of a gyroscope, an accelerometer, a hall effect sensor, a magnetometer, an orientation sensor, a proximity sensor, a temperature sensor, a time sensor; a geolocation sensor; an altimeter, a light sensor, a touch sensor, a magnetometer, a barometer, a gesture sensor, a biometric sensor, and/or a humidity sensor.

[0044] The WTRU 102 may include a full duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for both the UL (e.g., for transmission) and downlink (e.g., for reception) may be concurrent and/or simultaneous. The full duplex radio may include an interference management unit 139 to reduce and or substantially eliminate self-interference via either hardware (e.g., a choke) or signal processing via a processor (e.g., a separate processor (not shown) or via processor 118). In an embodiment, the WRTU 102 may include a half-duplex radio for which transmission and reception of some or all of the signals (e.g., associated with particular subframes for either the UL (e.g., for transmission) or the downlink (e.g, for reception)). [0045] FIG. 1C is a system diagram illustrating the RAN 104 and the CN 106 according to an embodiment. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the CN 106.

[0046] The RAN 104 may include eNode-Bs 160a, 160b, 160c, though it will be appreciated that the RAN 104 may include any number of eNode-Bs while remaining consistent with an embodiment. The eNode-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs 160a, 160b, 160c may implement MIMO technology. Thus, the eNode-B 160a, for example, may use multiple antennas to transmit wireless signals to, and/or receive wireless signals from, the WTRU 102a.

[0047] Each of the eNode-Bs 160a, 160b, 160c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and/or DL, and the like. As shown in FIG. 1 C, the eNode-Bs 160a, 160b, 160c may communicate with one another over an X2 interface.

[0048] The CN 106 shown in FIG. 1C may include a mobility management entity (MME) 162, a serving gateway (SGW) 164, and a packet data network (PDN) gateway (or PGW) 166. While each of the foregoing elements are depicted as part of the CN 106, it will be appreciated that any of these elements may be owned and/or operated by an entity other than the CN operator. [0049] The MME 162 may be connected to each of the eNode-Bs 162a, 162b, 162c in the RAN 104 via an S1 interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation/deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 may provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM and/or WCDMA.

[0050] The SGW 164 may be connected to each of the eNode Bs 160a, 160b, 160c in the RAN 104 via the S1 interface. The SGW 164 may generally route and forward user data packets to/from the WTRUs 102a, 102b, 102c. The SGW 164 may perform other functions, such as anchoring user planes during Inter-eNode B handovers, triggering paging when DL data is available for the WTRUs 102a, 102b, 102c, managing and storing contexts of the WTRUs 102a, 102b, 102c, and the like.

[0051] The SGW 164 may be connected to the PGW 166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[0052] The CN 106 may facilitate communications with other networks. For example, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. For example, the CN 106 may include, or may communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 106 and the PSTN 108. In addition, the CN 106 may provide the WTRUs 102a, 102b, 102c with access to the other networks 112, which may include other wired and/or wireless networks that are owned and/or operated by other service providers.

[0053] Although the WTRU is described in FIGS. 1A-1 D as a wireless terminal, it is contemplated that in certain representative embodiments that such a terminal may use (e.g., temporarily or permanently) wired communication interfaces with the communication network.

[0054] In representative embodiments, the other network 112 may be a WLAN.

[0055] A WLAN in Infrastructure Basic Service Set (BSS) mode may have an Access Point (AP) for the BSS and one or more stations (STAs) associated with the AP. The AP may have an access or an interface to a Distribution System (DS) or another type of wired/wireless network that carries traffic in to and/or out of the BSS. Traffic to STAs that originates from outside the BSS may arrive through the AP and may be delivered to the STAs. Traffic originating from STAs to destinations outside the BSS may be sent to the AP to be delivered to respective destinations. Traffic between STAs within the BSS may be sent through the AP, for example, where the source STA may send traffic to the AP and the AP may deliver the traffic to the destination STA. The traffic between STAs within a BSS may be considered and/or referred to as peer-to-peer traffic. The peer-to-peer traffic may be sent between (e.g., directly between) the source and destination STAs with a direct link setup (DLS). In certain representative embodiments, the DLS may use an 802.11e DLS or an 802.11z tunneled DLS (TDLS). A WLAN using an Independent BSS (IBSS) mode may not have an AP, and the STAs (e.g., all of the STAs) within or using the IBSS may communicate directly with each other. The IBSS mode of communication may sometimes be referred to herein as an "ad-hoc” mode of communication.

[0056] When using the 802.11 ac infrastructure mode of operation or a similar mode of operations, the AP may transmit a beacon on a fixed channel, such as a primary channel. The primary channel may be a fixed width (e.g, 20 MHz wide bandwidth) or a dynamically set width via signaling. The primary channel may be the operating channel of the BSS and may be used by the STAs to establish a connection with the AP. In certain representative embodiments, Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA) may be implemented, for example, in 802.11 systems. For CSMA/CA, the STAs (e.g., every STA), including the AP, may sense the primary channel. If the primary channel is sensed/detected and/or determined to be busy by a particular STA, the particular STA may back off. One STA (e.g., only one station) may transmit at any given time in a given BSS.

[0057] High Throughput (HT) STAs may use a 40 MHz wide channel for communication, for example, via a combination of the primary 20 MHz channel with an adjacent or nonadjacent 20 MHz channel to form a 40 MHz wide channel.

[0058] Very High Throughput (VHT) STAs may support 20MHz, 40 MHz, 80 MHz, and/or 160 MHz wide channels. The 40 MHz, and/or 80 MHz, channels may be formed by combining contiguous 20 MHz channels. A 160 MHz channel may be formed by combining 8 contiguous 20 MHz channels, or by combining two non-contiguous 80 MHz channels, which may be referred to as an 80+80 configuration. For the 80+80 configuration, the data, after channel encoding, may be passed through a segment parser that may divide the data into two streams. Inverse Fast Fourier Transform (IFFT) processing, and time domain processing, may be done on each stream separately. The streams may be mapped on to the two 80 MHz channels, and the data may be transmitted by a transmitting STA. At the receiver of the receiving STA, the above-described operation for the 80+80 configuration may be reversed, and the combined data may be sent to the Medium Access Control (MAC).

[0059] Sub 1 GHz modes of operation are supported by 802.11 af and 802.11 ah. The channel operating bandwidths, and carriers, are reduced in 802.11 af and 802.11 ah relative to those used in 802.11 n, and 802.11ac. 802.11 af supports 5 MHz, 10 MHz and 20 MHz bandwidths in the TV White Space (TVWS) spectrum, and 802.11 ah supports 1 MHz, 2 MHz, 4 MHz, 8 MHz, and 16 MHz bandwidths using non-TVWS spectrum. According to a representative embodiment, 802.11ah may support Meter Type Control/Machine-Type Communications, such as MTC devices in a macro coverage area. MTC devices may have certain capabilities, for example, limited capabilities including support for (e.g, only support for) certain and/or limited bandwidths. The MTC devices may include a battery with a battery life above a threshold (e.g., to maintain a very long battery life).

[0060] WLAN systems, which may support multiple channels, and channel bandwidths, such as 802.11 n, 802.11 ac, 802.11 af, and 802.11 ah, include a channel which may be designated as the primary channel. The primary channel may have a bandwidth equal to the largest common operating bandwidth supported by all STAs in the BSS. The bandwidth of the primary channel may be set and/or limited by a STA, from among all STAs in operating in a BSS, which supports the smallest bandwidth operating mode. In the example of 802.11 ah, the primary channel may be 1 MHz wide for STAs (e.g, MTC type devices) that support (e.g, only support) a 1 MHz mode, even if the AP, and other STAs in the BSS support 2 MHz, 4 MHz, 8 MHz, 16 MHz, and/or other channel bandwidth operating modes. Carrier sensing and/or Network Allocation Vector (NAV) settings may depend on the status of the primary channel. If the primary channel is busy, for example, due to a STA (which supports only a 1 MHz operating mode), transmitting to the AP, the entire available frequency bands may be considered busy even though a majority of the frequency bands remains idle and may be available.

[0061] In the United States, the available frequency bands, which may be used by 802.11 ah, are from 902 MHz to 928 MHz. In Korea, the available frequency bands are from 917.5 MHz to 923.5 MHz. In Japan, the available frequency bands are from 916.5 MHz to 927.5 MHz. The total bandwidth available for 802.11 ah is 6 MHz to 26 MHz depending on the country code.

[0062] FIG. 1D is a system diagram illustrating the RAN 113 and the CN 115 according to an embodiment. As noted above, the RAN 113 may employ an NR radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 113 may also be in communication with the CN 115.

[0063] The RAN 113 may include gNBs 180a, 180b, 180c, though it will be appreciated that the RAN 113 may include any number of gNBs while remaining consistent with an embodiment. The gNBs 180a, 180b, 180c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the gNBs 180a, 180b, 180c may implement MIMO technology. For example, gNBs 180a, 108b may utilize beamforming to transmit signals to and/or receive signals from the gNBs 180a, 180b, 180c. Thus, the gNB 180a, for example, may use multiple antennas to transmit wireless signals to, and/or receive wireless signals from, the WTRU 102a. In an embodiment, the gNBs 180a, 180b, 180c may implement carrier aggregation technology. For example, the gNB 180a may transmit multiple component carriers to the WTRU 102a (not shown). A subset of these component carriers may be on unlicensed spectrum while the remaining component carriers may be on licensed spectrum. In an embodiment, the gNBs 180a, 180b, 180c may implement Coordinated Multi-Point (CoMP) technology. For example, WTRU 102a may receive coordinated transmissions from gNB 180a and gNB 180b (and/or gNB 180c).

[0064] The WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using transmissions associated with a scalable numerology. For example, the OFDM symbol spacing and/or OFDM subcarrier spacing may vary for different transmissions, different cells, and/or different portions of the wireless transmission spectrum. The WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using subframe or transmission time intervals (TTIs) of various or scalable lengths (e.g., containing varying number of OFDM symbols and/or lasting varying lengths of absolute time). [0065] The gNBs 180a, 180b, 180c may be configured to communicate with the WTRUs 102a, 102b, 102c in a standalone configuration and/or a non-standalone configuration. In the standalone configuration, WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c without also accessing other RANs (e.g, such as eNode- Bs 160a, 160b, 160c). In the standalone configuration, WTRUs 102a, 102b, 102c may utilize one or more of gNBs 180a, 180b, 180c as a mobility anchor point. In the standalone configuration, WTRUs 102a, 102b, 102c may communicate with gNBs 180a, 180b, 180c using signals in an unlicensed band. In a non-standalone configuration WTRUs 102a, 102b, 102c may communicate with/connect to gNBs 180a, 180b, 180c while also communicating with/connecting to another RAN such as eNode-Bs 160a, 160b, 160c. For example, WTRUs 102a, 102b, 102c may implement DC principles to communicate with one or more gNBs 180a, 180b, 180c and one or more eNode-Bs 160a, 160b, 160c substantially simultaneously. In the non-standalone configuration, eNode-Bs 160a, 160b, 160c may serve as a mobility anchor for WTRUs 102a, 102b, 102c and gNBs 180a, 180b, 180c may provide additional coverage and/or throughput for servicing WTRUs 102a, 102b, 102c.

[0066] Each of the gNBs 180a, 180b, 180c may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the UL and/or DL, support of network slicing, dual connectivity, interworking between NR and E-UTRA, routing of user plane data towards User Plane Function (UPF) 184a, 184b, routing of control plane information towards Access and Mobility Management Function (AMF) 182a, 182b and the like. As shown in FIG. 1 D, the gNBs 180a, 180b, 180c may communicate with one another over an Xn interface.

[0067] The CN 115 shown in FIG. 1 D may include at least one AMF 182a, 182b, at least one UPF 184a, 184b, at least one Session Management Function (SMF) 183a, 183b, and possibly a Data Network (DN) 185a, 185b. While each of the foregoing elements are depicted as part of the CN 115, it will be appreciated that any of these elements may be owned and/or operated by an entity other than the CN operator.

[0068] The AMF 182a, 182b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N2 interface and may serve as a control node. For example, the AMF 182a, 182b may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, support for network slicing (e.g, handling of different PDU sessions with different requirements), selecting a particular SMF 183a, 183b, management of the registration area, termination of NAS signaling, mobility management, and the like. Network slicing may be used by the AMF 182a, 182b in order to customize CN support for WTRUs 102a, 102b, 102c based on the types of services being utilized WTRUs 102a, 102b, 102c. For example, different network slices may be established for different use cases such as services relying on ultra-reliable low latency (URLLC) access, services relying on enhanced massive mobile broadband (eMBB) access, services for machine type communication (MTC) access, and/or the like. The AMF 162 may provide a control plane function for switching between the RAN 113 and other RANs (not shown) that employ other radio technologies, such as LTE, LTE-A, LTE-A Pro, and/or non-3GPP access technologies such as WiFi. [0069] The SMF 183a, 183b may be connected to an AMF 182a, 182b in the CN 115 via an N11 interface. The SMF 183a, 183b may also be connected to a UPF 184a, 184b in the CN 115 via an N4 interface. The SMF 183a, 183b may select and control the UPF 184a, 184b and configure the routing of traffic through the UPF 184a, 184b. The SMF 183a, 183b may perform other functions, such as managing and allocating WTRU IP address, managing PDU sessions, controlling policy enforcement and QoS, providing downlink data notifications, and the like. A PDU session type may be IP-based, non-IP based, Ethernet-based, and the like.

[0070] The UPF 184a, 184b may be connected to one or more of the gNBs 180a, 180b, 180c in the RAN 113 via an N3 interface, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The UPF 184, 184b may perform other functions, such as routing and forwarding packets, enforcing user plane policies, supporting multi-homed PDU sessions, handling user plane QoS, buffering downlink packets, providing mobility anchoring, and the like.

[0071] The CN 115 may facilitate communications with other networks. For example, the CN 115 may include, or may communicate with, an IP gateway e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the CN 115 and the PSTN 108. In addition, the CN 115 may provide the WTRUs 102a, 102b, 102c with access to the other networks 112, which may include other wired and/or wireless networks that are owned and/or operated by other service providers. In one embodiment, the WTRUs 102a, 102b, 102c may be connected to a local Data Network (DN) 185a, 185b through the UPF 184a, 184b via the N3 interface to the UPF 184a, 184b and an N6 interface between the UPF 184a, 184b and the DN 185a, 185b.

[0072] In view of Figures 1A-1 D, and the corresponding description of Figures 1A-1 D, one or more, or all, of the functions described herein with regard to one or more of: WTRU 102a-d, Base Station 114a-b, eNode-B 160a-c, MME 162, SGW 164, PGW 166, gNB 180a-c, AMF 182a-ab, UPF 184a-b, SMF 183a-b, DN 185a-b, and/or any other device(s) described herein, may be performed by one or more emulation devices (not shown). The emulation devices may be one or more devices configured to emulate one or more, or all, of the functions described herein. For example, the emulation devices may be used to test other devices and/or to simulate network and/or WTRU functions.

[0073] The emulation devices may be designed to implement one or more tests of other devices in a lab environment and/or in an operator network environment. For example, the one or more emulation devices may perform the one or more, or all, functions while being fully or partially implemented and/or deployed as part of a wired and/or wireless communication network in order to test other devices within the communication network. The one or more emulation devices may perform the one or more, or all, functions while being temporarily implemented/deployed as part of a wired and/or wireless communication network. The emulation device may be directly coupled to another device for purposes of testing and/or may perform testing using over-the-air wireless communications. [0074] The one or more emulation devices may perform the one or more, including all, functions while not being implemented/deployed as part of a wired and/or wireless communication network. For example, the emulation devices may be utilized in a testing scenario in a testing laboratory and/or a non-deployed {e.g, testing) wired and/or wireless communication network in order to implement testing of one or more components. The one or more emulation devices may be test equipment. Direct RF coupling and/or wireless communications via RF circuitry {e.g., which may include one or more antennas) may be used by the emulation devices to transmit and/or receive data. [0075] In illustrated embodiments herein, an analytics and/or recommendations consumer, such as a policy control function (PCF), may be configured to be triggered to request {e.g, enhanced) analytics or recommendations related to policies and/or QoS policies. The PCF may include, in addition to a recommendation ID and/or analytics ID, a trigger ID, and/or performance assessment information, including recommendation or analytics sensitivity, accuracy. In one example embodiment, the PCF may receive the requested analytics or recommendations and determine to use them to decide and setup QoS and related policies. The PCF can determine after the setup of the policies the performance of the analytics or recommendation used for the final QoS and policies determination. In addition, the PCF may interact with other network functions such as NWDAF or performance evaluator to obtain/measure such performance. The PCF may determine to assess the performance on its own. For example, the PCF may determine to assess the QoS sustainability of the QoS flows of interest.

[0076] In one example, a method is provided in which a PCF is triggered to perform Optimized QoS policy determination using analytics and/or recommendations. The PCF may send a request to an analytics or recommendations producer to obtain recommendations {e.g, for QoS policies) or analytics related to QoS policies. The request may include, in addition to recommendation ID, analytics IDs of the analytics desired or analytics use to perform the recommendation ID, and trigger ID value and performance assessment information. The performance assessment information may include a required and/or preferred accuracy, sensitivity or performance level of the analytics or the recommendation and the analytics this recommendation requires. The performance assessment information may indicate the use of QoS sustainability as a performance evaluation metric for the recommendation The PCF may receive analytics results or recommendation results from the analytics or recommendations producer. The result may also include a measurement of the performance of the analytics or a performance measurement of the recommendation and the analytics it used, e.g, a level of sensitivity of the result to input parameters variations {e.g, time period variation), or accuracy of the recommendations or analytics. Additionally, the result may include information about the measurement of QoS sustainability of the determined recommendation (QoS policies), for a certain time period.

[0077] In the example, the PCF uses the received analytics or recommendations (for QoS policies) to determine QoS parameters/ and setup related policies. The PCF may determine, after using the analytics or recommendations and setting up policies, the performance of the network using those policies. The PCF may interact with other network entities {e.g, performance evaluator), or determine to perform the performance evaluation on its own. The consumer may determine the sensitivity of the recommendation or the QoS sustainability of the QoS policies that were set up in the network.

[0078] As contemplated herein, a certain logical entity that can be a Network Data Analytics Function (NWDAF) may perform recommendations. The NWDAF may obtain certain analytics related to the QoS or service performance. The NWDAF may determine to provide QoS recommendations or QoS policies recommendations.

[0079] Certain challenges may be provided in determining the metrics that may be used to assess the performance of such recommendations and how the telecommunications systems may allow for recommendations subscriber to request and obtain recommendations with certain guarantee in terms of performance, particularly since the recommendations may rely on different analytics as well as a certain logic to provide them. Some embodiments may be implemented to enhance existing network analytics and/or implement analytics that may help an analytics consumer, such as PCF, determine optimized policies or QoS parameters.

[0080] In some embodiments, data related to QoS flows is collected to be able to provide these analytics. However, when there are two or more QoS flows in a certain packet data unit (PDU) Session, these QoS flows may have an impact on each other, in terms of performance, depending on the different QoS parameters assigned to these QoS flows. Certain analytics may be implemented to account for correlation between QoS flows when training ML models to provide these analytics.

[0081] In some examples, an apparatus may be provided that may include at least one processor configured to detect a trigger to request a quality of service (QoS) recommendation. The processor may be configured to send, in response to the trigger, a request configured to request the QoS recommendation. The request may include, for example, at least one of a recommendation identifier, an analytics identifier configured to be used to determine the recommendation, and/or an identifier of the trigger. The processor may be configured to receive a message comprising the QoS recommendation and/or information related to the QoS recommendation. The information may include, for example, performance measurements for the recommendation that indicate at least one of an accuracy of analytics associated with the recommendation, an efficiency of the recommendation, a validity period associated with the recommendation, and/or a sensitivity of the recommendation. The processor may be configured to determine, based on the information related to the QoS recommendation, a performance of the received recommendation. The processor may be configured to determine, based on the performance of the received recommendation, one or more QoS policies. The processor may be configured to determine, after setting up QoS policies, to assess the performance of the recommendation and the QoS policies used.

[0082] Network Analytics sensitivity as a way to assess Analytics Performance may be implemented. In one aspect, systems and methods are disclosed herein to enhance the telecommunications systems, such as 5GS, to be able to use ML model sensitivity and/or analytics sensitivity as a performance measurement method. For instance, when a Network Analytics consumer of a network analytics service requests or subscribes to network analytics from a network analytics producer, the network analytics consumer provides certain input parameters to assist in determining the desired analytics. One example of such input parameters is a target period, which may be used for analytics reporting. The analytics consumer may use this parameter to indicate the time or time window in the past of future for which the analytics are requested.

[0083] The analytics consumer may also provide parameters related to the performance of the analytics or ML model. One existing metric is the accuracy of the analytics. For example, an analytics consumer may request analytics with a specific accuracy threshold, and/or a preferred accuracy level for the analytics. In addition to the accuracy of the analytics or ML models, the sensitivity of the ML model and/or analytics to input parameters may be considered, where the sensitivity values may indicate how network analytics vary during a set period or interval. In an example of an analytics target period, an analytics consumer may be interested in knowing whether and how the analytics values for a target period [x, x+T] vary when the target period varies slightly (e.g, for a period [x, x+T+delta] or [x+beta, x+ T+alpha]). The analytics consumer may be interested in obtaining analytics that are not very sensitive to variation in the target analytics period.

[0084] Other input parameters may be used to assess the sensitivity of the predictions and/or analytics to these parameters. According to examples, the analytics consumer may include a value delta that can be used for the sensitivity assessment. For instance, the analytics consumer may indicate that it wants analytics, in particular, analytics that have low sensitivity to changes in the values of a time window or target analytics period, when the length of the target period varies slightly, with a variation of target period not exceeding a value delta. The analytics producer may use this parameter to assess how robust the analytics are when the input value (e.g., time window) varies slightly with a variation up to delta. In that case, the analytics consumer may, similarly to the accuracy parameters, include a sensitivity value and/or a threshold for the network analytics. The network analytics consumer may also provide a preferred level of sensitivity for the analytics. For example, it may request analytics with a low sensitivity and/or high robustness, or medium sensitivity and so on.

[0085] The analytics consumer may also provide an output strategy. A robust and/or sensitivity compliant strategy may indicate that the analytics should be produced and provided when a certain or preferred level of sensitivity is satisfied. In one embodiment, an analytics producer such as NWDAF may use the analytics robustness and/or sensitivity related performance criteria, to perform ML model training and determining analytics that satisfy the sensitivity level (e.g., low sensitivity) of the analytics.

[0086] The NWDAF may use one or more performance criteria for the analytics. For example, the NWDAF may use both accuracy and sensitivity of analytics to assess analytics performance. In the context of this example, QoS sustainability may be considered as a special case of network analytics sensitivity, and as such, QoS sustainability may be used as a means to gauge the impact on the network performance as a result of a policy control setting. In other words, a network analytics consumer may request from a network analytics producer, feedback on network performance (e.g., recommendation on QoS performance) as a result of the current QoS parameter and/or policy settings configured by the PCF. [0087] The network analytics producer may then provide a measure of the QoS sustainability experience over a period of time, along with level of accuracy values, relative to the QoS sustainability observed during the requested time interval. In addition, the network analytics consumer may also indicate in the performance feedback request, the analytics or combination of analytics that need to be used when determining the QoS sustainability, over a period of time, and possibility for specific level of accuracy. This may be applied using any analytic that may be measured over a period of time, and for which change in value may be expected to remain within certain threshold as specified by the network analytics consumer (e.g, low sensitivity, medium sensitivity, and/or high sensitivity).

[0088] In some examples, an analytics or recommendations consumer, such as a PCF, may be configured to be triggered to request (e.g, enhanced) analytics or recommendations related to policies/QoS policies. The PCF may include, in addition to a recommendation ID and/or analytics ID, a trigger ID, and/or performance assessment information, including recommendation or analytics sensitivity, accuracy. In one embodiment, the PCF can receive the requested analytics or recommendations and determine to use them to decide and setup QoS and related policies. The PCF can determine after the setup of the policies the performance of the analytics or recommendation used for the final QoS and policies determination.

[0089] Enhancing analytics with inter-QoS flows dependency and/or correlation may be implemented. In another aspect, systems and methods are described to enhance the telecommunications system (e.g, 5GS) to enable the use of network analytics for policies or QoS determination. In certain embodiments, when two QoS flows (e.g, flows 1 and 2) are established within a certain PDU Session, the two QoS flows may play a role in impacting each other, and this may impact the QoS parameters of these QoS flows as well as the modification and/or release of these QoS flows. For example, the allocation and retention policy (ARP) value, the guaranteed flow bit rate (GFBR), maximum flow bit rate (MFBR) and QoS flow priority level that characterize a QoS flow 1, may impact on another QoS flow 2 within the same PDU Session, as described above and herein.

[0090] For some analytics (e.g, related to QoS flows, service experience, QoS sustainability, and so on) the relevant input data related to the two QoS flows may be included in the dataset sample and can be used by an NWDAF for ML model training, to allow for accounting for the impact and/or correlation between the two QoS flows, and their parameters, in the output that is desired.

[0091] In an illustrative example using input data related to two coexisting QoS flows to train a model that provides statistics and/or predictions related to service experience and QoS sustainability, when certain traffic flows are carried by these QoS flows, the input dataset is defined. One dataset sample may include as entries or features, QoS flowl ID, ARP1, GFBR1, MFBR1, priority levell, maximum data burst volume one (MDBV1), as well as QoS flow2 ID, ARP2, GFBR2, MFBR2, priority Ievel2, MDBV2. Such information may be collected from a network function like the PCF.

[0092] The sample may also include timestamp for the data sampling time, WTRU ID, PDU Session ID. The sample may also include information related to QoS flowl retainability, QoS flow2 retainability, QoS flowl modifications and/or release measurement, QoS flow2 modifications and/or release measurement. Such measurements may be collected from an entity such as, for example, the operations, administration and maintenance (OAM).

[0093] The sample may also include data related to service experience, for example, to traffic flow 1 carried by QoS flowl and traffic flow2 carried by QoS flow2. The two traffic flows may or may not be associated. For example, the two traffic flows may carry traffic related to the same application ID, or the traffic flows relate to two application (App) IDs, App ID1 and App ID2, that belong to the same application service provider (ASP).

[0094] Analytics may provide output in the form of statistics or predictions for service experience, QoS sustainability, and/or perhaps other aspects for the two QoS flows. For example, the output may represent statistics about observed service experience (OSE) 1 for a traffic carried in QoS flowl and observed service experience 2, observed for a traffic that is carried by QoS flow2, when both QoS flows coexists and have certain QoS parameters. [0095] The output related to the performance of the two QoS flows may be in the form of two separate outputs (e.g., (OSE1 , 0SE2) or (QoS sustainability 1 , QoS sustainability2)), or as a function of the two outputs (e.g., 0SE1 +0SE2 or OSETQoS sustainabilityl + OSE2*QoS sustainability2).The output here may represent statistics and/or predictions about the two QoS flows, and/or traffic flows carried by the two QoS flows, when the QoS flows coexist in the PDU Session. This may provide more informed analytics, that can allow better policy decisions or optimizations. A similar approach may be considered for three or more QoS flows and, in certain examples, may use information related to multiple co-existing QoS flows within a PDU Session.

[0096] FIGs. 2A and 2B shows an example procedure 200 where the PCF 205 (e.g., an example of analytics consumer) is triggered to request and/or subscribe to analytics that are related to co-existing QoS flows, and where the manner in which the PCF 205 may use such analytics to make optimized QoS determination. In example procedure 200, a traffic configured with round-trip latency requirement is shown. According to the example, two traffic flows may be bound by a round trip (RT) latency requirement. The PCF 205 may use the RT latency requirement to determine uplink (UL) packet delay budget one (PDB1) value for QoS flow 1 and downlink (DL) PDB2 for QoS flow2 (e.g., UL PDB1 + DL PDB2 is less than or equal to a predetermined threshold). The PCF 205 may use the enhanced analytics to determine the UL PDB1 and DL PDB2 combination that is most optimal, hence reducing policy and charging control (PCC) rule updates due to re-adjusting of UL PDB and DL PDB values.

[0097] Referring to FIGs. 2A and 2B, at 202, the application function (AF) 201 may send a request to the network exposure function (NEF) 203 to reserve resources for an AF session related to some traffic (e.g., extended reality and media services (XRM) traffic). The AF 201 may provide traffic flow information, for example for traffic flow 1 (e.g., UL pose information traffic), and traffic flow 2 (e.g., DL video traffic). The AF 201 may include a round-trip latency requirement for the round-trip delay that related both traffic flow 1 and traffic flow 2. The AF 201 may provide other QoS parameters (e.g., packet error rate (PER) or required data rate) to the NEF for each traffic flow. At 204, the NEF 203 may authorize the AF 201 request and forwards the request to the PCF 205. [0098] At 206, the PCF 205 may be triggered to perform optimized QoS policies procedure The PCF 205 may determine, based on the RT latency requirement provided by AF 201, service flow descriptions for traffic flow 1 and traffic flow 2, and other QoS requirements, that the two traffic flows may be carried in two QoS flows: QoS flow 1 for traffic flow 1 (e.g., pose information) and QoS flow 2 for traffic flow 2. The PCF 205 may use the RT latency requirement to determine a RT delay requirement for the UL/DL traffic round-trip delay. This determination may trigger the PCF 205 to use an optimized QoS policies procedure, in order to select stable or optimal UL PDB1 and DL PDB2 values. The PCF 205 may also use other factors to trigger optimized policies determination.

[0099] At 208, the PCF 205 may send a request to the NWDAF to obtain analytics related to inter-QoS flow performance. As shown in procedure 200, the PCF 205 may request an analytics ID for “inter-QoS flow sustainability and service experience". As set forth herein, the analytics ID may be configured as two separate outputs related to each QoS flow, or it may be a combined function of the performance of the two QoS flows. A joint analytics term may also be used such that the analytics may relate to both QoS flows or provide two separate analytics for each interrelated QoS flow. The PCF 205 may also include in the analytics request: WTRU ID as target for analytics reporting, RT delay requirement value, other QoS parameters (e.g., other than PDBs), QoS flow IDs, application ID(s), single network slice selection assistance information (S-NSSAI), data network name (DNN) and so on. The analytics request may also indicate an analytics target period. The PCF 205 may further include parameters related to the performance of the analytics. This may include analytics sensitivity parameters, such as referred analytics sensitivity level (e.g., low sensitivity or high robustness), an output strategy that prioritizes analytics sensitivity when providing the analytics.

[0100] At 210, once the NWDAF 207 receives the analytics request from the PCF 205, the NWDAF 207 may determine whether an existing ML model that provides the desired inter-QoS flow analytics and that satisfies the performance requirements is available. If the ML model is available, the NWDAF 207 may use this model to determine analytics. Otherwise, the NWDAF 207 may also collect data from different network functions (NFs), AF, and/or QAM (209), to train and/or retrain a ML model for the desired analytics ID. Data collected by the NWDAF 207 may correspond to the two QoS flows, as described herein. Once the ML model is trained to provide the desired analytics for the two QoS flows, the NWDAF 207 may use this ML model to determine analytics for the PCF 205, in which case the NWDAF 207 may provide for a set of combinations of values of (UL PDB1 , DL PDB2) for QoS flow 1 and QoS flow 2, and given RT delay requirement, the service experience of the two QoS flows, or traffic carried in the two QoS flows.

[0101] At 212, the NWDAF 207 may send the analytics result to the PCF 205. By way of example, the NWDAF 207 may send different values of UL PDB1 and DL PDB2 and the corresponding service experience and/or QoS sustainability predictions when such parameters are used. [0102] At 214, the PCF 205 may use the analytics result provided by the NWDAF 207 to determine optimal and/or optimized value for UL PDB1 and DL PDB2. For example, the PCF 205 may select the (UL PDB1, DL PDB2) combination that provides the most OSE1 + OSE2 output.

[0103] At 216, once the PCF 205 determines the optimized values for UL PDB1 and DL PDB2, the PCF 205 may generate and/or update PCC rules for the two traffic flows of interest and/or a PDU session modification procedure may be performed.

[0104] At 218, the PCF 205 may be triggered at some point after setting up the PCC rules with the UL PDB1 and DL PDB2 value to determine the performance of the network based on the decision the PCF took based on the analytics received. For example, the PCF 205 may determine to obtain and/or measure output related QoS sustainability of the QoS flows of interest.

[0105] Policy recommendations and/or performance evaluation methods may be implemented. In some examples, systems and methods may provide a recommendations producer, such as an NWDAF, which is configured to receive requests or subscription to provide certain recommendation, using a recommendation ID. The recommendations producer may perform or trigger data collection, or it may trigger ML model training to obtain certain network analytics, and/or use some internal logic and the analytics output to determine recommendation outputs for a consumer, such as PCF.

[0106] The performance of a recommendations output may be based on different factors. For instance, the recommendation performance may depend on which network analytics, or combination of network analytics, which may be identified by their analytics ID, are used to determine the recommendation. In one example, a QoS recommendation may be based on service experience analytics and/or QoS sustainability. In this sense, the metric for assessing the performance of the recommendation may be different when different network analytics, identified by their analytics IDs, are used.

[0107] The performance of the network analytics themselves may also be considered to assess the recommendation performance. For example, network analytics output accuracy, sensitivity level may be used as a parameter for QoS recommendation evaluation. The same and/or a different network analytics may be determined based on different features considered as dataset for training an ML model to produce different network analytics. For example, a service experience analytic may take QoS parameters, time window, WTRU location, and/or traffic performance as input parameters In other examples, performance other than service experience analytics may be considered, including performance based on an ML model trained with only QoS parameters, traffic flow performance, and/or time window (e.g, without WTRU location).

[0108] The evaluation of the performance of a recommendation may also be based on the performance of the recommendation given a certain level of accuracy and/or sensitivity of the network analytics the recommendation producer is using. In one example, the evaluation may address how efficient the recommendation logic is, including identifying when the analytics performance is good enough for different analytics performance levels. The way a recommendation is evaluated may also be impacted by the trigger for the recommendation consumer to request the recommendations. For example, the way a recommendation is evaluated may also be impacted by a PCF triggered to request QoS recommendation when the SMF has established a number of requests for QoS modification. The PCF may be triggered to request QoS recommendation when it receives an indication from an application function to perform optimized QoS, and/or due to WTRU location change and so on.

[0109] Depending on the trigger for the PCF to request recommendations, the recommendation may be evaluated using different metrics and/or different parameters (e.g., for accuracy, sensitivity or other). In one example, when a consumer such as PCF requests to obtain recommendations from a producer such as NWDAF, the PCF includes the trigger for the PCF to request the recommendations. The triggers may be identified with a trigger ID that associates a trigger ID value to certain triggers such as "WTRU location change”, “QoS modification requests reached a certain threshold,” and so on.

[0110] The NWDAF may also be configured to map the recommendation ID, and the trigger ID, with a performance evaluation metric. Using the recommendation requested by the PCF, and the trigger ID value, the NWDAF and/or recommendations producer may determine the performance evaluation method to use to evaluate the recommendation performance, as well as the related parameters. Alternatively, or additionally, the PCF may include an indication about the metric to use for the evaluation of the performance of the requested recommendation.

[0111] FIGs. 3A and 3B illustrates an example procedure 300 of a PCF 305 being triggered to request QoS recommendation from a NWDAF 309 to assess recommendation performance. For example, at 302, the application function (AF) 301 may request the 5G system (5GS) to reserve resources for an AF session. The AF 301 may include traffic flow descriptions and/or initial QoS parameters. The AF 301 may include an indication and/or notification that optimized QoS determination is preferred. Additionally, or alternatively, the AF 301 may provide a target and/or desired value for certain performance functions. For example, the AF 301 may indicate a certain level and/or range of acceptable observed service experience for the service flow. The AF 301 may also provide an indication that QoS sustainability needs to be maximized, while satisfying an acceptable level of observed service experience. Such indications may be used to trigger the PCF 305 to request QoS recommendations.

[0112] At 304, the NEF 303 may authorize the AF 301 request and forward the request with related assistance information to the PCF 305. At 306, the PCF 305 may use assistance information from the AF 301 request, together with other parameters, to be triggered to request QoS recommendations. For example, the PCF 305 may use the indication that optimized QoS is preferred, or the indication that QoS sustainability needs to be maximized while service experience is satisfied, or the service experience range provided by the AF 301 to determine to request that QoS recommendations. The PCF 305 may determine the trigger ID that triggered the PCF 305 to request the recommendation. The PCF 305 may also use information from the AF 301 or based on local policy, to provide certain parameters for the performance assessment, such as accuracy threshold or preferred sensitivity level. [0113] At 308, the PCF 305 may send a request to the NWDAF 307 to request recommendations The PCF may include a recommendation ID. The PCF 305 may also include which analytics ID need to be used to determine the recommendation. Additionally, the PCF 305 may provide a trigger ID value and input information related to network analytics accuracy or sensitivity preferred levels.

[0114] At 310, the NWDAF recommendations producer 307 may determine to obtain and/or request or subscribe to network analytics to another NWDAF 309. The NWDAF recommendations producer 307 may provide recommendations that include analytics ID, and input information for the performance assessment of the network analytics (e.g, accuracy and sensitivity thresholds and preferred levels).

[0115] At 312, the NWDAF 309 may determine to obtain network analytics. In some examples, if a suitable, existing ML model is available, the NWDAF 309 may use this model. Otherwise, the NWDAF 309 may perform data collection from different network entities to be able to train or retrain an ML model. The NWDAF 309 may determine network analytics output based on the trained ML model. The NWDAF 309 may provide one or more analytics IDs, enhanced analytics IDs, and/or joint analytics IDs depending on the request from the NWDAF recommendations producer 307.

[0116] At 314, the NWDAF 309 may provide the network analytics output results to the NWDAF recommendations producer 307 and/or the network analytics consumer (e.g., PCF 305). The NWDAF 309 output results may include the sensitivity and accuracy level of the network analytics output.

[0117] At 316, the NWDAF recommendations producer 307 may determine, based on received analytics IDs and/or internal logic, to produce QoS recommendations for the PCF 305. The NWDAF recommendations producer 307 may also determine to assess the performance of the recommendation ID output using the metric that may be associated with the trigger ID value provided by the PCF at 308, and the network analytics performance information provided by NWDAF 309 at 314.

[0118] At 318, the NWDAF recommendations producer 307 may send the QoS recommendations to the PCF 305. The NWDAF recommendations producer 307 may include the performance measurements for the recommendation (e.g., accuracy of analytics, efficiency of recommendation, validity period, and/or sensitivity of the recommendation). The recommendation consumer (e.g., the PCF 305) may determine to request or assess the performance of the received recommendation by interacting with other network functions that are responsible for performance evaluation or have that capability. Additionally, or alternatively, the PCF 305 may assess the performance on its own, depending on the scenario.

[0119] At 320, the PCF 305 may use the recommendation results provided by the NWDAF recommendations producer 307 to determine appropriate policies and/or QoS policies. The PCF 305 may use the QoS recommendation from the NWDAF recommendations producer 307 as is and/or determine QoS parameters directly from the QoS recommendation. The PCF 305 may also use the QoS recommendation and/or the PCF 305 own internal logic to determine optimized QoS policies or parameters. The PCF 305 may assess and/or request to assess the performance of the recommendation after the PCF 305 uses the recommendation at 320.

[0120] In one example, the PCF 305 may determine to evaluate the sensitivity of the recommendations (e.g. QoS policies or parameters). The PCF 305 may also determine to assess the QoS sustainability of the QoS parameters, after the PCF 305 has used the recommendation to determine and setup policies and QoS related policies, such as shown at 322.

Claims

CLAIMS:

1 . A first network device comprising: a processor configured to: detect a trigger to request a quality of service (QoS) recommendation; send, in response to the trigger, a request to a second network device for the QoS recommendation, wherein the request comprises at least one of a recommendation identifier (ID), an analytics ID configured to be used to determine the QoS recommendation, or a trigger ID; receive, from the second network device, a message comprising the QoS recommendation and information related to the QoS recommendation, wherein the information comprises performance measurements for the QoS recommendation, wherein the performance measurements indicate at least one of an accuracy of analytics associated with the QoS recommendation, an efficiency of the QoS recommendation, a validity period associated with the QoS recommendation, or a sensitivity of the QoS recommendation; and determine, based on the received QoS recommendation from the second network device, one or more QoS policies.

2. The first network device of claim 1 , wherein the processor is further configured to: determine, after setting up the one or more QoS policies, to assess performance of the QoS recommendation and the one or more QoS policies used.

3. The first network device of claim 1 , wherein the processor is further configured to: send, to the second network device, performance assessment parameters, wherein the performance assessment parameters comprise accuracy threshold or preferred sensitivity level.

4. The first network device of claim 1 , wherein the first network device comprises a policy control function (PCF).

5. The first network device of claim 1 , wherein the processor is further configured to: determine, after setting up the one or more QoS policies, to assess the sensitivity of the QoS recommendation and the one or more QoS policies used.

6. The first network device of claim 1 , wherein the processor is further configured to: determine, after using the one or more QoS policies, to assess a QoS sustainability metric of the one or more QoS policies used.

7. The first network device of claim 6, wherein the QoS sustainability metric comprises a measure of network analytics sensitivity.

8. The first network device of claim 1 , wherein the trigger to request the QoS recommendation comprises a request from a third network device.

9. A method performed by a first network device, the method comprising: detecting a trigger to request a quality of service (QoS) recommendation; sending, in response to the trigger, a request to a second network device for the QoS recommendation, wherein the request comprises at least one of a recommendation identifier (ID), an analytics ID configured to be used to determine the QoS recommendation, or a trigger ID; receiving, from the second network device, a message comprising the QoS recommendation and information related to the QoS recommendation, wherein the information comprises performance measurements for the QoS recommendation, wherein the performance measurements indicate at least one of an accuracy of analytics associated with the QoS recommendation, an efficiency of the QoS recommendation, a validity period associated with the QoS recommendation, or a sensitivity of the QoS recommendation; and determining, based on the received QoS recommendation from the second network device, one or more QoS policies.

10. The method of claim 9, wherein the method further comprises: determining, after setting up the one or more QoS policies, to assess performance of the QoS recommendation and the one or more QoS policies used.

11. The method of claim 9, wherein the method further comprises: sending, to the second network device, performance assessment parameters, wherein the performance assessment parameters comprise accuracy threshold or preferred sensitivity level.

12. The method of claim 9, wherein the first network device comprises a policy control function (PCF).

13. The method of claim 9, wherein the method further comprises: determining, after setting up the one or more QoS policies, to assess the sensitivity of the QoS recommendation and the one or more QoS policies used.

14. The method of claim 9, wherein the method further comprises: determining, after using the one or more QoS policies, to assess a QoS sustainability metric of the one or more QoS policies used.

15. The method of claim 14, wherein the QoS sustainability metric comprises a measure of network analytics sensitivity.

16 The method of claim 9, wherein the trigger to request the QoS recommendation comprises a request from a third network device.