WO2018145761A1 - Plan de commande à topologie adaptative, basé sur un id structuré, pour 5g - Google Patents
Plan de commande à topologie adaptative, basé sur un id structuré, pour 5g Download PDFInfo
- Publication number
- WO2018145761A1 WO2018145761A1 PCT/EP2017/053042 EP2017053042W WO2018145761A1 WO 2018145761 A1 WO2018145761 A1 WO 2018145761A1 EP 2017053042 W EP2017053042 W EP 2017053042W WO 2018145761 A1 WO2018145761 A1 WO 2018145761A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- logical
- resource element
- neighbors
- resource
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/12—Discovery or management of network topologies
- H04L41/122—Discovery or management of network topologies of virtualised topologies, e.g. software-defined networks [SDN] or network function virtualisation [NFV]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/40—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using virtualisation of network functions or resources, e.g. SDN or NFV entities
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
- H04L41/5041—Network service management, e.g. ensuring proper service fulfilment according to agreements characterised by the time relationship between creation and deployment of a service
Definitions
- the present invention relates to the control of 5G networks, and more particularly to a structured id- based control plane.
- the 5th generation (5G) mobile technology will be standardized and deployed by 2020.
- the devices and applications of the next generation network is expected to support many new types of connections between various devices such as cars, wearables, sensors and actuators, from both private and industrial environments.
- the new types of connections usually imply very distinct service requests about, for example, latency and data rate, which naturally asks for different treatments and thereby poses challenges to the control of the 5G networks.
- CN Core networks
- CN Core networks
- the various new service requests cause their properties to be heterogeneous.
- the sensor nodes may request to join the network in order to transmit sensing results, while there could be remote high speed vehicles communicating with each other at the same time. Effortlessly, distinct traffic patterns will be spontaneously generated to the network.
- the traffic patterns can be affected by the service duration, the node mobility patterns, and the requirements of quality-of-service (QoS) of the connection. Consequently, the service requests to the 5G network are hard to be predicted and a proper way to meet the requirements is to respond with agility to the incoming services in a dynamic way. However, efficiently responding to distinct and unpredictable service requests is not an easy task. Obviously, administratively pre-planning approaches with static network deployment do not work. Recently, network functions virtualization (NFV) and software-defined networking (SDN), as disclosed in: Open Networking Foundation, "Openflow Switch Specification", Version 1.5.1, 2015, have been mentioned as the central two enablers to realizing a flexible and programmable network infrastructure.
- NFV network functions virtualization
- SDN software-defined networking
- the idea behind the SDN is to abstract everything as a flow and to move the complexity of the flow treatment towards a single logical element called SDN controller (SDNC).
- SDNC SDN controller
- the SDN view reduces all the network elements to dump the flow treatment devices, which are only responsible for the flow processing.
- the SDN fuses all the management and control plane intelligence at the SDNC.
- the common abstraction and the locally available data simplify the development of the network control and management applications.
- the NFV further turns existing network functions (NF), which were mainly implemented by dedicated and specialized hardware formerly, into virtualized function modules that can be easily deployed or removed, as disclosed in: H. Hawilo, A. Shami, M. Mirahmadi and R.
- NFV next generation mobile networks
- vEPC next generation mobile networks
- Green networking advocates switching off underused resources or dynamically reroutes traffic to cheap energy nodes and links. Beyond that, new approaches to allow both sharing of excess resources and leasing extra resources lead to generalized usage of virtualization of both links and nodes in the infrastructures. Green networking and virtual networking call for support for network structural dynamics since links, nodes and their services cannot be presumed permanent, which are brought by SDN and NFV as well.
- NFV orchestrator and SDNC have to rely on a robust control plane connecting every network resource to transmit and realize their strategies operating the network behaviors. While the control plane deployment could be achieved manually or through some management solutions like, for example, the configuration management, the command line interface (CLI) management or the NETCONF configuration protocol as found in: M. Wasserman and T.
- a monitoring application e.g., an application of the sample theorem
- another application e.g., the NetFlow standard for network traffic monitoring
- different requirements will emerge in terms of necessary computational capacity and memory in the resource and in its controller.
- each switch connects by means of a TCP connection to the SDNC. Over this same connection, several different packet types such as PACKETJN for notification and FLOWMOD for flow rule modification are then sent.
- a solution to all the aforementioned problems would be to design a self-organizing control plane that can bootstrap autonomously without pre-configuration, provide multiple paths for failed control channel switching and be topological ⁇ adaptive to different service QoS requirements. Then, those key features should be supported by the resources, independently of the SDNC and the VIM, and they will therefore require a control specific intelligence in all controllable resources. In other words, the control plane should be spawned and constantly maintained by the resources without the need for the SDNC and VIM intervention.
- an IETF working group called autonomic networking integrated model and approach suggests that future networks shall be capable to manage themselves by some autonomous functions deployed in the infrastructure. Moreover, those autonomous functions shall be automatically connected and communicate with each other over an autonomous control plane (ACP) as found in: M. H. Behringer, T. Eckert and S. Bjarnason, "An Autonomic Control Plane", Internet Engineering Task Force, 2016.
- ACP autonomous control plane
- US-9,043,884-B2 US-9,130,837-B2 and US-9,391,959-B2 are specifically related to the control plane solution
- the existing routing protocol RFC6550 for low-power and lossy networks is suggested as the protocol to maintain an ACP, as found in T. Winter, P. Thuber, B. Brandt and others, "RFC 6550: IPv6 Routing Protocol for Low- Power and Lossy Networks," Internet Engineering Task Force (IETF) Request For Comments, 2008.
- the switches in LAN/WAN use RSTP or OSPF for interconnection. They both generate a spanning tree as an overlay.
- OSPF OSPF
- a fat-tree topology is used to connect computing resources.
- AODV ad hoc on-demand distance vector
- DSDV destination- sequenced distance-vector
- OLSR optimized link state routing
- the invention relates to a system comprising multiple resource elements within a communication network, wherein each resource element is adapted to support a resource- to-resource (R2R) protocol.
- the R2R protocol is adapted to assign a respective identification (id) value (rii) to each resource element amongst the multiple resource elements, each identification (id) value (n,) being a unique value, adapted to establish, for each resource element amongst the multiple resource elements, a connection to each of its physical or virtual neighbors by transmitting an id- based message (Hello message) from each resource element towards each of its physical or virtual neighbors, the physical or virtual neighbors of a resource element being the resource elements located in an underlay and with a network connection to it which is available for a connection establishment, and adapted to establish, by means of each resource element amongst the multiple resource elements and for each pair of logical neighbors, a logical connection between the respective two logical neighbors in order to form an id-based logical structure as an overlay and based on
- Each resource element can be defined as a node or a network node like, for example, a routing or forwarding element such as a switch and a probe.
- Each resource element can also be identified by a distinct unsigned integer identification value, namely an integer id regardless of length (in terms of bits).
- the integer id can, for example, be a public key of any binary length.
- the id value (n,) can be randomly drawn from a virtual id space.
- the resource-to-resource ( 2 ) protocol can be defined as an inter-resource protocol.
- each resource element amongst the multiple resource elements supports the 2 protocol through a respective resource control agent (RCA).
- RCA resource control agent
- each resource element can be a controllable resource element.
- each resource control agent can be locally implemented inside each respective resource element amongst the multiple resource elements as a local daemon.
- each resource element can intrinsically use some of its own components as an RCA. Then, those components will be locally modified by extending their functionalities in order to support the R2R protocol.
- each resource element amongst the multiple resource elements performs a local pairing operation in a distributed manner by transmitting a notification message (NotifyNb message) towards each of the two logical neighbors of each pair in order to establish the logical connection between the respective two logical neighbors, the logical connection between the respective two logical neighbors being established once the transmitted notification message (NotifyNb message) has been received by each of them.
- a notification message (NotifyNb message)
- the R2R protocol is adapted to solve a conflict of resource elements pairing the same pair of logical neighbors by allowing each pair of logical neighbors to be paired by means of a single resource element.
- the conflict is solved by allowing each of the two logical neighbors constituting the pair to make a same and independent choice to select the single resource element and to reject any other resource element by transmitting each a respective rejection message ( ejectNb message) towards each rejected resource element.
- the formation of the id-based logical structure can be improved using transmission of messages from the R2R protocol.
- the choice is based on calculating a respective logical distance from each of the pairing resource elements towards each of the two logical neighbors, the selected single resource element corresponding to the shortest logical distance between itself and one of the two logical neighbors of the pair.
- the present method based on the calculation of a logical distance offers the advantage to require no overhead and to accelerate the formation of the id-based logical structure by accepting the closer resource element doing the pairing operation.
- each resource element amongst the multiple resource elements has a routing table comprising all the pairs of logical neighbors paired by itself, all the logical paths going through itself and all the logical paths between its logical neighbors, and consisting of a set of data routing rules to be applied to the incoming data packets and a set of control routing rules to be applied to the incoming control packets, the set of control routing rules being maintained by the resource control agent (RCA).
- RCA resource control agent
- control channels can be established amongst all the resource elements, the routing table comprising control flow rules establishing the id-based structured overlay.
- the 2 protocol is adapted to establish, for at least one resource element amongst the multiple resource elements, logical connections to at least one finger resource element, the finger resource element corresponding to a resource element other than one of the two logical neighbors.
- the id-based logical structure can be grown with finger links or finger paths or finger connections.
- the logical connections to at least one finger resource element are selectively established using an algorithm of selection of finger resource elements.
- a resource element can select specific finger resource elements dynamically and keep connections to them so as to form a certain topology of the id-based logical structure, in particular for the control plane.
- a topology of the id-based logical structure is adaptively changed according to the network conditions.
- the change in the topology can be carried out in several ways.
- the topology of the id-based logical structure can be changed by adding or removing at least one logical connection to at least one finger resource element.
- a simple rewiring method can be obtained.
- the change in the topology can also be triggered either manually or autonomously and achieved using the algorithm of selection of finger resource elements.
- the finger links can be added or removed adaptively.
- the change in the topology can also be manually triggered by a controller or an administrator of the communication network.
- the finger links can be added or removed adaptively according to the own needs of the controller or the administrator.
- the change in the topology can also be autonomously triggered by each resource element amongst the multiple resource elements based on a performance indicator (KPI) of its connections to the logical neighbors and the finger resource elements.
- KPI performance indicator
- the id-based logical structure is an id-ordered ring-based logical structure.
- each resource element amongst the multiple resource elements can be a network- capable node.
- the resource element can be any network-capable element.
- the resource element can be any node like, for example, a network element, a compute node, a server, a storage server, a file server, a compute cluster or a domain controller amongst others.
- the invention relates to a control plane comprising the system as specified in the first aspect or any one of the implementations of the first aspect.
- a simple and self-organizing control plane can be obtained based on a full infrastructure controllability. Indeed, it is supported by the in-resource intelligence from all the controllable resources, with a distributed 2 protocol. Any network resource element locally hosts a resource control agent (RCA) talking to each other through the proposed R2R protocol.
- RCA resource control agent
- the self- organizing control plane is adapted to bootstrap autonomously without pre-configuration, provide multiple paths for failed control channel switching and be topological ⁇ adaptive to different service QoS requirements.
- a control-based intelligence is available in all the controllable resource elements such as the nodes, network-capable nodes and so on, which allows the control plane to be spawned and constantly maintained by the resources independently of the SDNC and VIM.
- the invention relates to a resource element of a system comprising multiple resource elements within a communication network according to the first aspect or any one of the implementations of the first aspect.
- the invention relates to a method for controlling multiple resource elements within a communication network.
- the method comprises the steps of assigning a respective identification (id) value (n,) to each resource element amongst the multiple resource elements, each identification (id) value (n,) being a unique value, establishing, for each resource element amongst the multiple resource elements, a connection to each of its physical or virtual neighbors by transmitting an id-based message (Hello message) from each resource element towards each of its physical or virtual neighbors, the physical or virtual neighbors of a resource element being the resource elements located in an underlay and with a network connection to it which is available for a connection establishment, and establishing, by means of each resource element amongst the multiple resource elements and for each pair of logical neighbors, a logical connection between the respective two logical neighbors in order to form an id-based logical structure as an overlay and based on the order of the identification (id) values (n,) of each resource element amongst the multiple resource elements, the two logical neighbors
- the method comprises the step of esta blishing, for at least one resource element amongst the multiple resource elements, logical connections to at least one finger resource element, the finger resource element corresponding to a resource element other than one of the two logical neighbors.
- the invention relates to a computer program comprising a program code for performing the method according to the fourth aspect or the first implementation of the fourth aspect when executed on a computer.
- the method can be performed in an automatic and repeatable manner.
- the computer program can be performed by the above apparatus.
- the above apparatus may be implemented based on a discrete hardware circuitry with discrete hardware components, integrated chips or arrangements of chip modules, or based on a signal processing device or chip controlled by a software routine or program stored in a memory, written on a computer-readable medium or downloaded from a network such as the Internet.
- a software routine or program stored in a memory written on a computer-readable medium or downloaded from a network such as the Internet.
- it can be a fully virtualized appliance, that is, for example, a virtual machine executed by any other machine including another virtual machine.
- the resource element can also be any computer program executed in some appropriate way on some hardware, which might in some situations be seen as not belonging to the resource element.
- the resource element can, if necessary and appropriate for the specific control purpose, be operationally delimited both in size (e.g., physical, geographical or logical protocol reach) and in depth (spanning all hardware and software or being rather limited to a specific layer in the protocol stack), while the definition of the resource element per se does not limit its applicability and generality.
- Fig. 1 shows a structured id-based control plane 100 consisting of an id-ordered ring-based logical structure (overlay) and a physical network structure (underlay) according to an embodiment of the present invention
- Fig. 2 shows an architecture 200 of the extended OVS node within an SDN scenario according to an embodiment of the present invention
- Fig. 3 shows a procedure of establishment of a physical path connecting two OVS nodes (S,, S, ) according to an embodiment of the present invention
- Fig. 4 shows a pairing procedure of establishment of a logical path connecting two logical neighbors (a,, b,) according to an embodiment of the present invention
- Fig. 5 shows an exemplary procedure of conflict resolution related to a pairing operation according to an embodiment of the present invention
- Fig. 6 shows a procedure of handling an OVS node joining (n x ) according to an em bodiment of the present invention
- Fig. 7 shows an exemplary procedure of handling an OVS node leaving (n x ) according to an em bodiment of the present invention
- Fig. 8 shows an exemplary procedure of topological adaptation with finger selection according to an embodiment of the present invention.
- Fig. 9 shows a system 300 of multiple resource elements with a distributed resource-to- resource ( 2 ) protocol according to an embodiment of the present invention.
- Fig. 1 shows a structured id-based control plane 100 consisting of an id-ordered ring-based logical structure (overlay) and a physical network structure (underlay) according to an em bodiment of the present invention.
- each resource element is a node such as a network-capa ble node (e.g., a routing element, a forwarding element, a network element, a compute node, a server, a storage server, a file server, a compute cluster or a domain controller amongst others) and is identified with a respective and unique identification (id) value (n,).
- the resource elements located in the physical network (denoted hereafter by u nderlay) are arranged in an embedded logical structure (denoted hereafter by overlay) through an id-based ma pping.
- the core structure of the overlay is an id-ordered ring-based topology grown with finger links connecting, for example, 3 ⁇ 4 to n 5 to n 7 a nd n 7 to n3 ⁇ 4 as depicted in the exemplary extended ring of Fig. 1.
- the ring-based topology can be replaced with any other topology exhibiting a circular form or with any other topology exhibiting a form other than circular.
- underlay and "overlay” are not limited to a respective physical and logical network, but are also applicable to any layers.
- the resource elements are physical elements.
- the resource elements are virtual machines running on any support, and to have them connected by virtual links, possibly using TCP/IP networking or VPN between each other.
- the corresponding virtual network or structure can then be used as an underlay, and the R2R protocol will span a new overlay on top for the control of the resource elements in this scenario.
- the resource elements are considered unstable and their connectivity is considered transient.
- the feature "unstable" means that the resource elements can come and go (this process is commonly called churn, and sometimes attrition) more or less at any time.
- the resource elements can also move, and the links or connections can also change, be dropped or new links or connections can be created by something other than the R2R protocol.
- the proposed R2R protocol is adapted to handle all these issues: new nodes, nodes leaving, links or connections dropped, links or connections failing, links or connections coming, and so on.
- this instability would be rather atypical if the resources were classical forwarding elements only. Indeed, a physical switch as a forwarding element cannot disappear from the rack, and cannot be easily duplicated.
- each identification (id) value (n,) is achieved by the proposed distributed R2R protocol, which is supported by the in-resource intelligence from each resource element through a respective resource control agent (RCA).
- the resource element is an Open vSwitch (OVS) node.
- OVS Open vSwitch
- the OVS node represents a well- known switch, which can be instantiated and run on a standard computing platform (e.g., in a virtual machine (VM)).
- VM virtual machine
- the OVS node has also the advantage to be already a commercial-grade product widely deployed in the real networks of many IT companies such as Google.
- the OVS node will be extended by integrating the R2R protocol therein, as it is depicted in Fig. 2 showing the architecture 200 of the extended OVS node within a SDN scenario.
- a virtual network using, for example, the prototyping environmental tool Mininet as described in: B. Lantz, B. Heller and N. McKeown, "A network in a laptop: rapid prototyping for software-defined networks", Proceedings of the 9th ACM SIGCOMM Workshop on Hot Topics in Networks, 2010, and consisting of a number of extended OVS nodes connected according to a predefined network topology, will be then created.
- the architecture 200 of the extended OVS node consists of two main parts: a kernel part and a userspace part, which are graphically separated by the dotted line.
- the kernel part contains a kernel datapath module 210, which implements the forwarding engine responsible for per-packet lookup, modification and forwarding.
- the userspace part contains an ovsdb-server 220, which is responsible for storing the information about the configuration of the switch, and an ovs- vswitchd module 230 as a local daemon, which can modify the kernel part by modifying, for example, the flow rule entries in the flow table.
- the flow table could then be a normal routing table consisting of a set of data routing rules to be applied to the incoming data packets and a set of control routing rules to be applied to the incoming control packets.
- the local daemon also exposes the external interface in order to allow the extended OVS node to communicate with a remote SDN controller through the depicted control port.
- the architecture 200 of the extended OVS node further comprises a RCA module (231), which is locally implemented inside the ovs-vswitchd module 230 in order to extend the functionalities of the local daemon (i.e., the ovs-vswitchd module 230) and allow the extended OVS node to support the R2R protocol of the present invention.
- the local daemon has been chosen because it has already been used successfully as a local agent responsible for many basic control tasks such as implementing the forwarding logic including media access control (MAC) learning, load balancing over bonded interfaces and communicating with the external SDN controller using the OpenFlow protocol.
- MAC media access control
- the extended local daemon can support the functionality to construct the structured id-based control plane of the present invention.
- the proposed R2R protocol utilizes four kinds of messages, which are designated as a Hello message, a NotifyNb message, a RejectNb message and a RemovePath message.
- the Hello message represents an id-based message that is periodically broadcast by an OVS node towards its physical or virtual neighbors in order to inform them about its own id value.
- the NotifyN b message represents a notification message that is broadcast by an OVS node towards its logical neighbors in order to inform them a bout other OVS nodes.
- the RejectNb message represents a rejection message that is broadcast by an OVS node having received a NotifyN b message towards the logical neighbor that has sent it the NotifyN b message in order to reject the received NotifyNb message.
- the RemovePath message represents a message that is broadcast by an OVS node towards all the peers whose connections take a leaving node as the next hop in order to remove a physical path towards the leaving node.
- the sym bol n will refer to the id value of an OVS node. Furthermore, since the proposed R2R protocol is fully distributed, it will be described from the point of view of the OVS node and all the other OVS nodes will behave identically to that OVS node.
- the R2R protocol starts with a step of bootstrapping, wherein each OVS node first generates its id value (n,) and afterwards, locates and esta blishes connections to its logical neigh bors and finger OVS nodes in order to form the structured overlay.
- the id generation ca n be carried out in different methods.
- the id value (n,) can be preconfigured in the OVS node or the id value (n,) can consider the location of the OVS node in the network such that the physically closer OVS nodes get closer id values (n,).
- the id value (n,) being randomly d rawn from a virtual id space will be chosen due to its simplicity.
- the id value (n,) should be generated with the aim of being unique thanks to a method allowing to minimize as much as possible the collision of generating the same id values (n,).
- each OVS node first identifies its physical or virtual neighbors that have availa ble network connections to itself and broadcasts a respective Hello message over all its port interfaces towards each of them, as illustrated in Fig. 3 showing the procedure of esta blishment of a physical path or connection between the two OVS nodes: S, and Sv with a respective port h and port i.
- the receiving OVS node learns that on the incoming port (i.e., the port h) it can reach the sending OVS node S, with the id value: n,.
- the method to form the structured overlay is inspired by the linearization algorithm as found in: M. Onus, A. W. icha and C. Scheideler, "Linearization: Locally Self-Stabilizing Sorting in Graphs", ALENEX, 2007. The algorithm turns any connected graph into a linear graph.
- each OVS node links its two peer OVS nodes if their id values (n,) are adjacent on the logical ring.
- the pairing operation can be illustrated in Fig. 4, which shows a pairing procedure of establishment of a logical path connecting two logical neighbors (a,, b,) in the case of one iteration.
- the OVS node S sorts the peer set N, according to the id values (n,) of the peers of the set N,.
- a second step (2) and for each consecutive pair, denoted by (a,, b,) in Fig. 4 and called a logical pair, the OVS node S, sends a respective NotifyNb message towards a, and b, over the respective paths or connections between itself (S,) and a, and between itself (S,) and b,.
- the OVS node S notifies a, about b, and b, about a,, respectively.
- the OVS node performing the pairing operation (e.g., the OVS node S, in Fig. 4) will be denoted in the following as a notifying node.
- both endpoints in the logical pair (a,, b,) receive their respective NotifyNb message, they add each other as logical neighbors if there is no better candidate so far.
- the suggested linearization-based algorithm guarantees the convergence, and the graph will be globally linearized.
- the suggested linearization-based algorithm can be considered self-stabilizing, and each OVS node will be able to find its global logical neighbor(s) eventually.
- the logic behind this is that the linearization-based pairing operation grows the peer set of other logical neighbors by notifying about a possible logical neighbor information.
- the notifying node e.g., the OVS node S, in Fig. 4
- Fig. 5 shows an exemplary proceed ure of conflict resolution related to a pairing operation according to an em bodiment of the present invention.
- This em bodiment provides a conflict resolution algorithm in the case that, when two notifying OVS nodes pair the same logical pair, two endpoints of that logical pair can make the same decision and independently agree to accept a pairing operation from the sa me notifying node.
- each OVS node a, and b receives a NotifyNb message from both the notifying OVS node x and the notifying OVS node y. Then, in a second step (2), the OVS nodes a, and b, use the minimum logical distances from each of the two notifying OVS nodes x and y to their respective endpoints to make, in a third step (3), their decision.
- the OVS node a calculates the logical distance from x to a
- the OVS node b calculates the logical distance from x to b
- the minimu m logical distance is chosen as ⁇ ⁇ .
- the same calculation applies to the OVS node y in order to calculate the logical distance from y to a, and bi, respectively, and obtaining the minimum logical distance ⁇ ⁇ .
- both a, and b choose in the third step (3) to linearize the OVS node x correspond ing to the minimum distance ⁇ ⁇ by sending each a respective RejectNb message towards the notifying OVS node y, otherwise (i.e., if ⁇ ⁇ ⁇ ⁇ ⁇ ⁇ ) they would have chosen the OVS node y.
- the pairing procedure by the accepted notifying OVS node x can be u ndertaken in order to pair (a,, b,) with a logical connection. It should be noted that the a bove conflict resolution can also be based on alternative criteria.
- each endpoint can use the path length given by the two notifying OVS nodes in order to make the decision.
- the criteria of determining the minimum logical distance between ⁇ ⁇ and 8y, as used in the em bodiment of Fig. 5, presents the advantage of requiring no overhead and also the advantage of accelerating the formation of the structured overlay, in particular the id- ordered ring-based overlay, due to the proximity with the notifying OVS node that is accepted.
- the present invention needs no special treatment to maintain the structured overlay. Indeed, any network dynamic results in logical neighbor set changes of related OVS nodes, which further trigger new linearization-based pairing operations and more particularly new notifications.
- the link change will be considered through the case of a node joining and another case of a node leaving.
- a node joining can be defined as a new node associating with some existing nodes in the network by creating new links.
- the new OVS node will be assumed to have its own id value n x , which is different from all the other id values of the existing OVS nodes in the network.
- the OVS node n x starts to esta blish its physical paths to the OVS nodes that are directly associated with it by sending and receiving Hello messages according to the procedure of Fig. 3.
- the OVS node S sorts its peer set and finds the new logical pair (a,, n x ).
- the adjacent logical neighbors execute the linearization-based pairing process and continuously notify the OVS node n x with the information a bout its possible logical neighbor(s).
- the OVS node n x grows iteratively, the OVS node n x also executes its own
- a node leaving can be defined as an existing node leaving the network while its adjacent links are removed. Such an event triggers the peer set changes of the nodes originally associated with the leaving node. Those nodes that directly observe the node leaving not only have to remove the physical paths to the leaving node, but also those virtual paths passing through the leaving node. As depicted in Fig. 7 showing a procedure of handling a node leaving according to an embodiment of the present invention, the OVS node S, observes the OVS node leaving n x . Thus, the invalid physical path from Si to n x will be removed from the flow table of S,, as illustrated in the first step (1).
- any endpoint of a path that takes the leaving node as the next hop will be notified with a RemovePath message to remove the respective invalid virtual path, as illustrated in the second step (2).
- all the rest is the same as the case of the node joining where the linearization-based pairing and notification processes triggered by the logical neighbor set changes will continue and eventually correct the globally linearized structure by removing the leaving node.
- the finger paths are established, when a node is kept being notified by possible logical neighbors from other nodes. Every node maintains paths or connections to all the notified logical neighbors.
- the topological adaptation requires a rewiring method.
- the present invention introduces a finger selection algorithm. More specifically, after the bootstrapping step of the suggested R2R protocol, a node will select particular finger nodes dynamically and keep connections to them so as to form a certain overlay topology for the control plane. Such a finger selection procedure takes a certain topological mode as its argument, which can generally switch between the balanced, scale-free and centralized modes.
- Fig. 8 shows a procedure of topological adaptation with finger selection according to an embodiment of the present invention.
- the finger selection algorithm starts with a first step (1) during which, when the OVS node S, with an id value establishes its finger paths, it starts by performing a lookup to know which are the OVS nodes n, with an id value equal to or greater than id + 2 1"1 , Vi 6 [l, k] . Then, instead of selecting this OVS node n, as the finger node directly, the finger node is picked up from an interval, and either the most or the least connected OVS node within the interval is chosen, as inspired by: S. Ktari, A. Hecker and H. Labiod, "A construction scheme for scale free dht-based networks", Global Telecommunications Conference, 2009.
- the size of the considered interval determines the number of super OVS nodes according to the following algorithm: - If the interval equals the ID space and encompasses all the OVS nodes with a high probability, then by preferring the most connected OVS node, the system will become a fully centralized system with essentially one central OVS node, to which each other OVS node will be connected;
- the system will become a fully distributed system, where all the OVS nodes have an equal degree. This also works for smaller intervals, so that the global ID space as interval is not a necessary requirement for that;
- DHT distributed hash table
- the interval to go from one calculated OVS node to the next (e.g., in the Chord system, which is an efficient distributed lookup service based on the Chord protocol, for an OVS node with an id value, the interval would be: [id + 2 1" 1 , id + 2' [), then decentralized systems can be created by increasing or decreasing the system centralization or distribution, respectively (i.e., the average OVS node's betweenness centrality). Different graph-theoretical properties (such as random graph, scale free graph, ring, tree and bus) can be achieved in this way at the system level by only using relatively simple and local algorithms within the respective resource nodes.
- this finger selection procedure of Fig. 8 does not require the OVS nodes to keep any additional information. What an OVS node only needs to know is to maintain the connections to its logical neighbors.
- the selection procedure can be performed at the remote node n, that is found in the lookup process (step 1) in a transparent way without adding any extra overhead.
- the OVS node S issues, in a second step (2), a query to set up a finger path and this query reaches the destination node (i.e., n,), instead of returning its own id, it can, in a fourth step (4), merely return the selected id and build the connection for the OVS node S,.
- changing to a certain overlay topology can be triggered either manually or autonomously.
- Manually changing the overlay topology for the control plane requires to propagate the decision from the controller or administrator of the network. Since the connectivity of the control plane will be always maintained by the control plane itself, the topology argument can be distributed across the network. After receiving the topology-changing command, each node runs the finger selection algorithm taking the topology argument.
- each node can have a different perception of its local network conditions (e.g., it could experience heterogeneous QoS connections in the control plane).
- Some parts of the network can be more dynamic, for example, at the edge of a mobile network, thereby rendering the connections not very relia ble.
- some parts of the network can be more static, for example, at the core of the mobile network.
- each concerned node autonomously senses the conditions of its connections to its logical neighbors and finger nodes. Once the KPI of the connections drops until reaching a threshold, each concerned node starts to run the finger selection algorithm switching to a balanced mode for resilience. Conversely, it switches to a centralized mode for efficiency if the situation is stable.
- the 2 protocol is designed to establish a connectivity in any possible situation.
- Fig. 9 shows a system 300 of multiple resource elements with a distributed resource-to-resource (R2R) protocol according to an embodiment of the present invention.
- R2R resource-to-resource
- the resource control agent (RCA) on each resource element keeps running the distributed R2R protocol to build and maintain the flow table consisting of a set of control flow rules, such that control channels are established amongst all the resource elements.
- the suggested structured overlay forms an in-resource control plane and, thus, resolves the aforementioned chicken-egg problem for the control need of both SDN and NFV, since it is established by a self-organizing and distributed R2R protocol, independent of SDNC and VIM. Thus, the operators do not have any longer to dedicate external resources, in particular, for establishing the control channel.
- the network Once the network is deployed, its control plane is built autonomously. This applies to physical networks, logical networks and virtual networks equally.
- the forwarding policy becomes simple and straightforward. Indeed, given a packet, whose destination is dst.id, the node greedily chooses the path to the node, whose id is the closest in the id- based logical structure to dst.id. Because of the logical order among all the nodes, the packet is forwarded by getting closer and closer towards the destination, until the logical distance becomes zero.
- Such a routing mechanism can be called an id-based routing. It is, thus, clear that each node only needs a partial information of the network topology, but it can still achieve any-to-any routing.
- each node is naturally independent to any data plane address strategies.
- the suggested structured overlay is immune to any topology change (except for full disconnect of network areas), and, for example, the node joining and node leaving will not cause fluctuations to the connectivity of the control plane. This is due to the fact that, once there is a node joining in the network, only its logical neighbors on the id-based logical structure will be involved to merge the new node, while all the other nodes do not have to update their routing information.
- the suggested structured overlay is quite scalable to the network size, which is a critical feature being required for the future control plane in 5G networks due to the expected large number of network nodes (e.g., small cells, smaller components in both software and data plane, dynamically composed to different slices, amongst others) and their high dynamics.
- network nodes e.g., small cells, smaller components in both software and data plane, dynamically composed to different slices, amongst others
- the suggested overlay provides a built-in multiple-path option. Specifically, when network dynamics (e.g., link failures resulting in broken paths) occurs, the affected node usually can route around the failed paths without requiring them to be repaired, as there are usually many routes between each pair of nodes. Since the R2R protocol keeps maintaining the structured overlay, the affected path between two logical neighbors will be reconnected eventually, if the disconnected logical neighbor is still in the network.
- Another advantage of the suggested structured overlay enables a topological adaptation, unlike the well-known routing protocols.
- the structured overlay provides enough flexibility for topology adaptation of the control plane. Specifically, the network nodes can add or remove finger connections in a distributed way but globally change the topology of the structured overlay.
- control plane For example, if each node connects to all the other nodes, the control plane becomes a full mesh network. If each node randomly chooses its finger nodes, the control plane becomes a balanced network. If a few nodes are selected by other nodes as finger nodes based on some criteria, the structured overlay will become a scale-free network.
- Such a topology adaptation enables the control plane to be adaptive to the QoS requirements of different control applications.
- the present invention relates to a self-organizing and adaptive control plane (CP) solution, which allows network nodes, namely all the controllable resource elements within the communication network, to autonomously establish and maintain an id-ordered ring-based logical structure without any pre-configuration.
- Each network node is identified by a distinct unsigned integer identification value, namely an integer id, and the entirety of the network nodes bootstrap and maintain the id-ordered ring-based logical structure for the control plane (CP), in which each node has logical connections to its predecessor and its successor as well as some finger peers.
- CP control plane
- each identification value is achieved by a distributed resource-to-resource ( 2 ) protocol, which is supported by the in-resource intelligence from each network node through a respective resource control agent (RCA).
- a distributed resource-to-resource ( 2 ) protocol which is supported by the in-resource intelligence from each network node through a respective resource control agent (RCA).
- RCA resource control agent
- the network nodes can adaptively add or remove logical connections to some finger peers in order to change the topology of the id-ordered ring-based logical structure and thereby to balance between resilience and efficiency.
- a computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.
- a suitable medium such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
La présente invention concerne une solution de plan de commande auto-organisée et adaptative qui permet à des nœuds de réseau contrôlables d'établir et de maintenir de manière autonome une structure logique en anneau ordonnée par ID, sans configuration préalable. Chaque nœud de réseau est identifié par une valeur d'identification entière non signée respective et unique, et la totalité des nœuds de réseau amorce et maintient la structure logique en anneau ordonnée par ID pour le plan de commande, chaque nœud ayant des connexions logiques avec son prédécesseur et son successeur ainsi que certains homologues doigts. L'attribution de chaque valeur d'identification est exécutée par un protocole de ressource à ressource distribué qui est pris en charge par l'intelligence en ressources de chaque nœud de réseau via un agent de contrôle de ressource respectif. Consécutivement à un déclenchement manuel ou autonome, les nœuds de réseau peuvent ajouter ou supprimer de manière adaptative des connexions logiques avec certains homologues doigts de sorte à modifier la topologie de la structure logique en anneau ordonnée par ID et l'équilibre entre résilience et efficacité.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/EP2017/053042 WO2018145761A1 (fr) | 2017-02-10 | 2017-02-10 | Plan de commande à topologie adaptative, basé sur un id structuré, pour 5g |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/EP2017/053042 WO2018145761A1 (fr) | 2017-02-10 | 2017-02-10 | Plan de commande à topologie adaptative, basé sur un id structuré, pour 5g |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2018145761A1 true WO2018145761A1 (fr) | 2018-08-16 |
Family
ID=58046643
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/EP2017/053042 Ceased WO2018145761A1 (fr) | 2017-02-10 | 2017-02-10 | Plan de commande à topologie adaptative, basé sur un id structuré, pour 5g |
Country Status (1)
| Country | Link |
|---|---|
| WO (1) | WO2018145761A1 (fr) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111638726A (zh) * | 2020-05-27 | 2020-09-08 | 北京理工大学 | 基于事件触发通信的多无人机编队一致性控制方法 |
| WO2020177540A1 (fr) * | 2019-03-01 | 2020-09-10 | 中兴通讯股份有限公司 | Procédé de structuration en couches de logique de réseau basée sur un protocole rift, dispositif, dispositif de réseau, et support d'informations |
| CN112312444A (zh) * | 2020-10-22 | 2021-02-02 | 深圳供电局有限公司 | 一种5g网络切片下资源备份方法 |
| CN112383422A (zh) * | 2020-11-04 | 2021-02-19 | 浙江大学 | 加快一致性分布式算法收敛速度的网络拓扑优化方法 |
| CN112600795A (zh) * | 2020-11-25 | 2021-04-02 | 中国人民解放军国防科技大学 | 一种不完全信息下的作战网络瓦解方法及系统 |
| CN114024856A (zh) * | 2020-07-17 | 2022-02-08 | 中兴通讯股份有限公司 | 路由优化方法、物理网络设备及计算机可读存储介质 |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9043884B2 (en) | 2013-01-25 | 2015-05-26 | Cisco Technology, Inc. | Autonomic network protection based on neighbor discovery |
| US9130837B2 (en) | 2012-05-22 | 2015-09-08 | Cisco Technology, Inc. | System and method for enabling unconfigured devices to join an autonomic network in a secure manner |
| US20160173338A1 (en) * | 2013-03-20 | 2016-06-16 | Wolting Holding B.V. | Compiler for and method for software defined networks |
| US9391959B2 (en) | 2013-01-15 | 2016-07-12 | Cisco Technology, Inc. | Automated control plane for limited user destruction |
-
2017
- 2017-02-10 WO PCT/EP2017/053042 patent/WO2018145761A1/fr not_active Ceased
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9130837B2 (en) | 2012-05-22 | 2015-09-08 | Cisco Technology, Inc. | System and method for enabling unconfigured devices to join an autonomic network in a secure manner |
| US9391959B2 (en) | 2013-01-15 | 2016-07-12 | Cisco Technology, Inc. | Automated control plane for limited user destruction |
| US9043884B2 (en) | 2013-01-25 | 2015-05-26 | Cisco Technology, Inc. | Autonomic network protection based on neighbor discovery |
| US20160173338A1 (en) * | 2013-03-20 | 2016-06-16 | Wolting Holding B.V. | Compiler for and method for software defined networks |
Non-Patent Citations (22)
| Title |
|---|
| A. DIXIT; F. HAO; S. MUKHERJEE; T. V. LAKSHMAN; R. KOMPELLA: "Towards an elastic distributed SDN controller", ACM SIGCOMM COMPUTER COMMUNICATION REVIEW, 2013 |
| A. R. CURTIS; J. C. MOGUL; J. TOURRILHES; P. YALAGANDULA; P. SHARMA; S. BANERJEE: "DevoFlow: scaling flow management for high-performance networks", ACM SIGCOMM COMPUTER COMMUNICATION REVIEW, 2011 |
| B. LANTZ; B. HELLER; N. MCKEOWN: "A network in a laptop: rapid prototyping for software-defined networks", PROCEEDINGS OF THE 9TH ACM SIGCOMM WORKSHOP ON HOT TOPICS IN NETWORKS, 2010 |
| C. E. PERKINS; P. BHAGWAT: "Highly dynamic destination-sequenced distance-vector routing (DSDV) for mobile computers", ACM SIGCOMM COMPUTER COMMUNICATION REVIEW, 1994 |
| C. L. HEDRICK, ROUTING INFORMATION PROTOCOL, 1988 |
| C. PERKINS; E. BELDING-ROYER; S. DAS, AD HOC ON-DEMAND DISTANCE VECTOR (AODV) ROUTING, 2003 |
| C.-C. TU; P.-W. WANG; T.-C. CHIUEH: "In-band control for an ethernet-based software-defined network", PROCEEDINGS OF INTERNATIONAL CONFERENCE ON SYSTEMS AND STORAGE, 2014 |
| D. B. JOHNSON; D. A. MALTZ: "Mobile computing", 1996, SPRINGER, article "Dynamic source routing in ad hoc wireless networks", pages: 153 - 181 |
| D. ORAN, OSI IS-IS INTRA-DOMAIN ROUTING PROTOCOL, 1990 |
| H. HAWILO; A. SHAMI; M. MIRAHMADI; R. ASAL: "Network", vol. 28, 2014, IEEE, article "NFV: state of the art, challenges, and implementation in next generation mobile networks (vEPC", pages: 18 - 26 |
| J. MOY, OPEN SHORTEST PATH FIRST ROUTING PROTOCOL (VERSION 2, 1994 |
| M. H. BEHRINGER; T. ECKERT; S. BJARNASON: "An Autonomic Control Plane", 2016, INTERNET ENGINEERING TASK FORCE |
| M. ONUS; A. W. RICHA; C. SCHEIDELER: "Linearization: Locally Self-Stabilizing Sorting in Graphs", ALENEX, 2007 |
| R. SHERWOOD; G. GIBB; K.-K. YAP; G. APPENZELLER; M. CASADO; N. MCKEOWN; G. PARULKAR: "Flowvisor: A network virtualization layer", OPENFLOW SWITCH CONSORTIUM, TECH. REP, 2009, pages 1 - 13 |
| S. KTARI; A. HECKER; H. LABIOD: "A construction scheme for scale free dht-based networks", GLOBAL TELECOMMUNICATIONS CONFERENCE, 2009 |
| S. SHARMA; D. STAESSENS; D. COLLE; M. PICKAVET; P. DEMEESTER: "Automatic bootstrapping of OpenFlow networks", LOCAL & METROPOLITAN AREA NETWORKS (LANMAN, 2013 |
| S. SHARMA; D. STAESSENS; D. COLLE; M. PICKAVET; P. DEMEESTER: "Enabling fast failure recovery in OpenFlow networks", DESIGN OF RELIABLE COMMUNICATION NETWORKS (DRCN, 2011 |
| S. SHARMA; D. STAESSENS; D. COLLE; M. PICKAVET; P. DEMEESTER: "Fast failure recovery for in-band OpenFlow networks", DESIGN OF RELIABLE COMMUNICATION NETWORKS (DRCN, 2013 |
| SOURABH JAIN ET AL: "VEIL: A Plug-&-Play Virtual (Ethernet) Id Layer for Below IP Networking", GLOBECOM WORKSHOPS, 2009 IEEE, IEEE, PISCATAWAY, NJ, USA, 30 November 2009 (2009-11-30), pages 1 - 6, XP031585793, ISBN: 978-1-4244-5626-0 * |
| T. CLAUSEN; P. JACQUET, OPTIMIZED LINK STATE ROUTING PROTOCOL (OLSR, 2003 |
| T. WINTER; P. THUBER; B. BRANDT: "RFC 6550: IPv6 Routing Protocol for Low-Power and Lossy Networks", INTERNET ENGINEERING TASK FORCE (IETF) REQUEST FOR COMMENTS, 2008 |
| W. WOJDAK: "Rapid Spanning Tree Protocol: A new solution from an old technology", REPRINTED FROM COMPACTPCI SYSTEMS, 2003 |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020177540A1 (fr) * | 2019-03-01 | 2020-09-10 | 中兴通讯股份有限公司 | Procédé de structuration en couches de logique de réseau basée sur un protocole rift, dispositif, dispositif de réseau, et support d'informations |
| US12132652B2 (en) | 2019-03-01 | 2024-10-29 | Zte Corporation | RIFT protocol-based network logical layering method and device, network device and storage medium |
| CN111638726A (zh) * | 2020-05-27 | 2020-09-08 | 北京理工大学 | 基于事件触发通信的多无人机编队一致性控制方法 |
| CN111638726B (zh) * | 2020-05-27 | 2021-07-27 | 北京理工大学 | 基于事件触发通信的多无人机编队一致性控制方法 |
| CN114024856A (zh) * | 2020-07-17 | 2022-02-08 | 中兴通讯股份有限公司 | 路由优化方法、物理网络设备及计算机可读存储介质 |
| US12476908B2 (en) | 2020-07-17 | 2025-11-18 | Zte Corporation | Route optimization method, physical network device and computer-readable storage medium |
| CN112312444A (zh) * | 2020-10-22 | 2021-02-02 | 深圳供电局有限公司 | 一种5g网络切片下资源备份方法 |
| CN112312444B (zh) * | 2020-10-22 | 2024-01-02 | 深圳供电局有限公司 | 一种5g网络切片下资源备份方法 |
| CN112383422A (zh) * | 2020-11-04 | 2021-02-19 | 浙江大学 | 加快一致性分布式算法收敛速度的网络拓扑优化方法 |
| CN112383422B (zh) * | 2020-11-04 | 2021-11-02 | 浙江大学 | 加快一致性分布式算法收敛速度的网络拓扑优化方法 |
| CN112600795A (zh) * | 2020-11-25 | 2021-04-02 | 中国人民解放军国防科技大学 | 一种不完全信息下的作战网络瓦解方法及系统 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6027629B2 (ja) | 最小限接続オブジェクト・ネットワークのための支援型インテリジェント・ルーティング | |
| EP1653687B1 (fr) | Réseau de commande séparé pour un routeur soft | |
| WO2018145761A1 (fr) | Plan de commande à topologie adaptative, basé sur un id structuré, pour 5g | |
| Ku et al. | Software-defined mobile cloud: Architecture, services and use cases | |
| US9634928B2 (en) | Mesh network of simple nodes with centralized control | |
| US9083627B2 (en) | Assisted traffic engineering for minimalistic connected object networks | |
| CN1819581B (zh) | 执行协议故障恢复的方法 | |
| US10893108B2 (en) | Maintaining application state of mobile endpoint device moving between virtualization hosts based on sharing connection-based metadata | |
| CN103428031B (zh) | 一种基于软件定义网络的域间链路快速故障恢复方法 | |
| US8406153B2 (en) | Affecting node association through load partitioning | |
| CN105052205A (zh) | 提供无服务中断的备份网络拓扑 | |
| Yuan et al. | OpenFlow based hybrid routing in Wireless Sensor Networks | |
| Vaghani et al. | A comparison of data forwarding schemes for network resiliency in software defined networking | |
| Municio et al. | Leveraging distributed protocols for full end-to-end softwarization in iot networks | |
| Rétvári et al. | OSPF for implementing self-adaptive routing in autonomic networks: A case study | |
| Bano et al. | A comparative analysis of hybrid routing schemes for SDN based wireless mesh networks | |
| EP2552151A1 (fr) | Procédé et système de contrôle centralisé d'associations de station mobile, routage et contrôle de débit dans les réseaux sans fil | |
| US20160105357A1 (en) | Method and network apparatus of establishing path | |
| Jamal et al. | VIBHAJAN: A lightweight and scalable control plane management for multi-controller SDN | |
| Vaghani et al. | Investigation of data forwarding schemes for network resiliency in POX software defined networking controller | |
| Abirami et al. | Depletion of vampire attacks in medium access control level using interior gateway routing protocol | |
| Malik et al. | Autonomous edge resource organization with Smallcell integration in 5G | |
| Fonti et al. | Enhancing Mobile Ad Hoc Networks (MANETs) with Software-Defined Networking (SDN): A Balanced Approach | |
| JP5898284B1 (ja) | 通信ネットワーク設計装置及び通信ネットワーク設計方法 | |
| Zhao et al. | Virtual Area Routing: A scalable intra-domain routing scheme |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17705346 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| 122 | Ep: pct application non-entry in european phase |
Ref document number: 17705346 Country of ref document: EP Kind code of ref document: A1 |