RU2825719C1 - Augmented reality for establishing internet connection - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to methods and systems for displaying a network device with augmented reality using a mobile device. Method comprises receiving an identification code from a network device by means of a mobile device; displaying captured images of a network device on a display of a mobile device, the images including a plurality of ports of the network device; retrieving information associated with the connection state of each of the plurality of ports using the identification code, wherein the extraction of information comprises accessing a remote database of identification data of devices connected to each of the plurality of ports and a library of virtual objects corresponding to the connected devices; determining, based on the remote database, identification data of devices connected to each of the plurality of ports; determining spatial relationship of mobile device relative to one or more ports using identification code; overlaying one or more virtual ports on top of a network device on images, wherein the virtual ports include information regarding the connection state of the network device in one or more virtual ports; and displaying icons of connected devices adjacent to virtual ports, wherein icons are displayed based on determined identification data of devices connected to each of plurality of ports.

EFFECT: generation of relevant information on the connection state of a network device and devices connected to it using an augmented reality mechanism.

35 cl, 36 dwg

Description

Cross-references to related applications

[0001] This patent application claims priority to U.S. Provisional Patent Application No. 62/900,403, entitled "AUGMENTED REALITY FOR INTERNET CONNECTIVITY INSTALLATION," filed September 13, 2019, and incorporated herein by reference in its entirety.

[0002] All publications and patent applications mentioned in this detailed description are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference.

Field of technology to which the invention relates

[0003] This disclosure generally relates to augmented reality applications, particularly as they relate to rendering augmented reality objects with images of network devices and related components, including servers, switches, access points, routers, and any of a number of other network devices.

State of the art

[0004] Computer networking equipment generally includes hardware devices that facilitate communication and interaction between devices in a computer network. Examples of network devices generally include switches, routers, gateways, and wireless access points that mediate the transmission of data in a computer network. When used in a business environment, network devices are typically used to connect computers and office equipment of various users, such as printers, to a local computer network and the Internet. A network device is typically mounted in a rack and organized in a rack with various other network devices. Sometimes the rack is located in a separate, temperature-controlled room with other racks of network devices. An IT specialist or technician must connect cables to the ports of the network devices to various other devices in the rack. From time to time, a network administrator must access one or more of the network devices to service or troubleshoot problems. Typically, the ports and cables look the same or similar to each other. Therefore, it can be difficult to identify which ports are used to connect to which network devices when looking at the various cables and network device ports in the rack.

[0005] When used in a home environment, network devices are typically used to wirelessly connect various personal computers and other electronic devices to the Internet. Consumers often establish their own Internet connection by connecting various cables to network devices or, in more complex cases, hire professionals to properly connect network devices. While user manuals can be helpful, the process can be confusing, especially for a non-technical person. If the Internet connection is not working, it can be difficult for the user to determine which devices are properly connected and working without time-consuming troubleshooting. Additionally, today's advancements and trends in home automation are leading to increased connectivity in the home environment. For example, real-time analytics, machine learning, sensor-based technologies, and embedded systems are converging to provide greater monitoring and control of various home appliances and systems in order to provide a high level of home automation. For example, lighting, HVAC, appliances, security systems, and other systems can be connected and centrally controlled, thereby providing improved convenience, comfort, energy efficiency, and security (such as in smart homes). As the connectivity network grows larger and more complex, the installation and maintenance of these networks will likely become more complex.

[0006] In general, it should be advantageous to provide software, systems and/or devices that enable a user to more easily install, maintain and troubleshoot network devices and related components, whether in a business or home environment.

The essence of the invention

[0007] This document describes augmented reality (AR) style systems and methods that include software that can be used to visualize network connectivity and other hidden aspects of electronic devices. The systems and methods use virtual objects as aids in installing, maintaining, managing, and/or repairing various network components. The systems and methods can be used by technical support personnel or information technology (IT) administrators in maintaining an organization's wireless and/or wired network system. In some cases, the systems and methods are used by a personal consumer of a network device during installation or troubleshooting of a home or home-office network environment.

[0008] Any of the methods, systems and devices described in this document can be used in combination with a mobile device, such as a mobile phone, a tablet computer, a laptop computer or a headset (e.g., a virtual reality headset). For example, a user can use a camera of a mobile device to capture an image or multiple images (e.g., in real-time imaging) of one or more network devices in order to view images of the network device on a display of the mobile device. In some cases, the methods are coded in an AR software application installed on and/or otherwise accessible by (e.g., via the Internet) the mobile device. In some cases, the software can be configured to operate in combination with other software (e.g., applications) accessible by the mobile device, such as commercially available AR software tools. Any AR software tool based on any operating system can be used. In some embodiments, AR tools and features of the ARKit development platform developed by Apple Inc., based in Cupertino, California, U.S.A., are used.

[0009] Although augmented reality technologies are known, the methods and systems described in this document include features that allow a user to quickly and visually determine the connection state of a network device in real time and, in some cases, various other network components. The network device may include an identification code or codes, such as an optical code, an RF code, etc. The optical code may include a visible marking that can be detected and correlated with the network device (e.g., a switch); examples of optical codes include, but are not limited to, QR codes, alphanumeric codes, symbols, etc. The optical code may correspond to a two-dimensional pattern code (e.g., a matrix bar code) that contains encoded information associated with the network device. The optical code may be displayed on the network device, for example, on the outer surface of the network device, or be otherwise easily accessible to the user. In some embodiments, the optical code is on a removable display, such as a touch screen display, that can be changed by the user to access different optical codes or other information.

[0010] Alternatively or additionally, the identification code of the network device may be a radio identification code corresponding to the network device. In some examples, the network device may transmit a radio signal, such as a Bluetooth signal, or may communicate with a mobile device via a radio frequency identification (RFID) code (tag) or a code (tag) according to the near field communication (NFC) standard. Radio identification may be used in combination with or instead of an optical code to identify the network device. Those skilled in the art may appreciate that the Bluetooth signal may have a frequency in the range from 2.402 Gigahertz to 2480 Gigahertz; and the radio frequency signal may have a frequency in the range from 20 Kilohertz to 300 Gigahertz.

[0011] The identification code may be uniquely associated with a specific network device. However, in some variations, the identification code may instead be associated with a specific subset of network devices, and the unique identification data of the device may be determined based on a secondary indicator, such as the identification data of the scanning device (e.g., a smartphone, a tablet computer, etc.), which may be associated with a specific user. For example, the identification code may identify a specific category, class, or subset of network devices, and the unique identification data of a specific device in this category, class, or subset of network devices may be uniquely identified by associated secondary information, such as the user identification data of the user performing the scan, the scanning device (e.g., a smartphone), one or more devices determined as associated or connected with the network device, etc.

[0012] An identification code (e.g., an optical code and/or a radio identification code) can be used to retrieve real-time information associated with the connection state of each of the ports of the network device. The identification code (e.g., an optical code and/or a radio identification code) can also be used to access a library of virtual objects based on the connection state of the port and the types of devices that are connected to the network. One or more virtual objects can then be combined (e.g., overlaid) with captured camera images of the network device to provide a visual representation of the ports, the various devices connected to the network, and other information related to the various devices connected to the network.

[0013] In some embodiments, the virtual objects include illustrations of communication ports (virtual ports) that are superimposed on corresponding captured images of the ports. For example, a user may view a port of a network device using a camera of a mobile device, and a virtual port may be displayed over at least a portion of the image of the port. The position of the virtual port may be automatically and dynamically adjusted based on the movement of the mobile device. For example, when the user moves the mobile device to capture an image of the network device from a different perspective, the position of the virtual port may be automatically adjusted to maintain its position over the image of the port such that the user experiences smooth transitions when moving and viewing different parts of the network device. In some embodiments, the position of the virtual object is stabilized by filtering one or more degrees of freedom used to determine the spatial relationship of the mobile device relative to the port(s).

[0014] The virtual objects may include one or more virtual labels with text, numbers, and/or symbols that convey information related to ports. For example, a virtual label on or near images of a port may include identification data of the port (e.g., a port number) and/or identification data of a device connected to the port. The virtual label may include a line that extends between the text, numbers, or symbols and the image of the port, indicating the relationship between the virtual label and the port.

[0015] The virtual objects may include illustrations of one or more devices connected or previously connected to the ports. The illustrations (icons) may provide a user with a quick visual cue as to which devices are associated with which ports. For example, an icon may be an illustration with sufficient detail for the user to determine whether the icon represents a personal computer, a printer, a router, an access point, a router, or another device. In some cases, information regarding the connected device is used to illustrate a particular model or brand of the device. For example, an icon may be an illustration of a particular brand or model of a telephone, television, laptop, headset, desktop computer, access point, media player, or tablet computer.

[0016] In some cases, the virtual object has a different appearance depending on the state of the port connection. For example, a virtual port may have a different color or opacity when the device is connected to the port compared to when the device is not connected to the port. In some cases, the icon has a different color or opacity when the device is connected to the port compared to when the device is not connected to the port. For example, the icon may have a grayed out or shaded appearance when the device is not currently connected, but was previously connected to the port for a specified time.

[0017] According to some embodiments, virtual objects can convey information regarding whether or not a device connected to a port of a network device has other devices connected to it. For example, if a network device connects to a wireless access point, a virtual label and/or illustration can indicate whether or not the access point device connects to one or more other devices, such as a telephone, a laptop, a tablet, or a headset. Virtual objects can also convey performance information of various devices connected to the network device, such as throughput or uptime. Virtual objects can convey such data in the form of a virtual graph or chart.

[0018] For example, the connection information may also or alternatively include port statistics regarding connected and/or unconnected ports on the network device. For example, a virtual display may include port statistics information for all or some connected ports; the user may toggle displaying/hiding any (all or a subset) of the port statistics information. The port statistics information may include, for example, throughput information (e.g., current throughput, average throughput for an hour, day, week, etc., MAC addresses, IP addresses, etc.). In some variations, the port statistics information may include historical information (e.g., showing one or more connections established in the past, when connected/disconnected, etc.). In some variations, the port statistics information may be displayed for currently unconnected ports. Port statistics may be displayed as text, icons, or some combination of both. This information may be displayed through user selection (e.g., selecting on a touch screen to expand or collapse additional information). Thus, port connectivity information may be displayed as part of a virtual object, in addition to a virtual object, or separately from a virtual object.

[0019] These and other features and advantages of AR methods, systems and devices are described in detail in this document.

[0020] Any of the methods described in this document may be performed by means of equipment (e.g., a system, device, etc.) configured to perform a method including the optional steps described in this document. For example, this document describes systems for displaying a network device with augmented reality using a mobile device. This system may include: receiving an identification code from a network device via a mobile device; displaying captured images of the network device taken by a camera on a display of the mobile device, wherein the images include a plurality of ports of the network device; extracting information associated with a connection state of each of the plurality of ports using the identification code; determining a spatial relationship of the mobile device relative to one or more ports using the identification code; and overlaying one or more virtual ports on top of the network device on the images, wherein the virtual ports include information regarding the connection of the network device in the one or more virtual ports.

[0021] The identification code may be an optical code (e.g., a bar code, QR codes, etc.) or an RF code (e.g., a Bluetooth code, NFC, etc.). Any of these methods may include accessing a library of virtual objects based on information associated with a port connection state, the virtual objects may include one or more icons corresponding to a type of device connected to one or more ports of a network device (e.g., a camera, a phone, a computer, an access point, etc.). The virtual ports may include one or more icons. When displayed, these icons may be used to indicate information regarding the connected device(s).

[0022] For example, a method for displaying a network device with augmented reality using a mobile device may include: displaying captured images of a network device taken by a camera on a display of the mobile device, wherein the images include: one or more ports of the network device and an optical code on the network device that include network connection information associated with the network device; extracting information associated with a connection state of the one or more ports using the optical code; accessing a library of virtual objects based on the information associated with the connection state of the one or more ports, wherein the virtual objects include an icon corresponding to a type of device connected to each of the one or more ports of the network device; determining a spatial relationship of the mobile device relative to the one or more ports using the optical code; and overlaying the one or more virtual ports, wherein the virtual ports include an icon corresponding to a type of device connected to the one or more ports, over the image of the network device, wherein the quality of the virtual port indicates the connection state of the port based on the extracted information.

[0023] A system for displaying a network device with augmented reality using a mobile device may include one or more processors; a memory device coupled to the one or more processors, wherein the memory device is configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: receiving an identification code from the network device via the mobile device; displaying captured images of the network device taken by a camera on a display of the mobile device, wherein the images include a plurality of ports of the network device; extracting information associated with a connection state of each of the plurality of ports using the identification code; determining a spatial relationship of the mobile device relative to the one or more ports using the identification code; and overlaying one or more virtual ports on top of the network device on the images, wherein the virtual ports include information regarding the connection of the network device in the one or more virtual ports.

[0024] One or more processors may be part of a mobile device (e.g., a smartphone, tablet computer, etc.).

[0025] Any of these methods and devices for implementing them may be designed to operate in real time or near real time.

[0026] For example, a method for displaying a network device with augmented reality using a mobile device, wherein the method comprises: capturing a video image of a network device using a mobile communication device, wherein the video image includes a plurality of ports; receiving, in the mobile communication device, an identification code specific to the network device, from the network device; determining, from the identification code, information regarding the connection of the plurality of ports of the network device; displaying on the video image, in real time, an overlay on the plurality of ports of the network device, wherein the overlay includes a network connection indicator specific to each port, wherein the network connection indicator comprises one or more of the following: connection status, connection speed, data traffic, connection identification data, connection duration, and the use of power over Ethernet (POE) technology.

[0027] A method for displaying a network device with augmented reality using a mobile device may include: capturing a video image of a network device using a mobile communication device, wherein the video image includes code specific to the network device; determining, from the code, information regarding the connection of one or more ports of the network device; displaying on the video image, in real time, an overlay on the one or more ports of the network device, including a network connection indicator specific to each port, wherein the network connection indicator comprises one or more of the following: connection status, connection speed, data traffic, connection identification data, connection duration, and the use of power over Ethernet (POE) technology.

[0028] The network device may be any appropriate network device having one or more ports, such as, for example, a switch, a router, an access point, etc. The mobile communication device may comprise a smartphone or a tablet computer. The code may comprise a QR code, such as a digital QR code. The digital QR code may be updated or modified to optically transmit information to a handheld device (e.g., a smartphone, a tablet computer, etc.).

[0029] Any of these methods may include determining the locations of each of one or more ports of the network device from the code and the image. For example, any of these methods may include determining the locations of each of one or more ports of the network device from the code and the image and the orientation of the mobile communication device.

[0030] The determination may comprise determining information regarding a connection of one or more ports of a network device by accessing a remote database using a mobile communication device and using a code to identify information regarding the one or more ports. In any of these methods, the one or more ports may be uniquely identified.

[0031] A method for displaying a network device with augmented reality using a mobile device may include: capturing a video image of a plurality of network devices using the mobile communication device, wherein each of the network devices comprises a plurality of ports and a unique identification code; receiving, in the mobile communication device, a unique identification code specific to each of the network devices, of the network devices of the plurality of network devices; determining, from the unique identification code specific to each of the network devices, information regarding the connection of one or more ports of each of the network devices; displaying on the video image, in real time, an overlay on each of the one or more ports of the network devices, including a network connection indicator specific to each port, wherein the network connection indicator comprises one or more of the following: a connection state, a connection speed, data traffic, connection identification data, a connection duration, and the use of power supply over Ethernet (POE) technology; and updating the display in real time as the network connection indicator changes.

[0032] For example, a method for displaying a network device with augmented reality using a mobile device may include: capturing a video image of a plurality of network devices using a mobile communication device, wherein the video image includes a digital QR code specific to each of the network devices, in the plurality of network devices; determining, from the digital QR codes, information regarding the connection of one or more ports of each of the network devices; displaying on the video image, in real time, an overlay on each of the one or more ports of the network devices, including a network connection indicator specific to each port, wherein the network connection indicator comprises one or more of the following: connection status, connection speed, data traffic, connection identification data, connection duration and use of power over Ethernet (POE) technology; and updating the display in real time as the network connection indicator changes.

[0033] Also described in this document are systems for displaying a network device with augmented reality using a mobile device. For example, a system may include: one or more processors; a memory device coupled to the one or more processors, wherein the memory device is configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: capturing a video image of a network device using a mobile communication device, wherein the video image includes a plurality of ports; receiving, at the mobile communication device, an identification code specific to the network device, from the network device; determining, from the identification code, information regarding the connection of the plurality of ports of the network device; displaying on the video image, in real time, an overlay on the plurality of ports of the network device, wherein the overlay includes a network connection indicator specific to each port, wherein the network connection indicator comprises one or more of the following: connection status, connection speed, data traffic, connection identification data, connection duration, and use of power over Ethernet (POE) technology.

[0034] A system for displaying a network device with augmented reality using a mobile device may include: one or more processors; a memory device associated with the one or more processors, wherein the memory device is configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: capturing a video image of a plurality of network devices using a mobile communication device, wherein each of the network devices comprises a plurality of ports and a unique identification code; receiving, in the mobile communication device, a unique identification code specific to each of the network devices, from the network devices of the plurality of network devices; determining, from the unique identification code specific to each of the network devices, information regarding a connection of one or more ports of each of the network devices; displaying on a video image, in real time, an overlay on each of the one or more ports of the network devices, including a network connection indicator specific to each port, wherein the network connection indicator comprises one or more of the following: connection status, connection speed, data traffic, connection identification data, connection duration, and use of power over Ethernet (POE) technology; and updating the display in real time as the network connection indicator changes.

[0035] The methods and equipment described in this document may be configured to be used with multiple network devices (e.g., multiple switches, routers, etc.) that include a plurality of ports. For example, in some variations, the method may include receiving an identification code (e.g., a unique identification code) for each of a plurality of network devices, such as a rack of devices. The identification codes may be optical, RF, or a combination of optical and RF codes. These methods and equipment may enable a user to switch between displays of virtual ports for each of the plurality of devices. For example, a user may move a mobile device up/down a rack of network devices to change the view showing the virtual ports corresponding to each; alternatively or additionally, the user may select a control for the mobile device to switch an augmented reality style view between different network devices. In some variations, several different network devices may be displayed together at the same time.

Brief description of the drawings

[0036] The novel features of the invention are set forth in detail in the following claims. A better understanding of the features and advantages of the present invention should be obtained by reference to the following detailed description, which sets forth illustrative embodiments in which the principles of the invention are used, and to the accompanying drawings, in which:

[0037] Fig. 1A and 1B illustrate an example of a user using the devices and methods described herein.

[0038] Fig. 2A illustrates an example of a rack of network devices having optical codes.

[0039] Fig. 2B illustrates an exemplary display screen for displaying one or more optical codes.

[0040] Figs. 3A-3C illustrate examples of optical codes generated using a code generator.

[0041] Figs. 4A-4F illustrate exemplary user interfaces for setting up and using a mobile device in AR mode.

[0042] Figs. 5A-5C illustrate other exemplary user interfaces for setting up a mobile device in AR mode.

[0043] Fig. 6 illustrates an example of an AR overlay for use in conjunction with captured images for viewing a network device.

[0044] Fig. 7A and 7B illustrate another example of user interfaces using a mobile device in AR mode.

[0045] Fig. 8 illustrates an exemplary user interface that includes various virtual objects related to the performance of one or more network devices.

[0046] Figs. 9A-9E illustrate aspects of example three-dimensional coordinate systems used to render AR objects.

[0047] Fig. 10A-10D illustrate a flow chart of a method and an architectural map representing an example process for setting up and using an AR application.

Detailed description of the invention

[0048] In general, this document describes augmented reality (AR) style systems, devices and methods for viewing hidden features of electronic devices. In particular embodiments, AR systems, devices and methods are used to visualize virtual objects that convey information related to a network connection of electronic devices. The virtual objects can provide a visual representation to a user regarding the network connection state of a network device, thereby simplifying the establishment and management of a computer network. The user can use a camera of a mobile device, such as a mobile phone or tablet computer, to capture a real-time image stream of the network device. The virtual objects can be visualized on the display of the mobile device along with the captured images to provide an augmented reality style experience for the user.

[0049] Fig. 1A and 1B illustrate one example of a user 101 using the AR devices and methods described herein. The user 101 may be any user, such as technical support personnel or an IT administrator who installs or manages an internal work network, or a personal consumer who installs or manages a home network. The user 101 may use the mobile device 100 to visually observe hidden aspects related to the network device 102 and which are to be used to install, troubleshoot, and/or check the status of the network device 102.

[0050] In general, the network device 102 may be any electronic device that is capable of connecting, wired and/or wireless, to one or more computer networks. The computer network may include one or more networks that range in size from wide area computer networks to a nanoscale computer network. The computer network may be a local area network, a wide area network, such as a cloud network (such as the Internet), or a combination of local and wide area networks. For example, the computer network may be a local area network that interconnects computers in an organization, apartment, or school, which may also be connected to the Internet. In some cases, the network device 102 is configured to mediate data transmission in one or more computer networks. The network device 102 may be a network switch (also called a "switched hub") that connects devices in a computer network using packet switching to receive, process and forward data to a target device. The network device 102 may be a gateway device that provides data flow between different networks. The network device 102 may be a router that forwards data packets between computers. The network device 102 may be a digital hub that receives information from various sensors and controllers (nodes) that can form an "intelligent" wirelessly connected network for a residential space (e.g., home, office, work, etc.). The network device 102 may be a network bridge that creates an aggregated network of several communication networks or network segments. The network device 102 may be a repeater that receives and retransmits signals. The network device 102 may be an Ethernet hub for connecting several Ethernet devices together. The network device 102 may be a hybrid network device, such as a multilayer switch, a protocol converter and/or a bridge router (bridge router). In some embodiments, the network device is a UniFi Dream Machine Pro (UDMP) device or another network device manufactured by Ubiquiti Networks, based in New York, New York, USA. The network device 102 may be an edge network device that is permanently located at the connection point of various networks, such as a proxy server, a firewall or a network address translator (NAT). The network device 102 may be an end station device used to create networks or dial-up connections, such as a network interface controller (NIC), a wireless network interface controller, a modem, an ISDN terminal adapter or a line control signal conditioner. The network device 102 may be a server that provides functionality for one or more clients. The network device 102 may be one of several network devices that are located on a rack of network devices 110, in this case, in a rack.

[0051] The devices and methods described in this document use virtual objects such that the user 101 can see hidden aspects of the network device 102 or of a number of network devices on a rack 110. To initiate the AR tools, the user 101 can use a scanning device, such as a mobile device 100, to scan an optical code 104 on the network device 102, as shown in Fig. 1A. For example, the user can use the camera of the mobile device 100 to capture one or more images of the optical code 104. The optical code 104 can be any machine-readable code that contains information regarding the network device 102. The optical code 104 can be a two-dimensional pattern that encodes information associated with the network device 102. For example, the optical code 104 can be a quick response (QR) code, an ARTag code, a bar code (for example, a linear bar code), an alphanumeric code, a numeric code, written letters, or any combination of the above. The optical code 104 can be located on the surface of the network device 102, for example, on an external surface that is easily accessible to the user 101. The optical code 104 can be attached to the surface using any means. For example, the optical code 104 can be painted on the surface of the network device or on a label adhered to the surface of the network device. In some cases, the optical code 104 is generated by means of a display on the network device 102.

[0052] The optical code 104 may include one or more identifiers (e.g., a unique identifier) associated with the network device 102, so that information associated with the network device 102 can be accessed. For example, when a user captures one or more images of the optical code 104, the mobile device 100 may send the identifier(s) to a network management system (e.g., to one or more remote servers). When receiving an identifier from the mobile device 100, the network management system can send information to the mobile device 100 and/or the network device 102 regarding the network device 102. For example, the mobile device 100 can have access to a database of information regarding the network device 102. The database can include a library of virtual objects based on the identification data of the network device 102, the connection state of the network device 102, and the identification data of any devices connected to the network device 102. The information can include a type (for example, a brand and model) of the network device 102, as well as information regarding one or more communication interfaces, such as hardware or software interfaces. The information can include information regarding one or more communication ports 108 of the network device 102. The port 108 can be any type of interface between the device and other computers or peripheral devices. Port 108 may be an input or output interface for a device. Port 108 may include a hardware interface, including but not limited to a universal serial bus (USB) port, a serial port, a video port, an audio port, a high-definition multimedia interface (HDMI) port, or a parallel port. In some cases, the communication interface is a wireless interface that wirelessly connects to one or more computers.

[0053] After the mobile device 100 receives information associated with the network device 102, this information can be used to render virtual objects on the display of the mobile device 100 along with the captured camera images in real time. In some embodiments, the virtual objects overlay the captured camera images. The virtual objects can include illustrations and/or text that convey information that is not normally obvious to the user 101. The virtual objects can include virtual ports 108, cables 106, or other illustrations of parts of the network device 102. The virtual objects can include text, numbers, and labels that convey information regarding the identification data and/or the state of any devices connected to the network device 102 via the ports 108 and/or via wireless communication.

[0054] The virtual objects may include icons corresponding to illustrations of various devices connected to the network device 102. In some cases, the icons may be displayed next to the captured images of the ports 108 so that the user 101 can quickly identify the type and connection of the various devices. The icons may illustrate any type of device connected to the network, such as a computer or another device (e.g., a sensor). For example, the computer may be a personal computer, such as a desktop computer, or a mobile device (e.g., a laptop, tablet, phone, or headset). The computer may be a server computer configured to be shared by one or more users of the network. The computer may be a server or a client. The computer may be a router, such as a wireless router. The computer may be a modem. The computer may be a printer or other computing device for setting up office work. The computer may be an access point device configured to allow Wi-Fi devices to connect to the network. The computer may be a smart home device such as a thermostat, smoke or carbon monoxide detector, security system, doorbell, smart lock, kitchen appliance. The computer may be a camera, speaker, or microphone device. The computer may be a Bluetooth device.

[0055] In some embodiments, the mobile device 100 can dynamically display virtual objects in real time. Thus, the virtual objects can be rendered on the display of the mobile device 100 during or close enough to the time that images captured by the camera of the mobile device are rendered on the display in such a way that the user can experience substantially continuous changes in the images, much like a video with virtual objects included therein. As the camera of the mobile device 100 captures a stream of images over time, the virtual objects can be rendered on the display along with the stream of images. Thus, when the user 101 moves the mobile device 100 relative to the network device 102, for example, when scanning different ports 108 of the network device 102, the virtual objects can dynamically update themselves based on the progress of changes in the captured images. For example, virtual objects can maintain their positions on the display relative to the corresponding ports 108 as the mobile device 100 moves. This spatial capability can be provided using a spatial coordinate system, which can depend on one or more motion sensors of the mobile device 100 to detect movement of the mobile device 100. For example, the mobile device 100 can include a motion sensor system, which can include an accelerometer and/or a gyroscope.

[0056] The virtual object(s) can update information regarding the connection state of the various ports 108 in real time. In this way, the user 101 can use the information provided by the virtual objects to troubleshoot the network device 102, for example, by connecting and disconnecting one or more cables 106 from the ports 108, or by turning on and off various devices connected to the network device 102. In some embodiments, in which data is transmitted in network packets (e.g., with packet switching), the virtual objects can provide the ability for the user 101 to perform inspection of the supplied packets for security controls. The user 101 can also check the connection state of each of the ports 108 in real time to determine, for example, whether or not the network device 102 is properly connected to various devices in the network. The virtual objects can transmit information regarding the state of various devices in the network, including secondary, tertiary, etc. devices that connect to these devices. Other network devices 102 on the rack 110 may similarly be viewable using AR, such that the user 101 can scan all the network devices on the rack 110. In this way, the AR tools described in this document may enable the user 101 to quickly and easily determine the state of the entire network system, including the network subsystems.

[0057] Fig. 2A shows an example of network devices 202a-202f having different optical codes 204a-204f that encode information related to the respective network devices 202a-202f. As shown, the optical codes 204a-204f may have any of a number of different patterns. For example, the optical codes 204a-204f may be a quick response (QR) code, an ARTag code, a bar code (e.g., a linear bar code), an alphanumeric code, a numeric code, written letters, or any combination of the above. The different patterns may encode different types of information based on the identification data and the state of the respective network device. In some cases, the optical codes 204a-204f are displayed on displays (e.g., flat panel displays) of the network devices 202a-202f. Fig. 2B illustrates an example of a display component that can be configured to display one or more optical codes. The displays can be of any type, such as liquid crystal displays (LCDs) and/or light-emitting diode (LED) displays (e.g., organic light-emitting diode (OLED) displays). In some embodiments, the displays are touch displays that are capable of displaying different images based on touch input. For example, touch displays can be configured to change what is displayed in response to a user swiping a finger across a touch screen.

[0058] In some cases, the network device(s) may not have an LCD display or other type of visually active screen, but may instead use a static optical code (such as a label or other printed identifier) associated with the network device.

[0059] As mentioned above, the identification code may be a radio frequency identification (e.g. RF) code, such as a radio identification code (e.g. Bluetooth, NFC, RFID), which can be used by a handheld device to identify (e.g. uniquely identify) a network device. The RF identification code may be dynamic (similar to an optical code on a screen), or it may be static (e.g. as a printed optical code attached to the outside of the network device).

[0060] Fig. 3A-3C show examples of three different optical codes that include different code templates. These examples illustrate how different templates can encode unique information based on a network device, including a connection state of the network device. Information from the optical codes can then be sent, for example, to a cloud management system that sends information associated with the network device to a mobile device and/or a network device, and which is used to generate virtual objects. In some examples, the optical codes are generated using a template generator program that (e.g., randomly) generates templates to provide unique optical codes.

[0061] Fig. 4A-4F illustrate examples of user interfaces for setting up and using the mobile device 400 in the AR mode, according to some embodiments. Fig. 4A shows a user using the camera of the mobile device 400 to view several network devices 402, each of which has a corresponding optical code (e.g., 404). The mobile device 400 can be considered an example of the mobile device 100. The user interface 420 can be displayed on the display screen of the mobile device and guide the user through a series of interface screens for the initial setup for the mobile device 400 in order to enter the AR mode. The user interface 420 can include the geographic location of the network devices, the types (e.g., brand and model) of the network devices, and/or illustrations of the network devices. The user interface 420 can be video images of the network devices 402 captured on the camera of the mobile device 400. The user interface 420 can include a button for entering the AR mode.

[0062] Fig. 4B1 and Fig. 4B2 illustrate two views of an exemplary user interface 421 before scanning an optical code of a network device. The user interface 421 may instruct the user to scan each of the network devices such that information regarding each of the network devices can be available and downloaded for use in the AR mode. The user interface 421 may include an optical code alignment frame 422 for aligning captured images of the optical code alignment frame 422 of each network device.

[0063] Fig. 4C1-4C4 illustrate four views of an exemplary user interface 422 during scanning of optical codes of multiple network devices of a rack. After scanning a first optical code of a first network device, the user interface may indicate to the user to scan a second optical code of a second network device. A visual indicator, such as a representation of the optical code alignment frame 422 with a different color or shade, may be used to indicate capturing of the optical code in the optical code alignment frame. Alternatively or additionally, another visual indicator, such as a check mark symbol 427, may become visible to indicate that the optical code has been successfully scanned. In some embodiments, a tactile and/or audio indicator, such as vibration and/or sound generated by a mobile device, may be used to indicate successful capturing and/or scanning of the optical code. The user interface may include a list 424 or table of network devices that is gradually updated as each of the optical codes of the network devices is successfully scanned. For example, an icon 425 illustrating a corresponding network device may populate the list 424 after the optical code for a network device is successfully scanned and/or recognized (e.g., by a network management system). The list 424 may then be populated with an additional icon 426 after the optical code for an additional network device is successfully scanned and/or recognized (e.g., by a network management system). This scanning of different optical codes for different network devices may continue, for example, until all network devices on the rack have been scanned.

[0064] Fig. 4D1-4D4 illustrate four views of an exemplary user interface 430 showing how AR settings can be applied after optical codes of network devices are successfully scanned. In one view, the user interface 430 may display a list 432 of scanned network devices that have been added, with an indication 433 (e.g., a button) for the user to respond whether or not additional network devices should be scanned. The user interface 430 may provide an indication 434 (e.g., a button) for the user to proceed to the next view of the user interface. After all desired network devices have been scanned, the user interface may provide an indication 435 (e.g., a button) for the user to optionally enter a name for a rack (e.g., a "rack") of network devices, and provide an indication 436 (e.g., a button) for the user to proceed to the next view of the user interface. Any of the user interface views 430 may include a return indication 438 to return to the available view. The AR settings for the network devices may then be applied to the network rack based on the information received via the optical codes and the information provided by the user. Once the settings are applied and completed, the full AR experience may be loaded at this point so that the mobile device may use the camera in AR mode for the newly created rack of network devices.

[0065] Fig. 4E shows an example of a mobile device after switching to AR mode. As shown, the display of the mobile device 400 can display a user interface 450, which can include one or more virtual objects 401 that overlay captured images of network devices and that can convey information related to a network device that is normally hidden from the user.

[0066] Fig. 4F shows a detailed view of the user interface 450 in AR mode according to some embodiments. The virtual objects may include text (e.g., numbers, letters, and/or symbols) and/or illustrations. For example, the virtual identifying text 452 of the network device, such as the name and/or Internet Protocol (IP) address of the network device, may be positioned next to/on top of (e.g., partially on top of) the image of the network device. The virtual objects may include a port area 454 that at least partially frames various virtual objects related to a particular network device. The virtual objects may include an optical code alignment frame 456 for aligning captured images of the optical code. The optical code alignment frame 456 can be used to help the user align the optical code in the capturing region A of the optical code alignment frame 456 for capturing the optical code. The capturing region A can be located in the central region of the optical code alignment frame 456. In some embodiments, the port region 454 and/or the optical code alignment frame 456 can have a transparent or semi-transparent quality that allows the basic image of the network device to be visible. In some cases, the virtual ports B overlay the corresponding ports of the network device. The virtual ports B can have the shape and size of the corresponding physical ports. The virtual ports B can include numbers that correspond to port numbers identified, for example, by the manufacturer of the network device. The virtual ports B can be filled with color C and/or be at least partially transparent to allow the basic image of the network device to be visible. In some cases, the color and/or translucent quality may vary depending on the state of the corresponding port. For example, virtual B ports that are actively connected to a computer may be filled with the first (e.g., darker) color and/or translucent quality, while virtual B ports that are not actively connected to a computer may be hollow (i.e., transparent) or filled with the second (e.g., lighter) color and/or translucent quality.

[0067] The virtual objects may include F icons (also called "virtual devices" or "virtual computers") that match the devices that connect to the corresponding ports. The F icons may be illustrations that have sufficient detail so that the user can identify the type of device (e.g., a telephone, a tablet computer, a laptop computer, a desktop computer, a television receiver, an access point, a virtual assistant device, a home appliance, or a security system device), and so that the user can easily and quickly distinguish different types of computers from each other. The F icons may be selected from a database (also called "libraries") of virtual object illustrations that are either automatically assigned or selected by the user. The relative position of the F icons of the computer may be contained in the E icon area.

[0068] The virtual objects may include a virtual device identification text H that describes the identification data of the computer. The virtual device identification text H may include information to help further identify the device that is connected or has previously been connected to the corresponding port. For example, the user name and/or the brand and model of the device may be identified in the virtual device identification text H. The virtual device identification text H may be automatically filled in based on information provided, for example, by a cloud network management system, or selected by the user. The relative position of the virtual device identification text H may be contained in the device identification text region G.

[0069] In some embodiments, the F icon and/or the H identification text of the virtual device may have a different quality based on whether the device is actively connecting to the corresponding port or not. For example, the F icon and/or the H identification text of the virtual device may have a fully colored or opaque appearance when the device is actively connecting to the port, have a shaded or grayed appearance when the device is not currently connecting but has previously connected to the port for a prescribed period of time, and be absent (i.e., areas E and G do not have an illustration or text) when the device should not connect to the port for a prescribed period of time. The prescribed period of time may vary. For example, the prescribed period of time may not exceed approximately 30 minutes, 1 hour, 2 hours, 3 hours, 6 hours, 12 hours, 24 hours, 3 days, or 1 week. In some embodiments, the prescribed period of time may be selected by the user. In some cases, the F icon and/or the H identification text of a virtual device include information regarding the operational state of the device. The operational state may include performance characteristics (e.g., real-time or statistical) of the device.

[0070] In some cases, the icon F and/or the identification text H of the virtual device may include an indicator regarding whether or not the device is connected to another device (e.g., a secondary, tertiary, etc. device). For example, an access point device may be functionally connected to one or more telephones, tablet computers, laptop computers, and/or headsets. The icon F may have a different appearance, such as being presented in a different color, surrounded by a halo or shadow, or provided with some other attribute that indicates a connection to one or more additional devices. Alternatively or additionally, the identification text H of the virtual device may include text that indicates a connection to one or more additional devices. In some cases, the icon F and/or the identification text H of the virtual device include information regarding the operational state of one or more additional devices.

[0071] Aspects of the user interface may vary, for example, depending on the display size, form factor, and/or operating system of the mobile device. For example, mobile phones may have a smaller height or width compared to tablet computers. The user interface may be designed to accommodate these differences.

[0072] Figs. 5A-5C illustrate other example user interfaces for setting up a mobile device in AR mode. Fig. 5A illustrates a user interface 521 before scanning an optical code, similar to the user interface 421 (Figs. 4B1-4B2), except that the user interface 521 includes a list 523 that can be filled in by network devices and that is laterally positioned relative to the optical code alignment frame 522. Fig. 5B illustrates a user interface 523 during scanning an optical code, similar to the user interface 423 (Figs. 4C1-4C4), except that the user interface 523 includes a list 526 that can be filled in by network devices and that is laterally positioned relative to the optical code alignment frame 522. 5C illustrates a user interface 530 for applying AR settings, similar to the user interface 430 (FIGS. 4D1-4D4), except that an indication 535 (e.g., a button) for the user to select to complete the AR settings is presented at the bottom side (e.g., right side) of the user interface 530.

[0073] As described herein, optical codes may include encoded information that provides access to a library of virtual objects. Optical codes may act as fiduciary markers to provide information regarding the appearance and placement of virtual objects. Relative positions of the virtual objects may be placed in an overlay that overlays captured images, such as by a camera of a mobile device. Fig. 6 illustrates an example of an overlay 600 according to some embodiments. At least a portion of the overlay 600 may be substantially transparent such that the base images can be viewed when rendered on a display of a mobile device. An optical code alignment frame 630 may define an area of the overlay 600 that is configured to scan and receive an image of an optical code. In some cases, the optical code alignment frame 630 may be outlined by a border or frame and/or have a central marker (e.g., a circle). In other cases, the optical code alignment frame 630 may not have boundaries or frames.

[0074] After the optical code is scanned and verified by the network management system (e.g., local or cloud-based) as associated with a specific network device, a library of virtual objects associated with the network device may be accessible and viewed on the display of the mobile device. For example, the optical code may inform the virtual object management regarding the brand and model of the network device and all associated physical characteristics, such as types, locations and sizes of ports. This information may be used to generate one or more virtual ports 634 corresponding to the actual port(s) of the network device, accessible from a database of various types of virtual ports. Additionally, the network management system may access information related to the connection state of the ports. This information may be used to determine various aspects of the virtual objects that convey information related to the connection state of one or more ports 634.

[0075] The overlay 600 may include a port region 632 that defines a region of the overlay 600 that includes virtual ports 634. The port region 632 may or may not include a visible outline or frame. In some cases, the port region 632 has a different color and/or transparency compared to other portions of the overlay 600. The virtual ports 634 may include text, such as numbers, to identify port numbers. In some cases, the virtual ports 634 include one or more status indicators (e.g., 638 and 639) that indicate the connection state of the port. The status indicators may have a different appearance (e.g., have different colors or shapes) based on whether the port is actively connected to the computer, has previously been connected to the computer for a prescribed period of time, or has not been connected to the computer for a prescribed period of time. Such status indicators (e.g. 638 and 639) may be used as a supplement or as an alternative to the fill quality described above with reference to Fig. 4E.

[0076] AR can provide video tracking characteristics that calculate the position and orientation of a camera relative to physical markers, such as an optical code on a network device, in real time. For example, after the position of the camera is known, a virtual camera can be positioned at an identical point, revealing a virtual object (for example, an optical code alignment frame 630) at the location of the optical code. Information from a network management system regarding a specific network device can be used to provide precise placement and alignment of virtual ports 634 relative to the optical code. This information can also be used to determine relative distances between virtual ports 634, as well as shapes and sizes of virtual ports 634. Information from the network management system can further be used to change aspects of the virtual objects in real time based on the connection state of various ports, as described in this document.

[0077] The spatial relationship between the mobile device and the network device can change in real time because the user can move the mobile device relative to the network device, thereby projecting images of the network device with different perspectives. In order to adapt to perspective changes, the formation of virtual objects is typically based on tracking the movement based on rotational degrees of freedom around fixed orthogonal axes (X, Y, Z axes), similar to a gimbal system. The movements can be tracked using traditional three-dimensional rotational movements, such as pitch movement in the lateral direction (rotation around the X axis), pitch movement in the longitudinal direction (rotation around the Y axis), and pitch movement about the vertical axis (rotation around the Z axis) to define six degrees of freedom. While this model is useful, the data associated with continuous monitoring and sensing of movements in real time can be significant, which can lead to a lag before virtual objects are formed on the display.

[0078] In any of the embodiments described herein, the spatial relationship may be simplified by eliminating one or more degrees of freedom. For example, the position of the network device is substantially stable, and the ports on the network device are generally oriented along a plane (e.g., on the back of the network device, perpendicular to the floor). Thus, one of the three-dimensional degrees of freedom may be assumed to be fixed and excluded from the calculations for determining the spatial relationship between the mobile device and the network device. In some embodiments, this is achieved by filtering out (e.g., ignoring) some raw data associated with three-dimensional movement of the mobile device that is not necessary for calculating the relative movement of the mobile device with respect to the ports. For example, in some embodiments, movements associated with pitching movement in the lateral direction (rotation around the X-axis) may be excluded from the calculation, thereby reducing the degrees of freedom and information required for the calculations by one third. Filtering data in this way allows for faster computations so that virtual objects (e.g., virtual ports) in captured images can be updated more quickly. Additionally, it can reduce the amount of correction required to update each image frame, thereby resulting in a virtual object having a more stable (e.g., less oscillating) location and appearance in real time.

[0079] Additionally, AR capabilities may use optical feature recognition to determine locations for virtual ports 634. In some examples, optical feature recognition may be used in addition to or instead of a spatial relationship between a mobile device and a network device. In some examples, optical feature recognition may include optical character recognition (OCR).

[0080] More specifically, using OCR, any relevant text on the hardware (e.g., port numbers, actual ports, etc.) can be identified by the camera to help place AR features in the overlay. This can reduce some sensor drift or other alignment issues that may occur through dead reckoning or other motion-based feature overlays. In addition, using OCR, specific identification information of the network device can be displayed and verified, and then a search for virtual ports can be performed.

[0081] For example, referring briefly again to Fig. 1, if the network device 102 is a 24-point network switch, then the text on the label of the network device 102 can be captured via OCR and can be identified as a 24-point switch. The mobile device 100 can then display identification information, namely, "This is a 24-point switch", and then the virtual ports of the network device can be searched for AR overlays, as mentioned above.

[0082] In some embodiments, after the OCR is performed, the mobile device 100 may display details of the expected configuration of the network device. For example, the mobile device 100 may display a list of which ports of the network device are expected to be connected and how they are expected to be connected by displaying a message (header) "Expected" or "Expected configuration", before starting the port search. In various embodiments, the above steps may be performed sequentially or in parallel.

[0083] Fig. 7A and 7B illustrate another example of a user interface 720 of a mobile device 700 in AR mode, according to some embodiments. Fig. 7A and 7B show images of a network device 720 rendered on a display of a mobile device 700 in real time, in two different perspective views. The user interface 720 may include one or more virtual objects that overlay portions of the displayed images of the network device 702. The virtual objects may include virtual ports 740 and 741, virtual cables 742, icons 750, and virtual labels 752 (e.g., text). In some cases, a port region 732 surrounding the virtual ports is outlined by a virtual line 745. The positions of the virtual objects may be configured to change in accordance with a change in perspective by the mobile device using the spatial coordinate model described herein. In this way, the virtual objects can actually maintain their alignment with the corresponding objects in the captured images. In this way, the virtual ports 740 and 741, the virtual cables 742, the icons 750 and the virtual labels 752 can appear to move as the mobile device 700 moves relative to the network device. One or more virtual objects can change based on the connection state of the corresponding one or more ports. For example, the virtual port 740 can have an outline of a first color (for example, white), indicating that the corresponding port is connected to the device, and the virtual port 741 can have an outline of a second color (for example, blue), indicating that the corresponding port is not connected to the device.

[0084] Any of the AR devices and methods described in this document may include virtual objects that indicate the performance of a network device and/or devices connected to the network device. Fig. 8 illustrates a detailed view of a user interface of a display of a mobile device in AR mode showing examples of virtual objects that transmit performance data. The virtual objects may include various selectable performance metrics, such as throughput 862 and uptime 864 of the network device. When a selectable performance metric is selected, one or more virtual charts or graphs 860 may be displayed. In Fig. 8, the graphs 860 indicate throughput values related to devices connected to the network device. The throughput metrics may include data related to the throughput of the devices, such as transmission control protocol (TCP) throughput or file transfer time. The uptime metrics may include the percentage of time that devices are up. Other metrics may include the usage of Internet traffic and/or the power consumption of the network device and/or devices connected to the network device. In some cases, the metrics may include the duration that one or more devices are connected to the network device. In some embodiments, a tree graph illustrating the various (e.g., primary, secondary, tertiary, etc.) devices that are connected or are at some point connected to the network device. In some cases, the metrics may include usage data related to a specific user of the device. For example, a graph or chart may be displayed indicating the amount of time the user uses social media or websites.

[0085] The user interface may include selectable virtual icons that the user can select by touching (e.g., using a touch screen of the mobile device) and/or by other selectable methods, such as an electronic pencil. The virtual icons may include a port icon 870, a power icon 872, a performance icon 874, and a settings icon 876. When the port icon 870 is selected, various virtual ports and virtual port labels may become visible, as described herein. When the power icon 872 is selected, virtual objects related to a power source and battery status and usage may become visible. When the performance icon 874 is selected, virtual objects related to performance, such as throughput 862 and uptime 864, may become visible. When the settings icon 876 is selected, virtual objects related to network configuration and settings may become visible.

[0086] As described herein, AR technologies may be based on the use of a three-dimensional coordinate system (3D XYZ). Figs. 9A-9E illustrate aspects of a three-dimensional XYZ axis system used as a basis for the AR methods described herein according to some embodiments, wherein X represents width, Y represents height, and Z represents depth. The three-dimensional XYZ system may ensure that spatial data used to generate AR objects is understandable (e.g., consistent). In some embodiments, a cross-platform software mechanism is used to establish the three-dimensional XYZ system. For example, the Unity3D game engine developed by Unity Technologies, based in San Francisco, California, USA, may be used. A point positioned in 3D space that has a width of 1 (on the X-axis), a height of 4 (on the Y-axis), and a depth of 3 (on the Z-axis) can be represented as a Vector3 {x:1, y:4, z:3} using the Unity3D system.

[0087] The three-dimensional spatial record may be captured during scanning of the optical code (e.g., see Fig. 4C1-4C4 or 5B). For example, an AR application on a mobile device (e.g., a phone, a tablet, or a headset) may use a camera and/or an orientation determination system (e.g., an accelerometer, a gyroscope, and/or a compass) of the mobile device to "capture" the three-dimensional spatial record of the network device and "remember" the configuration of the user's rack. This three-dimensional spatial capture may be a one-time layout. Fig. 9B illustrates an example of a three-dimensional spatial record for a rack 910 of network devices 902. The spatial record of each of the network devices 902 may be established as sets 909 of vectors in a three-dimensional XYZ coordinate system. The three-dimensional spatial record corresponds to a group of normalized positions (e.g., Vector3) based on trackers that store their local position. The three-dimensional spatial recording may be filtered as described in this document. In one implementation of the filtering, the local position may be considered to be on a "panel" (e.g., on a two-dimensional surface) since the orientation of the network device is typically fixed (e.g., with a bottom surface that is parallel to the ground), so that there is substantially no rotation about the X-axis. One of the trackers may be designated as an origin for the spatial recording, and the relative locations of the other trackers may be recorded relative to this origin. For example, the first optical code 904a (e.g., scanned first) may be designated as having an origin, and the relative locations of the remaining optical codes 904 (e.g., scanned thereafter) may be recorded relative to the first optical code 904a. In some cases, the starting optical code 904a is designated as the origin. The coordinate system may be based on any unit of measurement. In some embodiments, the coordinate system is based on a metric system (e.g., meters). It should be noted that in some embodiments, a Vector1 or Vector2 structure may be used instead of a Vector3. However, in some cases, a Vector3 may provide greater ability to support more complex network device layouts.

[0088] Data may flow between the 3D XYZ platform and native code of the mobile device, such as Swift or Java. Fig. 9C shows an example of code in JavaScript Object Notation (JSON) for a network setup device. Fig. 9D illustrates an example of using a Hypertext Transfer Protocol (HTTP) request implemented in native code and using additional data (rack_name for racks and model names and numbers for network devices) for the user interface during the spatial recording capture process. Fig. 9E also illustrates an example of JSON code for setting up multiple network devices.

[0089] Fig. 10A-10D illustrate a flow chart 1000 of a method and a corresponding architectural map 1100 representing an example process for setting up and using an AR application, according to some embodiments. Each of the steps in the flow chart 1000 of the method includes an indicator of a corresponding architectural component (e.g., Swift, Unity AR, network controller, or USW/LCM) in a corresponding map 1100. Map 1100 indicates the interaction of the components of the network controller, Swift, Unity AR, USW, and LCM.

[0090] Fig. 10A illustrates a portion of a flow chart 1000 of a method and a map 1100 corresponding to initializing an application with network information. Referring to the flow chart 1000, at 1002, a user can use a mobile device to retrieve a network application programming interface (API) for an AR application from a network controller to a Swift component. At 1004, the network JSON is filtered, for example, according to one or more filtering methods described in this document. At 1006, the network JSON data is parsed in a Unity AR component. At 1008, a spatial record is retrieved by means of a Swift component. At 1010, if the spatial record is successfully retrieved ("Yes"), the process continues to Fig. 10D, and if the spatial record is not successfully retrieved ("No"), the process continues to Fig. 10B.

[0091] Fig. 10B illustrates a portion of the flow chart 1000 of the method and the map 1100 corresponding to the distribution of the seed numbers of the tracer. Referring to the flow chart 1000 of the method, if it is determined that the spatial record is not successfully retrieved ("No") in Fig. 10A, at 1012, the MAC/tracer pairs are composed by the Unity AR component and sent to the Swift component. At 1014, the MAC/tracer pairs are published on the network controller by the Swift component. At 1016, the MAC/tracer pairs are distributed to the LCM and the USW, and at 1018, the tracer is formed from the seed numbers (e.g., to form QR codes).

[0092] Fig. 10C illustrates a portion of a flow chart 1000 of a method and a map 1100 corresponding to an arrangement (e.g., a one-time arrangement) for storing a spatial record. Referring to the flow chart 1000 of a method, at 1020, tracers are scanned and used to record a spatial record and sent to a Swift component. At 1022, the spatial record is placed in a network controller, and at 1024, the spatial record is received and stored by the network controller.

[0093] Fig. 10D illustrates a portion of a flow chart 1000 of a method and a map 1100 corresponding to resuming regular use of a mobile device and using an AR application based on a spatial record. Referring to the flow chart 1000 of the method, at 1026, the spatial record is retrieved by a Swift component such that the tracers in the LCM from the spatial record (USW/LCM) can be resumed. Additionally, the network controller responds to the spatial record at 1030. At 1032, the spatial record is parsed in the Unity AR component, and at 1034, any of the displayed tracers can be scanned. At 1036, the AR overlay for the rack can be recreated in a Unity AR component to compose an AR user interface (1038), with auto-switching based interaction (1040), and visualize port details (1042).

[0094] Although many of the examples described in this document relate to network devices such as switches, routers, and wireless access points, the AR technologies described in this document are not limited to these types of devices. For example, the AR technologies described in this document can be used to view any type of computer or other device capable of connecting to one or more computer networks. In some cases, the AR technologies described in this document are used to visualize hidden aspects of connectivity of mobile devices such as mobile phones, tablet computers, laptop computers, or headsets. The AR technologies described in this document can be used to visualize hidden aspects of connectivity of wireless communication features (i.e., not limited to hardware ports). For example, connectivity aspects related to one or more antennas (e.g., RF or Bluetooth chips) can be visualized using AR objects.

[0095] When a feature or element is referred to herein as being "in" another feature or element, it may be directly within the other feature or element, or intervening features and/or elements may also be present. In contrast, when a feature or element is referred to as being "directly within" another feature or element, intervening features or elements are not present. It should also be understood that when a feature or element is referred to as being "connected," "attached," or "linked" to another feature or element, it may be directly connected, attached, or linked to the other feature or element, or intervening features or elements may be present. In contrast, when a feature or element is referred to as being "directly connected," "directly attached," or "directly related" to another feature or element, intervening features or elements are not present. Although described or shown with respect to one embodiment, the features and elements so described or shown may be applied to other embodiments. Those skilled in the art will also appreciate that references to a structure or feature that is located "adjacent" to another feature may have portions that overlap or underlie the adjacent feature.

[0096] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. For example, as used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should be further understood that the terms "comprises" and/or "comprising," as used in this detailed description, specify the presence of the recited features, steps, operations, elements, or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, or groups thereof. As used herein, the term "and/or" includes any and all combinations of one or more associated recited elements and may be abbreviated as "/."

[0097] Spatially relative terms such as "under," "below," "lower," "above," "upper," and the like may be used herein for ease of description to describe the relationship of one element or feature to another element(s) or feature(s) as illustrated in the drawings. It should be understood that spatially relative terms are intended to cover various orientations of the device in use or operation, in addition to the orientation illustrated in the drawings. For example, if the device in the drawings is turned over, elements described as "under" or "below" other elements or features would then be oriented "above" the other elements or features. Thus, the exemplary term "under" may cover both an above and below orientation. The device may be oriented in other ways (rotated 90 degrees or in other orientations), and spatially relative descriptors used herein are interpreted accordingly. Similarly, the terms "up", "down", "vertical", "horizontal", etc. are used in this document for explanatory purposes only, unless otherwise expressly stated.

[0098] Although the terms "first" and "second" may be used herein to describe various features/elements (including steps), these features/elements should not be limited by these terms unless the context indicates otherwise. These terms may be used to distinguish one feature/element from another feature/element. Thus, the first feature/element explained below may be referred to as the "second feature/element", and similarly, the second feature/element explained below may be referred to as the "first feature/element", without departing from the teachings of the present invention.

[0099] As used herein in the detailed description and claims, including when used in the examples, and unless otherwise expressly indicated, all numbers may be read as if preceded by the word "about" or "approximately," even if the term is not explicitly shown. The phrase "about" or "approximately" may be used when describing an absolute value and/or position to indicate that the value and/or position described is within a reasonably expected range of values and/or positions. For example, a numerical value may have a value that is +/- 0.1% of a stated value (or range of values), +/- 1% of a stated value (or range of values), +/- 2% of a stated value (or range of values), +/- 5% of a stated value (or range of values), +/- 10% of a stated value (or range of values), etc. Any range of numerical values set forth herein is intended to include all subranges included within it.

[0100] Although various illustrative embodiments are described above, any number of changes may be made to the various embodiments without departing from the scope of the invention as described by the claims. For example, the order in which the various described method steps are performed may often be changed in alternative embodiments, and in other alternative embodiments, one or more method steps may be omitted altogether. Optional features of various embodiments of the device and system may be included in some embodiments and not in others. Accordingly, the foregoing description is provided primarily for exemplary purposes and should not be interpreted as limiting the scope of the invention as set forth in the claims.

[0101] The examples and illustrations included herein show, by way of illustration and not limitation, specific embodiments in which the inventive subject matter may be practiced. As mentioned above, other embodiments may be used and subtracted from them, and structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Such embodiments of the inventive subject matter may be referred to herein individually or collectively by the term "invention" merely for convenience and without the intent to intentionally limit the scope of this application to any one invention or inventive concept where more than one is actually disclosed. Thus, although specific embodiments are illustrated and described herein, any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of the various embodiments. Combinations of the above-described embodiments and other embodiments not specifically described herein will become apparent to those skilled in the art upon review of the above description.

Claims (80)

1. A method for displaying a network device with augmented reality using a mobile device, wherein the method comprises the steps of: - receive an identification code from a network device via a mobile device; - displaying captured images of the network device taken by the camera on the display of the mobile device, wherein the images include a plurality of ports of the network device; - extracting information associated with the connection state of each of the plurality of ports using an identification code, wherein extracting the information comprises accessing a remote database of identification data of devices connected to each of the plurality of ports and a library of virtual objects corresponding to the connected devices; - determine, based on a remote database, the identity of devices connected to each of a plurality of ports; - determining the spatial relationship of a mobile device relative to one or more ports using an identification code; - overlaying one or more virtual ports on top of the network device on the images, wherein the virtual ports include information regarding the connection state of the network device in the one or more virtual ports; and - display icons of connected devices adjacent to virtual ports, where the icons are displayed based on specific identification data of the devices connected to each of the plurality of ports. 2. The method according to claim 1, wherein the alignment of one or more virtual ports relative to the images captured by the camera is dynamically adjusted based on a certain spatial relationship. 3. The method according to claim 1, wherein each of the virtual ports additionally indicates the connection state of the port based on the extracted information. 4. The method according to claim 1, further comprising the step of superimposing a virtual label on the image, next to each of the virtual ports, wherein the virtual label includes text indicating one or more characteristics of a device connected to each of the ports corresponding to each of the virtual ports. 5. The method according to claim 1, wherein each of the virtual ports has a size and shape that substantially coincide with the size and shape of the image of one or more ports. 6. The method according to claim 1, wherein the position of the virtual port is dynamically changed in accordance with changes in the perspective of the camera of the mobile device relative to the network device. 7. The method according to claim 1, wherein each of the virtual ports has a different appearance when the corresponding port in the network device is connected, compared to the case when the corresponding port is not connected to the device. 8. The method of claim 1, further comprising the step of accessing a library of virtual objects based on information associated with the connection state of the port, wherein the virtual objects include one or more icons corresponding to a type of device connected to one or more ports of the network device. 9. The method of claim 8, wherein the device type is a mobile phone, a tablet computer, a laptop computer, a desktop computer, a wireless access point device, a virtual assistant device, a television receiver, a home appliance, or a security system device. 10. The method of claim 8, wherein one or more icons have a different appearance when the port is connected to the device, compared to the case where the port was previously connected and is no longer connected to the device. 11. The method of claim 1, wherein determining a spatial relationship of the mobile device relative to the network device comprises filtering movement data from a calculation for determining the spatial relationship. 12. The method of claim 11, wherein filtering the movement data comprises a step of excluding movement data associated with rotation about an axis in coordinate space. 13. The method according to claim 1, wherein the identification code is an optical code that is located on the touch screen of the network device. 14. The method according to claim 1, wherein the identification code is an RF code. 15. The method of claim 1, wherein the port connection state includes whether or not the port is connected to the device. 16. The method according to claim 1, wherein the port connection state includes information associated with the operating state of a computer connected to the port or that is connected to the port. 17. The method according to claim 1, wherein the port connection state includes information regarding whether or not a device that is connected or connected to the port is connected to a second device. 18. The method according to claim 17, wherein the port connection state includes an operating state of the device and an operating state of the second device. 19. The method according to claim 1, wherein the mobile device is a mobile phone, tablet computer, laptop computer or headset. 20. The method according to claim 1, wherein the network device is a network switch, router or access point. 21. The method of claim 1, further comprising the step of dynamically adjusting the overlay of one or more virtual ports. 22. A method for displaying a network device with augmented reality using a mobile device, wherein the method comprises the steps of: - displaying captured camera images of a network device on a display of a mobile device, wherein the images include: one or more ports of the network device and an optical code on the network device that includes network connection information associated with the network device; - extracting information associated with the connection state of one or more ports using an optical code, wherein the extracted information includes a database of identification data of devices connected to each of the one or more ports and a library of virtual objects corresponding to the connected devices; - determine, based on the retrieved database, the identity of devices connected to each of one or more ports; - access a library of virtual objects based on information associated with the connection state of one or more ports, wherein the virtual objects include an icon corresponding to the type of device connected to each of the one or more ports of the network device; - determining the spatial relationship of a mobile device relative to one or more ports using an optical code; - overlaying one or more virtual ports, wherein the virtual ports include an icon corresponding to the type of device connected to the one or more ports, over the image of the network device, wherein the color and/or opacity of the virtual port indicates the connection state of the port based on the information extraction; and simultaneously display icons of connected devices adjacent to the virtual ports, wherein the icons are displayed based on specific identification data of the devices connected to each of the plurality of ports. 23. A system for displaying a network device with augmented reality using a mobile device, the system comprising: - one or more processors; - a storage device associated with one or more processors, wherein the storage device is configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: - receiving an identification code from a network device via a mobile device; - displaying captured images of a network device taken by a camera on a display of a mobile device, wherein the images include a plurality of ports of the network device; - extracting information associated with the connection state of each of the plurality of ports using an identification code, wherein the extracted information includes a database of identification data of devices connected to each of the plurality of ports and a library of virtual objects corresponding to the connected devices; - determining the spatial relationship of a mobile device with respect to one or more ports using an identification code; - overlaying one or more virtual ports on top of a network device on images, wherein the virtual ports include information regarding the connection of the network device in the one or more virtual ports; and simultaneously display icons of connected devices adjacent to the virtual ports, wherein the icons are displayed based on specific identification data of the devices connected to each of the plurality of ports. 24. A method for displaying a network device with augmented reality using a mobile device, wherein the method comprises the steps of: - capture a video image of a network device using a mobile communication device, wherein the video image includes multiple ports; - receive, at the mobile communications device, an identification code specific to the network device, from the network device; - determining, from the identification code, connection information of a plurality of ports of the network device, wherein the connection information includes a database of identification data of devices connected to each of the plurality of ports and a library of virtual objects corresponding to the connected devices; and - displaying on a video image, in real time, an overlay on a plurality of ports of a network device, wherein the overlay includes a network connection indicator specific to each port, wherein the network connection indicator comprises one or more of the following: connection status, connection speed, data traffic, connection identification data, connection duration, icons of connected devices, and the use of power over Ethernet (POE) technology, In this case, the icons are determined from the connection information. 25. The method according to claim 24, wherein the network device comprises a switch. 26. The method according to claim 24, wherein the mobile communication device comprises a smartphone or a tablet computer. 27. The method according to claim 24, wherein the identification code comprises an optical code. 28. The method according to claim 24, wherein the identification code comprises a digital QR code. 29. The method according to claim 24, wherein the identification code comprises an RF code. 30. The method of claim 24, further comprising the step of determining the locations of each of the one or more ports of the network device from the identification code and the image. 31. The method according to claim 24, further comprising the step of determining the locations of each of the ports of the plurality of ports from the identification code and the image and orientation of the mobile communication device. 32. The method according to claim 24, wherein the determining comprises a step of determining information regarding the connection of a plurality of ports of the network device by accessing a remote database using a mobile communication device. 33. A method for displaying a network device with augmented reality using a mobile device, wherein the method comprises the steps of: - capture a video image of a plurality of network devices using a mobile communication device, wherein each of the network devices contains a plurality of ports and a unique identification code; - receive, in a mobile communication device, a unique identification code specific to each of the network devices, of the network devices from a plurality of network devices; - determining, from a unique identification code specific to each of the network devices, connection information of one or more ports of each of the network devices, wherein the connection information includes a database of identification data of devices connected to each of the plurality of ports and a library of virtual objects corresponding to the connected devices; - displaying on a video image, in real time, an overlay on each of the one or more ports of the network devices, including a network connection indicator specific to each port, wherein the network connection indicator comprises one or more of the following: connection status, connection speed, data traffic, connection identification data, connection duration, icons of connected devices, and use of Power over Ethernet (POE) technology, wherein the icons are determined from the connection information; and - Update the display in real time as the network connection indicator changes. 34. A system for displaying a network device with augmented reality using a mobile device, the system comprising: - one or more processors; - a storage device associated with one or more processors, wherein the storage device is configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: - capturing a video image of a network device using a mobile communication device, wherein the video image includes multiple ports; - receiving, in a mobile communications device, an identification code specific to a network device from a network device; - determining, from the identification code, connection information of a plurality of ports of a network device, wherein the connection information includes a database of identification data of devices connected to each of the plurality of ports and a library of virtual objects corresponding to the connected devices; - displaying on a video image, in real time, an overlay on a plurality of ports of a network device, wherein the overlay includes a network connection indicator specific to each port, wherein the network connection indicator comprises one or more of the following: connection status, connection speed, data traffic, connection identification data, connection duration, icons of connected devices and use of power over Ethernet (POE) technology, wherein the icons are determined from the connection information. 35. A system for displaying a network device with augmented reality using a mobile device, the system comprising: - one or more processors; - a storage device associated with one or more processors, wherein the storage device is configured to store computer program instructions that, when executed by the one or more processors, perform a computer-implemented method comprising: - capturing a video image of a plurality of network devices using a mobile communication device, wherein each of the network devices contains a plurality of ports and a unique identification code; - receiving, in a mobile communication device, a unique identification code specific to each of the network devices, from among the network devices of a plurality of network devices; - determining, from a unique identification code specific to each of the network devices, connection information of one or more ports of each of the network devices, wherein the connection information includes a database of identification data of devices connected to each of the plurality of ports and a library of virtual objects corresponding to the connected devices; - displaying on a video image, in real time, an overlay on each of one or more ports of network devices, including a network connection indicator specific to each port, wherein the network connection indicator comprises one or more of the following: connection status, connection speed, data traffic, connection identification data, connection duration, icons of connected devices, and use of Power over Ethernet (POE) technology, wherein the icons are determined from the connection information; and - Update display in real time as network connection indicator changes.