CN111316333B - Information prompting method and electronic equipment - Google Patents

Abstract

An information prompting method and an electronic device, the method comprising: when the electronic device operates a real-scene navigation function, the electronic device collects images of the surrounding environment of the user, generates a real scene image, and obtains information of points of interest from an electronic map, the electronic device Recognize the real scene image and determine the target object. The electronic device determines the label of the target object and the display position of the label in the real scene image according to the information of the point of interest and the location information of the electronic device. The electronic device display includes the label and the real scene In the image interface, the label is superimposed and displayed on the display position of the real scene image, so that the calculation workload of the electronic device can be reduced and power consumption can be saved.

Description

An information prompt method and electronic device

technical field

The present application relates to the technical field of terminals, in particular to an information prompting method and electronic equipment.

Background technique

Augmented reality (augmented reality, AR) is a new technology that integrates the real world and the virtual world by combining computing capabilities such as images and positions, targeting objects in the real world. At present, as some companies successively launch AR development tools and increase investment in the field of AR, the heat of this technology continues to be hot. Real-world navigation is one of the main applications of augmented reality. For example, the recently announced AR real-world navigation function in Google Map (Google Maps), which introduces AR technology to pedestrian navigation, has attracted widespread attention in the industry.

In practical applications, the application of the AR real-world navigation function is mainly reflected in two scenarios: (1) navigation scenarios with routes, that is, the user has a clear target address and may move forward according to the guidance of the navigation route or route label; (2) ) Browsing scene without roads, that is, the user has no clear goal, but will refer to the building information or store information provided in the AR real-world navigation function.

In the prior art, the building information or store information provided by the function of AR real-world navigation is usually obtained by extracting text from images collected on-site through optical character recognition technology, because electronic devices also need real-time The text content is analyzed, so the calculation amount of the electronic device is relatively large, resulting in a large power consumption of the electronic device.

Contents of the invention

The present application provides an information prompting method and an electronic device, which are used to realize a real-scene navigation function with a small amount of calculation in the electronic device.

In the first aspect, an embodiment of the present application provides an information prompt method, the method is applicable to an electronic device with a navigation function, and the method includes: when the electronic device runs the real-scene navigation function, the electronic device collects images of the user's surrounding environment , generate a real scene image, and obtain the information of the point of interest from the electronic map, then, the electronic device recognizes the real scene image, determines the target object, and then the electronic device information, determine the label of the target object and the display position of the label in the real scene image, and finally the electronic device displays an interface including the label and the real scene image, wherein the label is superimposed and displayed on the real scene image The display position of the real scene image.

In the embodiment of the present application, after the electronic device collects images of objects in the surrounding environment, it can only recognize typical buildings in the image, such as walls, ceilings, floors, building outlines, store floors, floor intervals, etc., without It is necessary to recognize the names of objects in many images that have been acquired, for example, it is not necessary to recognize the names of shops, so the calculation amount of electronic devices can be greatly reduced, and the power consumption of electronic devices can be saved.

In a possible design, the electronic device can acquire information of points of interest in a set area from an electronic map according to the shooting direction of the electronic device and the position information of the electronic device, wherein the set area includes the electronic device's Location information, the information of the point of interest includes the name of the point of interest and the coordinates of the point of interest. Then the electronic device determines the first relative positional relationship between the target object in the real scene image and the electronic device, and determines the relationship between the interest point and the electronic device according to the coordinates of the interest point and the position information of the electronic device. The second relative positional relationship of the user, and then the electronic device determines the target point of interest matching the target object according to the first relative positional relationship and the second relative positional relationship, and finally the electronic device determines the target point of interest according to the target point of interest information to generate the label of the target object. It can be seen that through the above method, the electronic device can obtain information such as the name of the target object from the point of interest, so the electronic device does not need to identify the names of the objects in many acquired images, and the calculation amount of the electronic device is reduced.

In a possible design, the electronic device determines the coordinates of the target object according to the coordinates of the target interest point; then the electronic device determines the center of the label of the target object in the real scene image according to the coordinates of the target object and the position information of the electronic device Coordinates, the final electronic device determines the display position of the label in the real scene image according to the center coordinates.

In the embodiment of the present application, the electronic device displays the label at the corresponding position of the real scene image by determining the center coordinates of the label. Since the label can include information such as the name of the store, it can provide clear and accurate information to the user.

In a possible design, the display position of the electronic device is at the upper middle position of the wall of the target object, and the display position is not in the display area of any of the following features of the real scene image: sky, ceiling, road, Floors, stairs, stair spaces. The advantage of this is that the location of the label display is more friendly and humanized, providing users with a better experience.

In a possible design, when the electronic device determines that the location information of the electronic device belongs to outdoors, the features recognized by the electronic device from the real scene image include any one or any combination of the following: walls, roads, sky, The outline of the building, and determine the target object according to the identified features; when the electronic device determines that the location information of the electronic device belongs to indoors, the features recognized by the electronic device from the real scene image include any one or any combination of the following : Ceiling, ground, road, stairs, stair space, store wall, and determine the target object according to the recognized features.

That is to say, in the embodiment of the present application, the electronic device will identify the corresponding features in a targeted manner according to the difference between indoor and outdoor, so that the target objects identified in different scenes can be more accurate.

In a possible design, when the electronic device determines that the brightness of the surrounding environment of the electronic device changes according to the information collected by the ambient light sensor of the electronic device, the electronic device determines the The display parameters of the label are adjusted, and the display parameters include any one or any combination of the following: font color, background transparency. Or, in a possible design, when the electronic device determines that the background brightness of the real scene image changes, the electronic device displays the label according to the current background brightness of the real scene image The parameters are adjusted, and the display parameters include any one or any combination of the following: font color, background transparency. In this way, the problem of unclear label display can be avoided. Even if the electronic device is in motion or in different time periods, the real scene image captured by the electronic device will change due to the change of the background environment of the image, which improves the user's Use experience.

In a possible design, when the electronic device determines that the electronic device enters the room from outdoors according to the positioning result of the electronic device, the electronic device automatically switches the label display style from the outdoor label display style to the indoor label display style. style; or, when the electronic device determines that the electronic device enters the outdoor from the indoor according to the positioning result of the electronic device, the electronic device automatically switches the display style of the label from the indoor label display style to the outdoor label display style.

Wherein, the outdoor label display style includes any one or any combination of the following: the content of the label includes the name of the target object and the distance between the target object and the electronic device, and the labels are loosely arranged in the real scene image, And the arrangement is irregular; the indoor label display style includes any one or any combination of the following: the content of the label includes the name of the target object, the distance between the target object and the electronic device, and the floor where the target object is located, The tags are arranged compactly in the real scene image, and are arranged according to the floor level where the target object is located.

In this way, in the real-scene navigation mode, the electronic device can adaptively display indoor and outdoor scenes and typical scene labels without manual selection by the user, which makes the real-scene navigation function more intelligent and improves user experience.

In a second aspect, an embodiment of the present application provides an electronic device, including a processor, a memory, a display, and a camera. Wherein, the memory is used to store one or more computer programs; when the one or more computer programs stored in the memory are executed by the processor, the electronic device can implement any possible design method of the first aspect.

In a third aspect, an embodiment of the present application further provides an electronic device, where the electronic device includes a module/unit for performing the method in the first aspect or any possible design of the first aspect. These modules/units can be realized by hardware, and can also be realized by executing corresponding software by hardware.

In the fourth aspect, the embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium includes a computer program, and when the computer program is run on the electronic device, the electronic device is made to execute any possible method of design.

In the fifth aspect, the embodiment of the present application further provides a method including a computer program product, and when the computer program product is run on the electronic device, the electronic device is made to execute any possible design method.

These or other aspects of the present application will be more concise and understandable in the description of the following embodiments.

Description of drawings

FIG. 1 is a schematic diagram of a communication scenario provided by an embodiment of the present application;

FIG. 2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of the architecture of an Android operating system provided in an embodiment of the present application;

FIG. 4 is a schematic diagram of an information prompt method provided by an embodiment of the present application;

Figure 5 and Figure 6 are schematic diagrams of an application scenario interface provided by the embodiment of the present application;

FIG. 7 is a schematic diagram of a spatial position relationship provided by an embodiment of the present application;

Fig. 8a is a schematic diagram of a three-dimensional spatial position relationship provided by the embodiment of the present application;

Fig. 8b is a schematic diagram of a display screen coordinate system provided by the embodiment of the present application;

Figure 8c and Figure 8d are schematic diagrams of the horizontal and vertical coordinate projections of a label provided by the embodiment of the present application;

FIG. 9 is a schematic diagram of an outdoor scene interface provided by the prior art;

FIG. 10 is a schematic diagram of an outdoor scene interface provided by an embodiment of the present application;

FIG. 11 is a schematic diagram of an indoor scene interface provided by an embodiment of the present application;

FIG. 12 is a schematic diagram of another outdoor scene interface provided by the embodiment of the present application;

Fig. 13 is a schematic structural diagram of the device provided by the embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Among them, in the description of the embodiments of the present application, the terms "first" and "second" are used for description purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features . Thus, a feature defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

The information prompt method provided by the embodiment of the present application can be applied to the communication scene shown in Figure 1, the electronic device 100 and the server 200, the electronic device 100 obtains the information of the points of interest in the electronic map from the server 200, and then calculates the information in the real scene image The label of the object and its display position in the real scene image.

Wherein, the electronic device and the server are interconnected through a communication network, and the communication network may be a local area network, or a wide area network through a relay (relay) device. When the communication network is a local area network, for example, the communication network may be a short distance communication network such as a wifi hotspot network, a wifi P2P network, a bluetooth network, a zigbee network or a near field communication (near fieldcommunication, NFC) network. When the communication network is a wide area network, exemplary, the communication network may be a third-generation communication technology (3rd-generation wireless telephone technology, 3G) network, a fourth-generation communication technology (the 4th generation mobile communication technology, 4G) network , the fifth-generation mobile communication technology (5th-generation mobile communication technology, 5G) network, the future evolution of the public land mobile network (public land mobile network, PLMN) or the Internet, etc.

或者其他操作系统的便携式电子设备。上述便携式电子设备也可以是其他便携式电子设备，诸如具有触敏表面(例如触控面板)的膝上型计算机(laptop)等。还应当理解的是，在本申请其他一些实施例中，上述电子设备100也可以不是便携式电子设备，而是具有触敏表面(例如触控面板)的台式计算机。In some embodiments of the present application, the electronic device 100 shown in FIG. 1 may be a portable electronic device with a navigation function and an image acquisition function, such as a mobile phone, a tablet computer, a wearable device with a wireless communication function (such as virtual reality glasses) wait. Exemplary embodiments of portable electronic devices include, but are not limited to

Or portable electronic devices with other operating systems. The aforementioned portable electronic device may also be other portable electronic devices, such as a laptop computer (laptop) with a touch-sensitive surface (such as a touch panel). It should also be understood that, in some other embodiments of the present application, the above-mentioned electronic device 100 may not be a portable electronic device, but a desktop computer with a touch-sensitive surface (such as a touch panel).

Exemplarily, as shown in FIG. 2 , the following uses the electronic device 100 as an example to describe the embodiment in detail.

The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a USB interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a communication module 150, a wireless communication module 160, Audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and SIM card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and the like.

It can be understood that, the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.

The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor ( image signal processor, ISP), controller, memory, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (Neural-network Processing Unit, NPU), etc. . Wherein, different processing units may be independent devices, or may be integrated in one or more processors. Wherein, the application processor is mainly used to start the camera when detecting the user's operation of enabling the real scene navigation function, and the graphics processor is used to recognize the real scene image collected by the camera.

Wherein, the controller may be the nerve center and command center of the electronic device 100 . The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.

A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.

In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuitsound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver (universal asynchronous receiver) /transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and/or A universal serial bus (universal serial bus, USB) interface, etc.

The USB interface 130 is an interface conforming to the USB standard specification, specifically, it may be a Mini USB interface, a Micro USB interface, a USB Type C interface, and the like. The USB interface can be used to connect a charger to charge the electronic device 100, and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.

It can be understood that the interface connection relationship between the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.

The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger.

The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the external memory, the display screen 194 , the camera 193 and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). The wireless communication function of the electronic device 100 can be realized by the antenna module 1, the antenna module 2 communication module 150, the wireless communication module 160, the modem processor and the baseband processor and so on.

Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Cellular network antennas can be multiplexed as WLAN diversity antennas. In other embodiments, the antenna may be used in conjunction with a tuning switch.

The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (Low Noise Amplifier, LNA) and the like. A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 . The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (wireless local area networks, WLAN), bluetooth (bluetooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near fieldcommunication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (code division multiple access, CDMA), wideband code Wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM , and/or IR technology, etc. The GNSS may include global positioning system (global positioning system, GPS), global navigation satellite system (global navigation satellite system, GLONASS), Beidou satellite navigation system (beidou navigation satellite system, BDS), quasi-zenith satellite system (quasi-zenith) satellite system, QZSS)) and/or satellite based augmentation systems (satellite based augmentation systems, SBAS).

The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

The display screen 194 is used to display virtual reality images, videos, etc. formed by superimposing labels and real scene images. The display screen 194 includes a display panel. The display panel can be LCD (liquid crystal display, liquid crystal display), OLED (organic light-emitting diode, organic light emitting diode), active matrix organic light emitting diode or active matrix organic light emitting diode (active-matrix organic light emitting diode, AMOLED), flexible light-emitting diode (flex light-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light emitting diodes (quantum dot light emitting diodes, QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens, where N is a positive integer greater than 1.

The electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.

The ISP is used for processing the data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 193 .

Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a complementary metal-oxide-semiconductor (complementary metal-oxide-semiconductor, CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some embodiments, the electronic device 100 may include 1 or N cameras, where N is a positive integer greater than 1.

Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.

Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in various encoding formats, such as: MPEG1, MPEG2, MPEG3, MPEG4 and so on.

The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 . The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.

The internal memory 121 can be used to store computer-executable program codes (for example, navigation applications, or other algorithms or applications callable by navigation applications, etc.), and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 . The memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as Baidu map, Gaode map, etc.) and the like. The storage data area can store data created during the use of the electronic device 100 (such as photos, etc.). In addition, the memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like.

The electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.

The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. Speaker 170A, also referred to as a "horn", is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music through speaker 170A, or listen to hands-free calls.

Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the receiver 170B can be placed close to the human ear to receive the voice.

The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. The earphone interface 170D is used for connecting wired earphones. The earphone interface may be a USB interface, or a 3.5mm open mobile terminal platform (open mobile terminal platform, OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

The pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, pressure sensor 180A may be disposed on display screen 194 .

The gyro sensor 180B can be used to determine the motion posture of the electronic device 100 .

The acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.

The distance sensor 180F is used to measure the distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 may use the distance sensor 180F for distance measurement to achieve fast focusing.

Proximity light sensor 180G may include, for example, light emitting diodes (LEDs) and light detectors, such as photodiodes. The light emitting diodes may be infrared light emitting diodes. The electronic device 100 emits infrared light through the light emitting diode. Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it may be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 . The electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to make a call, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, automatic unlock and lock screen in pocket mode.

The ambient light sensor 180L is used for sensing ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket, so as to prevent accidental touch.

Touch sensor 180K, also known as "touch panel". It can be set on the display screen 194 . Used to detect touches on or near it. The detected touch operation can be transmitted to the application processor to determine the type of touch event and provide corresponding visual output through the display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the electronic device 100 , which is different from the position of the display screen 194 .

The keys 190 include a power key, a volume key and the like. The keys may be mechanical keys. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 .

The motor 191 can generate a vibrating reminder. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects. The motor 191 may also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.

The SIM card interface 195 is used for connecting a subscriber identity module (subscriber identity module, SIM). The SIM card can be connected to and separated from the electronic device 100 by inserting it into the SIM card interface or pulling it out from the SIM card interface. The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support NanoSIM card, Micro SIM card, SIM card and so on. The same SIM card interface can insert multiple cards at the same time. The types of the multiple cards may be the same or different. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calling and data communication. In some embodiments, the electronic device 100 adopts an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .

The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the software structure of the electronic device 100 is exemplarily described by taking an Android system with a layered architecture as an example.

FIG. 3 is a block diagram of the software structure in the electronic device 100 according to the embodiment of the present invention.

The operating system is the Android system. The layered architecture divides the Android system into several layers, and each layer has a clear role and division of labor. Layers communicate through software interfaces. In some embodiments, the Android system is divided into four layers, which are, from top to bottom, the application program layer, the application program framework layer, the Android runtime (Android runtime) and the system library, and the kernel layer.

The application layer can consist of a series of application packages.

As shown in FIG. 3, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, and short message.

The application framework layer provides an application programming interface (application programming interface, API) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

As shown in Figure 3, the application framework layer can include window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

A window manager is used to manage window programs. The window manager can get the size of the display screen, determine whether there is a status bar, lock the screen, capture the screen, etc.

Content providers are used to store and retrieve data and make it accessible to applications. Said data may include video, images, audio, calls made and received, browsing history and bookmarks, phonebook, etc.

The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, and so on. The view system can be used to build applications. A display interface can consist of one or more views. For example, a display interface formed by superimposing labels and real scene images may include a view for displaying text and a view for displaying pictures.

The phone manager is used to provide communication functions of the electronic device 100 . For example, the management of call status (including connected, hung up, etc.).

The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, and so on.

The notification manager enables the application to display notification information in the status bar, which can be used to convey notification-type messages, and can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify the download completion, message reminder, etc. The notification manager can also be a notification that appears on the top status bar of the system in the form of a chart or scroll bar text, such as a notification of an application running in the background, or a notification that appears on the screen in the form of a dialog window. For example, prompting text information in the status bar, issuing a prompt sound, vibrating the electronic device, and flashing the indicator light, etc.

Android Runtime includes core library and virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

The core library consists of two parts: one part is the function function that the java language needs to call, and the other part is the core library of Android.

The application layer and the application framework layer run in virtual machines. The virtual machine executes the java files of the application program layer and the application program framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

A system library can include multiple function modules. For example: surface manager (surface manager), media library (Media Libraries), 3D graphics processing library (eg: OpenGL ES), 2D graphics engine (eg: SGL), etc.

The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.

The media library supports playback and recording of various commonly used audio and video formats, as well as still image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing, etc.

2D graphics engine is a drawing engine for 2D drawing.

The kernel layer is the layer between hardware and software. The kernel layer includes at least a display driver, a camera driver, an audio driver, and a sensor driver.

All of the following embodiments can be implemented in the electronic device 100 having the above hardware structure. In the following embodiments, the electronic device 100 is taken as an example to illustrate the information prompting method provided in the embodiments of the present application. The method mainly includes: the electronic device collects images of objects in the environment around the user to generate a real scene image, and then the electronic device further obtains the information of the points of interest in the electronic map to generate labels for each object in the real scene image, and finally Labels for individual objects are overlaid on real scene images. Because the information of the points of interest in the electronic map includes information such as the coordinates and names of the points of interest, and the location information of the electronic device can be obtained through positioning, the electronic device can determine the label of each object and the display of the label in the real scene image location, so that after the electronic device collects images of objects in the surrounding environment, it can only recognize typical buildings in the image, such as walls, ceilings, floors, building outlines, floors where shops are located, and floor intervals, etc. The names of objects in many acquired images are recognized, for example, there is no need to recognize the names of shops, so the calculation amount of electronic devices can be greatly reduced, and the power consumption of electronic devices can be saved.

Specifically, in combination with the structure of the electronic device described in FIG. 2, in the following embodiments of the present application, the specific process of the information prompt method is described in detail. Referring to FIG. 4, the specific process of the method may include:

Step 201, when the touch panel of the electronic device detects the user's operation of opening the AR real-world navigation mode of the navigation application, the touch panel detects the user's operation, and notifies the processor of the electronic device, the processor of the electronic device instructs the camera to turn on, and the camera Image acquisition is performed on the surrounding environment of the user, and at least one real scene image is generated to be displayed on the display screen.

Wherein, the real scene image may be an image directly captured by a camera, or may be an image that can represent the user's surrounding environment after performing real enhancement processing or other processing on the image captured by the camera, which is not limited in this embodiment of the present invention.

Step 202, the positioning module in the electronic device locates the position of the electronic device, and generates position information of the electronic device.

It can be understood that, the positioning module may specifically be a receiver of a positioning system such as a global positioning system (global positioning system, GPS) and a Beidou satellite navigation system.

It should be noted that this step and step 201 are not strictly performed sequentially, and step 201 can be performed first, and then step 202 can be performed, or step 202 can be performed first, and then step 201 can be performed, or the two steps can be performed simultaneously.

In step 203, the application processor of the electronic device obtains the information of the points of interest in the electronic map of the set area according to the position information of the electronic device, wherein the set area includes the position of the electronic device.

Step 204, the graphics processor of the electronic device recognizes the generated real scene image, determines the target object, and sends the identified features of the target object to the application processor, and the application processor combines the information of the point of interest with the target object Matching is performed, and the label of the target object is generated according to the matching result.

Step 205, the application processor of the electronic device determines the display position of the tag in the real scene image according to the position information of the electronic device and the coordinate information of the tag.

Step 206, the application processor of the electronic device superimposes the label and the real scene image according to the display position, and sends the superimposed graphics to the display screen, and the display screen displays the superimposed virtual reality image.

Specifically, there are two ways for an electronic device to obtain information about a point of interest in an electronic map of a set area. One way is: when the electronic device detects that the user has opened a navigation application, the electronic device can communicate with the navigation application. The application server of the application establishes a network connection. When the user enters the type or name of the point of interest, the electronic device can download the information of the point of interest of the electronic map of the set area from the application server to the electronic device; another way is: the electronic device It is also possible to download and install the information of the points of interest of the electronic map to the electronic device at the same time when downloading and installing the navigation application. In this way, when step 203 is performed, when the user inputs the type or name of the point of interest, the information can be directly retrieved from the database in the electronic device. Get information about a point of interest. Considering that the names of buildings and shops in the electronic map will change frequently, the former method is preferred in this application to obtain the information of points of interest. In addition, the setting area is generally determined based on empirical values. If the electronic device is outdoors, the setting area will be at most 200 meters, and if the electronic device is indoors, the setting area will be at most 30 meters.

In addition, it should be noted that the information of points of interest is a concept in the geographic information system, which generally includes information such as the name of the point of interest, the coordinates of the point of interest, the type of the point of interest, etc., and is usually collected in advance by the collector and stored in the data system . For example, the types of POIs may be restaurants, barbershops, and coffee shops, and the names of POIs corresponding to the type of restaurants may be KFC, McDonald's, Sichuan and Hunan restaurants, and the like. The coordinates of the point of interest refer to the latitude and longitude coordinates of the point of interest on the map, for example, the latitude and longitude coordinates of the KFC store on the map. In step 203, the information of the points of interest acquired by the electronic device may be obtained by filtering keywords input by the user, or may be obtained by filtering according to the type of points of interest set by the electronic device by default. For example, the electronic device is set by default that the POIs of the banking and dining types need to be displayed in the walking navigation, then the electronic device screens out the information of the POIs of the banking and dining types from the electronic map of the set area.

For the steps shown in Figure 4, the embodiment of the present application further describes the scenarios shown in Figure 5 and Figure 6. As shown in Figure 5, the electronic device detects that the user has acted on the operation of the application icon of Baidu Map, see Figure 5a. In response to this operation, the electronic device displays an electronic map interface, as shown in FIG. 5b. Further, referring to FIG. 6, when the electronic device detects that the user acts on the control of the route, in response to this operation, the electronic device displays the navigation route map, as shown in FIG. 6b. When the electronic device detects that the user acts on the AR In response to the operation of the control of the real-scene navigation line, the electronic device turns on the camera to collect images of the surrounding environment, and displays an interface diagram as shown in FIG. 6b.

Since the real scene image captured by the electronic device is an area in the set area, the electronic device recognizes the real scene image and determines that the target object is also an object in the set area. Therefore, the electronic device can determine the label of each object in the real scene image according to the position information of the electronic device and the coordinates in the information of the point of interest, wherein the label can include the name of the object and the linear distance between the object and the electronic device wait. For example, the real scene image captured by the electronic device is part of the scene in a certain university campus, the electronic device recognizes the real scene image, and the determined target objects are three buildings in the university campus. Further, the electronic device can match the corresponding interests of these buildings according to the position information of the electronic device, the spatial relative positional relationship between each object in the real scene image and the electronic device, and the coordinate information in the information of the acquired points of interest. The name of the point. For example, the names of the target objects are teaching building, library, and cafeteria respectively. Specifically, if the electronic device determines the spatial position relationship between the target object and the electronic device by recognizing the real scene image, one building is in front of the electronic device on the left, one building is directly opposite to the electronic device, and one building is in front of the electronic device on the right. According to the location information of the electronic device and the coordinate information in the information of the point of interest, the electronic device determines the relative positional relationship between the point of interest and the electronic device. The electronic equipment is directly opposite, and the point of interest of the cafeteria is in front of the electronic equipment on the right. After the two are matched, it can be determined that the building on the left front of the electronic device is a library, the building directly opposite is a teaching building, and the building on the right front is a cafeteria. Assume that the spatial position relationship between the target object and the electronic device is that a building is in front of the electronic device, and two buildings are directly opposite the electronic device. At this time, the orientation is different, and when the electronic device recognizes the object in the real scene image, it can recognize the electronic device. The distance between the two buildings directly opposite the device and the electronic device is different, so the electronic device can determine that the building on the left front of the electronic device is a library based on the location information of the electronic device and the coordinate information in the information of the point of interest. The building on the opposite side is the teaching building, and the building farther away is the canteen.

In some embodiments, the electronic device can further calculate the linear distance between the electronic device and the target object according to the position information of the electronic device and the coordinates of the point of interest corresponding to the target object. In this way, the final target generated by the electronic device The label of the object may include not only the name of the target object, but also distance information. For example, the label of the library generated by the electronic device may be: library, 50 meters away from the electronic device. Furthermore, the display shape of the label containing the target object may be a rectangular frame, a circular frame, etc., and the rectangular frame or the circular frame displays distance information and the name of the target object. This application does not limit the display shape of the label.

In some embodiments, after the electronic device recognizes the objects contained in the real scene image, it can further calculate the relative electron The azimuth of the device. In this way, the label of the target object finally generated by the electronic device may include not only the name of the target object, but also the azimuth angle. Among them, in the three-dimensional space, there are azimuth angles of the same dimensions as east, west, northwest, and up and down, while in the plane space, there are azimuth angles of the same dimensions as east, west, northwest.

As shown in Figure 7, for a flat space, the left and right viewing angles of the camera of the electronic device are generally about 90 degrees, and objects within the range of the left and right viewing angles may be in different orientations of the electronic device, such as left front, right front, front ahead. Therefore, the label of the library generated by the electronic device may be: library, in the southwest direction of the electronic device.

As shown in Figure 8, for a three-dimensional space, when it is defined that the electronic device is at the center of the sphere, the up and down viewing angles and left and right viewing angles are generally around 90 degrees. Objects of different levels may exist at different heights within the range of viewing angles. For example, in FIG. 8 , there are three levels of heights: the first floor, the second floor, and the first underground floor. Assuming that the electronic device is currently in the mall, the store label generated by the electronic device may be: KFC, on the south side of the second floor.

It is worth noting that, in the embodiment of the present application, the electronic device recognizes the object in the real scene image, mainly extracting the features of the object in the real scene image. For example, for the outdoor scene, the typical features extracted by the electronic device are Features such as walls, roads, sky, and building outlines; for indoor scenes, typical features extracted by electronic devices include ceilings, floors, roads, stairs, stair intervals, and shop walls. In the embodiment of the present application, it is not necessary to recognize the building names or shop names in the real scene image. For example, the electronic device only needs to recognize the building The name of the building on the first building, and the name of the building on the building that does not need to identify the teaching building, etc. The reason why electronic devices do not use image recognition to determine information such as building names or shop names is because if text recognition is performed on real scene images on the side, electronic devices need to perform a large number of calculations, which consumes more computing resources. Assuming that the text recognition of the real scene image is performed on the cloud server side, electronic devices are also required to upload the real scene image to the cloud server, which not only consumes data traffic, but also causes network delay due to the interactive transmission between the two, resulting in poor user experience.

That is to say, the electronic device only recognizes typical features in the real scene image, and combines the information of the points of interest obtained from the application server to generate labels for each object in the real scene image. Further, the electronic device then determines the display position of the label of each object in the real scene image.

In some embodiments, the acquired real scene image is a two-dimensional plane image, so it corresponds to a plane coordinate system, and the unit may be meters, while the electronic map is an ellipsoidal coordinate system, and the unit is latitude and longitude. When the electronic device calculates the display position of the label, it needs to convert the longitude and latitude coordinates of the information of the point of interest into the plane coordinates of the plane coordinate system. Among them, in the prior art, there are many tools for converting latitude-longitude coordinates and plane coordinates, for example, the arcgis tool can be used to realize the mutual conversion between latitude-longitude coordinates and plane coordinates.

It should be noted that the basic principle for the electronic device to convert the latitude and longitude coordinates and the plane coordinates is: the electronic device uses a certain coordinate on the display screen of the electronic device as the origin of the projection screen according to the position information of the electronic device and the coordinate information in the label. Combined with the orientation angle and tilt angle of the electronic device, as well as the width and height of the display screen of the electronic device, the viewing angle of the camera, and the relative offset of the camera, etc., the display position of the label of the target object on the electronic device is calculated according to the projection formula The coordinates of the center point of , that is, the abscissa of the center point and the ordinate of the center point.

Generally, as shown in Figure 8b, one definition of the coordinate system of the display screen of an electronic device is: when the screen is vertical, the upper left corner is the origin, and the coordinates are (0, 0). The lower coordinate is a positive value, perpendicular to the display screen, and the shooting direction of the camera is positive. Therefore, when the display screen is turned to landscape, it can be regarded as the X axis in the horizontal direction, the Y axis in the up and down direction, and the Z axis perpendicular to the XY plane. The abscissa of the screen projection origin is the coordinate of the X-axis center point of the display screen, and the vertical coordinate is the lower coordinate of the Y-axis of the display screen. The coordinate information of the screen projection origin is predefined known data.

Combined with Figure 8c, the calculation of the abscissa of the center point: when the vertical screen is known, the X coordinates of the upper left corner and the upper right corner of the display screen of the electronic device are known, and the viewing angle α of the camera is known, assuming it is 60 degrees, and the target object The angle β formed by P', the focal point O, and _P1 in the upper left corner of the camera's viewing range is the direction angle between the target object and the electronic device. The electronic device can calculate the target object P' on the display screen based on the above parameters. For the abscissa of the upper projection point p″, see Formula 1.

Wherein, p″ _X is the abscissa of the screen projection point p″ of the target object P’ on the display screen, β is the direction angle, α is the viewing angle of the camera, C _1x is the X coordinate of the upper left corner of the display screen of the electronic device, C _2x is the X coordinate of the upper right corner of the display screen of the electronic device.

Calculate the vertical coordinate of the center point: when the vertical screen is known, the coordinates of the upper left corner of the display screen of the electronic device and the vertical coordinate y0 of the origin of the projection screen can be calculated according to the coordinates of the label and the position information of the electronic device. The distance D between the devices, and the farthest distance Dmax that the electronic device can shoot is known, so the ordinate of the projection point p″ of the target object on the display screen can be calculated, see formula 2.

Among them, p" _y is the ordinate of the projected point p" of the target object P' on the display screen, D is the distance between the target object and the mobile terminal, D _max is the farthest shooting distance of the mobile terminal, and y is the electronic device The y coordinate of the upper left corner of the display screen, y ₀ is the ordinate of the screen projection origin of the electronic device.

In addition, the electronic device may also be tilted forward or backward when shooting, so the tilt angle of the display screen may also be considered when calculating the vertical coordinate.

Optionally, the electronic device can also reversely estimate the distance between the origin of the projection screen and the current position of the electronic device based on the ordinate of the screen projection origin, and then adjust the coordinates in the label of the target object to correspond to the position of the electronic device according to this distance The distance between the coordinates to get the ordinate of the optimized center point.

Considering that the display position of the label of the target object calculated by the electronic device according to the above method may still be unreasonable, for example, as shown in Figure 9, in the prior art, the label of company A in the real scene image captured by the mobile phone Displayed in the sky, the display position of the label has a large offset from the target object identified by the label, which is inconvenient for users to find the target object. In order to improve this problem, the present application further adjusts the coordinates of the central point calculated according to the above calculation method in combination with the recognition result of the real scene image. Specifically, for the real scene image captured by the electronic device in the outdoor scene, the electronic device judges whether the coordinates of the center point are in the area where these typical features are located by identifying the typical features such as the outdoor road, the ground, and the sky. , then adjust the center coordinates based on the area where the building of company A is located, and adjust the label of the target object to a relatively more reasonable height position. For example, as shown in Figure 10, adjust the label of company A to the building of company A exterior wall.

For another example, for the real scene image captured by the electronic device in the indoor scene, the electronic device obtains the position of the indoor vertical surface (such as a wall, shelf, etc.) by identifying typical features such as indoor roads, ceilings, walls, and floor intervals, and then Determine whether the coordinates of the center point are in the indoor vertical plane, if yes, do not adjust the coordinates of the center point; if not, adjust the coordinates of the center point, that is, put the coordinates of the center point in the vertical plane. Generally, the electronic device will adjust the coordinates of the center point in combination with preset rules, usually to the upper middle position of the wall of the target object. For example, the store name is generally displayed on 2/3 or In the 3/4 position, the product is generally displayed on the shelf, so the electronic device first estimates the display height of the display position of the label of the target object on the vertical plane. Then, according to the estimated display height, combined with the maximum limit of the label of the target object, the label of the target object is adjusted to a relatively more reasonable height position on the vertical plane.

Through the above methods, it can be seen that the embodiment of the present application uses a coarse-grained scene recognition, combined with the information of the points of interest obtained from the electronic map, to accurately display the label of the target object in the real scene image. Real scene navigation function reduces the calculation amount of electronic equipment at the same time.

In the embodiment of the present application, the electronic device can be positioned to obtain the location information of the electronic device, so the electronic device can further determine whether the electronic device is currently in an indoor scene or an outdoor scene according to the location information of the electronic device. In addition to extracting different features of objects in the real scene image according to different scenes, the electronic device can also perform corresponding features on the features of the object in the current real scene image in real time when the electronic device switches from an outdoor scene to an indoor scene. extraction, or when the electronic device switches from an indoor scene to an outdoor scene, the features of the object in the current real scene image are extracted in real time.

For example, when the electronic device locates the location where the electronic device is located as an outdoor scene, the typical features currently extracted by the electronic device from the real scene image include features such as walls, roads, sky, and building outlines; when the electronic device switches to an indoor In the scene, the typical features currently extracted by electronic devices from real scene images include ceilings, floors, roads, stairs, stair intervals, shop walls and other features.

For example, when the electronic device is currently performing pedestrian navigation outdoors, the electronic device can determine that the current location of the electronic device is an outdoor scene through information such as GPS and WIFI hotspots. When the electronic device performs image recognition on the real scene image collected by the camera, the extracted Typical features include features such as walls, roads, sky, and building outlines. As shown in Figure 10, objects recognized by mobile electronic devices include architectural information such as buildings, grass, roads, and squares. Obtain the information of the points of interest in the electronic map, so as to generate a label of company A building and fashion design plaza, and display the label at the corresponding position of the image. It can be seen that when the electronic device is in an outdoor scene, the labels in the image are relatively loose.

Further, when the electronic device determines that the electronic device enters the room from outdoors according to the positioning result of the electronic device, the electronic device automatically switches the label display style from the outdoor label display style to the indoor label display style; or When the electronic device determines that the electronic device enters the outdoor from the indoor according to the positioning result of the electronic device, the electronic device automatically switches the display style of the label from the indoor label display style to the outdoor label display style. Among them, the outdoor label display style includes any one or any combination of the following: the content of the label includes the name of the target object and the distance between the target object and the electronic device, and the labels are loosely arranged in the real scene image and arranged The way is irregular; the indoor label display style includes any one or any combination of the following: the content of the label includes the name of the target object, the distance between the target object and the electronic device, and the floor where the target object is located. The real scene images are arranged compactly and arranged according to the floor level where the target objects are located.

For example, when an electronic device enters a shopping mall from an outdoor pedestrian navigation, typical features currently extracted by the electronic device from real scene images include features such as ceilings, floors, roads, stairs, stair intervals, and shop walls. As shown in FIG. 11 , the objects recognized by the electronic device have features such as ceilings, floors, roads, stairs, floor intervals, and walls of shops. Then the electronic device obtains the information of each store in the electronic map according to the current position of the electronic device, thereby generating a label of each store, wherein the label includes not only the store name and distance, but also floor information. For example, if the label is shop A, F1 --- 10 meters, it means that the name of the shop is shop A, on the first floor, 10 meters away from the user. The electronic device displays the label on the wall of the store in the form of a rectangular frame. It can be seen that when the electronic device is in the indoor scene, compared with the outdoor scene, the number of tags will increase, and the tags will be arranged according to the floor level, and the tags will contain the floor information.

In addition, when the electronic device is in motion, or in different time periods, the real scene image captured by the mobile phone will have the problem of unclear label display due to changes in the background environment of the image. For example, in Fig. 12, Fig. 12a shows the superimposed display effect of the label and the real scene image under the condition of insufficient light in the afternoon in the prior art. It can be seen from Fig. 12a that the label of log cabin hotel is not easy for users to distinguish. For this reason, the information prompt method provided by the embodiment of the present application can also adjust the display parameters of the generated label according to the information collected by the ambient light sensor, such as adjusting the transparency of the font in the label, the color of the font, and the transparency of the label background, and finally make the label What is displayed is clearer. For example, as shown in Figure 12b, when the mobile electronic device captures the real scene image of the log cabin, the ambient light sensor detects that the current light is insufficient, so the electronic device reduces the transparency of the background of the log cabin hotel label, so that the electronic device can be more transparent. Get tab content clearly.

In addition, when the label is not clear after being superimposed on the real scene image, the electronic device can also adjust the background color in the real scene image, so that the electronic device can more clearly obtain the content of the label.

It is worth noting that the information prompt method provided by the embodiment of the present application is not only applicable to walking navigation scenarios, but also applicable to scenarios such as cycling or driving. In addition, when navigating, while the electronic device is displaying the virtual reality image superimposed by the label and the real scene image, the mobile phone can broadcast the object labels in various directions such as the left, right, left front, and right front of the electronic device in real time, for example, outdoor During the navigation process, the voice prompt information output by the mobile phone can be: a kindergarten is 500 meters to the left, a fountain square is 100 meters to the right, and a school supplies store is 50 meters in front. Or, during the indoor navigation process, the voice prompt information output by the mobile phone can be: there is women's clothing Ai Lian 10 meters ahead on the left side of the first floor, and there is a women's clothing only store 20 meters ahead on the right side of the first floor. Wherein, the speech language output by the mobile phone may also be English, Mandarin, Cantonese, etc., that is, the mobile phone broadcasts according to the language set by the electronic device.

It can be seen that in the embodiment of the present application, the electronic device performs text recognition on the objects in the real scene image collected by the electronic device by combining the information of the points of interest in the electronic map, and generates a label. In addition, the electronic device The coordinate information of the electronic device and the position information of the electronic device can calculate the display position of the label of each object in the image, and then superimpose the label with the real scene image, which can reduce the calculation workload of the electronic device and save power consumption. , the label content can be changed according to the indoor and outdoor scenes, and the background transparency of the label and the font of the label can also be adjusted according to the light and darkness of the real scene, which improves the clarity of the label in the virtual reality image generated by the superimposed display.

An embodiment of the present application also provides a computer-readable storage medium, the computer-readable storage medium includes a computer program, and when the computer program is run on an electronic device, the electronic device is made to execute any possible communication method described above. accomplish.

An embodiment of the present application further provides a computer program product, which, when the computer program product is run on an electronic device, causes the electronic device to execute any possible implementation of the above-mentioned communication method.

In some embodiments of the present application, the embodiment of the present application discloses a device, as shown in FIG. 13 , which is used to implement the methods described in the above method embodiments, which includes: a processing module 1101 and a display module 1102 . The processing module 1101 is used to support the electronic device to execute steps 201 to 205 in FIG. 4 . The processing module 1101 is also used to support the electronic device to execute the electronic device in FIG. It is used to support the electronic device to execute step 206 in FIG. 4 . The display module 1102 in FIG. 13 may be implemented by a display screen 194 , and the processing module 1101 may be implemented by a processor 110 . The device can be implemented in the application framework layer of the Android operating system, and the application framework layer can provide an interface for map applications to call. All relevant content of the steps involved in the above method embodiments can be referred to the function descriptions of the corresponding functional modules, and will not be repeated here.

Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated according to needs It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. For the specific working process of the above-described system, device, and unit, reference may be made to the corresponding process in the foregoing method embodiments, and details are not repeated here.

Each functional unit in each embodiment of the embodiment of the present application may be integrated into one processing unit, or each unit may physically exist separately, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage The medium includes several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: flash memory, hard disk, read-only memory, random access memory, magnetic disk or optical disk, and other media capable of storing program codes.

The above is only the specific implementation of the embodiment of the application, but the protection scope of the embodiment of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the application shall be covered by this application. Within the protection scope of the application embodiment. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.

Claims (17)

1. A method for prompting information, characterized in that it comprises: When the electronic device operates the real-scene navigation function, the electronic device collects images of the user's surrounding environment to generate a real scene image, and obtains settings from the electronic map according to the shooting direction of the electronic device and the position information of the electronic device Information about points of interest in the area, the set area includes location information of the electronic device, and the information about points of interest includes the name of the point of interest and the coordinates of the point of interest; The electronic device recognizes the real scene image to determine a target object; The electronic device determines a first relative positional relationship between the target object in the real scene image and the electronic device, and determines the relationship between the interest point and the electronic device according to the coordinates of the interest point and the position information of the electronic device. The second relative positional relationship of the user, according to the first relative positional relationship and the second relative positional relationship, determine the target point of interest matching the target object, and generate the target point of interest according to the information of the target point of interest a label for the object, the label including the name of the target object; The electronic device determines the display position of the label in the real scene image according to the information of the point of interest and the location information of the electronic device; The electronic device displays an interface including the label and the real scene image, wherein the label is superimposed and displayed at the display position of the real scene image. 2. The method according to claim 1, wherein the electronic device determines the display position of the label in the real scene image according to the information of the point of interest and the location information of the electronic device, include: The electronic device determines the coordinates of the target object according to the coordinates of the target point of interest; The electronic device determines the center coordinates of the label of the target object in the real scene image according to the coordinates of the target object and the position information of the electronic device; The electronic device determines the display position of the tag in the real scene image according to the center coordinates. 3. The method according to claim 1 or 2, wherein the display position is at the middle upper position of the wall of the target object, or on a shelf, and the display position is not at the center of the real scene image. In the display area of any of the following features: sky, ceiling, road, ground, stairs, stairwells. 4. The method according to claim 1 or 2, wherein the electronic device identifies the real scene image and determines the target object, including: The electronic device determines the environment in which the electronic device is located according to the location information of the electronic device; The electronic device recognizes typical features in the environment from the real scene image to determine the target object. 5. The method according to claim 1, wherein the electronic device identifies the real scene image and determines the target object, comprising: When the electronic device determines that the location information of the electronic device belongs to outdoors, the features recognized by the electronic device from the real scene image include any one or any combination of the following: walls, roads, sky, building outlines, and according to The identified features determine the target object; When the electronic device determines that the location information of the electronic device is indoors, the features recognized by the electronic device from the real scene image include any one or any combination of the following: ceiling, floor, road, stairs, stair interval, store The wall, and according to the identified features, determine the target object. 6. The method according to any one of claims 1 to 5, further comprising: When the electronic device determines that the brightness of the surrounding environment of the electronic device has changed according to the information collected by the ambient light sensor of the electronic device, the electronic device determines the brightness of the tag according to the current information collected by the ambient light sensor. The display parameters are adjusted, and the display parameters include any one or any combination of the following: font color, background transparency. 7. The method according to any one of claims 1 to 5, further comprising: When the electronic device determines that the background brightness of the real scene image changes, the electronic device adjusts the display parameters of the label according to the current background brightness of the real scene image, and the display parameters include the following Any one or any combination: font color, background transparency. 8. The method according to any one of claims 1 to 5, further comprising: The electronic device determines that when the electronic device enters the room from outdoors according to the positioning result of the electronic device, the electronic device automatically switches the label display style from the outdoor label display style to the indoor label display style; Or, when the electronic device determines that the electronic device enters the outdoor from the indoor according to the positioning result of the electronic device, the electronic device automatically switches the display style of the label from the indoor label display style to the outdoor label display style; Wherein, the outdoor label display style includes any one or any combination of the following: the content of the label includes the name of the target object and the distance between the target object and the electronic device, and the labels are loosely arranged in the real scene image, And the arrangement is irregular; The indoor label display style includes any one or any combination of the following: the content of the label includes the name of the target object, the distance between the target object and the electronic device, and the floor where the target object is located. The real scene images are arranged compactly and arranged according to the floor level where the target objects are located. 9. An electronic device, comprising a processor, a memory, a display and a camera; said memory is used to store one or more computer programs; When the processor runs the real scene navigation function, the camera collects images of the surrounding environment of the user to generate a real scene image; according to the shooting direction of the electronic device and the position information of the electronic device, the device is obtained from the electronic map The information of the point of interest in the specified area, the set area includes the location information of the electronic device, and the information of the point of interest includes the name of the point of interest and the coordinates of the point of interest; The processor obtains the information of the point of interest from the electronic map, and recognizes the real scene image to determine the target object; The processor determines the first relative positional relationship between the target object in the real scene image and the electronic device, and determines the relationship between the interest point and the electronic device according to the coordinates of the interest point and the position information of the electronic device. The second relative positional relationship of the user, according to the first relative positional relationship and the second relative positional relationship, determine a target point of interest matching the target object, and generate the target point of interest according to the information of the target point of interest The label of the object, according to the information of the point of interest and the position information of the electronic device, determine the display position of the label in the real scene image; wherein the label includes the name of the target object; The display displays an interface including the label and the real scene image, wherein the label is superimposed and displayed at the display position of the real scene image. 10. The electronic device according to claim 9, wherein the processor is further used for: determining the coordinates of the target object according to the coordinates of the target point of interest; determining the center coordinates of the label of the target object in the real scene image according to the coordinates of the target object and the position information of the electronic device; Determine the display position of the label in the real scene image according to the center coordinates. 11. The electronic device according to claim 9 or 10, wherein the display position is at the middle upper position of the wall of the target object, or on a shelf, and the display position is not in the real scene image In the display area of any of the following features: Sky, Ceiling, Road, Floor, Stair, Stair Space. 12. The electronic device according to claim 9, wherein the processor is specifically used for: determining the environment in which the electronic device is located according to the location information of the electronic device; The typical features in the environment are identified from the real scene image to determine the target object. 13. The electronic device according to claim 9, wherein the processor is specifically used for: When it is determined that the location information of the electronic device belongs to outdoors, the features identified from the real scene image include any one or any combination of the following: walls, roads, sky, building outlines, and the target object is determined according to the identified features ; When it is determined that the location information of the electronic device is indoors, the features identified from the real scene image include any one or any combination of the following: ceiling, floor, road, stairs, stair space, shop wall, and according to the identified features, Determine the target object. 14. The electronic device according to any one of claims 9 to 13, wherein the electronic device further comprises an ambient light sensor, and the processor is further configured to: When the processor determines that the brightness of the surrounding environment of the electronic device changes according to the information collected by the ambient light sensor, adjust the display parameters of the label according to the current information collected by the ambient light sensor, and the The display parameters include any one or any combination of the following: font color, transparency of the background. 15. The electronic device according to any one of claims 9 to 13, wherein the processor is further configured to: When it is determined that the background brightness of the real scene image changes, the display parameters of the label are adjusted according to the current background brightness of the real scene image, and the display parameters include any one or any combination of the following: font color , the transparency of the background. 16. The electronic device according to any one of claims 9 to 13, wherein the processor is further configured to: According to the positioning result of the electronic device, when it is determined that the electronic device enters the room from outdoors, the display style of the label is automatically switched from the outdoor label display style to the indoor label display style; Or, according to the positioning result of the electronic device, when it is determined that the electronic device enters the outdoor from the indoor, the display style of the label is automatically switched from the indoor label display style to the outdoor label display style; Wherein, the outdoor label display style includes any one or any combination of the following: the content of the label includes the name of the target object and the distance between the target object and the electronic device, and the labels are loosely arranged in the real scene image, And the arrangement is irregular; The indoor label display style includes any one or any combination of the following: the content of the label includes the name of the target object, the distance between the target object and the electronic device, and the floor where the target object is located. The real scene images are arranged compactly and arranged according to the floor level where the target objects are located. 17. A computer storage medium, characterized in that the computer-readable storage medium comprises a computer program, and when the computer program is run on an electronic device, the electronic device is made to execute the computer program according to any one of claims 1 to 8. method.

Images