WO2024174732A1 - Display device and speech control method - Google Patents

Abstract

A display device and a speech control method. The display device comprises a monitor (260), a sound collector and a controller (250), which is respectively connected to the monitor (260) and the sound collector, wherein the sound collector is configured to collect a speech control instruction of a user; and the controller (250) is configured to: determine at least one scrolling control which is comprised in the current display interface; construct a speech scrolling control word list of the scrolling control in the current display interface, wherein the speech scrolling control word list is used for representing the correlation between a scrolling direction of the scrolling control and a semantic control word; and in response to the speech control instruction of the user, on the basis of the speech scrolling control word list and speech control text corresponding to the speech control instruction, control a target scrolling control in the current display interface to execute a scrolling operation.

Description

Display device and voice control method

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Chinese patent application No. 202310155184.8, filed on February 22, 2023, the entire contents of which are incorporated by reference into this application.

Technical Field

The present application relates to the field of voice control technology, and in particular to a display device and a voice control method.

Background Art

With the development of voice control technology, voice functions are also integrated into various display devices (such as smart TVs), and users can jump to/open specified applications through voice commands.

Typically, the display interface of a display device maintains multiple controls that can respond to voice control commands, and the controls can display corresponding interface description words to the outside. After receiving the user's voice control command, the display device matches the voice control text corresponding to the voice control command with the interface description words of each control in the current display interface one by one. If the corresponding interface description word is matched, the control corresponding to the interface description word is controlled to perform related operations. For example, if a resource library control is displayed in the current display interface, then in response to the voice control command of "open resource library", the display device can automatically select the resource library control, and then open the resource library to display a variety of multimedia resources for users to choose from.

However, some display interfaces may contain some controls without interface description words, and therefore, it is not possible to perform voice control on them by using the above interface description word matching method.

Summary of the invention

The present application provides a display device and a voice control method, which can effectively control controls without interface description words in a display interface.

In a first aspect, the present application provides a display device, including a display, a sound collector, and a controller connected to the display and the sound collector respectively, wherein:

A display configured to display an image screen and a user interface;

A sound collector is configured to collect a user's voice control command;

The controller is configured as:

Determine at least one scroll control included in the current display interface;

Constructing a voice scroll control word list of a scroll control in the current display interface; the voice scroll control word list is used to indicate the correspondence between the scroll direction of the scroll control and the semantic control word;

In response to the user's voice control instruction, based on the voice scroll control word list and the voice control text corresponding to the voice control instruction, the target scroll control is controlled to perform a scroll operation in the scroll direction indicated by the voice control text; the target scroll control is a scroll control corresponding to the voice control text in the scroll controls contained in the current display interface.

In a second aspect, the present application provides a voice control method, comprising:

Determine at least one scroll control included in the current display interface;

Constructing a voice scroll control word list of a scroll control in the current display interface; the voice scroll control word list is used to indicate the correspondence between the scroll direction of the scroll control and the semantic control word;

In response to the user's voice control instruction, based on the voice scroll control word list and the voice control text corresponding to the voice control instruction, the target scroll control is controlled to perform a scroll operation in the scroll direction indicated by the voice control text; the target scroll control is a scroll control corresponding to the voice control text in the scroll controls contained in the current display interface.

In a third aspect, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a controller in a display device, it can implement some or all of the steps of the voice control method provided in the present application.

In a fourth aspect, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a controller in a display device, it can implement some or all of the steps of the voice control method provided in the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the implementation methods of some embodiments of the present application or related technologies, the following is a brief introduction to the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

FIG1 is an operation scenario between a display device and a control device shown in some embodiments of the present application;

FIG2 is a hardware configuration block diagram of a display device shown in some embodiments of the present application;

FIG3 is a software configuration block diagram of a display device shown in some embodiments of the present application;

FIG4 is a schematic diagram of control distribution of a display interface shown in some embodiments of the present application;

FIG5 is a flow chart of a method for executing voice control on a display device according to some embodiments of the present application;

FIG6 is a schematic diagram of horizontal scrolling of a scroll control shown in some embodiments of the present application;

FIG. 7 is a schematic diagram of vertical scrolling of a scroll control shown in some embodiments of the present application;

FIG8 is a schematic diagram of waterfall scrolling of a scroll control shown in some embodiments of the present application;

FIG9 is a schematic diagram of a flow chart of a voice-controlled scrolling control performing a scrolling operation according to some embodiments of the present application;

FIG10 is a schematic diagram of a distribution of multiple scrolling controls on a display interface shown in some embodiments of the present application;

FIG. 11 is a schematic diagram of the voice control logic of a scrolling control in a display interface shown in some embodiments of the present application.

DETAILED DESCRIPTION

In order to make the purpose and implementation method of the present application clearer, the exemplary implementation method of the present application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of the present application. Obviously, the described exemplary embodiments are only partial embodiments of the technical solution of the present application, rather than all embodiments.

It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and common meanings.

The terms "including" and "having" and related variations mentioned in the specification of this application and the above claims are intended to cover but not exclude inclusion. For example, a product/device containing a series of components is not necessarily limited to all the components clearly listed, but may include other components not clearly listed or inherent to these products/devices.

In the implementation mode of the present application, the display device can have various implementation forms, for example, it can be a television, a smart TV, a laser projection device, a monitor, an electronic whiteboard (electronic bulletin board), an electronic desktop (electronic table), it can also be a personal computer, a laptop computer, a smart phone, a tablet computer, a portable wearable device, etc.

Among them, the above-mentioned display device can be controlled by a control device, an intelligent control device, voice, action, gesture, trigger action, etc.

1 , in an exemplary control operation scenario between a display device and a control apparatus, a user may operate a display device 200 through a smart device 300 or a control apparatus 100 .

In some embodiments, the control device 100 may be a remote controller, and the communication between the remote controller and the display device includes infrared protocol communication or Bluetooth protocol communication, and other short-range communication methods, and the display device 200 is controlled wirelessly or wired. The user can input user commands through buttons on the remote controller, voice input, control panel input, etc. to control the display device 200.

In some embodiments, a smart device 300 (such as a mobile terminal, a tablet computer, a computer, a laptop computer, etc.) may also be used to control the display device 200. For example, the display device 200 is controlled using an application running on the smart device.

In some embodiments, the display device 200 may not use the smart device 300 or the control apparatus 100 to receive instructions, but may receive user control through touch or gestures.

In some embodiments, the display device 200 can also be controlled in a manner other than the control apparatus 100 and the smart device 300. For example, the user's voice command control can be directly received through a module for acquiring voice commands configured inside the display device 200; or the user's voice command control can be received through a voice control device provided outside the display device 200.

In some embodiments, the display device 200 also communicates data with the server 400, allowing the display device 200 to communicate through a local area network (LAN), a wireless local area network (WLAN), or other networks.

Among them, the server 400 can provide various contents and interactions to the display device 200.

As an example, server 400 can be a stand-alone server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

2 , the display device 200 may include at least one of a tuner 210 , a communicator 220 , a detector 230 , an external device interface 240 , a controller 250 , a display 260 , an audio output interface 270 , a memory 280 , a power supply, and a user interface.

In some embodiments, the tuner demodulator 210 receives broadcast television signals via wired or wireless reception, and demodulates audio and video signals from multiple wireless or wired broadcast television signals, such as Electronic Program Guide (EPG) data signals.

In some embodiments, the tuner-demodulator 210 and the controller 250 may be located in different separate devices, that is, the tuner-demodulator 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

In some embodiments, the communicator 220 is a component for communicating with an external device or server 400 according to various communication protocol types. For example, the communicator may include at least one of a Wifi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or a near field communication protocol chip, and an infrared receiver.

In some embodiments, the display device 200 can communicate with the external control device 100 or the server 400 through the communicator 220. Establish the sending and receiving of control signals and data signals.

For example, the display device can communicate with the cloud server through the communicator, send data information that needs to be processed to the cloud server, and obtain the data information processed by the cloud server.

In some embodiments, the detector 230 is used to collect signals of the external environment or the interaction between the display device 200 and the outside. For example, the detector 230 includes an image collector, such as a camera, which can be used to collect external environment scenes, user attributes or user interaction gestures; or the detector 230 includes a sound collector, such as a microphone, etc., which is used to collect external sounds.

In the interactive scenario of the voice-controlled display device shown in the present application, the sound collector can be used to collect the user's voice control instructions.

In some embodiments, the external device interface 240 may include, but is not limited to, any one or more of the following interfaces: a high definition multimedia interface (HDMI), an analog or digital high definition component input interface (component), a composite video input interface (CVBS), a universal serial bus (USB) input interface, an RGB port, etc. It may also be a composite input/output interface formed by the above multiple interfaces.

In some embodiments, the controller 250 includes: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (Random Access Memory, RAM), ROM (Read-Only Memory, ROM), a first interface to an nth interface for input/output, a communication bus (Bus), etc.

The controller 250 can control the operation of the display device 200 and respond to the user's operation through various software control programs stored in the memory 280. The controller 250 controls the overall operation of the display device 200. For example, in response to receiving a user command for selecting a UI object to be displayed on the display 260, the controller 250 can perform an operation related to the object selected by the user command.

In the interactive scenario of the voice-controlled display device shown in the present application, the controller is configured to determine at least one scroll control contained in the current display interface and construct a voice scroll control word list for the scroll control in the current display interface; then, after the sound collector receives the user's voice control instruction, based on the voice scroll control word list and the voice control text corresponding to the voice control instruction, in the scroll controls contained in the current display interface, the target scroll control corresponding to the voice control text is controlled to perform a scroll operation in the scroll direction indicated by the voice control text.

In some embodiments, the display 260 includes a display screen component for presenting images and a driving component for driving image display. The display 260 is used to receive the image signal output by the controller 250, and display the video content, image content, menu control interface components and user control UI interface.

As an example, the display 260 may be a liquid crystal display, an OLED display, and a projection display, and may also be a projection device and a projection screen.

The user may input a user command in a graphical user interface (GUI) displayed on the display 260, and the user interface receives the user input command through the GUI. Alternatively, the user may input a user command by inputting a specific sound or gesture, and the user interface receives the user input command by recognizing the sound or gesture through a detector.

In the interactive scenario of the voice-controlled display device shown in the present application, the display 260 is configured to display image screens and a user interface.

Among them, "user interface" is the medium interface for interaction and information exchange between application programs or operating systems and users. It realizes the conversion between the internal form of information and the form acceptable to users.

As an example, the user interface may be an interface element such as an icon, a window, a control, etc. displayed on the display of the display device 200 .

Among them, controls may include icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets and other visual interface elements.

Referring to FIG. 3 , taking the deployment of the Android software platform in the display device 200 as an example, the display device 200 divides the Android operating system into four layers, namely, from top to bottom, applications (Applications, referred to as "application layer"), application framework layer (Application Framework) layer (referred to as "framework layer"), Android runtime (Android runtime) and system library layer (referred to as "system runtime library layer"), and kernel layer.

In some embodiments, at least one application is running in the application layer. These applications may be window programs, system settings programs, clock programs, etc. that come with the Android operating system; or they may be applications developed by third-party developers.

It should be understood that the application programs included in the application layer are not limited to the above examples, but may also include other application programs.

The framework layer provides application programming interface (API) and programming framework for applications. The application framework layer includes some predefined functions.

In other words, the framework layer is equivalent to a processing center, which determines the actions of applications in the application layer. Applications can access resources in the Android system and obtain system services during execution through API interfaces.

In some embodiments, the framework layer includes managers, content providers, view component systems, etc.

Among them, the manager includes at least one of the following modules: an activity manager (Activity Manager) is used to interact with all activities running in the system; a location manager (Location Manager) is used to provide system services or applications with access to system location services; a package manager (Package Manager) is used to retrieve various information related to the application package currently installed on the device; a notification manager (Notification Manager) is used to control the display and clearing of notification messages; a window manager (Window Manager) is used to manage icons, windows, toolbars, wallpapers and desktop widgets on the user interface.

In some embodiments, the activity manager is used to manage the life cycle of each application and the usual navigation back function, such as controlling the exit, opening, and back of the application; the window manager is used to manage all window programs, such as obtaining the size of the display screen, determining whether there is a status bar, locking the screen, capturing the screen, and controlling display window changes (for example, reducing the display window, shaking the display, distorting the display, etc.).

In some embodiments, the system runtime layer provides support for the upper layer (ie, the framework layer). When the framework layer is used, the Android operating system will run the C/C++ library contained in the system runtime layer to implement the functions to be implemented by the framework layer.

In some embodiments, the kernel layer is a layer between hardware and software. The kernel layer includes at least one of the following drivers: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB driver, HDMI driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

After introducing the control interaction scenario, hardware structure and deployed software platform of the display device, next, based on the above-mentioned display device 200, the specific implementation process of the voice control method of the display device is described in detail.

With the development of intelligent information technology, the wider the voice control scenarios and coverage, the greater the market share. Voice functions are also integrated into various display devices. Users can view various multimedia resources and jump to/open specified applications through voice control commands.

Between the applications in the display device, voice control can be achieved through direct words to Direct jump/switching between applications. That is, when the voice control text corresponding to the voice control instruction input by the user is a direct word of a certain application, the display device responds to the voice control instruction and jumps from the current application to the target application indicated by the voice control text corresponding to the voice control instruction.

As an example, when a user needs to exit the video application currently being browsed to the home page, he or she can input the voice control command "return to the home page" to instruct the display device to switch from the currently running video application to the home page, making it easier for the user to select other applications/multimedia resources of interest on the home page.

It should be understood that the homepage is a user interface for displaying and directly accessing various application interfaces. Users can browse recommended multimedia resources and applications on the homepage to select multimedia resources of interest to play, or open applications of interest to play/browse related content in the application.

In some embodiments, for a single application, since there are various scenarios within the application, voice control cannot be implemented between the scenarios through the above direct words, and control can only be implemented through interface description words in different display interfaces. That is, when the voice control text corresponding to the voice control instruction input by the user is the interface description word of a control in the current display interface, the display device responds to the voice control instruction and opens/selects the target control indicated by the voice control text corresponding to the voice control instruction in the current display interface.

Among them, since the scenes within the application are relatively fixed, a voice interface control word list corresponding to each display interface in the application can be constructed during the first loading process after each application update. After receiving the user's voice control command, the target control that requires the user's request to operate can be quickly determined in the current display interface based on the voice interface control word list.

As an example, after the display device receives the user's voice control command, it matches the voice control text corresponding to the voice control command with the interface description words of each control in the voice interface control word list one by one. If the corresponding interface description word is matched, the target control corresponding to the interface description word is opened/selected in the current display interface.

In addition, in some display interfaces, in addition to the above-mentioned controls carrying interface description words, the display interface also includes some controls without interface description words, such as scroll controls.

It should be noted that the display interface containing scroll controls in the display device may include at least one of a home page, a web page, and a user interface within an application, and some embodiments of the present application do not impose any restrictions on this.

Among them, the scroll control is used to implement sliding and page turning operations. Since the scroll control has no interface description words and the scrolling direction is not unique, it is difficult to control it by voice by building a voice interface control word list and matching interface description words.

For ease of understanding, refer to Figure 4, taking the display interface of the device homepage as an example, the display interface includes multiple controls such as search, time, VIP, footprints, news, history, selections, multimedia resources (referred to as media resources), etc. The user can select a target control from the controls displayed in the display interface through the above-mentioned control device 100, the above-mentioned smart device 300 or voice control instructions and other control methods to open the content display interface of the target control.

As an example, if the user needs to operate control A in the display interface shown in Figure 4, the user inputs the voice control command of "activate VIP" to the display device. After the display device receives the voice control command, it traverses the interface description words of each control in the voice interface control word list of the display interface according to the voice control text corresponding to the voice control command. After matching the control corresponding to the interface description word of "activate VIP" as control A, it operates control A to open the content display interface for activating VIP, so that the user can further perform related operations for activating VIP rights in the content display interface.

However, there is no corresponding interface description word for the scroll control, the number of scroll controls included in a display interface may be different, and the scrolling direction of each scroll control is not unique. Therefore, it is impossible to determine the target scroll control that the user needs to operate from the display interface based on the user's voice control instructions through the above-mentioned interface description word matching method.

Continuing to refer to Figure 4, for the scrolling controls in the display interface that do not carry interface description words: B control and C control, the B control can slide up or down, and the C control can slide left or right, and the B control and the C control are only displayed as a sliding bar in the display interface, and there are no corresponding interface description words. Therefore, it is difficult to determine whether the target scrolling control to be operated is the B control or the C control in the display interface based on the user's voice control instructions.

Based on this, the present application provides a voice control method to identify a scroll control in the current display interface and construct a voice scroll control word list of the scroll control in the current display interface, so that after receiving the user's voice control instruction, the display device 200 can determine the target scroll control to be operated in the current display interface according to the voice scroll control word list and the voice control text corresponding to the voice control instruction, and control the target scroll control to perform a scroll operation in the scroll direction indicated by the voice control text. In this way, voice control of the scroll control in the display device is realized.

5 , when the display device 200 executes the voice control method provided in the present application, the controller 250 in the display device 200 is configured to perform the following steps:

Step 510: Determine at least one scroll control included in the current display interface.

It should be understood that the current display interface may not have a scroll control, or may contain one scroll control, or may contain multiple scroll controls. If the current display interface does not have a scroll control, the user can directly follow the interface description words of each control in the current display interface to directly voice control the corresponding control to perform operations such as opening/selecting.

If there is a scroll control in the current display interface, the technical solution provided in the present application is executed to determine at least one scroll control contained in the current display interface, and construct a list of voice scroll control words for the scroll control in the current display interface to implement voice control of the scroll control in the current display interface to perform scrolling operations.

As described above, the current display interface in some embodiments of the present application includes at least one of a home page, a web page, and a user interface within an application.

In some embodiments, if the current display interface is a home page, when executing the above step 510, the controller is further configured to: after the display device is powered on, obtain the scrolling control contained in the home page, and implement voice control of the scrolling control in the home page to perform a scrolling operation through the following steps 520-530.

In some embodiments, if the current display interface is a web page, then when executing step 510 above, the controller is further configured to: when a jump operation based on a web page link is detected, use web crawler technology to obtain web page related information of the jump web page, and obtain the web page content of the web page by parsing the web page related information, thereby determining whether there is a scroll control in the web page based on the web page content.

As an example, the total height of all content items in the webpage is determined based on the webpage content. If the total height of the content items is greater than the window height value of the display interface, there is a scroll control in the webpage, and the scrolling direction of the scroll control is vertical scrolling; if the total height of the content items is not greater than the window height value of the display interface, there is no scroll control in the webpage.

If it is determined that there is a scroll control in the webpage, the scroll control in the webpage can be voice-controlled to perform a scrolling operation through the following steps 520 to 530.

In some embodiments, if the current display interface is a user interface within an application, when executing the above step 510, the controller is further configured to: after the target application is started, monitor changes in the display interface of the target application; if the display interface changes, obtain control information of multiple controls contained in the current display interface after the change; and determine at least one scrolling control contained in the current display interface based on the control information.

It should be noted that the target application can be a system application that comes with the display device 200, an application installed by the user in the display device 200, or a third-party application launched through a network link. Some embodiments of the present application do not limit the source and type of the target application.

That is, the target application in the present application is any application triggered and started by the user in the display device 200 .

In one possible implementation, after the target application in the display device is started, it will register the Accessibility Service in the framework layer of the Android system. The Accessibility Service can receive some events triggered by the Android system, such as notification status, view-related events, fingerprints, button clicks (touch), etc.

Therefore, when the target application starts running and the target application is started, the accessibility service will monitor the changes in the display interface of the target application in real time. When the display interface changes, the callback interface of the accessibility service will receive the interface change notification. In this way, through the interface provided by the accessibility service, the root view node (Node Root) control of the current display interface after the change, as well as the sub-view node controls corresponding to each root view node control, can be obtained, and finally all the controls contained in the current display interface can be obtained.

Each control has control information such as type and function description information stored in the display device. In the display device, the control information of each control is stored in the form of a structure tree. Therefore, based on the above characteristics, the control information of all controls in the current display interface can be obtained through the structure tree of the current display interface.

Furthermore, a control node list (Node List) corresponding to the current display interface can be generated based on the control information of all controls contained in the current display interface.

It should be noted that the control node list includes controls with interface description words in the current display interface and scroll controls without interface description words.

Therefore, the control node list can be traversed to determine the type of each control in the control node list to determine the scroll control node list (Scroll Node List) corresponding to the current display interface.

In one possible implementation, the corresponding interface (such as the isScroll interface) is called through the accessibility service to determine whether the control corresponding to each node in the structure tree of the current display interface supports scrolling. If scrolling is supported, the control is determined to be a scroll control and is added to the scroll control node list.

In some embodiments, if the scroll control included in the display interface is relatively fixed (for example, the display position, type, function information, etc. of the scroll control have not changed within a preset time period), the corresponding relationship between the display interface and the scroll control node list can be stored after the scroll control corresponding to the display interface is determined for the first time. When the display interface is subsequently opened, the scroll control included in the display interface can be quickly determined according to the scroll control node list corresponding to the display interface.

Step 520: Construct a voice scrolling control word list for the scrolling control in the current display interface.

The voice scroll control word list is used to indicate the correspondence between the scroll direction of the scroll control and the semantic control words.

In some embodiments, when executing the above step 520, the controller is further configured to: obtain the display positions of multiple sub-controls controlled by the scrolling control in the current display interface; determine the scrolling direction of the scrolling control according to the display position of each sub-control; and construct a voice scrolling control word list for the scrolling control in the current display interface according to the scrolling direction.

The display position of the subcontrol may be the display area coordinates of the subcontrol in the current display interface.

In a possible implementation, a two-dimensional rectangular coordinate system is established with the upper left corner/center point of the current display interface as the coordinate center. Based on the two-dimensional rectangular coordinate system, the display position of each control (scroll control, sub-control of the scroll control, and other controls carrying interface description words) in the current display interface is determined.

As an example, the corresponding interface (e.g., getBounds interface) is called through the auxiliary function service to obtain the display position of each control in the current display interface. The display position may include the boundary coordinates of the control, such as the upper boundary coordinates, the lower boundary coordinates, the left boundary coordinates, and the right boundary coordinates.

Further, the display height of each sub-control is determined according to the display area coordinates of each sub-control in the current display interface. Then, according to the display height and display width, the scroll direction of the scroll control to which each sub-control belongs is determined.

In some embodiments, when determining the scrolling direction of the scrolling control, the controller is further configured to:

(1) If the display heights of the sub-controls in the current display interface are the same, the scrolling direction of the scrolling control is determined to be horizontal scrolling;

The display height is the difference between the upper and lower boundary coordinates of the control.

That is, when the display heights of each sub-control are the same, it means that multiple sub-controls are horizontally distributed in the current display interface. The scrolling direction of the scroll control to which the multiple sub-controls belong is controlled to be horizontal scrolling, so that the corresponding sub-controls can be browsed/selected/opened by sliding left and right/turning pages.

In one example, referring to FIG. 6 , taking the display interface as a user interface in a fitness application as an example, the fitness recommendation item in the display interface is a scroll control, and the scroll control can control the sliding of sub-controls corresponding to multiple fitness items.

Referring to Figure 6, the fitness recommendation items display icons of sub-controls such as finding courses, finding plans, fat loss, body shaping training, yoga, AI games, and screen casting fitness, as well as icons of other sub-controls provided by the fitness application but not displayed in the current display interface, such as icons corresponding to sub-controls such as dance and aerobics.

For ease of explanation, FIG6 only uses a triangular icon to represent the scroll control corresponding to the fitness recommendation item; of course, the scroll control can also be displayed in other forms, and some embodiments of the present application do not limit this.

In the display interface of the fitness application, since the display heights of the icons of the multiple fitness items included in the fitness recommendation item are all h, that is, the display heights of the multiple sub-controls included in the scrolling control corresponding to the fitness recommendation item are all the same, the scrolling direction of the scrolling control corresponding to the fitness recommendation item is horizontal scrolling.

In this way, by controlling the scroll control corresponding to the fitness recommendation item to perform a scroll operation in the horizontal direction, multiple fitness items recommended in the fitness application can be browsed in sequence, and then a fitness item of interest can be selected for practice.

(2) If the display widths of the sub-controls in the current display interface are the same, then the scrolling direction of the scrolling control is determined to be vertical scrolling;

The display width is the difference between the left and right border coordinates of the control.

That is, when the display widths of all sub-controls are the same, it means that multiple sub-controls are distributed vertically in the current display interface. The scrolling direction of the scroll control to which the multiple sub-controls belong is controlled to be vertical scrolling, so that the corresponding sub-controls can be browsed/selected/opened by sliding up and down/turning pages.

In one example, referring to FIG. 7 , taking the display interface as a user interface in a music application as an example, the playlist in the display interface (i.e., the XXX list in FIG. 7 ) is a scroll control, and the scroll control can control the sliding of sub-controls corresponding to multiple songs.

Referring to FIG. 7 , the playlist displays tabs for songs 01-09, as well as tabs for other sub-controls included in the playlist but not displayed in the current display interface, such as tabs corresponding to songs 10-121.

For ease of explanation, FIG7 only uses the arrow shown on the right to represent the scroll control corresponding to the playlist; of course, the scroll control can also be displayed in other forms, and some embodiments of the present application do not limit this.

In the display interface of the music application, since the display width of the tabs of the multiple songs included in the playlist is d, that is, the display width of the multiple sub-controls contained in the scroll control corresponding to the playlist is the same, the scrolling direction of the scroll control corresponding to the playlist is vertical scrolling.

In this way, by controlling the scroll control corresponding to the playlist to perform a scroll operation in the vertical direction, all songs included in the playlist can be browsed in sequence, thereby selecting songs of interest to play.

(3) If the display height and/or display width of each sub-control in the current display interface are different, determine the scroll control The scrolling direction is waterfall scrolling.

That is, if the display heights of the sub-controls are all different, or the display widths of the sub-controls are all different, or the display heights and display widths of the sub-controls are all different, it means that the multiple sub-controls are distributed in a waterfall flow manner in the current display interface, and the scrolling direction of the scroll control to which the multiple sub-controls belong is controlled to be waterfall flow scrolling, so that multiple sub-controls can be loaded by sliding up and down/turning pages, so that the user can browse/select/open the sub-controls of interest.

In one example, referring to FIG8 , a user interface of a picture viewing application is taken as an example, in which a scroll control is included, and the scroll control can control pictures corresponding to multiple sub-controls, and these sub-controls display corresponding pictures in a waterfall flow manner in the display interface.

Referring to FIG. 8 , the display interface includes pictures 1 to 8 , pictures corresponding to video 9 , and pictures corresponding to other sub-controls that are not displayed in the display interface.

For ease of explanation, FIG8 only uses the scroll bar on the right to represent the scroll control in the display page; of course, the scroll control can also be displayed in other forms, and some embodiments of the present application do not limit this.

In the display interface of the picture viewing application, since pictures 1-8 and video 9 are in different display areas in the current display interface, the boundary coordinates of the sub-controls corresponding to pictures 1-8 and video 9 are different. Therefore, the display heights and/or display widths of the sub-controls corresponding to pictures 1-8 and video 9 are different, and the scrolling direction of the scroll control is waterfall scrolling.

In this way, by controlling the scroll control in the display page to perform a scroll operation in the vertical direction, multiple pictures can be loaded at one time after each scroll operation, so that pictures of interest can be selected to perform operations such as previewing/downloading/processing.

In some embodiments, when constructing a voice scrolling control word list for a scrolling control in a current display interface, the controller is further configured to: set semantic control words corresponding to the scrolling control according to the scrolling direction of the scrolling control; and then generate a voice scrolling control word list for the scrolling control in the current display interface according to the scrolling direction and semantic control words of the scrolling control in the current display interface.

Among them, semantic control words include sliding semantic words and page turning semantic words.

As an example, for a scroll control with horizontal scrolling direction, the set sliding semantic words include "slide left" and "slide right"; for a scroll control with vertical scrolling direction, the set sliding semantic words include "slide up" and "slide down"; for a scroll control with waterfall scrolling direction, the set sliding semantic words include "slide up", "slide down", "slide left" and "slide right".

As an example, page turning semantic words include "previous page" and "next page".

In some embodiments, assuming that the scroll controls in the current display interface include M controls, N controls, and P controls, and the scroll direction of the M control is horizontal scrolling, the scroll direction of the N control is vertical scrolling, and the scroll direction of the P control is waterfall scrolling. For the current display interface, the following Table 1 provides an exemplary voice scroll control word list.

Table 1

Step 530: In response to the user's voice control instruction, based on the voice scrolling control word list and the voice control text corresponding to the voice control instruction, the target scrolling control is controlled to perform a scrolling operation in the scrolling direction indicated by the voice control text.

The target scroll control is a scroll control in the scroll controls included in the current display interface that corresponds to the voice-controlled text.

In some embodiments, referring to FIG. 9 , when implementing the above step 530, the controller is further configured to perform the following sub-steps:

Step 531: Obtain the voice control text corresponding to the voice control instruction.

Among them, the display device can establish a communication connection between the controller and the cloud server through the communicator, so as to parse the voice control instructions through the cloud server, thereby reducing the data processing volume and algorithm storage resource consumption of the display device.

In some embodiments, when implementing the above step 531, the controller is further configured to: send the voice control instruction to the cloud server to request the cloud server to parse and process the voice control instruction; and receive the voice control text corresponding to the voice control instruction sent by the cloud server.

That is, some embodiments of the present application store the voice analysis algorithm in a cloud server. After the cloud server receives the voice control instruction sent by the display device, it analyzes the voice control instruction and obtains the voice control text corresponding to the voice control instruction.

In some embodiments, if the cloud server parses the voice control instruction and determines that the voice control instruction is an invalid instruction (for example, the relevant control word character information is not parsed), it sends abnormal result feedback information to the display device to instruct the display device not to perform control operations on any controls for the time being and continue to detect the voice control instructions input by the user.

Step 532: Match the voice control text with the semantic control words in the voice scroll control word list to determine the target scroll control that the voice control text requests to control in the current display interface, and the target scroll direction of the target scroll control.

Among them, if there is only one matching scrolling control for the voice control text in the current display interface, the scrolling control is directly determined as the target scrolling control, and the target hybrid control is controlled to perform a scrolling operation in the target scrolling direction according to the voice control text, that is, step 533.

In some embodiments, if the current display interface contains multiple candidate scrolling controls corresponding to the voice-controlled text, when executing the above step 532, the controller is further configured to: obtain the control priority of each candidate scrolling control; and determine the target scrolling control from the multiple candidate scrolling controls according to the control priority.

In a possible implementation, the control priority of each candidate scrolling control is determined according to the management and control relationship between each candidate scrolling control and the display position of each candidate scrolling control in the current display interface.

Among them, in the structure tree corresponding to the current display interface, the priority of the scroll control in the parent node is higher than the priority of the scroll control in the child node.

Among them, the priority of the scroll control whose display position is on the upper left side of the current display interface is higher than the priority of the scroll control whose display position is on the lower right side of the current display interface.

As an example, referring to the display interface shown in FIG10 , the display interface includes multiple scroll controls: W control, X control, Y control and Z control. Among them, the W control can control the X control, the Y control and the Z control, and the priority of the W control is higher than that of the X control, the Y control and the Z control.

The control priorities of the X control, the Y control, and the Z control can be further determined according to their display positions in the display interface. The X control is at the top of the display interface and has the highest priority; the Z control is at the bottom of the display interface and has the lowest priority. Therefore, the control priorities of the X control, the Y control, and the Z control are: control X>control Y>control Z.

Therefore, the control priority of the four scroll controls in FIG10 is: control W> control X> control Y> control Z.

In this way, after the control priority of the scroll control in the current interface is determined, a target scroll control can be determined to perform a scroll operation according to the voice control instruction and the voice scroll control word list.

It should be understood that after receiving the user's voice control command, the controller can determine the control that the user requests to control in the current display interface in the order of "voice interface control word list first, then voice scrolling control word list"; the controller can also determine the control that the user requests to control in the current display interface in the order of "voice scrolling control word list first, then voice interface control word list", and some embodiments of the present application do not limit this.

In some embodiments, if it is determined that the current display interface includes a scroll control, after the display device receives the user's voice control instruction, it can first determine, based on the voice scroll control word list, whether the voice control text corresponding to the voice control instruction has a target scroll control requested to be controlled in the current display interface.

Furthermore, if the target scrolling control that matches the voice control text cannot be determined based on the voice scrolling control word list, then the voice control text is continued to be used to determine whether there is a target control for which control is requested in the voice interface control word list of the current display interface, wherein the interface description word of the target control in the voice interface control word list is the same as the voice control text.

Step 533: According to the voice control text, control the target scroll control to perform a scroll operation in the target scroll direction.

In some embodiments, when implementing the above step 533, the controller is further configured to: obtain a target scrolling distance of the target scrolling control according to the voice control text; and control the target scrolling control to perform a scrolling operation in the target scrolling direction according to the target scrolling distance.

Among them, the voice control text can include sliding voice text and page turning voice text.

In a possible implementation, if the voice control text is a sliding voice text, the scrolling distance may be a preset distance value. For example, the distance value may be set according to the display height, display width, display position interval, etc. of the multiple sub-controls controlled by the scroll control, and some embodiments of the present application do not limit this.

As an example, if the voice control text is a sliding voice text, see Figure 6, the scrolling distance of the scroll control can be the display position interval between the "Find Course" icon and the "Find Plan" icon; see Figure 7, the scrolling distance of the scroll control can be the display width of the tab of Song 01, that is, d; see Figure 8, the scrolling distance of the scroll control can be the display width or display height of any picture in the display interface.

In some embodiments, if the voice-controlled text is a page-turning voice text, when calculating the scrolling distance, the controller is further configured as follows: if the scrolling direction is horizontal scrolling, the scrolling distance is calculated based on the display position of the rightmost sub-control and the display position of the leftmost sub-control controlled by the target scrolling control; if the scrolling direction is vertical scrolling or waterfall scrolling, the scrolling distance is calculated based on the display position of the topmost sub-control and the display position of the bottommost sub-control controlled by the target scrolling control.

As an example, if the voice control text is a page-turning voice text, see Figure 6, the scrolling distance of the scroll control can be the total display width value between the "Find Course" icon and the "Cast Screen Training"icon; see Figure 7, the scrolling distance of the scroll control can be the total display width value between the "Find Course" icon and the "Cast Screen Training" icon. The distance can be the total display height value between the tabs of songs 01 to 09, that is, 9*d; referring to FIG8 , the scrolling distance of the scroll control can be the total display height value or the total display width value of multiple pictures in the current display interface. For example, the scrolling distance can be the total display height value of Picture 2 and Picture 6 in the display interface.

Based on the above content, the controller in the display device provided by the present application is connected to the display and the sound collector respectively. In the process of the display displaying the image screen and the user interface, the controller determines at least one scroll control included in the current display interface and constructs a voice scroll control word list of the scroll control in the current display interface. Wherein, the voice scroll control word list is used to represent the corresponding relationship between the scroll direction of the scroll control and the semantic control word. Further, after the sound collection device detects and collects the user's voice control instruction, the voice control instruction is sent to the controller, so that the controller responds to the voice control instruction, based on the pre-constructed voice scroll control word list and the voice control text corresponding to the voice control instruction, and controls the target scroll control corresponding to the voice control instruction in the current display interface to perform a scroll operation. In this way, by identifying the scroll control and constructing a voice scroll control word list for the scroll control, when the display device receives the user's voice control instruction, the target scroll control in the current display interface can be controlled to perform a scroll operation according to the voice control instruction, which overcomes the defect that the scroll control cannot be directly controlled by voice, realizes the full voice control of each control in the display device, improves the flexibility and convenience of the display device control method, and improves the user experience.

It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the steps or stages in other steps.

In addition, the present application also provides a voice control method, which can be applied to the above-mentioned display device 200 or other electronic devices, and the method includes:

Determine at least one scroll control included in the current display interface;

Constructing a voice scroll control word list of a scroll control in the current display interface; the voice scroll control word list is used to indicate the correspondence between the scroll direction of the scroll control and the semantic control word;

In response to the user's voice control instruction, based on the voice scrolling control word list and the voice control text corresponding to the voice control instruction, the target scrolling control is controlled to perform a scrolling operation in the scrolling direction indicated by the voice control text; the target scrolling control is a scrolling control corresponding to the voice control instruction among the scrolling controls contained in the current display interface.

In some embodiments, referring to FIG. 11 , the voice control logic of the scrolling control is as follows: after the target application is started, an accessibility service that can monitor page changes is registered, and changes in the display page are monitored through the accessibility service. When the controls in the display page change, that is, when the display page changes, the root view node control in the current display interface and the sub-view node control controlled by the root view node control are obtained according to the structure tree of the controls in the current display interface, and a control node list corresponding to the current display interface is generated according to the root view node control and the sub-view node control.

Furthermore, the control node list of the current display interface is traversed to determine at least one scroll control included in the current display interface, and a scroll control node list corresponding to the current display interface is generated.

Further, for any scroll control, the display positions of multiple sub-controls controlled by the scroll control are obtained, such as the upper boundary coordinates, lower boundary coordinates, left boundary coordinates and right boundary coordinates of the control; according to the display positions of each sub-control controlled by the scroll control, the scroll direction of the scroll control is determined, and the semantic control word corresponding to the scroll control is set; according to The scrolling direction and semantic control words of each scrolling control are used to generate a voice scrolling control word list of the scrolling control in the current display interface.

If a voice control command from the user is received, when the display device implements voice control, it traverses the voice scroll control word list according to the voice control text corresponding to the voice control command, determines the target scroll control corresponding to the voice control text, and then controls the target scroll control to perform a scroll operation.

It should be noted that, when the current display interface includes multiple scroll controls, the target scroll control in the current display interface is controlled to perform a scrolling operation according to the control priority.

Regarding the implementation principle and beneficial effects of the voice control method, please refer to the specific limitations of the various embodiments of the controller configuration content in the display device mentioned above, which will not be repeated here.

In some embodiments, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium may store a computer program, which, when called and executed by a controller in a display device or other electronic device, implements some or all steps of the voice control method provided in the present application.

As an example, the computer-readable storage medium may be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, or the like.

It should be understood that the technical solutions in some embodiments of the present application can be implemented by means of software plus a necessary general hardware platform. Therefore, the technical solutions in some embodiments of the present application can essentially or contribute to the prior art in the form of software products, which can be stored in a computer-readable storage medium.

In some embodiments, the present application further provides a computer program product, wherein the computer program product includes a computer program, which, when called and executed by a controller in a display device or other electronic device, can implement some or all steps of the voice control method provided in the present application.

For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above discussion in some embodiments is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. According to the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are intended to better explain the content of this application, so that those skilled in the art can better use the embodiments.

That is, for ordinary technicians in this field, any modifications, equivalent substitutions, improvements, etc. made without departing from the technical concept of this application should be included in the protection scope of this application.

Claims (10)

A display device, comprising: A display configured to display an image screen and a user interface; A sound collector is configured to collect a user's voice control command; A controller connected to the display and the sound collector respectively, wherein the controller is configured as follows: Determine at least one scroll control included in the current display interface; Constructing a voice scroll control word list of the scroll control in the current display interface; the voice scroll control word list is used to indicate the correspondence between the scroll direction of the scroll control and the semantic control word; In response to the user's voice control instruction, based on the voice scroll control word list and the voice control text corresponding to the voice control instruction, the target scroll control is controlled to perform a scroll operation in the scroll direction indicated by the voice control text; the target scroll control is a scroll control corresponding to the voice control text among the scroll controls contained in the current display interface. The display device according to claim 1, wherein the controller is further configured to: After the target application is started, monitoring the display interface changes of the target application; If the display interface changes, then obtain the control information of multiple controls included in the current display interface after the change; At least one scroll control included in the current display interface is determined according to the control information. The display device according to claim 1, wherein the controller is further configured to: Obtaining display positions of multiple sub-controls controlled by the scroll control in the current display interface; Determining the scrolling direction of the scrolling control according to the display position of each of the sub-controls; According to the scrolling direction, a voice scrolling control word list of the scrolling control in the current display interface is constructed. According to the display device of claim 3, the controller is further configured to: If the display heights of the sub-controls in the current display interface are the same, determining that the scrolling direction of the scrolling control is horizontal scrolling; If the display widths of the sub-controls in the current display interface are the same, determining that the scrolling direction of the scrolling control is vertical scrolling; If the display heights and/or display widths of the sub-controls in the current display interface are different, the scrolling direction of the scrolling control is determined to be waterfall scrolling. According to the display device according to any one of claims 1 to 4, the controller is further configured to: Obtaining a voice control text corresponding to the voice control instruction; Matching the voice control text with the semantic control words in the voice scroll control word list, determining the target scroll control that the voice control text requests to control in the current display interface, and the target scroll direction of the target scroll control; According to the voice control text, the target scroll control is controlled to perform a scroll operation in the target scroll direction. According to the display device of claim 5, the controller is further configured to: Acquiring a target scroll distance of the target scroll control according to the voice control text; The target scroll control is controlled to perform a scroll operation in the target scroll direction according to the target scroll distance. According to the display device of claim 6, the voice control text includes page turning voice text, and the controller is further configured to: If the target scrolling direction is horizontal scrolling, the target scrolling distance is calculated according to the display position of the rightmost subcontrol and the display position of the leftmost subcontrol controlled by the target scrolling control; If the target scrolling direction is vertical scrolling or waterfall scrolling, the target scrolling distance is calculated according to the display position of the uppermost subcontrol and the display position of the lowermost subcontrol controlled by the target scrolling control. According to the display device of claim 5, if the current display interface includes a plurality of candidate scroll controls corresponding to the voice-controlled text, the controller is further configured to: Obtaining the control priority of each of the candidate scroll controls; According to the control priority, the target scroll control is determined from the multiple candidate scroll controls. According to the display device of claim 5, the controller is further configured to: Sending the voice control instruction to a cloud server to request the cloud server to parse and process the voice control instruction; Receive the voice control text corresponding to the voice control instruction sent by the cloud server. A voice control method, comprising: Determine at least one scroll control included in the current display interface; Constructing a voice scroll control word list of the scroll control in the current display interface; the voice scroll control word list is used to indicate the correspondence between the scroll direction of the scroll control and the semantic control word; In response to the user's voice control instruction, based on the voice scroll control word list and the voice control text corresponding to the voice control instruction, the target scroll control is controlled to perform a scroll operation in the scroll direction indicated by the voice control text; the target scroll control is a scroll control corresponding to the voice control text among the scroll controls contained in the current display interface.