WO2021101039A1 - Display device and method for controlling display device - Google Patents

Abstract

A disclosed display device according to one embodiment comprises: a sensor that detects a user's gesture; a display that displays user interface (UI) elements; and a processor that controls the display to emphasize a first UI element on the basis of the user's gesture, to move the emphasized first UI element on the basis of the movement direction of the gesture, or to move and display a second UI element corresponding to the movement direction to the first UI element.

Description

Display device and control method of display device

Disclosed embodiments relate to a display device that displays feedback on a user's gesture without using an indicator such as a cursor, and a control method of the display device.

In general, a display device is an output device that visually displays received or stored image information to a user, and is used in various fields such as homes and businesses. For example, the display device is a monitor device connected to a personal computer or a server computer, a portable computer device, a navigation terminal device, a general television device, an Internet Protocol television (IPTV) device, a smart phone, a tablet. Personal computer (PC), personal digital assistant (PDA, Personal Digital Assistant), portable terminal device such as a cellular phone, various display devices used to reproduce images such as advertisements or movies in industrial sites, or various other devices There are kinds of audio/video systems and so on.

Recently, displays that provide various contents in addition to digital broadcasting functions have been developed. Such a display displays a user interface related to various contents, and allows a user interface displayed by a user's manipulation to be selected.

Meanwhile, the conventional display induces the user to select a user interface through a method of displaying a cursor on the screen. In the conventional display, feedback is performed by recognizing a user's gesture and moving a cursor on the screen through the user's gesture. However, this method requires precise control from the user, and when the cursor does not move to the desired position, the user has to make a gesture several times.

In addition, since the conventional display detects the user's gesture without considering the distance between the user and the device, the same cursor movement sensitivity and the same size of the user interface are provided according to the distance from the user.

An aspect disclosed is a display device and a control method for reducing user's concentration and reducing fatigue by omitting a result of recognizing a user's gesture such as a cursor and providing feedback through a change in a user interface.

The display device according to the disclosed embodiment includes a sensor for detecting a user's gesture; A display for displaying a UI (User Interface) element; And a first UI element is emphasized based on the user's gesture, and the highlighted first UI element is moved based on the movement direction of the gesture, or a second UI element corresponding to the movement direction is selected from the first UI element. And a processor that controls the display to move to and display.

The processor may set a preset boundary on the first UI element and the second UI element, respectively, and move the highlighted first UI element based on the user's gesture and the preset boundary.

When the movement of the gesture deviates from a preset boundary of the first UI element, the processor may release the emphasis of the first UI element and restore the moved first UI element.

When the movement of the gesture deviates from a preset boundary of the first UI element and enters a preset boundary of the second UI element, the processor may highlight the second UI element.

When the movement of the gesture deviates from a preset boundary of the first UI element and enters a preset boundary of the second UI element, the processor determines the color of the highlighted first UI element and the color of the second UI element. It can be changed sequentially.

The processor may highlight the first UI element based on the location of the user's gesture and a center point of the plurality of UI elements.

The at least one UI element includes the plurality of layers and icons, and highlights the first UI element based on rotation or movement of the plurality of layers, or the first UI element based on the size or shape of the icon. UI elements can be emphasized.

The processor may differently change the rotation angle of the first layer and the second layer among the plurality of layers based on the user's gesture.

The processor may determine a position at which the first UI element and the second UI element are displayed based on at least one of a length of the user's arm, a viewing angle of the user, or a distance between the user and the display device.

The processor may determine a size and a position at which the first UI element and the second UI element are displayed based on the length of the user's arm.

The processor may determine a position and a size at which the first UI element and the second UI element are displayed based on a distance between the user and the display device and a viewing angle of the user.

The processor may activate the highlighted first UI element based on the user's gripping operation.

The processor may adjust a setting value of the activated first UI element when detecting the movement of the hand while the user is grasping.

The processor may highlight the first UI element based on at least one of a line, a figure, a color, or a size of the first UI element.

A display device according to another disclosed embodiment includes a sensor for detecting a user's gesture; A display for displaying a UI (User Interface) element; And a processor that highlights the first UI element based on the user's gesture and moves the second UI element to the highlighted first UI element based on the movement direction of the gesture.

According to another disclosed embodiment, a method of controlling a display device includes: detecting a user's gesture; Display a UI (User Interface) element; Highlighting the first UI element based on the user's gesture; And controlling the display such that the highlighted first UI element moves or a second UI element corresponding to the movement direction moves to the first UI element based on the movement direction of the gesture.

The controlling includes: setting a preset boundary for each of the first UI element and the second UI element, and moving the highlighted first UI element based on the user's gesture and the preset boundary; can do.

The controlling may include, when the movement of the gesture deviates from a preset boundary of the first UI element, canceling the emphasis of the first UI element and returning the moved first UI element; have.

The controlling may include highlighting the second UI element when the movement of the gesture deviates from a preset boundary of the first UI element and enters a preset boundary of the second UI element.

The controlling is, when the movement of the gesture deviates from a preset boundary of the first UI element and enters a preset boundary of the second UI element, the highlighted color of the first UI element and the color of the second UI element To sequentially change; may include.

A display device and a control method according to an aspect of the disclosure omit a result of recognizing a user's gesture and provide feedback through a change in a user interface, thereby reducing a user's concentration and reducing fatigue.

A display device and a control method according to another aspect provide a user with ease of selection of a user interface through a gesture, thereby providing a quick coordinate recognition and a simple provision method.

The display device and the control method according to another aspect may provide convenience by changing the size and position of the user interface according to the distance of the user and the length of the viewing angle arm.

1 is a diagram illustrating a display device according to an exemplary embodiment.

2 is a control block diagram of a display device according to an exemplary embodiment.

3 to 5 are diagrams for explaining the operation of the display device.

6A to 6C are diagrams for explaining a reference required for an operation of a display device.

7 to 9 are diagrams of a specific embodiment of moving an highlighted UI element.

10A and 10B are diagrams for explaining an embodiment of emphasizing a UI element.

11 and 12 are diagrams for explaining a method of displaying a UI element that is changed according to the length of a user's arm.

13 and 14 are diagrams for explaining a method of displaying a UI element that is changed according to a viewing angle.

15 to 17 are diagrams of an embodiment of activating a UI element.

18 is a flowchart illustrating a method of controlling a display device according to an exemplary embodiment.

19 is a flowchart illustrating a method of controlling a display according to another exemplary embodiment.

The same reference numerals refer to the same elements throughout the specification. This specification does not describe all elements of the embodiments, and general content in the technical field to which the present invention belongs or overlapping content between the embodiments will be omitted. The term'unit, module, member, block' used in the specification may be implemented in software or hardware, and according to embodiments, a plurality of'units, modules, members, blocks' may be implemented as one component, It is also possible for one'unit, module, member, block' to include a plurality of components.

Throughout the specification, when a part is said to be "connected" with another part, this includes not only the case of being directly connected, but also the case of indirect connection, and the indirect connection includes connection through a wireless communication network. do.

In addition, when a part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

Throughout the specification, when a member is said to be positioned "on" another member, this includes not only the case where a member is in contact with the other member, but also the case where another member exists between the two members.

Terms such as first and second are used to distinguish one component from other components, and the component is not limited by the above-described terms.

Singular expressions include plural expressions, unless the context clearly makes exceptions.

In each step, the identification code is used for convenience of explanation, and the identification code does not describe the order of each step, and each step may be implemented differently from the specified order unless a specific sequence is clearly stated in the context. have.

Hereinafter, the operating principle and embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a diagram illustrating a display device according to an exemplary embodiment.

Referring to FIG. 1, the display device 1 may include a display panel 2, a housing 3 supporting the display panel 2, and a support 4.

The housing 3 forms an exterior of the display device 1, and includes a component inside the display device 1 to display various images or perform various functions. The housing 3 may be integrally formed with the display panel 2 or may be formed of a combination of a front housing and a rear housing.

The support 4 supports the housing 3. The support 4 may have various shapes depending on the designer's selection, and may be omitted if necessary. The support 4 can be attached to or detached from the housing 3 as necessary.

The display panel 2 is installed in front of the housing 3 and can display various images outside. Specifically, the display panel 2 may display at least one or more still images or moving images. The display panel 2 may be implemented using additional components such as a touch panel, if necessary.

A plurality of pixels are formed on the display panel 2, and an image displayed through the display panel 2 may be formed by a combination of light emitted from the plurality of pixels P. For example, a single image I may be formed on the screen 102 by combining light emitted from the plurality of pixels P like a mosaic.

As shown in FIG. 1, an image displayed by the display panel 2 may include various user interfaces 10. The user interface may include icons and shapes capable of representing various functions of the display device 1. Hereinafter, individual user interfaces included in the user interface will be described using the term UI (User Interface) element 10. Meanwhile, the user interface displayed by the display device 1 is not necessarily limited to the shape and shape shown in FIG. 1.

The display device 1 detects the user U and changes the UI element 10 to feed back the user U with a result of detecting a gesture that the user U takes.

2 is a control block diagram of a display device according to an exemplary embodiment.

Referring to FIG. 2, the display apparatus 1 includes a user input unit 110 receiving an input command from a user U, a content receiving unit 120 receiving image data from an external source device, and a user U. The sensor 130 for detecting a gesture of, and the image data received by the content receiving unit 120 and the input command received by the user input unit 110 are processed, and according to the user's gesture detected by the sensor 130 A processor 140 that provides feedback, an output unit 150 that outputs images and sounds processed by the processor 140, and a database (DB, 160) that stores data necessary for the operation of the processor 140 Includes.

The user input unit 110 may include an input button 111 for receiving a user input command. For example, the user input unit 110 includes a power button for turning on or off the display device 1, a source selection button for selecting one source device among a plurality of source devices, and an output from the display device 1. It may include a sound control button or the like for adjusting the sound volume.

The user input command implemented by the input button 111 may be implemented as a user interface, and may be displayed through the UI element 10 instead of the input button.

The input buttons 111 may each receive a user input and transmit an electrical signal corresponding to the user input to the processor 140, and the display device may be provided by various input means such as a push switch, a touch switch, a dial, a slide switch, and a toggle switch. It can be implemented in the housing 3 of (1).

The user input unit 110 includes a signal receiving unit 112 for receiving a remote control signal from the remote controller 112a.

The signal receiver 112 may receive a wireless signal corresponding to a user input from the remote controller 112a and transmit an electrical signal corresponding to the command, that is, an input command transmitted by the remote controller 112a, to the processor 140. .

The remote controller 112a may be provided separately from the display device 1, receive a user input command, and transmit a wireless signal corresponding to the user input to the display device 1. The remote controller 112a may more efficiently transmit a gesture of the user U. For example, the user U may operate the remote controller 112a instead of moving the user's hand. The display device 1 may recognize a user's gesture based on a difference in a signal transmitted from the remote controller 112a, and adjust the UI element 10 accordingly.

The content receiving unit 120 may include a wired receiving module 121 for receiving image data from a source device by wire, and a wireless receiving module 122 for receiving image data wirelessly from a source device.

The source device refers to providing various image data to the display apparatus 1 and may include all various configurations such as a set-top box, an external server, and a repeater of an Internet network.

The wired reception module 121 may receive image data from a source device through various types of image transmission cables. For example, the wired reception module 121 is a component (YPbPr/RGB) cable or a composite (composite video blanking and sync, CVBS) cable, or a High Definition Multimedia Interface (HDMI) cable or a universal serial bus ( Video data can be received from the source device through Universal Serial Bus, USB) cable or Ethernet (Ethernet, IEEE 802.3 technology standard) cable.

When receiving image data from a source device through the wired receiving module 121, the display apparatus 1 may receive image frame data from the source device. Here, the image frame data may represent uncompressed image data as a bit stream representing an image of one frame.

Since the wired reception module 121 receives image data through an image transmission cable, the data transmission rate is not limited. Accordingly, the wired reception module 121 may receive the image frame data from the source device as it is.

The wireless receiving module may receive content from a source device using various wireless communication standards.

For example, the wireless receiving module 122 uses Wi-Fi (WiFi, IEEE 802.11 technology standard), Bluetooth (Bluetooth, IEEE 802.15.1 technology standard) or ZigBee (ZigBee, IEEE 802.15.4 technology standard), etc. Thus, image data can be wirelessly received from the source device. In addition, the wireless reception module 122 may wirelessly receive image data from the source device using CDMA, WCDMA, GSM, Long Term Evolution (LET), WiBro, or the like.

When receiving image data from a source device through the wireless reception module 122, the display apparatus 1 may receive compressed/encoded image data from the source device. Here, the compressed/encoded image data may represent a bit stream in which an image of one frame or an image of a plurality of frames is compressed/encoded. For example, video frame data may be compressed/encoded according to video compression standards such as H.264/MPEG-4 AVC (Moving Picture Experts Group-4 Advance Vide Coding) or H.265/HEVC (High Efficiency Video Coding). I can. By compressing/encoding the image frame data, the compressed/encoded image data may have a smaller capacity (or size) than the original image frame data.

Since the wireless reception module 122 wirelessly receives image data, the data transmission rate is limited, and thus the wireless reception module 122 may receive compressed/encoded image data from the source device.

The content receiver 120 may receive image data from a source device via wire or wirelessly, and transmit the received image data to the processor 140.

The sensor 130 detects a gesture of the user U. Specifically, the sensor 130 may be a camera 131. The camera 131 generates an image (eg, a continuous frame) corresponding to the user's gesture in the camera recognition range. The recognition range of the camera 131 may be within 0.1 to 5 m from the display device 1 to the user U. The user gesture may include all movements of a user's body part or body part such as a user's face, facial expression, hand, fist, or finger.

The camera 131 may convert the received image into an electrical signal and transmit it to the processor 140. The camera 131 may include a lens (not shown) and an image sensor (not shown). The camera 131 may support optical zoom or digital zoom using a plurality of lenses and image processing. The recognition range of the camera 131 may be variously set according to the angle of the camera and environmental conditions. When the camera 131 is composed of a plurality of cameras, a 3D still image may be photographed using a plurality of cameras.

In addition to the camera 131, the sensor 130 may further include various sensors capable of determining a user's gesture. For example, the sensor 130 may further include various sensors such as an ultrasonic sensor and an IR sensor that sense a distance between the display device 1 and the user U.

The processor 140 controls the overall operation of the display device 1.

The processor 140 may control the content receiver 120 to receive image data from a source device, process the received image data, and output the image through the output unit 150. Specifically, the processor 140 may reconstruct the image frame data by decoding the compressed/encoded image data, and may render the image by processing the reconstructed image frame data.

In addition to image data, the processor 140 may display various user interfaces through the output unit 150. For example, when the display device 1 is connected to an Internet network, the display device 1 may perform various functions supported by the Internet network. The display device 1 may output, to the plurality of UI elements 10, an application capable of performing various functions received through a network and an embedded function during manufacturing.

The processor 140 highlights one UI element among the plurality of UI elements 10 based on the user's gesture detected by the sensor 130, or at least two UI elements among the plurality of UI elements 10. The output unit 150 may be controlled to sequentially emphasize the elements. Specifically, unlike a conventional display device, the processor 140 does not display a cursor corresponding to a user's gesture. When the user's gesture is detected, the processor 140 highlights the UI element 10 and moves the highlighted UI element 10 to correspond to the movement direction of the gesture, or by moving another UI element displayed in the movement direction. , The result of detecting the user's gesture is fed back. Through this, the user U can easily select a UI element without precisely controlling his or her movement corresponding to the cursor.

The processor 140 can highlight the UI element by changing at least one of the line, figure, color, or size of the UI element, and various embodiments of changing the UI element 10 will be described later through other drawings below. .

The output unit 150 includes a display panel 2 that visually displays an image, an image display unit 151 including a display driver (not shown) that drives the display panel 2, and an audio amplifier (not shown) that amplifies sound. City), a speaker (not shown) for aurally outputting the amplified sound, and a sound unit 152 including a microphone (not shown) for collecting ambient sound.

The display panel 2 may include pixels serving as a unit for displaying an image. Each pixel may receive an electrical signal representing an image from the display driver, and may output an optical signal corresponding to the received electrical signal. An image may be displayed on the display panel 2 by combining optical signals output from a plurality of pixels.

Display panels are applied to various types of panels such as Liquid Crystal Display Panel (LCD Panel), Light Emitting Diode Panel (LED Panel) or Organic Light Emitting Diode Panel (OLED Panel). It can be implemented by

The processor 140 may process sound data included in the transmitted image data and convert it into an sound signal.

The sound unit 152 may amplify the sound signal output through the audio amplifier. In addition, the speaker may convert the sound signal amplified by the audio amplifier into sound (sound wave). For example, the speaker may include a thin film that vibrates according to an electrical sound signal, and a sound wave may be generated by the vibration of the thin film. The microphone may collect ambient sound from the display device 1 and convert the collected sound into an electrical sound signal. The acoustic signal collected by the microphone may be transmitted to the processor 140.

The database 170 may store programs and data for controlling components included in the display device 1 and may store temporary control data issued while controlling components included in the display device 1.

The database 170 may store a program and data for decoding image data received by the content receiving unit 120 and a program and data for reproducing a stereoscopic image from the decoded image data. In addition, the database 170 may store temporary image data generated while decoding image data or while rendering a stereoscopic image.

The database 170 may store various ergonomic information that can be used to display or adjust the UI element 10. In terms of ergonomics, the maximum range of one arm that the user U can move in a comfortable position is 80 cm. The database 170 stores a range of movement of the arm corresponding to the size of the user U sensed by the sensor 130. When the user U is detected, the database 170 may provide arm length information to the processor 140. The processor 140 may determine the size and position of the UI element 10 based on the transmitted arm length, and display the UI element 10 through the output unit 150.

In addition, the database 170 stores information on the viewing angle of the user U. According to the ergonomic aspect, the viewing angle is the maximum cognitive area of 30 degrees from the center of both eyes of the user. The database 170 transmits the viewing angle of the user U corresponding to the distance between the display device 1 detected by the sensor 130 and the user U to the processor 140. The processor 140 determines the size and position of the UI element 10 based on the transmitted viewing angle, and displays the UI element 10 through the output unit 150.

The database 170 includes a nonvolatile memory such as a ROM (Read Only Memory) and a flash memory for storing data for a long period of time, and a static random access memory (S-RAM) (S-RAM) and D for temporarily storing data. -It may be provided as a volatile memory such as a dynamic random access memory (RAM). The database 170 may be a memory implemented as a separate chip from the processor 140, or may be implemented as a single chip with the processor 140.

Meanwhile, the display device 1 may further include various configurations in addition to the above-described configurations.

3 to 5 are diagrams for explaining the operation of the display device. It will be described together below in order to avoid redundant description.

Referring first to FIG. 3, the display device 1 may display a plurality of UI elements 11 to 16. The plurality of UI elements 11 to 16 may be implemented in various shapes, including an icon that allows a function to be guessed.

Specifically, the display device 1 is changed to a mode in which UI elements 11 to 16 can be selected according to an input command of the user U. The display device 1 detects the user U through the sensor 130 and determines the size and position of the UI elements 11 to 16 in correspondence with the distance to the user U, the viewing angle, and the length of the arm. The display device 1 displays the UI elements 11 to 16 as shown in FIG. 3 based on the determined size and position of the UI elements 11 to 16.

Meanwhile, the UI elements 11 to 16 are not necessarily limited to the shape shown in FIG. 3, and may have various shapes and arrangement structures.

Referring to FIG. 4, the display device 1 detects a gesture of a user U. For example, the user U may take a motion of raising the right arm. A conventional display device displays a cursor corresponding to a user's gesture. However, the disclosed display device 1 does not display UI elements such as a cursor.

Even if the cursor is not displayed, the display device 1 recognizes a position corresponding to the user's gesture. For convenience of explanation, the result of recognizing the user's gesture by the display device 1 is described as a virtual cursor 5.

Referring to FIG. 5, the display device 1 highlights a result of recognizing a user's gesture, that is, a UI element closest to the position of the virtual cursor 5 and a second UI element 12.

There may be various methods of emphasizing the second UI element 12 by the display device 1. For example, the display device 1 highlights the second UI element 12 by changing the second UI element 12 to a shape larger than the displayed size, changing the color inside the layer, or moving the shape of the icon. can do.

Through this, the user U can receive feedback of his or her gesture recognized by the display device 1 without moving the cursor. In addition, since the user U can take an additional gesture from the highlighted UI element 12, it may not be required to move the cursor precisely.

6A to 6C are diagrams for explaining a reference required for an operation of a display device. It will be described together below in order to avoid redundant description.

Referring to FIG. 6A, the UI element 10 may include a border 10a, a center point 10b of the UI element 10, and a boundary 10c of the UI element. In particular, the boundary 10c of the UI element has a preset width in proportion to the size of the UI element 10 to be displayed.

Referring to FIG. 6B, the display device 1 determines whether to highlight the UI element 10 based on the distance between the virtual cursor 5 and the center point of the UI element 10. As described above in FIGS. 3 to 5, the display device 1 does not display a specific cursor. Accordingly, as soon as the display device 1 detects a specific gesture of the user U, it highlights the UI element closest to the position corresponding to the gesture of the user (the position of the virtual cursor).

Specifically, the display device 1 calculates the distance between the position of the virtual cursor 5 and the center point of the plurality of UI elements 11 to 16. The display device 1 highlights the UI element having the closest calculated distance. As shown in FIG. 6B, the UI element having a center point closest to the position of the virtual cursor 5 may be the fourth UI element 14. The display device 1 emphasizes the fourth UI element 14.

The display device 1 may highlight a UI element selected from among the plurality of UI elements as shown in FIG. 6C.

In more detail, the display device 1 may change the size of the border 10a of the UI element and the icon included in the border. In addition, the display device 1 may move the UI element 10 to a position corresponding to the virtual cursor 5. However, the distance that the UI element 10 moves is based on the distance d between the center point 10b and the virtual cursor 5, and even if the edge 10a of the UI element moves, the location of the center point 10b is the UI element. Do not deviate from the boundary (10c) of.

7 to 9 are diagrams of a specific embodiment of moving an highlighted UI element. It will be described together below in order to avoid redundant description.

Referring to FIG. 7, the display device 1 displays the highlighted UI element 12 and detects an additional gesture of the user. For example, the user U may move his hand clockwise from the position of his right hand.

The display device 1 calculates a range of movement of a user's gesture and a direction of movement in the detected maximum range of the arm of the user U. When the user's gesture moves clockwise within a preset range, the display device 1 moves the highlighted second UI element 12 clockwise as shown in FIG. 7. In this case, the second UI element 12 is moved so that the center point 10b does not deviate from the boundary 10c of the UI element, and the icon 12d of the second UI element can also be moved within the frame 10a. .

Referring to FIG. 8, unlike FIG. 7, the display device 1 may detect an additional gesture of a user moving in a larger range.

The virtual cursor 5 corresponding to the movement of the user U may deviate from the boundary 10c of the UI element of the second UI element 12. When the virtual cursor 5 is closer to the third UI element 13 than the highlighted second UI element 12, it means that the movement of the hand among the gestures of the user U is changed to a certain distance or more. In this case, the display device 1 moves the emphasis to the third UI element 13.

The display device 1 moves the highlighted second UI element 12 to the third UI element 13 as shown in FIG. 8A.

As shown in FIG. 8B ), the display device 1 displays a color of only a certain area of the highlighted second UI element 12 in a dark color, and displays a color of a certain area of the third UI element 13 in a dark color.

When the virtual cursor 5 is further moved, the display device 1 moves the second UI element 12 to its original position, changes the size of the third UI element 13, and completely changes the color. .

The display device 1 according to the disclosed embodiment may provide feedback on the movement of the gesture to the user by sequentially changing the color of the emphasized second UI element 12 and the color of the third UI element 13. have.

Referring to FIG. 9, in the display device 1, the second UI element 12 and the third UI element 13 may be displayed at a sufficient distance from each other and spaced apart from each other at a preset interval. After the second UI element 12 is highlighted, the display device 1 may detect the direction of movement of the user U as shown in FIG. 8.

The display device 1 may move the highlighted second UI element 12 to the third UI element 13.

The virtual cursor 5 may be positioned between the boundary 12c of the second UI element 12 and the boundary 13c of the third UI element 13. The display device 1 changes the color of the emphasized second UI element 12. In addition, unlike FIG. 8, the display device 1 changes the size and color of the third UI element 13 while simultaneously moving the third UI element 13 to a position corresponding to the virtual cursor 5. I can.

Here, the emphasized second UI element 12 and the emphasized third UI element 13 emphasize a color only in a certain area, so that the user U can feed back the current movement path of the gesture.

When the virtual cursor 5 moves inside the boundary 13c of the third UI element 13, the display device 1 releases the emphasis on the second UI element 12. In addition, the display device 1 moves the second UI element 12 back to its original position. The display device 1 further emphasizes the moved third UI element 13.

10A and 10B are diagrams for explaining an embodiment of emphasizing a UI element. It will be described together below in order to avoid redundant description.

Referring to FIG. 10A, the UI element 20 according to another embodiment may include a plurality of layers 22 to 23 surrounding an icon (not shown). The plurality of layers 22 to 24 may be moved and rotated relative to the position of the virtual cursor 5.

As shown in FIG. 10A, the UI element 20 includes a first layer 22 skewed to the lower right of the center point 20b, a second layer 23 having the longest circumference, and a third layer skewed to the upper left ( 24).

When the virtual cursor 5 moves to the right, the first layer 22 closest to the moved virtual cursor 5 may be adjusted while having the largest rotation angle. Next, the second layer 23 may have a large movement and rotation angle, and the movement and rotation angle of the third layer 24 located farthest from the moved virtual cursor 5 may be the smallest.

10B shows the UI element 20 including the icon 21. 10B(a) is a state in which the UI element 20 is not highlighted. Based on the user's gesture, the UI element 20 may be emphasized as shown in b) of FIG. 10B. Specifically, the icon 21 of the UI element 20 is changed to a large size, and a plurality of layers 22 to 24 are generated. When an additional gesture of the user is detected, the changed icon 21 is rotated in the UI element 20, and each of the layers 22 to 24 rotates while having a different rotation angle.

Through the emphasized UI element 20, the user U can easily check the result of recognizing the gesture by the display device 1, and can easily receive feedback of an additional gesture through the highlighted UI element 20.

Meanwhile, the display device 1 includes the position of the UI element 10 displayed based on at least one of the length of the arm of the user U, the viewing angle of the user U, or the distance between the user U and the display device. You can adjust the size.

Specifically, the display device 1 can adjust the size and position of the UI element 10 displayed according to the distance from the user, and adjust the size and position of the UI element 10 according to the viewing angle of the user U. . In addition, the display device 1 adjusts the size and position of the UI element 10 to the length of the user U's arm. In addition, the display device 1 may adjust the size and position of the UI element 10 based on the distance and viewing angle from the user, and the size of the UI element 10 based on the distance from the user and the length of the user's arm. And it is also possible to adjust the position. In addition, the display device 1 may adjust the size and position of the UI element 10 in consideration of all of the distance to the user, the viewing angle, and the length of the user's arm.

11 and 12 are diagrams for explaining a method of displaying a UI element that is changed according to the length of a user's arm. It will be described together below in order to avoid redundant description.

11 and 12, the display device 1 detects a user U through a sensor 130. Specifically, the camera 131 may photograph the user U. The user U may have various arm lengths according to physical conditions. For example, the length of the arm of the user U having an adult body as shown in FIG. 11 is different from the length of the arm of the user U'having a minor body as shown in FIG. 12.

As shown in FIG. 11, the user U with a long arm has a wider range than the user U'of FIG. 12 in which the UI element can be manipulated. If the UI elements 11 to 16 displayed for the user U of FIG. 11 are displayed to the user U'of FIG. 12, the user U'may be inconvenient to operate.

The display device 1 analyzes the distance to the user U and the captured image, and then determines the length of the arm of the current user U. If the length of the user U's arm exceeds a preset range, for example, 60 cm, the display device 1 may arrange the size of the plurality of UI elements 11 to 16 to have a small and wide spacing. have.

However, as shown in FIG. 12, when the user U'has a small movable range, the display device 1 may arrange a plurality of UI elements 11 to 16 to have a narrower interval compared to FIG. 11.

Through this, since the display device 1 does not require the user's movement to an area outside the range of motion of the hand, it is possible to reduce the difficulty of operation and the degree of fatigue.

13 and 14 are diagrams for explaining a method of displaying a UI element that is changed according to a viewing angle. It will be described together below in order to avoid redundant description.

Referring to FIG. 13, the database 170 stores information on the viewing angle of the user U. According to the ergonomic characteristics, the display device 1 sets a person's viewing angle to 30°. The display device 1 adjusts the size of the displayed plurality of UI elements 11 to 16 in consideration of the viewing angle according to the distance from the user U. Through this, when the user makes a gesture, it is possible to select and change the UI element 10 with only a minimum of motion.

Meanwhile, information on the viewing angle stored in the database 170 may be changed by a user's input, and is not necessarily limited to 30°.

Referring to FIG. 14, it can be seen that the display device 1 photographs the user U, and the user U turns his head and looks at the upper right corner. In addition, the display device 1 may detect that the user U is located diagonally instead of in front. In this case, the display device 1 may display the plurality of UI elements 11 to 16 at the top of the display instead of at the center.

As described above, the display device 1 may display a plurality of UI elements 11 to 16 according to the user's position, while considering the viewing angle and the length of the arm together. That is, when the user looks at the display device 1 and is located close to the display device 1 at the same time as shown in FIG. 14, the display device 1 is The positions and sizes of the elements 11 to 16 can be changed and displayed. Through this, the user (U) can perform the minimum operation to change the UI element, thereby reducing the degree of difficulty and fatigue.

15 to 17 are diagrams of an embodiment of activating a UI element. It will be described together below in order to avoid redundant description.

The display device 1 may detect a gesture of the user U and highlight the first UI element 11. By changing the size of the first UI element 11 as shown in FIG. 15, the display device 1 indicates that the area corresponding to the position of the current user U's hand corresponds to the position of the first UI element 11. Feedback.

If the user U holds his hand, the display device 1 activates the first UI element 11. That is, the emphasized first UI element 11 is only changed to replace the role of the cursor for the user's gesture, but does not mean that the first UI element 11 is selected.

When the user U is grasped through the sensor 130, the display device 1 activates the first UI element 11. For example, if the first UI element 11 is a user interface for controlling the sound and sound unit 152, the display device 1 is switched to a user interface screen that displays sound output as numbers.

As shown in FIG. 17, the display device 1 may detect a gesture of moving a hand held by the user U. The display apparatus 1 may feed back the user U that the degree of sound output is changed by changing the number displayed on the top of the icon 11a based on the movement direction of the hand of the hand.

Meanwhile, FIGS. 15 to 17 are only examples of activating the UI element, and the UI element may be activated by various gestures. In addition, the user interface displayed by the activated UI element may also vary.

18 is a flowchart illustrating a method of controlling a display device according to an exemplary embodiment.

Referring to FIG. 18, the display device 1 detects a user (200).

The display device 1 may detect a user through the sensor 130, and the sensor 130 according to an exemplary embodiment may use the camera 131.

The display device 1 displays the UI element 10 (210).

The displayed UI element 10 may be at least one or more, and is a user interface in which various functions provided by the display device 1 are formed.

The display device 1 detects a user's gesture (220) and highlights the UI element 10 (230).

Specifically, the display device 1 emphasizes the UI element 10 by changing at least one of a size, a position, and a color. For another example, when the user's gesture is detected after the UI element 10 is displayed, the UI element 100 may change the size and shape of the icon or change the rotation angle of a plurality of layers.

The display device 1 detects an additional gesture of the user (240) and changes the highlighted UI element 10 (240).

The display device 1 may move the UI element 10 based on the movement of the gesture.

For example, the UI element 10 may include a plurality of layers 22 to 24 and an icon 21. In addition, the UI element 10 may include a preset boundary 10c. When the virtual cursor 5 corresponding to the user's gesture moves near the border 10c of the UI element 10 that is highlighted, the position of the icon 21 of the UI element 10 can be moved near the border.

If the virtual cursor 5 corresponding to the user's gesture deviates from the boundary 10c of the highlighted UI element 10, the emphasis of the UI element 10 may be released and moved to the original position.

19 is a flowchart illustrating a method of controlling a display according to another exemplary embodiment.

Referring to FIG. 19, the display device 1 emphasizes the first UI element 11 (300 ).

The display device 1 may display a plurality of UI elements 11 to 16. When detecting the user's gesture, the display device 1 highlights the first UI element 11 at a position closest to the position corresponding to the gesture.

The display device 1 detects an additional gesture of the user (310) and moves the highlighted first UI element 11 (320).

As described above with reference to FIG. 10, the additional gesture may be a movement having a user's direction.

The center point 10a of the moving first UI element 11 does not deviate from the boundary 10c. However, the virtual cursor 5 corresponding to the user's gesture may enter the boundary 12c of the second UI element while leaving the boundary 10c of the first UI element (330).

The display device 1 moves the second UI element 12 (340) and changes the colors of the first UI element 11 and the moved second UI element 12 (350).

Specifically, the display device 1 may display a color of only a certain area of the first UI element 11 in a dark color, and may display a color of only a certain area of the second UI element 12 in a dark color.

The virtual cursor 5 corresponding to the user's gesture may enter the boundary of the second UI element 12 (360).

The meaning of entering the boundary is that the center point 12a of the second UI element 12 moved in correspondence with the position of the virtual cursor 5 is located inside the boundary 12c of the second UI element.

The display device 1 releases and moves the emphasis of the first UI element 11 (370 ), and highlights the moved second UI element 12 (380 ).

Through this, the display apparatus 1 omits the result of recognizing the user's gesture and provides feedback through a change in the user interface, thereby reducing the user's concentration and reducing fatigue. In addition, the display device 1 provides a quick coordinate recognition and a simple provision method by providing ease of selection of a user interface. The display device 1 may provide convenience by changing the size and position of the user interface according to the user's distance and the viewing angle and the length of the arm.

Claims (20)

A sensor that detects a user's gesture; A display for displaying a UI (User Interface) element; And The first UI element is emphasized based on the user's gesture, and the highlighted first UI element is moved based on the movement direction of the gesture, or a second UI element corresponding to the movement direction is converted to the first UI element. A display device comprising a; a processor that controls the display to move and display. The method of claim 1, The processor, A display device configured to set a preset boundary on the first UI element and the second UI element, respectively, and to move the highlighted first UI element based on the user's gesture and the preset boundary. The method of claim 2, The processor, When the movement of the gesture deviates from a preset boundary of the first UI element, the display device cancels the emphasis of the first UI element and returns the moved first UI element. The method of claim 3, The processor, A display device for emphasizing the second UI element when the gesture movement deviates from a preset boundary of the first UI element and enters a preset boundary of the second UI element. The method of claim 3, The processor, When the movement of the gesture deviates from the preset boundary of the first UI element and enters the preset boundary of the second UI element, the color of the highlighted first UI element and the color of the second UI element are sequentially changed. Display device. The method of claim 1, The processor, A display device for emphasizing the first UI element based on the position of the user's gesture and a center point of the plurality of UI elements. The method of claim 1, The first UI element includes the plurality of layers and icons, The processor, A display device that emphasizes the first UI element based on the rotation or movement of the plurality of layers or the first UI element based on the size or shape of the icon. The method of claim 7, The processor, A display device configured to differently change a rotation angle of a first layer and a rotation angle of a second layer among the plurality of layers based on the user's gesture. The method of claim 1, The processor, A display device configured to determine a position at which the first UI element and the second UI element are displayed based on at least one of a length of the user's arm, a viewing angle of the user, or a distance between the user and the display device. The method of claim 9, The processor, A display device configured to determine a size and a position at which the first UI element and the second UI element are displayed based on the length of the user's arm. The method of claim 9, The processor, A display device configured to determine a location and a size at which the first UI element and the second UI element are displayed based on a distance between the user and the display device and a viewing angle of the user. The method of claim 1, The processor, A display device that activates the highlighted first UI element based on the user's gripping motion. The method of claim 12, The processor, A display device configured to adjust a setting value of the activated first UI element when a movement of a hand is sensed while the user is grasping. The method of claim 1, The processor, A display device that highlights the first UI element based on at least one of a line, figure, color, or size of the first UI element. A sensor that detects a user's gesture; A display for displaying a UI (User Interface) element; And A processor that highlights the first UI element based on the user's gesture and moves a second UI element to the highlighted first UI element based on a movement direction of the gesture. Detect a user's gesture; Display a UI (User Interface) element; Highlighting the first UI element based on the user's gesture; And Controlling a display such that the highlighted first UI element moves or a second UI element corresponding to the movement direction moves to the first UI element based on the movement direction of the gesture; Way. The method of claim 16, To control the above, Controlling a display device comprising: setting a preset boundary to each of the first UI element and the second UI element, and moving the highlighted first UI element based on the user's gesture and the preset boundary. Way. The method of claim 17, To control the above, And when the movement of the gesture deviates from a preset boundary of the first UI element, releasing the emphasis of the first UI element and returning the moved first UI element. The method of claim 18, To control the above, And when the movement of the gesture deviates from a preset boundary of the first UI element and enters a preset boundary of the second UI element, highlighting the second UI element. The method of claim 18, To control the above, When the movement of the gesture deviates from the preset boundary of the first UI element and enters the preset boundary of the second UI element, the color of the highlighted first UI element and the color of the second UI element are sequentially changed. A method for controlling a display device comprising a.

Images