KR102758246B1 - Electronic device for controlling function associated to stylus pen by air gesture, method for operating thereof and storage medium - Google Patents

Abstract

An electronic device includes a display device including a sensing panel, at least one sensor, at least one processor operatively connected with the display device and the at least one sensor, and a memory operatively connected with the at least one processor, wherein the memory may store instructions configured to, when executed, cause the at least one processor to, when a first input signal corresponding to an air gesture is detected in a proximity area of the electronic device by the at least one sensor and a second input signal corresponding to an input by the stylus pen is not detected through the sensing panel, control a first function related to a screen display in response to the first input signal, and when the first input signal is detected in a state where the second input signal is detected, control a second function related to the stylus pen in response to the first input signal. Other embodiments are possible.

Description

{ELECTRONIC DEVICE FOR CONTROLLING FUNCTION ASSOCIATED TO STYLUS PEN BY AIR GESTURE, METHOD FOR OPERATING THEREOF AND STORAGE MEDIUM}

Various embodiments relate to an electronic device, a method of operating the same, and a storage medium for controlling functions associated with a stylus pen by air gestures.

Electronic devices are being developed to receive various inputs from users through a specified input device (e.g., a stylus pen) connected to the electronic device via wireless communication. The electronic device can identify a location on the electronic device specified by an input device having a pen function (hereinafter, for convenience of explanation, may be referred to as a “stylus pen”) and perform a corresponding function.

The electronic device can detect a magnetic field generated from a stylus pen using an electromagnetic resonance (EMR) method. The electronic device can identify the position of the stylus pen based on the induced electromotive force generated by the magnetic field for each channel.

The stylus pen can be connected to an electronic device via short-range communication (e.g., Bluetooth low energy (BLE)). The stylus pen can transmit information about the pressed state of a button disposed on the housing of the stylus pen to the electronic device via short-range communication, and the electronic device can perform a designated action based on the received information.

By using the stylus pen, the user can input handwriting or draw pictures on the screen of the electronic device as if writing a note. In addition, the note application allows the user to not only draw pictures, but also output the drawing marks by changing various pen drawing options such as the thickness of the pen, brush, and color. In this way, the user can conveniently and intuitively provide input to the electronic device using the stylus pen.

However, in the case of applications that support various functions based on a stylus pen, effective control may be difficult because the user must select and set the various functions one by one. For example, when a note application is running, in order for the user to set pen drawing options, the user may only be able to access the lower paths through a sequential path, such as first selecting a palette, then selecting a brush, and then selecting a thickness. In addition, since a palm reject function is applied to prevent touch malfunction, it may be difficult to apply multi-touch interaction.

In addition, in order to repeat a specific action, the user may have to repeat the operation using the stylus pen. For example, the user may have to frequently and repeatedly perform the action of setting the pen drawing option and adjusting the detailed setting value, which may cause inconvenience to the user. Therefore, a method may be required that can control various functions based on the stylus pen quickly and easily without interruption.

According to one embodiment, an electronic device includes a display device including a sensing panel, at least one sensor, at least one processor operatively connected with the display device and the at least one sensor, and a memory operatively connected with the at least one processor, wherein the memory stores instructions configured to, when executed, cause the at least one processor to: if a first input signal corresponding to an air gesture is detected in a proximity area of the electronic device by the at least one sensor and a second input signal corresponding to an input by the stylus pen is not detected through the sensing panel, control a first function related to a screen display in response to the first input signal; and if the first input signal is detected in a state where the second input signal is detected, control a second function related to the stylus pen in response to the first input signal.

According to one embodiment, a method for controlling a function associated with a stylus pen by an air gesture in an electronic device may include an operation of identifying whether at least one of a first input signal corresponding to the air gesture and a second input signal corresponding to an input by the stylus pen is detected in a proximity area of the electronic device by at least one sensor, an operation of controlling a first function associated with a screen display in response to detection of the first input signal in a state in which the second input signal is not detected, and an operation of controlling a second function associated with the stylus pen in response to detection of the first input signal in a state in which the second input signal is detected.

According to one embodiment, a storage medium storing commands may include: a processor configured to cause the processor to perform at least one operation when the commands are executed by at least one processor, wherein the at least one operation may include: an operation of identifying whether at least one of a first input signal corresponding to an air gesture and a second input signal corresponding to an input by a stylus pen is detected in a proximity area of the electronic device by at least one sensor; an operation of controlling a first function related to a screen display in response to detection of the first input signal in a state in which the second input signal is not detected; and an operation of controlling a second function related to the stylus pen in response to detection of the first input signal in a state in which the second input signal is detected.

According to one embodiment, when an application supporting various functions based on a stylus pen is running, functions associated with the stylus pen can be conveniently and intuitively used through air gestures.

According to one embodiment, functions associated with a stylus pen can be easily controlled through air gestures of the hand opposite to the hand holding the stylus pen, thereby increasing work efficiency through two-handed operation.

According to one embodiment, when drawing using a stylus pen, by using air gestures in a proximity area of an electronic device rather than an execution screen, not only is uninterrupted drawing possible with a hand holding the stylus pen, but drawing concentration can also be improved.

According to one embodiment, since the control of the application execution screen or the change of properties related to the stylus pen can be performed in response to air gestures rather than a fixed pop-up window, for example, a fixed palette panel, not only can the user obtain immediate feedback, but also can be efficient in terms of space utilization by not covering at least a part of the execution screen.

FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.
FIG. 2 is a perspective view of an electronic device including a stylus pen according to one embodiment.
FIG. 3a is a block diagram illustrating a stylus pen according to one embodiment.
FIG. 3b is an exploded perspective view of a stylus pen according to one embodiment.
FIG. 4 is an internal block diagram of an electronic device for processing air gestures associated with a stylus pen according to one embodiment.
FIG. 5 is an example screen diagram for explaining control of an application execution screen using air gestures according to one embodiment.
FIG. 6 is an example screen diagram for explaining changes in stylus pen properties when hovering using air gestures according to one embodiment.
FIG. 7 is a flowchart illustrating an operation of an electronic device for processing air gestures associated with a stylus pen according to one embodiment.
Figure 8 is a table illustrating types of air gestures according to one embodiment.
FIG. 9a is an example of an application execution screen before activating air gesture mode according to one embodiment.
FIG. 9b is an example screen diagram for explaining changes in the tool panel when the air gesture mode is activated according to one embodiment.
FIG. 9c is an example screen diagram for explaining changes in the tool panel when activation of the air gesture mode according to one embodiment is completed.
FIG. 9d is an example screen diagram for explaining the hidden state of the tool panel when activation of the air gesture mode is completed according to one embodiment.
FIG. 10A is an example screen diagram providing status information upon entry into air gesture mode according to one embodiment.
FIG. 10b is an example screen diagram providing status information according to an air gesture input according to one embodiment.
FIG. 10c is an example screen diagram providing status information according to non-detection of an air gesture according to one embodiment.
FIG. 11A is an example screen diagram for explaining a pen property change operation based on an air gesture according to one embodiment.
FIG. 11b is an example screen diagram providing an icon panel according to a change in pen properties according to one embodiment.
FIG. 11c is an example screen diagram showing a hidden state of an icon panel according to a change in pen properties according to one embodiment.
FIG. 12 is an example screen diagram for explaining the display operation of an icon panel according to hovering according to one embodiment.
Figure 13a is an example screen diagram for right-hand mode according to one embodiment.
Figure 13b is an example screen for left-hand mode according to one embodiment.
FIG. 14 is an example screen diagram for explaining an operation using an air gesture when removing a stylus pen according to one embodiment.
FIG. 15 is an example screen diagram for explaining the operation of a translator using air gestures according to one embodiment.

The terms used in this document are only used to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly indicates otherwise. All terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person having ordinary skill in the art of the present invention. Terms defined in commonly used dictionaries may be interpreted as having the same or similar meaning in the context of the related art, and shall not be interpreted in an ideal or excessively formal meaning unless explicitly defined in this document. In some cases, even if a term is defined in this document, it cannot be interpreted to exclude embodiments of the present invention.

FIG. 1 is a block diagram of an electronic device (101) in a network environment (100a) according to various embodiments. Referring to FIG. 1, in the network environment (100a), the electronic device (101) may communicate with the electronic device (102) through a first network (198) (e.g., a short-range wireless communication network), or may communicate with the electronic device (104) or the server (108) through a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) through the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input device (150), an audio output device (155), a display device (160), an audio module (170), a sensor module (176), an interface (177), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the display device (160) or the camera module (180)), or may include one or more other components. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, a sensor module (176) (e.g., a fingerprint sensor, an iris sensor, or a light sensor) may be implemented embedded in a display device (160) (e.g., a display).

The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may load a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) into a volatile memory (132), process the command or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor), and a secondary processor (123) (e.g., a graphic processing unit, an image signal processor, a sensor hub processor, or a communication processor) that may operate independently or together therewith. Additionally or alternatively, the auxiliary processor (123) may be configured to use less power than the main processor (121), or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121), or as a part thereof.

The auxiliary processor (123) may control at least a portion of functions or states associated with at least one of the components of the electronic device (101) (e.g., the display device (160), the sensor module (176), or the communication module (190)), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. According to one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)).

The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or nonvolatile memory (134).

The program (140) may be stored as software in memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

The input device (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input device (150) can include, for example, a microphone, a mouse, a keyboard, or a digital pen (e.g., a stylus pen).

The audio output device (155) can output an audio signal to the outside of the electronic device (101). The audio output device (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive an incoming call. According to one embodiment, the receiver can be implemented separately from the speaker or as a part thereof.

The display device (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display device (160) can include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display device (160) can include touch circuitry configured to detect a touch, or a sensor circuitry configured to measure a strength of a force generated by the touch (e.g., a pressure sensor).

The audio module (170) can convert sound into an electric signal, or vice versa, convert an electric signal into sound. According to one embodiment, the audio module (170) can obtain sound through an input device (150), or output sound through an audio output device (155), or an external electronic device (e.g., an electronic device (102) (e.g., a speaker or headphone)) directly or wirelessly connected to the electronic device (101).

The sensor module (176) can detect an operating state (e.g., power or temperature) of the electronic device (101) or an external environmental state (e.g., user state) and generate an electric signal or data value corresponding to the detected state. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). According to one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

The haptic module (179) can convert an electrical signal into a mechanical stimulus (e.g., vibration or movement) or an electrical stimulus that a user can perceive through a tactile or kinesthetic sense. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) can include one or more lenses, image sensors, image signal processors, or flashes.

The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery (189) can power at least one component of the electronic device (101). In one embodiment, the battery (189) can include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module (190) may support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., the electronic device (102), the electronic device (104), or the server (108)), and performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., the application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). A corresponding communication module among these communication modules can communicate with an external electronic device via a first network (198) (e.g., a short-range communication network such as Bluetooth, WiFi direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a cellular network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can identify and authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module can include one antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) can include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), can be selected from the plurality of antennas by, for example, the communication module (190). A signal or power can be transmitted or received between the communication module (190) and the external electronic device via the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., an RFIC) can be additionally formed as a part of the antenna module (197).

At least some of the above components may be connected to each other and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

According to one embodiment, a command or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the electronic devices (102, 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform at least a part of the function or service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

FIG. 2 is a perspective view of an electronic device (101) including a stylus pen (201) (e.g., the electronic device (102) of FIG. 1) according to one embodiment. According to one embodiment, the stylus pen (201) may correspond to the input device (150) of FIG. 1 rather than the electronic device (102) of FIG. 1.

Referring to FIG. 2, an electronic device (101) of one embodiment may include the configuration illustrated in FIG. 1, and may include a structure into which a stylus pen (201) may be inserted. The electronic device (101) includes a housing (210), and a portion of the housing (210), for example, a portion of a side surface (210a), may include a hole (211). The electronic device (101) may include a first internal space (212), which is a storage space connected to the hole (211), and the stylus pen (201) may be inserted into the first internal space (212). According to the illustrated embodiment, the stylus pen (201) may include a first button (201a) that can be pressed at one end to facilitate taking the stylus pen (201) out of the first internal space (212) of the electronic device (101). When the first button (201a) is pressed, a rebound mechanism (e.g., a rebound mechanism by at least one elastic member (e.g., a spring)) connected to the first button (201a) is operated, so that the stylus pen (201) can be detached from the first internal space (212).

FIG. 3a is a block diagram illustrating a stylus pen (e.g., the stylus pen (201) of FIG. 2) according to one embodiment.

Referring to FIG. 3A, a stylus pen (201) according to one embodiment may include a processor (220), a memory (230), a resonant circuit (287), a charging circuit (288), a battery (289), a communication circuit (290), an antenna (297), a trigger circuit (298), and/or a sensor (299). In some embodiments, the processor (220), at least a portion of the resonant circuit (287), and/or at least a portion of the communication circuit (290) of the stylus pen (201) may be configured on a printed circuit board or in the form of a chip. The processor (220), the resonant circuit (287), and/or the communication circuit (290) may be electrically connected to the memory (230), the charging circuit (288), the battery (289), the antenna (297), the trigger circuit (298), and/or the sensor (299).

A processor (220) according to an embodiment may include a generic processor configured to execute a customized hardware module or software (e.g., an application program). The processor (220) may include a hardware component (function) or a software element (program) including at least one of various sensors provided in the stylus pen (201), a data measurement module, an input/output interface, a module for managing the status or environment of the stylus pen (201), or a communication module. The processor (220) may include, for example, one or a combination of two or more of hardware, software, or firmware. According to one embodiment, the processor (220) may be configured to transmit information indicating a pressed state of a button (e.g., button (337)), sensing information acquired by a sensor (299), and/or information calculated based on the sensing information (e.g., information associated with a position of a stylus pen (201)) to the electronic device (101) via the communication circuit (290).

A resonant circuit (287) according to an embodiment of the present invention may resonate based on an electromagnetic field signal generated from a digitizer (e.g., a display device (160)) of an electronic device (101) and may radiate an electro-magnetic resonance (EMR) input signal (or a magnetic field) by resonance. The electronic device (101) may identify a position of a stylus pen (201) on the electronic device (101) using the electromagnetic resonance input signal. For example, the electronic device (101) may identify the position of the stylus pen (201) based on the magnitude of an induced electromotive force (e.g., an output current) generated by the electromagnetic resonance input signal in each of a plurality of channels (e.g., a plurality of loop coils) in the digitizer. Meanwhile, in the above, the electronic device (101) and the stylus pen (201) have been described as operating based on the EMR method, but this is merely an example, and the electronic device (101) may also generate a signal based on an electric field based on an ECR (electrically coupled resonance) method. The resonance circuit of the stylus pen (201) may be resonated by an electric field. The electronic device (101) may check the potential in a plurality of channels (e.g., electrodes) due to resonance in the stylus pen (201), and may also check the position of the stylus pen (201) based on the potential. The stylus pen (201) may be implemented using an AES (active electrostatic) method, and those skilled in the art will understand that there is no limitation on the type of implementation. Additionally, the electronic device (101) may detect the stylus pen (201) based on a change in capacitance (self capacitance or mutual capacitance) associated with at least one electrode of the touch panel. In this case, the stylus pen (201) may not include a resonant circuit. In the present disclosure, the “panel” or “sensing panel” may be used as a term encompassing a digitizer and a TSP (touch screen panel).

The memory (230) according to one embodiment may store information related to the operation of the stylus pen (201). For example, the information may include information for communication with the electronic device (101) and frequency information related to the input operation of the stylus pen (201). In addition, the memory (230) may store a program (or an application, an algorithm, or a processing loop) for calculating information (e.g., coordinate information and/or displacement information) about the position of the stylus pen (201) from the sensing data of the sensor (299). The memory (230) may also store a communication stack of the communication circuit (290). Depending on the implementation, the communication circuit (290) and/or the processor (220) may include a dedicated memory.

A resonant circuit (287) according to an embodiment may include a coil (or an inductor) and/or a capacitor. The resonant circuit (287) may resonate based on an input electric field and/or magnetic field (e.g., an electric field and/or a magnetic field generated from a digitizer of the electronic device (101). When the stylus pen (201) transmits a signal by the EMR method, the stylus pen (201) may generate a signal including a resonant frequency based on an electromagnetic field generated from an inductive panel of the electronic device (101). When the stylus pen (201) transmits a signal by the AES method, the stylus pen (201) may generate a signal by using capacitive coupling with the electronic device (101). When the stylus pen (201) transmits a signal by the ECR method, the stylus pen (201) can generate a signal including a resonant frequency based on an electric field generated from a capacitive device of an electronic device. According to one embodiment, the resonant circuit (287) can be used to change the intensity or frequency of an electromagnetic field according to a user's manipulation state. For example, the resonant circuit (287) can provide various frequencies for recognizing a hovering input, a drawing input, a button input, or an erasing input. For example, the resonant circuit (287) can provide various resonant frequencies according to a connection combination of a plurality of capacitors, or can provide various resonant frequencies based on a variable inductor and/or a variable capacitor.

According to one embodiment, the charging circuit (288) may, when connected to the resonant circuit (287) based on the switching circuit, rectify the resonant signal generated from the resonant circuit (287) into a DC signal and provide it to the battery (289). According to one embodiment, the stylus pen (201) may use the voltage level of the DC signal detected by the charging circuit (288) to determine whether the stylus pen (201) is inserted into the electronic device (101). Alternatively, the stylus pen (201) may check a pattern corresponding to a signal detected by the charging circuit (288) to determine whether the stylus pen (201) is inserted.

A battery (289) according to one embodiment may be configured to store power required for the operation of the stylus pen (201). The battery (289) may include, for example, a lithium-ion battery or a capacitor, and may be rechargeable or replaceable. According to one embodiment, the battery (289) may be charged using power (for example, a direct current signal (direct current power)) provided from a charging circuit (288).

A communication circuit (290) according to an embodiment may be configured to perform a wireless communication function between a stylus pen (201) and a communication module (190) of an electronic device (101). According to an embodiment, the communication circuit (290) may transmit information related to status information, input information, and/or location of the stylus pen (201) to the electronic device (101) using a short-range communication method. For example, the communication circuit (290) may transmit direction information (e.g., motion sensor data) of the stylus pen (201) obtained through a trigger circuit (298), voice information input through a microphone, or remaining amount information of a battery (289) to the electronic device (101). For example, the communication circuit (290) may transmit sensing data obtained from a sensor (299) and/or information related to a location of the stylus pen (201) confirmed based on the sensing data to the electronic device (101). For example, the communication circuit (290) can transmit information about the state of a button (e.g., button (337)) provided in the stylus pen (201) to the electronic device (101). As an example, the short-range communication method may include at least one of Bluetooth, BLE (Bluetooth low energy), NFC, and Wi-Fi direct, but there is no limitation on the type.

An antenna (297) according to an embodiment may be used to transmit a signal or power to an external device (e.g., the electronic device (101)) or receive a signal or power from an external device. According to an embodiment, the stylus pen (201) may include a plurality of antennas (297), and among them, at least one antenna (297) suitable for a communication method may be selected. Through the selected at least one antenna (297), the communication circuit (290) may exchange a signal or power with an external electronic device.

A trigger circuit (298) according to an embodiment may include at least one button or sensor circuit. According to an embodiment, the processor (220) may check an input method (e.g., touch or press) or a type (e.g., EMR button or BLE button) of a button of the stylus pen (201). According to an embodiment, the trigger circuit (298) may transmit a trigger signal to the electronic device (101) using an input signal of the button or a signal through the sensor (299).

The sensor (299) according to one embodiment may include an accelerometer, a gyro sensor, and/or a geomagnetic sensor. The acceleration sensor may sense information about the linear movement of the stylus pen (201) and/or acceleration of the stylus pen (201) about three axes. The gyro sensor may sense information related to the rotation of the stylus pen (201). The geomagnetic sensor may sense information about the direction in which the stylus pen (201) is facing within the absolute coordinate system. According to one embodiment, the sensor (299) may include at least one of a sensor for measuring movement, as well as a sensor for generating an electric signal or data value corresponding to an internal operating state or an external environmental state of the stylus pen (201), for example, a battery level detection sensor, a pressure sensor, a light sensor, a temperature sensor, and a biometric sensor. According to various embodiments, the processor (220) may transmit information acquired from the sensor (299) to the electronic device (101) through the communication circuit (290). Alternatively, the processor (220) may transmit information related to the position of the stylus pen (201) (e.g., coordinates of the stylus pen (201) and/or displacement of the stylus pen (201)) to the electronic device (101) through the communication circuit (290) based on the information acquired from the sensor (299).

FIG. 3b is an exploded perspective view of a stylus pen (e.g., the stylus pen (201) of FIG. 2) according to one embodiment.

Referring to FIG. 3b, the stylus pen (201) may include a pen housing (300) that constitutes the outer shape of the stylus pen (201) and an inner assembly inside the pen housing (300). In the illustrated embodiment, the inner assembly may be inserted into the pen housing (300) in a single assembly operation while several components mounted inside the stylus pen (201) are combined together.

The pen housing (300) may have a shape that is elongated between the first end (300a) and the second end (300b), and may include a second internal space (301) therein. The pen housing (300) may have an elliptical cross-section formed by a long axis and a short axis, and may be formed in an elliptical column shape overall. The first internal space (212) of the electronic device (101) described above in FIG. 2 may also have an elliptical cross-section formed corresponding to the shape of the pen housing (300). According to one embodiment, the pen housing (300) may include at least a portion of a synthetic resin (e.g., plastic) and/or a metallic material (e.g., aluminum). According to one embodiment, the first end (300a) of the pen housing (300) may be composed of a synthetic resin material. Various other embodiments may be applied to the material of the pen housing (300).

The inner assembly may have an elongated shape corresponding to the shape of the pen housing (300). The inner assembly may be largely divided into three configurations along the length direction. For example, the inner assembly may include a coil portion (310) positioned corresponding to the first end (300a) of the pen housing (300), an ejection member (320) positioned corresponding to the second end (300b) of the pen housing (300), and a circuit board portion (330) positioned corresponding to the body of the pen housing (300).

The coil portion (310) may include a pen tip (311) exposed to the outside of the first end (300a) when the internal assembly is fully inserted into the pen housing (300), a packing ring (312), a coil (313) wound multiple times, and/or a pressure sensing portion (314) for obtaining a change in pressure according to the pressurization of the pen tip (311). The packing ring (312) may include epoxy, rubber, urethane, or silicone. The packing ring (312) may be provided for the purpose of waterproofing and dustproofing, and may protect the coil portion (310) and the circuit board portion (330) from submersion or dust. According to one embodiment, the coil (313) can form a resonant frequency in a set frequency band (e.g., 500 kHz), and can be combined with at least one element (e.g., a capacitive element) to adjust the resonant frequency formed by the coil (313) within a certain range.

The ejection member (320) may include a configuration for ejecting the stylus pen (201) from the first internal space (212) of the electronic device (e.g., 101 of FIG. 2). According to one embodiment, the ejection member (320) may include a shaft (321) and an ejection body (322) and a button portion (323) (e.g., the first button (201a) of FIG. 2) that are arranged around the shaft (321) and form the overall outer shape of the ejection member (320). When the internal assembly is completely inserted into the pen housing (300), a portion including the shaft (321) and the ejection body (322) is surrounded by the second end (300b) of the pen housing (300), and at least a portion of the button portion (323) may be exposed to the outside of the second end (300b). A plurality of unillustrated components, for example, cam members or elastic members, may be arranged within the ejection body (322) to form a push-pull structure. In one embodiment, the button portion (323) may be substantially coupled with the shaft (321) to perform a linear reciprocating motion with respect to the ejection body (322). According to various embodiments, the button portion (323) may include a button having a catch structure formed so that a user can pull out the stylus pen (201) using a fingernail. According to one embodiment, the stylus pen (201) may include a sensor that detects the linear reciprocating motion of the shaft (321), thereby providing another input method.

The circuit board portion (330) may include a printed circuit board (332), a base (331) surrounding at least one surface of the printed circuit board (332), and an antenna. According to one embodiment, a substrate mounting portion (333) on which the printed circuit board (332) is placed is formed on an upper surface of the base (331), and the printed circuit board (332) may be fixed in a state of being mounted on the substrate mounting portion (333). According to one embodiment, the printed circuit board (332) may include a first surface and a second surface, and a variable capacity capacitor or switch (334) connected to a coil (313) may be placed on the first surface, and a charging circuit, a battery (336), or a communication circuit may be placed on the second surface. Here, the first side and the second side of the printed circuit board (332) may mean different laminated sides in a structure laminated above and below, according to one embodiment, and may mean different parts of the printed circuit board arranged along the length direction of the printed circuit board (332) according to another embodiment. The battery (336) may include an EDLC (electric double layered capacitor). The charging circuit is located between the coil (313) and the battery, and may include a voltage detector circuitry and a rectifier. Here, the battery (336) may not necessarily be arranged on the second side of the printed circuit board (332). The location of the battery (336) may be set in various ways according to various mounting structures of the circuit board (330), and may be arranged in a different location than that illustrated in the drawing.

The antenna may include an antenna structure (339) such as the example illustrated in FIG. 3B and/or an antenna embedded in a printed circuit board (332). According to various embodiments, a switch (334) may be provided on the printed circuit board (332). A second button (337) provided on the stylus pen (201) is used to press the switch (334) and may be exposed to the outside through a side opening (302) of the pen housing (300). The second button (337) is supported by a support member (338), and when there is no external force acting on the second button (337), the support member (338) provides an elastic restoring force to restore or maintain the second button (337) in a state where it is placed at a certain position. Meanwhile, the second button (337) can be implemented in any one of a physical key, a touch key, a motion key, a pressure key, and a key less method, and there is no limitation on the implementation form of the button.

The circuit board portion (330) may include, for example, a packing ring such as an O-ring. According to one embodiment, an O-ring made of an elastic body may be placed at both ends of the base (331) to form a sealing structure between the base (331) and the pen housing (300). In some embodiments, the support member (338) may partially form a sealing structure by closely contacting the inner wall of the pen housing (300) around the side opening (302). For example, the circuit board portion (330) may also include a waterproof and dustproof structure similar to the packing ring (312) of the coil portion (310) at least in one portion.

The stylus pen (201) may include a battery mounting portion (333a) in which a battery (336) is placed on the upper surface of the base (331). The battery (336) that may be mounted on the battery mounting portion (333a) may include, for example, a cylinder type battery.

The stylus pen (201) may include a microphone (not shown) and/or a speaker. The microphone and/or speaker may be directly connected to the printed circuit board (332) or may be connected to a separate flexible printed circuit board (FPCB) (not shown) connected to the printed circuit board (332). According to various embodiments, the microphone and/or speaker may be positioned parallel to the second button (337) in the longitudinal direction of the stylus pen (301).

FIG. 4 is an internal block diagram (400) of an electronic device for processing air gestures associated with a stylus pen according to one embodiment.

Referring to FIG. 4, a display device (460) (e.g., the display device (160) of FIG. 1) that detects an input of a stylus pen (201) (e.g., the stylus pen (201) of FIG. 2) may be provided to an electronic device (101) (e.g., the electronic device (101) of FIG. 1). The display device (460) may not only support input/output functions of data simultaneously, but may also detect a touch.

According to one embodiment, the display device (460) may include a sensing panel (461), a display panel (462), and a display control unit (463). In FIG. 4, the sensing panel (461) is described as being included in the display device (460), but it may also form a layer structure with the display panel (462) and operate as an input panel different from the display device (160).

The sensing panel (461) can detect the position of the touch input of the stylus pen (201), and the display panel (462) can output an image. The display control unit (463) can be a driving circuit that controls the display panel (462) to output an image through the display panel (462).

The sensing panel (461) may be configured as an input pad of the EMR method or the EMI (electro-magnetic interface) method using an electromagnetic sensor when the stylus pen (201) supports the EMR (electro-magnetic resonance) method. This is merely an example, and may also be configured as an input pad of the ECR (electrically coupled resonance) method.

The sensing panel (461) can receive a magnetic field from the stylus pen (201) and detect the position of the stylus pen (201) therefrom. The sensing panel (461) can be composed of one or more panels forming a mutual layer structure to detect input using a plurality of sensors.

In addition, the sensing panel (461) may be implemented as a touch screen panel (TSP), and if implemented as a touch screen panel, the position of the stylus pen (200) may be identified based on an output signal from an electrode. The stylus pen (201) may be implemented in an AES (active electrostatic) manner, and those skilled in the art will understand that there is no limitation on the type of implementation.

In addition, the sensing panel (461) can detect the contact or proximity of the stylus pen (201) or the user's finger. For example, the sensing panel (461) can detect a hovering input by the stylus pen (201). The sensing panel (461) can generate an input signal corresponding to the hovering input and transmit the same to the processor (120). The sensing panel (461) can generate an input signal including information on the hovering input based on the location where the hovering input occurs, the stylus pen (201), and the operating state by the button provided on the stylus pen (201).

In this way, hardware implemented as an EMR method or a touch screen panel capable of sensing a signal from a stylus pen may be referred to as a sensing panel (461). This sensing panel (461) may detect a touch input by a human body or an input by a stylus pen (201). The configuration of the display device (460) illustrated in FIG. 4 is only an example, and the type and number of panels constituting the display device (460) and the upper and lower positions of the panels may vary depending on the manufacturing technology of the electronic device (101).

The display device (460) may, for example, display a pop-up window for items related to stylus pen properties according to the execution of an application. Since the user must change the stylus pen properties only through the pop-up window, a significant portion of the application execution screen may be covered by the pop-up window each time. In addition, since the display device (460) applies a palm reject function to prevent touch malfunction, a new input method other than the existing touch input method may be required when using the stylus pen (201).

To this end, according to one embodiment, while using a stylus pen (201), an air gesture using the opposite hand rather than the hand holding the stylus pen (201) can be used as a new input method. According to one embodiment, by using an air gesture using the opposite hand, it is possible to control an application execution screen using the stylus pen (201) or change properties related to the stylus pen when executing an application. Therefore, from the user's perspective, frequent and repetitive tasks such as selecting options and adjusting detailed settings can be easily changed when using the stylus pen (201), thereby improving user accessibility.

In addition, when drawing using a stylus pen (201), by using air gestures in the vicinity of the electronic device (101) rather than the execution screen, not only is uninterrupted drawing possible with the hand holding the stylus pen (201) but also the concentration on drawing can be improved, and the productivity increase effect utilizing two-hand operation can be expected. In addition, since it is possible to control the application execution screen or change properties related to the stylus pen in response to air gestures rather than a fixed pop-up window, for example, a fixed palette panel, not only can the user obtain immediate feedback, but at least a part of the execution screen is not covered, so it can be efficient in space utilization.

According to one embodiment, the sensor (476) (e.g., the sensor module (176) of FIG. 1) may sense an air gesture by a user's hand. The air gesture operation may be an operation in which the user moves a hand or finger without touching the touch screen at a predetermined distance from the electronic device (101). According to one embodiment, an air gesture associated with a stylus pen may mean a movement for controlling a related function by the stylus pen (201) on the electronic device (101).

For example, the sensor (476) may include at least one sensor for sensing an air gesture by a hand other than the hand holding the stylus pen (201). If such a sensor (476) is used, it is possible to identify in which direction among the left and right directions with respect to the front center axis of the electronic device (101) an air gesture by the user's hand is detected when the rear side of the electronic device (101) is placed on the floor. In addition, since the sensor (476) can detect the pose of the hand holding the stylus pen (201) when the user is holding it, it is possible to distinguish whether the hand holding the stylus pen (201) is the right or left hand.

As described above, based on the sensing data by the sensor (476), the hand holding the stylus pen (201) and the hand not holding the stylus pen (201) can be distinguished. For example, if the user is holding and using the stylus pen (201) with his or her right hand, a right-hand graphic UI can be displayed on the display device (460) in response to an air gesture made by the left hand. For example, a graphic UI listing a tool set consisting of tools corresponding to the operation state of the stylus pen (201) can be moved to the right on the display device (460) so that a right-handed user can use it more easily. Conversely, if the user is holding and using the stylus pen (201) with his or her left hand, a left-hand graphic UI can be displayed on the display device (460) in response to an air gesture made by the right hand. Alternatively, for example, if a user is holding and using a stylus pen (201) with his/her left hand, the components of the graphic UI may be placed to the right or displayed and then disappeared on the right so that the user can utilize more space on the left side of the screen of the electronic device.

As an example of a sensor for detecting the air gesture as described above, according to an embodiment, the sensor (476) may be an RF spectrum type sensor. The sensor (476) may use the frequency of a 60 GHz WiFi wireless antenna. For example, the sensor may transmit an RF signal toward a target object, receive a reflected radio wave signal, and analyze the pattern of the RF signal that returns after the radiated RF signal is reflected by the target object. The user's hand gesture may be identified based on the pattern of the RF signal. For example, air gesture recognition based on an RF signal may use a frequency domain of 5 GHz to 60 GHz and have a field of view (FOV) of 160 to 180 degrees. In the case of an air gesture using such an RF signal, fine control of a related function by a stylus pen (201) may be possible.

According to one embodiment, the sensor (476) may be any one of an infrared sensor, an ultrasonic sensor, a motion sensor, a 3D sensor, and a vision sensor, and the type of the sensor is not limited thereto. Information sensed from at least two of these sensors may be combined and utilized to detect an air gesture made by a user's hand. For example, an air gesture made by a user's hand may be detected based on outputs from an ultrasonic sensor and a camera.

Sensing data from the sensor (476) may be transmitted to the processor (420), and the sensing data may be processed by the processor (420).

According to one embodiment, the processor (420) can identify an air gesture (e.g., an air gesture type) based on sensing data of the sensor (476), and perform an operation corresponding to the air gesture. According to one embodiment, the processor (420) can distinguish between the hand holding the stylus pen (201) and the other hand among the user's two hands based on sensing data by the sensor (476). According to one embodiment, the processor (420) can recognize whether the air gesture corresponds to any one of the other hand holding the stylus pen (201), a gesture using one finger of the other hand, a gesture using two fingers of the other hand, a gesture using multiple fingers, a fist gesture, and a palm-out gesture.

Air gestures can be used to call up a menu related to the stylus pen (201) or control a current execution screen with one hand while holding the stylus pen (201) with the other hand without interfering with the use of the stylus pen (201). According to one embodiment, when an air gesture is input by the other hand while holding the stylus pen (201), the processor (420) can transmit a function mapped to the air gesture to the application.

For example, the processor (420) may provide at least one icon panel (e.g., composed of icons that can change pen functions) related to pen properties (e.g., thickness and/or color) or change pen properties (e.g., thickness and/or color) based on an air gesture. Or, the processor (420) may perform a control operation on a currently running application screen based on an air gesture. Here, at least one icon panel is a graphic UI that lists a tool set utilized for a task using the stylus pen (201), and may be implemented to be temporarily displayed on the application execution screen and then disappear. For example, the tool set may include tools that can be used in an application that is running using the stylus pen (201). In addition to the above, at least one icon panel may include various pen functions related to the stylus pen (201), such as a translator and a smart select.

According to one embodiment, the processor (420) can map a function related to a stylus pen (201) corresponding to an air gesture, and can check and perform the mapped function.

According to one embodiment, the processor (420) may recognize the user's two hands within a proximity area of the electronic device (101), for example, a spatial range for recognizing an air gesture, based on the sensing data of the sensor (476), but may recognize the hand holding the stylus pen (201) and the other hand sequentially or simultaneously. Alternatively, the hand not holding the stylus pen (201) may be recognized first, and then the hand holding the stylus pen (201) may be recognized. Alternatively, after first recognizing which hand is holding the stylus pen (201), the processor (420) may perform a sensing operation to detect an air gesture in the opposite direction of the recognized hand.

According to one embodiment, in order to detect an air gesture, the processor (420) may activate a sensing function for recognizing an air gesture when the stylus pen (201) is removed from the storage space of the electronic device (101) or when a button of the stylus pen (201) is pressed upon detecting hovering by the stylus pen (201). Accordingly, according to one embodiment, the processor (420) may identify which hand among the two hands is to be used as an input interface based on sensing data of the sensor (476), for example, when the stylus pen (201) is removed from the storage space of the electronic device (101).

According to one embodiment, the process (420) may recognize an air gesture when detecting hovering by the stylus pen (201), but even when not in a hovering state, for example, when the stylus pen (201) is prepared while the user selects an icon for activating the air gesture mode displayed on the screen, that is, when the stylus pen (201) is prepared on the screen to be used even when there is no hovering, the air gesture may be recognized. In addition, when a user inputs a predetermined touch input for activating the air gesture mode, the operation of the air gesture may be recognized while the stylus pen (201) is prepared.

According to one embodiment, the process (420) can identify a direction corresponding to a recognized hand and detect air gestures only for the identified direction, thereby preventing the input interface from changing during application execution. For example, an air gesture by a hand holding a stylus pen (201) can be prevented from being recognized.

According to one embodiment, the memory (430) may store information related to the operation of the stylus pen (201). According to one embodiment, the memory (430) may also store a program (or an application, an algorithm, or a processing loop) for performing an operation corresponding to an air gesture. The memory (430) may store instructions for causing the electronic device (101) or the stylus pen (201) to perform a specific operation corresponding to an air gesture.

According to one embodiment, an electronic device (101) includes a display device (460) including a sensing panel (461), at least one sensor (476), at least one processor (420) operatively connected with the display device and the at least one sensor (476), and a memory (430) operatively connected with the at least one processor (420), wherein the memory (430) may store instructions configured to, when executed, cause the at least one processor (420) to, when a first input signal corresponding to an air gesture is detected in a proximity area of the electronic device (101) by the at least one sensor (476) and a second input signal corresponding to an input by the stylus pen is not detected through the sensing panel, control a first function related to a screen display in response to the first input signal, and when the first input signal is detected in a state where the second input signal is detected, control a second function related to the stylus pen in response to the first input signal. According to one embodiment, the first function related to the screen display may be a control function for a running application screen, and the second function related to the stylus pen may be a function related to a property of the stylus pen. For example, when a user makes an air gesture with the other hand while holding the stylus pen, the second input signal may not be detected but the first input signal may be detected, in which case the display of the current screen may be adjusted in response to the air gesture. For example, in response to the air gesture, controls such as screen zoom-in, screen zoom-out, upward scroll, downward scroll, and screen rotation may be performed, and functions that can be mapped in relation to screen control may not be limited thereto.

For example, when a user holds a stylus pen close to a screen or in a hovering state, a second input signal may be detected, and a first input signal corresponding to an air gesture may be detected simultaneously while the second input signal is being detected. In this case, a second function related to the stylus pen may be controlled in response to the air gesture. For example, a property of the stylus pen may be changed in response to the air gesture.

According to one embodiment, the instructions may be configured to cause the at least one processor (420) to identify whether at least one of the first input signal and the second input signal is detected after removal of the stylus pen from the electronic device is detected.

According to one embodiment, the instructions may be configured to control the at least one processor (420) to change a second function related to the stylus pen in response to the first input signal when the first input signal is detected in a state in which the second input signal is detected, and to display information about the changed second function on the display device for a specified period of time and then be removed. In addition, for example, if the second input signal is detected again within a predetermined period of time after the second input signal is detected, an item related to the currently changed stylus pen attribute may be displayed on the display device (460) for a specified period of time and then removed.

According to one embodiment, the instructions may be configured to cause the at least one processor (420) to, when removal of the stylus pen (201) from the electronic device (101) is detected, display a plurality of executable applications related to the stylus pen (201), and change a function of one of the plurality of applications in response to the first input signal.

According to one embodiment, the instructions may be configured to cause the at least one processor (420) to change a second function associated with the stylus pen in response to the first input signal when the first input signal is detected in a state in which a third input signal indicating that a button (e.g., button (337)) of the stylus pen (420) is pressed is detected.

In one embodiment, the instructions may be configured to cause the at least one processor (420) to change a second function associated with the stylus pen in real time based on the first input signal after a third input signal indicating that the button of the stylus pen is pressed is detected and before a fourth input signal indicating that the button of the stylus pen is released is detected.

According to one embodiment, the instructions may be configured to cause the at least one processor (420) to display an icon indicating an air gesture activation state on an application execution screen displayed on the display device (460) when the first input signal is detected after removal of the stylus pen from the electronic device is detected.

According to one embodiment, the instructions may be configured to cause the at least one processor (420) to display an icon indicating an activated state of the air gesture, and then, if the first input signal is not detected for a predetermined period of time, display an icon indicating a deactivated state of the air gesture on an application execution screen displayed on the display device (460).

According to one embodiment, the instructions may be configured such that, when the at least one processor (420) detects the first input signal in an area corresponding to a left direction of the electronic device (101), rearrange at least one item for the second function related to the stylus pen on the display device (460) of the electronic device (101) to correspond to a right-hand mode, and when the at least one processor (420) detects the first input signal in an area corresponding to a right direction of the electronic device (101), rearrange at least one item for the second function related to the stylus pen on the display device (460) of the electronic device (101) to correspond to a left-hand mode.

According to one embodiment, the at least one sensor (476) may include at least one of an RF spectrum type sensor, an infrared sensor, an ultrasonic sensor, a motion sensor, a 3D sensor, and a vision sensor capable of detecting a first input signal corresponding to the air gesture in a proximity area of the electronic device (101).

FIG. 5 is an example screen diagram (500a, 500b) for explaining control of an application execution screen using air gestures according to one embodiment.

As illustrated in 500a of FIG. 5, depending on which direction the air gesture is detected in, for example, in the proximity area (515) in the first direction (e.g., the left side of the electronic device (101)) or in the proximity area (520) in the second direction (e.g., the right side of the electronic device (101)), within the spatial range for recognizing the air gesture of the electronic device (101), the electronic device (101) can determine whether the hand to be used as the input interface corresponding to the air gesture is the right hand or the left hand. For example, as illustrated in 500a of FIG. 5, if the air gesture is detected in the proximity area (515) in the first direction (e.g., the left side of the electronic device (101)), it can be recognized that the hand to be used as the input interface is the left hand (510).

According to various embodiments, the electronic device (101) may change the UI layout in real time to correspond to the identified left or right hand after identifying the user's left or right hand. For example, if the electronic device (101) recognizes that the hand to be used as the input interface corresponding to the air gesture is the left hand in order to increase user convenience, the electronic device (101) may optimally arrange the UI for the right hand using the stylus pen (201). Conversely, if the electronic device (101) recognizes the right hand, the UI may optimally arrange the UI for the left hand using the stylus pen (201). According to one embodiment, the UI may include icons related to the stylus pen function. The electronic device (101) may set and store the layout of UI components related to the stylus pen function by matching them to the left or right-hand mode in advance.

Meanwhile, if the user recognizes that the hand to be used as an input interface for an air gesture is the left hand (510), as shown in 500b of FIG. 5, when the user draws a picture using a stylus pen (201) with the right hand and then momentarily lifts the hand from the screen (525), and makes a pinch out gesture of bringing the thumb and index finger of the recognized left hand (510) together and then spreading them apart without touching the screen of the electronic device (101), the screen (530) can be enlarged and displayed in response to the pinch out gesture. In this way, the user can control the application execution screen by making a hand gesture without touching the screen to which the palm rejection function is applied while the application is running.

FIG. 6 is an example screen diagram (600a, 600b) for explaining a change in stylus pen properties when hovering using an air gesture according to one embodiment.

In 600a of FIG. 6, it is exemplified that an air gesture is detected in a proximity area (620) in a first direction (e.g., the left side of the electronic device (101)) within a spatial range for recognizing an air gesture of the electronic device (101). In one embodiment, when an air gesture is detected in a state where a signal indicating hovering is input, the electronic device (101) may perform a different operation from a case where an air gesture is detected in a state without hovering.

According to one embodiment, when an air gesture is detected while a signal indicating hovering (640) by a stylus pen (201) is input, an action mapped to the detected air gesture may change. For example, since the signal indicating hovering (640) being input means that the user is preparing to use the stylus pen (201) on the screen, the electronic device (101) may provide at least one tool UI related to the stylus pen (201) or change pen properties based on the air gesture.

For example, based on sensing data detected in a proximity area (620) in a first direction (e.g., the left side of the electronic device (101)), if it is recognized that the hand to be used as an input interface for an air gesture is the left hand (610), as illustrated in 600b of FIG. 6, if the user draws a picture using a stylus pen (201) with the right hand and then hovers (640) over the screen (625), and then makes a pinch out gesture of bringing the thumb and index finger of the recognized left hand (610) together and then spreading them apart without touching the screen (635) of the electronic device (101), the stylus pen (201) property may be changed in response to the pinch out gesture. In the above, the case where an air gesture is recognized while hovering (640) is maintained was described as an example, but even when not in a hovering state, for example, when the user is ready to use a stylus pen (201) or there is a touch input using the stylus pen (201), that is, when the stylus pen (201) is being prepared on the screen to be used even when there is no hovering, an air gesture can be recognized.

For example, when an input by a stylus pen (201) is not detected but is in a state of preparation, the operation of an air gesture is recognized in real time, and thus the application execution screen can be controlled in response to the operation of the air gesture. On the other hand, when an input by a stylus pen (201) is detected according to a hovering input or a touch input by the stylus pen (201), the properties or functions related to the stylus pen (201) can be immediately changed by recognizing the operation of an air gesture in real time.

For example, properties such as thickness and opacity of the stylus pen (201) may be changed in response to a pinch-out gesture. At this time, a tool UI (630) related to the stylus pen properties may be temporarily displayed and then disappear on at least a portion of the application execution screen (635) so that the user can see at a glance what the changed stylus pen properties are.

According to one embodiment, in the case of a pinch-out gesture, a tool UI (630) indicating stylus pen properties that gradually change in response to the degree to which the thumb and index fingers are spread apart may be displayed on the application execution screen (635). For example, if the pen properties controlled by the pinch-out gesture are pen thickness and opacity, a pinch-out gesture of spreading or closing two fingers apart may correspond to an action of adjusting the pen thickness and opacity. In this way, the actions of the air gesture may be implemented to correspond to adjusting or changing properties related to the stylus pen. Accordingly, the user may control the execution screen of the electronic device (101) using the stylus pen (201) or change properties related to the stylus pen (201) simply by making a hand motion in an area close to the electronic device (101) without touching each related item to change properties related to the stylus pen.

FIG. 7 is a flowchart (700) of operations of an electronic device for processing air gestures associated with a stylus pen according to one embodiment.

Referring to FIG. 7, the operating method may include operations 705 to 720. Each step/operation of the operating method of FIG. 7 may be performed by at least one of an electronic device (e.g., the electronic device (101) of FIG. 1), at least one processor of the electronic device (e.g., the processor (120) of FIG. 1 and the processor (420) of FIG. 4). In one embodiment, at least one of operations 705 to 720 may be omitted, the order of some operations may be changed, or another operation may be added.

Below, a case will be described where an application related to a stylus pen (201) is being run and a function related to the stylus pen (201) is controlled using air gestures.

Here, the fact that an application related to the stylus pen (201) is running may mean that the stylus pen (201) is detached from the electronic device (101), and that various functions related to the stylus pen (201) can be controlled according to air gestures in the detached state.

According to one embodiment, the electronic device (101) can detect an air gesture initiation event. For example, the stylus pen (201) can be removed from the storage space of the electronic device (101), and the air gesture initiation event can be detected upon removal. That is, an operation for detecting an air gesture can be performed upon removal.

Alternatively, for example, the electronic device (101) may identify an air gesture initiation event based on information included in a communication signal from the stylus pen (201). The air gesture initiation event may be, for example, detection of a pressed state of a button (e.g., button (337)) provided on the stylus pen (201). The button may be implemented in any one of a physical key, a touch key, a motion key, a pressure key, and a key less manner, and there is no limitation on the implementation form. After identifying the pressed state of the button, the stylus pen (201) may transmit a communication signal including information indicating the pressed state of the button to the electronic device (101).

In operation 705, the electronic device (101) can identify whether a first input signal corresponding to an air gesture is detected in a proximity area of the electronic device (101). If the first input signal corresponding to the air gesture is detected in the proximity area of the electronic device (101), the first input signal can be confirmed. For example, the form of the air gesture corresponding to the first input signal can be identified, and the electronic device (101) can recognize whether the air gesture corresponds to any one of a plurality of air gestures, such as a gesture using one finger of the opposite hand holding the stylus pen (201), a gesture using two fingers of the opposite hand, a gesture using multiple fingers, a fist gesture, and a palm-out gesture.

It can be identified whether a second input signal corresponding to an input by a stylus pen (201) is detected in the 710 operation. According to one embodiment, the input by the stylus pen (201) may include a hovering input.

If the second input signal corresponding to the input by the stylus pen (201) is not detected, that is, if only the first input signal corresponding to the air gesture is detected, in operation 720, the electronic device (101) can control the application execution screen in response to the detection of the first input signal in a state where the second input signal is not detected. According to one embodiment, the electronic device (101) can adjust the application execution screen according to the air gesture corresponding to the first input signal. For example, the application execution screen can be rotated in a direction corresponding to the rotation gesture according to the rotation gesture.

On the other hand, when a second input signal corresponding to the hovering input by the stylus pen (201) is detected, that is, when the first input signal is detected in a state in which the second input signal corresponding to the hovering input by the stylus pen (201) is detected, the electronic device (101) can change the stylus pen property in response to the first input signal in a state in which the second input signal corresponding to the hovering input is detected in operation 715. As described above, the electronic device (101) can determine whether there is a hovering input when an air gesture is input, and control a corresponding function according to the determination result. In addition, although the above-described example illustrates a case in which it is determined whether there is a hovering input in a state in which an air gesture is input, it is also possible to determine whether there is a touch input in which an actual user uses the stylus pen (201) in addition to the hovering input. Accordingly, a screen control operation based on an air gesture can be performed during hovering or during a touch input.

According to one embodiment, when an air gesture is detected in a state where a signal indicating hovering by the stylus pen (201) is input, an action mapped to the air gesture may be different compared to an air gesture detected in a state where there is no hovering. For example, since the fact that a signal indicating hovering is input means that the user is preparing to use the stylus pen (201) on the screen, the electronic device (101) may provide a UI for at least one tool (e.g., composed of an icon that can change pen thickness, transparency, brush, color, and/or color) related to the stylus pen (201) based on the air gesture, or change pen properties (e.g., pen thickness, transparency, brush, color, and/or color).

According to one embodiment, a method for controlling a function associated with a stylus pen by an air gesture in an electronic device (e.g., the electronic device (101) of FIG. 1) may include an operation of identifying whether at least one of a first input signal corresponding to the air gesture and a second input signal corresponding to an input by the stylus pen is detected in a proximity area of the electronic device by at least one sensor, an operation of controlling a first function associated with a screen display in response to detection of the first input signal in a state in which the second input signal is not detected, and an operation of controlling a second function associated with the stylus pen in response to detection of the first input signal in a state in which the second input signal is detected.

In one embodiment, the act of identifying whether at least one of the first input signal and the second input signal is detected may include an act of identifying whether at least one of the first input signal and the second input signal is detected after removal of the stylus pen from the electronic device is detected.

According to one embodiment, the operation of controlling the second function associated with the stylus pen in response to detection of the first input signal while the second input signal is detected may include the operation of controlling the second function associated with the stylus pen to be changed in response to the first input signal while the second input signal is detected, and the operation of displaying information about the changed second function for a specified period of time and then removing it.

According to one embodiment, the method may further include, when removal of the stylus pen from the electronic device is detected, displaying a plurality of executable applications related to the stylus pen and changing a function of one of the plurality of applications in response to the first input signal.

According to one embodiment, the method may further include an operation of changing a second function associated with the stylus pen in response to the first input signal when the first input signal is detected in a state in which a third input signal indicating that a button of the stylus pen is pressed is detected.

According to one embodiment, the method may further include an operation of changing a second function associated with the stylus pen in real time based on the first input signal after a third input signal indicating that the button of the stylus pen is pressed is detected and before a fourth input signal indicating that the button of the stylus pen is released is detected.

According to one embodiment, the method may further include an operation of displaying an icon indicating an activated state of the air gesture on the application execution screen when the first input signal is detected after removal of the stylus pen from the electronic device is detected, and an operation of displaying an icon indicating an inactivated state of the air gesture after displaying the icon indicating the activated state of the air gesture and then, if the first input signal is not detected for a predetermined period of time.

According to one embodiment, the method may further include, if the first input signal is detected in an area corresponding to a left direction of the electronic device, displaying at least one item for a second function related to the stylus pen by rearranging it to correspond to a right-hand mode, and if the first input signal is detected in an area corresponding to a right direction of the electronic device, displaying at least one item for a second function related to the stylus pen by rearranging it to correspond to a left-hand mode.

FIG. 8 is a table (800) illustrating types of air gestures according to one embodiment.

As illustrated in FIG. 8, a table (800) exemplifying the types of air gestures may include the shape of air gestures (810), a description of air gestures (820), an action by air gestures (830), an action by air gestures during hovering (840), and a UI for stylus pen-related items (850).

The pinch in/out gesture represents a motion of bringing the thumb and index finger together and then spreading them apart or bringing them together again, and in a non-hovering state, it can zoom in/out the canvas, that is, zoom in/out the application execution screen. On the other hand, if the pinch in/out gesture is made in a hovering state, that is, when the stylus pen (201) is brought close to the screen, the stylus pen properties can be changed in response to the pinch in/out gesture. For example, values such as pen thickness and pen transparency can be adjusted as stylus pen properties, and a UI (851) in which the pen thickness and pen transparency are adjusted can be displayed on at least a part of the application execution screen so that the user can identify the changed state.

A rotation gesture indicates a motion of turning a thumb and an index finger to the left or right, and in a state without hovering, it can rotate a canvas, that is, rotate an application execution screen. On the other hand, in a state with hovering, a type of brush can be changed as a stylus pen property in response to a rotation gesture. For example, brushes can be sequentially changed in a UI (852) in which types of brushes are listed in response to a rotation direction and degree of rotation so that a user can identify the type of brush being changed. For example, when a user makes a rotation gesture indicating a rotation motion to the right, brushes can be sequentially changed in the right direction, and when the rotation gesture is stopped, the type of brush corresponding to the selected brush can be determined.

The up-down gesture represents a motion of moving the index finger up and down or left and right, and in a state without hovering, it can scroll up and down or move left and right on the application execution screen. On the other hand, in a state with hovering, the color or saturation can be adjusted as a stylus pen property in response to the up-down gesture. For example, the color or saturation value can be adjusted in response to the degree of the up-down gesture or the degree of the left and right movement gesture, and a UI (853) in which the color or saturation value is adjusted in response thereto can be displayed.

A double tap gesture indicates a user's action of tapping twice with one finger, and in a non-hovering state, a canvas can be enlarged/reduced, for example, at a fixed ratio, on an application execution screen. On the other hand, a color picker may be called/terminated in response to a double tap gesture in a hovering state. For example, a color picker may be called in response to a double tap gesture in a hovering state, or a color may be changed to a color specified by the user as a stylus pen property. For example, a UI (854) in which a color changed in response to a double tap gesture is selected may be displayed.

The palm swipe gesture represents an action of the user rubbing the palm left and right, and can undo/redo the content executed on the application execution screen. For example, if the user rubs the palm left, undo can be performed, and if the user rubs the palm right, redo can be performed. For example, UI (855) representing an action executed in response to the palm swipe gesture can be displayed.

The clench gesture represents a user making a fist and can perform a full erase of the content displayed on the application launch screen. For example, if the user makes a clench gesture while the application launch screen is displayed, a UI (856) may be displayed indicating that all drawings will be deleted.

According to one embodiment, the UI (850) for the stylus pen related item may be implemented to be displayed on at least a portion of the application execution screen for a certain period of time and then disappear so as not to cover the application execution screen as much as possible.

According to one embodiment, the UI (850) for the stylus pen related item may be displayed close to the hovering position of the stylus pen (201) when hovering. Alternatively, the UI (851) may be temporarily displayed at the top of the application execution screen when hovering and then hidden.

The above describes the types of air gestures and the functions mapped to them, but various air gestures can be corresponded to control the application execution screen and change the properties of the stylus pen. For example, instead of the palm swipe gesture, an eraser gesture representing a motion of rubbing a flat surface with a finger can be used to perform an action of erasing part or all of the contents on the application execution screen. In addition to the above, motions used in real life, such as applying a gesture of picking up an object to a motion of selecting an object on the screen, can be applied as air gestures as they are, and the mappable motions related to the stylus pen in response to air gestures may not be limited thereto.

FIG. 9A is an example of an application execution screen before activating the air gesture mode according to one embodiment. FIG. 9A illustrates a case where a panel (900) that is always exposed at the top of the application execution screen is displayed, and a panel (905) for changing properties related to a stylus pen (201) is displayed. Here, the application is an application that can draw letters or pictures using a stylus pen (201), and may be called a note application, a memo application, or a drawing board application. Such an application provides tool panels (900, 905) that can change properties such as pen thickness and color, and such tool panels (900, 905) may cover a part of the application execution screen at all times or whenever the properties are changed. According to one embodiment, when the air gesture mode is activated, the panel may be adjusted to a minimized size as illustrated in FIGS. 9B to 9D. Figures 9b to 9d illustrate the process of sequentially changing the size of the panel when the air gesture mode is activated.

FIG. 9b is an example screen diagram for explaining changes in the tool panel when the air gesture mode is activated according to one embodiment.

As illustrated in FIG. 9b, when an air gesture (910) is detected in a proximity area of the electronic device (101), the air gesture mode may be activated, and an interactive visual effect may be provided so as not to interfere with a user's work when the air gesture mode is activated. For example, a visual interaction may be provided in which at least one tool panel (e.g., composed of pen thickness, transparency, brush, color, and/or an icon that can change color) (915) related to a property of the stylus pen (201) disappears in a first direction (920).

FIG. 9C is an example screen diagram for explaining changes in the tool panel upon completion of activation of the air gesture mode according to one embodiment. As shown in FIG. 9C, the display of the tool panel (925) may temporarily stop before disappearing in the first direction (930), thereby allowing the user to recognize the currently selected pen property related to the stylus pen (2010).

FIG. 9d is an example screen diagram for explaining the hidden state of the tool panel when the air gesture mode activation is completed according to one embodiment. FIG. 9d exemplifies a case where the tool panel (925) of FIG. 9c is in a hidden state.

As illustrated in FIGS. 9b to 9d, when the air gesture mode is activated, the tool panel display may be sequentially changed to a size that minimizes the display. In addition, as illustrated in FIGS. 9b to 9d, when the activation of the air gesture mode is completed, that is, when the control of a function related to the stylus pen using the air gesture is possible, the color, brightness, transparency, shape, saturation, and color effect (e.g., gradient effect, blinking effect) of a graphic object (e.g., pointer, indicator) (945) indicating hovering by the stylus pen (201) on the screen may be changed. For example, the air gesture mode may be indicated to be activated by changing the color of the graphic object to a different color or changing the shape of the graphic object, e.g., the shape of the pointer. In addition, a status icon (940) indicating whether the air gesture is activated may be provided on the screen so that the user can intuitively recognize the activation of the air gesture mode.

As described above, when air gestures are activated, graphic objects are displayed differently on the screen depending on whether air gesture mode is entered or completed, so that the user can easily recognize the activation status of air gesture mode.

FIG. 10A is an example screen diagram providing status information according to an embodiment of the present invention when entering the air gesture mode. FIG. 10A illustrates an example of indicating an entry status into the air gesture mode when an air gesture is detected by a user's hand (1010) in a proximity area of an electronic device (101). Similar to FIG. 9D, FIG. 10A may provide a pointer (1015) by a stylus pen (201) and an icon (1005) indicating whether the air gesture is activated on an application execution screen.

FIG. 10b is an example screen diagram providing status information according to an air gesture input according to an embodiment. FIG. 10b illustrates a case where, when an air gesture (1010) by a user's hand is detected while the air gesture mode is activated, a UI (1025) is displayed for confirming the air gesture and changing the property related to the stylus pen (201) corresponding to the confirmed air gesture. At this time, as illustrated in FIG. 10b, when an air gesture (1010) by a user's hand is detected while the air gesture mode is activated, the color of an icon (1005) indicating whether the air gesture is activated may be changed. For example, the color of the icon (1005) indicating whether the air gesture is activated may be displayed by switching to a preset color, such as a different color, while the air gesture is input. In addition, by changing the brightness, transparency, shape, saturation, and color effect (e.g., gradient effect, blinking effect) in addition to the color of the icon (1005), whether the air gesture is activated may be indicated. Additionally, according to one embodiment, if the confirmed air gesture is a rotation gesture, a UI (1025) may be temporarily displayed and then disappear, sequentially selecting one of the types of brushes in response to the rotation gesture.

FIG. 10C is an example screen diagram providing status information according to non-detection of an air gesture according to an embodiment. FIG. 10C illustrates a screen when the air gesture mode is activated but a hand is not detected within a proximity area. As illustrated in FIG. 10C, when a hand indicating an air gesture is absent within a proximity area for a certain period of time, that is, when a hand (1010) is not detected even after a specified period of time, the air gesture mode may be automatically terminated, and accordingly, the color of an icon (1030) indicating whether the air gesture is activated may be changed to a color, for example, light gray, to indicate a deactivated state.

As shown in FIGS. 10A to 10C, the display status of icons (or indicators) indicating whether air gestures are activated on the application execution screen may vary depending on whether a hand performing an air gesture is detected in a proximity area of the electronic device (101).

FIG. 11A is an example screen diagram for explaining a pen property change operation based on an air gesture according to an embodiment. As illustrated in FIG. 11A, when the air gesture mode is activated by an icon (1105) indicating whether the air gesture is activated, a user can make a desired air gesture using one hand (1110), and an icon panel (1100) including a pen property that can be changed in an overlay form can be displayed on the pointer position of the stylus pen (201) in response to the air gesture. For example, real-time movement (1100) between types of brushes can be possible in response to the degree of the air gesture, for example, the rotation direction and degree of the rotation gesture. This UI (1100) can disappear, for example, after N seconds, when a brush type is selected in response to the air gesture.

FIG. 11B is an example screen diagram providing an icon panel according to a change in pen properties according to an embodiment. As illustrated in FIG. 11B, when a brush type is selected in response to an air gesture, an icon panel (1115) indicating the selected brush may be popped up for a certain period of time (e.g., N seconds). At this time, since the brush selection using the air gesture is complete, the color of an icon (1120) indicating whether the air gesture is activated may be changed on the screen. For example, information on the selected brush, such as the brush type and the current color, may be temporarily shown and then made to disappear as illustrated in FIG. 11C. Here, FIG. 11C is an example screen diagram showing a hidden state of an icon panel according to a change in pen properties according to an embodiment. As illustrated in FIG. 11C, the electronic device (101) may display the icon panel (1125) in a hidden state on one side of the screen, and may maintain the color of the icon (1120) indicating whether the air gesture is activated.

As shown in the above-described FIGS. 11a to 11c, by providing an icon panel that is temporarily displayed when an air gesture is input, the user is informed of the currently set pen properties, and at the same time, the user can continue drawing without interruption. In addition to the above, as shown in FIG. 12, an icon panel that is temporarily displayed depending on the hovering state may also be provided.

FIG. 12 is a screen example diagram for explaining the display operation of the icon panel according to hovering according to one embodiment. FIG. 12 illustrates a case where the display of the icon panel changes depending on whether hovering is not performed for N seconds during hovering and whether hovering is input again. For example, a user may draw letters or pictures using a stylus pen (201) and then temporarily remove the stylus pen (201) from the screen, but may then start writing or drawing again. In this case, the electronic device (101) may temporarily display an icon panel so that the user can know the properties of the stylus pen (201) that was being used just before removing the stylus pen (201).

According to one embodiment, as illustrated in FIG. 12, the electronic device (101) may display the icon panel (1205) in a hidden state during hovering, and when the hovering is released and there is no hovering for N seconds, display an icon panel (1210) indicating the current pen properties on at least a part of the screen, for example, on the upper left of the screen, and then display the icon panel (1215) in a hidden state when there is hovering again. As described above, by informing the user of the current pen properties right before using the stylus pen (201), the user can know which pen is currently being used.

Figure 13a is an example screen diagram for right-hand mode according to one embodiment.

As illustrated in FIG. 13a, when an air gesture (1310a) by the user's left hand is detected in an area corresponding to the left side of the electronic device (101), items related to the stylus pen properties may be rearranged and displayed to correspond to the right-hand mode using the stylus pen (201). For example, some tools (1305, 1315) on the icon panel may be rearranged to correspond to the right-hand mode, and the order of options may be flipped left/right. Accordingly, when an air gesture by the left hand is detected, an icon panel may be displayed in which the most frequently used tools (1325) are arranged on the left side, as in the first screen (1320). In addition, a hidden icon panel (1335) may also be arranged to be located on the left side, as in the second screen (1330).

FIG. 13b is an example of a screen for a left-hand mode according to one embodiment. As illustrated in FIG. 13b, when an air gesture (1310b) by a user's right hand is detected in an area corresponding to the right direction of the electronic device (101), items related to the stylus pen properties may be rearranged and displayed to correspond to the left-hand mode using the stylus pen (201). For example, the icon panel (1340) may be hidden to the right, and the location of the icon panel (1355) may be rearranged to correspond to the left-hand mode as in the first screen (1350), and the hidden icon panel (1365) may also be positioned to the right as in the second screen (1360). This switching of screens for the right-hand or left-hand mode is intended to facilitate the user's operation, and may facilitate direct screen interface operation with the right hand in the absence of hovering.

As described above, if an air gesture is detected in an area corresponding to the left direction of the electronic device, the electronic device (101) can rearrange and display items related to the stylus pen (201) properties to correspond to the right-hand mode. Conversely, if an air gesture is detected in an area corresponding to the right direction of the electronic device, the electronic device (101) can rearrange and display items related to the stylus pen (201) properties to correspond to the left-hand mode.

FIG. 14 is an example screen diagram for explaining an operation using an air gesture when removing a stylus pen according to one embodiment.

As illustrated in 1400a of FIG. 14, the stylus pen (201) can be removed (1405) from the storage space of the electronic device (101). For example, the user can take the stylus pen (201) out from the storage space of the electronic device (101). According to one embodiment, the electronic device (101) can detect the removal of the stylus pen (201). For example, the electronic device (101) can detect the removal of the stylus pen (201) based on a failure to receive a signal from the stylus pen (201), but there is no limitation on the method of detecting the removal. On the other hand, the stylus pen (201) can also detect the removal of the stylus pen (201). For example, the stylus pen (201) can detect removal of the stylus pen (201) based on non-reception of a signal from the electronic device (101), but there is no limitation on the method of detecting removal.

If a removal is detected, a plurality of applications executable in relation to the stylus pen (201) may be displayed on the screen (1420) as illustrated in 1400b of FIG. 14. Alternatively, icons corresponding to frequently used applications in relation to the stylus pen (201) may be displayed on the screen. According to one embodiment, the plurality of applications executable in relation to the stylus pen (201) may include various functions such as creating a new note, viewing all notes, smart select, writing after capture, live message, and translator. For example, in the case of the smart select function, the user may select a desired area using an air gesture instead of the stylus pen (201) and save or share it. For example, if there is content to be captured, the user may select the smart select function, select the shape of the area to be captured using an air gesture, and then select the area by dragging.

As illustrated in 1400b of FIG. 14, when the electronic device (101) recognizes an air gesture (1410) by a user who selects a smart select function among a plurality of applications, the electronic device (101) may indicate that the smart select function (1425) has been selected. Then, in response to the selection of the smart select function (1425), the smart select function is executed, and then, for example, an area may be selected in response to an air gesture indicating an action of dragging an area to be captured, and when an air gesture different from the air gesture is additionally input, an action mapped to the additionally input air gesture may be performed for the selected area. The action mapped to each of these air gestures may vary depending on whether the stylus pen (201) is in a hovering state or not. For example, when an air gesture is input in a hovering state, the air gesture may be identified and an area may be selected based on the identified air gesture. That is, the shape of the selected area may vary depending on the form of the air gesture. For example, if an air gesture is input without hovering, one of the sub-functions of the smart select function, such as sharing the selected area or saving it to the gallery, may be performed, as mapped to the input air gesture.

FIG. 15 is an example screen diagram for explaining the operation of a translator using air gestures according to one embodiment.

Referring to FIG. 15, if an air gesture is received while a translation function is running among multiple applications, the electronic device (101) can confirm the air gesture and perform an operation mapped to the translation function in response to the air gesture. For example, the language (1515) of the translation device can be changed using the air gesture (1510) or a portion to be translated can be selected (1520) and then the translation can be controlled to be executed. For example, if an air gesture is received while the stylus pen (201) is brought close to a word requiring translation, the electronic device (101) can display the translation result.

In addition, for example, if there is a word or sentence to be translated, the translation function can be executed using an air gesture while the stylus pen (201) is removed from the electronic device (101), and if another air gesture is input while the translation function is executed, the language to be used in the window displayed at the top of the screen may be automatically changed in response to the other air gesture. At this time, the translation text selection unit can be changed to a word or a sentence, and such change can also be set by an air gesture. However, this is exemplary, and various functions related to the stylus pen (201) that can be mapped to an air gesture in the electronic device (101) according to various embodiments may not be limited thereto. That is, the operation according to the air gesture described above is merely exemplary, and the operations related to the stylus pen that can be mapped to an air gesture are not limited thereto.

Electronic devices according to various embodiments disclosed in this document may be devices of various forms. The electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliance devices. Electronic devices according to embodiments of this document are not limited to the above-described devices.

It should be understood that the various embodiments of this document and the terminology used herein are not intended to limit the technical features described in this document to specific embodiments, but rather to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first) is referred to as "coupled" or "connected" to another (e.g., a second) component, with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

The term "module" as used in this document may include a unit implemented in hardware, software or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integrally configured component or a minimum unit of the component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more commands stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, ‘non-transitory’ simply means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded between a seller and a buyer as a commodity. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store ^TM ) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a part of the computer program product may be at least temporarily stored or temporarily generated in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or an intermediary server.

According to various embodiments, each of the components (e.g., modules or programs) described above may include a single or multiple entities. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., modules or programs) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the components of the plurality of components identically or similarly to those performed by the corresponding component of the plurality of components prior to the integration. According to various embodiments, the operations performed by the modules, programs or other components may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

In a storage medium storing commands, the commands are configured to cause the at least one processor to perform at least one operation when executed by the at least one processor, wherein the at least one operation may include: identifying whether at least one of a first input signal corresponding to an air gesture and a second input signal corresponding to an input by a stylus pen is detected in a proximity area of the electronic device by at least one sensor; controlling a first function related to a screen display in response to detection of the first input signal in a state where the second input signal is not detected; and controlling a second function related to the stylus pen in response to detection of the first input signal in a state where the second input signal is detected.

And the embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to easily explain the technical contents according to the embodiments of the present invention and to help understand the embodiments of the present invention, and are not intended to limit the scope of the embodiments of the present invention. Therefore, the scope of the various embodiments of the present invention should be interpreted as including all changes or modified forms derived based on the technical ideas of the various embodiments of the present invention in addition to the embodiments invented herein.

101: electronic device, 120: processor, 130: memory, 150: input device, 160: display device, 190: communication module

Claims (20)

In electronic devices,
A display device including a sensing panel;
At least one sensor capable of detecting an air gesture in a proximity area of said electronic device;
At least one processor operatively connected to said display device and said at least one sensor; and
comprising a memory operatively connected to at least one processor;
The above memory, when executed, causes the electronic device to:
Distinguishing between a user's hand holding the stylus pen and the user's other hand by at least one sensor,
By identifying the direction corresponding to the opposite hand, an air gesture is detected for the identified direction,
When an air gesture is detected in a proximity area of the electronic device by at least one sensor, and an input signal corresponding to an input by the stylus pen is not detected through the sensing panel, a first function related to a screen display is controlled in response to the air gesture,
An electronic device storing instructions configured to control a second function related to the stylus pen in response to the input signal when the air gesture is detected in a state where the input signal is detected, arrange an icon panel on the display device in response to the air gesture, and rearrange the arranged icon panel by moving it to the left on the display device when the opposite hand is the left hand, and to rearrange the arranged icon panel by moving it to the right on the display device when the opposite hand is the right hand. In the first paragraph, the instructions cause the electronic device to:
After the removal of the stylus pen from the electronic device is detected, identifying whether at least one of the air gesture and the input signal is detected;
When the air gesture is detected while the input signal is detected, the second function related to the stylus pen is controlled to be changed in response to the air gesture.
An electronic device configured to display information about the changed second function on the display device for a specified period of time and then remove it. delete In the first paragraph, the instructions cause the electronic device to:
An electronic device configured to, when removal of the stylus pen from the electronic device is detected, display a plurality of executable applications related to the stylus pen, and change the function of one of the plurality of applications in response to the air gesture. In the first paragraph, the instructions cause the electronic device to:
When the air gesture is detected while a second input signal indicating that the button of the stylus pen is pressed is detected, the second function related to the stylus pen is changed in response to the air gesture.
An electronic device configured to change a second function associated with the stylus pen in real time based on the air gesture after a second input signal indicating that the button of the stylus pen is pressed is detected and before a third input signal indicating that the button of the stylus pen is released is detected. delete In the first paragraph, the first function related to the screen display is,
An electronic device having a control function for a running application screen, wherein the second function related to the stylus pen is a function related to the properties of the stylus pen. In the first paragraph, the instructions cause the electronic device to:
After the removal of the stylus pen from the electronic device is detected, if the air gesture is detected, an icon indicating the activation status of the air gesture is displayed on the application execution screen displayed on the display device,
An electronic device configured to display an icon indicating an activated state of the air gesture on an application execution screen displayed on the display device, if the air gesture is not detected for a certain period of time after displaying an icon indicating an activated state of the air gesture. delete delete In the first paragraph, at least one sensor,
An electronic device comprising at least one of an RF spectrum type sensor, an infrared sensor, an ultrasonic sensor, a motion sensor, a 3D sensor, and a vision sensor capable of detecting an air gesture in a proximity area of the electronic device. A method for controlling a function associated with a stylus pen by air gesture in an electronic device,
An action of distinguishing between a user's hand holding the stylus pen and the user's other hand by at least one sensor;
An operation of identifying a direction corresponding to the opposite hand, detecting an air gesture for the identified direction, and identifying whether at least one of an air gesture in a proximity area of the electronic device and an input signal corresponding to an input by the stylus pen is detected by the at least one sensor;
An operation for controlling a first function related to screen display in response to detection of the air gesture in a state where the input signal is not detected; and
A method for controlling a function associated with a stylus pen by an air gesture, comprising: controlling a second function associated with the stylus pen in response to detection of the air gesture in a state where the input signal is detected; arranging an icon panel on a display device, in which at least one item for the second function is arranged in response to the air gesture; and moving the arranged icon panel to the left on the display device and rearranging it if the opposite hand is a left hand, and moving the arranged icon panel to the right on the display device and rearranging it if the opposite hand is a right hand. delete delete delete delete delete delete delete In a storage medium storing commands, the commands are configured to cause the electronic device to perform at least one operation when executed by at least one processor of the electronic device, the at least one operation comprising:
An action of distinguishing between a user's hand holding a stylus pen and the user's other hand by at least one sensor;
An operation of identifying a direction corresponding to the opposite hand, detecting an air gesture for the identified direction, and identifying whether at least one of an air gesture in a proximity area of the electronic device and an input signal corresponding to an input by the stylus pen is detected by the at least one sensor;
An operation for controlling a first function related to screen display in response to detection of the air gesture in a state where the input signal is not detected; and
A storage medium comprising an operation of controlling a second function related to the stylus pen in response to detection of the air gesture in a state where the input signal is detected, arranging an icon panel on a display device in which at least one item for the second function is arranged in response to the air gesture, and moving the arranged icon panel to the left on the display device and rearranging it when the opposite hand is a left hand, and moving the arranged icon panel to the right on the display device and rearranging it when the opposite hand is a right hand.

Images