[go: up one dir, main page]

WO2020153785A1 - Dispositif électronique et procédé pour fournir un objet graphique correspondant à des informations d'émotion en utilisant celui-ci - Google Patents

Dispositif électronique et procédé pour fournir un objet graphique correspondant à des informations d'émotion en utilisant celui-ci Download PDF

Info

Publication number
WO2020153785A1
WO2020153785A1 PCT/KR2020/001161 KR2020001161W WO2020153785A1 WO 2020153785 A1 WO2020153785 A1 WO 2020153785A1 KR 2020001161 W KR2020001161 W KR 2020001161W WO 2020153785 A1 WO2020153785 A1 WO 2020153785A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
emotion information
processor
electronic device
graphic object
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/KR2020/001161
Other languages
English (en)
Korean (ko)
Inventor
정글지
배주윤
우영찬
이미영
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Publication of WO2020153785A1 publication Critical patent/WO2020153785A1/fr
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T13/00—Animation
    • G06T13/20—3D [Three Dimensional] animation
    • G06T13/40—3D [Three Dimensional] animation of characters, e.g. humans, animals or virtual beings
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815—Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/60—Editing figures and text; Combining figures or text
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/04—Segmentation; Word boundary detection
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G10L25/63—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for estimating an emotional state
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/90—Pitch determination of speech signals

Definitions

  • Various embodiments of the present invention relate to a method of providing a graphic object corresponding to emotion information.
  • SNS social network services
  • messenger services messenger services
  • users can share their daily lives with others.
  • a user writes his or her daily life or thoughts in writing, shares it with the other person, emojis with the text, and a predefined emoji based on the facial feature area extracted from the user's image (eg, hairstyle, glasses, hat) , Stickers) to express their feelings directly or indirectly to the other party.
  • a predefined emoji based on the facial feature area extracted from the user's image (eg, hairstyle, glasses, hat) , Stickers) to express their feelings directly or indirectly to the other party.
  • the electronic device provides a user with a predefined emoticon or emoji, but is limited in expressing the user's emotions because the predefined types of emoticons or emoji are limited.
  • the user cannot take a picture of his face, and accordingly, cannot transmit his feelings to the other party.
  • the electronic device may determine emotion information of an object based on a feature region of an object included in an image, and provide a graphic object corresponding to the determined emotion information.
  • the electronic device may determine emotion information based on input voice data, and provide graphic objects related to the voice data corresponding to the emotion information by mapping the recognized object in the image.
  • the electronic device may apply one's emotion to another object (eg, an animal or an object) obtained from the camera in an AR environment in which the user's own face cannot be captured.
  • another object eg, an animal or an object
  • an electronic device includes a display, a microphone, a processor operatively connected to the display and the microphone, and a memory operatively connected to the processor, wherein the memory, upon execution ,
  • the processor displays an image on the display, recognizes at least one object from the displayed image, extracts at least one feature region for the recognized at least one object, and at least one through the microphone
  • voice data is received
  • first emotion information is determined based on the received at least one voice data
  • at least one graphic object related to the at least one voice data is generated based on the first emotion information
  • a region to synthesize the generated at least one graphic object among the at least one feature region is determined based on the content of speech of the at least one voice data, and the generated at least one location at a position corresponding to the determined region.
  • an electronic device includes a display, a processor operatively connected to the display, and a memory operatively connected to the processor, wherein the memory, when executed, includes the processor, Displaying an image on the display, recognizing at least one object from the displayed image, and determining emotion information based on at least one of data related to the image, context information, or information related to a user of the electronic device, When an area in the image to synthesize at least one graphic object corresponding to the determined emotion information is determined, the display displays one or more graphic objects corresponding to the determined emotion information to be synthesized in the area based on characteristics of the determined area. Instructions to display and synthesize the at least one selected graphic object at a position corresponding to the determined area when a user input for selecting at least one of the displayed one or more graphic objects is detected. ).
  • a method of providing a graphic object corresponding to emotion information of an electronic device includes displaying an image on a display, recognizing at least one object from the displayed image, and recognizing the recognized image. Extracting at least one feature region for at least one object, determining at least one voice information based on the received at least one voice data when at least one voice data is received through the microphone, the Generating at least one graphic object related to the at least one voice data based on first emotion information, and the generated at least one of the at least one feature region based on the utterance content of the at least one voice data
  • the method may include determining an area to synthesize the graphic object, and synthesizing the generated at least one graphic object at a position corresponding to the determined area.
  • a method of providing a graphic object corresponding to emotion information of an electronic device includes: displaying an image on a display, recognizing at least one object from the displayed image, and Determining emotion information based on at least one of related data, context information, or information related to a user of the electronic device, and determining an area in the image to synthesize at least one graphic object corresponding to the determined emotion information , Displaying one or more graphic objects corresponding to the determined emotion information to be synthesized in the area on the display based on characteristics of the determined area, and selecting at least one graphic object from the displayed one or more graphic objects When a user input is sensed, it may include an operation of synthesizing the selected at least one graphic object at a position corresponding to the determined area.
  • the electronic device may determine emotion information for an image based on data related to the image or input voice data, and synthesize a graphic object corresponding to the determined emotion information into the image.
  • the user can communicate with the other party by using the image in which the graphic object corresponding to the emotion information is synthesized, so that the user's emotion can be more abundantly transmitted to the other party.
  • the emotions are reinterpreted according to a characteristic area of a person in an object (or animal) obtained from the camera. It can provide a fun factor to the user.
  • FIG. 1 is a block diagram of an electronic device in a network environment, according to various embodiments.
  • FIG. 2 is a block diagram illustrating an electronic device according to various embodiments of the present disclosure.
  • FIG. 3 is a flowchart illustrating a method of providing a graphic object corresponding to emotion information according to various embodiments.
  • FIG. 4 is a flowchart illustrating a method of providing a graphic object corresponding to emotion information according to various embodiments.
  • FIG. 5 is a diagram for describing a method of determining emotion information by extracting a feature region for at least one object recognized from an image according to various embodiments.
  • FIG. 6 is a diagram for describing a method of generating at least one graphic object based on determined emotion information according to various embodiments.
  • FIG. 7 is a diagram for describing a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
  • FIG. 8 is a diagram for explaining a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
  • 9A and 9B are diagrams illustrating a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
  • FIG. 10 is a diagram for describing a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
  • 11 is a diagram illustrating signal flow between electronic devices that provide a graphic object corresponding to emotion information according to various embodiments.
  • FIG. 12 is a diagram illustrating signal flow between electronic devices that provide graphic objects corresponding to emotion information according to various embodiments.
  • FIG. 1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments.
  • the electronic device 101 communicates with the electronic device 102 through the first network 198 (eg, a short-range wireless communication network), or the second network 199. It may communicate with the electronic device 104 or the server 108 through (eg, a remote wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108.
  • the first network 198 eg, a short-range wireless communication network
  • the server 108 e.g, a remote wireless communication network.
  • the electronic device 101 may communicate with the electronic device 104 through the server 108.
  • the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, a sensor module ( 176), interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ).
  • the components for example, the display device 160 or the camera module 180
  • the sensor module 176 eg, a fingerprint sensor, an iris sensor, or an illuminance sensor
  • the display device 160 eg., a display
  • the sensor module 176 eg, a fingerprint sensor, an iris sensor, or an illuminance sensor
  • the processor 120 executes software (eg, the program 140) to execute at least one other component (eg, hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and can perform various data processing or operations. According to one embodiment, as at least part of data processing or computation, the processor 120 may receive instructions or data received from other components (eg, the sensor module 176 or the communication module 190) in the volatile memory 132. Loaded into, process instructions or data stored in volatile memory 132, and store result data in non-volatile memory 134.
  • software eg, the program 140
  • the processor 120 may receive instructions or data received from other components (eg, the sensor module 176 or the communication module 190) in the volatile memory 132. Loaded into, process instructions or data stored in volatile memory 132, and store result data in non-volatile memory 134.
  • the processor 120 includes a main processor 121 (eg, a central processing unit or an application processor), and an auxiliary processor 123 (eg, a graphics processing unit, an image signal processor) that can be operated independently or together. , Sensor hub processor, or communication processor). Additionally or alternatively, the coprocessor 123 may be set to use less power than the main processor 121, or to be specialized for a specified function. The coprocessor 123 may be implemented separately from the main processor 121 or as a part thereof.
  • a main processor 121 eg, a central processing unit or an application processor
  • an auxiliary processor 123 eg, a graphics processing unit, an image signal processor
  • the coprocessor 123 may be set to use less power than the main processor 121, or to be specialized for a specified function.
  • the coprocessor 123 may be implemented separately from the main processor 121 or as a part thereof.
  • the coprocessor 123 may replace, for example, the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 may be active (eg, execute an application) ) With the main processor 121 while in the state, at least one component of the components of the electronic device 101 (eg, the display device 160, the sensor module 176, or the communication module 190) It can control at least some of the functions or states associated with.
  • the coprocessor 123 eg, image signal processor or communication processor
  • may be implemented as part of other functionally relevant components eg, camera module 180 or communication module 190). have.
  • the memory 130 may store various data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176).
  • the data may include, for example, input data or output data for software (eg, the program 140) and instructions related thereto.
  • the memory 130 may include a volatile memory 132 or a nonvolatile memory 134.
  • the program 140 may be stored as software in the memory 130, and may include, for example, an operating system 142, middleware 144, or an application 146.
  • the input device 150 may receive commands or data to be used for components (eg, the processor 120) of the electronic device 101 from outside (eg, a user) of the electronic device 101.
  • the input device 150 may include, for example, a microphone, mouse, keyboard, or digital pen (eg, a stylus pen).
  • the audio output device 155 may output an audio signal to the outside of the electronic device 101.
  • the audio output device 155 may include, for example, a speaker or a receiver.
  • the speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive an incoming call.
  • the receiver may be implemented separately from, or as part of, the speaker.
  • the display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user).
  • the display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device.
  • the display device 160 may include a touch circuitry configured to sense a touch, or a sensor circuit (eg, a pressure sensor) configured to measure the strength of the force generated by the touch. have.
  • the audio module 170 may convert sound into an electrical signal, or vice versa. According to an embodiment, the audio module 170 acquires sound through the input device 150, or an external electronic device (eg, directly or wirelessly connected to the sound output device 155 or the electronic device 101) Sound may be output through the electronic device 102 (eg, a speaker or headphones).
  • an external electronic device eg, directly or wirelessly connected to the sound output device 155 or the electronic device 101
  • Sound may be output through the electronic device 102 (eg, a speaker or headphones).
  • the sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state can do.
  • the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biological sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.
  • the interface 177 may support one or more designated protocols that can be used for the electronic device 101 to directly or wirelessly connect to an external electronic device (eg, the electronic device 102).
  • the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.
  • HDMI high definition multimedia interface
  • USB universal serial bus
  • SD card interface Secure Digital Card interface
  • audio interface audio interface
  • the connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ).
  • the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).
  • the haptic module 179 may convert electrical signals into mechanical stimuli (eg, vibration or movement) or electrical stimuli that the user can perceive through tactile or motor sensations.
  • the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.
  • the camera module 180 may capture still images and videos. According to one embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.
  • the power management module 188 may manage power supplied to the electronic device 101.
  • the power management module 188 may be implemented, for example, as at least part of a power management integrated circuit (PMIC).
  • PMIC power management integrated circuit
  • the battery 189 may supply power to at least one component of the electronic device 101.
  • the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.
  • the communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishing and performing communication through the established communication channel.
  • the communication module 190 operates independently of the processor 120 (eg, an application processor) and may include one or more communication processors supporting direct (eg, wired) communication or wireless communication.
  • the communication module 190 may include a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg : Local area network (LAN) communication module, or power line communication module.
  • a wireless communication module 192 eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module
  • GNSS global navigation satellite system
  • LAN Local area network
  • Corresponding communication module among these communication modules includes a first network 198 (eg, a short-range communication network such as Bluetooth, WiFi direct, or infrared data association (IrDA)) or a second network 199 (eg, a cellular network, the Internet, Or, it may communicate with an external electronic device through a computer network (eg, a telecommunication network such as a LAN or WAN).
  • a computer network eg, a telecommunication
  • the wireless communication module 192 uses a subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 in a communication network such as the first network 198 or the second network 199.
  • IMSI International Mobile Subscriber Identifier
  • the antenna module 197 may transmit a signal or power to the outside (eg, an external electronic device) or receive it from the outside.
  • the antenna module 197 may include a single antenna including a conductor formed on a substrate (eg, a PCB) or a radiator made of a conductive pattern.
  • the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network 198 or the second network 199, is transmitted from the plurality of antennas by, for example, the communication module 190. Can be selected.
  • the signal or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna.
  • other components eg, RFIC
  • other than the radiator may be additionally formed as part of the antenna module 197.
  • peripheral devices for example, a bus, a general purpose input and output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)
  • GPIO general purpose input and output
  • SPI serial peripheral interface
  • MIPI mobile industry processor interface
  • commands or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199.
  • Each of the electronic devices 102 and 104 may be the same or a different type of device from the electronic device 101.
  • all or some of the operations performed on the electronic device 101 may be performed on one or more external electronic devices of the external electronic devices 102, 104, or 108.
  • the electronic device 101 when the electronic device 101 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 101 instead executes the function or service itself.
  • one or more external electronic devices may be requested to perform at least a portion of the function or the service.
  • the one or more external electronic devices receiving the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and deliver the result of the execution to the electronic device 101.
  • the electronic device 101 may process the result, as it is or additionally, and provide it as at least part of a response to the request.
  • cloud computing, distributed computing, or client-server computing technology can be used, for example.
  • FIG. 2 is a block diagram 200 illustrating an electronic device according to various embodiments.
  • the electronic device 201 (eg, the electronic device 101 of FIG. 1) includes a wireless communication circuit 210 (eg, the communication module 190 of FIG. 1 ), a camera 220 (eg: Camera module 180 of FIG. 1, memory 230 (eg, memory 130 of FIG. 1), sensor unit 240 (eg, sensor module 176 of FIG. 1), audio processing circuit 250 (Eg, the audio module 170 of FIG. 1), a touch screen display 260 (eg, the display device 160 of FIG. 1), and a processor 270 (eg, the processor 120 of FIG. 1). can do.
  • a wireless communication circuit 210 eg, the communication module 190 of FIG. 1
  • a camera 220 eg: Camera module 180 of FIG. 1, memory 230 (eg, memory 130 of FIG. 1)
  • sensor unit 240 eg, sensor module 176 of FIG. 1
  • audio processing circuit 250 Eg, the audio module 170 of FIG. 1
  • a touch screen display 260 eg, the display device 160 of FIG
  • the wireless communication circuit 210 (eg, the communication module 190 of FIG. 1) includes an electronic device 201 (eg, the electronic device 101 of FIG. 1) and an external electronic device (eg: Communication between the electronic device 102, the electronic device 104 of FIG. 1, or a server (eg, the server 108 of FIG. 1) may be connected.
  • an electronic device 201 eg, the electronic device 101 of FIG. 1
  • an external electronic device eg: Communication between the electronic device 102, the electronic device 104 of FIG. 1, or a server (eg, the server 108 of FIG. 1) may be connected.
  • the camera 220 (eg, the camera module 180 of FIG. 1) transmits the collected image as a preview image to the display 261 so that the user can view the image projected through the camera 220. Can be confirmed.
  • the camera 220 may generate an image data by photographing the collected image at the time when the input requesting the photographing occurs in response to an input requesting photographing.
  • one or more cameras 220 may be provided.
  • the memory 230 (eg, the memory 130 of FIG. 1) is a program for generating and synthesizing a graphic object corresponding to emotion information into an image and a program according to embodiments of the present invention.
  • a graphic object composite image may be stored.
  • the memory 230 may store a reference value for determining emotion information based on information related to a feature region extracted from an object, data in an image, context information, and voice data.
  • the memory 230 may store at least one graphic object corresponding to emotion information.
  • the memory 230 may store attribute values corresponding to emotion information to be applied to text for voice data.
  • the memory 230 may store an image obtained by combining at least one graphic corresponding to emotion information.
  • the sensor unit 240 detects an external environmental state (eg, a user state) of the electronic device 201 and generates electricity corresponding to the sensed state. You can generate signal or data values.
  • the sensor unit 240 may include a gesture sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.
  • the audio processing circuit 250 (eg, the audio module 170 of FIG. 1) is not illustrated, but may include a speaker and a microphone.
  • the audio processing circuit 250 may further include an earphone and a microphone.
  • the microphone may receive sound (eg, voice data), convert it into an electrical signal, and provide it to the processor 270.
  • the touch screen display 260 (eg, the display device 160 of FIG. 1) may be configured as an integral type including the display 261 and the touch panel 263.
  • the touch screen display 260 may display an image under the control of the processor 270.
  • the touch screen display 260 may display at least one graphic object corresponding to emotion information generated under the control of the processor 270.
  • the touch screen display 260 may synthesize at least one object generated in a specific area in the image under the control of the processor 270 and display an image in which the at least one object is synthesized.
  • the processor 270 controls the overall operation of the electronic device 201 and the signal flow between the internal components of the electronic device 201, and processes data. And control the supply of power from the battery to the components.
  • the processor 270 may display an image on the display 261.
  • the displayed image may include at least one image obtained from at least one camera 220, or an image stored in the memory 230.
  • the processor 270 can recognize at least one object from the displayed image. have.
  • the at least one object may include at least one of a person, an animal, or an object.
  • the processor 270 may extract at least one feature region (eg, a facial feature region including an eye region, a nose region, and/or a mouth region) for the recognized at least one object.
  • the processor 270 may determine emotion information based on at least one of the extracted at least one feature area, image-related data, or context information.
  • the data related to the image includes the background of the image, the object placed in the foreground, the type of the object (e.g. gun, knife, pointed object), the color of the image (e.g. saturation, brightness), within the image.
  • the included audio and/or text included in the video may be included, and the context information may include a current time, a current location, and/or an event schedule.
  • the processor 270 when at least one voice data is received from at least one user through a microphone (not shown), in one embodiment, the processor 270 includes a speech content for the received at least one voice data, each of the at least one voice data It may analyze at least one of the included user's voice tone, pitch, frequency, or sound pressure. The processor 270 may determine emotion information for each of the at least one voice data based on the analyzed information. The processor 270 may generate at least one graphic object to be synthesized in an image based on the determined emotion information for each of the at least one voice data. For example, the at least one graphic object may include a sticker corresponding to the determined emotion information, an emoji, and/or text applied with attributes related to the at least one voice data.
  • the processor 270 determines an area to synthesize at least one graphic object generated among at least one feature area based on the utterance content of the at least one voice data, and the location in the image corresponding to the determined area At least one graphic object generated in the can be synthesized.
  • the processor 270 may store an image in which at least one graphic object is synthesized in the memory 230.
  • FIG. 3 is a flowchart 300 illustrating a method of providing a graphic object corresponding to emotion information according to various embodiments.
  • the processor may display an image on the display (eg, the display 261 of FIG. 2) in operation 301.
  • the displayed image may include at least one image obtained from at least one camera (eg, the camera 220 of FIG. 2 ), or an image stored in a memory (eg, the memory 230 of FIG. 2 ). .
  • the processor may recognize at least one object from the displayed image.
  • the at least one object may include at least one of a person, an animal, or an object.
  • the at least one object may be automatically recognized, or may be recognized by a user input selecting at least one object in an image.
  • the processor may extract at least one feature region for the recognized at least one object.
  • the at least one feature region is, for example, a facial feature region, and may include an eye region, a nose region, and/or a mouth region.
  • the at least one feature region may be automatically extracted, or may be extracted by a user input selecting at least one feature region.
  • the processor may generate a control point (eg, a marker) on the contour of the extracted at least one feature area, for example, the contour of the left and right eye areas, the contour of the nose area, and/or the mouth area.
  • the processor may sense a user input adjusting a control point generated in the contour of the extracted at least one feature area, and adjust the extracted at least one feature area based on the detected user input.
  • At least one feature region for at least one object may be accurately extracted through a user input adjusting a control point.
  • the processor may further perform an operation of tracking changes in at least one feature region (eg, eye movement or mouth movement) for at least one object based on a difference image between image frames.
  • a change in at least one feature area for example, eye movement or mouth movement, may be used to determine first emotion information of operation 307 described later.
  • the processor determines the first emotion information based on at least one of the extracted at least one feature region, the image-related data, or context information in operation 307.
  • the first emotion information may include at least one of happiness, anger, anxiety, sadness, boredom, disgust, or daily life.
  • the processor may determine the first emotion information based on the extracted at least one feature region, for example, the distance between the left and right eyes and/or the shape of the mouth.
  • the present invention is not limited thereto, and the processor may determine the first emotion information based on the movement of at least one feature region, for example, the movement of the eyes, movement of the lips, and/or movement of the face.
  • data related to an image includes an image background, an object disposed in the foreground, an object type (eg, gun, knife, pointed object), and an image color (eg, saturation, brightness).
  • object type e.g., gun, knife, pointed object
  • image color e.g, saturation, brightness
  • the processor may determine first emotion information based on data related to the image.
  • the context information may include the current time, current location, and/or event schedule.
  • the processor may determine the first emotion information based on the context information.
  • the first emotion information may be determined by analyzing an event according to the current time. When the event over time is birthday or Christmas, the first emotion information may be determined as positive emotion information (for example, happiness). When the event over time is Halloween, the first emotion information is negative emotion information (for example: Fear).
  • the first emotion information may be determined as positive emotion information (eg, happiness), and when the current location is a ghost house, the first emotion information may be negative emotion information (eg : Fear).
  • the priority may be set in at least one feature area, data related to an image, or context information.
  • the processor may determine the first emotion information based on at least one feature area, image-related data, or context information having a higher priority.
  • the processor may receive at least one voice data through a microphone (eg, the audio processing circuit 250 of FIG. 2). For example, at least one voice data may be received from at least one user.
  • the processor may analyze the received at least one voice data, and based on this, determine second emotion information for each of the at least one voice data.
  • the second emotion information may include at least one of happiness, anger, anxiety, sadness, boredom, disgust, or daily life.
  • the processor may select at least one of speech content for each of the received at least one voice data, a user's voice tone, pitch, frequency, or sound pressure included in each of the at least one voice data. Can be analyzed.
  • the processor may generate at least one graphic object to be synthesized in the image based on at least one of the first emotion information or the second emotion information in operation 313.
  • at least one graphic object is applied with attributes related to at least one voice data generated based on stickers, emojis, and/or second emotion information corresponding to at least one of the first emotion information or the second emotion information.
  • the text related to the at least one voice data generated based on the second emotion information may include text to which at least one of attributes corresponding to the second emotion information, such as font size, style, color, thickness, or motion, is applied.
  • the processor may synthesize at least one of the first emotion information and the second emotion information in the image based on the emotion information having the highest priority. You can create a graphic object.
  • the processor may display the text to which the attribute is applied on the display. In one embodiment, when there are multiple texts to which an attribute has been applied, the processor may detect a user input selecting one of the plurality of texts. The processor may synthesize text selected by a user input into an image.
  • the text to which the attribute is applied may be customized by the user.
  • At least one graphic object corresponding to emotion information may be generated through customizing the text to which the attribute is applied.
  • the processor may determine an area to synthesize at least one graphic object generated among the at least one feature area based on the content of speech of the at least one voice data. For example, if the speech content of the at least one voice data is content related to the mouth (eg, “delicious”), the processor determines, for example, the mouth region as the region to synthesize the at least one graphic object generated from among the at least one feature region. Can. For another example, if the speech content of the at least one voice data is content related to the mouth (eg “delicious”), and content related to the nose (eg “smells good”), the processor may, for example, select one of the at least one feature area, for example. , The mouth region and the nose region may be determined as regions to synthesize at least one generated graphic object.
  • the processor may determine an area selected by the user among at least one feature area as an area to synthesize at least one graphic object in an image.
  • the processor may synthesize at least one graphic object generated at a position in the image corresponding to the determined region.
  • the processor may store an image in which at least one graphic object is synthesized in a memory.
  • operation 315 is described, but is not limited thereto.
  • the processor may further receive voice data from the user.
  • the processor may generate at least one graphic object based on the first emotion information, and provide a list thereof.
  • the processor may generate at least one graphic object based on the second emotion information and provide a list for it.
  • the processor may determine the third emotion information based on the first emotion information and the second emotion information.
  • the processor may generate at least one graphic object related to the third emotion information based on the third emotion information, and provide a list for it.
  • the user can select a graphic object to be synthesized in the image from a list corresponding to each emotion information.
  • the third emotion information is determined by fusion of the first emotion information and the second emotion information, and a graphic object is provided, an optimal graphic object for at least one object recognized in the image and the image may be obtained.
  • FIG. 4 is a flowchart 400 illustrating a method of providing a graphic object corresponding to emotion information according to various embodiments.
  • a processor may display an image in operation 401.
  • the emotion information may be determined based on at least one of data related to an image, context information, or information related to a user.
  • the data related to the image may include audio information included in the image and/or motion information of at least one object included in the image.
  • the context information may include the current time, current location, and/or event schedule.
  • the information related to the user may include at least one biometric information, at least one gesture information, and/or at least one voice data.
  • the processor may determine an area in the image to synthesize the at least one graphic object. For example, the processor may determine an area in which to place the at least one graphic object based on a user input selecting an area to synthesize at least one graphic object in the image.
  • the processor may provide one or more graphic objects corresponding to emotion information to be synthesized based on the determined characteristics of the region. For example, one or more provided graphic objects may be provided differently based on the determined region in the image. If the determined region in the image is the eye region, the processor may provide eye images corresponding to emotion information. If the area in the determined image is an area in which the animation character object is displayed, the processor may provide corresponding character images in which emotion information is reflected in the animation character object. If the determined region in the image is a region displaying a background image, the processor may determine the color, brightness, and/or transparency of the background image excluding the foreground based on emotion information and provide it.
  • the processor may detect an input for selecting at least one graphic object to be synthesized in an image from among the provided one or more graphic objects.
  • the at least one graphic object selected at the location in the image corresponding to the determined area may be synthesized, and in operation 413, the image obtained by combining the at least one graphic object may be stored in the memory.
  • the processor may determine emotion information based on at least one object selected by a user input.
  • the processor may generate at least one graphic object corresponding to the determined emotion information, and synthesize the generated at least one graphic object in an image.
  • the processor may determine a plurality of emotion information based on the plurality of objects when there are a plurality of objects selected by the user input.
  • the processor may provide at least one graphic object to be synthesized in an image generated based on each of the determined emotion information.
  • the processor may provide at least one graphic object generated based on the first emotion information and at least one graphic object generated based on the second emotion information.
  • the processor may synthesize the selected graphic object in the image in response to an input of selecting one of the at least one graphic object generated based on each emotion information provided.
  • the processor may segment an image into a specific number of regions, and recognize one or more objects included in the divided region.
  • the processor may determine emotion information based on each of the recognized one or more objects.
  • the processor may provide at least one graphic object to be synthesized in the divided region based on each of the determined emotion information.
  • the processor may synthesize the selected graphic object in the corresponding area in response to detection of an input for selecting one of the at least one graphic object generated based on each emotion information for each area provided.
  • the processor may determine the emotion information based on each of the one or more objects recognized in the image, and then receive a plurality of voice data.
  • the processor may segment the received plurality of voice data. For example, the processor may divide the received plurality of voice data based on the speech rate, sound tone, pitch, frequency, and/or sound pressure for each of the received plurality of voice data.
  • the processor may determine emotion information for each of the plurality of voice data based on at least one of speech content, sound tone, pitch, frequency, or sound pressure of each of the received plurality of voice data.
  • the processor may provide at least one graphic object based on the emotion information for each of the determined voice data.
  • the processor may generate at least one graphic object corresponding to emotion information for each of the determined voice data.
  • the processor may determine at least one graphic object to be synthesized based on emotion information of each of the one or more objects, and synthesize at least one graphic object determined in a corresponding feature area of each of the one or more objects.
  • a user input for selecting at least one graphic object among the plurality of graphic objects may be sensed.
  • the processor may synthesize the selected at least one graphic object in the feature area of each of the recognized one or more objects in response to a user input.
  • FIG. 5 is a diagram 500 illustrating a method of determining emotion information by extracting a feature region for at least one object recognized from an image according to various embodiments.
  • At least one feature region (eg, at least one of a human 550, an object 560, or an animal 570) that is a reference for determining emotion information (at least one feature region)
  • characteristics of the eye area, nose area, and/or mouth area may be stored in a memory (eg, the memory 230 of FIG. 2).
  • the characteristics of the at least one feature region may include at least one of the shape of the eye (eg, the degree of sagging of the eye, the focus of the eye, the size of the eye, and the wrinkles of the eye) or the shape of the mouth.
  • the processor eg, the processor 270 of FIG.
  • emotional information may include sadness 510, angry 520, surprise 530, and/or happiness 540.
  • emotion information may include anxiety, boredom, and/or hate.
  • FIG. 6 is a diagram 600 illustrating a method of generating at least one graphic object based on determined emotion information according to various embodiments.
  • a graphic object generated based on emotion information and attributes of the graphic object may be stored in a memory (eg, the memory 230 of FIG. 2 ).
  • the processor eg, the processor 270 of FIG. 2
  • the processor may include a graphic object based on emotion information stored in memory and emotion information 650 determined based on the attributes of the graphic object, such as happiness 610, surprise ( Font style 660, color 670, motion 680, or sticker 690 corresponding to each emotion information according to surprised 620, angry 630, or sad 640 ).
  • FIG. 7 is a diagram 700 illustrating a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
  • the processor (eg, the processor 270 in FIG. 2) recognizes the face 713 of the object 711 included in the image, as shown in reference numerals ⁇ 710> and ⁇ 720>, At least one feature region, for example, an eye region 721 and a mouth region 723 may be extracted from the recognized face 713.
  • the processor may determine emotion information on the recognized object 711 based on the extracted eye region 721 and the mouth region 723. For example, based on the shape of the extracted eye region 721 and the mouth region 723, the processor may determine emotion information, for example, as “angry”.
  • the processor may analyze the background 731 of the image (eg, an area excluding the object 711 recognized in the image) as illustrated by reference numeral ⁇ 730>.
  • the processor may analyze the background 731 of the image and determine an atmosphere for the image based on this.
  • the processor may determine the mood for the image based on the color (or saturation, brightness) of the background 731.
  • the present invention is not limited to this, and the processor includes the types of objects included in the image (for example, objects such as weapons and skeletons, objects such as animals and dolls), and the movement of the objects (for example, fast movement and slow movement), and the frequency of the image.
  • the mood for the image may be determined based on the components (eg, edge, high-frequency component of the boundary line, low-frequency component), and/or text included in the image. For example, if an object such as a weapon or a skeleton is recognized, or if the movement of the object exceeds a specified speed, or if the frequency component of the image is a high frequency component, or if the text contained in the image is text representing negative emotions, the processor sends emotion information. You can decide to be “horror” or “anger”.
  • the processor displays the emotion information as “happy. ”, and/or “pleasure”.
  • the atmosphere determined based on the image in FIG. 7 according to an embodiment will be described on the assumption that it is a creepy and scary atmosphere.
  • the processor includes an object 711 recognized from an image, at least one feature region 721, 723 for the object 711, determined emotional information (eg angry), and mood (eg spooky and scary)
  • the atmosphere may be stored in a memory (eg, the memory 230 of FIG. 2).
  • the processor is based on the extracted eye 721 and the shape of the mouth 723, at least one graphic object based on the emotion information (eg, angry) determined and the determined atmosphere (eg, spooky and scary atmosphere) Can generate
  • the processor may receive voice data such as “watch out” 735 through the microphone.
  • the processor may analyze the received voice data, such as the content of the “watch out” 735, the characteristics of the user voice included in the voice data (eg, tone, pitch, frequency, and/or sound pressure).
  • the processor determines emotion information for the input voice data based on the content of the analyzed voice data and characteristics of the user's voice, as shown in reference numeral ⁇ 740>, and based on this, the at least one graphic An object 741 may be generated.
  • the processor may determine, for example, “angry” of emotion information based on a result of analyzing the voice data.
  • the processor may display at least one graphic object 751 to which an attribute corresponding to the generated emotion information is applied as shown in reference numeral ⁇ 750>.
  • the attribute may include at least one of font size, style, color, thickness, or motion.
  • the at least one graphic object to which the attribute is applied may include voice data, for example, text generated based on “watch out” 735, stickers generated based on emotion information and mood.
  • the processor may display a confirmation button 753 with at least one graphic object 751 to which an attribute corresponding to emotion information is applied.
  • the processor is based on at least one graphic object to which an attribute is applied, for example, emotion information (eg, anger) for voice data as shown in reference numeral ⁇ 760>.
  • Generated text 751 e.g. font with sharp angled borders and text “watch out” with red
  • each of the text 751 and the sticker 761 may be synthesized by being disposed in a corresponding feature region, for example, the mouth region 723 and the eye region 721 among at least one extracted feature region.
  • an image in which at least one graphic object to which the attribute of reference numeral ⁇ 760> is applied is displayed is displayed.
  • the processor displays reference number ⁇ 760>.
  • An image obtained by combining at least one graphic object to which an attribute is applied may be displayed.
  • the processor may display a button for customizing the graphic object 751 together with at least one graphic object 751 to which attributes corresponding to emotion information are applied. At least one graphic object corresponding to emotion information intended by the user may be accurately generated through a user input of selecting a button for customizing.
  • FIG. 8 is a diagram 800 for describing a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
  • the processor (eg, the processor 270 of FIG. 2) recognizes the face of the object included in the image as shown in reference numeral ⁇ 810>, and at least one feature area, for example, from the recognized face , The eye region 811, the nose region 813, and/or the mouth region 815 may be extracted.
  • the processor may receive voice data, such as “smells delicious” 817 from the user.
  • the processor may include at least one graphic generated based on voice data such as “smells delicious” 817 as shown in reference numeral ⁇ 820>, such as attributes (eg, font size, style, color, thickness, and/or
  • attributes eg, font size, style, color, thickness, and/or
  • the first graphic object 821, the second graphic object 823, and the third graphic object 825 to which different motions are applied may be displayed on the display.
  • the processor detects a user input 827 for selecting a first graphic object 821 from among the displayed first graphic object 821, the second graphic object 823, and the third graphic object 825. can do.
  • the processor may determine a position in the image to place the selected first graphic object 821 as shown in reference numeral ⁇ 830>. have.
  • the processor may analyze the utterance content of the voice data, and determine a position to place the selected first graphic object 821 in the image.
  • the processor analyzes the voice data “smells delicious” 817, and “smells” is placed 831 in the feature region, eg, nose region 813, and “delicious” is the feature region, eg, mouth region 815. It can be decided to be placed 833.
  • the processor may receive an input for setting a position in an image in which a selected graphic object is to be placed.
  • the processor may receive inputs 841 and 843 for setting motions of the first graphic object 821, for example, “smells” and “delicious”, as shown at reference numeral ⁇ 840>. .
  • the processor may set motions of each of the first graphic object 821, for example, “smells” and “delicious” based on the inputs 841 and 843 for setting the received motion.
  • the processor images the first graphic object 821 on the basis of the motions 851 and 853 set in the first graphic object 821, for example, “smells” and “delicious”, as shown in reference numeral ⁇ 850>. Can be synthesized in.
  • 9A and 9B are diagrams for explaining a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments (900) )to be.
  • the processor (eg, the processor 270 of FIG. 2) includes the first face 913 and the second object of the first object 911 included in the image as shown in reference numeral ⁇ 910>.
  • the second face 917 of 915 may be recognized.
  • the processor may include at least one feature area, for example, an eye area, a nose area, from the second face 917 of the first face 913 and the second face 917 recognized as illustrated by reference numeral ⁇ 920>. And/or the mouth region can be extracted.
  • the processor may determine emotion information for the second object 915 based on the extracted at least one feature region. For example, emotion information determined based on the extracted at least one feature region is described assuming “anger”.
  • the processor may receive first voice data, eg, “what are you looking at” 925. At least one of the second faces 917 of the second object 915 is based on the received first voice data, for example, “what are you looking at” 925, as shown in reference numeral ⁇ 930>.
  • a graphic object 931 of can be generated and displayed. For example, the at least one graphic object may include text for the received voice data.
  • the processor may determine emotion information on the first voice data based on at least one of the utterance content, sound tone, pitch, frequency, and sound pressure of the received first voice data. For example, the emotion information for the first voice data will be described assuming “tension” and “angry”.
  • the processor determines the attribute (eg, font size, style, color, thickness, or motion) of the text for the first voice data based on the emotion information (eg, tension, anger) about the first voice data, and , The text “what are you looking at” (931) to which the determined attribute is applied may be displayed.
  • the attribute eg, font size, style, color, thickness, or motion
  • the processor is configured to generate second voice data such as “nothing... ”935.
  • the second voice data may be received after the first voice data is received and a designated time has elapsed.
  • the processor may determine emotion information for the second voice data based on at least one of the utterance content, acoustic tone, pitch, frequency, or sound pressure of the received second voice data.
  • the emotion information for the second voice data is described assuming “worried”.
  • the processor determines the attribute of the text for the second speech data based on the emotion information (eg, worry) about the second speech data, and the text “nothing...” to which the determined attribute is applied is applied. ”941 can be displayed.
  • the processor generates at least one graphic object (eg, text and/or stickers) among at least one feature area based on the utterance content of the voice data and the determined emotion information, as indicated by reference numeral ⁇ 950>. )
  • the processor may include at least one graphic object created in the determined area (for example, text “what are you looking at” (931) for the first voice data, text “nothing...” (941) for the second voice data, and/or Alternatively, the sticker 951 corresponding to the emotion information determined based on the feature region for the second object 915 may be synthesized.
  • voice data for example, first voice data and second voice data are received from a user at designated time intervals, but is not limited thereto.
  • the processor may simultaneously receive voice data from each of a plurality of users.
  • the processor displays the first face 963 of the first object 961 and the second face 957 of the second object 965 included in the image as shown in reference numeral ⁇ 960>.
  • the processor may include at least one feature region (eg, eye region, nose region, and/or mouth region) extracted from the recognized first face 963 and second face 967 as shown in reference numeral ⁇ 970>.
  • the processor receives the first voice data “what are you looking at” 973 from a plurality of users, for example, the first user, and the second voice data “nothing...” from the second user. ”(977).
  • the processor may determine emotion information for the first voice data and the second voice data based on at least one of the utterance content, sound tone, pitch, frequency, and sound pressure of the received first voice data and the second voice data.
  • the processor includes first voice data such as “what are you looking at” 973 and second voice data such as “nothing... Based on ”977, at least one graphic object may be generated and displayed.
  • the at least one graphic object is text and second voice data for the first voice data to which the attribute of the text determined based on emotion information (eg, at least one of font size, style, color, thickness, or motion) is applied. May contain text for
  • the processor may compare emotion information for the first object 961 and the second object 965 with emotion information for the first voice data and the second voice data.
  • the operation of comparing emotion information may be an operation of analyzing whether similarity of emotion information and/or emotion information of the same category (eg, positive emotion information, negative emotion information).
  • the processor may determine an area in the image in which at least one graphic object generated based on the first audio data and the second audio data is to be placed.
  • emotion information for the first object 961 is “sadness”
  • emotion information for the second object 965 is “angry”
  • emotion information for the first voice data is “angry” and the second voice data.
  • the emotional information about the home is explained assuming “worries”.
  • the processor determines the text for the second voice data having the emotion information “worry” as an area in which the feature area of the first object 961 having the emotion information “sadness” is arranged, and the first having the emotion information “angry”.
  • the text for voice data may be determined as an area in which a feature area of the second object 965 having emotion information “angry” is to be arranged.
  • the processor includes at least one graphic object (eg, text “what are you looking at” (981) for the first voice data) (981) generated in the determined area as shown in reference numeral ⁇ 980>, the second The text “nothing...” 983 for voice data, and/or the sticker 985 corresponding to the emotion information determined based on the feature area for the second object 965 may be synthesized and displayed.
  • graphic object eg, text “what are you looking at” (981) for the first voice data
  • the second The text “nothing...” 983 for voice data and/or the sticker 985 corresponding to the emotion information determined based on the feature area for the second object 965 may be synthesized and displayed.
  • FIG. 10 is a diagram 1000 illustrating a method of synthesizing at least one graphic object generated at a position corresponding to a feature region of at least one object based on determined emotion information according to various embodiments.
  • a processor (eg, the processor 270 of FIG. 2) includes a plurality of objects included in an image, for example, a first face of the first object 1011 (as shown in reference number ⁇ 1010>). 1013) and the second face 1017 of the second object 1015 may be recognized.
  • the processor may receive voice data, eg, “welcome back” 1019.
  • the processor generates at least one graphic object based on the received voice data, for example, “welcome back” 1019, as shown in reference numeral ⁇ 1020>, and displays the generated at least one graphic object 1021 on the display.
  • the at least one graphic object may include text for the received voice data.
  • the attribute of text for voice data may be determined based on at least one of speech content, acoustic tone, pitch, frequency, or sound pressure of the received voice data. For example, emotion information determined based on voice data will be described assuming “happy”.
  • the processor may separate text for speech data based on the number of recognized objects, as indicated by reference numeral ⁇ 1030>. For example, text may be separated into letters constituting a word. For example, the graphic object 1031 to be placed on the first face 1013 of the first object 1011 is “welcome”, and the graphic object 1033 to be placed on the second face 1017 of the second object 1015 Can be determined as “back”.
  • the processor provides at least one graphic object, eg, stickers 1041, 1043, 1045, corresponding to emotion information, for example, “happiness” determined based on voice data, as shown at reference numeral ⁇ 1040>. It can be synthesized on the first face 1013 of one object 1011 and the second face 1017 of the second object 1015.
  • the processor displays each of the graphic objects 1031 “welcome” and the graphic objects 1033 “back” together with the stickers 1041, 1043, 1045 as shown in reference numeral ⁇ 1050>, for example,
  • the first face 1013 of the first object 1011 and the second face 1017 of the second object 1015 may be synthesized.
  • FIG. 11 is a diagram 1100 illustrating a signal flow between electronic devices that provide a graphic object corresponding to emotion information according to various embodiments.
  • the performance of the external electronic device 1120 may be higher than that of the electronic device 1110, and the high performance external electronic device 1120 analyzes information related to an image and analyzes the results Can be provided to the electronic device 1110.
  • the electronic device 1110 may display an image in operation 1130.
  • the image may include at least one image photographed from at least one camera (eg, the camera 220 of FIG. 2 ), and an image stored in a memory (eg, the memory 230 of FIG. 2 ). .
  • the electronic device 1110 may determine information related to an image to be analyzed in operation 1135.
  • the information related to the image to be analyzed is a second image different from the image displayed in operation 1130, audio information included in the image, text information included in the image, biometric information of the user of the electronic device 1110, or context information It may include at least one of.
  • the second image is an image including an image displayed in operation 1130 (eg, one of a frame image of the image, an image including at least a part of a face, body, and object in the first image), a barcode, and a two-dimensional barcode ( Examples: QR code, data matrix code), markers, or text images.
  • the audio information may include audio, audio signal, and/or video audio track information received by the electronic device 1110.
  • Biometric information is biometric information (eg, HRM, HRV, SPO2, blood pressure, blood sugar, EEG, skin resistance, electromyogram, electrocardiogram, respiratory rate per minute, or body temperature) received through a biometric sensor of the electronic device 1110, electronic device Biometric information (eg, blood flow change image) obtained by analyzing an image through a camera of 1110, and/or body pose or body motion information by a motion sensor may be included.
  • the text information may include text input by a pen input, a virtual keyboard, or a keypad, a symbol, a symbol, a caption track of an image, associated caption file information, or subtitle information included or associated with music content.
  • the context information includes current time, current location, and schedule related information (eg, birthday, Christmas) for the electronic device 1110, profile information (eg, preference, food, personal information) for the user of the electronic device 1110 (Eg, gender, age, visit information).
  • profile information eg, preference, food, personal information
  • emotion analysis target information may be included in an image or may be separately received (eg, voice data).
  • the electronic device 1110 may transmit information related to an image to be analyzed to the external electronic device 1120 in operation 1140.
  • the external electronic device 1120 analyzes information related to an image received from the electronic device 1110 in operation 1145 and determines emotion information based on the information. For example, the external electronic device 1120 is based on at least one of a second image different from the displayed image, audio information included in the image, text information included in the image, biometric information of the user of the electronic device 1110, or context information. Emotion information.
  • the external electronic device 1120 may determine at least one graphic object based on the determined emotion information in operation 1150. In another embodiment, the at least one graphic object may be determined based on the preferences of the user of the electronic device 1110 or another user.
  • the external electronic device 1120 may transmit the determined emotion information and at least one graphic object to the electronic device 1110 in operation 1155.
  • the electronic device 1110 may determine a position to synthesize at least one graphic object in the image.
  • the electronic device 1110 may determine a graphic object to be synthesized in an image among the received at least one graphic object, and in operation 1170, the determined graphic object may be synthesized at the determined location.
  • FIG. 12 is a diagram 1200 illustrating signal flow between electronic devices that provide graphic objects corresponding to emotion information according to various embodiments.
  • the first external electronic device 1210 receives analysis target information and provides the received analysis target information to the electronic device 1220 (eg, the electronic device 1110 of FIG. 11 ).
  • the first external electronic device 1210 may include a smart watch, head mounted display (HMD), augmented reality glass (AR glass), drone, robot, and Internet of Things (IoT) device.
  • the first external electronic device 1210 may recognize and process image or audio information, and may also receive and transmit the image to the electronic device 1220.
  • the performance of the second external electronic device 1230 is higher than that of the electronic device 1220 and the first external electronic device 1210.
  • the second external electronic device 1230 having high performance may analyze information related to an image and provide the analyzed result to the electronic device 1220.
  • information related to an image may include an image, audio, sound-to-text (STT), barcode, marker, and emotion information (or emotion, mood).
  • the second external electronic device 1230 may manage preference information based on a user profile received from electronic devices and provide a graphic object based on the preference information.
  • the second external electronic device 1230 may store and manage graphic objects and transmit the graphic object to the electronic device 1220.
  • the determined candidate graphic object and image may be received to perform synthesis and editing operations.
  • the second external electronic device 1230 includes a server, a smart home hub device, a cloud computing device, a mobile edge comp. (MEC), an access point device, an Internet message server (IMS), and multimedia streaming It may include either a server or a video and video communication host device.
  • MEC mobile edge comp.
  • IMS Internet message server
  • multimedia streaming It may include either a server or a video and video communication host device.
  • the first external electronic device 1210 may obtain context information about the first external electronic device 1210.
  • the context information may include image information, audio information, text data, and/or biometric information obtained from the first external electronic device 1210.
  • the electronic device 1220 may display an image in operation 1243.
  • the first external electronic device 1210 may transmit the obtained context information to the electronic device 1220 in operation 1245.
  • the electronic device 1220 may analyze at least one of at least one feature region extracted from the image, data related to the image, and context information obtained from the first external electronic device 1210 in operation 1247. have. In operation 1249, the electronic device 1220 may transmit the analysis result to the second external electronic device 1230.
  • the second external electronic device 1230 may determine emotion information based on an analysis result received from the electronic device 1220 in operation 1251.
  • the electronic device 1220 analyzes at least one of at least one feature region extracted from an image, data related to the image, and context information obtained from the first external electronic device 1210 and then analyzes the result. It has been described as transmitting to the second external electronic device 1230, but is not limited thereto.
  • the electronic device 1220 determines at least one of at least one feature region extracted from the image, data related to the image, and context information obtained from the first external electronic device 1210 as information related to the image to be analyzed and removes it. 2 to the external electronic device 1230.
  • the second external electronic device 1230 analyzes information related to an image received from the electronic device 1220 and determines emotion information based on the information.
  • the second external electronic device 1230 may transmit the determined emotion information to the electronic device 1220 in operation 1253.
  • the electronic device 1220 may determine a location to display the graphic object in the image. In operation 1257, the electronic device 1220 may request the second external electronic device 1230 a graphic object to be synthesized in the image.
  • the second external electronic device 1230 determines at least one graphic object based on the determined emotion information in operation 1259, and in step 1261, determines the determined at least one graphic object to the electronic device 1220. Can transmit.
  • the electronic device 1220 may determine a graphic object to be synthesized in an image among at least one graphic object received from the second external electronic device 1230. In operation 1265, the electronic device 1220 may request the second external electronic device 1230 to synthesize the determined graphic object in the image.
  • the second external electronic device 1230 may synthesize the graphic object determined at the determined location in the image in operation 1267, and transmit the synthesized graphic object image to the electronic device 1220 in operation 1269. .
  • the electronic device 1220 may display an image in which a graphic object received from the second external electronic device 1230 is synthesized.
  • the processor may include at least one feature region extracted for at least one object recognized in the image, data related to the image, context information, or information related to the analyzed voice data.
  • Emotion information may be determined based on at least one, and at least one graphic object may be generated based on the determined emotion information.
  • the processor may synthesize at least one object generated based on emotion information into an image and store it in a memory so that it can communicate with other users.
  • the text applied with attributes corresponding to at least one feature area, image-related data and context information as well as emotion information related to received voice data is synthesized, the user's emotion can be more richly transmitted to other users.
  • the electronic device may be various types of devices.
  • the electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device.
  • a portable communication device eg, a smart phone
  • a computer device e.g., a smart phone
  • a portable multimedia device e.g., a portable medical device
  • a camera e.g., a portable medical device
  • a camera e.g., a portable medical device
  • a camera e.g., a portable medical device
  • a wearable device e.g., a smart bracelet
  • a home appliance device e.g., a home appliance
  • any (eg first) component is referred to as a “coupled” or “connected” to another (eg second) component, with or without the term “functionally” or “communically” If mentioned, it means that any of the above components can be connected directly to the other components (eg by wire), wirelessly, or through a third component.
  • module may include units implemented in hardware, software, or firmware, and may be used interchangeably with terms such as, for example, logic, logic blocks, components, or circuits.
  • the module may be an integrally configured component or a minimum unit of the component or a part thereof performing one or more functions.
  • the module may be implemented in the form of an application-specific integrated circuit (ASIC).
  • ASIC application-specific integrated circuit
  • Various embodiments of the present disclosure may include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). It may be implemented as software (eg, the program 140) including them.
  • a processor eg, processor 120
  • a device eg, electronic device 101
  • the one or more instructions may include code generated by a compiler or code executable by an interpreter.
  • the storage medium readable by the device may be provided in the form of a non-transitory storage medium.
  • non-transitory only means that the storage medium is a tangible device, and does not include a signal (eg, electromagnetic waves), and this term is used when data is stored semi-permanently in a storage medium. It does not distinguish between temporary storage cases.
  • a signal eg, electromagnetic waves
  • a method according to various embodiments disclosed in this document may be provided as being included in a computer program product.
  • Computer program products can be traded between sellers and buyers as products.
  • the computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store TM ) or two user devices ( For example, it can be distributed directly between smartphones) and online (eg, downloaded or uploaded).
  • a portion of the computer program product may be stored at least temporarily on a storage medium readable by a device such as a memory of a manufacturer's server, an application store's server, or a relay server, or may be temporarily generated.
  • each component (eg, module or program) of the above-described components may include a singular or a plurality of entities.
  • one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added.
  • a plurality of components eg, modules or programs
  • the integrated component may perform one or more functions of each component of the plurality of components the same or similar to that performed by the corresponding component among the plurality of components prior to the integration. .
  • operations performed by a module, program, or other component may be executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be executed in a different order, or omitted Or, one or more other actions can be added.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Computational Linguistics (AREA)
  • Multimedia (AREA)
  • Acoustics & Sound (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Signal Processing (AREA)
  • Psychiatry (AREA)
  • Hospice & Palliative Care (AREA)
  • General Health & Medical Sciences (AREA)
  • Child & Adolescent Psychology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Engineering & Computer Science (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente invention concerne, selon divers modes de réalisation, un dispositif électronique pouvant comprendre : un dispositif d'affichage ; un microphone ; un processeur connecté de manière fonctionnelle au dispositif d'affichage et au microphone ; et une mémoire connectée de manière fonctionnelle au processeur, la mémoire stockant des instructions qui, lorsqu'elles sont exécutées, amènent le processeur à : afficher une image sur le dispositif d'affichage ; reconnaître au moins un objet dans l'image affichée ; déduire une ou plusieurs zones caractéristiques dudit objet reconnu ; lorsqu'au moins un élément de données vocales est reçu par le biais du microphone, déterminer des premières informations d'émotion sur la base desdites données vocales reçues ; générer au moins un objet graphique associé audit élément de données vocales sur la base des premières informations d'émotion ; sélectionner une zone dans laquelle combiner ledit objet graphique généré, à partir de la ou des zones caractéristiques sur la base de contenus d'énoncé du ou des éléments de données vocales ; et combiner ledit objet graphique généré avec la zone sélectionnée de telle sorte que celui-ci se trouve à un emplacement correspondant à la zone sélectionnée. En plus de divers modes de réalisation décrits ici, divers autres modes de réalisation sont également possibles.
PCT/KR2020/001161 2019-01-24 2020-01-23 Dispositif électronique et procédé pour fournir un objet graphique correspondant à des informations d'émotion en utilisant celui-ci Ceased WO2020153785A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR10-2019-0009477 2019-01-24
KR1020190009477A KR102667547B1 (ko) 2019-01-24 2019-01-24 전자 장치 및 이를 이용한 감정 정보에 대응하는 그래픽 오브젝트를 제공하는 방법

Publications (1)

Publication Number Publication Date
WO2020153785A1 true WO2020153785A1 (fr) 2020-07-30

Family

ID=71736404

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2020/001161 Ceased WO2020153785A1 (fr) 2019-01-24 2020-01-23 Dispositif électronique et procédé pour fournir un objet graphique correspondant à des informations d'émotion en utilisant celui-ci

Country Status (2)

Country Link
KR (1) KR102667547B1 (fr)
WO (1) WO2020153785A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022127486A1 (fr) * 2020-12-17 2022-06-23 展讯通信(天津)有限公司 Procédé et appareil de commutation de thème d'interface, terminal et support de stockage
CN115238111A (zh) * 2022-06-15 2022-10-25 荣耀终端有限公司 一种图片显示方法及电子设备

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112053370A (zh) * 2020-09-09 2020-12-08 脸萌有限公司 基于增强现实的显示方法、设备及存储介质
KR20230013987A (ko) * 2021-07-20 2023-01-27 삼성전자주식회사 전자 장치 및 실시간 메시지 생성 방법
KR20230021924A (ko) 2021-08-06 2023-02-14 삼성전자주식회사 전자 장치 및 전자 장치에서 복수의 상태들의 우선 순위에 기반한 캐릭터 오브젝트 표시 방법
CN115035898B (zh) * 2022-05-12 2025-08-26 广州市迪声音响有限公司 用于汽车音响功放器的控制系统及方法
KR102860087B1 (ko) 2023-04-26 2025-09-15 주식회사 엘지유플러스 딥 러닝에 기반한 이미지 처리 장치 및 방법

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0970030A (ja) * 1995-08-31 1997-03-11 Mitsubishi Electric Corp 仮想会議システムの画像表示方法並びに仮想会議用端末装置
KR20070006337A (ko) * 2005-07-08 2007-01-11 삼성전자주식회사 휴대단말기의 이미지편집 방법
JP2010066844A (ja) * 2008-09-09 2010-03-25 Fujifilm Corp 動画コンテンツの加工方法及び装置、並びに動画コンテンツの加工プログラム
US20100094634A1 (en) * 2008-10-14 2010-04-15 Park Bong-Cheol Method and apparatus for creating face character based on voice
JP2012191662A (ja) * 2012-06-15 2012-10-04 Casio Comput Co Ltd 電子データ編集装置、電子データ編集方法及びプログラム
KR101189053B1 (ko) * 2009-09-05 2012-10-10 에스케이플래닛 주식회사 아바타 기반 화상 통화 방법 및 시스템, 이를 지원하는 단말기

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002279443A (ja) * 2001-03-16 2002-09-27 Minolta Co Ltd 映像再生装置、テレビ放送システム、映像送信装置、および映像データ送信方法
KR20130084543A (ko) * 2012-01-17 2013-07-25 삼성전자주식회사 사용자 인터페이스 제공 장치 및 방법
KR20170027589A (ko) * 2015-09-02 2017-03-10 삼성전자주식회사 기능 제어 방법 및 그 방법을 처리하는 전자 장치

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0970030A (ja) * 1995-08-31 1997-03-11 Mitsubishi Electric Corp 仮想会議システムの画像表示方法並びに仮想会議用端末装置
KR20070006337A (ko) * 2005-07-08 2007-01-11 삼성전자주식회사 휴대단말기의 이미지편집 방법
JP2010066844A (ja) * 2008-09-09 2010-03-25 Fujifilm Corp 動画コンテンツの加工方法及び装置、並びに動画コンテンツの加工プログラム
US20100094634A1 (en) * 2008-10-14 2010-04-15 Park Bong-Cheol Method and apparatus for creating face character based on voice
KR101189053B1 (ko) * 2009-09-05 2012-10-10 에스케이플래닛 주식회사 아바타 기반 화상 통화 방법 및 시스템, 이를 지원하는 단말기
JP2012191662A (ja) * 2012-06-15 2012-10-04 Casio Comput Co Ltd 電子データ編集装置、電子データ編集方法及びプログラム

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022127486A1 (fr) * 2020-12-17 2022-06-23 展讯通信(天津)有限公司 Procédé et appareil de commutation de thème d'interface, terminal et support de stockage
CN115238111A (zh) * 2022-06-15 2022-10-25 荣耀终端有限公司 一种图片显示方法及电子设备
CN115238111B (zh) * 2022-06-15 2023-11-14 荣耀终端有限公司 一种图片显示方法及电子设备

Also Published As

Publication number Publication date
KR20200092207A (ko) 2020-08-03
KR102667547B1 (ko) 2024-05-22

Similar Documents

Publication Publication Date Title
WO2020153785A1 (fr) Dispositif électronique et procédé pour fournir un objet graphique correspondant à des informations d'émotion en utilisant celui-ci
WO2020171540A1 (fr) Dispositif électronique permettant de fournir un mode de prise de vue sur la base d'un personnage virtuel et son procédé de fonctionnement
WO2019164266A1 (fr) Dispositif électronique permettant de produire une image contenant un avatar 3d reflétant le mouvement du visage grâce à un avatar 3d correspondant au visage et procédé de fonctionnement de celui-ci
KR102667064B1 (ko) 전자 장치 및 이를 이용한 카메라 기능과 연동하여 이모지 편집을 위한 사용자 인터페이스를 제공하는 방법
US20200402304A1 (en) Electronic device and method for managing custom object on basis of avatar
WO2021172832A1 (fr) Procédé de modification d'image basée sur la reconnaissance des gestes, et dispositif électronique prenant en charge celui-ci
WO2021242005A1 (fr) Dispositif électronique et procédé de génération d'autocollant d'émoji basés sur un avatar d'utilisateur
WO2020130691A1 (fr) Dispositif électronique et procédé pour fournir des informations sur celui-ci
WO2020171621A1 (fr) Procédé de commande d'affichage d'avatar et dispositif électronique associé
US10409324B2 (en) Glass-type terminal and method of controlling the same
WO2021020814A1 (fr) Dispositif électronique de mise en place d'avatar et son procédé d'exploitation
EP3616050A1 (fr) Appareil et procédé pour contexte de commande vocale
WO2018124633A1 (fr) Dispositif électronique et procédé de délivrance d'un message par celui-ci
KR20180057366A (ko) 이동 단말기 및 그 제어방법
WO2020130281A1 (fr) Dispositif électronique et procédé de fourniture d'un avatar sur la base de l'état émotionnel d'un utilisateur
WO2019125029A1 (fr) Dispositif électronique permettant d'afficher un objet dans le cadre de la réalité augmentée et son procédé de fonctionnement
EP3779660B1 (fr) Appareil et procédé d'affichage d'éléments graphiques en fonction d'un objet
WO2020159302A1 (fr) Dispositif électronique permettant d'assurer diverses fonctions dans un environnement de réalité augmentée et procédé de fonctionnement associé
EP3655204A1 (fr) Dispositif électronique apte à se déplacer et son procédé de fonctionnement
WO2023128469A1 (fr) Dispositif électronique fournissant un effet d'image basé sur une image et son procédé de commande
WO2021149930A1 (fr) Dispositif électronique et procédé de création d'histoire
WO2020085718A1 (fr) Procédé et dispositif de génération d'avatar sur la base d'une image corrigée
WO2019088338A1 (fr) Dispositif électronique et procédé de commande associé
WO2022215823A1 (fr) Procédé et dispositif de génération d'image
CN111292743B (zh) 语音交互方法及装置、电子设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20745031

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20745031

Country of ref document: EP

Kind code of ref document: A1