JP7725435B2 - Communication System - Google Patents

Description

The present invention relates to a communication system in which a plurality of people communicate with each other two-way through sight, hearing and touch, and more particularly to a communication system using a sofa-type two-way communication device .

Recently, remote communication devices and software that enable online interpersonal communication between distant locations have become widely used. Furthermore, the use of physical communication devices and software that involve the five senses is also increasing.

Non-Patent Document 1 discloses a sofa-type embodied communication device that utilizes haptic technology in addition to audiovisual technology to enable online embodied communication across remote locations, involving the whole body, such as sitting next to someone or patting their back.

"Development of Sync Sofa, a sofa-type communication system that uses haptic technology to connect people even when they're apart," KDDI Research, Inc. Press Release 2021112501

The physical communication devices and audio-tactile technology adjustment programs disclosed in the prior art are designed for one-way sensory presentation and are unable to provide the desired two-way sensory presentation.

Specifically, in bidirectional five-sensory presentation, which simultaneously acquires and presents information from all five senses, including visual, auditory, and tactile information, there was a problem in that when one-way five-sensory presentation was performed on both sides, howling occurred due to physical interference between the devices involved in acquisition and presentation and the respective pieces of information, and the desired results were not achieved.

An object of the present invention is to solve the above technical problems and to provide an embodied communication system that can realize two-way remote communication via visual, auditory, and tactile senses using adjacent communication modes.

To achieve the above objectives, the present invention is characterized by a communication device used for remote communication, which has the following configuration:

(1) A chair-shaped device equipped with at least visual information acquisition and presentation means from among tactile information acquisition and presentation means, auditory information acquisition and presentation means, and visual information acquisition and presentation means, and the visual information presentation means is provided on at least one of one end and the other end of the chair shape.

(2) The information presented by each acquisition means is transmitted to other communication devices via a network, and each presentation means receives and presents the information acquired by each acquisition means of the other communication devices via the network.

(3) The device is provided with a first loop suppression means for switching the microphone used to acquire the auditory information to be transmitted based on the distance between the speaker that presents the received auditory information and each microphone.

(4) A first loop suppression means is provided for switching the microphone used to acquire the auditory information to be transmitted based on the distance between the speaker that presents the received auditory information and each microphone and the frequency characteristics of the received auditory information.

(5) The first loop suppression means further includes means for suppressing the presentation of auditory information.

(6) A second loop suppression means is provided that switches the acceleration sensor used to obtain the tactile information to be transmitted based on the distance between the vibration actuator that presents the received tactile information and each acceleration sensor.

(7) A second loop prevention means is provided that switches the acceleration sensor used to obtain the tactile information to be transmitted based on the distance between the vibration actuator that presents the received tactile information and each acceleration sensor and the frequency characteristics of the received tactile information.

(8) The second loop suppression means further includes means for suppressing the presentation of tactile information.

This invention makes it possible to realize remote communication involving two-way presentation of the five senses using adjacent communication modes.

Furthermore, the present invention can prevent feedback that can occur when auditory or tactile information is looped between communication devices, thereby enabling high-quality two-way communication with minimal discomfort.

1 is a diagram illustrating a sofa-type communication system according to an embodiment of the present invention. FIG. 1 is a diagram showing the configuration of the main parts of Sync Sofa. FIG. 10 is a diagram showing an example in which the user of one Sync Sofa is displayed on the large display of the other Sync Sofa. FIG. 1 is a functional block diagram showing the configuration of a control system that realizes two-way communication between Sync Sofas. This is a diagram showing a schematic example of two-way communication using three or more Sync Sofas.

Embodiments of the present invention will now be described in detail with reference to the drawings. Figure 1 is a diagram illustrating a sofa-type communication system 1 according to one embodiment of the present invention, which is configured by connecting a pair of remotely located two-way communication sofas (hereinafter referred to as Sync Sofas) 1A and 1B via a network.

Each Sync Sofa 1A, 1B is equipped with a camera as a visual information acquisition means, a microphone as an auditory information acquisition means, an acceleration sensor as a tactile information acquisition means, a speaker as an auditory information presentation means, and a vibration actuator as a tactile information presentation means (all of which are omitted in Figure 1), as well as a large display 10 as a visual information presentation means, allowing users of each Sync Sofa 1A, 1B to communicate two-way visually, auditorily, and tactilely via the control system.

Each Sync Sofa 1A, 1B has a sofa shape with a seat and a backrest, and the large display 10 is placed on at least one of the left and right sides of the seat so that it faces the seat. The position of the large display 10 on one Sync Sofa is notified to the other Sync Sofa in advance.

Figure 2 shows the main components of each Sync Sofa 1A, 1B, with the various cushioning materials and surface coverings removed. The Sync Sofa is equipped with at least one large display 10, multiple cameras 11, multiple microphones 12, multiple acceleration sensors 13, multiple speakers 14, and multiple vibration actuators 15, as well as a control system that controls each component to enable two-way communication.

The camera 11 mainly captures images of users seated on the seats of each Sync Sofa. It is desirable, but not essential, for the camera 11 to be equipped with a depth sensor. The microphone 12 mainly detects the voice and movement sounds, such as the rustling of clothes, of users seated on the seats of each Sync Sofa. In this embodiment, an audio microphone that mainly detects voice and a movement sound microphone that mainly detects movement sounds are provided separately.

The acceleration sensor 13 mainly detects the behavior of a user seated on the seat of each Sync Sofa. The acceleration sensor 13 can be any of a variety of sensors capable of detecting user behavior, such as a vibration sensor. In the description of this embodiment, the various sensors are collectively referred to as the acceleration sensor 13.

The speaker 14 plays back the voice and gesture sounds detected by the microphone 12 on the Sync Sofa of the communication partner. The vibration actuator 15 presents the vibrations detected by the acceleration sensor 13 on the Sync Sofa of the communication partner. Images captured by the camera 11 on one Sync Sofa are displayed on the large display 10 on the other Sync Sofa.

Figure 3 shows an example in which the user of Sync Sofa 1A is displayed on the large display 10 of Sync Sofa 1B. The user of Sync Sofa 1B is also displayed on the large display 10 of Sync Sofa 1A, realizing two-way communication based on visual information.

In this embodiment, vibration actuators 15 are provided in a total of ten locations: four on the seat surface 8 and six on the backrest 9. Each vibration actuator 15 is positioned based on the width of a person's shoulders and buttocks, so that it comes into contact with the left, right and center of the back, and the left and right buttocks, respectively. This is to stimulate the tactile sensation on each side of the person's back and buttocks.

The vibration actuator 15 does not come into direct contact with the human body and is covered with cushioning material approximately 2 cm thick and sturdy fabric approximately 1 mm thick, and the vibrations it presents are transmitted through the cushioning material and fabric. The vibrations presented to one Sync Sofa are a signal synthesized from the vibrations detected by the acceleration sensors 13 located on the seat 8 and backrest 9 of the other Sync Sofa and the movement sounds detected by the microphone 12.

Figure 4 is a functional block diagram showing the configuration of the control system 20 that enables two-way communication between Sync Sofa 1A and 1B.

In addition to well-known control functions for controlling the large display 10, camera 11, microphone 12, acceleration sensor 13, speaker 14, vibration actuator 15, and communication control unit 16, the control system 20 also includes a motion parallax image generation unit 201, presentation position control unit 202, and feedback suppression unit 203, and each Sync Sofa 1A, 1B is interconnected via the communication control unit 16 and a network. The control system 20 may be provided for each Sync Sofa, or may be provided on the cloud so that each Sync Sofa can share the control system.

The motion parallax image generation unit 201 generates motion parallax images based on multiple camera images with different viewpoints and the estimated position of the subject person. The presentation position control unit 202 localizes auditory information and tactile information to predetermined positions based on the estimated position of the subject person. The howling suppression unit 203 includes a first loop suppression unit 2031 and a second loop suppression unit 2032, and suppresses howling of auditory information and tactile information.

Such a control system 20 can be configured by implementing applications (programs) that realize the functions detailed below on a general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated or single-function machine in which part of the application is implemented as hardware or software.

In this embodiment, two-way communication is achieved by Sync Sofa 1A and Sync Sofa 1B, which have substantially identical configurations, operating cooperatively via control system 20. Therefore, the operation of this embodiment will be explained here, focusing mainly on the operation of Sync Sofa 1A. To make the explanation easier to understand, the components on the Sync Sofa 1A side are distinguished by adding the suffix A to each component, and the components on the Sync Sofa 1B side are distinguished by adding the suffix B to each component.

The motion parallax video generation unit 201A of Sync Sofa 1A edits the visual information received from Sync Sofa 1B into a video that reflects the motion parallax of user UA of Sync Sofa 1A. In this embodiment, motion parallax video is generated by rendering foreground and background polygon models placed in 3D space with a virtual camera.

The foreground is a person object of user UB, whose person area extracted from the visual information transmitted by Sync Sofa 1B is mapped onto a single rectangular polygon. From the perspective of providing a sense of proximity and realism, it is desirable for the background to be objects that resemble a sofa, wall, floor, etc. The person object is placed in 3D space so that it is displayed life-size on large display 10A.

The position of the virtual camera can be obtained as a result of tracking the position information of the user UA's head obtained from the depth sensor camera 11A. This allows user UA on Sync Sofa 1A to realize eye contact with user UB on Sync Sofa 1B while reflecting their own motion parallax. Note that motion parallax images can also be generated by switching between foreground images captured simultaneously using multiple cameras depending on the viewpoint position, or by synthesizing intermediate images.

The presentation position control unit 202A localizes the audio information and gesture sounds received from Sync Sofa 1B at predetermined positions. The presentation position control unit 202A can also localize the tactile information of vibrations and gesture sounds received from Sync Sofa 1B at predetermined positions.

For auditory information, in this embodiment, Sync Sofa 1A is equipped with full-range speakers 14 in six locations: four on the left and right sides of the seat surface 8 and two on the left and right sides of the backrest 9. Each speaker 14 is positioned at a distance sufficient to localize a sound image.

In this embodiment, assumed gesture sounds include the sound of the user's clothing rubbing against the surface fabric of the Sync Sofa 1 when changing posture, the frictional sounds made when crossing or fidgeting their legs, and the collision sounds made when the user sits on the Sync Sofa 1. The gesture sounds of each user are localized to the user's seat position, and the sounds are localized to the mouth of each user as displayed on the large display 10.

Although tactile information has a lower frequency than auditory information, the tactile presentation position may be varied depending on the position of each user, just as in the case of auditory information.

For example, if the received visual information detects movement of the user's hands, back, buttocks, etc., the vibration presentation position can be changed, or the presentation position can be continuously changed in accordance with the movement. Therefore, if a movement such as stroking the back is detected from the visual information, the vibration actuator 15 corresponding to that position can be selectively driven.

To prevent feedback during two-way communication, the feedback suppression unit 203A includes a first loop suppression unit 2031A that suppresses the looping of auditory information and a second loop suppression unit 2032A that suppresses the looping of tactile information.

The feedback of auditory information in Sync Sofa 1A occurs when the movement sounds and voices received from Sync Sofa 1B are output from speaker 14A, picked up by microphone 12A, and then transmitted to Sync Sofa 1B for presentation, creating an unnecessary loop in the transmission path of auditory information, causing this loop to oscillate.

It is known that the frequency of the auditory information loop depends on the distance between speaker 14A and microphone 12A and the impulse response. Therefore, the occurrence of a loop can be suppressed by increasing the distance between the presentation position of the auditory information controlled by presentation position control unit 202A and the position of microphone 12A, or by switching microphone 12A depending on the frequency characteristics of the auditory information.

In this embodiment, the first loop suppression unit 2031A monitors the position where the gesture sounds and voice received from Sync Sofa 1B are localized (the position where the auditory information is presented), and can switch between the microphone 12A that captures the gesture sounds and the microphone 12A that captures the voice based on the distance between the localization position and each microphone 12A.

Alternatively, based on the frequency characteristics (e.g., high, mid, and low frequencies) within a short-time frame of auditory information and the distance between each microphone 12A and the position where the movement sounds and voice are localized, a microphone 12A located in a position where loops are unlikely to occur may be selected for each frequency band of auditory information, such as microphone 12A_1 for the high frequency range, microphone 12A_2 for the mid frequency range, and microphone 12A_3 for the low frequency range.

Alternatively, if the power within a short-time frame of auditory information exceeds a set threshold, more specifically, if the ratio of the average power of all frequencies within the short-time frame to the power of a certain frequency exceeds a pre-set threshold, it may be determined that a sound has occurred and the output of speaker 14A may be suppressed.

The feedback of haptic information on Sync Sofa 1A occurs when haptic information received from Sync Sofa 1B is output from vibration actuator 15A, acquired by acceleration sensor 13A, and then transmitted to Sync Sofa 1B for presentation, creating an unnecessary loop in the transmission path of the haptic information, causing this loop to oscillate.

Here, it is known that the frequency of the haptic information loop depends on the distance between the vibration actuator 15A and the acceleration sensor 13A and the impulse response. Therefore, the occurrence of a loop can be suppressed by increasing the distance between the presentation position of the haptic information controlled by the presentation position control unit 202A and the position of the acceleration sensor 13A, or by switching the acceleration sensor 13A depending on the frequency characteristics of the haptic information.

In this embodiment, the second loop suppression unit 2032A monitors the position where the vibrations and gesture sounds received from Sync Sofa 1B are localized (the position where haptic information is presented), and can switch the acceleration sensor 13A that acquires the vibrations based on the distance between the localization position and each acceleration sensor 13A.

Alternatively, based on the frequency characteristics (e.g., high, mid, and low frequencies) within the short-time frame of the haptic information and the distance between the position where the vibration and movement sound are localized and each acceleration sensor 13A, an acceleration sensor 13A located in a position where loops are unlikely to occur may be selected for each frequency band of the haptic information, such as acceleration sensor 13A_1 for the high frequency range, acceleration sensor 13A_2 for the mid frequency range, and acceleration sensor 13A_3 for the low frequency range.

Alternatively, if the power within a short-time frame of haptic information exceeds a set threshold, more specifically, if the ratio of the average power of all frequencies within the short-time frame to the power of a certain frequency exceeds a preset threshold, it may be determined that a sound has occurred and the output of vibration actuator 15A may be suppressed.

In the above embodiment, two-way communication is achieved through visual, auditory, and tactile information, but the present invention is not limited to this and can be applied to any communication system that achieves two-way communication through at least visual information.

In addition, in the above embodiment, the present invention has been described as being applied to a sofa-shaped Sync Sofa, but the present invention is not limited to this and can be similarly applied to any structure that has at least a seat, such as a bench or chair. Furthermore, while it is desirable for the seat to have a backrest, this is not a required component of the present invention.

Furthermore, in the above embodiment, the large display 10 is described as being provided on at least one of the left and right sides of the seat, but the present invention is not limited to this. For chairs with circular seats or chairs without a concept of left and right, such as balance balls, the large display 10 can be provided on at least one of the opposing ends of the seating area.

Furthermore, in the above embodiment, the present invention was described using an example of two-way communication using two Sync Sofas, but the present invention is not limited to this and can also be applied to two-way communication using three or more Sync Sofas, as shown in Figure 5.

Furthermore, according to the above-described embodiment, two-way remote communication can be realized through adjacent communication modes, using sight, hearing, and touch, making it possible to contribute to Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient, and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations.

1...Sofa-type communication system, 1A, 1B, 1C...Sync Sofa, 8...Seat, 9...Backrest, 10...Large display, 11...Camera, 12...Microphone, 13...Acceleration sensor, 14...Speaker, 15...Vibration actuator, 16...Communication control unit, 20...Control system, 201...Motion parallax image generation unit, 202...Presentation position control unit, 203...Howling suppression unit

Claims (10)

In a communication system configured by connecting a plurality of communication devices for remote communication with each other via a network ,
Each communication device
a chair-shaped device having at least one of a tactile information acquisition means and a presentation means, an auditory information acquisition means and a presentation means, and a visual information acquisition means and a presentation means;
the information acquired by each of the acquisition means is transmitted to another communication device via a network, and each of the presentation means receives and presents the information acquired by each of the acquisition means of the other communication device via the network;
the visual information presenting means is provided on at least one of one end side and the other end side of the chair shape,
a means for localizing at least one of the auditory information and the tactile information to be presented at a predetermined position based on position information of a person area estimated based on visual information received from the other communication device;
and loop suppression means for suppressing looping of at least one of the auditory information and the tactile information based on a position at which at least one of the auditory information and the tactile information is localized . a display is provided as the means for presenting the visual information;
The communication system according to claim 1, further comprising a presentation position control means for controlling the image to be displayed on the display based on the relationship between the visual information received from the other communication device and the visual information transmitted to the other communication device . A plurality of microphones are provided as means for acquiring auditory information,
The communication system described in claim 1 or 2 , characterized in that the loop suppression means includes a first loop suppression means that switches the microphone used to acquire the auditory information to be transmitted based on the distance between the position where the auditory information is localized and each microphone. A plurality of microphones are provided as means for acquiring auditory information,
The communication system described in claim 1 or 2, characterized in that the loop suppression means includes a first loop suppression means that switches the microphone used to acquire the auditory information to be transmitted based on the distance between the position where the auditory information is localized and each microphone and the frequency characteristics of the auditory information . 4. The communication system according to claim 3 , wherein the first loop suppression means further comprises means for suppressing presentation of the auditory information. 5. The communication system according to claim 4 , wherein the first loop suppression means further comprises means for suppressing presentation of the auditory information. A plurality of acceleration sensors are provided as means for acquiring tactile information,
The communication system described in claim 1 or 2, characterized in that the loop prevention means includes a second loop prevention means that switches the acceleration sensor used to acquire the tactile information to be transmitted based on the distance between the position where the tactile information is localized and each acceleration sensor . A plurality of acceleration sensors are provided as means for acquiring tactile information,
The communication system described in claim 1 or 2, characterized in that the loop prevention means includes a second loop prevention means that switches the acceleration sensor used to acquire the tactile information to be transmitted based on the distance between the position where the tactile information is localized and each acceleration sensor and the frequency characteristics of the tactile information. 8. The communication system according to claim 7 , wherein the second loop suppression means further comprises means for suppressing presentation of the tactile information. 9. The communication system according to claim 8 , wherein the second loop suppression means further comprises means for suppressing presentation of the tactile information.