WO2025215713A1 - Information notification device - Google Patents

Abstract

An information notification device according to one aspect of the present invention comprises a reception unit, an acquisition unit, a recognition unit, an expansion unit, and a transmission unit. The reception unit receives a notification destined for a user. The acquisition unit acquires user characteristic information indicating the superiority and inferiority of the user with respect to a plurality of modals. The recognition unit recognizes the situation of the user with respect to a terminal of the user. On the basis of the user characteristic information acquired by the acquisition unit and the situation of the user recognized by the recognition unit, the expansion unit expands the notification, received by the reception unit, to one or more pieces of modal information not including modals serving as distractor stimuli for the user among the plurality of modals. The transmission unit transmits, to the user, a notification including the one or more pieces of modal information expanded by the expansion unit.

Description

Information notification device

One aspect of the present invention relates to an information notification device.

When receiving notification information such as news, email, or SNS (Social Networking Service) via a network, including the Internet, other modal information not included in the received information, such as text, voice, or facial expressions, is generated to expand the notifications provided to the user.

For example, in Patent Document 1, when an email is received, an avatar's movements are generated from the language input information to provide an enhanced notification.

Regarding modal information, for example, as disclosed in Non-Patent Document 1, Mehrabian's experiment is known, which investigated which modal takes priority in influencing impressions when visual, auditory, and linguistic information do not match.

Furthermore, non-patent document 2 explains the cognitive characteristics of understanding, organizing, memorizing, and expressing information received through sensory organs such as the eyes and ears.

JP 2011-138464 A

Kurita, "How did emojis originate and spread around the world?", Institute of Electronics, Information and Communication Engineers, Communications Society Magazine 11 (3), 2017 Honda Mami, "A Doctor's Smartness Test", Kobunsha Shinsho, June 15, 2012

However, when notifications are expanded to include modal information such as text, voice, and facial expressions, they may contain modalities that act as distracting stimuli depending on the characteristics or situation of the user receiving the notification.

For example, regarding user characteristics, in an extreme case, audio notifications may cause distress to users with hearing sensitivity. More generally, audio notifications may distract users with hearing sensitivity or those with hearing sensitivity, potentially interfering with their activities.

Furthermore, with regard to the user's situation, for example, if the user is listening to music, providing the user with an audio notification would interfere with the user's music listening activity.

Not only voice notifications, but notifications using other modal information may also disrupt user activities. Activities that may be disrupted by such notifications include, for example, researching, viewing, and creating materials using a work terminal such as a PC (Personal Computer) or smartphone, watching and creating content, programming, remote meetings, phone calls, email, chat, and social media.

This invention was made in consideration of the above circumstances, and aims to provide technology that enables enhanced notifications without interfering with the user's activities.

In order to solve the above problems, an information notification device according to one embodiment of the present invention comprises a receiving unit, an acquiring unit, a comprehension unit, an expansion unit, and a distribution unit. The receiving unit receives a notification addressed to the user. The acquiring unit acquires user characteristic information indicating the user's superiority and inferiority with respect to multiple modals. The comprehension unit comprehends the user's situation with respect to the user's device. The expansion unit expands the notification received by the receiving unit to one or more modal information out of the multiple modals, excluding any modal that poses a disruptive stimulus to the user, based on the user characteristic information acquired by the acquiring unit and the user's situation ascertained by the comprehension unit. The distribution unit distributes a notification including the one or more modal information expanded by the expansion unit to the user.

According to one aspect of this invention, modal information that does not include modals that may be disruptive to the user is delivered based on user characteristic information and the user's situation, thereby providing a technology that enables expanded notifications without interfering with the user's activities.

FIG. 1 is a block diagram showing an example of the hardware configuration of a user terminal to which an information notification device according to a first embodiment of the present invention is applied. FIG. 2 is a block diagram showing an example of the software configuration of the user terminal. FIG. 3 is a diagram illustrating an example of the contents stored in the modal characteristics storage unit. FIG. 4 is a diagram schematically illustrating each modal in the example of the modal characteristics shown in FIG. FIG. 5 is a diagram illustrating an example of the contents stored in the modal selection rule storage unit. FIG. 6 is a flowchart showing an example of the processing procedure and processing content of the information notification processing executed by the control unit of the user terminal. FIG. 7 is a diagram showing an example of a received notification. FIG. 8 is a diagram showing an example of a plurality of pieces of modal information that are extensions of a received notification. FIG. 9 is a diagram illustrating an example of a grasp situation. FIG. 10 is a diagram schematically showing modalities selected in the example shown in FIG. FIG. 11 is a diagram showing extended modal information included in the notification delivered in the example shown in FIG. FIG. 12 is a diagram showing another example of the grasp situation. FIG. 13 is a diagram schematically showing modalities selected in the example shown in FIG. FIG. 14 is a diagram showing extended modal information included in the notification delivered in the example shown in FIG. FIG. 15 is a diagram showing another example of the grasp situation. FIG. 16 is a diagram schematically showing modalities selected in the example shown in FIG. FIG. 17 is a diagram showing extended modal information included in the notification delivered in the example shown in FIG. FIG. 18 is a block diagram showing an example of the software configuration of a user terminal to which an information notification device according to the second embodiment of the present invention is applied. FIG. 19 is a flowchart illustrating an example of the processing procedure and processing content of an information notification process executed by the control unit of the user terminal in the second embodiment. FIG. 20 is a block diagram showing an example of the hardware configuration of an information notification system to which an information notification device according to the third embodiment of the present invention is applied. FIG. 21 is a block diagram showing an example of the software configuration of the information notification system. FIG. 22 is a flowchart showing an example of the procedure and content of an information notification process executed by the control unit of the distribution server included in the information notification system. FIG. 23 is a flowchart showing an example of the processing procedure and processing content of the information notification processing executed by the control unit of the user terminal included in the information notification system.

Embodiments of the present invention will be described below with reference to the drawings.

[First embodiment]
(Configuration example)
1 and 2 are block diagrams showing an example of the hardware and software configurations of a user terminal 1 to which an information notification device according to a first embodiment of the present invention is applied. The user terminal 1 is an information processing device such as a PC, a smartphone, or a tablet terminal that can be connected to a network including the Internet.

The user terminal 1 is equipped with a control unit 11 that uses a hardware processor such as a CPU, and is connected to this control unit 11 via a bus 16 with a storage unit having a program storage unit 12 and a data storage unit 13, a communication interface unit 14, and an input/output interface unit 15. Note that in the diagram, "interface" is abbreviated as "IF."

The control unit 11 is a hardware processor such as a CPU (Central Processing Unit). For example, by using a multi-core and multi-threaded CPU, it is possible to execute multiple information processes simultaneously. The control unit 11 may be equipped with multiple hardware processors.

Under the control of the control unit 11, the communication interface unit 14 sends and receives information to and from other devices using a communication protocol defined by the network.

The input/output interface unit 15 is connected to an input device 171, an audio input device 172, a video input device 173, an audio output device 174, a video output device 175, and the like. Note that the user terminal 1 does not necessarily have to be equipped with all of these devices, and some of the devices may be connected as external devices.

The input device 171 is used by the user of the user terminal 1 to input instructions necessary for the operation of the user terminal 1. If the user terminal 1 is a personal computer, the input device 171 includes, for example, a keyboard, a pointing device such as a mouse or a touchpad, etc. If the user terminal 1 is a smartphone or tablet terminal, the input device 171 includes, for example, a touch panel and operation buttons arranged on the display screen of the video output device 175. The input device 171 may also include a reader device for reading information from a recording medium that stores various types of information such as programs and data.

The voice input device 172 is equipped with a microphone and captures voices uttered by the user.

The video input device 173 is equipped with a camera such as a webcam or a general video camera, and captures video of the user.

The audio output device 174 is equipped with a speaker and transmits various information to the user, who is the operator of the user terminal 1, by sound.

The video output device 175 includes a display such as an LCD monitor, an organic EL (Electro Luminescence) monitor, or a projector screen, and displays various information to the user who is operating the user terminal 1.

The program storage unit 12 is configured, for example, by combining non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD), solid state drive (SSD), or EEPROM (registered trademark) (electrically erasable programmable read-only memory), as a storage medium, with non-volatile memory such as read-only memory (ROM). In addition to middleware such as an operating system (OS), the program storage unit 12 stores application programs necessary for operation as the information notification device according to the first embodiment of the present invention. Hereinafter, the OS and each application program will be collectively referred to as the program.

The data storage unit 13 is, for example, a combination of non-volatile memory that can be written to and read from at any time, such as a HDD, SSD, EEPROM, or memory card, as a storage medium, and volatile memory such as RAM (Random Access Memory). The data storage unit 13 includes, in its storage area, a modal characteristics storage unit 131 and a modal selection rule storage unit 132 as storage units necessary for implementing the first embodiment.

The modal characteristic storage unit 131 pre-stores user characteristic information indicating the dominance/inferiority of the user of the user terminal 1 for each of multiple modals, such as text, voice, and facial expressions. This user characteristic information for each modality can be determined, for example, with reference to Mehrabian's experiments, as described in Non-Patent Document 1. The user characteristic information may also be determined with reference to cognitive characteristic tests, as disclosed in Non-Patent Document 2. Alternatively, the user characteristic information may be determined based on the self-evaluation input of the user of the user terminal 1. Figure 3 is a diagram showing an example of the contents stored in the modal characteristic storage unit 131. For example, as shown in Figure 3, the modal characteristic storage unit 131 can store the dominance/inferiority of each modal as a numerical ratio. Figure 4 is a diagram schematically showing each modal in the example of modal characteristics shown in Figure 3. In other words, Figure 4 is a diagram schematically illustrating the dominance/inferiority of each modal in an easy-to-understand manner. Figures 3 and 4 show an example where facial expression is determined to be the inferior modal and voice is determined to be the dominant modal.

The modal selection rule memory unit 132 stores selection rules for determining which modal to use based on user characteristic information about modals and the user's status relative to the user terminal 1. The user's status can include whether the user terminal 1 is operating, i.e., the operating status of the user terminal 1. The user's status can also include the usage status of the user terminal 1, which is determined from the orientation of the user's face toward the user terminal 1, and the input and output status of text, audio, and video on the user terminal 1. Here, "face orientation" refers to whether the user's face is facing the user terminal 1. The input and output status of text, audio, and video on the user terminal 1 can include the status of keyboard input, mouse operation, audio input and output, and video input and output. "Keyboard input status" refers to the extent to which input is being made using the keyboard, which is one of the input devices 171. "Mouse operation status" refers to the extent to which operation is being made using the mouse, which is one of the input devices 171. "Audio input and output status" refers to whether audio input and output is being used by the audio input device 172 or the audio output device 174. "Video input/output status" refers to whether the video input/output is being used by the video input device 173 or the video output device 175.

The modal selection rule storage unit 132 can store, as a selection rule, a rule related to the user's concentration level, such as "use the inferior modal when the face is facing directly toward the user terminal 1" in relation to "face orientation status."

Furthermore, the modal selection rule storage unit 132 can store language-related rules regarding "keyboard input status," such as "text modal should not be used when keyboard input is being used."

Furthermore, the modal selection rule storage unit 132 can store rules related to concentration level, such as "when mouse operation is involved, use the inferior modal whenever possible" in relation to "mouse operation status."

Furthermore, the modal selection rule storage unit 132 can store rules related to hearing, such as "Do not use the audio modal when audio input/output is fully used," regarding the "audio input/output situation."

Furthermore, the modal selection rule storage unit 132 can store a visual rule regarding "video input/output status," such as "when video output is fully used, facial expression modals should not be used."

The modal selection rules stored in the modal selection rule storage unit 132 may be learned using a decision tree or an SVM (Support Vector Machine).

Figure 5 is a diagram showing an example of the contents stored in the modal selection rule storage unit 132. While Figure 5 is shown in table format, it is of course not limited to this. Furthermore, Figure 5 shows an extracted portion of the user characteristic information for modals stored in the modal characteristic storage unit 131, corresponding to the case where facial expression is the inferior modal and voice is the dominant modal, as shown in Figures 3 and 4. It goes without saying that the modal selection rule storage unit 132 stores rules for various modal combinations. Furthermore, if the user terminal 1 is used only by a specific user and the dominance/inferiority of multiple modals is fixed, the modal selection rule storage unit 132 may store only the rules appropriate for that user. In this case, the modal characteristic item in the example shown in Figure 5 may be omitted. Alternatively, it is possible to omit the modal characteristic storage unit 131 by substituting this modal characteristic item. In the modal characteristic items in Figure 5, a circle indicates a dominant, a cross indicates a subordinate, and a triangle indicates a neutral position. Additionally, in the comprehension status section in Figure 5, a circle indicates a situation in which it is fully used, a cross indicates a situation in which it is not used, and a triangle indicates a situation in which it is not fully used. In the selection modal section in Figure 5, a circle indicates a selection, and a hyphen indicates a non-selection.

The control unit 11 includes a notification receiving unit 111, a notification expanding unit 112, a status grasping unit 113, a notification selecting unit 114, and a notification distribution unit 115 as processing function units according to the first embodiment of the present invention. All of these processing function units 111 to 115 are realized by causing the hardware processor of the control unit 11 to execute an information notification program for the information notification device according to the first embodiment of the present invention, which is stored in the program storage unit 12.

In addition to storing the information notification program in advance in the program storage unit 12, the information notification program may be read out from a storage medium storing the information notification program when necessary and stored in the program storage unit 12. Alternatively, the information notification program may be downloaded from a program server (not shown) and stored in the program storage unit 12. In addition, at least some of the processing functions of at least one of the processing function units 111 to 115 may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), DSP (Digital Signal Processor), FPGA (Field-Programmable Gate Array), or GPU (Graphics Processing Unit), instead of being realized by the information notification program and the hardware processor of the control unit 11.

The notification receiving unit 111 receives notifications addressed to the user of the user terminal 1 that are sent over the network via the communication interface unit 14. The notification receiving unit 111 can store the received notifications in a temporary storage area (not shown) provided in the data storage unit 13.

The notification expansion unit 112 generates modal information for all of the multiple modalities, such as text, voice, and facial expression, for the notification received by the notification receiving unit 111. Specifically, the notification expansion unit 112 adds modal information and expands the notification by generating modal information such as the facial expression of the notification sender, the voice of the message, and the text of the message. The notification expansion unit 112 can store the multiple pieces of generated modal information in a temporary storage area of the data storage unit 13.

With regard to text modal information, if the notification received by the notification receiving unit 111 is text-based, the notification expansion unit 112 can use the text included in the notification. On the other hand, if the received notification is audio or video-based, the notification expansion unit 112 generates text modal information using known voice recognition or image recognition functions. The notification expansion unit 112 may also use, for example, AI (Artificial Intelligence) to generate text with different content related to the notification content based on the text included in the received notification or the recognized text.

With regard to audio modal information, if the notification received by the notification receiving unit 111 is audio-based, the notification expanding unit 112 can use the audio contained in the notification. On the other hand, if the received notification is text-based, the notification expanding unit 112 generates audio modal information using a known audio synthesis function. Furthermore, if the received notification is video-based, the notification expanding unit 112 uses a known character recognition function or image recognition function to extract text contained in the image, and then generates audio modal information from the extracted text using a known audio synthesis function. Note that when generating this audio modal information, audio of any voice tone may be synthesized, or a specific voice tone may be selected and used for each sender.

Facial expression modal information is visual information that can identify the sender of the notification. If the notification received by the notification receiving unit 111 is video-based, the notification expansion unit 112 generates facial expression modal information using the video of the sender included in the notification. For example, AI can be used to extract the sender's facial expression from the video. Also, if the received notification is text-based or audio-based, the notification expansion unit 112 can generate facial expression modal information based on information about the sender of the notification. Facial expressions may be generated for any facial image or avatar, or for a specific sender, facial images with multiple facial expressions may be stored in advance in the data storage unit 13 and selected for use.

The situation understanding unit 113 acquires input information from the input device 171, audio input device 172, and video input device 173 via the input/output interface unit 15, and also acquires information indicating the output status from the audio output device 174 and video output device 175. The input information and/or information indicating the output status may not be acquired from the input and/or output devices, but may be acquired from another process being executed by the control unit 11 that controls the input and/or output by those devices. Based on this acquired information, the situation understanding unit 113 determines the input and output status of text, audio, and video on the user terminal 1, the user's facial orientation toward the user terminal 1, and so on, and understands the user's situation relative to the user terminal 1 from the results of this determination. The situation understanding unit 113 transmits this understood user situation to the notification selection unit 114.

The notification selection unit 114 acquires user characteristic information by reading user characteristic information indicating the user's dominance and inferiority with respect to multiple modals from the modal characteristic storage unit 131. Based on the acquired user characteristic information and the user's situation grasped by the situation grasping unit 113, the notification selection unit 114 selects one or more pieces of modal information from the multiple pieces of modal information generated by the notification expansion unit 112 that do not include modals that are disruptive stimuli for the user. More specifically, the notification selection unit 114 selects one or more pieces of modal information that are not disruptive stimuli for the user in accordance with selection rules based on the user characteristic information and the user's situation stored in the modal selection rule storage unit 132.

The notification distribution unit 115 distributes to the user an expanded notification that includes one or more pieces of modal information selected by the notification selection unit 114 based on both the user characteristic information and the user's situation. Specifically, if text modal information or facial expression modal information is selected, the notification distribution unit 115 outputs the modal information from the video output device 175 via the input/output interface unit 15. Furthermore, if audio modal information is selected, the notification distribution unit 115 outputs the modal information from the audio output device 174 via the input/output interface unit 15.

(Example of operation)
Next, an example of the operation of the user terminal 1 to which the information notification device configured as described above is applied will be described. Fig. 6 is a flowchart showing an example of the processing procedure and processing content of the information notification processing executed by the control unit 11 of the user terminal 1. The information notification processing shown in this flowchart is executed while the user terminal 1 is operating.

As shown in FIG. 6, the control unit 11 of the user terminal 1 first operates as a notification receiving unit 111 and determines whether a notification sent from any sender via the network has been received via the communication interface unit 14 (step S101). If it determines that a notification has not been received, the control unit 11 performs the processing of step S101 again. In this way, the control unit 11 waits to receive a notification.

If it is determined that a notification has been received, the control unit 11 operates as the notification expansion unit 112 and expands the received notification in all of the multiple modalities, such as text, voice, and facial expressions (step S102). In other words, the control unit 11 generates multiple modal information items corresponding to the received notification.

The control unit 11 also operates as a situation assessment unit 113, determining the input and output status of text, audio, and video on the user terminal 1, the orientation of the user's face toward the user terminal 1, and other factors, and assessing the user's situation relative to the user terminal 1 from the results of this assessment (step S103).

Then, the control unit 11 operates as the notification selection unit 114 and selects modal information to use from the generated plurality of modal information (step S104). Specifically, the control unit 11 reads user characteristic information indicating the user's superiority and inferiority with respect to the plurality of modals from the modal characteristic storage unit 131, and selects one or more pieces of modal information from the generated plurality of modal information based on this user characteristic information and the grasped user situation, which do not include modals that act as disruptive stimuli for the user.

The control unit 11 operates as the notification distribution unit 115 and distributes to the user a notification including one or more pieces of modal information selected based on both the user characteristic information and the user's situation (step S105). Specifically, the control unit 11 outputs facial expression and text modal information from the video output device 175 via the input/output interface unit 15, and outputs audio modal information from the audio output device 174. Thereafter, the control unit 11 proceeds to the processing of step S101 above.

The following provides an explanation using a specific example. Here, we will assume that the user characteristic information stored in the modal characteristic storage unit 131 is in the form of superiority and inferiority as shown in Figure 3, and that the modal selection rule storage unit 132 stores selection rules as shown in Figure 5.

Figure 7 shows an example of a received notification. Here, it is assumed that the user terminal 1 has received an email notification MN. In the example of Figure 7, the email notification MN includes the sender "Labor Manager" indicated in "From:" and the email body indicated in "Message" which reads "Please check the work schedule...".

FIG. 8 is a diagram showing an example of multiple pieces of modal information that are extensions of a received notification. For the email notification MN in the example shown in FIG. 7, the control unit 11 generates text modal information TX, voice modal information VO, and facial modal information FE, as shown in FIG. 8.

9 is a diagram showing an example of a grasp situation. In this example, the facial orientation, keyboard input, and mouse operation statuses are marked with circles, and the audio input/output status and video input/output status are marked with crosses. This is a situation in which the user is concentrating on, for example, creating documents, without looking away from the screen of the user terminal 1, and is making full use of keyboard input and mouse operation. In such a situation, according to the selection rules shown in FIG. 5, facial expression modal information is selected, but audio and text modal information are not selected. FIG. 10 is a diagram schematically showing the modals selected in the example shown in FIG. 9. In FIG. 10, dotted lines indicate that no selection is made. In the user's situation shown in FIG. 9, the user's facial orientation is not averted and there is mouse operation, so the control unit 11 selects facial expression, which is the inferior modal. Furthermore, the control unit 11 does not select audio, as it is the dominant modal, and does not select the text modal because there is keyboard input. FIG. 11 is a diagram showing the extended modal information included in the notification delivered in the example shown in FIG. 9. In a user situation such as that shown in Figure 9, only facial expression modal information FE selected based on both the user characteristic information and the user situation will be notified, as shown in Figure 11.

12 is a diagram showing another example of the recognition status. In this example, the video input/output status is indicated by a circle, the facial orientation status and audio input/output status are indicated by a triangle, and the keyboard input status and mouse operation status are indicated by a cross. This situation is, for example, when a user is participating in a remote conference but is not making any comments, is facing away from the screen of the work terminal, and is not using keyboard input or mouse operation. In this situation, according to the selection rules shown in FIG. 5, audio and text modal information is selected, but facial expression modal information is not selected. FIG. 13 is a diagram schematically showing the modals selected in the example shown in FIG. 12. In the user's situation shown in FIG. 12, video input/output is used, so the control unit 11 does not select the facial expression modal. Furthermore, because audio input/output is not used at all and the user's face is facing away from the screen of the user terminal 1, the control unit 11 selects the audio modal, which is the dominant modal. Furthermore, because keyboard input is not used, the control unit 11 selects the text modal. FIG. 14 is a diagram showing extended modal information included in the notification delivered in the example shown in FIG. 12. In a user situation such as that shown in Figure 12, speech modal information VO and text modal information TX selected based on both the user characteristic information and the user situation will be notified, as shown in Figure 14.

15 is a diagram showing another example of the grasp status. In this example, only the facial orientation status is marked with a triangle, while the other statuses, such as keyboard input status, mouse operation status, audio input/output status, and video input/output status, are all marked with a cross. This indicates a situation where, for example, when work such as document creation or literature research is stalled, the user's face tends to be turned away from the screen of the user terminal 1, and there is little keyboard input or mouse operation. In such a situation, according to the selection rule shown in FIG. 5, facial expression, audio, and text modal information are all selected. FIG. 16 is a diagram schematically showing the modals selected in the example shown in FIG. 15. In the user's situation shown in FIG. 12, video input/output is not being used entirely, so the control unit 11 selects the facial expression modal. Furthermore, audio input/output is not being used entirely, and the user's face tends to be turned away from the screen of the user terminal 1, so the control unit 11 selects the audio modal, which is the dominant modal. Furthermore, since keyboard input is not being used, the control unit 11 selects the text modal. FIG. 17 is a diagram showing expanded modal information included in the notification delivered in the example shown in FIG. 15. In the user situation shown in Figure 15, facial expression modal information FE, voice modal information VO, and text modal information TX selected based on both the user characteristic information and the user situation will be notified, as shown in Figure 17.

(Actions and Effects)
As described above, in the user terminal 1 to which the information notification device according to the first embodiment is applied, the notification receiving unit 111 receives a notification addressed to the user, and the situation grasping unit 113 grasps the user's situation with respect to the user terminal 1. The notification expanding unit 112 and the notification selecting unit 114 then acquire user characteristic information indicating the user's dominance and inferiority with respect to multiple modals from the modal characteristic storage unit 131. Based on the acquired user characteristic information and the user's situation grasped by the situation grasping unit, the notification received by the notification receiving unit 111 is expanded to one or more pieces of modal information among the multiple modals, excluding any modal that constitutes a disturbing stimulus for the user. The notification delivering unit 115 delivers to the user a notification including one or more pieces of modal information expanded based on both the user characteristic information and the user's situation. In this manner, the notification expanding unit 112 and the notification selecting unit 114 are examples of an expanding unit. The notification selecting unit 114 is an example of a selecting unit.
As described above, the information notification device according to the first embodiment delivers modal information that does not include modals that may be disruptive to the user based on both user characteristic information and the user's situation, thereby providing a technology that enables enhanced notifications without interfering with the user's activities. Depending on the usage situation of the user terminal 1, the user terminal 1 to which the information notification device according to the first embodiment is applied enhances notifications so as not to include modals that may be disruptive due to specific sensory sensitivities or sensory dominance. This enables clear notifications to be provided without interfering with the activities of users with sensory sensitivities or dominance, such as document research, viewing, and creation, content viewing and production, programming, and various types of communication, using the user terminal 1, such as a PC or smartphone.

In addition, in a user terminal 1 to which the information notification device of the first embodiment is applied, the notification expansion unit 112 generates multiple modal information by expanding the notification received by the notification receiving unit 111 for all of the multiple modals, and the notification selection unit 114 selects one or more modal information from the multiple modal information generated by the notification expansion unit 112 based on both the user characteristic information and the user's situation, which does not constitute a disruptive stimulus for the user, and the notification distribution unit 115 distributes a notification including the one or more modal information selected by the notification selection unit 114.
Therefore, in the information notification device of the first embodiment, by selecting modal information to be delivered from multiple modal information expanded for all modalities based on both user characteristic information and the user's situation, it is possible to deliver a notification including one or more modal information that does not pose a disruptive stimulus to the user.

In addition, in a user terminal 1 to which the information notification device of the first embodiment is applied, a modal selection rule memory unit 132 stores selection rules based on user characteristic information and the user's situation, and the notification selection unit 114 selects one or more modal information items that do not constitute a disturbing stimulus for the user in accordance with the selection rules stored in the modal selection rule memory unit 132.
Therefore, in the information notification device according to the first embodiment, by storing selection rules in advance, it becomes possible to easily select modal information.

Second Embodiment
In the first embodiment, expanded modal information is generated for all modals, including text, voice, and facial expressions, in response to a received notification, and then modal information to be actually used is generated based on both user characteristic information and the user's situation. Alternatively, a modal to be used may be selected first based on both the user characteristic information and the user's situation, and then modal information may be generated only for the selected modal. The second embodiment is an example of such selective generation of modal information.

Hereinafter, configurations and processes similar to those in the first embodiment will be assigned the same reference numerals as those in the first embodiment, and their explanations will be omitted, and only the parts that differ from the first embodiment will be explained.

(Configuration example)
FIG. 18 is a block diagram showing an example of the software configuration of a user terminal 1 to which an information notification device according to the second embodiment of the present invention is applied.

In this second embodiment, the notification receiving unit 111 transmits the received notification to the notification selecting unit 114 in addition to the notification expanding unit 112.

In this second embodiment, the notification selection unit 114 acquires user characteristic information from the modal characteristic storage unit 131, and selects one or more modals from among multiple modals, excluding modals that pose a disruptive stimulus to the user, based on both this acquired user characteristic information and the user's situation grasped by the situation grasp unit 113. More specifically, the notification selection unit 114 selects one or more modals that do not pose a disruptive stimulus to the user, in accordance with selection rules based on the user characteristic information and the user's situation stored in the modal selection rule storage unit 132. The notification selection unit 114 transmits the result of the modal selection based on both this user characteristic information and the user's situation to the notification expansion unit 112.

In this second embodiment, the notification expansion unit 112 generates expanded modal information for only the modal selected by the notification selection unit 114 based on both the user characteristic information and the user's situation, in response to the notification received by the notification reception unit 111. The notification expansion unit 112 transmits this modal information generated based on both the user characteristic information and the user's situation to the notification distribution unit 115.

(Example of operation)
19 is a flowchart showing an example of the processing procedure and processing content of an information notification process executed by the control unit 11 of the user terminal 1 to which the information notification device according to the second embodiment is applied. In this embodiment, when it is determined in step S101 that a notification has been received, the control unit 11 operates as the situation grasping unit 113 and grasps the user's situation with respect to the user terminal 1 (step S103).

The control unit 11 also operates as the notification selection unit 114 and determines which modal to use from among the multiple modals (step S111). That is, the control unit 11 reads user characteristic information indicating the user's dominance and inferiority with respect to the multiple modals from the modal characteristic storage unit 131, and selects one or more modals from among the multiple modals based on this user characteristic information and the user's understood situation, excluding any modalities that may be disruptive stimuli for the user.

Then, the control unit 11 operates as a notification expansion unit 112 and expands the received notification only for modals selected from multiple modals such as text, voice, and facial expressions based on both the user characteristic information and the user's situation (step S112). As a result, the control unit 11 generates one or more pieces of modal information for the received notification that do not include modals that may be disruptive stimuli for the user.

In step S105, the control unit 11 operates as the notification delivery unit 115 and delivers to the user a notification including one or more pieces of modal information generated based on both the user characteristic information and the user's situation. The control unit 11 then proceeds to the processing of step S101 above.

(Actions and Effects)
As described above, in the user terminal 1 to which the information notification device of the second embodiment is applied, modal information that does not include modals that act as disruptive stimuli for the user can be delivered, similar to the first embodiment, based on both user characteristic information and the user's situation.

Furthermore, in a user terminal 1 to which the information notification device of the second embodiment is applied, the notification selection unit 114 selects one or more modals that do not pose a disruptive stimulus to the user based on both the user characteristic information and the user's situation, the notification expansion unit 112 generates one or more pieces of modal information by expanding the notification received by the notification receiving unit 111 to one or more modals selected by the notification selection unit 114 based on both the user characteristic information and the user's situation, and the notification distribution unit 115 distributes a notification including one or more pieces of modal information based on both the user characteristic information generated by the notification expansion unit 112 and the user's situation.
Therefore, in a user terminal 1 to which the information notification device of the second embodiment is applied, a modal to be used is selected from a plurality of expandable modals based on both user characteristic information and the user's situation, and expanded modal information is generated only for the selected modal, thereby making it possible to deliver a notification including one or more modal information that does not pose a disruptive stimulus to the user.

[Third embodiment]
In the first and second embodiments, an example in which the information notification device is applied to one user terminal 1 has been described, but some functions may be provided in a server computer installed on the web or in the cloud. This will be described below as the third embodiment. Note that although the description will be made in correspondence with the first embodiment, it goes without saying that the description can also be made in correspondence with the second embodiment.

(Configuration example)
20 and 21 are block diagrams showing an example of the hardware and software configurations of an information notification system to which an information notification device according to the third embodiment of the present invention is applied. The information notification system includes a plurality of user terminals 1, a distribution server 2, and a sender terminal 3, which are connected to each other so as to be able to communicate with each other via a network NW. Note that there may be a plurality of sender terminals 3, and FIGS. 20 and 21 show only one of them as a representative.

The user terminals 1 may belong to different users, or one user may be able to operate multiple user terminals 1. The hardware configuration of each user terminal 1 is the same as in the first embodiment. With regard to the software configuration, as shown in Figure 21, the control unit 11 of the user terminal 1 comprises a situation grasping unit 113 and a notification distribution unit 115 as processing function units according to the third embodiment of the present invention. Note that, due to space limitations, the program storage unit 12 is omitted from Figure 21.

The situation grasping unit 113 grasps the user's situation as described in the first embodiment, and transmits the grasped user's situation to the distribution server 2 via the network NW using the communication interface unit 14. However, in this embodiment, the situation grasping unit 113 operates in response to receiving, via the communication interface unit 14, an inquiry transmitted from the distribution server 2 via the network NW. The notification distribution unit 115 receives, via the communication interface unit 14, a notification including extended modal information transmitted from the distribution server 2 via the network NW. The notification distribution unit 115 then outputs the extended modal information contained in the received notification to the audio output device 174 and/or video output device 175 via the input/output interface unit 15.

As shown in FIG. 20, the distribution server 2 has a control unit 21 that uses a hardware processor such as a CPU, and this control unit 21 is connected to a storage unit having a program storage unit 22 and data storage unit 23, and a communication interface unit 24 via a bus 25. The control unit 21, program storage unit 22, data storage unit 23, communication interface unit 24, and bus 25 correspond to the control unit 11, program storage unit 12, data storage unit 13, communication interface unit 14, and bus 16 in the user terminal 1.

Regarding the software configuration of the distribution server 2, as shown in FIG. 21, the control unit 11 includes a notification receiving unit 211, a notification expansion unit 212, and a notification selection unit 214 as processing function units according to the third embodiment of the present invention. Furthermore, the data storage unit 23 includes, in its storage area, a modal characteristics storage unit 231, a modal selection rule storage unit 232, and a user information storage unit 233 as storage units necessary for implementing the third embodiment. Note that, due to space limitations, the program storage unit 22 is omitted from FIG. 21.

The modal characteristic storage unit 231 stores user characteristic information common to all users. Alternatively, the modal characteristic storage unit 231 may store different user characteristic information for each user. The modal selection rule storage unit 232 is similar to the modal selection rule storage unit 232 in the first embodiment. The user information storage unit 233 stores information for communicating with the user terminal 1 used by each user, such as an IP address, in other words, information for identifying each user's user terminal 1. The user information storage unit 233 may also store information for identifying multiple user terminals 1 for a single user. If the modal characteristic storage unit 231 stores user characteristic information for each user, it is possible to omit the user information storage unit 233 by storing information for identifying the user terminal 1 in association with each piece of user characteristic information in the modal characteristic storage unit 231.

The notification receiving unit 211 receives notifications addressed to each user, sent from one of the sender terminals 3 via the network NW, via the communication interface unit 24. The notification expansion unit 212 is similar to the notification expansion unit 112 in the first embodiment. Like the notification selection unit 114 in the first embodiment, the notification selection unit 214 selects one or more pieces of modal information that do not constitute a disruptive stimulus for the user, in accordance with selection rules based on the user characteristic information and the user's status stored in the modal selection rule storage unit 232. At this time, the notification selection unit 214 acquires the user's status from the corresponding user terminal 1 by sending a user status inquiry via the communication interface unit 24 to the status understanding unit 113 of the user terminal 1 of the user to whom the notification received by the notification receiving unit 211 is addressed, in accordance with information for identifying the user terminal 1 stored in the user information storage unit 233. Note that if a user can use multiple user terminals 1, some user terminals 1 may not be in operation. The user's status is not acquired from inoperational user terminals 1.

(Example of operation)
Figure 22 is a flowchart showing an example of the processing procedure and processing content of the information notification processing executed by the control unit 21 of the distribution server 2 included in the information notification system, and Figure 23 is a flowchart showing an example of the processing procedure and processing content of the information notification processing executed by the control unit 11 of the user terminal 1 included in the information notification system.

As shown in FIG. 22, the control unit 21 of the distribution server 2 first operates as a notification receiving unit 211 and determines whether a notification sent from any sender via the network NW has been received via the communication interface unit 24 (step S201). If it determines that a notification has not been received, the control unit 21 performs the processing of step S201 again. In this way, the control unit 21 waits to receive a notification.

If it is determined that a notification has been received, the control unit 21 operates as the notification expansion unit 212 and expands the received notification in all of the multiple modalities, such as text, voice, and facial expressions (step S202). In other words, the control unit 21 generates multiple modal information items corresponding to the received notification.

The control unit 21 also inquires about the status of the destination user of the received notification (step S203). Specifically, the control unit 21 obtains information for identifying the destination user's user terminal 1 from the user information storage unit 233, and uses the communication interface unit 24 to inquire about the user's status from the corresponding user terminal 1 via the network NW.

As shown in FIG. 23, the control unit 11 of the user terminal 1 first operates as the status assessment unit 113 and determines whether an inquiry transmitted from the distribution server 2 via the network NW has been received via the communication interface unit 14 (step S121). If it is determined that an inquiry has not been received, the control unit 11 proceeds to processing in step S123.

If it is determined that an inquiry has been received, the control unit 11 operates as the situation assessment unit 113, determining the input/output status of text, audio, and video at the user terminal 1, the orientation of the user's face toward the user terminal 1, and other information, and from the results of this assessment, assesses the user's situation relative to the user terminal 1 (step S103).

Then, the control unit 11 transmits the grasped user status to the distribution server 2 via the network NW via the communication interface unit 14 (step S122). The control unit 11 then proceeds to the processing of step S123.

As shown in FIG. 22, the control unit 21 operating as the notification selection unit 214 of the distribution server 2 acquires the user status information returned from the inquired user terminal 1 via the network NW via the communication interface unit 24 (step S204). Note that if the inquired user terminal 1 is not operating, the user status information will not be returned. Therefore, the status of the destination user acquired in step S204 includes the operating status of the user terminal 1 in addition to the user status.

Then, the control unit 21 selects modal information to use from the generated plurality of modal information (step S205). Specifically, the control unit 21 reads user characteristic information indicating the user's dominance and inferiority with respect to the plurality of modals from the modal characteristic storage unit 231, and selects one or more pieces of modal information from the generated plurality of modal information based on both this user characteristic information and the grasped user situation, which do not include any modal that will act as a disruptive stimulus for the user.

Furthermore, based on the acquired operating status of the user terminal 1, the control unit 21 determines the user terminal 1 to which a notification containing one or more pieces of modal information selected based on both this user characteristic information and the user status will be sent (step S206).

Then, the control unit 21 transmits a notification containing one or more pieces of modal information selected based on both this user characteristic information and the user's situation to the determined destination user terminal 1 via the network NW via the communication interface unit 24 (step S207). The control unit 21 then proceeds to the processing of step S201.

As shown in FIG. 23, if the control unit 11 of the user terminal 1 determines in step S121 that it has not received an inquiry, or after transmitting the user's status in step S122, it operates as the notification distribution unit 115 and determines whether it has received, via the communication interface unit 14, a notification containing one or more pieces of modal information selected based on both the user characteristic information and the user's status, transmitted from the distribution server 2 via the network NW (step S123). If it determines that it has not received such a notification, the control unit 11 proceeds to the processing of step S121.

If it is determined that a notification has been received, the control unit 11 delivers to the user a notification including one or more pieces of modal information selected based on both the received user characteristic information and the user's situation (step S105). Specifically, the control unit 11 outputs facial expression modal information and text modal information from the video output device 175 via the input/output interface unit 15, and outputs audio modal information from the audio output device 174. The control unit 11 then proceeds to the processing of step S121 above.

(Actions and Effects)
As described above, in an information notification system to which the information notification device of the third embodiment is applied, as in the first embodiment, modal information that does not include modals that act as disruptive stimuli for the user can be delivered based on both user characteristic information and the user's situation.

Furthermore, in the third embodiment, notifications addressed to multiple users can be processed collectively on the distribution server 2.

Furthermore, in the third embodiment, compared to the first and second embodiments in which all functions of the information notification device are performed by the user terminal 1, the user terminal 1 only needs to execute some of the functions related to the information notification device, which reduces the amount of operating resources used by the user terminal 1 when it is performing some kind of information processing task, minimizing the reduction in processing power that affects that information processing task.

[Other embodiments]
The present invention is not limited to the above-described embodiment.

For example, while text, voice, and facial expressions have been used as examples of modalities to be expanded, other modals may also be used, or other modals may also be included.

Furthermore, the flow of each process described with reference to the flowchart is not limited to the procedure described. For example, the order of the processes in step S102 and step S103 shown in Figure 6 may be reversed, or they may be performed simultaneously in parallel. In this way, the order of some steps may be changed, or some steps may be performed simultaneously in parallel. Furthermore, the processing content of some steps may be modified.

The program may be transferred in a state where it is stored in an electronic device, or in a state where it is not stored in an electronic device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory, tangible medium. The recording medium is a computer-readable medium. The form of the recording medium is not important, as long as it is capable of storing the program and is computer-readable, such as a CD-ROM or memory card.

Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations corresponding to the embodiments may be adopted as appropriate.

In short, this invention is not limited to the above-described embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all of the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 1...User terminal 2...Distribution server 3...Sender terminal 11, 21...Control unit 12, 22...Program storage unit 13, 23...Data storage unit 14, 24...Communication interface unit 15...Input/output interface unit 16, 25...Bus 111, 211...Notification receiving unit 112, 212...Notification expansion unit 113...Situation grasping unit 114, 214...Notification selection unit 115...Notification distribution unit 131, 231...Modal characteristic storage unit 132, 232...Modal selection rule storage unit 171...Input device 172...Audio input device 173...Video input device 174...Audio output device 175...Video output device 233...User information storage unit FE...Facial expression modal information NW...Network TX...Text modal information VO...Audio modal information

Claims (4)

a receiving unit for receiving a notification addressed to a user;
an acquisition unit that acquires user characteristic information indicating superiority and inferiority of the user with respect to a plurality of modals;
a grasping unit configured to grasp a status of the user with respect to the user's terminal;
an expansion unit that expands the notification received by the receiving unit to one or more modal information of the plurality of modals, based on the user characteristic information acquired by the acquisition unit and the user's situation grasped by the grasping unit, the modal information not including a modal that is a disturbing stimulus for the user;
a distribution unit that distributes a notification including the one or more pieces of modal information expanded by the expansion unit to the user;
An information notification device comprising: The extension portion is
a notification extension unit that generates a plurality of modal information by extending the notification received by the receiving unit for all of the plurality of modalities;
a selection unit that selects, from the plurality of pieces of modal information generated by the notification extension unit, one or more pieces of modal information that do not constitute a disturbing stimulus for the user, based on the user characteristic information and the user's situation;
Equipped with
The information notification device according to claim 1 , wherein the distribution unit distributes a notification including the one or more pieces of modal information selected by the selection unit. The extension portion is
a selection unit that selects one or more modals that do not become a disturbing stimulus for the user based on the user characteristic information and the user's situation;
a notification extension unit that generates the one or more pieces of modal information by extending the notification received by the receiving unit with respect to the one or more modals selected by the selecting unit;
Equipped with
The information notification device according to claim 1 , wherein the distribution unit distributes a notification including the one or more pieces of modal information generated by the notification extension unit. a storage unit that stores a selection rule based on the user characteristic information and the user situation,
The information notification device according to claim 2 , wherein the selection unit performs the selection in accordance with the selection rule stored in the storage unit.