JP7135335B2 - VIEWING HISTORY RECORDING SYSTEM, VIEWING HISTORY RECORDING DEVICE, RECORDING METHOD AND PROGRAM - Google Patents

Description

The present invention relates to a viewing history recording system, a viewing history recording device, a recording method, and a program.

In audience rating surveys, viewers are stratified into categories such as M1-M3, F1-F3, C, and T according to the gender and age of the viewers, and a method that is useful for program creation and marketing is adopted. . For example, the advertising industry distinguishes between men aged 20-34 as M1, men aged 35-49 as M2, and men over 50 as M3. Similarly, categories F1 to F3 are used for females. For children aged 4 to 12 and teenagers, C and T are used as the initials of these characters regardless of gender.

Patent Literature 1 discloses an information processing system capable of collecting attributes of an unspecified number of viewers and measuring viewing effects. According to this document, a wireless communication device transmits viewer data including a storage unit storing user data associated with a user and part or all of the user data stored in the storage unit. and a data transmission unit. Then, it is described that the viewing effect of the content reproduced by the viewing device is analyzed on the information processing device side based on the viewer data.

Patent Literature 2 discloses a television receiver capable of performing user-specific processing considered appropriate for a plurality of viewers. Specifically, it is disclosed that the viewer is specified from the characteristics of the operation of the remote controller, and that the viewer is specified from the viewing tendency of past programs.

Patent Literature 3 discloses a content display device capable of calculating the degree of concentration of a user on content as a degree of concentration without performing prior calibration.

JP 2010-41194 A JP 2008-277995 A JP 2016-111612 A

The following analysis is given by the present invention. As in Patent Literatures 1 and 2, there is known a configuration in which a user is specified and an audience rating is analyzed. I have a problem that I can't. For example, it is difficult to determine what the viewer is paying attention to on the screen of the television device with either of Patent Documents 1 and 2.

It is an object of the present invention to provide a viewing history recording system, a viewing history recording device, a recording method, and a program that can collect information about an object that a user is paying attention to in content and use it for marketing and content creation. do.

According to the first aspect, a photographing means for photographing a face image of a user viewing content being reproduced on a predetermined reproduction device, the face image of the user photographed by the photographing means, and a pre-registered authentication specifying means for specifying the user by using data for identifying the user; gaze area specifying means for specifying an area that the user is gazing at in the display area of the content from the face image of the user; and specifying the user. a creating means for creating viewing area transition information representing a transition of an area watched by the user watching the content in association with the attribute-based viewing history of the content using the result. A system of record is provided.

According to a second aspect, there is provided a viewing history recording device arranged together with a reproducing device in a viewing history recording system. This viewing history recording device can take the form of, for example, a device called a set-top box, or a peripheral device for a content reproducing device or a television device.

According to a third aspect, a viewing history recording device comprising a processor, a memory, and a photographing means for photographing a facial image of a user who is viewing content being reproduced by a predetermined reproduction device is provided with the photographing means. The user is identified by using the face image of the user photographed in and the authentication data registered in advance, and the area that the user is paying attention to in the display area of the content is identified from the face image of the user. creating viewing area transition information representing a transition of an area watched by the user viewing the content in association with the attribute-based viewing history of the content using the identification result of the user; There is provided a content viewing history recording method characterized by: This method is tied to a specific machine represented by the viewing history recording system and viewing history recording device described above.

According to a fourth aspect, a viewing history recording device equipped with a photographing means for photographing a face image of a user viewing content being reproduced on a predetermined reproducing device, and a computer provided with a processor and a memory. a process of identifying the user by using the face image of the user photographed by the photographing means and authentication data registered in advance; A transition of the region watched by the user watching the content is represented by associating a process of identifying the region watched by the user with the viewing history by attribute of the content using the identification result of the user. A program for executing a process of creating viewing area transition information is provided. This program can be recorded in a computer-readable (non-transitory) storage medium. That is, the present invention can also be embodied as a computer program product.

According to the present invention, it is possible to collect information about an object that a user is paying attention to in content, and use the information for marketing and content creation.

It is a figure which shows the structure of one Embodiment of this invention. 1 is a diagram showing the configuration of a viewing history recording device according to a first embodiment of the present invention; FIG. FIG. 4 is a diagram for explaining the operation (viewing history recording) of the viewing history recording device according to the first embodiment of the present invention; FIG. 4 is a diagram for explaining the operation (gazing area specification) of the viewing history recording device according to the first embodiment of the present invention; FIG. 4 is a diagram for explaining an example of transition data of a user's attention area recorded by the viewing history recording device according to the first embodiment of the present invention; FIG. 4 is a diagram for explaining the effects of the viewing history recording device according to the first embodiment of the present invention; FIG. 10 is a diagram showing the configuration of a viewing history recording device according to a second embodiment of the present invention; FIG. 11 is a diagram showing the configuration of a viewing history recording device according to a third embodiment of the present invention; FIG. 11 is a diagram for explaining the operation (facial expression determination) of the viewing history recording device according to the third embodiment of the present invention; FIG. 11 is a diagram for explaining an example of transition data of a user's facial expression recorded by the viewing history recording device according to the third embodiment of the present invention; FIG. 13 is a diagram showing a modified configuration of the viewing history recording device according to the third embodiment of the present invention; FIG. 11 is a diagram showing the configuration of a viewing history recording device according to a fourth embodiment of the present invention; FIG. 12 is a diagram for explaining the operation (orientation determination) of the viewing history recording device according to the fourth embodiment of the present invention; FIG. 12 is a diagram for explaining an example of user posture transition data recorded by the viewing history recording device according to the fourth embodiment of the present invention; It is a figure which shows the modified structure of the viewing-and-listening history recording device of the 4th Embodiment of this invention. 1 is a diagram showing the configuration of a computer that constitutes the viewing history recording apparatus of the present invention; FIG.

First, an outline of an embodiment of the present invention will be described with reference to the drawings. It should be noted that the drawing reference numerals added to this overview are added to each element for convenience as an example to aid understanding, and are not intended to limit the present invention to the illustrated embodiments. Also, connection lines between blocks in drawings and the like referred to in the following description include both bidirectional and unidirectional connections. The unidirectional arrows schematically show the flow of main signals (data) and do not exclude bidirectionality. Moreover, in the following description, "A and/or B" is used to mean at least one of A and B.

In one embodiment of the present invention, as shown in FIG. 1, a viewing history recording system 10 comprising photographing means 11, specifying means 12, creating means 13a, and gaze area specifying means 14 is realized. can. It should be noted that the above-described means of the viewing history recording system 10 may be arranged in a single device such as a set-top box, or part of the means may be arranged on the content reproduction device or the network side. . That is, the viewing history recording system 10 of the present invention can also be configured by combining a plurality of devices.

More specifically, the photographing means 11 is composed of a camera or the like for photographing a facial image of a user viewing content being reproduced by a predetermined reproducing apparatus.

The specifying means 12 specifies the user by using the face image of the user photographed by the photographing means 11 and authentication data registered in advance. For example, the identification unit 12 calculates feature amounts from the user's face image. Then, the user having the feature closest to the calculated feature amount (that is, the face is similar) is specified from the authentication data.

Then, the gaze area specifying means 14 specifies an area that the user is gazing at (hereinafter also referred to as "gazing area") in the display area of the content from the face image of the user. A gaze area specifying means 14 estimates the direction of the user's line of sight from the user's face image. Various methods can be used for estimating the line-of-sight direction. can be estimated from the amount of deviation from the position of (see Patent Document 3). Of course, if the orientation of the user's face can be estimated from the glasses worn by the user or from other wearable sensors, the orientation of the user's face may be estimated using the data acquired from these devices.

Then, the gaze area identifying means 14 identifies the area that the user is gazing at based on the estimated direction of the user's line of sight and the relative position between the photographing means and the playback device. Note that the area that the user is gazing at may be an approximate range, and very simply, the gazing area specifying means 14 determines which of the three areas of the content, left, center, and right, is the user's gaze. It may be in a specified form. In a more desirable form, the attention area specifying means 14 divides the content display area of the playback device into nine areas such as upper left, left, lower left, upper, center, lower, upper right, right, and lower right. may be specified as the fixation area of . Of course, it is also possible to employ a form in which the gaze area specifying means 14 specifies the user's gaze area in the form of coordinate information or the like.

The creating means 13a creates viewing area transition information representing a transition of an area watched by the user watching the content, in association with the attribute-based viewing history of the content using the identification result of the user. do.

The viewing history recording system 10 configured as described above collects, in addition to the attribute-based viewing history 20 of the content, the gaze region of the user viewing the content as viewing region transition information. As shown in FIG. 1, this viewing area transition information may be recorded as gaze area transition information 21, separate from the attribute-based viewing history 20, or may be recorded as accompanying data of the content attribute-based viewing history. may be recorded as

According to the above-described embodiments, in addition to viewing history information represented by audience ratings, it is possible to collect information about the target of the user's attention in the content and use it for marketing and content creation. Become.

For example, according to Patent Literatures 1 and 2, it is possible to obtain information that the audience rating of the M1 layer contributes to the audience rating of a certain program, or conversely that the program is not watched at all by the F3 layer. According to the present invention, it is possible to obtain information such as when a performer appearing in a program contributes greatly, or when the content of the program itself is popular.

Further, in a more desirable form, it is preferable that the gaze area specifying means 14 can record the time and the degree of gaze of the user on each area. By doing so, it is possible to grasp the region that has been watched by the user for a long time and the movement of the user's line of sight. Further, the degree of fixation may be presented in the form of a map in which colors or shades are used for each of the areas. Furthermore, the size of the area for recording the gaze area may be changed according to the attributes of the user and the movement of the line of sight. For example, when the movement of the user's line of sight is small, i.e., when the user is looking at the screen vaguely, the line of sight is recorded in three regions, left, center, and right. In this way, the recording granularity, that is, the size of the recording area may be changed dynamically.

[First embodiment]
Next, a first embodiment in which the present invention is applied to an audience rating survey of television broadcasting will be described in detail with reference to the drawings. FIG. 2 is a diagram showing the configuration of the viewing history recording device according to the first embodiment of the present invention. Referring to FIG. 2, a viewing history recording device 10a including photographing means 11, specifying means 12, recording means 13, gaze area specifying means 14, and authentication data storage means 16, and storage on the network. 15 are shown. Such a viewing history recording device can take the form of a small information appliance such as a smart speaker (AI speaker) as well as a set-top box (STB) form.

The photographing means 11 is composed of a camera or the like for photographing a facial image of a user who is viewing program content displayed on a predetermined television receiver. From the viewpoint of estimating the direction of the user's line of sight, it is preferable that the photographing means 11 be fixed at a predetermined position with respect to the display surface of the television receiver.

The specifying means 12 specifies the user by using the face image of the user photographed by the photographing means 11 and the authentication data stored in the authentication data storage means 16 . For example, the identification unit 12 calculates feature amounts from the user's face image. Then, the user having the feature closest to the calculated feature amount (that is, the face is similar) is specified from the authentication data. The authentication data is composed of, for example, data in which the feature amount of the user's face image is associated with information about the user's attributes. In this embodiment, it is assumed that the information on the user's attribute is a category representing the eight types of user layers M1 to M3, F1 to F3, C, and T described above. The identification unit 12 preferably has a function of receiving registration of authentication data using the user's face image photographed by the photographing unit 11 and registering it in the authentication data storage unit 16 . More desirably, the identifying means 12 has a function of automatically determining a category representing the user group from the user's face image photographed by the photographing means 11 .

The gaze area specifying means 14 specifies an area (hereinafter also referred to as a "gazing area") that the user watching the program content is watching from the user's face image captured by the photographing means 11 . Since the method of specifying the gaze area is the same as that of the gaze area specifying means 14 of the viewing history recording system 10, the explanation is omitted.

A recording means 13 stores an attribute-based viewing history in which channel selection information sent from a television receiver, the viewing history of the content of the specified user, and information about the attribute of the specified user are associated with each other. to record. Also, the recording means 13 records the transition of the gaze area specified by the gaze area specifying means 14 in the storage 15 as the gaze area transition information in association with the channel selection information sent from the television receiver. Therefore, in the present embodiment, the recording means 13 functions as the creating means 13a described above.

The storage 15 is arranged on the Internet or a network of a viewing rate analysis company, and is used as a storage destination of viewing history by attribute and attention area transition information. Also, by using this storage 15 as a storage destination for attribute-based viewing histories and attention area transition information of other viewing history recording devices, attribute-based viewing histories and attention area transition information are collected. The data accumulated in the storage 15 is provided to an external node such as a predetermined analysis server and used for audience rating analysis.

Next, the operation of this embodiment will be described in detail with reference to the drawings. FIG. 3 is a diagram for explaining the operation (viewing history recording) of the viewing history recording device according to the first embodiment of the present invention. In the following description, it is assumed that users F2, M1, and M3 and their face images are registered in advance in the identifying means 12, and user M1 among them has started viewing on the television receiver.

First, when the user M1 starts viewing on the television receiver, the photographing means 11 photographs the face image of the user M1 (11 in FIG. 3).

Next, the specifying unit 12 refers to the authentication data stored in the authentication data storage unit 16, and determines whether the user who started viewing is the user M1 from the face image of the user M1 photographed by the photographing unit 11. (12 in FIG. 3).

Next, the recording means 13 associates the specified user attribute with the channel selection information, creates an attribute-specific viewing history, and starts recording in the storage 15 (13 in FIG. 3).

The lower part of FIG. 3 shows an example of viewing history by attribute. The example in the lower part of FIG. 3 shows attribute-specific viewing histories in which viewing start date/time, user attribute, viewing content (channel selection information), and viewing end date/time are associated with each other. By collecting such data, it becomes possible to obtain not only the audience rating of a certain program but also the breakdown of the audience rating by attribute.

Next, a process of identifying a region of interest by the viewing history recording device 10a of the present embodiment will be described. FIG. 4 is a diagram for explaining the attention area specifying process of the viewing history recording device 10a of the present embodiment.

Referring to FIG. 4, first, the photographing means 11 transmits a photographed image to the gaze area specifying means 14 (step S001). Here, the photographed image to be transmitted to the gaze area specifying means 14 may be the face image of the user photographed for specifying the user when creating the viewing history by attribute shown in FIG. Of course, the image may be an image separately photographed by the photographing means 11 in order to specify the gaze area by the gaze area specifying means 14 .

Next, the gaze area identifying means 14 identifies the user's position and line of sight direction from the user's face image received from the photographing means 11 (step S002). Here, the user's position can be calculated from the position and size of the user's face image. In addition, the direction of the user's line of sight can be estimated from the amount of deviation between the position of each organ in the user's face image and the position of each organ of the face calculated in advance from the authentication data (see Patent Document 3). ).

Next, the gaze area specifying means 14 calculates the user's gaze area on the display screen of the television receiver (step S003). The gaze region of the user can be calculated based on the estimated direction of the user's line of sight and the relative position between the photographing means 11 and the television receiver.

Then, the gaze area specifying means 14 transmits the calculated gaze area of the user to the recording means 13 (step S004).

The recording means 13 sequentially records the transition of the user's gaze area received from the gaze area specifying means 14 as attached data of the attribute-based viewing history (step S005). While the user continues to watch the program, the photographing means 11, the attention area specifying means 14, and the recording means 13 repeat the processes of steps S001 to S005 to create transition data of the user's attention area. In this way, by recording the transition data of the user's attention area in parallel with program viewing, the transition data of the user's attention area corresponding to the time of the viewing history by attribute is created.

FIG. 5 shows an example of transition data of a user's attention area recorded by the viewing history recording device 10a. In the example of FIG. 5, data recording changes in the user's attention area during program viewing is recorded as attention area transition in association with attribute-based viewing history information. Note that the gaze area may not be recorded each time a face image of the user is received from the photographing means 11, but may be recorded only when the gaze area changes.

By aggregating a large number of transition data of the region of interest as described above, it is possible to cause an arbitrary external analysis server or the like to perform the following analysis. For example, as shown in FIG. 6, the analysis server aggregates the most focused regions of the user group (attribute) of interest for each scene broadcasted at 18:00 on February 1, 2018, and compares them with the content of the program. Then, data is created that enables grasping which part of the program attracts the attention of the target user class (attribute).

For example, in the example of FIG. 6, two guests appear from 18:07 during the program. At that time, it can be seen that the line of sight of the M1 layer was concentrated on the right side of the screen. As a result, it can be seen that the line of sight was concentrated on Mr. A among the two guests. Similarly, at 18:15 during the program, the program advertisement by the guests starts, but the region of the highest attention of the M1 layer at that time is the lower part of the screen. From this, it can be seen that the effect of the program advertisement by the guest was sufficient for at least the M1 layer. Similarly, the most frequently watched area of the M1 layer during the CM (Commercial Message) period from 18:20 during the program is the left side of the screen. As a result, for example, it can be seen that M1 is interested in the product on the left side of the three products displayed on the screen.

Furthermore, by performing the same analysis as above for other user groups and comparing them, it is possible to guess performers and performances that are highly appealing to each user group. As a result, for example, it is possible to create programs and commercials that take into account the attributes of major customers of program sponsors.

Further, in the above-described embodiment, at least the photographing means 11, the specifying means 12, and the gaze area specifying means 14 of the viewing history recording device 10a are arranged in the home of the user. Then, the recording means 13 records in the storage 15 as attribute-based viewing history information in which information of individual users is converted (anonymized) into information of a user group. The analysis of the audience rating and the like are performed by accessing the storage 15 that stores the anonymized data. Therefore, sensitive personal information such as the user's facial image and the associated viewing history will not be leaked to the outside of the user.

In the above example, a single user starts viewing the content, and the attention area is recorded. In the procedure, it is also possible to identify and record the viewing history by attribute for each user. Also, in this case, the transition of the gazed area is recorded for each user.

[Second embodiment]
Next, a second embodiment in which prior registration of authentication data can be omitted will be described in detail with reference to the drawings. FIG. 7 is a diagram showing the configuration of a viewing history recording apparatus according to the second embodiment of the present invention.

Referring to FIG. 7, a configuration of a viewing history recording device 10b having an estimating means 12a and a user attribute estimating engine 16a instead of the combination of the identifying means 12 and the authentication data storage means 16 in the first embodiment is shown. ing. Since other configurations are the same as those of the first embodiment, the differences will be mainly described below.

The estimating means 12a estimates user attributes using a user attribute estimation engine 16a. For example, the estimating means 12a performs predetermined preprocessing on the face image of the user photographed by the photographing means 11, and then sends it to the user attribute estimating engine 16a. The user attribute estimation engine 16a estimates gender and age group as attributes of the user based on features such as hairstyle, wrinkles, and expression of the user's facial image received from the estimation means 12a. As a specific example of the user attribute estimation engine 16a, FieldAnalyst (registered trademark) or the like, which is commercialized by the applicants, can be used.

Since the basic operation of this embodiment is the same as that of the first embodiment except that the estimation unit 12a estimates the attributes of the user in user authentication, the description is omitted.

As described above, this embodiment has an advantage over the first embodiment in that authentication data need not be prepared in advance. This advantage means that attribute-based viewing history data of an unspecified number of users can be acquired. This makes it possible to efficiently collect viewing history data for genders and age groups with a small amount of data.

[Third Embodiment]
Next, in addition to the viewing history, a detailed description will be given with reference to the drawings of a third embodiment in which transitions in the user's facial expression during viewing of content can be collected. FIG. 8 is a diagram showing the configuration of a viewing history recording device according to the third embodiment of the present invention.

Referring to FIG. 8, there is shown the configuration of a viewing history recording device 10c in which facial expression determination means 17 is added to the configuration of the first embodiment. Since other configurations are the same as those of the first embodiment, the differences will be mainly described below.

The facial expression determination means 17 determines the facial expression of the user from the user's face image photographed by the photographing means 11 . Various methods can be adopted as the facial expression determination method in the facial expression determination means 17 . For example, a method can be employed in which the shape and position of each organ of the face in the authentication data of the user are rendered expressionless, and the facial expression is determined by using the change or amount of movement of each organ from the facial expression. As another method, a user's facial expression is compared by comparing pre-registered facial expression patterns such as "smile", "sad face", and "expressionless" with features extracted from the user's facial image. A determination method can also be adopted. Also, at this time, face images such as a "smiling face", a "sad face", and an "expressionless face" may be prepared for each user in advance for verification. As a result, it is possible to grasp the habits of each user's facial expression and determine the facial expression with high accuracy.

Subsequently, facial expression determination processing by the viewing history recording device 10c of the present embodiment will be described. FIG. 9 is a diagram for explaining facial expression determination processing of the viewing history recording device 10c of the present embodiment.

Referring to FIG. 9, first, the photographing means 11 transmits a photographed image to the facial expression determining means 17 (step S001). Here, the photographed image to be transmitted to the facial expression determination means 17 may be the face image of the user photographed to identify the user when creating the viewing history by attribute shown in FIG. Of course, it may be an image separately photographed by the photographing means 11 for facial expression judgment by the facial expression judging means 17 .

Next, the facial expression determining means 17 determines the facial expression of the user from the user's facial image received from the imaging means 11 (step S102).

Next, the facial expression determination means 17 transmits the determined facial expression of the user to the recording means 13 (step S104).

The recording unit 13 sequentially records the transition of the user's facial expression received from the facial expression determination unit 17 as attached data of the viewing history by attribute (step S005). While the user continues to watch the program, the photographing means 11, the facial expression determining means 17 and the recording means 13 repeat the processes of steps S001 to S005 to create transition data of the user's facial expression. In this embodiment as well, by recording the transition data of the user's facial expression in parallel with viewing the program, the transition data of the user's facial expression corresponding to the time of the viewing history by attribute is created. Therefore, in the present embodiment, the recording means 13 also functions as means for creating transition data of the user's facial expression described above.

FIG. 10 shows an example of user facial expression transition data recorded by the viewing history recording device 10c. In the example of FIG. 10, data recording changes in the user's facial expression while viewing a program is recorded as facial expression transitions in association with attribute-specific viewing history information. The information about the facial expression of the user may be recorded only when there is a change in the facial expression instead of being recorded each time the image of the user's face is received from the photographing means 11 .

By aggregating a large number of facial expression transition data as described above, the following analysis can be performed. For example, assume that a change in facial expression as shown in FIG. 10 is obtained from a user belonging to the user layer F2 for the broadcast on February 2, 2018 at 21:00. By collecting similar data from users belonging to the other user layer F2, it is possible to grasp whether or not there was a response intended by the producer from the target user layer in each scene of the program.

Further, as in the first embodiment, more detailed analysis can be performed by combining information about the user's gaze area. For example, as shown in FIG. 6, it is possible to grasp the facial expression of the user in addition to the fact that the line of sight is focused on the guest appearing in the program. In addition, it is possible to collect information such as what kind of facial expressions were made by those who were not interested in the guests.

In the above-described embodiment, as shown in FIG. 8, the viewing history recording device 10c has been described as having the gaze area specifying means 14. However, as shown in FIG. 11, the gaze area specifying means 14 is omitted. It is also possible to adopt a configuration with

[Fourth embodiment]
Next, a detailed description will be given of a fourth embodiment in which it is possible to collect not only the viewing history but also the transition of the user's posture while viewing the content, with reference to the drawings. FIG. 12 is a diagram showing the configuration of a viewing history recording device according to the fourth embodiment of the present invention.

Referring to FIG. 12, there is shown the configuration of a viewing history recording device 10e in which posture determination means 18 is added to the configuration of the first embodiment. Since other configurations are the same as those of the first embodiment, the differences will be mainly described below.

Posture determination means 18 determines the posture of the user from an image including the face image of the user photographed by photographing means 11 . Various methods can be adopted as a facial expression determination method in the posture determination means 18 . For example, if the face image of the user is lying down, it may be determined that the user is lying down and watching. As another method, pre-registered posture patterns such as "laying down", "watching with arms crossed", "watching while sitting on a sofa or the like and relaxing", and an image containing the user's face image are extracted. It is also possible to adopt a method of determining the posture of the user by comparing it with information. In addition, at this time, for each user, reference images are prepared in advance for each user's posture, such as "laying down", "viewing with arms folded", and "viewing while sitting on a sofa or the like in a relaxed manner", and collation is performed. can be As a result, it becomes possible to determine the posture with high accuracy by understanding the habits of each user's posture.

Subsequently, posture determination processing by the viewing history recording device 10e of the present embodiment will be described. FIG. 13 is a diagram for explaining posture determination processing of the viewing history recording device 10e of the present embodiment.

Referring to FIG. 13, first, the photographing means 11 transmits a photographed image to the posture determining means 18 (step S001). Here, the photographed image to be transmitted to the posture determination means 18 may be the face image of the user photographed to identify the user when creating the viewing history by attribute shown in FIG. Of course, the image may be an image separately photographed by the photographing means 11 for facial expression judgment by the posture judging means 18 .

Next, posture determination means 18 determines the posture of the user from the image including the user's face image received from photographing means 11 (step S202).

Next, posture determining means 18 transmits the determined user posture to recording means 13 (step S204).

The recording unit 13 sequentially records the transition of the user's posture received from the posture determination unit 18 as attached data of the viewing history by attribute (step S005). While the user continues to watch the program, the photographing means 11, posture determining means 18, and recording means 13 repeat the processes of steps S001 to S005, thereby creating posture transition data of the user. In this embodiment as well, by recording the user's posture transition data in parallel with program viewing, the user's posture transition data corresponding to the time of the viewing history by attribute is created. Therefore, in the present embodiment, the recording means 13 also functions as means for creating the transition data of the user's posture described above.

FIG. 14 shows an example of user posture transition data recorded by the viewing history recording device 10e. In the example of FIG. 14, data recording changes in the user's posture while viewing a program are recorded as posture transitions in association with attribute-specific viewing history information. The information about the posture of the user may not be recorded each time a face image of the user is received from the photographing means 11, but may be recorded only when there is a change in facial expression.

By aggregating a large number of posture transition data as described above, the following analysis can be performed. For example, assume that a change in posture as shown in FIG. 14 is obtained from a user belonging to the user layer F2 for the broadcast on February 2, 2018 at 21:00. By collecting similar data from users who belong to the other user group F2, it is possible to grasp the attitude with which the target user group of the program viewed the program.

Further, as in the first embodiment, more detailed analysis can be performed by combining information about the user's gaze area. For example, as shown in FIG. 6, in addition to the fact that the line of sight is focused on the guest appearing in the program, it is possible to grasp the posture of the user. In addition, it is possible to collect information such as what kind of postures those who were not interested in guests were doing.

In the above-described embodiment, as shown in FIG. 12, the viewing history recording device 10e is provided with the gaze area specifying means 14. However, as shown in FIG. 15, the gaze area specifying means 14 is omitted. It is also possible to adopt a configuration with

Although each embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiments, and further modifications, replacements, and adjustments can be made without departing from the basic technical idea of the present invention. can be added. For example, the network configuration, the configuration of each element, and the form of expression of messages shown in each drawing are examples for helping understanding of the present invention, and are not limited to the configurations shown in these drawings. For example, in the above-described embodiment, users are classified into eight categories M1 to M3, F1 to F3, C, and T according to their gender and age. It is not limited. Very simply, it may be a rough division such as male, female, adult, and minor.

Further, in each of the above-described embodiments, an example has been given assuming an audience rating survey of terrestrial television broadcasting, but it is possible to apply the present invention to surveys of user reactions to other video content in general. Such video content includes cable television, satellite broadcasting, on-demand distribution content, and the like. In addition, the present invention is not limited to such content producers, and can be used, for example, as a tool for showing viewers' reaction to a user who has posted certain video content.

Moreover, each of the embodiments described above can be combined as appropriate. For example, by combining the third embodiment and the fourth embodiment, it is possible to configure a viewing history recording apparatus that records transitions of the user's facial expression and posture while viewing a program. Similarly, based on the viewing history recording device 10b of the second embodiment, it is also possible to employ a form in which one or both of the third embodiment and the fourth embodiment are combined.

Furthermore, in each of the above-described embodiments, transitions of the user's facial expression and posture while watching a program were recorded. may be recorded as auxiliary data. Examples of such physical characteristics include information obtained directly from facial images, such as whether glasses are worn, hairstyle, degree of fatigue, and degree of rough skin. Also, the user may extract body structure, clothing, and the like from the face image and the image of the neck part of the facial image, and record them as auxiliary data of the viewing history data. In this case, the viewing history recording device is provided with physical feature identifying means for identifying the physical features of the user.

As a result, for example, we can conduct activities such as soliciting sponsors from companies that want to sell products, sending individual direct mails, and reflecting them in interlocking advertisements for those who have received a favorable response to certain content. It can be done more effectively.

Further, the procedures shown in the first to fourth embodiments described above cause the computers (9000 in FIG. 16) functioning as the viewing history recording devices 10a to 10f to realize the functions as the viewing history recording devices 10a to 10f. It can be realized by a program. Such a computer is exemplified by a configuration including a CPU (Central Processing Unit) 9010, a communication interface 9020, a memory 9030, and an auxiliary storage device 9040 in FIG. That is, the CPU 9010 in FIG. 16 may execute a preprocessing program for a user's face image and a program for specifying a gaze area, and update processing of each calculation parameter held in the auxiliary storage device 9040 or the like may be executed.

That is, each part (processing means, function) of the viewing history recording devices 10a to 10f shown in each of the above-described embodiments executes each of the above-described processes using the hardware of the processors installed in these devices. It can be implemented by a computer program that causes

Finally, preferred forms of the invention are summarized.
[First form]
(See the viewing history recording system from the first viewpoint above.)
[Second form]
In the viewing history recording system described above,
The attributes of the user include at least the user's gender and age group,
The creation means may take a configuration that creates the viewing area transition information for each gender and age group of the user.
[Third form]
In the viewing history recording system described above,
The viewing area transition information is stored in a predetermined storage and used for audience rating analysis.
[Fourth form]
The photographing means, the specifying means, and the attention area specifying means of the viewing history recording system described above are arranged in the user's home,
A configuration may be adopted in which the predetermined storage provides, in response to a request from the outside, the attribute-based viewing history of the content and the transition of the area watched by the user who is viewing the content.
[Fifth form]
The viewing area transition information may be provided in association with the content viewed by the user.
[Sixth form]
It is also preferable that the viewing history recording system described above uses the viewing area transition information to provide map information indicating the degree of gaze of the user based on the time the user gazed or the movement of the user's gaze.
[Seventh form]
In the viewing history recording system described above,
It is also preferable that the creating means creates the viewing area transition information by changing the size of the area to be recorded based on the movement of the user's line of sight.
[Eighth mode]
Instead of identifying means for identifying the user by using the face image of the user photographed by the photographing means and authentication data registered in advance,
It is also possible to adopt a configuration including estimation means for estimating attributes of the user based on the face image of the user photographed by the photographing means.
[Ninth form]
In the viewing history recording system described above,
moreover,
facial expression determination means for determining the facial expression of the user viewing the content from the facial image of the user;
It is also possible to employ a configuration including a second creation means for creating facial expression transition information representing facial expression transitions of the user for each gender and age group of the user.
[Tenth mode]
In the viewing history recording system described above,
The facial expression transition information may be provided in association with content viewed by the user.
[Eleventh form]
In the viewing history recording system described above,
moreover,
Posture determining means for determining a posture of the user viewing the content from the face image of the user;
A configuration including third creating means for creating posture transition information representing the posture transition of the user for each gender and age group of the user can also be adopted.
[Twelfth form]
In the viewing history recording system described above,
The posture transition information may be provided in association with content viewed by the user.
[Thirteenth Mode]
(Refer to the viewing history recording device from the second viewpoint above)
[14th mode]
(See the recording method of the viewing history from the third viewpoint above.)
[15th mode]
(Refer to the program from the fourth viewpoint above)
It should be noted that the thirteenth to fifteenth modes can be developed into the second to twelfth modes in the same manner as the first mode.

It should be noted that each disclosure of the above patent documents is incorporated herein by reference. Within the framework of the full disclosure of the present invention (including the scope of claims), modifications and adjustments of the embodiments and examples are possible based on the basic technical concept thereof. Various combinations or selections of various disclosure elements (including each element of each claim, each element of each embodiment or example, each element of each drawing, etc.) are possible within the framework of the disclosure of the present invention. is. That is, the present invention naturally includes various variations and modifications that can be made by those skilled in the art according to the entire disclosure including claims and technical ideas. In particular, any numerical range recited herein should be construed as specifically recited for any numerical value or subrange within that range, even if not otherwise stated.

10 viewing history recording system 10a to 10f viewing history recording device 11 photographing means 12 identifying means 12a estimating means 13 recording means 13a creating means 14 gaze area identifying means 15 storage 16 authentication data storage means 16a user attribute estimating engine 17 facial expression determining means 18 Posture Determining Means 20 Viewing History by Attribute 21 Gaze Area Transition Information 9000 Computer 9010 CPU
9020 Communication interface 9030 Memory 9040 Auxiliary storage device

Claims (14)

a photographing means for photographing a facial image of a user viewing content being reproduced by a predetermined reproduction device;
identifying means for identifying the user by using the face image of the user photographed by the photographing means and authentication data registered in advance;
gaze region specifying means for specifying, from the face image of the user, a region that the user is gazing at in the display region of the content;
creating means for creating viewing area transition information representing a transition of an area watched by the user watching the content in association with an attribute-by -attribute viewing history for each attribute representing a user's gender and age group ;
an estimation unit that analyzes the viewing area transition information and infers an element with a high appeal effect in the content;
facial expression determination means for determining the facial expression of the user viewing the content from the facial image of the user;
a second creation means for creating facial expression transition information representing the transition of the user's facial expression;
analysis means for analyzing the facial expression of the user when the line of sight is focused on the element with the high appeal effect, based on the facial expression transition information;
A viewing history recording system comprising: 2. The viewing history recording system according to claim 1 , wherein said viewing area transition information is stored in a predetermined storage and used for viewing rate analysis. at least the photographing means, the specifying means, and the gaze area specifying means are arranged in the user's home,
3. The viewing history recording system according to claim 2 , wherein said viewing area transition information is provided in response to an external request via said predetermined storage. 4. The viewing history recording system according to claim 1 , wherein the viewing area transition information is provided in association with the content viewed by the user. 5. The viewing history recording system according to claim 4 , wherein said viewing area transition information is provided in the form of map information indicating the degree of gaze of said user based on the time or line of sight movement of said user. 6. The creation means creates the viewing area transition information by changing the size of the area specified by the gaze area specifying means and which the user is gazing , based on the movement of the user's line of sight. Any one viewing history recording system. Instead of identifying means for identifying the user by using the face image of the user photographed by the photographing means and authentication data registered in advance,
7. The viewing history recording system according to any one of claims 1 to 6 , further comprising estimating means for estimating attributes of said user based on said user's face image photographed by said photographing means. 7. The viewing history recording system according to any one of claims 1 to 6 , wherein said facial expression transition information is provided in association with content viewed by said user. moreover,
Posture determining means for determining a posture of the user viewing the content from the face image of the user;
9. The viewing history recording system according to any one of claims 1 to 8 , further comprising third creating means for creating posture transition information representing the posture transition of the user for each gender and age group of the user. 10. The viewing history recording system according to claim 9 , wherein said posture transition information is provided in association with content viewed by said user. a photographing means for photographing a facial image of a user viewing content being reproduced by a predetermined reproduction device;
estimating means for estimating attributes representing the gender and age group of the user based on the facial image of the user captured by the imaging means;
gaze region specifying means for specifying, from the face image of the user, a region that the user is gazing at in the display region of the content;
creating means for creating viewing area transition information representing a transition of an area watched by the user watching the content in association with the attribute-based viewing history for each attribute;
an estimation unit that analyzes the viewing area transition information and infers an element with a high appeal effect in the content;
facial expression determination means for determining the facial expression of the user viewing the content from the facial image of the user;
a second creation means for creating facial expression transition information representing the transition of the user's facial expression;
analysis means for analyzing the facial expression of the user when the line of sight is focused on the element with the high appeal effect, based on the facial expression transition information;
A viewing history recording system comprising: a photographing means for photographing a facial image of a user viewing content being reproduced by a predetermined reproduction device;
identifying means for identifying the user by using the face image of the user photographed by the photographing means and authentication data registered in advance;
gaze region specifying means for specifying, from the face image of the user, a region that the user is gazing at in the display region of the content;
creating means for creating viewing area transition information representing a transition of an area watched by the user watching the content in association with an attribute-by -attribute viewing history for each attribute representing a user's gender and age group ;
an estimation unit that analyzes the viewing area transition information and infers an element with a high appeal effect in the content;
facial expression determination means for determining the facial expression of the user viewing the content from the facial image of the user;
a second creation means for creating facial expression transition information representing the transition of the user's facial expression;
analysis means for analyzing the facial expression of the user when the line of sight is focused on the element with the high appeal effect, based on the facial expression transition information;
A viewing history recording device. A viewing history recording device comprising a processor, a memory, and an imaging means for capturing an image of the face of a user viewing content being reproduced by a predetermined reproducing device,
identifying the user by using the face image of the user captured by the imaging means and authentication data registered in advance;
identifying an area that the user is gazing in the display area of the content from the face image of the user;
creating viewing area transition information representing the transition of the area where the user is watching the content, in association with the attribute-specific viewing history for each attribute representing the user's gender and age group ;
Analyzing the viewing area transition information to infer an element with a high appeal effect in the content,
determining the expression of the user while viewing the content from the face image of the user;
creating facial expression transition information representing the transition of the user's facial expression;
analyzing the facial expression of the user when the user's line of sight is focused on the element with a high appeal effect, based on the facial expression transition information ;
A content viewing history recording method characterized by: A computer equipped with a viewing history recording device equipped with a photographing means for capturing a facial image of a user viewing content being reproduced on a predetermined reproducing device, and having a processor and a memory,
a process of identifying the user by using the face image of the user captured by the imaging means and authentication data registered in advance;
a process of identifying an area that the user is gazing at in the display area of the content from the face image of the user;
A process of creating viewing area transition information representing the transition of the area where the user is watching the content, in association with the attribute-specific viewing history for each attribute representing the user's gender and age group ;
a process of analyzing the viewing area transition information and estimating an element with a high appeal effect in the content;
a process of determining the facial expression of the user while viewing the content from the face image of the user;
a process of creating facial expression transition information representing the transition of the user's facial expression;
a process of analyzing the facial expression of the user when the line of sight is focused on the element with the high appeal effect, based on the facial expression transition information;
program to run.

Images