JP2023104686A - Information processing method, computer program, information processing device and information processing system - Google Patents

Abstract

To provide an information processing method, a computer program, an information processing apparatus, and an information processing system capable of authenticating a group including a plurality of users.SOLUTION: In the information processing method according to an embodiment of the present invention, an information processing apparatus associates a group to which a plurality of users belong and characteristic information indicating characteristics of users belonging to the group, and stores the information in a database, acquires a photographed image of the user, and, based on the acquired photographed image and the characteristic information stored in the database, identifies a group to which the user shown in the photographed image belongs. The information processing apparatus can identify the group to which the user belongs when at least one user among the plurality of users belonging to the group is shown in the acquired photographed image.SELECTED DRAWING: Figure 1

Description

TECHNICAL FIELD The present invention relates to an information processing method, a computer program, an information processing apparatus, and an information processing system for realizing authentication of a customer who uses a service or the like.

2. Description of the Related Art Conventionally, face authentication technology for authenticating a user based on a photographed image of the user's face or the like has been widely used. In recent years, face authentication using a learning model generated by machine learning, so-called AI (Artificial Intelligence), has been performed to improve authentication accuracy.

In Patent Document 1, a visitor's face image is captured by an imaging device of a locking system installed at the entrance of an occupied space, and information about the captured face image is compared with recorded information about the visitor's face image. Then, a face authentication system is described that compares the date and time when a face image is captured by an imaging device with visit date and time information that has been registered in advance, and authenticates the visitor based on the results of both comparisons.

JP 2021-140795 A

Conventional face authentication technology authenticates an individual user. The face authentication system described in Patent Literature 1 is also a system that authenticates an individual visitor.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide an information processing method, a computer program, an information processing apparatus, and an information processing method capable of authenticating a group including a plurality of users. It is to provide a system.

In an information processing method according to an embodiment, an information processing apparatus stores in a database a group to which a plurality of users belong and characteristic information indicating characteristics of users belonging to the group in association with each other, and photographs the user. A photographed image is acquired, and a group to which the user shown in the photographed image belongs is specified based on the acquired photographed image and the characteristic information stored in the database.

According to one embodiment, it can be expected to authenticate a group containing multiple users.

1 is a schematic diagram for explaining an overview of an information processing system according to an embodiment; FIG. 1 is a block diagram showing the configuration of a server device according to an embodiment; FIG. It is a schematic diagram which shows an example of group DB. 2 is a block diagram showing the configuration of a terminal device according to this embodiment; FIG. It is a schematic diagram which shows an example of the authentication screen which a terminal device displays. FIG. 4 is a schematic diagram showing an example of a group name confirmation screen displayed by a terminal device; 6 is a flow chart showing the procedure of group authentication processing performed by the server device according to the present embodiment; 6 is a flow chart showing the procedure of group authentication processing performed by the terminal device according to the present embodiment; 6 is a flow chart showing a procedure of group registration processing performed by the server device according to the present embodiment; 7 is a flow chart showing a procedure of user addition processing by the server device according to the present embodiment.

A specific example of an information processing system according to an embodiment of the present invention will be described below with reference to the drawings. The present invention is not limited to these examples, but is indicated by the scope of the claims, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

<System overview>
FIG. 1 is a schematic diagram for explaining an outline of an information processing system according to this embodiment. An information processing system according to the present embodiment is a system that performs processing for authenticating a group to which a plurality of users belong. In this embodiment, a group is a gathering of people such as family members or friends. However, the group handled by the information processing system according to the present embodiment may include a group to which only one user belongs.

The illustrated example is a case where the information processing system according to the present embodiment is applied to a store (facility) such as a restaurant. are grouped. The information processing system according to the present embodiment is not limited to stores such as restaurants, and may be used in various facilities such as theme parks, shopping centers, and movie theaters where users can visit in groups. . The information processing system according to the present embodiment includes a server device 1, a terminal device 3 used by a user, and a terminal device 5 used by a store clerk (employee). In a store such as a restaurant, for example, one terminal device 3 is installed at one table used by one group, and the user can use the terminal device 3 to browse menus and order products. . The order details input by the user using the terminal device 3 are transmitted to the server device 1 through communication via the network, and then transmitted from the server device 1 to the terminal device 5 of the store clerk and displayed. The store clerk can provide the product to the user according to the order content displayed on the terminal device 5 .

In the information processing system according to the present embodiment, a group of users is treated as a group as described above, and for example, points are given to a group visiting a store, coupons are distributed, or payment is discounted. We provide services that provide incentives such as For this reason, in the information processing system according to the present embodiment, the faces of users belonging to a group who have visited a store are photographed by the terminal device 3, the photographed image is obtained by the server device 1, and is copied into the photographed image. A process for authenticating that the group to which the received user belongs is a pre-registered group. Incentives given to a group are available to all users belonging to this group.

In the information processing system according to the present embodiment, for example, a user who visits a store operates the terminal device 3 installed on the table to perform procedures for starting to use the service. At this time, the terminal device 3 prompts the user to take a picture of the face by displaying a message or the like requesting the user to take a picture of the face. The user uses the camera provided in the terminal device 3 to photograph the face of himself or another user in the group. The terminal device 3 transmits the captured image of the user's face to the server device 1 .

Based on the face image of the user received from the terminal device 3, the server device 1 performs processing for authenticating the group to which the user belongs. In the information processing system according to the present embodiment, group IDs and facial features of users belonging to the group are associated with each other and stored in the database. The server device 1 calculates the feature amount of the user's face from the image received from the terminal device 3, and searches the database for a feature corresponding to the calculated feature amount to identify the group to which the user belongs. Based on this identification result, the server apparatus 1 authenticates that the group of users who visited the store is the group stored in the database. When the server device 1 succeeds in group authentication, the server device 1 notifies the terminal device 3 and the terminal device 5 to that effect, and may give incentives such as store visit points or service coupons to the authenticated group.

In the information processing system according to this embodiment, one user belongs to one group and does not belong to a plurality of groups. However, the information processing system may allow one user to belong to multiple groups. In this case, the server device 1 may, for example, present as candidates a plurality of groups to which the user depicted in the image received from the terminal device 3 belongs, and may accept group selection from the user. Further, for example, when the server device 1 cannot specify a group by one user, the server device 1 requests the photographing of the faces of the second and subsequent users, and in response to this request, a plurality of users are photographed based on the images acquired from the terminal device 3. You may specify one group to which you belong.

Further, in the information processing system according to the present embodiment, if it is confirmed that at least one user has visited the store, the group to which this user belongs is authenticated, and this group is treated as having visited the store. . However, the information processing system may verify the visit of two or more users to authenticate the group. In addition, in the information processing system according to the present embodiment, there is no upper limit to the number of users belonging to one group. An upper limit may be set for

Further, the server device 1 searches the database based on the user's face image received from the terminal device 3, the feature amount of the user's face is not stored in the database, and the group to which the user belongs does not exist. In this case, a process of newly registering a group to which this user belongs is performed. The server device 1 also performs a process of adding and registering a new user (face feature amount) to a group already stored in the database. The server device 1 can also perform processing for deleting a predetermined user from a group registered in the database.

When the server device 1 notifies the terminal device 5 used by the store clerk that the group has been authenticated, the terminal device 5 displays information about the authenticated group, that is, the group that has visited the store. The terminal device 5 may display, for example, facial images of users belonging to an authenticated group, and may display information such as the history of products ordered by this group in the past. Information such as troubles that occurred in the past may be displayed, or any information other than these may be displayed.

<Device configuration>
FIG. 2 is a block diagram showing the configuration of the server device 1 according to this embodiment. The server device 1 according to the present embodiment includes a processing unit 11, a storage unit (storage) 12, a communication unit (transceiver) 13, and the like. In this embodiment, the explanation is given assuming that the processing is performed by one server device, but the processing may be performed by a plurality of server devices in a distributed manner.

The processing unit 11 includes an arithmetic processing unit such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit) or a quantum processor, a ROM (Read Only Memory) and a RAM (Random Access Memory). It is configured using The processing unit 11 reads out and executes the server program 12a stored in the storage unit 12, thereby performing processing for authenticating the group to which the user shown in the photographed image acquired from the terminal device 3 belongs, and processing for authenticating the group to the database. , and various other processing such as registration processing.

The storage unit 12 is configured using, for example, a large-capacity storage device such as a hard disk. The storage unit 12 stores various programs executed by the processing unit 11 and various data required for processing by the processing unit 11 . In the present embodiment, the storage unit 12 stores a server program 12a executed by the processing unit 11. FIG. The storage unit 12 also includes a group DB (database) 12b that stores various information used for group authentication, and a learning model storage unit 12c that stores information on machine-learned learning models used for group authentication processing and the like. is provided.

In this embodiment, the server program (computer program, program product) 12a is provided in a form recorded in a recording medium 99 such as a memory card or an optical disc, and the server device 1 reads the server program 12a from the recording medium 99 and stores it. Store in section 12 . However, the server program 12a may be written in the storage unit 12 during the manufacturing stage of the server device 1, for example. Further, for example, the server program 12a may be delivered by another remote server device or the like, and the server device 1 may acquire the program through communication. For example, the server program 12 a may be recorded in the recording medium 99 and read by a writing device and written in the storage unit 12 of the server device 1 . The server program 12 a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 99 .

FIG. 3 is a schematic diagram showing an example of the group DB 12b. The group DB 12b according to the present embodiment is a database in which, for example, "group ID", "feature information", "incentive information", "history information", etc. are stored in association with each other. “Group ID” is identification information assigned to a group to which one or more users belong. The "group ID" is automatically determined by the server device 1 when the group is registered, or determined by user input when the group is registered.

The “feature information” is information relating to a feature amount calculated from an image of the face of one or more users belonging to the group. In this embodiment, the feature amount of each user is, for example, tens to hundreds of dimensions of vector information. In the information processing system according to the present embodiment, a feature amount is calculated from an image of the user's face using a learning model that has undergone machine learning in advance so as to output the feature amount of the image in response to the input of the image. do. For example, when registering a group, the server device 1 acquires a photographed image of a user from the terminal device 3, inputs the acquired image to a learning model to calculate a feature amount, and uses the calculated feature amount as the user's image. is stored in the group DB 12b in association with the group to which it belongs.

Further, in the information processing system according to the present embodiment, one of at least two types of feature amounts for one user is calculated and stored in the group DB 12b. The first feature amount is a feature amount that indicates the features of the user's entire face, and the second feature amount is a feature amount that indicates the features of a portion of the user's face. Specifically, the second feature amount is a feature amount indicating the features of the user's entire face excluding the mouth and nose and their surroundings (that is, the user's entire face is covered with a mask). This is a feature quantity that indicates the features of the part excluding the part that is In other words, the second feature amount is a feature amount that includes the features of the user's eyes and does not include the features of the user's mouth.

"Incentive information" is information about incentives such as points or coupons given to a group, for example. In the illustrated example, coupons such as "5% discount" and "30% discount" are given to groups. "History information" is information such as the history of use of the store by the group, such as the date and time when the store was previously used, or the products ordered when the store was previously used. In the illustrated example, the number of times the group has used the store is stored as "history information", such as "2nd visit" and "10th day of visit".

Note that the configuration of the group DB 12b shown in FIG. 3 is an example, and is not limited to this. For example, the group DB 12b may not include "incentive information" and "history information". Further, for example, in the "feature information" of the group DB 12b, instead of or together with the feature amount, a photographed image of the user's face taken at the time of registration of the group may be stored. When the captured image is stored in the group DB 12b and the feature amount is not stored, the server device 1 can calculate the user's feature amount as necessary based on the stored captured image. Also, for example, the group DB 12b may store attribute information such as gender or age of users belonging to the group.

In the information processing system according to the present embodiment, the group DB 12b stores a "group ID" as identification information for identifying a group. Identification information that identifies each user, such as an address, is not stored. The information processing system according to the present embodiment can be expected to avoid the risk of leakage of personal information by not storing such information that can identify an individual user in the database. However, identification information such as user names or user IDs may be stored in the group DB 12b.

The learning model storage unit 12c of the storage unit 12 stores information about a learned learning model that the server device 1 uses for group authentication processing. The information about the learning model may include, for example, configuration information indicating how the learning model is configured, and information such as internal parameter values of the learning model determined by machine learning processing. In the present embodiment, the server device 1 does not perform machine learning processing for generating a learning model, acquires a learned learning model generated by another device, and stores the acquired learning model in the learning model storage unit 12c. to use. However, the server device 1 may perform machine learning processing to generate a learning model.

The learning model used by the server device 1 in the present embodiment includes, for example, a mask estimation model for estimating whether or not a person photographed in an image is wearing a mask, and a feature amount of the entire face of a person photographed in an image. and a learning model such as a second feature value calculation model for calculating a feature value of a specific part of a person captured in an image. The mask estimation model is a classification model that accepts an image of a person's face as an input and classifies whether or not the person in the image is wearing a mask. The mask estimation model is a learning model having a configuration such as DNN (Deep Neural Network) or CNN (Convolutional Neural Network). For example, the mask estimation model is learned by attaching a flag to the image of a person wearing a mask to indicate that they are wearing a mask, and attaching a flag to the image of a person not wearing a mask to indicate that they are not wearing a mask. It can be generated by performing so-called supervised machine learning using data for training (teaching data).

The first feature value calculation model is a learning model that receives, for example, an image of a person's face as an input and outputs a vector of feature values representing the features of the entire face. For example, a learning model of an encoder/decoder having an encoder that converts an input image into a feature amount vector and a decoder that converts the feature amount vector into the original image is generated by machine learning, and the encoder of this learning model is the first It can be used as a feature amount calculation model. For example, the first feature value calculation model can be generated by performing so-called unsupervised machine learning using a plurality of images of human faces as learning data.

The second feature amount calculation model is a learning model that receives, for example, an image of a person's face as an input and outputs a feature amount vector that indicates the features of a specific portion of the face. In the present embodiment, the second feature amount calculation model outputs a vector of feature amounts indicating the features of the specific portion, with the eye and the peripheral portion of the human face as the specific portion. However, the specific part is not limited to this, and various parts of a person's face may be adopted as the specific part. The second feature value calculation model includes, for example, a learning model that extracts an image region of eyes and their surroundings from an input photographed image of a person's face, and a learning model that outputs a vector of feature values for the extracted image region. can be configured using An existing learning model for object detection or segmentation, for example, can be used as the learning model for extracting image regions. A learning model that outputs a vector of feature values of an image region can be generated by performing machine learning using an image of a specific portion of a person's face in the same manner as the first feature value calculation model.

The communication unit 13 of the server device 1 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN (Local Area Network), the Internet, and the like. In this embodiment, the communication unit 13 communicates with one or more terminal devices 3 and 5 via the network N. FIG. The communication unit 13 transmits the data given from the processing unit 11 to other devices, and gives the data received from the other devices to the processing unit 11 .

Note that the storage unit 12 may be an external storage device connected to the server device 1 . The server device 1 may be a multicomputer including a plurality of computers, or may be a virtual machine virtually constructed by software. The server device 1 is not limited to the above configuration, and may include, for example, a reading unit that reads information stored in a portable storage medium, an input unit that receives operation inputs, or a display unit that displays images. .

Further, in the server device 1 according to the present embodiment, the processing unit 11 reads out and executes the server program 12a stored in the storage unit 12, so that the captured image acquisition unit 11a, the feature amount calculation unit 11b, the group authentication unit 11c, the registration processing unit 11d, the incentive giving unit 11e, the information transmission unit 11f, and the like are implemented in the processing unit 11 as software functional units. In the figure, functional units related to processing for authenticating a group to which a user belongs are illustrated as functional units of the processing unit 11, and functional units related to other processing are omitted.

The captured image acquisition unit 11 a performs processing for acquiring an image of the user's face captured by the terminal device 3 . In the information processing system according to the present embodiment, a user who visits a store uses the terminal device 3 to photograph his/her own face, and the terminal device 3 transmits the photographed image of the user's face to the server device 1. do. The photographed image acquisition unit 11a of the server device 1 acquires the photographed image of the user's face transmitted from the terminal device 3 at this time by receiving it through the communication unit 13, and temporarily stores the acquired photographed image in the storage unit 12. memorize. Further, the captured image acquisition unit 11a may request the terminal device 3 to re-capture the captured image when the acquired captured image is not suitable for authentication, for example, the image is not clear.

Based on the photographed image of the user's face acquired by the photographed image acquisition unit 11a, the feature amount calculation unit 11b uses one or a plurality of learning models stored in the learning model storage unit 12c to determine the characteristics of the photographed image. A process of calculating a first feature amount or a second feature amount relating to the user's face is performed. The feature amount calculation unit 11b determines whether or not the user in the captured image is wearing a mask, and if the user is not wearing a mask, calculates a first feature amount indicating the features of the entire face. . On the other hand, when the user is wearing a mask, the feature quantity calculation unit 11b calculates a second feature quantity indicating the features of the specific portions of the user's eyes and their surroundings. Note that the feature amount calculator 11b may calculate both the first feature amount and the second feature amount when the user does not wear a mask.

The group authentication unit 11c performs processing for authenticating the group to which the user shown in the captured image belongs, based on the feature amount calculated by the feature amount calculation unit 11b. The group authentication unit 11c calculates the degree of similarity between the feature amount (first feature amount or second feature amount) calculated by the feature amount calculation unit 11b and the feature amount registered in the group DB 12b. For example, cosine similarity can be used as similarity, but other similarities may be used. The group authentication unit 11c searches the group DB 12b for a feature amount that has the highest degree of similarity with the feature amount calculated by the feature amount calculation unit 11b and that exceeds a predetermined threshold, and selects a group in which this feature amount is registered. The group to which the user belongs is specified, and this group is authenticated as a valid group that uses the store. Further, if the group DB 12b does not register a feature amount whose degree of similarity with the feature amount calculated by the feature amount calculation unit 11b exceeds a predetermined threshold, the group authentication unit 11c determines that the group to which is belongs is unregistered, and causes the terminal device 3 to perform processing such as registration of a new group.

The registration processing unit 11d performs a process of registering information about a new group in the group DB 12b, a process of adding a new user to an existing group, and the like. The registration processing unit 11d adds a new group ID to the group DB 12b, and adds a first feature amount and a second feature amount calculated from the user's face image captured by the terminal device 3 to this group ID. are stored in association with each other to register a new group. Further, when the terminal device 3 requests additional registration of the user, the registration processing unit 11d registers the first feature amount and the second feature amount of the user shown in the captured image acquired from the terminal device 3, It is added and stored in association with the registered group ID. In the present embodiment, when registering a new group and when additionally registering a user, the user is requested to take a picture of the face without a mask, and the photographed face of the user is requested. Both the first feature amount and the second feature amount are calculated from the image and stored in the group DB 12b. Moreover, the registration processing unit 11d may perform processing for confirming that the feature amount of the user to be newly registered has not already been registered in the group DB 12b before registering these feature amounts.

The incentive giving unit 11e performs a process of giving incentives such as points or coupons to groups authenticated by the group authenticating unit 11c. For example, once a day, the incentive granting unit 11e grants, as an incentive, a coupon that applies a predetermined discount to the payment amount to a group authenticated by visiting the store, and adds the coupon to the incentive information in the group DB 12b. Information to that effect is stored. The incentive given by the incentive giving unit 11e is not limited to points or coupons, and may be of any kind.

The information transmission unit 11f performs a process of transmitting information about the group authenticated by the group authentication unit 11c to the terminal device 5 of the store. In the present embodiment, the information transmitting unit 11f reads information stored in the history information of the group DB 12b for the authenticated group, and transmits the read history information to the terminal device 5 of the store. As a result, the terminal device 5 can display history information such as which group the group of users who visited the store belongs to and the number of times this group has used this store before, and present it to the store clerk. .

FIG. 4 is a block diagram showing the configuration of the terminal device 3 according to this embodiment. The terminal device 3 according to the present embodiment includes a processing unit 31, a storage unit (storage) 32, a communication unit (transceiver) 33, a display unit (display) 34, an operation unit 35, a camera 36, and the like. . The terminal device 3 is, for example, a device used by a user such as a customer visiting a store, and may be configured using an information processing device such as a tablet terminal device or a personal computer. In the present embodiment, the terminal device 3 is assumed to be installed in a store, but the terminal device 3 is not limited to this, and may be a smartphone or a tablet terminal device possessed by the user, for example.

The processing unit 31 is configured using an arithmetic processing unit such as a CPU or MPU, a ROM, and the like. The processing unit 31 reads and executes the program 32a stored in the storage unit 32 to perform various processes such as a process of authenticating a group of users who have visited the store and a process of registering information about the group.

The storage unit 32 is configured using, for example, a nonvolatile memory device such as a flash memory or a storage device such as a hard disk. The storage unit 32 stores various programs executed by the processing unit 31 and various data required for processing by the processing unit 31 . In the present embodiment, the storage unit 32 stores a program 32a executed by the processing unit 31. FIG. In this embodiment, the program 32a is distributed by a remote server device or the like, and the terminal device 3 acquires it through communication and stores it in the storage unit 32. FIG. However, the program 32a may be written in the storage unit 32 during the manufacturing stage of the terminal device 3, for example. For example, the program 32a may be stored in the storage unit 32 after the terminal device 3 reads the program 32a recorded in the recording medium 99 such as a memory card or an optical disc. For example, the program 32 a may be recorded in the recording medium 99 and read by a writing device and written in the storage unit 32 of the terminal device 3 . The program 32 a may be provided in the form of distribution via a network, or may be provided in the form of being recorded on the recording medium 99 .

The communication unit 33 communicates with various devices via a network N including a mobile phone communication network, a wireless LAN, the Internet, and the like. In the present embodiment, the communication unit 33 communicates with the server device 1 via the network N. FIG. The communication unit 33 transmits data received from the processing unit 31 to other devices, and provides the processing unit 31 with data received from other devices.

The display unit 34 is configured using a liquid crystal display or the like, and displays various images, characters, etc. based on the processing of the processing unit 31 . The operation unit 35 receives a user's operation and notifies the processing unit 31 of the received operation. For example, the operation unit 35 receives a user's operation using an input device such as mechanical buttons or a touch panel provided on the surface of the display unit 34 . Further, for example, the operation unit 35 may be an input device such as a mouse and a keyboard, and these input devices may be detachable from the terminal device 3 .

The camera 36 is configured using an imaging device such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The camera 36 is provided on the housing of the terminal device 3 or the like, and provides the processing section 31 with an image captured by the imaging element.

Further, in the terminal device 3 according to the present embodiment, the processing unit 31 reads out and executes the program 32a stored in the storage unit 32, so that the authentication processing unit 31a, the registration processing unit 31b, the display processing unit 31c, etc. are software. is implemented in the processing unit 31 as a functional unit. The program 32a may be a program dedicated to the information processing system according to the present embodiment, or may be a general-purpose program such as an Internet browser or web browser.

The authentication processing unit 31a performs processing for authenticating the group to which the user who visited the store belongs. For example, when the user starts using the terminal device 3, the authentication processing unit 31a requests the user to photograph the face. In response to this request, the user uses the camera 36 provided in the terminal device 3 to photograph his own face or the faces of other users in the group. The authentication processing unit 31a acquires an image of the user's face captured by the camera 36, transmits the acquired captured image to the server device 1, and requests group authentication. The authentication processing unit 31a receives information about whether or not group authentication by the server device 1 is permitted, and if the authentication is successful, notifies the user to that effect, and if the authentication is unsuccessful, registers a new group. Require the user.

The registration processing unit 31b performs processing for registering information about a new group, processing for adding a new user to an existing group, and the like. The registration processing unit 31b performs new registration of a group when, for example, the user selects an item for new registration of a group from a screen such as a menu, or when a registered group cannot be specified in authentication processing. . The registration processing unit 31b displays a new group registration screen to request the user to photograph his/her face, transmits an image of the user's face photographed by the camera 36 to the server apparatus 1, and requests new registration of the group. . At this time, the registration processing unit 31b may receive an input of the group ID from the user and transmit it to the server device 1 together with the captured image. Note that the group ID may be created, for example, by the server device 1 generating an appropriate character string. In this case, after registering a new group, the server device 1 transmits the group ID to the terminal device 3, Device 3 notifies the user of the group ID.

Further, the registration processing unit 31b performs additional user registration for this group when, for example, the user selects an item for additional user registration from a screen such as a menu after authentication of the group or registration of a new group. The registration processing unit 31b requests the photographing of the face of the user to be added, transmits the image of the user's face photographed by the camera 36, and the group ID of the group to which the user is to be added to the server device 1, and the user is registered. request additional registration of

Further, the registration processing unit 31b receives information about the success or failure of the new registration of the group or the additional registration of the user from the server device 1, and displays the information about the success or failure on the display unit . For example, when the captured image is not clear, the server device 1 notifies the terminal device 3 of the registration failure. In response to this notification, the registration processing unit 31b of the terminal device 3 may request the user to re-photograph the face, re-transmit the re-photographed image to the server device 1, and perform registration processing.

The display processing unit 31c performs processing for displaying various messages and various images for the user on the display unit 34 during authentication processing performed by the authentication processing unit 31a and registration processing performed by the registration processing unit 31b. For example, when the user starts using the terminal device 3, the display processing unit 31c displays on the display unit 34 a message requesting that the user take a picture of the face for group authentication. The image of the face is displayed on the display section 34 .

In the information processing system according to the present embodiment, the terminal device 5 used by the store clerk or the like is configured using an information processing device such as a tablet terminal device or a personal computer, and the terminal device used by the user 3, the illustration of the block diagram is omitted. In the present embodiment, the terminal device 5 receives the information about the user group transmitted from the server device 1 and displays the received information on the display unit, thereby performing processing for presenting the information to the store clerk.

<Group authentication process>
In the information processing system according to the present embodiment, a plurality of tables provided in a store such as a restaurant, for example, are each provided with a terminal device 3, and a user uses the terminal device 3 to display menus, accept orders, and the like. do. When a group including one or more users visits a store such as a restaurant and a user belonging to this group starts using the terminal device 3 provided on the table for order etc., the terminal device 3 is sent to the user. requests authentication processing. After displaying, for example, a message prompting the authentication process on the display unit 34 , the terminal device 3 displays an authentication screen for the authentication process on the display unit 34 .

FIG. 5 is a schematic diagram showing an example of an authentication screen displayed by the terminal device 3. As shown in FIG. The terminal device 3 displays a title string of "authentication screen" at the top of the authentication screen, and provides two areas for displaying images below the title string in a vertically arranged manner. The terminal device 3 displays a real-time image (moving image) captured by the camera 36 in a display area provided above the authentication screen. The terminal device 3 also provides a button labeled "shoot" below this display area, and accepts a user's face shooting operation by a touch operation or a tap operation on this "shoot" button.

In addition, the terminal device 3 displays a captured image (still image) captured in response to the operation of the "capture" button in a display area provided below the authentication screen. That is, the terminal device 3 displays a real-time moving image captured by the camera 36 in the upper display area of the authentication screen, and obtains a still image of one frame from this moving image when the "capture" button is operated. and display it in the lower display area. A button labeled “Authenticate” is provided below the display area of the captured image. It should be noted that the terminal device 3 displays the "authentication" button as disabled (in a state in which the operation is not accepted) immediately after the start of displaying the authentication screen, and after the photographing is performed by the operation of the "shooting" button, the "authentication" button is displayed. Enable (state to accept operations).

When the authentication screen is displayed on the display unit 34 of the terminal device 3, the user directs the camera 36 of the terminal device 3 to his or another user's face to view a real-time image displayed in the upper display area of the authentication screen. Check the moving image. The user operates the "shoot" button while the face is in the upper display area, and takes an image of the face for authentication processing. The captured image is displayed in the lower display area of the authentication screen, and the user checks the image displayed in the lower display area and operates the "Authenticate" button to perform the authentication process. can.

The terminal device 3 transmits the captured image of the user's face captured on the authentication screen to the server device 1 and requests group authentication. Upon receiving this request, the server device 1 acquires the photographed image of the user's face from the terminal device 3 and determines whether or not the user shown in the photographed image is wearing a mask. At this time, the server device 1 inputs the photographed image to the learning model for classifying whether or not there is a mask stored in the learning model storage unit 12c, and acquires the information output by the learning model, so that the It can be determined whether the user is wearing a mask.

When the user does not wear a mask, the server apparatus 1 calculates a first feature quantity indicating features of the user's entire face from the captured image. The server device 1 calculates similarities between the calculated first feature amount and the plurality of first feature amounts registered in the group DB 12b. The server device 1 searches the group DB 12b for the registered first feature amount having the highest similarity to the first feature amount calculated from the captured image, and determines whether the similarity exceeds a preset threshold. determine whether or not When the highest degree of similarity exceeds the threshold, the server apparatus 1 acquires the group ID of the group in which this first feature amount is registered from the group DB 12b, and the user shown in the captured image belongs to the group of this group ID. identify that the user belongs to On the other hand, if the highest degree of similarity does not exceed the threshold, the server apparatus 1 determines that the group to which the user depicted in the captured image belongs is not registered.

When the user wears a mask, the server apparatus 1 calculates a second feature amount representing features of the user's eyes and their surroundings from the captured image. The server device 1 calculates similarities between the calculated second feature amount and the plurality of second feature amounts registered in the group DB 12b. The server device 1 searches the group DB 12b for the registered second feature amount having the highest similarity to the second feature amount calculated from the captured image, and determines whether the similarity exceeds a preset threshold. determine whether or not When the highest degree of similarity exceeds the threshold, the server device 1 acquires the group ID of the group in which the second feature is registered from the group DB 12b, and the user pictured in the captured image belongs to the group of this group ID. identify that the user belongs to On the other hand, if the highest degree of similarity does not exceed the threshold, the server apparatus 1 determines that the group to which the user depicted in the captured image belongs is not registered.

When the server device 1 can identify the group to which the user shown in the photographed image belongs, the server device 1 notifies the terminal device 3 that the authentication of the group has been successful, and provides points, coupons, or the like to this group. provide incentives for At this time, the server device 1 reads, for example, the group ID of the authenticated group, incentive information, history information, etc. from the group DB 12b and transmits them to the terminal device 3, and the terminal device 3 receives this information and displays it on the display unit 34. do. On the other hand, if the server device 1 determines that the group to which the user shown in the captured image belongs is not registered, the server device 1 notifies the terminal device 3 that the authentication of the group has failed.

FIG. 6 is a schematic diagram showing an example of a group name confirmation screen displayed by the terminal device 3. As shown in FIG. The illustrated group name confirmation screen is displayed on the display unit 34 by the terminal device 3 that has received a notification from the server device 1 that the group has been successfully authenticated. The terminal device 3 displays the title character string "group name confirmation" at the top of the group name confirmation screen, and displays the group ID (group name) of the authenticated group below it. In the illustrated example, it is assumed that the group ID of the authenticated group is "3 good friends ○○○ group", and the terminal device 3 displays the character string "3 good friends ○○○ group". .

Also, the terminal device 3 displays a message based on the information transmitted from the server device 1 below the group ID. In the illustrated example, the terminal device 3 displays the messages "Thank you for coming to our store for the second time." This is a message based on the history information and incentive information stored in the group DB 12b. The message shown in FIG. 5 displayed by the terminal device 3 is merely an example, and the terminal device 3 may display various messages based on information obtained from the server device 1 .

The terminal device 3 also displays a button labeled "Confirm" below these messages. When the user's operation on the "confirm" button is received, the terminal device 3 hides the group plane confirmation screen and displays another screen. For example, if the store is a restaurant, the terminal device 3 can display a screen for accepting orders from users.

FIG. 7 is a flow chart showing the procedure of group authentication processing performed by the server device 1 according to the present embodiment. The photographed image acquisition unit 11a of the processing unit 11 of the server device 1 according to the present embodiment acquires the photographed image by receiving the photographed image of the user's face transmitted from the terminal device 3 at the communication unit 13. (Step S1). The feature amount calculation unit 11b of the processing unit 11 uses the mask estimation model stored in the learning model storage unit 12c of the storage unit 12, for example, to determine whether the user in the captured image acquired in step S1 is wearing a mask. (step S2).

If the user in the captured image is not wearing a mask (S2: NO), the feature amount calculation unit 11b uses the first feature amount calculation model stored in the learning model storage unit 12c of the storage unit 12, for example. Then, a first feature amount relating to the entire face of the user captured in the image is calculated (step S3). The group authentication unit 11c of the processing unit 11 calculates the degree of similarity between the first feature amount calculated in step S3 and the plurality of first feature amounts stored in the group DB 12b (step S4). The process proceeds to S7.

If the user in the captured image is wearing a mask (S2: YES), the feature amount calculation unit 11b uses the second feature amount calculation model stored in the learning model storage unit 12c of the storage unit 12, for example. Then, a second feature amount relating to the specific portion of the user captured in the image is calculated (step S5). The group authentication unit 11c calculates the degree of similarity between the second feature amount calculated in step S5 and the plurality of second feature amounts stored in the group DB 12b (step S6), and proceeds to step S7. proceed.

The group authentication unit 11c specifies the feature quantity with the highest similarity among the plurality of feature quantities stored in the group DB 12b based on the similarity calculated in step S4 or S6 (step S7). The group authentication unit 11c determines whether or not the similarity calculated in step S4 or S6 is equal to or greater than a threshold for the feature amount specified in step S7 (step S8). If the similarity is equal to or greater than the threshold (S8: YES), the group authentication unit 11c identifies the group associated with the feature amount identified in step S7 as the group to which the user belongs (step S9). is a valid registered group. The incentive giving unit 11e of the processing unit 11 gives incentives such as points or coupons to the group identified in step S9 (step S10), and ends the process.

If the similarity is not equal to or greater than the threshold, that is, if the similarity is less than the threshold (S8: NO), the group authentication unit 11c determines that the feature amount of the user shown in the captured image is not registered in the group DB 12b, and that the user determines that the group to which is belongs is not registered, notifies the terminal device 3 of failure of group authentication (step S11), and terminates the process.

FIG. 8 is a flowchart showing the procedure of group authentication processing performed by the terminal device 3 according to this embodiment. The display processing unit 31c of the processing unit 31 of the terminal device 3 according to the present embodiment displays, for example, the authentication screen shown in FIG. 5 on the display unit 34 (step S21). The authentication processing unit 31a of the processing unit 31 determines whether or not the "shooting" button provided on the authentication screen has been operated (step S22). If the "capture" button has been operated (S22: YES), the authentication processing unit 31a acquires an image (still image) captured by the camera 36 (step S23). The authentication processing unit 31a activates the "authenticate" button provided on the authentication screen (step S24), and returns the process to step S21.

If the "shooting" button has not been operated (S22: NO), the authentication processing unit 31a determines whether or not the "authentication" button provided on the authentication screen has been operated (step S25). If the "authenticate" button has not been operated (S25: NO), the authentication processing unit 31a returns the process to step S21. If the "authenticate" button has been operated (S25: YES), the authentication processing unit 31a transmits the captured image acquired in step S23 to the server device 1 (step S26), and requests group authentication.

The authentication processing unit 31a receives the information about the result of the group authentication transmitted from the server device 1, and determines whether or not the group authentication has succeeded (step S27). If the group authentication has not succeeded (S27: NO), the authentication processing unit 31a returns the process to step S21. If the authentication is successful (S27: YES), the display processing unit 31c displays the group name confirmation screen shown in FIG. 6 on the display unit 34 (step S28). The display processing unit 31c determines whether or not the "confirm" button on the group name confirmation screen has been operated (step S29). If the "confirm" button has not been operated (S29: NO), the display processing unit 31c returns the process to step S28 to continue displaying the group name confirmation screen. If the "confirm" button has been operated (S29: YES), the display processing unit 31c ends the process.

<Group registration process>
In the information processing system according to this embodiment, for example, when a group including one or more users visits a store for the first time, it is necessary to register the group. For example, a group user can register a group by operating the terminal device 3 provided with the table and selecting a group registration item on a menu screen or the like. The terminal device 3 displays a group registration screen on the display unit 34 when the group registration item is selected.

Although illustration is omitted, the group registration screen displayed by the terminal device 3 may have, for example, the same configuration as the authentication screen shown in FIG. "Register Group" is displayed, and instead of the "Authenticate" button, a "Register" button labeled "Register" is provided. On the group registration screen, the user operates the "photograph" button to take an image of the face, and operates the "registration" button to register the group. When the "Register" button is operated, the terminal device 3 transmits the captured image to the server device 1 and requests registration of the group. Note that the terminal device 3 may provide a text box or the like for entering a group name on the group registration screen to receive an input of the group ID (group name) from the user, and transmit the group ID to the server device 1 together with the captured image. In the information processing system according to the present embodiment, the server device 1 appropriately creates the group ID.

The server device 1 receives a group registration request from the terminal device 3 together with the photographed image, and registers a new group in the group DB 12b. The server device 1 determines whether or not the user shown in the captured image is wearing a mask, and if the user is wearing a mask, notifies the terminal device 3 of the fact and re-captures the image. ask for In response to the re-photographing request, the terminal device 3 displays a message requesting re-photographing without wearing a mask, for example, displays a group registration screen, and re-photographs the user's face.

If the user in the captured image is not wearing a mask, the server apparatus 1 calculates both the first feature amount and the second feature amount using the learning model based on the captured image. The server device 1 appropriately creates a group ID for a new group, associates the created group ID with the first feature amount and the second feature amount calculated from the captured image, and stores them in the group DB 12b. , to register a new group. At this time, the server device 1 may provide an incentive for group registration, for example. The server device 1 transmits the created group ID to the terminal device 3 and notifies the group registration completion. The server device 1 may also notify the terminal device 5 of the store that a new group has been added.

The terminal device 3 that has received the group registration completion notification displays on the display unit 34 a message notifying of the group registration completion together with the group ID created by the server device 1 . At this time, the terminal device 3 may receive a change of the group ID from the user. The group ID to be changed may be, for example, a user ID obtained by accepting a character string input from the user, or may be re-presented to the user after the server device 1 appropriately recreates the group ID. Further, for example, the server device 1 may create a plurality of group IDs as candidates before registering the group, present them to the user, and adopt the group ID selected by the user from among the plurality of candidates.

After the registration of the group is completed, the terminal device 3 accepts additional registration of the second and subsequent users to the group. For example, the terminal device 3 provides an “additional registration” button for additionally registering a user to the group on a screen for notifying the group registration completion, and when the operation of the “additional registration” button is accepted, the additional registration is performed. A registration screen is displayed on the display unit 34 to accept additional registration of the user. Note that the terminal device 3 may accept additional user registration in the same way after the group has been authenticated in the group authentication process.

The additional registration screen displayed by the terminal device 3 may be a screen having the same configuration as the authentication screen and the group registration screen. For example, the additional registration screen may have a configuration in which the title character string is set to "additional registration" and an "additional registration" button is provided in place of the "authentication" button with respect to the authentication screen shown in FIG. On the additional registration screen, the second and subsequent users operate the "shoot" button to take a face image, and operate the "additional registration" button to perform additional registration to the registered group. When the "additional registration" button is operated, the terminal device 3 transmits the captured image and the group ID to be added to the server device 1, and requests additional registration of the user.

The server device 1 receives from the terminal device 3 a request for additional registration in the group together with the photographed image and the group ID, calculates the first feature amount and the second feature amount from the photographed image, and adds them to the specified group ID. The first feature amount and the second feature amount calculated in association are stored in the group DB 12b. At this time, if the user in the captured image is wearing a mask, the server device 1 may notify the terminal device 3 of this fact and request re-capture of the image. The server device 1 that has additionally registered the user in the group notifies the terminal device 3 of the completion of the additional registration of the user. Upon receiving this notification, the terminal device 3 displays on the display unit 34 a message notifying that the additional registration of the user has been completed.

FIG. 9 is a flow chart showing the procedure of group registration processing performed by the server device 1 according to the present embodiment. The photographed image acquisition unit 11a of the processing unit 11 of the server device 1 according to the present embodiment acquires the photographed image by receiving the photographed image of the user's face transmitted from the terminal device 3 at the communication unit 13. (Step S41). The feature amount calculation unit 11b of the processing unit 11 uses, for example, the mask estimation model stored in the learning model storage unit 12c of the storage unit 12 to determine whether the user in the captured image acquired in step S41 is wearing a mask. (step S42).

If the user in the photographed image is wearing a mask (S42: YES), the registration processing unit 11d of the processing unit 11 informs the terminal device 3 that registration is not possible because the user is wearing a mask. (step S43), requests re-imaging without wearing a mask, and terminates the process.

If the user in the captured image is not wearing a mask (S42: NO), the feature amount calculation unit 11b uses the first feature amount calculation model stored in the learning model storage unit 12c of the storage unit 12, for example. Then, a first feature amount relating to the entire face of the user captured in the image is calculated (step S44). Further, the feature amount calculation unit 11b calculates a second feature amount related to a specific portion of the user captured in the image, for example, using the second feature amount calculation model stored in the learning model storage unit 12c of the storage unit 12. (Step S45).

The registration processing unit 11d generates a group ID by generating an appropriate character string (step S46). The registration processing unit 11d associates the group ID generated in step S46, the first feature amount calculated in step S44, and the second feature amount calculated in step S45, and stores them in the group DB 12b. (Step S47). The registration processing unit 11d notifies the terminal device 3 that the registration of the group has been completed together with the group ID generated in step S46 (step S48), and terminates the process.

FIG. 10 is a flow chart showing a procedure of user addition processing by the server device 1 according to the present embodiment. The photographed image acquisition unit 11a of the processing unit 11 of the server device 1 according to the present embodiment acquires the photographed image by receiving the photographed image of the user's face transmitted from the terminal device 3 at the communication unit 13. (Step S61). Also, the registration processing unit 11d of the processing unit 11 acquires the group ID transmitted from the terminal device 3 together with the captured image (step S62).

The feature amount calculation unit 11b of the processing unit 11 uses, for example, the mask estimation model stored in the learning model storage unit 12c of the storage unit 12 to determine whether the user in the captured image acquired in step S61 is wearing a mask. (step S63). If the user in the captured image is wearing a mask (S63: YES), the registration processing unit 11d notifies the terminal device 3 that registration cannot be performed because the user is wearing a mask (step S64), requesting re-imaging without wearing a mask, and terminating the process.

If the user in the captured image is not wearing a mask (S63: NO), the feature amount calculation unit 11b uses the first feature amount calculation model stored in the learning model storage unit 12c of the storage unit 12, for example. Then, a first feature amount relating to the entire face of the user captured in the image is calculated (step S65). Further, the feature amount calculation unit 11b calculates a second feature amount related to a specific portion of the user captured in the image, for example, using the second feature amount calculation model stored in the learning model storage unit 12c of the storage unit 12. (Step S66).

The registration processing unit 11d associates the group ID acquired in step S62 with the first feature amount calculated in step S65 and the second feature amount calculated in step S66, and stores the group DB 12b. (step S67). The registration processing unit 11d notifies the terminal device 3 that the additional registration of the user for the group has been completed (step S68), and terminates the process.

<Others>
(1) Group Authentication with Multiple Users In the information processing system described above, the server device 1 can perform group authentication by acquiring a photographed face image of one user in a group who has visited a store. can. For this reason, at least one user belonging to the group may photograph the face using the terminal device 3 . For example, for a group in which four users are registered, the information processing system according to the present embodiment does not require all four users to take photographs of their faces for group authentication. If it can be confirmed that at least one user is matched with the facial features registered, the group that has visited the store is authenticated as a registered group. It is not necessary for all four users registered as a group to visit the store, and the group is authenticated when at least one user visits the store. Also, the user may visit the store together with other users not included in the group (unregistered users or users included in another group).

However, the information processing system may be configured to request photographing of the faces of two or more users, and authenticate the group when matching of the feature amounts of the two or more users can be confirmed. For example, the terminal device 3 transmits captured images of faces of two users to the server device 1, and the server device 1 calculates feature amounts for the two users. The server device 1 calculates the similarity between the feature amount calculated for the first user and the feature amount registered in the group DB 12b, and the feature amount having the highest similarity and exceeding the threshold value is registered. Identify the group as the first candidate. Next, the server device 1 calculates the similarity between the feature amount of the other user registered in the first candidate group and the feature amount calculated for the second user, and when the similarity exceeds the threshold, This first candidate group is identified as the group to which the two users belong, and this group is authenticated as a registered valid group.

If the degree of similarity between the feature amount of another user registered in this group and the feature amount calculated for the second user does not exceed the threshold, the server device 1 determines that the first user is second most similar. A group is specified as a second candidate in which a feature amount having a high degree of similarity and exceeding a threshold value is registered. The server device 1 calculates the degree of similarity between the feature amount of another user registered in the second candidate group and the feature amount calculated for the second user. The second candidate group is specified as the group to which the two users belong, and this group is authenticated as a registered valid group. In this way, the server device 1 sequentially identifies group candidates and determines whether or not to permit authentication, and fails group authentication when there is no group that can be a candidate in the group DB 12b. In this case, authentication failure may occur, for example, when at least one of the two users is unregistered, or when the two users are registered in different groups.

In the conventional face authentication technology, a user and the user's characteristic information are associated with each other and registered in a database, and the user shown in the captured image is identified and authenticated. In this case, the degree of similarity between the registered feature information and the feature image obtained from the captured image is calculated, and if the degree of similarity exceeds a predetermined threshold, it is determined that the photographed person is the registered person. be done. If the threshold for comparison with similarity is set to a high value, the number of cases where face authentication cannot be performed increases, and if it is set to a low value, erroneous authentication increases.

The information processing system according to the present embodiment identifies and authenticates the group to which the user belongs, rather than identifying and authenticating the individual user based on the feature information of the user captured in the captured image. . By adopting a configuration in which a group is specified and authenticated based on the feature information of multiple users, it is possible to set a low threshold value for comparison with the similarity, and even if the threshold value is set to a low value, erroneous authentication will occur. can be expected to suppress the increase in

Note that the terminal device 3 may transmit a plurality of captured images captured for a plurality of users to the server device 1, for example, each time the images are captured, or may collectively transmit a plurality of images to the server device 1, for example. good. For example, the terminal device 3 prompts all users included in a group who have visited a store to take pictures of their faces, and when all the users' faces have been taken, the terminal device 3 can transmit a plurality of taken images to the server device 1 . can. However, the plurality of captured images transmitted in this case may include captured images of unregistered users or captured images of users belonging to different groups. In such a case, the server device 1 may identify and authenticate a plurality of groups based on, for example, a plurality of captured images, or identify and authenticate one group that includes the largest number of users, for example. However, the terminal device 3 may be notified of users who are not included in this group.

(2) Use of Number of Users for Group Authentication In the information processing system described above, the server device 1 can authenticate a group based on a photographed image of at least one user belonging to the group. However, when registering a group, it is preferable to photograph the faces of a plurality of users (preferably all users) belonging to the group and register the feature amounts. For example, if only the user who is the representative of the group registers the feature amount, and other users are using the information processing system according to the present embodiment without registering the feature amount, the representative is absent and the other users If you visit the store by yourself, group authentication may not be performed and you may not be able to receive incentives.

For multiple users belonging to a group, in order to encourage as many users as possible to register, the information processing system provides an incentive amount according to, for example, the number of registered users included in the group or the number of users participating in group authentication. may be increased or decreased. For example, if the feature amount of four users included in the group is registered in the group DB 12b, the information processing system can set the value of 1% when the face of one person is photographed when performing group authentication at a store. A discount incentive is given, and if two faces are photographed, a 2% discount incentive is given, and if three faces are photographed, a 3% discount incentive is given, and four faces are photographed. The incentive may be increased or decreased, such as giving an incentive of 4% discount when the image is taken. Note that this increase/decrease in the amount of incentive given is merely an example, and the present invention is not limited to this.

(3) Search order of group DB 12b In the above-described information processing system, the server device 1 calculates the user's feature amount from the photographed image acquired from the terminal device 3, and the most It is necessary to search for features with high similarity. In order to shorten this search time, the server device 1 may limit the search range based on, for example, user attribute information (gender, age, etc.).

For example, the server device 1 uses a learning model for estimating the gender of the user depicted in the captured image to estimate the gender of the user depicted in the captured image captured by the terminal device 3 at the time of group registration, The gender estimation result is stored in association with the feature amount calculated from the captured image. At the time of group authentication, the server device 1 estimates the sex of the user from the captured image captured by the terminal device 3, and searches for feature values corresponding to the estimated sex. When a feature amount whose degree of similarity exceeds the threshold cannot be found, the server device 1 searches for a feature amount corresponding to a gender other than the estimated gender.

Further, for example, the server device 1 uses a learning model for estimating the age of the user depicted in the captured image to estimate the age of the user depicted in the captured image captured by the terminal device 3 at the time of group registration. , the age estimation result is stored in association with the feature amount calculated from the photographed image. Note that the age estimated by the learning model does not have to be a highly accurate age for each year, but can be an approximate age such as 20 years old or younger, 20 to 39 years old, 40 to 59 years old, and 60 years old or older. you can At the time of group authentication, the server device 1 estimates the age of the user from the captured image captured by the terminal device 3, and searches for the feature amount corresponding to the estimated age. If a feature amount whose similarity exceeds the threshold cannot be found, the server device 1 searches for a feature amount corresponding to an age other than the estimated age.

Further, for example, when there are multiple stores, the server device 1 associates the store name, store ID, or the like with which group registration has been performed in the group DB 12b and stores them in the group DB 12b. At the time of group authentication, together with the photographed image photographed by the terminal device 3, the name of the store where the terminal device 3 is installed, etc. are acquired, and the user included in the group of the group ID corresponding to the acquired store name, etc. Search is performed for the feature amount of . If a feature amount whose similarity exceeds the threshold cannot be found, the server device 1 selects a store other than the acquired store (for example, the closest store), and finds the feature amount of the corresponding group such as the store name of this store. Do a search for

In this way, it can be expected that the search time can be shortened by limiting the range in which the server device 1 searches for similar feature amounts from the group DB 12b at the time of group authentication based on conditions such as gender, age, and store. Note that these conditions may be used in combination. For example, the search range of the feature amount may be limited by the condition of gender and age.

(4) Updating the database The server device 1 calculates a feature amount from the user's photographed image taken by the terminal device 3 at the time of group authentication, searches for a similar feature amount from the group DB 12b, and authenticates the group. . The server apparatus 1 may update the feature amount by associating the feature amount calculated at this time with the group ID of the authenticated group and storing it in the group DB 12b. When storing a new feature amount in the group DB 12b by updating, the server apparatus 1 may store the new feature amount by overwriting the previous feature amount, or may store the new feature amount while leaving the previous feature amount. feature amount may be stored.

(5) Collective Shooting of Multiple Users In the information processing system according to the above-described embodiments, one user is shot with the camera 36 of the terminal device 3, and multiple users included in the group are individually shot. However, the information processing system is not limited to this, and the information processing system may be configured to be able to photograph a plurality of users in one photographing. For example, the server device 1 uses a learning model for detecting human faces to detect the faces of a plurality of users captured in the images captured by the terminal device 3, and extracts images of each user's face from the captured images. can be extracted.

The information processing system according to the present embodiment does not install the terminal device 3 on each table in the store, but takes pictures of the user with one or a plurality of cameras such as surveillance cameras installed at suitable places in the store, for example. may When photographing with such a camera, a method of collectively photographing a plurality of users with one camera is preferable.

(6) Calculation of Feature Amount In the information processing system according to the above-described embodiment, the server device 1 acquires a photographed image of the user's face photographed by the terminal device 3, and based on the acquired photographed image, the server device Although the device 1 calculates the feature amount, it is not limited to this. The information processing system may be configured such that the terminal device 3 calculates a feature amount from the captured image and transmits the feature amount to the server device 1 . The terminal device 3 includes a mask estimation model for estimating whether or not a mask is worn, a first feature value calculation model for calculating a first feature value of the entire face of the user captured in the captured image, and a captured image. A learning model such as a second feature amount calculation model for calculating a second feature amount of a specific part of the user captured in the image is stored, and the feature amount is calculated from the captured image using these learning models.

<Summary>
In the information processing system according to the present embodiment having the above configuration, the server device 1 associates groups to which a plurality of users belong and feature information of the users belonging to the groups, and stores them in the group DB 12b. The server device 1 acquires a photographed image of the user photographed by the camera 36 of the terminal device 3, and based on the acquired photographed image and the characteristic information stored in the group DB 12b, the user shown in the photographed image belongs to the user. Identify the group and authenticate that the group to which this user belongs is a registered valid group. Accordingly, the information processing system according to the present embodiment can be expected to authenticate a group including a plurality of users.

Further, in the information processing system according to the present embodiment, when at least one of a plurality of users belonging to a group is shown in a captured image captured by the terminal device 3, this user belongs to the group. Authorize the group to use. As a result, the information processing system can identify and authenticate a group by acquiring a photographed image of at least one user without acquiring photographed images of all users belonging to the group.

Further, in the information processing system according to the present embodiment, the photographed image of the user's face or the feature amount calculated from this image is stored in the group DB 12b as feature information. The server device 1 calculates the feature amount of the user's face shown in the photographed image, and based on the calculated feature amount and the image or the feature amount stored in the group DB 12b, the user photographed in the photographed image belongs to. Authorize the group to use. Accordingly, the information processing system according to the present embodiment can be expected to accurately authenticate a group based on a photographed image of a user's face.

Further, the feature amount handled by the information processing system according to the present embodiment includes a first feature amount for the entire face of the user and a second feature amount for a specific portion of the user's face. The second feature amount is a feature amount of a specific portion of the user's face that includes the eyes and does not include the mouth, such as the user's eyes and their peripheral portions. The server device 1 determines whether or not the user shown in the captured image is wearing a mask, performs group authentication based on the first feature amount for users not wearing the mask, and wears the mask. Group authentication based on the second feature quantity is performed for users who have Accordingly, the information processing system according to the present embodiment can be expected to accurately authenticate a group regardless of whether or not the user is wearing a mask.

Further, in the information processing system according to the present embodiment, attribute information such as gender or age of the user may be stored in the group DB 12b in association with the feature quantity. The server device 1 estimates the attribute information of the user shown in the captured image acquired from the terminal device 3, and among the feature amounts stored in the group DB 12b, the associated attribute information matches or matches the estimated attribute information. Similar feature amounts are preferentially read, and a group is identified based on the similarity of the read feature amounts and the feature amounts calculated from the captured image. As a result, the information processing system according to the present embodiment can be expected to increase the processing speed related to group authentication.

Further, in the information processing system according to the present embodiment, the server device 1 accepts new registration of a group to which the user shown in the captured image acquired from the terminal device 3 belongs, and the new group ID and the captured image are stored in the group DB 12b. Register the feature value calculated from The server device 1 also accepts additional registration of another user to the registered group, and registers the feature amount calculated from the photographed images of the second and subsequent users in association with the registered group ID in the group DB 12b. As a result, the information processing system according to the present embodiment can store information about groups to which a plurality of users belong in the group DB 12b, and perform group authentication based on captured images of one or more users.

Further, in the information processing system according to the present embodiment, the server device 1 gives incentives such as points or coupons to the authenticated group. As a result, the information processing system according to the present embodiment can be expected to prompt the user to perform group authentication, and can be expected to prompt the user to visit a shop or the like where group authentication is performed.

The embodiments disclosed this time are illustrative in all respects and should be considered not restrictive. The scope of the present invention is indicated by the scope of the claims rather than the meaning described above, and is intended to include all modifications within the scope and meaning equivalent to the scope of the claims.

1 server device (information processing device, computer)
3 terminal device 5 terminal device (second terminal device)
11 processing unit 11a captured image acquisition unit (acquisition unit)
11b feature amount calculation unit 11c group authentication unit (specification unit)
11d registration processing unit (storage processing unit)
11e incentive provision unit 11f information transmission unit 12 storage unit 12a server program (computer program)
12b Group DB (database)
12c learning model storage unit 13 communication unit 31 processing unit 31a authentication processing unit 31b registration processing unit 31c display processing unit 32 storage unit 32a program 33 communication unit 34 display unit 35 operation unit 36 camera 98, 99 recording medium N network

Claims (13)

The information processing device
A group to which a plurality of users belong is associated with feature information indicating features of users belonging to the group and stored in a database;
Acquiring a photographed image of a user,
Identifying a group to which the user in the captured image belongs based on the acquired captured image and the feature information stored in the database;
Information processing methods. The information processing device identifies a group to which the user belongs when at least one of a plurality of users belonging to a group is shown in the acquired captured image;
The information processing method according to claim 1 . The feature information includes an image of the user's face or a feature amount calculated from the image,
The information processing device is
calculating a feature amount of the user's face captured in the acquired captured image;
Identifying a group to which the user depicted in the captured image belongs based on the calculated feature amount and the image or feature amount stored in the database;
The information processing method according to claim 1 or 2. The feature amount includes a first feature amount for the entire user's face and a second feature amount for a specific part of the user's face,
The information processing method according to claim 3. The second feature amount is a feature amount of a specific portion of the user's face that includes the eyes and does not include the mouth.
The information processing method according to claim 4. The information processing device
Determining whether the user photographed in the acquired captured image is wearing a mask,
Identifying a group based on the first feature amount for users who do not wear a mask,
For users wearing masks, specifying a group based on the second feature amount,
The information processing method according to claim 4 or 5. The feature information includes user attribute information associated with the user's face image or feature amount,
The information processing device
estimating attribute information of the user captured in the acquired captured image;
Among the images or feature amounts stored in the database, preferentially read out images or feature amounts whose associated attribute information matches or is similar to the estimated attribute information to identify the group;
The information processing method according to any one of claims 3 to 6. The information processing device
Receiving new registration of a group to which the user photographed in the photographed image belongs,
accepting additional registration of another user for the group;
The information processing method according to any one of claims 1 to 7. The information processing device provides an incentive to the specified group,
The information processing method according to any one of claims 1 to 8. to the computer,
A group to which a plurality of users belong is associated with feature information indicating features of users belonging to the group and stored in a database;
Acquiring a photographed image of a user,
A computer program for executing a process of identifying a group to which a user shown in the photographed image belongs, based on the acquired photographed image and the feature information stored in the database. a storage processing unit that performs a process of associating a group to which a plurality of users belong and characteristic information indicating characteristics of users belonging to the group and storing the information in a database;
an acquisition unit that acquires a photographed image of a user;
An information processing apparatus, comprising: a specifying unit that specifies a group to which a user depicted in the captured image belongs, based on the acquired captured image and the feature information stored in the database. a terminal device that is provided in a facility and transmits a captured image of a user who has visited the facility by a capturing unit to an information processing device;
a database that associates groups to which a plurality of users belong and feature information indicating features of the users belonging to the groups;
An acquisition unit that acquires a photographed image of a user from the terminal device, and an identification unit that identifies a group to which the user depicted in the photographed image belongs, based on the acquired photographed image and feature information stored in the database. and an information processing device having a transmission unit that transmits information related to the group identified by the identification unit to the terminal device,
The terminal device receives information related to the group transmitted by the information processing device and displays the information on a display unit.
Information processing system. comprising a second terminal device for use by employees of said facility;
The transmission unit of the information processing device transmits information related to the specified group to the second terminal device,
The second terminal device receives information related to the group transmitted by the information processing device and displays it on a display unit.
The information processing system according to claim 12.

Images