WO2011083732A1 - Information processing system - Google Patents

Abstract

Disclosed is an information processing system that creates relevant information, marketing information, and the like on the basis of evaluation of specific objects that receive attention in scenes. The information processing system can evaluate search requests from a plurality of users in accordance with scenes in terms of specific objects recognized in accordance with image search requests from the users on the basis of a mother database (MDB) reconfigured from a database for configuring objects, and can sequentially supply information related to the searched objects, marketing information, and the like to user groups that issued the image search requests in accordance with the degree of attention.

Description

Information processing system

The present invention relates generally to an information processing system, and more particularly to an information processing system that performs evaluation based on the number of reference links to a database.

Information retrieval technology has advanced as the number of viewers and the amount of information on the network increase with the spread of the Internet in recent years. For example, there are a number of portal sites that can use powerful search engines. In addition, a technology has also been developed that analyzes the search keywords and access history of readers, and delivers Web pages and advertisements that meet the preferences of the readers, and marketing based on the keywords used by readers will also be conducted. The

For example, there is an information providing apparatus which can provide useful information accurately and easily for the user (Patent Document 1). The information providing apparatus stores access frequency information representing access frequency information indicating the frequency of access to each content by the user in association with user identification information identifying the user, and access tendency to content among the users The inter-user similarity calculation means for calculating the inter-user similarity representing the similarity between the users based on the access frequency information stored in the access history storage means, and the inter-user similarity between the user and each user Content score calculating means for calculating a content score which is information indicating the usefulness of the content for the user from the access frequency information of each user weighted by the content score calculating means, and the content score calculated by the content score calculating means Content score is associated with the user identification information Storing index storage means, query input means for receiving an input of a query including user identification information transmitted from the communication terminal device, and acquiring content identification information of content conforming to the query received by the query input means A provision information generation unit that generates provision information from the acquired content identification information by referring to the content score stored in the index storage unit in association with user identification information included in the query; And a provision information output unit configured to output the provision information generated by the provision information generation unit to the communication terminal apparatus.

In addition, there is a search server that presents the season of search keywords and the reason for the popularity (Patent Document 2). This search server includes a blog article collecting means for collecting blog articles, a period determining means for determining a target period for displaying the appearance frequency of a search keyword according to information received from a client, and a section for each section determined from the target period. The appearance frequency of the search keyword is counted for the blog article updated in the above, and the appearance frequency counting means for generating time series information of appearance frequency, the peak interval detection means for detecting the peak interval of the appearance frequency, and the peak interval The related keyword extraction means for extracting the related keyword of the search keyword from the blog article updated in 4 and the graph information generation means for generating the graph information which graphed the time-series information and transmitting it to the client A related keyword is displayed as a label of the peak section.

In addition, there is an advertisement distribution system in which the advertiser side can easily grasp the advertising effect of the banner advertisement, and provides the banner advertisement having a higher advertising effect to each user (Patent Document 3). This advertisement delivery system refers to the user profile of the user corresponding to the user terminal from among a plurality of advertisement data stored in advance. Then, the advertisement data to be distributed to the user terminal is extracted based on the user profile by reference. Next, the present invention is characterized in that the extracted advertisement data is distributed to the user terminal via the network to display an advertisement.

JP, 2009-265754, A JP 2008-262301 A JP 2002-236846 A

However, the tabulation of reference history and access history in the conventional system is only to total the access to the entire page in units of the location (URL etc.) of the web page, and for example, to any part of the web page It was not possible to collect attention information that led to details, such as whether the viewer was interested.

Furthermore, also in market analysis based on search keywords and search queries, text processing is mainly performed, such as counting the frequency of use of keywords and calculating the degree of association between keywords. In addition, technology to improve the delivery efficiency of advertisements is also limited to analysis and analysis of profile information such as user's gender, age group, preferences, and areas of interest to extract and deliver highly relevant advertisements. An analysis of an object that the viewer specifically (or not) focused on is desired.

Therefore, an object of the present invention is to provide an information processing system that analyzes a specific object (the object) focused by a viewer and distributes information, advertisements, and the like related to the object of interest. .

In the information processing system according to the present invention, a plurality of objects for which a specific object is recognized in response to an image search request from a user based on a mother database (MDB) reconstructed from a database for constructing the object The search request from the user is evaluated according to the scene, and it is possible to sequentially provide the user who made the image search request with the advertisement and / or service related to the target object according to the degree of attention.

Further, the information processing system according to the present invention makes it possible to provide statistical data generated from the degree of attention to a plurality of objects to a user and / or a third party.

Further, in the information processing system according to the present invention, when there is an unregistered specific object recognition request in a mother database (MDB) reconstructed from a database for forming an object, a plurality of users and If an unknown object discovery notification (attention) is given to three parties, and if it is possible to distinguish an object based on the collective intelligence from the user and the information provided by the third party, the detailed database pertaining to the object will be MDB After registration, a plurality of users and third parties are notified that the unknown object has been identified, and advertisements and / or services relating to the target object can be sequentially provided to the user.

Further, the information processing system according to the present invention enables the user to change the attribute of a part of the object according to the user's preference using the MDB of the object for which the specific object is recognized, and the user's preference or request To provide to third parties as marketing information.

In addition, the information processing system according to the present invention preliminarily registers, in the MDB, an object which is likely to be put into the market in advance, and can promptly respond to an image search request from a future user.

Further, in the information processing system according to the present invention, when the specific object recognition request from the user in the scene transits to another object, the degree of attention is evaluated based on the transition history information, and the evaluation is performed based on the evaluation. Statistical data can be provided to third parties as marketing information.

According to the information processing system of the present invention, the following effects can be obtained.
(1) With regard to an object focused by a viewer in a specific image (including still images and moving images, the same applies hereinafter), it is useful for the viewer based on the degree of attention (frequency and / or attention rank) of the object It is possible to realize an information processing system that provides additional information.
(2) It is possible to realize an information processing system which additionally displays a display such as an advertisement and / or a service related to the target object based on the degree of attention of an object which the viewer focuses on in a specific image scene.
(3) It is possible to provide an information processing system that performs marketing based on the transition of the degree of attention of the viewer with respect to an individual object in a specific scene such as an image.

It is a system configuration figure in one embodiment of the information processing system concerning the present invention. It is an explanatory view explaining a server and a functional block of a mother database (MDB) in one embodiment of an information processing system concerning the present invention. It is an explanatory view explaining a functional block of a terminal unit in one embodiment of an information processing system concerning the present invention. It is an explanatory view explaining the data flow in one embodiment of the information processing system concerning the present invention. It is an explanatory view explaining ADB, MDB, and a data flow of a marketing information generation part in one embodiment of an information processing system concerning the present invention. It is an explanatory view explaining an attention object in a scene in one embodiment of an information processing system concerning the present invention. It is an explanatory view explaining ADB in one embodiment of an information processing system concerning the present invention. It is explanatory drawing explaining the example of analysis of the view object in the scene in the other embodiment of the information processing system concerning the present invention (example of generation of marketing data). It is an explanatory view explaining an operation flow in one embodiment of an information processing system concerning the present invention. It is an explanatory view explaining an operation flow in one embodiment of an information processing system concerning the present invention. It is an explanatory view explaining an operation flow in one embodiment of an information processing system concerning the present invention. It is an explanatory view explaining an operation flow in one embodiment of an information processing system concerning the present invention. It is an explanatory view explaining an operation flow in one embodiment of an information processing system concerning the present invention.

Hereinafter, modes for implementing the information processing system according to the present invention will be described in detail.

FIG. 1 shows a system configuration of an information processing system according to an embodiment of the present invention.
The information processing system 100 includes an information processing server 101, an MDB 102, and terminal devices 105a to 105d used by a user. The server 101 and the MDB 102 are connected by a connection 103, and the server 101 and the terminal device group 105 are networks It is connected to 104.

The server in the present specification is a computer system that provides some service in response to a request from a client, means a computer group in which a plurality of server systems are coupled, and is driven by a plurality of software. Thus, in the present specification, the meaning of the server is positioned as described above.

The computer as hardware is, in its most basic configuration, an electronic computer provided with an arithmetic logic unit, a control unit, a storage device, and an input / output device connected by an instruction bus and a data bus. Arithmetic operation, logical operation, comparison operation, shift operation, etc. are executed in the arithmetic logic unit based on information (bit data) input from the input / output device via the input / output interface. The executed data is stored in the storage device as needed and output from the input / output device. The series of processes are controlled by a software program stored in a storage device. Each server machine in the present invention is also hardware that is minimally equipped with the above-described basic functions as a computer, and is controlled by programs such as an operating system, device drivers, and application software.

FIG. 2 shows functional blocks of the information processing server 101 and the MDB 102 in an embodiment of the information processing system according to the present invention. The information processing server 101 includes, as software functional blocks, an area processing unit 201, a general object recognition unit 202, a specific object recognition unit 203, a network communication control unit 204, a data search processing unit 205, and an MDB search unit 206. And an MDB learning unit 207, an MDB management unit 208, a scene processing unit 210, a marketing information generation unit 211, and a focus point processing unit 212. The functional blocks of the information processing server 101 are not necessarily limited to these, but these representative functions will be briefly described.

The area processing unit 201 performs area division in the image, cutting out of partial images, and the like. The general object recognition unit 202 recognizes an object included in an image with a general name (category). Further, the object recognition unit 203 collates the information registered in the MDB 102 to identify an object. A network communication control unit 204 performs image input / output processing, information communication control with a terminal, and the like. The data search processing unit 205 collects information from the link destination, performs an inquiry of collective intelligence, collection, and a search. The MDB search unit 206 searches for tag data such as the name of an object. The MDB learning unit 207 performs addition of detailed information on an object, registration of time information, registration, addition, and the like of incidental information. The MDB management unit 208 extracts feature points and feature amounts from object configuration data, extracts category information from incidental information, registers in category data, extends, divides, updates, integrates, and corrects category classification in category data. Register new categories, etc.

The scene processing unit 210 performs processing to recognize the same scene in a still image or a moving image, and the marketing information generation unit 211, which will be described later, is a database (DB) structure (attention database extracted from the degree of attention to a specific object : Generate marketing information based on the information of ADB). The focus point processing unit 212 performs specific object recognition based on the MDB 102, processes a reference to a target object, etc., and stores the totaled result in the ADB 2101.

Further, the MDB 102 includes object configuration data 251, incidental information data 252, feature amount data 253, category data 254, and nonspecific object data 255. Although the MDB 102 is not necessarily limited to these, these representative functions will be briefly described.

The object configuration data 251 is a database for configuring an object, and configures or manufactures an object, such as structure, shape, dimensions of the object, connection information of parts, layout drawing, movable part, movable range, weight, rigidity, etc. Holds basic information necessary for
The additional information data 252 holds all the information on the object such as the name of the object, the manufacturer, the part number, the date and time, the material, the composition, and the processing information.
The feature amount data 253 holds feature points and feature amount information of individual objects generated based on the above information.
The category data 254 holds information used when performing category classification of an object in the general object recognition unit.
The unspecified number data 255 stores an object not recognized by the specific object recognition unit. If a specific object is recognized by the evolution of MDB, it is removed.

FIG. 3 shows a terminal device in an embodiment of the information processing system according to the present invention. The terminal devices 105a to 105d are client terminal devices widely used by users, and include computers, personal digital assistants (PDAs), cellular phones, and the like. That is, the terminal devices 105a to 105d represent a state in which many kinds of electronic information devices are connected to the network. Hereinafter, when the terminal device 105 is described, one of the terminal devices 105a to 105d connected to the network is referred to. Naturally, the terminal devices 105a to 105d do not have to be all the same model. It may be a terminal device provided with equivalent functions (or minimum functions that can be performed). Here, typical functional blocks of the terminal device 105 will be described.

The terminal devices 105a to 105d include an input / output unit 301, a network connection unit 302, a CPU 303, a storage unit 304, and a power supply unit 305. The input / output unit 301 is, for example, an input device such as a keyboard, a mouse, a pen, a jog dial, a joystick, a touch pad (including one incorporated in a display), a microphone for voice recognition, and an output such as a display, a speaker, or a vibrator Configured with devices. The network connection unit 302 is an interface for connecting to a network, and may be wired or wireless.

The CPU 303 is a central processing unit, the storage unit 304 is a storage device such as a hard disk and a memory, and the power supply unit 305 is a battery or the like.

In addition to this, a sensor for detecting inclination or acceleration, a GPS receiver, or the like can be mounted as needed.

FIG. 4 shows a data flow in an embodiment of the information processing system according to the present invention. In the information processing system according to the present invention, the user 401 browses the scene 402 using the terminal device 105, and the information processing server 101 is inquired about an object focused on the scene 402. This inquiry is made by pointing an object of interest in the scene 402 with a pointing device of the terminal device 105 (not shown).

Here, in the case of a two-dimensional scene, not only a single frame cut out as a still image such as a photograph but also an aggregate thereof becomes the same scene if a plurality of photographs capture the same situation. In the case of a moving image, a frame group recording one continuous situation is the same scene. Even in the case of a moving image, when a plurality of moving images record the same scene, an aggregate of the plurality of moving images similarly constitutes the same scene.
The scene is stored, for example, on a third party server (not shown) and can be viewed by a plurality of users.
For example, the user can inquire of the object of interest to the information processing server 101 via the software according to the present invention incorporated in the terminal device 105.

On the other hand, although not shown in FIG. 4, the information processing server 101 identifies an object detectable based on the MDB 102 from various still images and moving images existing on the network in advance, In response to the query for the object recognition request, the recognition result of the specific object in the scene 402 can be promptly returned to the user. Similarly, in the case of a still image or moving image uploaded and distributed in real time from the user and the third party 404, detection of a specific object is started in real time based on the MDB 102, and the detection to the user and the third party 404 is performed. Return the result promptly. That is, the information processing server 101 tries to observe which object in the scene the user focused on by performing object recognition in response to an inquiry about a target object from the user for each scene. By these, the third party 404 can provide the user with the outgoing information 405 from the advertiser and the service provider.

The concept of the scene assumed in the present invention is a wide range. For example, assume a fashion show in which many models appear. When shooting this fashion show with multiple cameras from different angles, the stage stage constitutes one scene, and the images from each camera constitute one view. The scene in the present invention assumes not only the view at each camera but also the former scene.

In addition, a scene assumed by the present invention is reconstructed from a plurality of contents created by the user's hand and called UCG (user-generated content) or CGC (consumer generated content), and uploaded to a server or shared site. Ru. For example, still images and moving images of sports watching games uploaded from a user to a shared site can also constitute the same scene. Alternatively, the same scene is reconstructed from the video of the surveillance camera taken from a plurality of angles.

In order to utilize the scene, not only the content as described above, but also third parties 404 and advertisers etc. prepare in advance still images and moving images including objects that the user wants to see, In some cases.

The information processing server 101 in FIG. 4 performs a process of recognizing the same scene in a still image or a moving image, and recognizes a specific object based on the MDB 102 and processes a reference to a target object, etc. A processing unit 212, an ADB 2101 for recording aggregation results processed by the focus point processing unit 212, and a marketing information generation unit 211 for generating marketing information to be described later based on the information recorded in the ADB 2101. And are included.
In addition, "focused rank" means a focused index by a plurality of users (for example, how many users focused on it, etc.), and "focused frequency" means the frequency that a specific user focuses on the same specific object. Do.

The information processing server 101 can also solve the unconfirmed object by inquiring of collective intelligence and third parties about the unconfirmed object for which the specific object could not be identified (FIG. 4).

Also, a third party can provide object configuration data such as CAD for the product of the third party as specific object information to the information processing server 101 in order to obtain more accurate marketing information (FIG. 4). The information processing server 101 can also provide the advertiser 405 or a third party 406 with the statistical data or the marketing information generated by the marketing information generation unit 211 (FIG. 4).

FIG. 5 illustrates ADB, MDB and data flow of the marketing information generation unit in an embodiment of the information processing system according to the present invention.

In FIG. 5, in the information processing server 101, the data structure of a recognition object in each scene is managed in the ADB 2101 for each scene. For example, in FIG. 5, in the scene 1 (501), specific objects 501a and 501b are recognized. As a result of recognition, (1) category (general name level of object), (2) specific object information (name and type of specific product), (3) attention degree, and (4) attention object for each specific object Transition information is stored in a predetermined data structure. Here, transition information can be used to generate statistical data based on transitions within categories, transitions between categories, as well as transitions along a time axis as described later.

As apparent from the above description, the information processing server 101 has a feature of grasping a scene and analyzing the degree of attention of a specific object in the scene.

A specific example of scene and attention level analysis in an embodiment of the information processing system according to the present invention will be described with reference to FIG.

FIG. 6A is a specific example of a scene. As shown in the figure, in the scene 402, the Champs Elysees with the view of the Etoile Arc de Triomphe 601 seen at a distance is photographed, and on the Champs Elysees, an infinite number of cars are traveling.

The scene 402 is shared by a plurality of users, and the user can designate an object of interest using a pointing device or the like of the terminal device 105 (not shown). The user can point to various objects in the scene 402, but for simplicity of explanation, the triumphal arch 601, the car 602 and the car 603 are taken as representative examples of the pointed objects, and the degree of attention obtained respectively is obtained Will be described based on FIG. 6B.

FIG. 6B is a specific example of ADB. First, as shown in the upper table in FIG. 6B, the scene 402 is a view assigned an ID of 402, and the type is photo. And, it constitutes one view of a scene (c1010) called "Chants elysée where Etoile Arc de Triomphe can be seen". According to the table in the upper part of FIG. 6B, in addition to the view identified by the ID 402, another view 403 (ID 403, type is a moving image (movie)) configuring the scene (c 1010) is registered. That is, since the content (not shown) of the ID 403 also belongs to the scene (c1010), the scene "Chant élysées where the Etoile Arc de Triomphe can be seen" is configured similarly to the ID 402.
The upper table of FIG. 6B has the following data structure in one embodiment.

In the “link to target object” field, a link to one or more target objects focused in the scene (or view) is provided. In FIG. 6B, since the Arc of Triumph 601, the car 602 and the car 603 are focused as described above, links to these are provided. Further, since the triumphal arch 601 is included in the view 403 (content not shown), a link to the triumphal arch 601 is provided.

Next, in the middle table of FIG. 6B, structure data summarized for each target object is shown. The object 601 focused in the scene 402 is recognized as "Etoile Arc de Triomphe 601" by the recognition processing (Fig. 12 etc.) described later, the focused object 602 is recognized as "car type A (602)", and the focused object 603 is "car type B (603) "is recognized. Then, the link to the MDB of the recognized object and the degree of attention of the object are recorded. In the middle table of FIG. 6B, the attention degree of the Etoile Arc de Triomphe 601 is 3557, the attention degree of the car type A (602) is 15, and the attention degree of the car type B (603) is 135.
The calculation of the degree of attention is appropriately weighted as well as the number of times of reference by the user.
The middle table of FIG. 6B has the following data structure in one embodiment.

In the "link to view" field, a link to one or more views in which the object of interest exists is provided. In FIG. 6B, since the Triumphal Arch 601 is included in the views 402 and 403 as described above, links to these views are provided.

The table at the bottom of FIG. 6B shows a data structure for managing the focus time for each of the scene, view, and target object. In the first row at the bottom of FIG. 6B, it is recorded that the object 602 is focused on January 5, 2010 at 13:23:45 in the view 402 of the scene C 1010, and a link to the object 602 is established. It can be seen that this object is a car type A of a car (in FIG. 6B middle row).
The lower table of FIG. 6B has the following data structure in one embodiment.

In the lower part of FIG. 6B, there is only one item for recording time, but in place of this, along with the user (can be identified by ID, IP address, etc.) focusing on multiple histories of the focused time. It may be managed as a list.

An example of analysis of an object of interest (example of generation of marketing data) in a scene in another embodiment of the information processing system according to the present invention will be described with reference to FIG. FIG. 7A shows statistical information of a scene generated from the accumulation of the degree of attention of the user for each object, which is detected for each scene by the information processing system 100, and FIG. , 1 hour, 6 hours, 24 hours, 1 week, January, 3 months, half a year, 1 year, 10 years, etc.), and is statistical information generated from a change in the degree of attention for each object in each scene. The generation of these statistical information will be described in detail along with the underlying idea.

It is assumed that the user A instructs the specific object recognition of the object 1 (the car 1) in one view (for example, a photo) of a certain scene. In this case, the information processing system 100 identifies the object 1 (the car 1), and stores the information stored in the MDB (for example, manufacturer, model name, year, color, model (for example, 2 doors, 3 doors, hatchback) , One box, etc.), displacement, etc.) and / or options that can be identified (eg sunroof, roof rack, fog lights, wheels etc.), and links to the information in each MDB to the objects in the scene It records with an attention degree as a thing regarding the object 1 in ADB. If the user further performs specific object recognition of another object in the object 1 (car 1), for example, the wheel N, the information of MDB as the object N in the ADB of the scene (for example, the manufacturer name, etc.) Record links to model number, size, model year, retail price, material, etc.).

Next, when the same user A performs a specific object of another object 2 (car 2), the specific object recognition is similarly performed, and the detectable object information unique to the car is recorded as the ADB 2 of the object. Furthermore, even when the user gives an instruction to specify an object other than a car, the information is recorded in the ADB as focused information of each object.

Further, it is assumed that another user B performs specific object recognition of a certain car in a certain view of the same scene. In this case, the information processing system 100 identifies a car as described above. As a result, if the car is already detected, the degree of attention of the object A in the registered ADB is updated. If it is a newly detected car, it is newly registered as an object 3 in the ADB. Furthermore, when the user instructs specific object recognition of the wheel N of the car, the degree of attention of the wheel N is similarly updated.

Thus, by repeating the above process, the information processing system 100 can accumulate the degree of attention of each user for each object detected for each scene, and generate statistical information of the scene (FIG. 7A). ).

Further, in parallel with the above processing, the information processing system 100 takes a unit of a period (for example, 1 hour, 6 hours, 24 hours, 1 week, January, 3 months, half a year, 1 year, 10 years, etc.) It is possible to generate statistical information on the change in the degree of attention of the object in a unit of period related to the degree of attention of the object in each scene (FIG. 7B).

The above generation example is an example, and various other marketing data can be extracted.
For example, the following data can be generated.
1. Degree of attention in the category to which the object belongs (for example, car). For example, data generation such as model name, model, color, year model, etc. of top N cars with high degree of attention of user in car X car in scene.
2. Data generation of top N object categories, names, attention ranks, and locations of objects in the scene, among all objects in the scene.
3. Data generation such as the category, name, degree of attention, etc. of the high-ranked objects for each view in the scene

FIG. 8 illustrates an operation flow (in particular, object recognition in a scene and generation flow of ADB) in an embodiment of the information processing system according to the present invention. First, when the steady process step in the information processing system 100 is started in S801, the process proceeds to S802, and data such as an image is input. For example, the data includes a plurality of images such as still images and moving images, and these data are arranged in time series. Next, in step S803, a scene is cut out from the input data. And it cuts out a view for each cut out scene.

In S804, an object is cut out from the cut out view this time using a segmentation method or the like. Next, in step S805, recognition processing of a specific object (see FIG. 12 described later) is performed.

Then, if a specific object is recognized in S806 (for example, see FIG. 12), ADB is generated and a link with MDB is established. Next, in step S807, it is determined whether there is an image capable of recognizing a specific object in the view. If YES, the process returns to step S805 to repeat the subsequent processing.

In the case of No in the above-mentioned S807, that is, finally when the scene, the view, and all the specific objects present in it are recognized, the process proceeds to S808 and the processing is ended. At this point in time, scenes and views will have links to MDBs, so it will be possible to reconstruct them and create new scenes.

FIG. 9 illustrates an operation flow (in particular, an ADB update and an object reference history registration flow) in an embodiment of the information processing system according to the present invention. First, in S901, processing of a focused point instructed by the user using a pointing device or the like of the portable terminal 105 (not shown) is started. Next, in step S902, category recognition based on general object recognition and object identification processing based on specific object recognition are performed on an area (target object) specified by the user in a view in a scene. Then, in S903, it is determined whether or not the target object has been recognized. If it can not be recognized in S903 (No in S903), the processing is ended, but if it can be recognized (Yes in S903), the process proceeds to step S904.

In S904, it is determined whether the recognized object has already been recognized in the scene. Specifically, the object structure of interest is searched and investigated. If the object is a detected object (Yes in S904), the process proceeds to S906. On the other hand, if it is a newly detected object (No in S904), the process proceeds to S905.

In S905, registration processing of object information to ADB is performed. That is, since what is registered is a newly detected object, an object ID, a category, a link to an MDB are registered in the object structure of interest in FIG. 6B, and further, a scene ID, a link to a view ID, and an image type (Photo or video) is registered.

Next, the process proceeds to S 906, where the degree of attention of the ADB (FIG. 6B) regarding the object of interest is updated. Then, the process proceeds to S 907, the reference history regarding the object of interest is registered in the ADB (FIG. 6B) as a scene ID, view ID, time, and a link to the object, and the process for the object of interest ends (S 908).

The operation | movement flow (The advertisement selection flow based on the information of ADB) in one Embodiment of the information processing system based on this invention is demonstrated to FIG. First, when the process in the case where the user focuses on an object in S1001 is started, the process proceeds to S1002 and the user focuses on a specific object. For example, the object is clicked with a mouse, or the object is surrounded. Then, in S1003, it is determined whether the object has already been recognized by the information processing system. If it is recognized (Yes in S1003), the process advances to S1004 to obtain the ADB of the object. If it is not recognized (No in S1003), the process proceeds to S1005, the process of FIG. 12 is executed, and the specific object is recognized. Next, the process proceeds to S1006, and when the specific object is recognized in the process of FIG. 12, an ADB is generated and a link with the MDB is established.

Next, in S1007, the attention degree rank of ADB is updated. This value makes it possible to estimate how many people are paying attention to the object. In S1008, an advertisement and / or service appropriate for the user is selected from the attention level and transition information of the ADB. Then, the selected advertisement and / or service is presented to the user (S1009), and the process is ended (S1010).

FIG. 11 shows an operation flow (generation of marketing information based on attention degree and information related to the object (for example, scene, transition, place, time, etc.) in one embodiment of the information processing system according to the present invention. explain. First, when the process is started in S1101, the process proceeds to S1102, and the user focuses on a specific object. For example, the object is clicked with a mouse, or the object is surrounded. Then, in S1103, it is determined whether the object has already been recognized by the information processing system. If it is recognized (Yes in S1103), the object attention degree structure of the object is obtained (S1104). If not recognized (S1104), the process of FIG. 12 is executed to recognize the specific object (S1105). In S1106, when the specific object is recognized in the process of FIG. 12, an ADB is generated and a link with the MDB is established. In S1107, the attention degree rank of ADB is updated. This value indicates how many people pay attention to the object.

Next, proceeding to S1108, marketing information is generated from the attention degree rank of the ADB, transition information, and information related to the object. Then, the selected marketing information is provided to the third party (S1109), and this step is ended (S1110).

The flow of object recognition using the MDB 201 will be described based on FIG.

[General object recognition]
The start of the image recognition process (S1201) starts with the input of an original image in the terminal device 105 or the clipping of a scene by the information processing server 101 (S1202). The original image may be a two-dimensional image or a three-dimensional image. In addition, when an original image is input, an instruction of the target area of any object in the original image is made through a device (not shown) such as a pointing device, and the entire original image is input without processing an instruction of the target point. (For example, clipping of a scene by the information processing server 101). Next, general object recognition processing is performed in S1204. For example, a method of BOF (Bag-Of-Features) can be adopted for the general object recognition processing. In this general object recognition process, recognition of the detected object category (general name of object) is performed. However, when an instruction of a focused point is given, the process branches depending on whether the category can be recognized or not, and the determination is made in S1205. If the category can not be recognized, the process proceeds to S 1206, where it is judged about the handling of the existing category (S 1207), but if the category recognition of the object is possible regardless of the presence or absence of the instruction at the focus point The process advances to step S1209 to proceed to object recognition processing.

[Specific object recognition]
In the case of proceeding to the specific object recognition processing in the determination of S1205, first, the cutout processing of the individual object image is performed in S1209. Then, a specific object recognition process is performed on the clipped individual object image (S1210). In the specific object recognition process, identification of an object is attempted by an evaluation function that calculates the degree of coincidence based on feature amount data extracted from object configuration data registered in the MDB 201.

[Learning process (1)]
On the other hand, if general object recognition can not be performed in the determination of S1205, the process advances to S1206, and the information distance between the feature amount of the object of interest and the feature amount of the object belonging to the existing category that the MDB 201 grasps Based on the determination, it is determined whether a new category including the object of interest is registered (S1207) or whether the extension of the existing category close to the object of interest is considered (S1208). If a new category is to be registered (S1207), the process returns to S1204, and if an existing category is expanded (S1208), the process proceeds to S1209.

[After specific object recognition learning processing (2)]
In S1211, it is determined whether identification of a specific object has been achieved. If the specific object can be identified, the process advances to step S1213, and it is determined whether the individual object image extracted in step S1209 includes more detailed information than the detailed data of the object registered in the MDB 201. If YES in S1213, the process advances to S1214, the detailed data of the object in the MDB 201 is updated by the MDB learning unit, and more detailed information comes to be included. On the other hand, if it is determined No at S1213, the process proceeds to S1215, and the next determination is made.

In step S1215, it is determined that the general object can not be recognized in step S1205, and the process advances to steps S1208, S1209, and S1210 in the determination of step S1206, and is determined when the specific object is recognized (Yes in step S1211). In S1215, if the specified object is an existing category, the definition of the existing category registered in the MDB 201 is expanded or divided if the information distance of the objects in the category is dispersed by the expansion, Alternatively, if the information distance from the proximity category is equal to or less than the information distance between objects in the category, the defect of the information of the existing object is found by integration or registration of the specified object. A correction is made and category data is updated (S1216). On the other hand, if the specified object is not an existing category in S1215, the process transitions to S1207 and is registered as a new category.

In S1211, when the specific object can not be recognized, the object is registered as an “unconfirmed object” in the MDB 201, and the recognition process is ended in preparation for future processing (S1217). Also in the case where the existing category is expanded and updated in S1216, the recognition process is ended (S1217).

The learning function of the MDB management 202 in the above description improves the specific object recognition capability by being appropriately reflected in the operation of the information processing system according to the present invention.

[Other embodiments]
In the information processing system according to the present invention, the following embodiment can be taken.

(1) With regard to an object for which a specific object has been recognized in the above-described embodiment, the user directly uses the MDB for the object to directly set the attributes (eg, color, surface treatment, presence / absence of options, etc.) of the object in the scene. Allows changes such as zooming, viewpoint movement, and composition so that they can be picked up visually and interactively according to taste. As a result, from the situation where the product introduction corresponding to various tastes of the user is not necessarily performed in response to mass production, the change according to the user's taste, new specification request items, etc. are more detailed as marketing information It becomes possible to collect and realize effective multi-product production.

(2) An object to be put into the market is registered in advance in the MDB, and the image search request from the user can be promptly responded. For example, a third party, such as a manufacturer, pre-registers some of the CAD data of the new product family with the MDB and embeds it in a scene that can be shared by many users. After that, it will also be possible to introduce products with high attention to the market. On the other hand, for products that are not focused on the user for a long time, it is a good decision material for considering design changes, sales suspension, and so on.
In addition, it is desirable that prior registration of an object which may be put into the market in advance into the MDB be promptly deregistered (deleted) from the MDB as an absent object when not put into the market.

(3) Although the result of the specific object recognition is stored in ADB, embedding the metadata (text) as the recognition result into a predetermined area of the original scene or view image (for example, JPEG image) format Can contribute to the conventional image retrieval system.

(4) When there is an unregistered specific object recognition request in an MDB reconstructed from a database for constructing an object, notification of unknown object detection to a plurality of user groups and third parties with respect to the object (attention) And if it is possible to determine an object based on the collective intelligence from the user and information provided by a third party, a detailed database pertaining to the object is object configuration data 251 of MDB and / or incidental information data 252 After registering in, it is possible to notify the identification of unknown objects to a plurality of user groups and third parties, and to sequentially provide advertisements and / or services relating to the target object to the user groups.
Then, the unknown object also in the above is stored in the unspecified object data 255 of the MDB for future analysis.
As a method of object identification using collective intelligence, there are the following. For example, it is carried out by searching an encyclopedia on the Internet or automatically posting it on a Q & A bulletin board. In order for the system to search the encyclopedia on the Internet by itself, the search query is created using the category obtained by general object recognition and the feature quantity generated by the MDB, and the search is executed. Then, a new feature quantity is extracted from the returned content, and it is tried again whether identification of the object is possible. Also, when automatically posting on the Q & A notice board, the original image is uploaded to the notice board together with the category obtained by general object recognition. At this time, automatically edit the fixed form prepared in advance, and "Please tell me the model of this ○○" or "Please tell the website where the design information is disclosed about this △" Post a query such as Then, from other users (including human beings), "It is xx-xxxx" or "The design data of that Δ 入手 is obtained from http://www.aaabbb.com/cad/data.dxf "Can be done" advice. The system analyzes and evaluates these advices, accesses a designated URL, and attempts to download object design data and the like. If the object identification is successful based on the newly obtained design data, the obtained new data is added to the MDB and the database is updated.

100 information processing system 101 information processing server 102 mother database (MDB)
103 connection (network)
104 Network 105a to 105d Terminal device

Claims (6)

According to the scene, the search request from a plurality of users is performed on the object for which the specific object is recognized according to the image search request from the user based on the mother database (MDB) reconstructed from the database for constructing the object An information processing system capable of sequentially providing, to a user who made an image search request, an advertisement and / or a service related to a target object according to a degree of attention and evaluation. The information processing system according to claim 1, wherein statistical data generated from the degree of attention to a plurality of objects can be provided to a user and / or a third party. In the information processing system according to claim 1 or 2, when there is an unregistered specific object recognition request in a mother database (MDB) reconstructed from a database for forming an object, a plurality of specific object recognition requests for the object are performed. An alert (attention) to the user and the third party to find out the unknown object, and if it is possible to discriminate the object based on the collective intelligence from the user and the information provided from the third party, a detailed database related to the object An information processing system which guides a plurality of users and third parties to identification of an unknown object and registers an advertisement and / or a service related to a target object sequentially to the user after registering in the MDB. The information processing system according to any one of claims 1 to 3, wherein the user can change the attribute of a part of the object according to the user's preference using the MDB of the object for which the specific object is recognized An information processing system that enables third parties to provide their preferences and requirements as marketing information. In the information processing system according to any one of claims 1 to 4, it is possible to preliminarily register in the MDB an object which may be put into the market in advance, and to promptly respond to an image search request from a future user. Information processing system. In the information processing system according to any one of claims 1 to 5, when the specific object recognition request from the user in the scene transits to another object, the attention degree is determined based on the transition history information. An information processing system that makes it possible to evaluate and provide statistical data based on the evaluation to a third party as marketing information.

Images