WO2016187681A1 - Method for building an image data base, image recognition method, image recognition system and uses thereof - Google Patents

Abstract

The present invention relates to a method for building an image data base, to an image recognition method, to an image recognition system and to the uses thereof. The invention belongs to the field of information technology, more specifically to the areas of computer graphics, computer vision, image processing and recognition, and can be used for automatic identification and recognition of areas of interest on the basis of their visual representation.

Description

 IMAGE BASE CONSTRUCTION METHOD, IMAGE RECOGNITION METHOD, IMAGE RECOGNITION SYSTEM

 IMAGES AND THEIR USES.

 FIELD OF INVENTION

 [1] The present invention relates to a method of constructing an image base, an image recognition method, an image recognition system and its uses.

 [2] It is inserted in the field of information technology, more specifically in the areas of computer graphics, computer vision, image processing and recognition, having application in the identification and automatic recognition of places of interest from their visual representation.

 BACKGROUND OF THE INVENTION AND TECHNICAL STATE

 [3] Tourism is a billion dollar industry and major source of resources in many countries. In general, tourist activity involves the exploration of the visited place, with emphasis on the so-called local tourist spots that, for historical, geographical or leisure reasons, characterize the visited place. Being out of their environment, tourists often have difficulty recognizing a certain point, and do not know detailed information about that location.

[4] In this sense, many initiatives have been devised in order to give the tourist (or in general terms, a user about to know a new locality) tools that promote this identification, without the need for external guides, and preferably using the technological apparatus. available, mainly using mobile devices, such as smartphones or tablets, with an internet connection.

 [5] US document 2011/0176734 Al is aimed at recognizing specific areas of an image, especially areas containing buildings and constructions. However, the technology in question adopts the strategy of grouping characteristic points to perform identification, which ends up being a filtering process that reduces the ability to recognize image details as well as utilizes pose estimation, which leads to uncertainties and compromise the effectiveness of recognition. The scientific works of Liu (2014) and Hu (2006) also have similar application in the recognition of building facades, but restricted to this type of structure.

 [6] US 2009/7565139 B2 presents a proposal to provide the description of an image by combining optical character recognition, rigid objects, and face recognition. However, its application is restricted to the description of images (and their components), and is not intended to identify the photographed location.

[7] Some technologies, in turn, use geolocation data from the device in question to match images or photos in a collection of images, such as latitude / longitude coordinates or device orientation. This is the case with patents US 2008/0147730, US 2012/8189964 B2, US 2013/0198176 Al and US 8467810, and scientific work by Chen (2013), Zheng (2009), Abe (2010), Kim (2012) and Byungsoo (2009). [8] Alternatively, location recognition alternatives are dealt with in US patent 8483715 and in the works by Knopp (2010) and Chen (2009), who use images available on the Internet, on large bases such as Panoramio, Google Street Vie, Flickr, among others, associating with the recognition process geolocation or label information produced by the authors of these images.

 [9] Overall, the technologies found do not establish a well-defined protocol for reference image acquisition or database construction, and are primarily based on principles such as location tags or geolocation information from mobile devices. (smartphones, tablets). These solutions, however, have as a critical point the fact that they are supported by geolocation information labels that do not allow to identify in which direction the camera is pointing. Such a situation may lead to misidentification of a place of interest that, for example, is behind the user and not in front of him. Additionally, geolocation tags require the need for a GPS device for both the image base formation and the user's device. Such features make these solutions less robust.

[10] There is still no technology that describes the structured creation of an image base, which provides for the reference of a place of interest in different lighting conditions and viewpoints, and which promotes the processing efficiency and reliability of the image. system. In addition, the present invention provides a robust method for identifying places of interest from images based on identifying feature points; of images, criterion for acceptance or rejection WI ^ uu = ^I U characteristic ^W and found criterion for image identification confirmed from the characteristic points.

 [11] It is therefore concluded that none of the methods described in the prior art contemplates the differential provided by the present invention, since they do not solve the problem of the correct identification of places of interest with reliability, especially considering different conditions of registration of the photograph to be identified ^ as lighting or direction.

BRIEF DESCRIPTION OF THE INVENTION

 [12] The present invention relates to a method of constructing an image base, an image recognition method, an image recognition system and its uses.

 [13] 0. The method of constructing an image base comprises the steps of:

 (a) Acquisition of reference images from different positions at a sufficient distance for framing, preferably arranged in a circle around the reference and equally spaced, and under different lighting conditions, depending on the time of day and condition atmospheric.

 (b) preprocessing of the images;

 (c) extraction of the characteristic points of the image, and

(d) Storage in the database of the characteristic points of each location.

[14] The image recognition method is responsible for identifying and recognizing an input image by through efficient comparison with records in the database. Its steps are:

 (a) Receiving an input image to be identified;

 (b) image preprocessing;

 (c) extraction of characteristic points;

 (d) Comparison of feature points extracted with feature points from the database;

 (e) Identification of the photographed location.

 [15] Furthermore, the image recognition system comprises the application of the above methods together with the image acquisition and processing equipment, namely at least one mobile device with camera and Internet connection, and a server for remote processing.

 Finally, its various uses are still considered objects of the present invention, such as recognition of sights, building facades, landscapes, regions and buildings by aerial photography, in addition to use in augmented reality environments. The invention is not limited to the presented applications and can be used for the recognition of photographs and general visual representations.

 BRIEF DESCRIPTION OF THE FIGURES

 - Figure 1 illustrates the steps of the image base construction method.

 - Figure 2 illustrates the spatial arrangement of the viewpoints of reference photographs.

- Figure 3 illustrates the different recording conditions of the images under the different driving conditions, view, time, weather, and camera model.

- Figure 4 illustrates the steps of the image recognition method.

 - Figure 5 illustrates the criterion for excluding characteristic point pairs found as a function of their deviation from other pairs.

 - Figure 6 illustrates the image recognition system and its operation.

 - Figure 7 illustrates the comparison of two images with the marked characteristic points and the lines joining them.

- Figure 8 illustrates the comparison of the two images from another angle with the pairs of characteristic points joined by lines, but with the presence of two lines that are highly divergent from the others.

 - Figure 9 illustrates the situation that the tourist spot in question is photographed from the opposite angle to previous situations. ; .

DETAILED DESCRIPTION OF THE INVENTION

 [17] The present invention relates to a method of constructing an image base, an image recognition method, an image recognition system and its uses.

 [18] The method of building an image base comprises the steps of:

(a) Acquisition of reference images from different positions at a sufficient distance for framing, preferably arranged in a circle around the reference and equally spaced, and under different lighting conditions, depending on the time of day and condition atmospheric. (b) preprocessing of the images;

 (c) Extraction of characteristic points from the image;

(d) Storage in the database of the characteristic points of each location.

 [19] Figure 1 illustrates the steps of the image base construction method from acquiring reference images from a place of interest (101). The construction of the database and definition of the indexing and search mechanisms is performed by pre-processing these images (102), extracting their characteristic points.

(103) and storage in the database of the characteristic points of each site (104).

[20] The image acquisition step 101 follows a well-defined protocol illustrated by Figures 2 and 3. Each site of interest or reference to be employed in the construction _. The image base should be photographed from different positions, preferably arranged in a circle around the reference, equally spaced _; and at a distance sufficient for your frame. Optionally, the photographs can also be taken in the opposite direction of the place of interest, associating their surroundings and giving robustness to the image base (figure 2).

[21] This procedure should be repeated for different lighting conditions, taking into account the time of day (morning, afternoon and night) and the atmospheric condition (clear or cloudy). Optionally, different camera models and resolutions can be used to increase the variability of the generated reference images (Figure 3).

[22] It is important to note that each reference site may require a specific procedure, the image acquisition protocol being customizable. The more images acquired from a place of interest, the more likely it is that a photo of that place will be associated with an image from the image base. Places of interest with access or restricted viewing positions at certain angles do not require a total of 8 positions to be used for image acquisition, requiring only photos from positions without access impediment.

 [23] Preprocessing these images (102) is to resize each image to ensure that the largest side of all images is the same size. This resizing occurs to maintain the original aspect ratio (width divided by height) of the image. In embodiments of this invention, dimensions between 300 to 1000 pixels have been used.

 [24] On the resized image, a feature detection and extraction algorithm (103) is applied, which is transformed into a data representation to be recorded in the database. This representation contains all the information necessary for the recognition method to function, and it is no longer necessary to keep the original image stored.

[25] In one implementation of this invention, the Speed-Up Robust Features (SURF) algorithm was used for the extraction of feature points from the image, and the JSON format for data representation in the database. The SIFT algorithm {Scale-Invariant Feature Transform) can be used as an alternative to SURF, however, being less performance-efficient and less robust when applied to images with different transformations. Another suitable data representation format may also be used without compromising the essence of the invention.

 [26] At the end of this method, a data structure is created to maintain associations between the place of interest data representations of a photograph and the place of interest name associated with that photograph.

(104), resulting in the image base itself.

 [27] The image recognition method, which is responsible for identifying and recognizing an input image through efficient comparison with the records in the database, is illustrated in Figure 4, and can be described according to its main steps:

 (a) Receiving an input image (401) to be identified;

 (b) Image preprocessing (402);

 (c) extraction of characteristic points (403);

 (d) Comparison of extracted points with database points (404);

 (e) Identification of the photographed location (409 or 410).

[28] The recognition method works by receiving an input image to be identified (401), which must also go through a preprocessing step (402), where it resizes while maintaining the aspect ratio. and extracting its feature points (403), which can also be used SURF algorithm and the representation of JSON data in ^one: the implementations of this invention. [29] The method then traverses the database and compares the characteristic points of the input image with the characteristic points of the reference images that make up a _; looking for similarities (404). This step seeks to find correspondences or common characteristic points in the two images, then generating a list of the pairs of points that met the proposed similarity requirement. :

[30] Then the median of the angles between the lines formed by each is calculated _. points of this list, excluding the pairs whose line angle deviates from the median of the angles. In one embodiment of the invention disregarded straight-angle pairs of points diverging more than 2.5 ° from the median of the angles. Figure 5 shows that the dashed line (formed by a matching pair) between points in the input photo (501) and the database photo

(502) deviates from the pattern found for the other lines, so that their associated pair of points is excluded from the analysis.

 [31] Groups of feature point pairs that have the same feature point in the input image and different feature points in the reference image are also discarded. The same is true for the reverse situation. After this process, the final number of characteristic point pairs (405), which is above the previously defined threshold, is considered a positive association of images referring to the same place, identifying the input (410).

[32] The threshold value influences the robustness of the solution. A small threshold value results in a less demanding recognition process and can lead to false acknowledgments. Too high a value is restrictive and may lead to non-recognition of the point of interest.

[33] This threshold can range from 10 pairs for smaller images (largest side near 300 pixels) to 20 pairs for larger images (over 1000 pixels), and values ^; were determined by comparing various images of different resolutions and verifying the results of the comparisons for different threshold values.

 [34] The 15-pair value has been shown to be empirically adequate to be used as the threshold for any pair of images without significant impairment to method processing and reliability.

 [35] In one implementation of this invention, a threshold value of 15 point pairs was used, and the positive association found was obtained with 100% credibility. Using ever larger values as a threshold, the algorithm becomes more restrictive. For smaller and smaller values, the algorithm becomes more relaxed. For values between 10 and 20 pairs as threshold, the best results were observed.

[36] If the number of pairs found is below the defined threshold (406), the best association (with the highest number of pairs) found (407) that has at least 2 pairs of characteristic points is considered the correct solution after every scan. the reference stock photography (408), obtaining the result ^'closest (409), but with reduced reliability.

[37] The above process can be performed using threads to parallelize the database comparison process, increasing its speed, with each thread being responsible for comparing the input image with a part of the database. An additional processing time-based stopping criterion can also be defined to avoid a long search. In addition, process reliability and speed can be further enhanced by using image-associated metadata such as GPS data, photo registration time, device direction, etc., which may eventually be available if used. a mobile device, such as a smartphone or tablet. In these cases, the device used sends to the server the image to be identified and other available metadata (photo time, latitude and longitude, device direction, etc.). This metadata is then used by the server to guide the search for database images that must first be compared to the input image, allowing you to speed up the input image identification process by reducing the number of comparisons required until a match is found. satisfactory. To optimize database scanning, database records that correspond to metadata images that have a greater degree of similarity to the metadata associated with the input image receive higher priority when compared to the input image.

[38] Still, the image recognition system comprises the application of the methods mentioned above, was combined with the image acquisition and processing equipment, namely at least one mobile device with camera and internet connection, and also a server for remote processing. Its general operation is illustrated in Figure 6 and consists of a mobile photography user identifying a place of interest (1), register the location by means of a photograph (2) and send your photo to a server (3), which remotely analyzes the photo and, in the case of a positive identified association, returns with the name, other identifying information, of the location of interest identified and additional information registered in the database (4).

 Finally, its various uses are still considered objects of the present invention, such as recognition of sights, building facades, landscapes, regions and buildings by aerial photography, in addition to use in augmented reality environments. The invention is not limited to the presented applications and can be used for the recognition of photographs and general visual representations.

 CONCRETIZATION EXAMPLES

 [40] Figures 7, 8 and 9 illustrate the application of the invention in question to the identification of the São Francisco de Assis da Pampulha Church in Belo Horizonte. |

 [41] Figure 7 illustrates the comparison between the image to be identified (7b) and the image from the database (7a), with the characteristic points marked, as well as the lines joining similar pairs of characteristic points at? Two images.

[42] Figure 8, whose image to be identified (8b) is taken from another angle, and compared to the database image (8a), also illustrates the pairs of characteristic points joined by lines. In this situation, two straight lines are observed with angles very different from the median of the straight angles of all pairs of characteristic points found. The Point pairs of these two lines are discarded by the straight line acceptance criterion of the algorithm.

 [43] Figure 9, in turn, illustrates the situation that the tourist spot in question is photographed from its other end, highlighting the importance of the capture protocol for setting up a multi-image base. angles to a particular place of interest. In the upper pair of images, the image being identified (9b) is not associated with the database image (9a). In the pair of images at the bottom of the figure, where the capture angles are similar in both the database image (9c) and the image to be identified (9b), there is a positive identification of the place of interest.

References :

 - Fei Liu and Stefan Seipel. Detection of Façade Regions in Street Vie Images from Split-and-Merge of Perspective Patches Journal of Iraage and Graphics, Vol. 2, No. 1, pp. 8-14, June 2014.

 - Jingjing Hu, Joshua Sawyer, and Jean-Yves Herve Building Detection and Recognition for an Automated Tour Guide. Systems, Man and Cybernetics, 2006. SMC '06. IEEE International Conference on (Volume 1) 2006. pp. 283-289.

- Shioen Chen, Yi-Hao Chung, Hsin-Fu Chien and Chueh-Wei Chang A SURF Feature Based Recognition System for Distinctive Architectures, Information Technology Convergence. Lecture Notes in Electrical Engineering Volume 253, 2013, pp 111-119

- Yan-Tao Zheng, Ming Zhao, Yang Song, Hartwig Adam Ulrich Buddemeier, Alessandro Bissacco, Fernando Brucher Tat _T Seng Chua, and Hartmut Neven, Tour the World: building a ^: web- scale landmark recognition engineer, Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. 2009. pp 1085-1092

 - Naoyuki Abe, Atogu Oogami, Atsushi Shimada, Hajime Nagahara, Rin-ichiro Taniguchi. Clickable Real World: Interaction with Real-world Landmarks using Mobile Phone Camera TENCON 2010 - 2010 IEEE Region 10 Conference; 21-24 Nov. 2010. 914 - 917;

 - Daehoon Kim and Eenjun Hwang, Seungmin Rho, Location-based Large-scale Landmark Recognition Scheme for Mobile Devices, Mobile, Ubiquitous, and Intelligent Computing (MUSIC), 2012 Third FTRA International Conference on. 2012. pp 47 - 52

- Byungsoo Lim and Joonoo Kim. 2009. Efficient database reduction method of building recognition using global positioning systeirt on mobile device. In Proceedings of the 4th International Conference on Wireless Pervasive Computing (ISWPC'09). IEEE Press, Piscataway, NJ, USA, 168- 172. ^5;

 - Jan Knopp, Josef Sivic, and Tomas Pajdla. 2010. Avoiding confusing features in place recognition. In Proceedings of the 11th European Conference on Computer Vision: Part I (ECCV'10), Kostas Daniilidis, Petros Maragos, and Nikos Paragios (Eds.). Springer-Verlag, Berlin, Heidelberg, 748-761.

 - Tao Chen, Zhen Li, Kim-Hui Yap, Kui Wu, Lap-Pui Chau A Multi-scale Learning Approach for Landmark Recognition Using Mobile Devices. Information, Communications and Signal Processing, 2009. ICICS 2009. 7th International Conference on, 8-10 Dec. 2009, 1 - 4.

Claims

 1. Method of building an image base comprising the steps of:  (a) Acquisition of reference images from different positions at a sufficient distance for framing, preferably arranged in a circle around the reference and equally spaced, and under different lighting conditions, depending on the time of day and condition atmospheric. (b) preprocessing of the images;  (c) Extraction of characteristic points from the image.  (d) Storage in the database of the characteristic points of each location.  Method according to claim 1, characterized in that step (a) is optionally also carried out in the opposite direction of the place of interest.  Method according to claim 1, characterized in that step (a) is repeated using different image resolutions.  Method according to claim 1, characterized in that step (a) is performed from at least eight (8) different positions in a direction directly to the place of interest under at least two (2) different lighting conditions, and using at least 1 (one) resolution.  Method according to claim 1, characterized in that step (b) comprises resizing each image and equalizing the largest side of all images while maintaining their original aspect ratio. Method according to claim 1, characterized in that step (c) comprises detecting and extracting stitches. characteristic.  Method according to claim 1, characterized in that step (c) is preferably performed by the Speed-Up Robust Features (SURF) algorithm or the Scale-Invariant Feature Transform (SIFT) for the extraction of the characteristic points.  Method according to claim 1, characterized in that step (d) comprises the transformation of the characteristic points into a data representation and their registration in a database.  Method according to claim 1, characterized in that step (d) preferably uses the JSON format for representation of data in the database.  Method according to claim 1, characterized in that it obtains a data structure which maintains the associations between the data representations of the places of interest corresponding to a photograph and the name of the place of interest associated with that photograph.  11. Image recognition method comprising the steps of:  (a) Receiving an input image to be identified;  (b) image preprocessing;  (c) extraction of characteristic points;  (d) Comparison of feature points extracted with feature points from the database;  (e) Identification of the photographed location. Method according to claim 11, characterized in that step (b) comprises resizing the image to be identified while maintaining its aspect ratio. A method according to claim 11, characterized in that step (c) comprises extracting its characteristic points and their data representation.  Method according to claim 11, characterized in that step (d) comprises inspecting the database and comparing the characteristic points of the input image with the characteristic points of the reference images that make up the base.  Method according to claim 11, characterized in that step (d) seeks common matches or characteristic points in the two images, and then generates a list of the pairs of points that meet the proposed similarity requirement.  Method according to claim 11, characterized in that step (e) comprises calculating the median of the angles between the lines formed by each pair of points in this list, and subsequently excluding the pairs whose line angle deviates from the median of the angles. .  A method according to claim 11, characterized in that step (e) excludes groups of feature point pairs that have the same feature point in the input image and different feature points in the reference image, and vice versa.  Method according to claim 11, characterized in that the threshold for positive association is between 10 and 20 pairs, preferably 15. Method according to claim 11, characterized in that step ⁽ e ⁾ counts the final number of characteristic point pairs, and if this number is above a threshold defined, consider a positive association of images referring to the same place and identify the input.  Method according to claim 11, characterized by step (e), if no image reaches the threshold value defined after all reference image scan, choose the best association (with most pairs) found and that has at least 2 pairs of characteristic points as the correct solution.  Method according to claim 11, characterized in that it alternatively uses metadata associated with the image, such as GPS data, time of photo recording, device direction, etc.  Method according to claim 11, characterized in that the metadata prioritizes the comparison of database records according to the similarity with the metadata associated with the input image. ■ 23. image recognition system comprises the application of methods as described from 1 to 21 together with at least one mobile camera and with connection to the Internet, and a server for remote processing.  System according to claim 23, characterized in that it has thread-parallel processing.  System according to claim 23, characterized in that it comprises a stopping criterion based on processing time. Use of the system as defined in claims 23 to 25, characterized in that it is for point recognition sights, building facades, landscapes, regions and buildings by aerial photography.  Use of the system as defined in claims 23 to 25 characterized in that it is for augmented reality environments, photo recognition and general visual representations.