CN111506772B - Image searching method and system based on image feature extraction - Google Patents

Abstract

The invention discloses a picture searching method and system based on image feature extraction, comprising a feature extraction and comparison module, a picture feature extraction module and a picture searching module, wherein the source video is compressed and stored in a compressed video database through the feature extraction and comparison module, clustering is performed to form a cluster group, and a cluster sequence set corresponding to all source videos and a cluster sequence set of pictures to be queried are obtained, so that comparison is performed in the picture searching module to obtain a comparison result, and an actual frame picture and video are obtained by combining indexes stored in the compressed video database, so that further video acquisition is performed. The beneficial effects are that: and (3) carrying out mutual comparison of feature extraction on each frame in the video by utilizing an image feature extraction technology, and eliminating repeated frame feature sequences. The method has the advantages that the low occupied memory is realized, the memory occupation is reduced after video compression, compared with the prior art, the image search and retrieval speed is increased, and the positioning is accurate.

Description

Image search method and system based on image feature extraction

technical field

The present invention relates to the technical field of image search, in particular to a method and system for image search based on image feature extraction.

Background technique

With the development of the film and television industry, in order to record human cultural and artistic heritage, or some for-profit film and television viewing platforms or radio and television groups, a large number of videos are preserved. However, due to the accumulation of time, there is still a huge amount of data after classification and preservation.

With the development of a large number of people's use and intelligence, people begin to look for more convenient, fast and accurate video search methods. In the prior art, when video is saved, it is usually to name each video by a keyword or set text information such as recording time to save. People search by entering keywords or time, but due to the large amount of data, keywords are often repeated. Even if the search is narrowed down, there is still a lot of data. When searching, people can only search by watching related video content. Data The amount is large, the search speed is slow, and time is wasted.

With the development, people began to propose the use of pictures for targeted search of film and television videos, that is, to search for which video belongs to a huge database by randomly obtaining pictures in any video. It takes a long time to compare the frames of each video with the acquired pictures one by one, and has high server requirements, large memory usage, and low applicability.

Therefore, in the prior art, there is still an improvement direction for the above-mentioned image search technology, and it is necessary to propose a image search technology to speed up the image search.

Contents of the invention

In view of the above problems, the present invention provides a method and system for image search video based on image feature extraction, which extracts image features from videos and saves features concisely.

In order to achieve the above object, the concrete technical scheme that the present invention adopts is as follows:

A method of image search based on image feature extraction, the key technology of which is that the specific steps include:

Step 1: After performing feature extraction and compression on each frame of pictures in the M videos, M video feature point sets are obtained;

Step 2: Send M video feature point collections into the compressed video database, and obtain cluster clusters and cluster sequence sets of M video feature points through clustering operations;

Step 3: The picture acquisition terminal acquires the picture to be queried, and obtains the feature sequence of the picture to be queried after the picture feature extraction module; and sends it to the compressed video database for clustering, and obtains the cluster sequence of the picture to be queried;

Step 4: Compare the image cluster sequence to be queried with the cluster sequence set of M video feature points to obtain the sequence similarity; and extract the video feature point set and frame feature sequence corresponding to the highest sequence similarity of the image cluster sequence to be queried ;

Step 5: According to the video feature point set and the index of the frame feature sequence, extract the frame picture in the corresponding video, and compare the frame picture with the picture to be queried to obtain the picture similarity;

Step 6: If the picture similarity is greater than the picture similarity threshold, output the corresponding video or the index of the corresponding video.

Through the above design, after feature extraction and compression of M videos, M video feature point sets are correspondingly obtained. And in the compressed video database, a clustering cluster and a cluster sequence set of M video feature points corresponding to the M videos are formed by clustering. When any picture is obtained, after feature extraction and clustering, the cluster sequence of the picture to be queried is obtained. By comparing the cluster sequence of the picture to be queried with the cluster sequence set of M video feature points, the M video feature points Find the frame picture with the maximum similarity and the index of the corresponding video in the cluster sequence set of . By comparing the frame pictures that meet the requirements with the actual pictures to be queried, the accuracy of the found videos is further confirmed and the search accuracy is improved. By designing a new module to realize video feature extraction and compression, the memory usage of video will be reduced, and the features of continuous frame similarity in video will be used to delete frame pictures with large similarity, and the frames in video will be greatly reduced. And combining the uniqueness of the picture feature sequence, the feature sequence extracted from each picture is different, after feature extraction, the memory of the reserved video feature point set is further reduced. Wherein, M is a positive integer.

Further, in step 1, after implementing feature extraction and compression for each frame of pictures in the M videos, the specific steps for obtaining M video feature point sets are:

Step 11: The video acquisition end sequentially acquires a section of video Ax from M video feature points, and stores it in the storage module to be extracted;

Step 12: The storage module to be extracted obtains the total number of frames N of the video Ax, and sends each frame in the video Ax to the feature extraction and comparison module in turn;

Step 13: The feature extraction and comparison module is used to specify the reference frame, and perform preprocessing, image feature extraction, comparison, screening, deletion, and compression on the video Ax according to the reference frame to obtain a set of video feature points. The specific content is:

S131: The feature extraction and comparison module obtains the first frame, makes the first frame a reference frame, sorts the extracted features after extracting the image features of the first frame, obtains a reference frame feature sequence, and saves the reference frame feature sequence in In the feature sequence storage module;

S132: The feature extraction and comparison module obtains the next frame, and makes the frame the frame to be compared, extracts the image features of the frame to be compared, and sorts the extracted features to obtain a feature sequence of the frame to be compared;

S133: Compare the feature sequence of the frame to be compared with the feature sequence of the reference frame;

If the similarity between the two is greater than or equal to the set similarity threshold, discard the feature sequence of the frame to be compared and enter step S134;

If the similarity is less than the set similarity threshold, then make the feature sequence of the frame to be compared a new reference frame feature sequence, that is: let the corresponding frame to be compared be a reference frame; and save the new reference frame feature sequence in sequence In the feature sequence storage module, enter step S134;

S134: Determine whether the Nth frame in the video Ax has been compared; if so, the feature sequence storage module forms a video feature point set with all reference frame feature sequences preserved, and outputs the video feature point set of the video Ax; otherwise, return to step S132 ;

Step 14: Determine whether all the M videos have been compressed, if so, end step 1, otherwise return to step 11.

Since the video is composed of frames arranged in sequence, and the process of forming an animation, the image similarity of adjacent frames is large. Through the above design, in order to simplify and compress the video database, extract the image features of each frame of the video, compare the adjacent frames one by one, delete the repeated frames, and finally retain the repetition rate. Low set of video feature points. When performing the image search function, after extracting the image features of the image, you can quickly find the corresponding video feature point set. Compared with the existing technology, the process cycle of image search is short, which is convenient for popularization and use. The time position of the picture appearing in the video can be more accurately positioned.

When the feature extraction and comparison module acquires a picture of any frame, it needs to preprocess the picture first, wherein the preprocessing content includes: picture size reset processing and grayscale processing, wherein the picture size reset process is about to process the picture Reset to a uniform size, where the picture size can be customized according to the system. Among them, the grayscale value for grayscale processing is also customized, and the grayscale value ranges from 0 to 255.

Among them, the similarity threshold is custom-set. After comparing the feature sequence of the frame to be compared with the feature sequence of the reference frame, if the similarity is greater than the similarity threshold, it is considered that only the reference frame needs to be saved for the two frames, and so on. After video feature extraction and saving only part of the feature sequence, the video occupation capacity used for image search is greatly reduced without affecting the realization of image search, and the search speed is accelerated.

Where x in Ax is a positive integer.

Furthermore, in step 2, the specific steps for obtaining cluster clusters by clustering the M video feature point sets are as follows:

Step 21a: randomly select k features from M video feature point sets as the cluster center of the initial cluster;

Step 21b: Calculate the clustering distances from all the features in the M video feature point sets to k features as cluster centers, and set the minimum clustering distance to divide the features;

Step 21c: Select the cluster whose cluster center changes, and calculate the mean value of the cluster distance of the corresponding cluster, and determine the cluster center of the cluster according to the mean value of the cluster distance,

Step 21d: If the cluster center does not change any more, then terminate the clustering, output the cluster cluster formed by k clusters and the corresponding cluster center, otherwise return to step 21b;

In step 2, the specific steps of obtaining the cluster sequence set of M video feature points through the clustering operation of the M video feature point sets are: numbering the k cluster clusters in the cluster cluster groups;

Calculate the clustering distances from each feature to k clusters in the M video feature point sets in turn;

Sort the k clustering distances of each feature, and classify the feature into the cluster corresponding to the minimum clustering distance, and obtain the cluster number corresponding to the feature, until all the video feature points in each video feature point set After the features are classified, M cluster sequence sets are obtained.

Through the clustering analysis algorithm, the features of the video are classified, and the unique cluster clusters of the video features are obtained, and the features of M videos are classified. When comparing the images to be queried, a comparison basis is provided. After extracting the features in the picture to be queried, put them into the clustering clusters to classify each feature, and compare them one by one with the M cluster sequence sets corresponding to the M videos, and compare whether there are any similarities in the M cluster sequence sets. The image features to be queried are classified into the same sequence. Through the clustering algorithm, the features of the pictures are clustered, which is convenient for feature similarity comparison, and provides a comparison basis for image search.

To further describe, in step 3, the picture acquisition terminal acquires the picture to be queried, and obtains the feature sequence of the picture to be queried after feature extraction; and sends it to the compressed video database for clustering, and the specific steps to obtain the cluster sequence of the picture to be queried are:

Step 31: After the image acquisition terminal obtains the image to be queried, it sends the image to be queried to the image feature extraction module;

Step 32: the picture feature extraction module combines the third-party visual library to extract the picture feature sequence to be queried and send it to the compressed video database;

Step 33: Calculate the clustering distances of all the features in the feature sequence of the picture to be queried, and classify the feature into the cluster corresponding to the minimum clustering distance, and obtain the cluster sequence of the picture to be queried corresponding to the feature sequence of the picture to be queried.

Using the above scheme, feature extraction and classification are performed on the image to be queried, and it is used for comparison with M cluster sequence sets to find out the corresponding frame image and source video.

To further describe, the feature extraction and comparison module and the picture feature extraction module are all connected to a third-party visual library; the feature extraction and comparison module is at least provided with a frame sending unit and a feature sequence receiving unit, and the frame sending unit is used for The frame to be compared is sent to the third-party visual library for image feature extraction of the frame to be compared, and the obtained frame feature sequence to be compared is fed back to the feature sequence receiving unit; the picture feature extraction module is provided with a picture sending unit and a frame to be compared. The query picture feature sequence receiving unit, the picture sending unit is used to send the picture to be queried to the third-party visual library for feature extraction, and feed back the obtained picture feature sequence to be queried to the picture feature sequence receiving unit to be queried.

The third-party vision library is an open source library for computer vision, which can be OpenCV, JavaCV, Torch3Vision, ImLab, CIMG, Generic Image Library (GIL)-boost integration, etc.

A further technical solution is: the third-party vision library is an OpenCV vision library.

OpenCV is an open source computer vision application platform. The project was initiated by the Russian team of Intel Corporation's R&D Center. It has an open source BSD certificate. The goal of OpenCV is to achieve real-time computer vision. It is a cross-platform computer vision library. It has developed rapidly since the day it was developed, and has received strong support and contributions from many companies and industry leaders. Because it is a BSD open source license, it can be used in scientific research and commercial applications for free. As a powerful computer vision open source library, OpenCV largely refers to the implementation details and style of MatLab. For example, after OpenCV2. All are exactly the same (such as imread, imshow, imwriter, etc.). This approach not only shortens the distance between product development and academic research, but also greatly improves the R&D efficiency of developers. It has to be said that Intel is really a great company. In computer memory, digital images are stored and operated in the form of matrices. For example, in MatLab, an image corresponds to a matrix after being read, and the same is true in OpenCV. In the early versions of OpenCV1.x, image processing was implemented through the structure of IplImage (the name comes from Intel Image Processing Library, another open source library of Intel, abbreviated as IplImage). The early OpenCV was written in C language, so the interface provided is also a C language interface, and its source code is completely C programming style. The IplImage structure is the basic data structure for OpenCV matrix operations. Up to the OpenCV2.x version, the OpenCV open source library introduced the idea of object-oriented programming, and a large number of source codes were rewritten in C++. The Mat class (abbreviation for Matrix) is a package class introduced by OpenCV for image processing. In terms of function, the Mat class is further enhanced on the basis of the IplImage structure, and, due to the introduction of C++ advanced programming features, the scalability of the Mat class is greatly improved, and the content of the Mat class is continuously enriched in later versions. By viewing the Mat class Definition, you will find that its design and implementation is very comprehensive and specific, basically covering the basic requirements of computer vision for image processing.

OpenCV already includes the following application domain functions: 2D and 3D feature toolbox, motion estimation, face recognition system, pose recognition, human-computer interaction, mobile robot, motion understanding, object identification, segmentation and recognition, stereo vision, motion tracking , augmented reality (AR technology).

Based on the above function realization needs, OpenCV also includes the following statistical machine learning libraries: Boosting algorithm, Decision Tree (decision tree) learning, Gradient Boosting algorithm, EM algorithm (expectation maximization), KNN algorithm, naive Bayesian classification, Artificial neural network, random forest, palm vector machine.

Most of the modules in OpenCV are implemented based on C++, and a small part of them are implemented based on C language. The current SDK provided by OpenCV already supports C++, Java, Python and other language application development. Currently, the newly developed algorithms and module interfaces of OpenCV itself are all based on C++. OpenCV supports application development on almost all mainstream OS systems, including Windows, Mac, Linux, FreeBSD, OpenBSD, etc. The mobile platform supports Android, IOS, BlackBerray and other platforms. Users can obtain related SDK downloads, development documents and environment configuration information from the OpenCV official website. Since the release of version 1.0, OpenCV has immediately attracted the attention of many companies and has been widely used in product development and innovation in many fields. Related applications include satellite map and electronic map stitching, image noise processing in medicine, object detection, and security monitoring in the field of safety and security. Intrusion detection, automatic monitoring and alarm, product quality inspection in manufacturing and industry, camera calibration. UAV flight in the military field, unmanned driving and underwater robots and many other fields.

A further technical solution is: the feature extraction and comparison module is provided with a feature point extraction tool, and the feature point extraction tool is provided with a video reading function block, a frame fetching function block, an AKAZE algorithm function block, and feature point normalization Function block, write file function block;

The feature extraction and comparison module is also provided with video feature parameters, wherein the video feature parameters include the similarity threshold, video input path, and video output path.

Through the above solution, the feature extraction and comparison module is used to realize functions such as video reading, frame reading, frame picture sending, and feature receiving, and realize source video search through paths.

Furthermore, the unique feature attributes of frame features in the frame feature sequence include feature abscissa, feature ordinate, feature angle, feature size, feature weight, feature extension, and feature pyramid layer; The same features of attributes such as angles and different enlarged sizes are extracted. The feature extraction is not easy to lose, the extraction integrity is good, and it is convenient for different angles.

Still further, all reference frame feature sequences in the video feature point set are arranged in sequence according to the order of preservation. When the feature sequence of the frame to be compared is compared with the feature sequence of the reference frame, and the similarity is less than the set similarity threshold, the feature sequence of the frame to be compared is stored in the feature sequence storage module and arranged behind the feature sequence of the previous reference frame. After using the final saved video feature point set, through any feature sequence of unknown position, after comparison, the approximate position where the feature sequence appears can be obtained for positioning.

Furthermore, any frame feature sequence or image feature sequence to be queried all includes sequentially connected frame number, feature sequence start label, feature sequence content, and feature sequence end label; in order to mark each frame, set frame number, feature Sequence start number, feature sequence end number, each frame can be distinguished by marking, and its start and end can be digitally identified and marked.

Still further, the start label of the feature sequence is composed of an integer of X bytes; the end label of the feature sequence is composed of an integer of Y bytes; the starting point of the video feature point set is connected with a start label; The reference frame feature sequences are connected by feature connectors.

That is, for video A1, the starting label of its video feature point set is: video A1 feature sequence.

A system for searching video by image based on image feature extraction, its key technology is: comprising a video acquisition terminal and a picture acquisition terminal; wherein, the video acquisition terminal is connected to a storage module to be extracted, and the storage module to be extracted is connected to the Connected with the comparison module, a frame sending unit and a feature sequence receiving unit are arranged in the feature extraction and comparison module, and a third-party visual library connection terminal is provided on the feature extraction and comparison module, and the third-party visual library connection terminal is used for all The frame sending unit and the feature sequence receiving unit are respectively connected with a third-party visual library, and a feature sequence storage module is also connected to the feature extraction and comparison module, and the feature sequence storage module is connected with a compressed video database, and the compressed video database Connect with the image search module;

The picture acquisition terminal is connected with the compressed video database through the picture feature extraction module, and a third-party visual library connection terminal is arranged on the picture feature extraction module, which is used to extract the picture feature sequence to be queried, and the picture acquisition terminal also It is connected with the image search module, and the image search module is connected with the source video library.

Through the above system, the acquisition, preservation, generation of feature sequences, video frame extraction, and video frame deletion functions of the source video are realized, and the most streamlined set of video feature points is output, and the clustering of video features is formed by compressing the video database. group, combined with the clustering cluster group to obtain a cluster sequence set of M video feature points. And through the above system to achieve picture acquisition, picture feature extraction and classification, after comparing with the cluster sequence set of M video feature points, classify and compare the results, obtain pictures with high similarity, combine the pictures to be queried and frame pictures After the actual pictures are compared, the video search results are further determined, so as to obtain the corresponding video or the location where the video is saved in combination with the source video index in the compressed video database, so as to realize fast and accurate video search.

Still further, the compressed video database is provided with a clustering unit and a data unit;

The clustering unit is used to perform a clustering operation on M sets of video feature points to obtain a cluster sequence set of cluster clusters and M video feature points; and cluster all the features in the feature sequence of the picture to be queried, Obtain the image cluster sequence to be queried;

M video feature point sets, cluster sequence sets of M video feature points, source video index of each video, any picture feature sequence to be queried and picture cluster sequence to be queried are stored in the data unit; The video search module is provided with a clustering comparison unit, a picture acquisition unit, a frame picture acquisition unit, and a source video acquisition unit;

The cluster comparison unit is used to compare the cluster sequence of the picture to be queried with the set of cluster sequences of M video feature points to obtain sequence similarity; the picture acquisition unit is used to obtain the picture to be queried; the frame picture The obtaining unit is used to obtain the frame picture corresponding to the highest sequence similarity;

The source video obtaining unit is used to obtain the source video corresponding to the highest sequence similarity.

Through the above solution, the clustering unit performs a clustering operation on each feature in the M videos, and classifies each feature. The data unit is used to save the feature data, feature classification data of all videos and the corresponding index of each video in the source video. When searching for images by images, the obtained pictures are classified and compared through the clustering comparison unit, and the most similar sequences are found from the cluster sequence sets of M video feature points.

The beneficial effect of the present invention is to use the image feature extraction technology to perform feature extraction and comparison of each frame in the video, and to eliminate repeated frame feature sequences. Realize low memory usage, due to video compression, the amount of data is reduced. When performing image search, based on the clustering clusters obtained by the clustering algorithm, the acquired pictures are classified, and the similarity is determined through the classification results. Compare the process blocks, and locate the approximate frame number where the picture appears in the video. Compared with the prior art, the invention can speed up the retrieval speed and has accurate positioning.

Description of drawings

Fig. 1 is a system structure block diagram of the present invention;

Fig. 2 is a flow chart of the method for image search with pictures of the present invention;

Fig. 3 is the flow chart of the equivalent video compression storage method of the present invention;

Figure 4 is an architecture diagram of the MADlib open source machine learning library;

Figure 5 is a list of detected videos;

Fig. 6 is an instantaneous state diagram of an equivalent video compression operation.

Figure 7 is a picture to be queried;

Fig. 8 is a schematic diagram of the original frame image search results;

Fig. 9 is a schematic diagram of the image search result of the frame image with cropping size 1;

FIG. 10 is a schematic diagram of image search results for frame images with a cropping size of 2. FIG.

Detailed ways

The specific implementation manner and working principle of the present invention will be further described in detail below in conjunction with the accompanying drawings.

Combining Figure 1 and Figure 2, it can be seen that an image search method based on image feature extraction, the specific steps include: Step 1: After performing feature extraction and compression on each frame of M videos, M videos are obtained. Video feature point set; Step 2: Send M video feature point sets into the compressed video database, and obtain cluster clusters and M video feature point cluster sequence sets through clustering operations; Step 3: The image acquisition terminal treats the query image Acquisition, after the image feature extraction module, the feature sequence of the picture to be queried is obtained; and sent to the compressed video database for clustering, to obtain the cluster sequence of the picture to be queried; step 4: the cluster sequence of the picture to be queried and the cluster of M video feature points Compare the sequence sets to obtain the sequence similarity; and extract the video feature point set and frame feature sequence corresponding to the highest sequence similarity of the picture cluster sequence to be queried; Step 5: According to the index of the video feature point set and frame feature sequence, extract A frame picture in the corresponding video, and compare the frame picture with the picture to be queried to obtain the picture similarity;

Step 6: If the picture similarity is greater than the picture similarity threshold, output the corresponding video or the index of the corresponding video.

In this embodiment, a sequence similarity threshold is also set, and the sequence similarity threshold is 80%. If the highest sequence similarity is lower than the threshold, the next step will not be performed and it will be displayed that no relevant content has been retrieved. When the highest sequence similarity is greater than or equal to the threshold, extract the video feature point set and frame feature sequence corresponding to the highest sequence similarity of the image cluster sequence to be queried. In this embodiment, the picture similarity threshold is 90%. It can be seen from Fig. 3 that in step 1, after feature extraction and compression of each frame of pictures in M videos, the specific steps to obtain M video feature point sets are as follows:

Step 11: The video acquisition terminal obtains a section of video Ax from M video feature points in turn, and stores it in the storage module to be extracted; Step 12: The storage module to be extracted obtains the total number of frames N of the video Ax, and saves each video Ax in the video Ax One frame is sent to the feature extraction and comparison module in turn; Step 13: The feature extraction and comparison module is used to specify the reference frame, and preprocess the video Ax according to the reference frame, extract image features, compare, filter, delete, and compress to obtain A collection of video feature points, the specific content is:

S131: The feature extraction and comparison module obtains the first frame, makes the first frame a reference frame, sorts the extracted features after extracting the image features of the first frame, obtains a reference frame feature sequence, and saves the reference frame feature sequence in In the feature sequence storage module; S132: the feature extraction and comparison module obtains the next frame, and makes the frame a frame to be compared, extracts the image features of the frame to be compared, and sorts the extracted features to obtain a feature sequence of the frame to be compared;

S133: Compare the feature sequence of the frame to be compared with the feature sequence of the reference frame;

If the similarity between the two is greater than or equal to the set similarity threshold, then discard the feature sequence of the frame to be compared, and enter step S134; if the similarity is less than the set similarity threshold, then make the feature sequence of the frame to be compared a new benchmark Frame feature sequence, that is: let the corresponding frame to be compared be the reference frame; and store the new reference frame feature sequence in the feature sequence storage module in turn, and enter step S134;

S134: Determine whether the Nth frame in the video Ax has been compared; if so, the feature sequence storage module forms a video feature point set with all reference frame feature sequences preserved, and outputs the video feature point set of the video Ax; otherwise, return to step S132 ;

Step 14: Determine whether all the M videos have been compressed, if so, end step 1, otherwise return to step 11.

In this embodiment, when preprocessing each frame of the picture in the video Ax, after the picture size reset process, the picture is reset to a size of 400×300, and the gray value is set to 50%. In this embodiment, the feature extraction and comparison module is connected with the third-party vision library OpenCV; in this embodiment, the OpenCV vision library is provided with the AKAZE algorithm.

In the present embodiment, in step 2, the concrete steps of obtaining the cluster clusters through the clustering operation of the M video feature point sets are as follows:

Step 21a: Select k features arbitrarily from the M video feature point sets as the cluster center of the initial cluster; Step 21b: Calculate the clustering of all features to k features in the M video feature point sets as the cluster center Class distance, and set the minimum clustering distance to divide the features;

Step 21c: Select the cluster whose cluster center changes, and calculate the mean value of the cluster distance of the corresponding cluster, and determine the cluster center of the cluster according to the mean value of the cluster distance,

Step 21d: If the cluster center does not change any more, then terminate the clustering, output the cluster cluster formed by k clusters and the corresponding cluster center, otherwise return to step 21b;

In step 2, the specific steps of obtaining the cluster sequence set of M video feature points through the clustering operation of the M video feature point sets are: numbering the k cluster clusters in the cluster cluster groups;

Calculate the clustering distances from each feature to k clusters in the M video feature point sets in turn;

Sort the k clustering distances of each feature, and classify the feature into the cluster corresponding to the minimum clustering distance, and obtain the cluster number corresponding to the feature, until all the video feature points in each video feature point set After the features are classified, M cluster sequence sets are obtained.

In this embodiment, image feature clustering is realized by connecting with the MADlib open source machine learning library, and the clustering method adopted is the k-means algorithm.

MADlib is an open source machine learning library jointly developed by Pivotal and the University of Berkeley. It provides a variety of data conversion, data exploration, statistics, data mining and machine learning methods. It can easily analyze and mine structured data. Users can easily load MADlib into the database to expand the analysis function of the database. In July 2015, MADlib became an incubator project of the Apache Software Foundation. After two years of development, it graduated in August 2017 and became a top Apache project. Its current latest version is MADlib 1.15, which can seamlessly integrate with database systems such as PostgreSQL, Greenplum and HAWQ. In this embodiment, a plug-in integrated into PostgreSQL is used in the pre-research.

As can be seen from Figure 4, the MADlib open source machine learning library architecture, the top layer of the architecture is the user interface. As mentioned earlier, users only need to call the functions provided by MADlib in the SQL query statement to complete the data mining work. The SQL syntax here should match the specific database management system. The bottom layer is the database management system such as Greenplum, PostgreSQL, etc., and finally they process the query request.

It can also be seen from Figure 4 that the MADlib system architecture consists of the following four main components from top to bottom: the driver function implemented by Python calling the SQL template, the high-level abstraction layer implemented by Python, the core function implemented by C++, and the low-level database implemented by C++ abstraction layer.

The main design idea driving the MADlib architecture is consistent with Hadoop, reflected in:

Manipulate local data within the database without unnecessary data movement across multiple runtime environments.

Take full advantage of database engine capabilities, but separate data mining logic from database-specific implementation details. Take advantage of the parallelism and scalability provided by MPP shared-nothing technologies, such as Greenplum or HAWQ database systems. The maintenance activities performed are open to the Apache community and ongoing academic research.

Among them, the basic idea of the k-means algorithm is: the basic idea of the k-means clustering method is: divide a given set of N data records into K groups, and each group represents a cluster. K<N. And these K groups meet the following conditions: each group contains at least one data record. Each data record belongs to one and only one group.

The algorithm first gives an initial grouping, and then changes the grouping through repeated iterations, so that the grouping scheme after each improvement is better than the previous one, and the so-called good standard is: the closer the objects in the same grouping, the better ( has converged, repeated iterations until there is almost no difference in the data within the group), and the farther the distance between objects in different groups, the better.

The working principle of the k-means algorithm is:

First randomly select K points from the data set, each point initially represents the center of each cluster, then calculate the distance from the remaining samples to the center point, assign it to the nearest cluster, and then recalculate the average value of each cluster As a new center point, the whole process is repeated continuously. If there is no significant change in the two adjacent adjustments, it means that the cluster formed by data clustering has converged. A characteristic of this algorithm is to examine whether the classification of each sample is correct in each iteration. If it is not correct, it needs to be adjusted. After all the samples are adjusted, the center point is modified to enter the next iteration. This process is repeated until a certain termination condition is met, which can be any of the following: No objects are reassigned to a different cluster. The cluster centers no longer change.

The sum of squared errors is a local minimum.

The k-means algorithm is a typical distance-based clustering algorithm, which uses distance as the evaluation index of similarity, that is, the closer the distance between two objects, the greater the similarity. The algorithm considers that clusters are composed of close objects, so the final goal is to obtain compact and independent clusters. The input of the k-means algorithm is the number of clusters k and n data objects, and the output is k clusters that meet the minimum error standard. The processing flow is as follows: randomly select k objects from n data objects as initial centers.

Calculate the distance of each object from these central objects and divide the corresponding objects according to the minimum distance. Recalculate the mean of each changed cluster as the new center.

Loop 2, 3 until each cluster no longer changes. The termination condition is generally to minimize the sum of the squares of the distances from the object to its cluster center: the collection category is greater than but not limited to the following types of TV programs:

During the implementation process, firstly, we collect classified films, so that the classification of ethnic groups can be as standard, objective and reasonable as possible, and the classification of ethnic groups produced later can be better distinguished as much as possible.

The collection categories are greater than but not limited to the following types of TV programs: news, variety shows, film and television dramas, sports events, scenery, animals, foreign blockbusters, domestic blockbusters, war films, indoor dramas...

In this embodiment, in step 3, the picture acquisition terminal acquires the picture to be queried, and obtains the feature sequence of the picture to be queried after feature extraction; and sends it to the compressed video database for clustering, and the specific steps for obtaining the picture cluster sequence to be queried are as follows: :

Step 31: After the image acquisition terminal obtains the image to be queried, it sends the image to be queried to the image feature extraction module;

Step 32: the picture feature extraction module combines the third-party visual library to extract the picture feature sequence to be queried and send it to the compressed video database;

Step 33: Calculate the clustering distances of all the features in the feature sequence of the picture to be queried, and classify the feature into the cluster corresponding to the minimum clustering distance, and obtain the cluster sequence of the picture to be queried corresponding to the feature sequence of the picture to be queried.

A further technical solution is: the feature extraction and comparison module and the picture feature extraction module are all connected to a third-party visual library; the feature extraction and comparison module is at least provided with a frame sending unit and a feature sequence receiving unit, and the frame sending unit is used to The frame to be compared is sent to the third-party visual library for image feature extraction of the frame to be compared, and the obtained frame to be compared feature sequence is fed back to the feature sequence receiving unit;

The picture feature extraction module is provided with a picture sending unit and a picture feature sequence receiving unit to be queried, and the picture sending unit is used to send the picture to be queried to the third-party visual library for feature extraction, and obtain the feature of the picture to be queried The sequence is fed back to the image feature sequence receiving unit to be queried.

In this embodiment, the feature extraction and comparison module is provided with a feature point extraction tool, and the feature point extraction tool is provided with a video reading function block, a frame-taking function block, an AKAZE algorithm function block, and a feature point normalization function block, write file function block;

The feature extraction and comparison module is also provided with video feature parameters, wherein the video feature parameters include the similarity threshold, video input path, and video output path. Among them, the written shell script is used to call the feature point extraction tool in turn, and the feature point comparison and input and output are performed according to the feature point extraction tool combined with a similarity threshold of 90%, video input path, and video output path. During the feature extraction, the operating environment is in the docker container, and combined with the written shell script, the feature point extraction tool is sequentially invoked to start feature extraction.

In this embodiment, the third-party vision library is the OpenCV vision library. In this embodiment, the AKAZE algorithm is set in the OpenCV vision library. In this embodiment, the extracted feature points are output as a csv file in a specified format; and then packaged into a jar file.

In this embodiment, the unique feature attributes of frame features in the frame feature sequence include feature abscissa, feature ordinate, feature angle, feature size, feature weight, feature extension, and feature pyramid layer;

All reference frame feature sequences in the video feature point set are arranged in sequence according to the order of preservation;

Any frame feature sequence or image feature sequence to be queried includes sequentially connected frame number, feature sequence start label, feature sequence content, and feature sequence end label;

The start number of the characteristic sequence is composed of an integer of X bytes; the end number of the characteristic sequence is composed of an integer of Y bytes; in this embodiment, X=Y=4.

The starting points of the set of video feature points are connected with a start label; two pairs of the reference frame feature sequences are connected by feature connectors.

Then the feature sequence formed by the video Ax is:

Video Ax feature sequence + feature connector + frame number of the first frame + start label of the first frame feature sequence + content of the first frame feature sequence + end label of the first frame feature sequence + feature connector + frame number of the i1th frame + The start label of the i1th frame feature sequence + the i1th frame feature sequence content + the i1th frame feature sequence end label + feature connector + the frame number of the i2th frame + the i2th frame feature sequence start label + the i2th frame feature sequence content + the i2th frame feature sequence content + the i2th frame i2 frame feature sequence end label + ... where i1 and i2 are integers greater than 1, and i1>i2.

In this embodiment, the video feature point compression package is a storage file of a binary sequence that has undergone a gzip compression process. Combining with Figure 1, it can be seen that an image search system based on image feature extraction includes a video acquisition terminal and an image acquisition terminal;

Wherein, the video acquisition terminal is connected with the storage module to be extracted, the storage module to be extracted is connected with the feature extraction and comparison module, and a frame sending unit and a feature sequence receiving unit are arranged in the feature extraction and comparison module, and the feature extraction The comparison module is provided with a third-party visual library connection end, which is used for the frame sending unit and the feature sequence receiving unit to connect with the third-party visual library respectively, and on the feature extraction and comparison module. A feature sequence storage module is connected, and the feature sequence storage module is connected with a compressed video database, and the compressed video database is connected with a picture search module;

The picture acquisition terminal is connected with the compressed video database through the picture feature extraction module, and a third-party visual library connection terminal is arranged on the picture feature extraction module, which is used to extract the picture feature sequence to be queried. The video module is connected, and the image search module is connected with the source video library.

It can also be seen in conjunction with Fig. 1 that a clustering unit and a data unit are arranged in the compressed video database;

The clustering unit is used to perform a clustering operation on M sets of video feature points to obtain a cluster sequence set of cluster clusters and M video feature points; and cluster all the features in the feature sequence of the picture to be queried, Obtain the image cluster sequence to be queried;

In the data unit, there are M video feature point sets, M video feature point cluster sequence sets, the source video index of each video, any picture feature sequence to be queried, and the picture cluster sequence to be queried;

The described image search module is provided with a cluster comparison unit, a picture acquisition unit, a frame picture acquisition unit, and a source video acquisition unit; the cluster comparison unit is used to compare the picture cluster sequence to be queried with M video features The cluster sequence sets of points are compared to obtain the sequence similarity;

The picture acquisition unit is used to acquire the picture to be queried;

The frame picture acquisition unit is used to acquire the frame picture corresponding to the highest sequence similarity;

The source video obtaining unit is used to obtain the source video corresponding to the highest sequence similarity.

To further illustrate this embodiment, 18 videos from the TV program "Speak Out Loud" and any two small videos obtained randomly are used as source videos.

Among them, the 18 video download addresses of "Speak Out Loud" are: http://qjcq.cbg.cn/dsscl/1.shtml.

Using crawler technology to collect and download some of the videos on the first page of the above-mentioned webpages, a total of 18 videos. Among them, most of the videos are about 30 minutes long, one is 45 minutes long, and the other is 1 hour long. The other two videos are about 3-5 minutes long. There are 20 videos in total. See attached picture 5 for the video list diagram.

Combined with Figure 3, naming instructions: 170101001 and 170101002 are short videos of 3-5 minutes, and the rest of the names are composed of the broadcast time + number of a certain episode of "Speak Out Loud". For example, 170524001 represents a program "Suspicion" broadcast on May 24, 2017 according to the natural order. The rest of the names are in accordance with the above rules. The total size of the above video files is: 2.49GB

The equivalent video compression storage system is installed in a host computer with a machine configuration of 2U, 4 cores, 16 threads in total, and 32G memory. The CPU is used for extraction, and the computing power is limited, and the average load during the operation is 30%+. See Figure 6 for details on the instantaneous state of the operation. After extracting feature points from the entire video, the resulting csv file takes up 1.4GB of disk space. It is then packaged into a tar.gz file, and the disk space occupied by the finally obtained video feature point compression package is 480MB.

As can be seen from Figure 7, it is a schematic diagram of pictures to be inspected. According to the infringement characteristics of the infringing videos in reality: most of them cut part of the source video and even add their own logo information. However, it is relatively rare to use the source video information as a part of the infringing video screen. In the embodiment of this city, the test process is mainly to perform verification on a certain frame in the video: original frame interception: test01; cropping size 1: test02; cropping size 2: test05.

The verification results are shown in Figure 8-10. After the above tests, we can see that as the cropping size increases, fewer and fewer results can be found. Because we considered the infringement features of the infringing video when calculating the feature points, we also cropped the source video and then extracted the feature points. The current version used for the above display is 3 minutes * 2 segments * 4 zoom scales * 80% frames developed and tested (total 4W frames), the maximum compatible cropping top, bottom, left, and right dimensions are 10% 10% 20%.

It should be noted that the above description is not intended to limit the present invention, and the present invention is not limited to the above-mentioned examples. Those skilled in the art may make changes, modifications, additions or replacements within the scope of the present invention. It should also belong to the protection scope of the present invention.

Claims (9)

1. A picture searching method based on image feature extraction is characterized by comprising the following specific steps:

step 1: extracting and compressing the characteristics of each frame of picture in M videos to obtain M video characteristic point sets;

step 2: sending the M video feature point sets into a compressed video database, and obtaining a clustering cluster and a cluster sequence set of the M video feature points through clustering operation;

step 3: the picture obtaining end obtains a picture to be queried, and a picture feature sequence to be queried is obtained after the picture feature extraction module; sending the image clusters into a compressed video database for clustering to obtain a to-be-queried image cluster sequence;

step 4: comparing the image cluster sequence to be queried with a cluster sequence set of M video feature points to obtain a sequence similarity; extracting a video characteristic point set and a frame characteristic sequence which correspond to the sequence similarity of the picture cluster sequence to be queried, wherein the video characteristic point set corresponds to the highest sequence similarity of the picture cluster sequence to be queried;

step 5: extracting a frame picture in a corresponding video according to the video feature point set and the index of the frame feature sequence, and comparing the frame picture with a picture to be queried to obtain a picture similarity;

step 6: if the picture similarity is greater than the picture similarity threshold, outputting the corresponding video or the index of the corresponding video;

In step 1, the specific steps of obtaining M video feature point sets after feature extraction and compression are performed on each frame of picture in M videos are as follows:

step 11: the video acquisition end sequentially acquires a section of video Ax from M video feature points and stores the video Ax in a storage module to be extracted;

step 12: the storage module to be extracted obtains the total frame number N of the video Ax, and each frame in the video Ax is sequentially sent to the feature extraction and comparison module;

step 13: the feature extraction and comparison module is used for designating a reference frame, preprocessing the video Ax according to the reference frame, extracting, comparing, screening, deleting and compressing the image features to obtain a video feature point set, wherein the specific content is as follows:

s131: the feature extraction and comparison module acquires a first frame, enables the first frame to be a reference frame, sequences the extracted features after extracting the features of the first frame image to obtain a reference frame feature sequence, and stores the reference frame feature sequence in the feature sequence storage module;

s132: the feature extraction and comparison module obtains the next frame, enables the frame to be a frame to be compared, extracts the image features of the frame to be compared, and then sorts the extracted features to obtain a frame feature sequence to be compared;

S133: comparing the frame characteristic sequence to be compared with the reference frame characteristic sequence;

if the similarity is greater than or equal to the set similarity threshold, discarding the feature sequence of the frame to be compared, and entering step S134;

if the similarity is smaller than the set similarity threshold, the frame feature sequence to be compared is made to be a new reference frame feature sequence, namely: the corresponding frames to be compared are made to be reference frames; sequentially storing the new reference frame characteristic sequences in a characteristic sequence storage module, and entering step S134;

s134: judging whether the Nth frame in the video Ax is completely compared; if yes, the feature sequence storage module forms all the stored reference frame feature sequences into a video feature point set, and outputs the video feature point set of the video Ax; otherwise, returning to the step S132;

step 14: and judging whether all M videos are compressed, if yes, ending the step 1, otherwise, returning to the step 11.

2. The image search method based on image feature extraction of claim 1, wherein: in the step 2, the specific step of obtaining a clustering cluster group by clustering M video feature point sets is as follows:

step 21a: randomly selecting k features from the M video feature point sets as the clustering centers of the initial clusters;

Step 21b: respectively calculating the clustering distances from all the features to k features in the M video feature point sets as the centers of the clustering clusters, and setting the minimum clustering distance to divide the features;

step 21c: selecting a cluster with a changed cluster center, calculating a cluster distance average value of the corresponding cluster, determining the cluster center of the cluster according to the cluster distance average value,

step 21d: if the cluster center does not change any more, ending the clustering, outputting a cluster group formed by the k cluster groups and the corresponding cluster center, otherwise, returning to the step 21b;

in the step 2, the specific steps of obtaining the cluster sequence set of the M video feature points through clustering operation by the M video feature point sets are as follows:

numbering k cluster groups in the cluster group;

calculating the clustering distance from each feature to k clustering clusters in M video feature point sets in sequence;

and sequencing k clustering distances of each feature, classifying the feature into a cluster corresponding to the minimum clustering distance, and acquiring a cluster number corresponding to the feature until all the features in each video feature point set are classified, so as to obtain M cluster sequence sets.

3. The image searching method based on image feature extraction as claimed in claim 2, wherein in step 3, the image obtaining end obtains the image to be queried, and the feature sequence of the image to be queried is obtained after feature extraction; and sending the image clusters into a compressed video database for clustering, wherein the specific steps for obtaining the image cluster sequence to be queried are as follows:

Step 31: after the picture obtaining end obtains the picture to be queried, the picture to be queried is sent to a picture feature extraction module;

step 32: the picture feature extraction module is combined with a third party visual library, extracts a picture feature sequence to be queried and sends the picture feature sequence to the compressed video database;

step 33: and calculating all feature clustering distances in the feature sequence of the picture to be queried, classifying the features into the cluster corresponding to the minimum clustering distance, and obtaining the picture cluster sequence to be queried corresponding to the feature sequence of the picture to be queried.

4. The image search method based on image feature extraction of claim 1, wherein: the feature extraction and comparison module and the picture feature extraction module are connected with a third-party visual library;

the characteristic extraction and comparison module is at least provided with a frame sending unit and a characteristic sequence receiving unit, wherein the frame sending unit is used for sending the frame to be compared to the third-party visual library to extract the characteristic of the image of the frame to be compared, and feeding the obtained characteristic sequence of the frame to be compared back to the characteristic sequence receiving unit;

the image feature extraction module is provided with an image sending unit and an image feature sequence receiving unit to be queried, and the image sending unit is used for sending the image to be queried to the third-party visual library for feature extraction, and feeding the obtained image feature sequence to be queried back to the image feature sequence receiving unit to be queried.

5. The image search method based on image feature extraction of claim 4, wherein: the feature extraction and comparison module is provided with a feature point extraction tool, and the feature point extraction tool is provided with a video reading function block, a frame taking function block, an AKAZE algorithm calling function block, a feature point normalization function block and a file writing function block;

and the feature extraction and comparison module is also internally provided with video feature parameters, wherein the video feature parameters comprise the similarity threshold value, a video input path and a video output path.

6. The image search method based on image feature extraction of claim 1, wherein: the third party visual library is an OpenCV visual library.

7. The image search method based on image feature extraction of claim 1, wherein: the unique characteristic attributes of the frame characteristics in the frame characteristic sequence comprise a characteristic abscissa, a characteristic ordinate, a characteristic angle, a characteristic size, a characteristic weight, a characteristic expansion and a characteristic pyramid layer;

all the reference frame feature sequences in the video feature point set are sequentially arranged according to the stored sequence;

any frame characteristic sequence or picture characteristic sequence to be inquired comprises a frame sequence number, a characteristic sequence starting label, characteristic sequence content and a characteristic sequence ending label which are connected in sequence;

The feature sequence start label consists of an integer of X bytes; the characteristic sequence end mark consists of an integer of Y bytes;

the video feature point set starting point is connected with a starting mark; and every two reference frame feature sequences are connected through feature connectors.

8. The image search system based on image feature extraction is characterized in that: the method comprises a video acquisition end and a picture acquisition end;

the video acquisition end is connected with a storage module to be extracted, the storage module to be extracted is connected with a feature extraction and comparison module, a frame sending unit and a feature sequence receiving unit are arranged in the feature extraction and comparison module, a third-party visual library connecting end is arranged on the feature extraction and comparison module and used for respectively connecting the frame sending unit and the feature sequence receiving unit with a third-party visual library, a feature sequence storage module is also connected with the feature extraction and comparison module, the feature sequence storage module is connected with a compressed video database, and the compressed video database is connected with a picture searching module;

the image acquisition end is connected with the compressed video database through an image feature extraction module, a third-party visual library connection end is arranged on the image feature extraction module and used for extracting an image feature sequence to be queried, the image acquisition end is also connected with the image searching module, and the image searching module is connected with a source video library;

The specific working steps of the image search system based on image feature extraction are as follows:

step 1: extracting and compressing the characteristics of each frame of picture in M videos to obtain M video characteristic point sets;

step 2: sending the M video feature point sets into a compressed video database, and obtaining a clustering cluster and a cluster sequence set of the M video feature points through clustering operation;

step 3: the picture obtaining end obtains a picture to be queried, and a picture feature sequence to be queried is obtained after the picture feature extraction module; sending the image clusters into a compressed video database for clustering to obtain a to-be-queried image cluster sequence;

step 4: comparing the image cluster sequence to be queried with a cluster sequence set of M video feature points to obtain a sequence similarity; extracting a video characteristic point set and a frame characteristic sequence which correspond to the sequence similarity of the picture cluster sequence to be queried, wherein the video characteristic point set corresponds to the highest sequence similarity of the picture cluster sequence to be queried;

step 5: extracting a frame picture in a corresponding video according to the video feature point set and the index of the frame feature sequence, and comparing the frame picture with a picture to be queried to obtain a picture similarity;

step 6: if the picture similarity is greater than the picture similarity threshold, outputting the corresponding video or the index of the corresponding video;

In step 1, the specific steps of obtaining M video feature point sets after feature extraction and compression are performed on each frame of picture in M videos are as follows:

step 11: the video acquisition end sequentially acquires a section of video Ax from M video feature points and stores the video Ax in a storage module to be extracted;

step 12: the storage module to be extracted obtains the total frame number N of the video Ax, and each frame in the video Ax is sequentially sent to the feature extraction and comparison module;

step 13: the feature extraction and comparison module is used for designating a reference frame, preprocessing the video Ax according to the reference frame, extracting, comparing, screening, deleting and compressing the image features to obtain a video feature point set, wherein the specific content is as follows:

s131: the feature extraction and comparison module acquires a first frame, enables the first frame to be a reference frame, sequences the extracted features after extracting the features of the first frame image to obtain a reference frame feature sequence, and stores the reference frame feature sequence in the feature sequence storage module;

s132: the feature extraction and comparison module obtains the next frame, enables the frame to be a frame to be compared, extracts the image features of the frame to be compared, and then sorts the extracted features to obtain a frame feature sequence to be compared;

S133: comparing the frame characteristic sequence to be compared with the reference frame characteristic sequence;

if the similarity is greater than or equal to the set similarity threshold, discarding the feature sequence of the frame to be compared, and entering step S134;

if the similarity is smaller than the set similarity threshold, the frame feature sequence to be compared is made to be a new reference frame feature sequence, namely: the corresponding frames to be compared are made to be reference frames; sequentially storing the new reference frame characteristic sequences in a characteristic sequence storage module, and entering step S134;

s134: judging whether the Nth frame in the video Ax is completely compared; if yes, the feature sequence storage module forms all the stored reference frame feature sequences into a video feature point set, and outputs the video feature point set of the video Ax; otherwise, returning to the step S132;

step 14: and judging whether all M videos are compressed, if yes, ending the step 1, otherwise, returning to the step 11.

9. The image feature extraction based image search system of claim 8, wherein: a clustering unit and a data unit are arranged in the compressed video database;

the clustering unit is used for carrying out clustering operation on the M video feature point sets to obtain a clustering cluster and a cluster sequence set of the M video feature points; clustering all the features in the feature sequence of the picture to be queried to obtain a cluster sequence of the picture to be queried;

M video feature point sets, M cluster sequence sets of video feature points, a source video index of each video, any picture feature sequence to be queried and any picture cluster sequence to be queried are stored in the data unit;

the image searching module is provided with a cluster comparison unit, a picture acquisition unit, a frame picture acquisition unit and a source video acquisition unit;

the cluster comparison unit is used for comparing the image cluster sequence to be queried with a cluster sequence set of M video feature points to obtain a sequence similarity;

the picture acquisition unit is used for acquiring a picture to be queried;

the frame picture acquisition unit is used for acquiring a frame picture corresponding to the highest sequence similarity;

the source video acquisition unit is used for acquiring a source video corresponding to the highest sequence similarity.

Images