CN111639359A - Method and system for detecting and early warning privacy risks of social network pictures - Google Patents

Abstract

A method for risk detection and early warning of image privacy in social networks, comprising: step 1: extracting key elements in the image by using a target detection framework and obtaining information of the key elements; step 2: collecting a data set of whether the image is private or not, Perform the operation of step 1 for each image in the collection, and then perform statistics on the entire data set to obtain the correlation between various key elements and private and public images, and then build a knowledge map based on this; Step 3: Use neural network Extract the features of the whole picture and the key elements of the picture, and use the knowledge map in step 2 to construct a graph neural network, which is used to fuse the features of the whole picture and the key elements of the picture, and obtain the final expression of the picture; and step 4: Based on step 3 The final expression of the picture in the neural network is used to predict the privacy risk of the picture.

Description

A method and system for social network picture privacy risk detection and early warning

technical field

The present invention relates to the privacy protection of social networks, in particular to a method and system for privacy risk detection and early warning in social network picture sharing.

Background technique

With the popularity of the mobile Internet, social networking has become a part of people's daily life. Devices such as smartphones and cameras provide easy access to pictures, so a large number of pictures are shared on social networks to share people's daily lives. After 2014, images have surpassed plain text to become the most numerous form of sharing on social networks, with more than 100 million and 300 million images uploaded every day on Instagram and Facebook, respectively.

The shared pictures contain a lot of information, which is likely to reveal the privacy of users. Instead of textual content that needs to be thought and typed, the user simply presses the shutter to get a picture. Therefore, although social network providers allow users to set the visibility of content to protect user privacy, many users are unaware of the privacy risks in pictures. In one study, researchers surveyed users' expectations about the privacy settings of images by describing the content of images to users. The results showed that there was a discrepancy between user expectations and the actual privacy settings of these images. Therefore, there is a need for an effective method that can infer privacy risks in user-shared pictures, identify pictures that may reveal privacy, and give early warnings.

Some existing methods using machine learning are mainly divided into two categories. The first category uses the entire image for classification and gives suggestions for privacy settings. Due to the inexplicable problem of machine learning, the suggestions given by this method are difficult to be understood by users, and it is difficult for users to find out which areas in the image are more likely to leak privacy; another type of method based on object detection is easy to understand, but subject to Limited to predefined object categories, it cannot deal with privacy leakage caused by objects outside the list.

SUMMARY OF THE INVENTION

In view of the above existing problems, the present invention proposes a method and system for the detection and early warning of privacy risk of social network pictures, the purpose is to predict the privacy risk of pictures uploaded by social network users, and to give early warning to pictures with high privacy risks, reminding Users upload with caution.

The present invention provides a method for detecting and pre-warning the privacy risk of social network pictures, which includes: step 1: extracting key elements in the picture by using a target detection framework and obtaining information of the key elements; step 2: collecting information about whether the picture is private or not Data set, perform step 1 operation on each picture in the data set, and then make statistics on the entire data set to obtain the correlation between various key elements and privacy and public pictures, and build a knowledge map accordingly; step Three: Use the neural network to extract the features of the whole picture and the key elements of the picture, and use the knowledge map in step 2 to construct a graph neural network, which is used to fuse the features of the whole picture and the key elements of the picture, and obtain the final expression of the picture; and steps 4: Based on the final expression of the picture in step 3, use the neural network to predict the privacy risk of the picture.

The present invention also provides a system for detection and early warning of privacy risks in social network pictures, which includes: a picture element extraction module: extracting key elements in the picture by using a target detection framework and obtaining the information of the key elements; a knowledge map building module: collecting For the data set of whether the picture is private or not, the operation in the picture element extraction module is performed on each picture in the data set, and then statistics are performed on the entire data set to obtain the correlation between various key elements and private and public pictures, and According to this, the knowledge map is constructed; the image information fusion module: use the neural network to extract the characteristics of the whole picture and the key elements of the image, and use the knowledge map in the knowledge map building module to build a graph neural network, which is used to fuse the whole image and the key elements of the image. And the privacy detection and early warning module: based on the final expression of the picture in the picture information fusion module, the neural network is used to predict the privacy risk of the picture and give early warning.

The present invention comprehensively considers the key elements in the picture and the whole picture, utilizes the constructed knowledge graph to fuse the information, thereby obtains the privacy risk of the picture, warns the pictures with high privacy risk, and reminds the user to upload it cautiously.

The present invention is described in detail below with reference to the accompanying drawings and specific embodiments, but is not intended to limit the present invention.

Description of drawings

FIG. 1 is a schematic flowchart of a method for detecting and pre-warning the privacy risk of a social network picture according to the present invention.

Detailed ways

Below in conjunction with accompanying drawing, structure principle and working principle of the present invention are described in detail:

The first step is to extract key elements based on target detection.

In order to obtain the position and category information of key elements in the picture, extraction is carried out with the help of mature target detection frameworks, such as Faster-RCNN, MASK-RCNN, etc. Of course, there are other target detection frameworks, and the present invention is not limited to this. Therefore, the key elements obtained are all categories that can be detected by the target detection framework.

The second step is to build a knowledge graph based on key elements.

Collect data sets of social network pictures on social network platforms that want to protect the privacy of pictures, and use them to obtain labels for whether users think the pictures will reveal privacy. According to the data, the pictures in the dataset are divided into two categories: private pictures and public pictures. If image datasets on social networking platforms are difficult to obtain, other annotated image privacy datasets published in previous studies can also be used as an alternative.

After obtaining the dataset, perform the operations in the first step for each image, extract the key elements on each image and obtain the information of the key elements. After the key elements and their information are extracted, statistics are carried out on the entire image dataset, in order to obtain the frequency of various key elements appearing in private images and public images, respectively, and then use the number of images in their respective categories. Unification, and use this as the strength of the association between each key element and the two categories of privacy and publicity.

To build a knowledge graph, the graph contains two types of nodes: the first type of two nodes, representing the two categories of privacy and publicity, respectively, so they are category nodes; the second type of nodes, the number of which is equal to the type of key elements, so they are key elements node. An edge is established between the category node and the two types of key elements, and the weight of the edge is the strength of the association between the key element and the two categories of privacy and publicity.

The third step is the fusion of global information and key element information based on graph neural network.

For a picture, the neural network is used to extract the overall features of the picture as the initialization of the two node features of the first category, namely privacy and publicity. If the image contains a certain type of key element, the feature of the corresponding area of this type of key element is cut out as the corresponding second type of node, that is, the initialization of the key element node. The remaining uninitialized node features are zeroed.

After all nodes are initialized, a graph neural network is used to fuse the global information contained in the category nodes with the local information contained in the key element nodes. Finally, for the fused features, different weights are assigned to the features of different key element nodes, and spliced with the fused global features to form the final expression of the picture.

The fourth step is image privacy risk inference based on fusion information.

Based on the final expression of the picture obtained in the third step, the neural network is used to predict the privacy risk of the picture and give an early warning.

The above model is trained based on the collected dataset, resulting in a model that can be applied in practice. After the user uploads the picture, the model is used to make predictions. When the privacy risk predicted by the model is high, the detected key elements are marked to remind the user to share carefully and pay attention to the key elements that may reveal privacy, thereby reducing the privacy of social network image sharing. risk.

Of course, the present invention can also have other various embodiments, without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding Changes and deformations should belong to the protection scope of the appended claims of the present invention.

Claims (14)

1. a method for social network picture privacy risk detection and early warning, is characterized in that, comprising: Step 1: Use the target detection framework to extract the key elements in the picture and obtain the information of the key elements; Step 2: Collect the data set of whether the image is private or not, perform the operation of step 1 on each image in the data set, and then make statistics on the entire data set to obtain the correlation between various key elements and private and public images. And according to this to build a knowledge map; Step 3: extract the features of the whole picture and the key elements of the picture by using the neural network, and use the knowledge map in step 2 to construct a graph neural network, which is used to fuse the features of the whole picture and the key elements of the picture, and obtain the final expression of the picture; and Step 4: Based on the final expression of the picture in Step 3, use the neural network to predict the privacy risk of the picture. 2. The method for social network picture privacy risk detection and early warning according to claim 1, wherein the target detection framework in the step 1 can adopt Faster-RCNN or MASK-RCNN. 3. the method for social network picture privacy risk detection and early warning as claimed in claim 1, is characterized in that, the data set in described step 2 can adopt the picture data set collected on the social platform of protecting picture privacy or Use existing image datasets marked with privacy or not. 4. the method for social network picture privacy risk detection and early warning as claimed in claim 3, is characterized in that, the described picture data set collected on the social platform of protecting picture privacy is based on whether the platform user thinks that picture can leak privacy Most of the annotations are used as the basis for judgment, and the pictures are divided into two categories: private pictures and public pictures. 5. the method for social network picture privacy risk detection and early warning as claimed in claim 1, is characterized in that, the association degree of various key elements in described step 2 and privacy, public picture is respectively with various key elements It is represented by the frequency of appearance in private and public pictures. 6. The method for social network picture privacy risk detection and early warning according to claim 1, wherein the knowledge graph in the step 2 comprises two types of nodes: Category nodes, representing two categories, privacy and public; and Key element nodes, the number is equal to the type of key elements; Among them, an edge is established between the two types of nodes, and the weight of the edge is the degree of association between the key element and the private/public category. 7. The method for social network picture privacy risk detection and early warning according to claim 6, wherein the step 3 specifically comprises: For a picture, the neural network is used to extract the overall features of the picture as the initialization of the two node features of the first type, and for a certain type of key elements contained in the picture, the features of the corresponding area are cropped, as the corresponding node of the second type. initialization; After all nodes are initialized, a graph neural network is used to fuse the global information contained in the category nodes and the local information contained in the key element nodes; For the fused features, different weights are given to the features of different key element nodes, and spliced with the fused global features to form the final expression of the picture. 8. A system for social network picture privacy risk detection and early warning, comprising: Image element extraction module: use the target detection framework to extract the key elements in the image and obtain the information of the key elements; Knowledge graph building module: collect data sets of image privacy or not, perform operations in the image element extraction module for each image in the data set, and then conduct statistics on the entire data set to obtain the relationship between various key elements and privacy, publicity The relevance of the picture, and build the knowledge graph accordingly; Image information fusion module: use the neural network to extract the features of the whole picture and the key elements of the image, and use the knowledge map in the knowledge map building module to build a graph neural network, which is used to fuse the features of the whole image and the key elements of the image, and obtain the image information. final expression; and Privacy detection and early warning module: Based on the final expression of the picture in the picture information fusion module, the neural network is used to predict the privacy risk of the picture and give early warning. 9. The system for social network picture privacy risk detection and early warning according to claim 8, wherein the target detection framework in the picture element extraction module can adopt Faster-RCNN or MASK-RCNN. 10. The system for social network picture privacy risk detection and early warning according to claim 8, wherein the data set in the picture information fusion module can adopt picture data collected on a social platform protecting picture privacy set or use existing image datasets marked with privacy or not. 11. The system for social network picture privacy risk detection and early warning according to claim 10, wherein the picture data set collected on the social platform for protecting picture privacy is based on whether the platform user thinks that the picture will leak privacy Most of the annotations are used as the basis for judgment, and the pictures are divided into two categories: private pictures and public pictures. 12. The system for social network picture privacy risk detection and early warning according to claim 8, wherein various key elements in the knowledge graph building module are associated with privacy and public pictures using various key elements. The frequency of elements appearing in private and public images is represented respectively. 13. The system for social network picture privacy risk detection and early warning according to claim 8, wherein the knowledge graph in the knowledge graph building module comprises two types of nodes: Category nodes, representing two categories, privacy and public; and Key element nodes, the number is equal to the type of key elements; Among them, an edge is established between the two types of nodes, and the weight of the edge is the degree of association between the key element and the private/public category. 14. The method for social network picture privacy risk detection and early warning according to claim 13, wherein the specific operations in the picture information fusion module include: For a picture, the neural network is used to extract the overall features of the picture as the initialization of the two node features of the first type, and for a certain type of key elements contained in the picture, the features of the corresponding area are cropped, as the corresponding node of the second type. initialization; After all nodes are initialized, a graph neural network is used to fuse the global information contained in the category nodes and the local information contained in the key element nodes; For the fused features, different weights are given to the features of different key element nodes, and spliced with the fused global features to form the final expression of the picture.

Images