CN113378859B - Image privacy detection method with interpretability - Google Patents

Image privacy detection method with interpretability Download PDF

Info

Publication number
CN113378859B
CN113378859B CN202110723826.0A CN202110723826A CN113378859B CN 113378859 B CN113378859 B CN 113378859B CN 202110723826 A CN202110723826 A CN 202110723826A CN 113378859 B CN113378859 B CN 113378859B
Authority
CN
China
Prior art keywords
node
privacy
formula
ith
classification
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110723826.0A
Other languages
Chinese (zh)
Other versions
CN113378859A (en
Inventor
张兰
于海阔
李向阳
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
University of Science and Technology of China USTC
Original Assignee
University of Science and Technology of China USTC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University of Science and Technology of China USTC filed Critical University of Science and Technology of China USTC
Priority to CN202110723826.0A priority Critical patent/CN113378859B/en
Publication of CN113378859A publication Critical patent/CN113378859A/en
Application granted granted Critical
Publication of CN113378859B publication Critical patent/CN113378859B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/95—Retrieval from the web
    • G06F16/951—Indexing; Web crawling techniques

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Evolutionary Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses an interpretable image privacy detection method, which comprises the following steps: 1. constructing a data set of the privacy classification; 2. extracting semantic information contained in the data set image; 3, constructing a weighted directed graph corresponding to the data set image; 4. constructing a graph neural network and training the output of the second classification of the nodes of the weighted directed graph based on the dataset; 5. determining a coding pixel area of the input image by using the neural network obtained by training; 6. and generating the privacy rule. The invention provides automatic positioning of image desensitization based on a deep learning theory, and can explain a privacy classification result at an object level to obtain a privacy rule at the object level, thereby improving interpretability of privacy classification.

Description

Interpretable image privacy detection method
Technical Field
The invention relates to the field of data privacy and machine learning, in particular to an image privacy detection method with interpretability.
Background
With the development of the information age and the popularization of smart phones, people are used to share and communicate on a social platform, and according to incomplete statistics, Facebook generates 3.5 hundred million photos and 1 hundred million hours of video browsing every day. Instagram users shared 9500 million photos and videos per day. Meanwhile, the personal picture is also perceived and collected by the outside world in many other different ways, such as mobile phone application access, cloud storage, and the like. However, while sharing and spreading of social pictures brings great convenience to people, the risk of privacy disclosure is also increased by the large amount of rich personal information contained in the published pictures, for example, the background in the pictures exposes the geographic location and whereabouts, the people appearing in the pictures expose the social relationships of privacy, and the like.
When people release photos, the protection of their privacy is often ignored, and even if people notice that the images are desensitized by a manual coding mode in most cases, the mode is complicated and not safe enough. When privacy disclosure events occur continuously, people need to research privacy information in pictures, so that the users can be helped to judge which personal pictures are private and which are not private, research privacy rules with consensus and personalization is carried out, and end-to-end help is provided for desensitization of the users.
In the existing privacy detection technology, some works are to divide the picture into private and non-private pictures by means of deep learning or support vector machines and the like directly from the view point of picture classification, and the classification result of the technology depends on a training set and has no interpretability, so that the technology has no available value. Some technologies research objects contained in pictures, research the correlation between the objects and the privacy of the pictures, and thereby evaluate the privacy degree of the pictures, and such methods have the problems of unclear privacy definition, limited detection capability in model output, poor interpretability of privacy rules, and the like. And the existing data set generally has the problems of poor interpretability of privacy classification and the like.
Disclosure of Invention
Aiming at the problems of the existing method, the invention provides an image privacy detection method with interpretability, so that the automatic positioning of image desensitization can be provided based on a deep learning theory, and the privacy classification result can be interpreted at an object level to obtain the privacy rule at the object level, thereby improving the interpretability of the privacy classification.
In order to achieve the purpose, the invention adopts the following technical scheme:
the invention relates to an interpretable image privacy detection method which is characterized by comprising the following steps:
step 1, constructing a data set of privacy classification;
step 1.1, collecting N code printing images subjected to desensitization processing on a social platform by using a web crawler;
step 1.2, marking frame information and categories of all objects and backgrounds in each code printing image, so as to obtain the number K of the object categories of N code printing images;
step 1.3, marking the object and the background in each coded image as 1 if desensitization treatment is carried out on the object and the background, otherwise marking the object and the background as 0, thereby completing privacy classification of the object in each coded image and obtaining a desensitization image data set D;
step 2, extracting semantic information contained in the image;
step 2.1, training the fast-rcnn model based on the desensitization image data set D to obtain a target detection model M1；
Any input image I passes through the target detection model M1Post-output detection result M1(I)＝{o1,o2,…,oi,…,onIn which o isiIndicates that the frame information B is includediAnd the ith object information of the classified confidence probability, n represents the target detection model M1The number of objects detected in the input image I, I ∈ [1, n ]]；
Step 2.2, training the vgg model based on the place365 data set to obtain a background classification model M2；
Any input image I passes through a background classification model M2Post-output background classification result M2(I)＝{s1,f1In which s1Confidence probability representing the classification of the background, f1A feature vector representing an output of the background classification model;
step 3, constructing a weighted directed graph G corresponding to the input image II＝{VI,EI}；
Step 3.1 construction of node set VI：
For the ith object information oiConstructing and using corresponding embedded vectors as corresponding nodes
Figure GDA0003635017210000021
Thereby obtaining n nodes { v }1,v2,…,vn}; wherein,
Figure GDA0003635017210000022
indicates the ith object information oiA corresponding confidence probability vector of length K,
Figure GDA0003635017210000023
indicates the ith object information oiThe confidence probability for the kth class,
Figure GDA0003635017210000024
indicates the ith object information oiThe information of the corresponding frame is stored in the storage device,
Figure GDA0003635017210000025
representing the coordinates of the center point of the frame in the pixel matrix of the input image I,
Figure GDA0003635017210000026
representing the relative size of the bounding box;
confidence probability s using a multi-tier perceptron as shown in equation (1)1And feature vector f1Mapping to a background node v of length K +3n+1Thereby obtaining a node set VI＝{v1,v2,…,vn+1}；
vn+1＝(s1||f1)×Wn+1 (1)
In the formula (1), Wn+1A parameter matrix representing a multilayer perceptron, · | |, representing splicing of vectors;
step 3.2 construct adjacency matrix EI；
Calculating any ith node viFrame box ofiAnd the jth node vjBox of (2)jWhether or not to overlap, if so, calculating the overlapped pixel area S (box) by using the formula (2)i∩boxj) Occupy the frame boxiRatio of areas EI[i,j]If there is no overlap, let the ratio EI[i,j]0; thereby obtaining an adjacency matrix EI＝{EI[i,j]|i＝1,2,…,n；j＝1,2,…,n}；
Figure GDA0003635017210000031
In the formula (2), S represents a pixel area;
step 4, constructing a general formula1Layer diagramConvolutional layers and2graph neural network M composed of layer perceptron3And trained on the basis of the desensitized image dataset D to utilize a neural network M3Get weighted directed graph GI＝{VI,EITwo classification outputs for each node in the tree;
step 4.1 construct a graph convolution layer using equation (3):
hm+1＝σ(EIhmwm),0≤m≤l1-1 (3)
in the formula (3), l1Indicates the number of convolutional layers, hmFeature vector, w, representing the convolution layer of the m-th layer mapmA parameter matrix representing the mth layer map convolution layer; σ represents an activation function; when m is equal to 0, the compound has the following structure,
Figure GDA0003635017210000032
step 4.2 construction of l Using formula (4)2The layer perceptron enables each node to share one parameter matrix, and therefore the output O of the multilayer perceptron shown in the formula (5) is obtained;
Figure GDA0003635017210000033
Figure GDA0003635017210000034
in the formulae (4) and (5),
Figure GDA0003635017210000035
a feature vector representing the k-th layer perceptron,
Figure GDA0003635017210000036
a parameter matrix representing the k-th layer perceptron,
Figure GDA0003635017210000037
the eigenvectors representing the k +1 layer perceptron,
Figure GDA0003635017210000038
denotes the l2Feature vectors of layer perceptrons,/2Representing the number of layers of the multilayer perceptron, and sig () representing a sigmoid function for outputting a binary result of each node;
step 4.3, constructing a loss function L (theta) by using the formula (6):
Figure GDA0003635017210000039
in formula (6), O (v)i) Represents the ith node viThe classification result of (2); theta denotes the neural network M of the graph3And is given by
Figure GDA0003635017210000041
yiRepresents the ith node viThe privacy classification label value of (1);
step 4.4, training the loss function L (theta) based on the desensitization image data set D by using a gradient descent method until the model converges, thereby obtaining a trained graph neural network M3And outputs a weighted directed graph GI＝{VI,EIThe result of the classification of each node in the Chinese character- };
step 5, using the trained graph neural network M3Determining a coding pixel area of an input image I;
step 5.1, if the ith node viClassification result of (v) O (v)i) If the value is larger than the set threshold value, the node v represents the ith node viIs a privacy node, otherwise, represents the ith node viIs a non-privacy node;
step 5.2, assume the ith node viFor the privacy node, if i belongs to [1, n ]]Then utilize the ith node viCorresponding frame information
Figure GDA0003635017210000042
Determining a coding area; if i is n +1, it means the ith node viMarking the coding region as a background node, and removing the coding region from the whole input image I{B1,B2,…,BnPixel areas left behind the frame of the contained object;
step 6, generating a privacy rule:
step 6.1, constructing a weighted directed graph G by using the formula (7)I＝{VI,EIIth node v iniSymbol vector sv ofi：
Figure GDA0003635017210000043
In formula (7), sv [ j ]]Represents the ith node viSign of the jth value of (v)i[j]Represents the ith node viThe jth value of (d);
step 6.2, solving the classification result by using the formula (8) for the ith node viGradient vector gv of nodei：
Figure GDA0003635017210000044
In formula (8), gv [ j ]]Represents the ith node viThe gradient of the jth value of (a);
step 6.3, solving the ith node v by using the formula (9)iImportance Im (v) to classification resultsi)：
Figure GDA0003635017210000045
Step 6.4, if Im (v)i) If the value is larger than the set privacy threshold value tau, the node v indicates the ith node viIs a privacy-related node;
step 6.5, constructing a weighted directed graph G by using the formula (10)I＝{VI,EISign matrix of the middle edges se:
Figure GDA0003635017210000051
formula (10)) In se [ i, j ]]Represents the ith node viAnd the jth node vjThe sign of the edge of (a);
step 6.6, solving a gradient matrix ge of the classification result for each edge by using the formula (11):
Figure GDA0003635017210000052
in formula (11), ge [ i, j ]]Represents the ith node viAnd the jth node vjThe gradient of the edge of (a);
step 6.7, solve the adjacency matrix Ime (E) of privacy-related edges between privacy nodes using equation (12)I)：
Ime(EI)＝se·ge (12)
Step 6.8, making the privacy sub-graph SGI＝{{vi|Im(vi)≥τ，1≤i≤n},Ime(EI) The privacy rule of the input image I is used as the input image I;
and 7, taking the coded pixel area and the privacy rule as the privacy detection result of the input image I.
Compared with the prior art, the invention has the beneficial effects that:
1. according to the method, the privacy degrees of the objects are classified by constructing the graph neural network, the number of parameters of the traditional convolutional neural network is greatly reduced, and simultaneously the classes of the objects in the images and the contact information between the objects are effectively combined, so that the efficiency and the accuracy of image-level privacy classification and object-level privacy classification are effectively improved.
2. The invention provides a method for evaluating the privacy relevance of an object in an image by utilizing the interpretability of a neural network; by extracting gradient information of the input vector, the object information and the object privacy classification result are associated, and the reason of object privacy classification in the image is explained at a semantic level, so that the interpretability of the image privacy protection system is effectively improved.
3. The invention has high expandability, can be used in combination with a plurality of image information extraction models, for example, the system is easily expanded to different object detection models and corresponding heat map generation methods, and can effectively position the pixel position of an object needing desensitization in an image, thereby improving the usability of the image privacy protection system.
4. The invention designs a method for the real information data set containing the image privacy rule for the first time, breaks through the limitation that the traditional method can only use a public data set to simulate the privacy data set, and provides a new scheme and a data collection method for further and deeply researching the image privacy in the future.
Drawings
FIG. 1 is a flow chart of privacy rule generation in the present invention;
FIG. 2 is an example of a data set image of the present invention;
FIG. 3a is an input image of a test set according to the present invention;
FIG. 3b is a node classification diagram corresponding to the input images in the test set according to the present invention.
Detailed Description
In this embodiment, as shown in fig. 1, an interpretable image privacy detection method is performed as follows:
step 1, constructing a data set of privacy classification;
step 1.1, collecting N code printing images subjected to desensitization processing on a social platform by using a web crawler; in this embodiment, the main mobile phone platforms include multiple social platforms such as microblogs, knows, small red books, and the like, and the final coding image set is obtained by using an image modification detection program and manual screening. FIG. 2 is an exemplary image of a data set according to the present invention.
Step 1.2, marking frame information and categories of all objects and backgrounds in each code printing image, so as to obtain the number K of the object categories of the N code printing images; in this embodiment, 101 objects and scenes such as a human face, various cards, a person, an automobile, a bag, a mobile phone, a medicine, a notebook, an indoor space, an outdoor space, and the like are mainly marked.
Step 1.3, marking the object and the background in each code printing image as 1 if the object and the background are subjected to desensitization treatment, and marking the object and the background as 0 if the object and the background are not subjected to desensitization treatment, so that the privacy classification of the object in each code printing image is completed, and a desensitization image data set D is obtained;
step 2, extracting semantic information contained in the image;
step 2.1, training the faster-rcnn model based on the desensitization image data set D to obtain a target detection model M1；
Any input image I passes through the target detection model M1Post-output detection result M1(I)＝{o1,o2,…,oi,…,onIn which oiIndicates that the frame information B is includediAnd the ith object information of the confidence probability of classification, n represents the target detection model M1The number of objects detected in the input image I, I ∈ [1, n ]]；
Step 2.2, training the vgg model based on the place365 data set to obtain a background classification model M2(ii) a The data set is used as a data set of a scene classification model, and application scenes matched with the method are compared.
Any input image I passes through a background classification model M2Post-output background classification result M2(I)＝{s1,f1In which s is1Confidence probability representing background classification, f1A feature vector representing an output of the background classification model;
step 3, constructing a weighted directed graph G corresponding to the input image II＝{VI,EI}；
Step 3.1 construction of node set VI：
For the ith object information oiConstructing corresponding embedded vectors and using the embedded vectors as corresponding nodes
Figure GDA0003635017210000061
Thereby obtaining n nodes { v1,v2,…,vn}; wherein,
Figure GDA0003635017210000062
indicates the ith object information oiA corresponding confidence probability vector of length K,
Figure GDA0003635017210000063
indicates the ith object information oiFor the confidence probability of the kth class,
Figure GDA0003635017210000071
indicates the ith object information oiThe information of the corresponding frame is displayed on the display,
Figure GDA0003635017210000072
representing the coordinates of the center point of the frame in the pixel matrix of the input image I,
Figure GDA0003635017210000073
representing the relative size of the bounding box;
confidence probability s is determined using a multi-tier perceptron as shown in equation (1)1And a feature vector f1Mapping to a background node v of length K +3n+1Thereby obtaining a node set VI＝{v1,v2,…,vn+1}；
vn+1＝(s1||f1)×Wn+1 (1)
In the formula (1), Wn+1A parameter matrix representing a multi-layer perceptron, | | | represents the splicing of vectors;
step 3.2 construct adjacency matrix EI；
Calculating any ith node viFrame box ofiAnd the jth node vjBox of (2)jWhether or not to overlap, and if so, calculating the overlapped pixel area S (box) by using the formula (2)i∩boxj) Occupy the frame boxiRatio of areas EI[i,j]If there is no overlap, let the ratio EI[i,j]0; thereby obtaining an adjacency matrix EI＝{EI[i,j]1,2, …, n; j ═ 1,2, …, n }. Representing v using a size-proportional relationship of areasiAnd vjAnd further describes the closeness of the relationship between the objects.
Figure GDA0003635017210000074
In the formula (2), S represents a pixel area;
step 4, constructing a general formula1Layer map convolution layer and2graph neural network M composed of layer perceptron3In this embodiment, the method sequentially includes: the system comprises a No. 1 graph volume layer, a No. 1 full connection layer, a No. 2 graph volume layer, a No. 2 full connection layer, a shared 2-layer sensing machine and an activation function layer; and trained on the basis of the desensitized image dataset D, thereby utilizing the neural network M of the map3Get a weighted directed graph GI＝{VI,EITwo classification outputs for each node in the tree;
step 4.1 construct a graph convolution layer using equation (3):
hm+1＝σ(EIhmwm),0≤m≤l1-1 (3)
in the formula (3), l1Indicates the number of convolutional layers, hmFeature vector, w, representing the convolution layer of the mth layer mapmA parameter matrix representing the convolution layer of the m-th layer map; σ represents an activation function; when m is equal to 0, the compound is,
Figure GDA0003635017210000075
step 4.2 construction of l Using formula (4)2The layer perceptron enables each node to share one parameter matrix, and therefore the output O of the multilayer perceptron shown in the formula (5) is obtained;
Figure GDA0003635017210000081
Figure GDA0003635017210000082
in the formulae (4) and (5),
Figure GDA0003635017210000083
a feature vector representing the k-th layer perceptron,
Figure GDA0003635017210000084
a parameter matrix representing the k-th layer perceptron,
Figure GDA0003635017210000085
the eigenvectors representing the k +1 layer perceptron,
Figure GDA0003635017210000086
denotes the l (th)2Feature vectors of layer perceptron,/2Representing the number of layers of the multi-layer perceptron and sigma () representing the activation function, in this example the relu function and the leakyrelu function are used. sig () represents a sigmoid function for outputting a result of the classification by two for each node;
step 4.3, constructing a loss function L (theta) by using the formula (6):
Figure GDA0003635017210000087
in formula (6), O (v)i) Represents the ith node viThe classification result of (2); theta denotes graph neural network M3Is a parameter of
Figure GDA0003635017210000088
yiRepresents the ith node viThe privacy classification label value of (1);
step 4.4, training the loss function L (theta) by using a gradient descent method based on the desensitization image data set D until the model converges, thereby obtaining a trained graph neural network M3And outputs a weighted directed graph GI＝{VI,EIThe result of the classification of each node in the Chinese character- }; fig. 3a and 3b show images in the test set of the data set, and as can be seen from the node classification results in the figures, the coded object in fig. 3a and the node privacy classification result in fig. 3b are basically consistent.
Step 5, using the trained graph neural network M3Determining a coding pixel area of an input image I;
step 5.1, if the ith node viClassification result of (d) O (v)i) Is greater than the set threshold value and is,then it represents the ith node viIs a privacy node, otherwise, represents the ith node viIs a non-privacy node; in this embodiment, the threshold is set to 0.
Step 5.2, assume the ith node viFor the privacy node, if i belongs to [1, n ]]Then utilize the ith node viCorresponding frame information
Figure GDA0003635017210000089
Determining a coding area; if i is n +1, it means the ith node viMarking the coding region as the whole input image I except for the { B ] as the background node1,B2,…,BnPixel area left behind the border of the contained object;
and 6, generating a privacy rule, wherein the reason of the classification result needs to be explained for the privacy classification result output by the model, so that the classification result can be trusted. Whether a certain object is coded or not is related to the object itself, and may also be related to the background in the image and other objects appearing in the image. The privacy rule of the image I coding object is GIBy solving the privacy rules contained in the sub-graph representation image:
step 6.1, constructing a weighted directed graph G by using the formula (7)I＝{VI,EIIth node v iniSymbol vector sv ofi：
Figure GDA0003635017210000091
In formula (7), sv [ j ]]Represents the ith node viSign of the jth value of (v)i[j]Represents the ith node viThe jth value of (d);
step 6.2, solving the classification result for the ith node v by using the formula (8)iGradient vector gv of nodei：
Figure GDA0003635017210000092
In formula (8), gv [ j ]]Represents the ith node viThe gradient of the jth value of (a);
step 6.3, solving the ith node v by using the formula (9)iImportance Im (v) to classification resultsi)：
Figure GDA0003635017210000093
Step 6.4, if Im (v)i) If the value is larger than the set privacy threshold value tau, the node v indicates the ith node viFor privacy-related nodes, in this embodiment, the threshold τ is set to 0.5.
Step 6.5, constructing a weighted directed graph G by using the formula (10)I＝{VI,EISign matrix se of middle edge:
Figure GDA0003635017210000094
in formula (10), se [ i, j]Represents the ith node viAnd the jth node vjThe sign of the edge of (a);
step 6.6, solving a gradient matrix ge of the classification result for each edge by using the formula (11):
Figure GDA0003635017210000095
in formula (11), ge [ i, j]Represents the ith node viAnd the jth node vjThe gradient of the edge of (a);
step 6.7, solve the adjacency matrix Ime (E) of privacy-related edges between privacy nodes using equation (12)I)：
Ime(EI)＝se·ge (12)
Step 6.8, making the privacy sub-graph SGI＝{{vi|Im(vi)≥τ，1≤i≤n},Ime(EI) The privacy rule of the input image I is used as the input image I;
and 7, the coding pixel area and the privacy rule are jointly used as the privacy detection result of the input image I, in the traditional work, the privacy judgment condition is often a black box model, and reasonable explanation cannot be given, but the method uses the privacy subgraph to represent the privacy rule, so that the relation between objects and the influence on the privacy rule can be seen semantically, the defect that the traditional method cannot be explained is overcome, and the output coding area at the pixel level can provide accurate positioning for subsequent desensitization operation.
Due to the machine learning interpretability technology, the method has the advantages of being capable of interpreting privacy reasons at a semantic level and positioning privacy, and is suitable for scenes such as cloud uploading, social network sharing and mobile phone application reading.

Claims (1)

1. An interpretable image privacy detection method is characterized by comprising the following steps:
step 1, constructing a data set of privacy classification;
step 1.1, collecting N coded images subjected to desensitization treatment on a social platform by using a web crawler;
step 1.2, marking frame information and categories of all objects and backgrounds in each code printing image, so as to obtain the number K of the object categories of N code printing images;
step 1.3, marking the object and the background in each coded image as 1 if desensitization treatment is carried out on the object and the background, otherwise marking the object and the background as 0, thereby completing privacy classification of the object in each coded image and obtaining a desensitization image data set D;
step 2, extracting semantic information contained in the image;
step 2.1, training the faster-rcnn model based on the desensitization image data set D to obtain a target detection model M1；
Any input image I passes through the target detection model M1Post-output detection result M1(I)＝{o1,o2,…,oi,…,onIn which o isiThe representation includes frame information BiAnd the ith object information of the classified confidence probability, n represents the target detection model M1Detected on the input image INumber of objects, i ∈ [1, n ]]；
Step 2.2, training the vgg model based on the place365 data set to obtain a background classification model M2；
Any input image I passes through a background classification model M2Post-output background classification result M2(I)＝{s1,f1In which s is1Confidence probability representing background classification, f1A feature vector representing an output of the background classification model;
step 3, constructing a weighted directed graph G corresponding to the input image II＝{VI,EI}；
Step 3.1 construct node set VI：
For the ith object information oiConstructing corresponding embedded vectors and using the embedded vectors as corresponding nodes
Figure FDA0003635017200000011
Thereby obtaining n nodes { v1,v2,…,vn}; wherein,
Figure FDA0003635017200000012
indicates the ith object information oiA corresponding confidence probability vector of length K,
Figure FDA0003635017200000013
indicates the ith object information oiFor the confidence probability of the kth class,
Figure FDA0003635017200000014
indicates the ith object information oiThe information of the corresponding frame is displayed on the display,
Figure FDA0003635017200000015
representing the coordinates of the center point of the frame in the pixel matrix of the input image I,
Figure FDA0003635017200000016
indicating a relatively large borderSmall;
confidence probability s is determined using a multi-tier perceptron as shown in equation (1)1And feature vector f1Mapping to a background node v of length K +3n+1Thereby obtaining a node set VI＝{v1,v2,…,vn+1}；
vn+1＝(s1||f1)×Wn+1 (1)
In the formula (1), Wn+1A parameter matrix representing a multi-layer perceptron, | | | represents the splicing of vectors;
step 3.2 construct adjacency matrix EI；
Calculating any ith node viFrame box ofiAnd the jth node vjBox of (2)jWhether or not to overlap, and if so, calculating the overlapped pixel area S (box) by using the formula (2)i∩boxj) Occupy the frame boxiRatio of areas EI[i,j]If there is no overlap, let the ratio EI[i,j]0; thereby obtaining an adjacency matrix EI＝{EI[i,j]|i＝1,2,…,n；j＝1,2,…,n}；
Figure FDA0003635017200000021
In formula (2), S represents a pixel area;
step 4, constructing a group I1Layer map convolution layer sum2Graph neural network M composed of layer perceptron3And trained on the basis of the desensitized image dataset D, thereby utilizing the neural network M of the map3Get a weighted directed graph GI＝{VI,EITwo classification outputs of each node in the tree;
step 4.1 construct a graph convolution layer using equation (3):
hm+1＝σ(EIhmwm),0≤m≤l1-1 (3)
in the formula (3), l1Indicates the number of convolutional layers, hmFeature vector, w, representing the convolution layer of the mth layer mapmRepresents the m-th layerA parameter matrix of the graph convolution layer; σ represents an activation function; when m is equal to 0, the compound is,
Figure FDA0003635017200000022
step 4.2 construction of l Using formula (4)2The layer perceptron enables each node to share one parameter matrix, and therefore output O of the multilayer perceptron shown in the formula (5) is obtained;
Figure FDA0003635017200000023
Figure FDA0003635017200000024
in the formulae (4) and (5),
Figure FDA0003635017200000025
a feature vector representing the k-th layer perceptron,
Figure FDA0003635017200000026
a parameter matrix representing the k-th layer perceptron,
Figure FDA0003635017200000027
representing the feature vector of the k +1 th layer perceptron,
Figure FDA0003635017200000028
denotes the l (th)2Feature vectors of layer perceptron,/2Representing the number of layers of the multilayer perceptron, and sig () representing a sigmoid function and used for outputting a binary classification result of each node;
step 4.3, constructing a loss function L (theta) by using the formula (6):
Figure FDA0003635017200000031
in the formula (6), O (v)i) Represents the ith node viThe classification result of (2); theta denotes the neural network M of the graph3And is given by
Figure FDA0003635017200000032
yiRepresents the ith node viThe privacy classification label value of (1);
step 4.4, training the loss function L (theta) based on the desensitization image data set D by using a gradient descent method until the model converges, thereby obtaining a trained graph neural network M3And outputs a weighted directed graph GI＝{VI,EIThe result of classification of each node in the tree is obtained;
step 5, using the trained graph neural network M3Determining a coding pixel area of an input image I;
step 5.1, if the ith node viClassification result of (v) O (v)i) If the value is larger than the set threshold value, the node v represents the ith node viIs a privacy node, otherwise, represents the ith node viIs a non-privacy node;
step 5.2, assume ith node viFor the privacy node, if i belongs to [1, n ]]Then utilize the ith node viCorresponding frame information
Figure FDA0003635017200000033
Determining a coding area; if i is n +1, it means the ith node viMarking the coding region as the whole input image I except for the { B ] as the background node1,B2,…,BnPixel areas left behind the frame of the contained object;
step 6, generating a privacy rule:
step 6.1, constructing a weighted directed graph G by using the formula (7)I＝{VI,EIThe ith node v iniSymbol vector sv ofi：
Figure FDA0003635017200000034
In formula (7), sv [ j ]]Represents the ith node viSign of the jth value of (v)i[j]Represents the ith node viThe jth value of (d);
step 6.2, solving the classification result by using the formula (8) for the ith node viGradient vector gv of nodei：
Figure FDA0003635017200000035
In formula (8), gv [ j ]]Represents the ith node viThe gradient of the jth value of (a);
step 6.3, solving the ith node v by using the formula (9)iImportance Im (v) to classification resultsi)：
Figure FDA0003635017200000036
Step 6.4, if Im (v)i) If the value is larger than the set privacy threshold value tau, the node v represents the ith node viIs a privacy-related node;
step 6.5, constructing a weighted directed graph G by using the formula (10)I＝{VI,EISign matrix of the middle edges se:
Figure FDA0003635017200000041
in formula (10), se [ i, j]Represents the ith node viAnd the jth node vjThe sign of the edge of (a);
step 6.6, solving a gradient matrix ge of the classification result for each edge by using the formula (11):
Figure FDA0003635017200000042
in formula (11), ge [ i, j ]]Represents the ith node viAnd the jth node vjThe gradient of the edge of (a);
step 6.7, solve the adjacency matrix Ime (E) of privacy-related edges between privacy nodes using equation (12)I)：
Ime(EI)＝se·ge (12)
Step 6.8, making the privacy sub-graph SGI＝{{vi|Im(vi)≥τ，1≤i≤n},Ime(EI) The privacy rule of the input image I is used as the input image I;
and 7, taking the coded pixel area and the privacy rule as the privacy detection result of the input image I.
CN202110723826.0A 2021-06-29 2021-06-29 Image privacy detection method with interpretability Active CN113378859B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110723826.0A CN113378859B (en) 2021-06-29 2021-06-29 Image privacy detection method with interpretability

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110723826.0A CN113378859B (en) 2021-06-29 2021-06-29 Image privacy detection method with interpretability

Publications (2)

Publication Number Publication Date
CN113378859A CN113378859A (en) 2021-09-10
CN113378859B true CN113378859B (en) 2022-07-15

Family

ID=77579699

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110723826.0A Active CN113378859B (en) 2021-06-29 2021-06-29 Image privacy detection method with interpretability

Country Status (1)

Country Link
CN (1) CN113378859B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116244738B (en) * 2022-12-30 2024-05-28 浙江御安信息技术有限公司 Sensitive information detection method based on graph neural network

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105260676B (en) * 2015-10-16 2017-10-03 合肥工业大学 Image privacy decision-making commending system and its method
US11397818B2 (en) * 2019-12-13 2022-07-26 Here Global B.V. Method, apparatus and computer program product for determining a semantic privacy index
CN111242196B (en) * 2020-01-06 2022-06-21 广西师范大学 Differential privacy protection method for interpretable deep learning
CN111859454B (en) * 2020-07-28 2024-03-29 桂林慧谷人工智能产业技术研究院 Privacy protection method for defending link prediction based on graph neural network

Also Published As

Publication number Publication date
CN113378859A (en) 2021-09-10

Similar Documents

Publication Publication Date Title
CN109978893B (en) Training method, device, equipment and storage medium of image semantic segmentation network
CN109815843B (en) Image processing method and related product
Sun et al. Rural building detection in high-resolution imagery based on a two-stage CNN model
Wang et al. Deep sub-region network for salient object detection
Bhavana et al. Hand sign recognition using CNN
CN107944450A (en) A kind of licence plate recognition method and device
CN106650619A (en) Human action recognition method
CN108764269A (en) A kind of cross datasets pedestrian recognition methods again based on space-time restriction incremental learning
Chandran et al. Missing child identification system using deep learning and multiclass SVM
CN106156777A (en) Textual image detection method and device
CN113378859B (en) Image privacy detection method with interpretability
CN114155556B (en) Human body posture estimation method and system based on stacked hourglass network added with channel shuffling module
CN111507416A (en) Smoking behavior real-time detection method based on deep learning
CN114943937A (en) Pedestrian re-identification method and device, storage medium and electronic equipment
Sarkar et al. Grad-cam aware supervised attention for visual question answering for post-disaster damage assessment
Guo et al. DeepSSN: A deep convolutional neural network to assess spatial scene similarity
Santos et al. RECOGNIZING AND EXPLORING AZULEJOS ON HISTORIC BUILDINGS’FACADES BY COMBINING COMPUTER VISION AND GEOLOCATION IN MOBILE AUGMENTED REALITY APPLICATIONS
CN115953832A (en) Semantic decoupling-based combined action recognition method of self-attention model
Orhei Urban landmark detection using computer vision
Gunawan et al. Semantic segmentation of aerial imagery for road and building extraction with deep learning
CN114550016A (en) Unmanned aerial vehicle positioning method and system based on context information perception
Sreelekshmi et al. Deep forgery detect: enhancing social media security through deep learning-based forgery detection
CN114463685A (en) Behavior recognition method and device, electronic equipment and storage medium
CN116453194B (en) Face attribute discriminating method and device
Rai et al. Auto Neural Architecture Generator for UAV-Based Geospatial Surveillance for Aerial Crisis Mapping in Dark

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant