CN113837393A - Wireless sensing model robustness detection method based on probability and statistical evaluation - Google Patents

Wireless sensing model robustness detection method based on probability and statistical evaluation Download PDF

Info

Publication number
CN113837393A
CN113837393A CN202111030938.4A CN202111030938A CN113837393A CN 113837393 A CN113837393 A CN 113837393A CN 202111030938 A CN202111030938 A CN 202111030938A CN 113837393 A CN113837393 A CN 113837393A
Authority
CN
China
Prior art keywords
vector
test sample
inconsistency
perception model
probability
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111030938.4A
Other languages
Chinese (zh)
Other versions
CN113837393B (en
Inventor
翟双姣
童维媛
汤战勇
刘博�
房鼎益
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Northwest University
Original Assignee
Northwest University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Northwest University filed Critical Northwest University
Priority to CN202111030938.4A priority Critical patent/CN113837393B/en
Publication of CN113837393A publication Critical patent/CN113837393A/en
Application granted granted Critical
Publication of CN113837393B publication Critical patent/CN113837393B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/245—Classification techniques relating to the decision surface
    • G06F18/2451—Classification techniques relating to the decision surface linear, e.g. hyperplane

Landscapes

  • Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Evolutionary Biology (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Mathematical Optimization (AREA)
  • Probability & Statistics with Applications (AREA)
  • Pure & Applied Mathematics (AREA)
  • Computing Systems (AREA)
  • Medical Informatics (AREA)
  • Operations Research (AREA)
  • Algebra (AREA)
  • Databases & Information Systems (AREA)
  • Mobile Radio Communication Systems (AREA)

Abstract

The invention discloses a method for detecting robustness of a wireless perception model based on probability and statistical evaluation, which comprises the following steps of training a bottom layer perception model, and respectively calculating a probability vector and a statistical vector in the process of predicting a test sample by the wireless perception model, wherein the statistical vector is realized by the following steps: defining an inconsistency measurement function according to an inconsistency measurement theory of the conformal prediction theory and the machine learning algorithm in the step 1; the inconsistency measure function evaluates that a test sample is many different from a previous set of samples, a greater inconsistency measure indicating that the test sample is more dissimilar from the previous set of samples; then defining a calibration data set, calculating an inconsistency measurement score and calculating a statistical vector; and finally, judging whether the prediction of the bottom layer perception model on the test sample is correct by using an anomaly detector. The method can be used on any machine learning-based wireless perception model to detect the robustness of the wireless perception model in a deployment stage.

Description

Wireless sensing model robustness detection method based on probability and statistical evaluation
Technical Field
The invention belongs to the field of robustness research of a wireless sensing scheme based on machine learning, and particularly relates to a wireless sensing model robustness detection method based on probability and statistical evaluation.
Background
Wireless sensing technology is the basis for many emerging applications, such as smart home personalized customization, fall monitoring, emotion detection, vital sign monitoring, and the like. The wireless sensing technology is used for sensing the surrounding environment and the activity and physiological information of human beings by using ubiquitous wireless signals (such as WiFi, RFID, ultrasonic waves and the like). At present, wireless networks are widely popularized in the global scope, the deployment cost is obviously reduced by using a mode of sensing the physical world by using wireless signals, and important breakthroughs are made in the aspects of usability, universality and the like. Existing wireless signals (sound, light, radio frequency signals and the like) in the environment can be used for sensing the environment in an 'additional' mode while the tasks (illumination, communication and the like) of the user are completed. Taking radio frequency signals as an example, the wireless signals generated by the signal transmitting end can generate physical phenomena such as direct incidence, reflection, scattering and the like in the process of propagation, thereby forming a plurality of propagation paths. Thus, the multipath superimposed signal formed at the signal receiving end carries spatial information reflecting the signal propagation process. The wireless sensing technology obtains the channel variation characteristics of the signal transmission process by analyzing the variation of the wireless signals in the transmission process, thereby realizing wide space physics and human physiology sensing.
Among all wireless sensing technologies, wireless sensing technologies based on machine learning have achieved a breakthrough. Machine learning techniques predict test data by learning the relationship of signal characteristics to target activity over a set of labeled training samples. The wireless sensing technology based on machine learning mainly depends on a precise algorithm of the machine learning technology to realize feature extraction and automatic classification, and the premise of the machine learning technology is that training data and test data are distributed consistently. However, wireless signals are highly susceptible to environmental changes, for example, changes in room layout can cause the reflected path of the signal to change, further causing the test data distribution to differ greatly from the training data distribution, which can lead to unreliable prediction results in machine learning. The unreliable prediction result is reflected in the learning-based wireless sensing technology, so that the robustness of the wireless sensing technology is greatly challenged.
In order to improve the robustness of the perception model (the trained model has high prediction accuracy to different environments and is called robustness), scientific researchers have made a lot of attempts and efforts, and the models can be roughly divided into two types, one type is a data enhancement method; another class is methods that extract more robust features. The data enhancement method is mainly characterized in that a conversion function is designed by a professional or learned through a neural network, and then simulated data of different scenes are obtained through the conversion function, so that a machine learning model suitable for the scene can be ensured to exist in each scene, and further, the test data of the scene can have better robustness; the method for extracting more robust features mainly extracts some more robust features irrelevant to the environment through a plurality of receiving devices or through a domain adaptive network, for example, the action of 'waving' can be recognized for people no matter how the environment changes, so that some features of 'waving' can be found theoretically, and the features are not changed no matter how the environment changes. After the characteristics which are not changed along with the environment are found, a robust perception model can be trained by using the characteristics and can be deployed to the actual environment.
Although existing methods are effective for certain specific tasks or environmental changes, these methods can only address some of the changes that can be expected at the system design stage to affect model robustness. Neither of these approaches yields better performance with respect to unpredictable changes in the post-deployment environment of the system, such as changes in the location and manner in which a chair is moved or gestures are performed. For the data enhancement method, it is impossible to predict all environmental changes in advance and design a transfer function for each environmental change; for a method for extracting robust features irrelevant to the environment, it is almost impossible to find features with robustness to all environments, so that the existing method is difficult to solve the problem of robustness of a wireless perception model based on learning in a deployment stage, and improvement on the existing method or a new method for solving the problem is urgently needed.
Disclosure of Invention
Aiming at the robustness problem of a wireless perception model based on machine learning in a deployment stage, the invention aims to provide a wireless perception model robustness detection method based on probability and statistical evaluation.
In order to realize the task, the invention adopts the following technical solution:
a wireless perception model robustness detection method based on probability and statistical evaluation is characterized by comprising the following steps:
step 1, training a bottom layer perception model
For a wireless perception task, firstly collecting a group of training samples and marking class labels, then extracting signal characteristics of different classes, and finally selecting a machine learning algorithm to train a wireless perception model based on machine learning;
step 2, calculating the probability vector of the wireless perception model in the process of predicting the test sample
When the trained wireless perception model is deployed in an actual environment, for monitoring the robustness of the perception model in real time, for each test sample, calculating a probability vector of the perception model in the process of predicting the test sample so as to evaluate the credibility of the prediction; the probability vector can be calculated by using a predict _ proba method in a scimit-spare packet in python;
step 3, calculating statistical vectors in the process of predicting test samples by the wireless perception model
Introducing a statistical vector, wherein the statistical vector is realized by the steps of:
step 3.1, defining an inconsistency measurement function according to an inconsistency measurement theory of the conformal prediction theory and the machine learning algorithm in the step 1; the inconsistency measure function evaluates that a test sample is many different from a previous set of samples, a greater inconsistency measure indicating that the test sample is more dissimilar from the previous set of samples;
step 3.2, define the calibration data set.
Dividing a training data set into equal k parts, using k-1 parts as the training set, using the remaining k-th part as a calibration data set, repeating the steps for k times, wherein k is 10, and all samples in the training data set are subjected to one-time calibration data set;
step 3.3, calculate the inconsistency measure score
The inconsistency measure scores of the calibration data set and the test data set are calculated using the inconsistency measure function defined in step 3.1:
for the calibration data set, knowing the true label, an inconsistency measurement score of a calibration set vector is obtained; for the test data set, calculating an inconsistency measurement score with each candidate class according to the unknown real label;
and 3.4, calculating a statistical vector.
Firstly, sorting the inconsistency measurement scores of the calibration data set from small to large according to categories; for the test samples, calculating a ratio of the inconsistency measure score in the calibration set to be at least the same as the inconsistency measure score of the temporarily marked test samples;
step 4, judging whether the prediction of the bottom layer perception model on the test sample is correct or not by using an anomaly detector
The anomaly detector mainly finds a tight boundary in the feature space, so that data points outside the boundary are considered as anomalous events;
the abnormal detector is a single-class support vector machine, the boundary of the abnormal detector is a hyperplane, and the hyperplane is used for separating a normal data point from an origin in a feature space, so that the boundary of the hyperplane is as close to the normal data point as possible; during deployment, the anomaly detector checks whether the input test sample is within the boundary constructed by the perceptual model training samples, and if not, the test sample is considered to be an outlier, i.e., the prediction of the test sample by the underlying perceptual model is incorrect.
According to the invention, the method for the anomaly detector to judge the prediction realization of the bottom layer perception model on the test sample is as follows:
dividing the training data set into 10 equal parts, wherein 9 parts are used as the training set to train the wireless perception model, and then calculating 1 part of probability vectors and statistical vectors according to the wireless perception model, so that the probability vectors and the statistical vectors of all the training data sets are obtained by repeating for 10 times;
connecting the probability vector and the statistical vector into a vector as a feature training anomaly detector; training an anomaly detector for each candidate class;
connecting the obtained probability vector and the statistical vector into a vector, inputting the vector into a candidate anomaly detector corresponding to the prediction label, and if the output result is 1, the prediction is correct; otherwise the prediction is wrong.
Compared with the prior art, the wireless perception model robustness detection method based on probability and statistical evaluation has the following beneficial technical effects:
(1) the first method comprises the steps of detecting the robustness of a wireless perception system in a deployment stage by combining probability evaluation and statistical evaluation;
(2) the method comprises the steps that firstly, an anomaly detector and a classification rejection strategy are used for improving the robustness of a wireless perception model based on machine learning;
(3) the method can be used on any wireless perception model based on machine learning to detect the robustness of the wireless perception model in a deployment stage; the method has good performance on 11 existing wireless perception models based on machine learning.
Drawings
FIG. 1 is a general flowchart of the method for detecting robustness of a wireless sensing model based on probability and statistical evaluation according to the present invention.
Fig. 2 is a controllable environment setup diagram.
FIG. 3 is a graph of different environmental changes in a controlled environment.
Fig. 4 is a diagram of different position and orientation environmental changes.
Fig. 5 is the performance of the different models themselves in the static setting and the dynamic setting.
Fig. 6 is a graph of the performance of detecting drift samples in different models.
The invention is described in further detail below with reference to the figures and specific embodiments.
Detailed Description
For the robustness problem of the wireless sensing model based on machine learning in the deployment stage, firstly, the maximum difference between the robustness problem of the wireless sensing model in the deployment stage and the robustness problem in the design stage is determined. The robustness of the wireless perception model in the design stage mainly depends on the design of a feature extraction algorithm and a classification algorithm, and the robustness in the deployment stage mainly depends on the adaptability of the model to the environment, so that the robustness can be detected in time before the environment changes to have a great influence on the performance of the model. Therefore, the applicant proposes a new idea to detect the robustness of the machine learning-based wireless perception model in the deployment phase, i.e. to detect when the wireless perception model fails due to environmental changes in the deployment phase.
Generally, for a machine learning based wireless perception model, a predicted output is always given no matter what the input is, but whether the output is correct or not is not guaranteed. Whether the prediction of the model is correct is judged mainly according to the intermediate result in the calculation process of the machine learning algorithm, the probability and a statistical evaluation method, and only the correct prediction result is output. Thus, when a prediction of a model is determined to be a misprediction, the model is deemed to be invalid in that environment. Therefore, the wireless sensing model robustness detection method based on probability and statistical evaluation provided by the application is complementary to the existing robustness method of the design stage model, and when the failure of the deployment stage model is detected, the robustness of the model can be enhanced by using the robustness enhancement scheme of the design stage model. However, how to detect model failures remains the biggest challenge for the robustness of the deployment phase model.
As shown in fig. 1, the present embodiment provides a method for detecting robustness of a wireless sensing model based on probability and statistical evaluation, which includes the following steps:
step 1, training a bottom layer perception model
Generally, for a wireless sensing task, a group of training samples are collected and labeled with class labels, then signal features of different classes are extracted, and finally a machine learning algorithm is selected to train a wireless sensing model based on machine learning. Since the aim is to detect the robustness of a trained perceptual model in the deployment process, it is assumed that a wireless perceptual model with n classes is trained.
Step 2, calculating a probability vector [ c ] in the process of predicting the test sample by the wireless perception model1,c2,c3...cn]
When the trained wireless perception model is deployed in an actual environment, in order to monitor the robustness of the perception model in real time, for each test sample, a probability vector of the perception model in the process of predicting the test sample is calculated so as to evaluate the credibility of the prediction. The probability vector represents the probability that a test sample is attributed to each candidate class by the underlying wireless perception model, and the sum of the probabilities of all classes is 1. If the probability of the selected prediction class is more significant than that of other classes, the higher the reliability of the prediction, and the reliability of the prediction can be expressed by using a probability vector compared with the case of directly outputting the prediction result.
The probability vector can be obtained by calculating by using a predict _ proba method in a scimit-learn packet in python, wherein the scimit-learn packet in python almost comprises all machine learning algorithms, such as a support vector machine, a random forest, a decision tree and the like, and can meet the requirements of basic machine learning algorithms.
Although the probability vector can indicate the reliability of prediction, an abnormal probability vector may occur because the sum of the probabilities of all candidate classes must be 1. For example, a classification algorithm has two candidate classes, and for a test sample that does not belong to any one of the candidate classes, if the probability that the classification algorithm will assign the test sample to class 1 is low, e.g., c1At 0.0001, in order to satisfy the condition that the sum of all probabilities is 1, the test sample is assigned to the probability c of class 221-0.0001-0.9999, which results in a distorted probability vector c1,c2]＝[0.0001,0.9999]. Due to the fact thatThis is not sufficient to use only the probability vectors to represent the confidence of the prediction.
Step 3, calculating a statistical vector [ p ] in the process of predicting the test sample by the wireless perception model1,p2,p3...pn]
Due to the limitation that the sum of the probabilities in the probability vector is 1, statistical vectors are introduced by means of conformal prediction theory to compensate for this disadvantage. If the probability vector is said to represent the probability that a test sample belongs to each class as compared to other candidate classes, then the statistical vector represents the probability that a test sample belongs to each class as compared to the sample before each candidate class, and thus the statistical vector has no constraint of a sum of 1. The method specifically comprises the following substeps:
and 3.1, defining an inconsistency measurement function according to an inconsistency measurement theory of the conformal prediction theory and the machine learning algorithm in the step 1. The inconsistency measure function evaluates how different a test sample is from a previous set of samples. A greater inconsistency measure indicates that the test sample is more dissimilar to the previous sample;
step 3.2, define calibration data set
The calibration data set should be representative to reflect the condition of the training data set. Therefore, the training data set is first divided into k equal parts, k-1 of the k equal parts is used as the training set, the remaining k-th part is used as the calibration data set, and the process is repeated k times, and all samples in the training data set are subjected to one-time calibration data set. In the present embodiment, k is set to 10.
Step 3.3, calculate the inconsistency measure score
The inconsistency measure scores of the calibration data set and the test data set are calculated using the inconsistency measure function defined in step 3.1. For the calibration data set, the true tags are known, e.g., there are m calibration sets { [ x ]1,y1],[x2,y2],...,[xm,ym]Will result in an m 1 vector inconsistency measure score [ A11,A22...Amm]. For test data sets, unknown true tags, calculating inconsistencies with each candidate classThe score is measured. For example, there is a test sample S whose inconsistency measure score [ A ] is calculated with respect to n candidate classesS1,AS2...ASn]。
Step 3.4, calculating statistical vectors
The inconsistency measure scores of the calibration data sets are first sorted from small to large by category, and if there are t samples in each category, the sorted inconsistency measure scores of the calibration data sets are as follows { [ A ]11,A21...At1],[A12,A22...At2],....,[A1n,A2n...Atn]In which [ A ]11,A21...At1]Represents the sorted inconsistency measure score of the first type of sample in the calibration data set, [ A ]1n,A2n...Atn]Representing the sorted inconsistency measure score of the nth sample in the calibration data set. For the test samples, an inconsistency measure score in the calibration set is calculated that is at least the same proportion as the inconsistency measure score of the temporarily marked test samples. For example, a test sample is temporarily labeled as Category 1, with an inconsistency measure scored AS1Let A beS1 Category 1 inconsistency measure scores [ A ] arranged in the calibration data set11,A21...At1]The k-th bit of the test sample is (t +1-k)/t, and the statistical measurement value of the test sample belonging to the class 1 is repeated n times, so that the statistical measurement value of each class of the test sample belonging to the test sample, namely the statistical vector [ p ]1,p2...pn]。
Step 4, judging whether the prediction of the bottom layer perception model on the test sample is correct or not by using an anomaly detector
The anomaly detector essentially looks for a tight boundary in the feature space so that data points outside the boundary are considered anomalous events. In the specific experimental example of applicants' embodiment, the feature space is defined by the probability vectors and statistical vectors calculated in steps 3 and 4. In this embodiment, the anomaly detector is a one-class support vector machine (one-class SVM), and the boundary of the one-class SVM is a hyperplane for separating the normal (i.e., perceptual model training samples) data points in the feature space from the origin, so that the hyperplane boundary is as close to the normal data points as possible. During deployment, the anomaly detector checks whether the input test samples are within the boundaries constructed by the perceptual model training samples. If not, the test sample is considered to be an outlier, i.e., the prediction of the test sample by the underlying perceptual model is incorrect. The specific implementation steps are as follows:
step 4.1, train anomaly Detector
The training data set is divided into 10 equal parts, wherein 9 parts are used as the training set to train the wireless perception model, and then 1 part of the probability vectors and the statistical vectors are calculated according to the wireless perception model, so that the probability vectors and the statistical vectors of all the training data sets are obtained after repeating for 10 times. Concatenating the probability vector and the statistical vector into a vector c1,c2...cn,p1,p2...pn]The anomaly detector is trained as a feature. An anomaly detector is trained for each candidate class.
Step 4.2, calculating the probability vector and the statistical vector of the test sample
And calculating a probability vector and a statistical vector of the test sample according to the step 2 and the step 3.
Step 4.3, judging whether the prediction of the bottom layer perception model on the test sample is correct
Connecting the probability vector obtained in the step 4.2 and the statistical vector into a vector, inputting the vector into a candidate anomaly detector corresponding to the prediction label, and if the output is 1, the prediction is correct; otherwise the prediction is wrong.
Experimental design part:
in order to evaluate the method for detecting robustness of the wireless sensing model based on probability and statistical evaluation provided by the above embodiments, the applicant applies the method to 11 existing wireless sensing models based on machine learning, as shown in table 1, including different sensing tasks, different wireless signals, different signal characteristics, and different machine learning algorithms.
Table 1: wireless perception method of evaluation
Figure BDA0003245206730000101
1)WiG：He W，Wu K，Zou Y，et al.WiG:WiFi-Based Gesture Recognition System[C]//International Conference on Computer Communication&Networks.IEEE，2015。
2)WiAG：Virmani A,Shahzad M.Position and orientation agnostic gesture recognition using wifi[C]//Proceedings of the 15th Annual International Conference on Mobile Systems，Applications，and Services.2017：252-264。
3)TACT：Wang Y，Zheng Y.Modeling RFID signal reflection for contact-free activity recognition[J].Proceedings of the ACM on Interactive，Mobile，Wearable and Ubiquitous Technologies，2018，2(4):1-22。
4)AllSee：Kellogg B，Talla V，Gollakota S.Bringing gesture recognition to all devices[C]//11th{USENIX}Symposium on Networked Systems Design and Implementation({NSDI}14).2014：303-316。
5)EI：Jiang W，Miao C，Ma F，et al.Towards environment independent device free human activity recognition[C]//Proceedings of the 24th Annual International Conference on Mobile Computing and Networking.2018：289-304。
6)WiWho：Zeng Y，Pathak P H，Mohapatra P.WiWho：WiFi-based person identification in smart spaces[C]//2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks(IPSN).IEEE,2016：1-12。
7)WifiU：Wang W，Liu A X，Shahzad M.Gait recognition using wifi signals[C]//Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing.2016：363-373。
8)ibWrite：Liu J，Wang C，Chen Y，et al.VibWrite:Towards finger-input authentication on ubiquitous surfaces via physical vibration[C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security.2017：73-87。
9)Taprint：Chen W：Chen L，Huang Y，et al.Taprint：Secure text input for commodity smart wristbands[C]//The 25th Annual International Conference on Mobile Computing and Networking.2019：1-16。
10)UDO-Free：Ustev Y E，Durmaz Incel O，Ersoy C.User,device and orientation independent human activity recognition on mobile phones：Challenges and a proposal[C]//Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication.2013：1427-1436。
11)M-Touch：Song Y，Cai Z，Zhang Z L.Multi-touch authentication using hand geometry and behavioral information[C]//2017IEEE symposium on security and privacy(SP).IEEE，2017：357-372。
1. Experimental scenario
In the experiment, two scenes, an ideal controllable environment and a daily actual environment are set. The controlled environment is a 3.5m long by 2.5m wide by 2m high radio frequency darkroom as shown in FIG. 2. The wall, the ceiling and the floor in the darkroom can absorb the wireless signals, so that the influence of the uncontrollable multipath superposition of the signals on the experiment can be effectively reduced. The daily real environment is in a daily office and outdoor environment.
2. Experimental setup
The set up was performed according to the experiments in the original paper, with the perception task as a unit.
Gesture recognition: in this case study, six representative gestures are considered, including "push and pull", "draw a circle", "throw", "slide", "sweep", and "draw zig" glyphs, all of which are commonly supported by existing approaches. In a controlled environment, WiFi, RFID and ultrasonic signals are collected. For reproducibility, a programmable robotic arm is used to simulate gestures. There are five settings in the controlled environment, labeled S1 through S5 in fig. 3. For WiAG (a data enhancement method), WiFi signals are collected from the controlled environment and the daily office environment, respectively. In addition to assessing WiAG in a controlled environment at settings S1 through S5, two experimental variants were created, WiAG-C and WiAG-O. Following the experimental setup described in the WiAG original paper, WiAG-C and WiAG-O represent collected signals varying five different positions (positions at different distances and angles from the sender, labeled L1-L5) in a controlled environment and an office environment, respectively, as shown in FIG. 4.
Gait recognition: in this case study, gait data was collected from a controlled environment using WiFi signals for 15 volunteers (8 women) at the S1 to S5 settings.
And (3) activity recognition: in this case study, the input identifications of VibWrite and Taprint are shown using VibWrite-R and Taprint-R, respectively, and VibWrite-A and Taprint-A, respectively, the user identifications of VibWrite and Taprint, respectively, with reference to the settings of the original paper. The user identification methods of VibWrite and Taprint can predict which target users have entered text. Using data collected from smartphone Inertial Measurement Unit (IMU) sensors, UDO-Free can identify human activities, including "running," walking, "" cycling, "" standing, "and" sitting. The M-Touch uses data collected from IMU sensors for user authentication. The participants were also those 15 users who performed gait recognition.
3. Device setup
WiFi: two micro-PCs, each with an Intel 5300 Network Interface Card (NIC), are used as a transmitter and receiver to collect WiFi signals. Channel State Information (CSI) measurements are collected using an open source CSI measurement tool. As is conventional in the previous work, the sender is configured to send 1000 packets per second to the receiver.
RFID: signals were sent and received using an H47 RFID tag and a directional antenna powered by the Impinj R420 RFID reader. The same number of tags as in the original papers of the related methods were used, 4 for the TACT experiments and 3 for the allse experiments.
Ultrasonic: a modulated 19kHz ultrasound was transmitted using a commercial speaker (JBL Jembe) and the ultrasound signal was collected using a microphone (SAMSON meter mic). The loudspeaker and the microphone are connected to the notebook computer for data processing.
Vibration: with reference to the setup of the Vibwrite experiment, a vibration motor and a piezoelectric sensor were connected to a notebook computer to transmit and receive vibration signals.
Sensor data: for Taprint, 15 volunteers were asked to wear an LR G smart watch on the left hand and to tap the 12 knuckles of the left hand 50 times each with the right hand. Sensor data is collected using an Inertial Measurement Unit (IMU) of the smart watch. A millet smartphone is also used to run a customized android application to collect IMU sensor data for other tasks.
4. Environmental change
For each case, various changes to the environment were made by changing the location and execution of the activity, or adding furniture to introduce additional multiple paths, the specific changes being shown with reference to table 2 below.
Table 2: environmental Change for each case study
Figure BDA0003245206730000141
5. Evaluating settings
Consider two training test scenarios: static settings and dynamic settings. In a static setting, the training and test samples are from the same environment, e.g., for gesture recognition, both the training and test set samples are from the S1 environment. In a dynamic setting, data samples from different environments are mixed, including the environment used to collect the sensory model training samples and data from other environmental settings. Specifically, k-fold cross-validation is applied to data obtained from different environments, with the data collected for each environment being a set, the perceptual model is trained using the data collected for four of the environments, and then the trained model is tested on the data collected for the remaining environments.
6. Evaluation results
As shown in fig. 5, in the static setting, the prediction accuracy of all perception methods exceeds 93%. However, in dynamic settings, their performance may be affected, with this experiment observing an average drop in prediction accuracy of 22%. For wireless signals (such as WiFi and ultrasonic waves), the influence of environmental changes is also obvious, and the prediction precision is reduced by more than 40% through experimental findings. This is not surprising, since these signals are sensitive to environmental changes.
Fig. 6 shows the performance of the wireless perceptual model robustness detection method (RISE) based on probability and statistical evaluation, which is presented in the above embodiment, in detecting samples (i.e., drift samples) that are predicted incorrectly by the underlying perceptual model. For most perceptual methods, RISE gives an average accuracy of 92% (at least 89%). This means that it rarely filters out correctly predicted samples. For some perceptual approaches, such as Taprint-R, the average precision of RISE is 89%. This translates to a false positive rate of 3.7% (i.e., incorrectly predicting normal samples as drift samples), which means that RISE sometimes rejects correct perceptual prediction. This is found to be limited by the underlying model, which performs poorly in dynamic environments, and its predicted probability vector becomes noisy, which in turn affects the RISE filtered drift samples. Similar trends in recall and F1-score were also observed. Overall, the average accuracy of RISE was 94.5% and the recall was 92.3% in all case studies. The results showed that the false positive rate was 1.8% and the false negative rate (i.e. missed detection) was 7.7%.

Claims (2)

1. A wireless perception model robustness detection method based on probability and statistical evaluation is characterized by comprising the following steps:
step 1, training a bottom layer perception model
For a wireless perception task, firstly collecting a group of training samples and marking class labels, then extracting signal characteristics of different classes, and finally selecting a machine learning algorithm to train a wireless perception model based on machine learning;
step 2, calculating the probability vector of the wireless perception model in the process of predicting the test sample
When the trained wireless perception model is deployed in an actual environment, for monitoring the robustness of the perception model in real time, for each test sample, calculating a probability vector of the perception model in the process of predicting the test sample so as to evaluate the credibility of the prediction; the probability vector can be calculated by using a predict _ proba method in a scimit-spare packet in python;
step 3, calculating statistical vectors in the process of predicting test samples by the wireless perception model
Introducing a statistical vector, wherein the statistical vector is realized by the steps of:
step 3.1, defining an inconsistency measurement function according to an inconsistency measurement theory of the conformal prediction theory and the machine learning algorithm in the step 1; the inconsistency measure function evaluates that a test sample is many different from a previous set of samples, a greater inconsistency measure indicating that the test sample is more dissimilar from the previous set of samples;
step 3.2, define calibration data set
Dividing a training data set into equal k parts, using k-1 parts as the training set, using the remaining k-th part as a calibration data set, repeating the steps for k times, wherein k is 10, and all samples in the training data set are subjected to one-time calibration data set;
step 3.3, calculate the inconsistency measure score
The inconsistency measure scores of the calibration data set and the test data set are calculated using the inconsistency measure function defined in step 3.1:
for the calibration data set, knowing the true label, an inconsistency measurement score of a calibration set vector is obtained; for the test data set, calculating an inconsistency measurement score with each candidate class according to the unknown real label;
and 3.4, calculating a statistical vector.
Firstly, sorting the inconsistency measurement scores of the calibration data set from small to large according to categories; for the test samples, calculating a ratio of the inconsistency measure score in the calibration set to be at least the same as the inconsistency measure score of the temporarily marked test samples;
step 4, judging whether the prediction of the bottom layer perception model on the test sample is correct or not by using an anomaly detector
The anomaly detector mainly finds a tight boundary in the feature space, so that data points outside the boundary are considered as anomalous events;
the abnormal detector is a single-class support vector machine, the boundary of the abnormal detector is a hyperplane, and the hyperplane is used for separating a normal data point from an origin in a feature space, so that the boundary of the hyperplane is as close to the normal data point as possible; during deployment, the anomaly detector checks whether the input test sample is within the boundary constructed by the perceptual model training samples, and if not, the test sample is considered to be an outlier, i.e., the prediction of the test sample by the underlying perceptual model is incorrect.
2. The method of claim 1, wherein the anomaly detector determines that the prediction of the test sample by the underlying perceptual model is implemented by:
dividing the training data set into 10 equal parts, wherein 9 parts are used as the training set to train the wireless perception model, and then calculating 1 part of probability vectors and statistical vectors according to the wireless perception model, so that the probability vectors and the statistical vectors of all the training data sets are obtained by repeating for 10 times;
connecting the probability vector and the statistical vector into a vector as a feature training anomaly detector; training an anomaly detector for each candidate class;
connecting the obtained probability vector and the statistical vector into a vector, inputting the vector into a candidate anomaly detector corresponding to the prediction label, and if the output result is 1, the prediction is correct; otherwise the prediction is wrong.
CN202111030938.4A 2021-09-03 2021-09-03 Wireless perception model robustness detection method based on probability and statistical evaluation Active CN113837393B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111030938.4A CN113837393B (en) 2021-09-03 2021-09-03 Wireless perception model robustness detection method based on probability and statistical evaluation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111030938.4A CN113837393B (en) 2021-09-03 2021-09-03 Wireless perception model robustness detection method based on probability and statistical evaluation

Publications (2)

Publication Number Publication Date
CN113837393A true CN113837393A (en) 2021-12-24
CN113837393B CN113837393B (en) 2023-10-24

Family

ID=78962221

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111030938.4A Active CN113837393B (en) 2021-09-03 2021-09-03 Wireless perception model robustness detection method based on probability and statistical evaluation

Country Status (1)

Country Link
CN (1) CN113837393B (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108846412A (en) * 2018-05-08 2018-11-20 复旦大学 A kind of method of extensive zero sample learning
CN110717602A (en) * 2019-09-29 2020-01-21 南京大学 Machine learning model robustness assessment method based on noise data
CN111753986A (en) * 2020-06-28 2020-10-09 浙江工业大学 Dynamic testing method and device for deep learning model
CN111914919A (en) * 2020-07-24 2020-11-10 天津大学 Open set radiation source individual identification method based on deep learning
CN112036433A (en) * 2020-07-10 2020-12-04 天津城建大学 CNN-based Wi-Move behavior sensing method
US20200410403A1 (en) * 2019-06-27 2020-12-31 Royal Bank Of Canada System and method for detecting data drift

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108846412A (en) * 2018-05-08 2018-11-20 复旦大学 A kind of method of extensive zero sample learning
US20200410403A1 (en) * 2019-06-27 2020-12-31 Royal Bank Of Canada System and method for detecting data drift
CN110717602A (en) * 2019-09-29 2020-01-21 南京大学 Machine learning model robustness assessment method based on noise data
CN111753986A (en) * 2020-06-28 2020-10-09 浙江工业大学 Dynamic testing method and device for deep learning model
CN112036433A (en) * 2020-07-10 2020-12-04 天津城建大学 CNN-based Wi-Move behavior sensing method
CN111914919A (en) * 2020-07-24 2020-11-10 天津大学 Open set radiation source individual identification method based on deep learning

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
王科迪;易平;: "人工智能对抗环境下的模型鲁棒性研究综述", 信息安全学报, no. 03 *

Also Published As

Publication number Publication date
CN113837393B (en) 2023-10-24

Similar Documents

Publication Publication Date Title
Zou et al. Towards occupant activity driven smart buildings via WiFi-enabled IoT devices and deep learning
AU2016200905B2 (en) A system and method for identifying and analyzing personal context of a user
Ma et al. WiFi sensing with channel state information: A survey
Abdelnasser et al. A ubiquitous WiFi-based fine-grained gesture recognition system
Ahmed et al. Device free human gesture recognition using Wi-Fi CSI: A survey
Guo et al. HuAc: Human activity recognition using crowdsourced WiFi signals and skeleton data
Zou et al. Robust WiFi-enabled device-free gesture recognition via unsupervised adversarial domain adaptation
Jannat et al. Efficient Wi-Fi-based human activity recognition using adaptive antenna elimination
Xu et al. Attention-based gait recognition and walking direction estimation in wi-fi networks
Bu et al. TransferSense: towards environment independent and one-shot wifi sensing
Zhu et al. NotiFi: A ubiquitous WiFi-based abnormal activity detection system
Fang et al. Eyefi: Fast human identification through vision and wifi-based trajectory matching
Zou et al. Joint adversarial domain adaptation for resilient WiFi-enabled device-free gesture recognition
Hao et al. CSI‐HC: A WiFi‐Based Indoor Complex Human Motion Recognition Method
US10397736B2 (en) Mobile terminal
Zheng et al. Missile: A system of mobile inertial sensor-based sensitive indoor location eavesdropping
WO2018178207A1 (en) Apparatus and method
KR20190102507A (en) Display apparatus and operation method of the same
CN113837393B (en) Wireless perception model robustness detection method based on probability and statistical evaluation
Jawad et al. Human Activity and Gesture Recognition Based on WiFi Using Deep Convolutional Neural Networks
Konings et al. A stacked neural network-based machine learning framework to detect activities and falls within multiple indoor environments using Wi-Fi CSI
US20220262164A1 (en) Apparatus, method and system for person detection and identification utilizing wireless signals and images
US10496879B2 (en) Multiple-detection gesture recognition
Mostafa et al. A survey of indoor localization systems in multi-floor environments
Choujaa et al. Activity recognition from mobile phone data: State of the art, prospects and open problems

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant