CN113837393B - Wireless perception model robustness detection method based on probability and statistical evaluation - Google Patents
Wireless perception model robustness detection method based on probability and statistical evaluation Download PDFInfo
- Publication number
- CN113837393B CN113837393B CN202111030938.4A CN202111030938A CN113837393B CN 113837393 B CN113837393 B CN 113837393B CN 202111030938 A CN202111030938 A CN 202111030938A CN 113837393 B CN113837393 B CN 113837393B
- Authority
- CN
- China
- Prior art keywords
- vector
- perception model
- statistical
- probability
- samples
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000008447 perception Effects 0.000 title claims abstract description 62
- 238000010972 statistical evaluation Methods 0.000 title claims abstract description 14
- 238000001514 detection method Methods 0.000 title claims abstract description 11
- 239000013598 vector Substances 0.000 claims abstract description 78
- 238000012360 testing method Methods 0.000 claims abstract description 68
- 238000000034 method Methods 0.000 claims abstract description 54
- 238000012549 training Methods 0.000 claims abstract description 44
- 238000005259 measurement Methods 0.000 claims abstract description 40
- 238000010801 machine learning Methods 0.000 claims abstract description 31
- 230000008569 process Effects 0.000 claims abstract description 16
- 230000006870 function Effects 0.000 claims abstract description 14
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 13
- 238000012544 monitoring process Methods 0.000 claims description 5
- 238000012706 support-vector machine Methods 0.000 claims description 4
- 230000007613 environmental effect Effects 0.000 description 12
- 238000013461 design Methods 0.000 description 10
- 230000000694 effects Effects 0.000 description 10
- 238000005516 engineering process Methods 0.000 description 9
- 238000002474 experimental method Methods 0.000 description 9
- 230000008859 change Effects 0.000 description 5
- 238000011156 evaluation Methods 0.000 description 5
- 230000003068 static effect Effects 0.000 description 4
- 238000007635 classification algorithm Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000002354 daily effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000005021 gait Effects 0.000 description 3
- 238000002604 ultrasonography Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000006855 networking Effects 0.000 description 2
- 244000062793 Sorghum vulgare Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 230000001351 cycling effect Effects 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 235000019713 millet Nutrition 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/245—Classification techniques relating to the decision surface
- G06F18/2451—Classification techniques relating to the decision surface linear, e.g. hyperplane
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Software Systems (AREA)
- Mathematical Physics (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Probability & Statistics with Applications (AREA)
- Pure & Applied Mathematics (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Operations Research (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Mobile Radio Communication Systems (AREA)
Abstract
The application discloses a wireless perception model robustness detection method based on probability and statistical evaluation, which comprises the following steps of firstly training a bottom perception model, respectively calculating probability vectors and statistical vectors in the process of predicting test samples by the wireless perception model, wherein the implementation steps of the statistical vectors comprise: defining an inconsistency measurement function according to an inconsistency measurement theory of the conformal prediction theory and the machine learning algorithm in the step 1; the inconsistency measurement function evaluates that a test sample differs from a previous set of samples in a number, a larger inconsistency measurement indicating that the test sample is less similar to the previous set of samples; then defining a calibration data set, calculating an inconsistency measurement score and calculating a statistical vector; finally, an anomaly detector is used for judging whether the prediction of the underlying perception model on the test sample is correct. The method can be used on any machine learning based wireless awareness model to detect its robustness in the deployment phase.
Description
Technical Field
The application belongs to the field of robustness research of wireless perception schemes based on machine learning, and particularly relates to a wireless perception model robustness detection method based on probability and statistical evaluation.
Background
Wireless sensing technology is the basis for many emerging applications, such as smart home personalized customization, fall monitoring, emotion detection, vital sign monitoring, and the like. Wireless sensing technology refers to sensing the surrounding environment and human activity and physiological information by using ubiquitous wireless signals (e.g., wiFi, RFID, ultrasound, etc.). At present, wireless networks have been widely popularized in the global scope, and the manner of sensing the physical world by using wireless signals will significantly reduce the deployment cost, and make important breakthroughs in the aspects of usability, universality and the like. Existing wireless signals (acoustic, optical, radio frequency signals, etc.) in the environment may be used "in addition" to sensing the environment while performing the task (lighting, communication, etc.). Taking radio frequency signals as an example, the wireless signals generated by the signal transmitting end can generate physical phenomena such as direct irradiation, reflection, scattering and the like in the propagation process, so that a plurality of propagation paths are formed. In this way, the multipath superimposed signal formed at the signal receiving end carries spatial information reflecting the signal propagation process. The wireless sensing technology obtains the channel variation characteristic of the signal propagation process by analyzing the variation of the wireless signal in the propagation process, thereby realizing wide space physical and human physiological sensing.
Among all wireless sensing technologies, the machine learning-based wireless sensing technology has achieved a breakthrough progress. Machine learning techniques predict test data by learning the relationship of signal characteristics to target activity over a set of labeled training samples. The wireless sensing technology based on machine learning mainly relies on a precise algorithm of the machine learning technology to realize feature extraction and automatic classification, and the premise of the machine learning technology is that training data and test data are distributed consistently. However, wireless signals are highly susceptible to environmental changes, such as changes in the room layout that can cause changes in the reflected path of the signal, further resulting in a test data profile that differs significantly from the training data profile, which can lead to unreliable predictions for machine learning. This unreliable prediction is reflected in learning-based wireless sensing techniques where robustness is highly challenging.
In order to improve the robustness of the perception model (the trained model has high prediction accuracy for different environments is called robustness), scientific researchers have made a great deal of attempts and efforts, and the method can be roughly divided into two types, namely a data enhancement method; another class is the method of extracting more robust features. The data enhancement method mainly comprises the steps of designing by a professional or learning a conversion function by a neural network, and obtaining simulation data of different scenes by the conversion function, so that each scene can be ensured to have a machine learning model suitable for the scene, and further, the test data of the scene can have better robustness; the method for extracting the more robust features mainly extracts some more robust features irrelevant to the environment through a plurality of receiving devices or a domain self-adaptive network, for example, the action of "recruiting hands" can be recognized for people no matter how the environment changes, so that some features of "recruiting hands" can be found theoretically, and the features cannot change no matter how the environment changes. Once these environmentally invariant features are found, a robust perceptual model can be trained using these features and deployed to the actual environment.
While existing methods are effective for certain specific tasks or environmental changes, these methods can only address some of the foreseeable changes in model robustness at the system design stage. Neither of these methods yields good performance for unpredictable changes in the environment after deployment of the system, such as moving a chair or performing gestures where and how the gestures are performed. For the data enhancement approach, it is not possible to predict all environmental changes in advance and design a transfer function for each environmental change; for the method of extracting the robust features irrelevant to the environment, it is almost impossible to find the features with robustness to all environments, so the existing method is difficult to solve the problem of robustness of the wireless sensing model based on learning in the deployment stage, and improvement on the existing method or searching for a new method is needed to solve the problem.
Disclosure of Invention
Aiming at the problem of robustness of a wireless perception model based on machine learning in a deployment stage, the application aims to provide a wireless perception model robustness detection method based on probability and statistical evaluation.
In order to achieve the above task, the present application adopts the following technical solutions:
the wireless perception model robustness detection method based on probability and statistical evaluation is characterized by comprising the following steps of:
step 1, training a bottom layer perception model
For a wireless sensing task, firstly, collecting a group of training samples and marking class labels, then extracting signal characteristics of different classes, and finally, selecting a machine learning algorithm to train a wireless sensing model based on machine learning;
step 2, calculating probability vectors in the process of predicting test samples by the wireless perception model
When the trained wireless perception model is deployed in an actual environment, for monitoring the robustness of the perception model in real time, for each test sample, calculating a probability vector of the perception model in the process of predicting the test sample so as to evaluate the credibility of the prediction; the probability vector may be calculated by using the predictive_proba method in the scikit-learn packet in python;
step 3, calculating a statistical vector in the process of predicting test samples by the wireless perception model
Introducing a statistical vector, wherein the implementation step of the statistical vector comprises the following steps:
step 3.1, defining an inconsistency measurement function according to an inconsistency measurement theory of a conformal prediction theory and a machine learning algorithm in the step 1; the inconsistency measurement function evaluates that a test sample differs from a previous set of samples in a number, a larger inconsistency measurement indicating that the test sample is less similar to the previous set of samples;
step 3.2, defining a calibration data set.
Firstly dividing a training data set into k equal parts, then taking k-1 parts of the training data set as a training set and the rest k parts as a calibration data set, repeating the steps k times, wherein k=10, and all samples in the training data set are subjected to one calibration data set;
step 3.3, calculating an inconsistency measurement score
Calculating an inconsistency measure score for the calibration data set and the test data set using the inconsistency measure function defined in step 3.1:
for the calibration dataset, knowing the true tag will result in an inconsistency measurement score for the calibration dataset vector; for the test dataset, calculating an inconsistency measure score with each candidate class for the unknown real label;
and 3.4, calculating a statistical vector.
Firstly, sorting the inconsistent measurement scores of the calibration data set from small to large according to categories; calculating, for the test samples, a proportion of the inconsistency measurement scores in the calibration set that is at least the same as the inconsistency measurement scores of the temporarily marked test samples;
step 4, judging whether the prediction of the bottom layer perception model to the test sample is correct or not by using an anomaly detector
The anomaly detector primarily finds a tight boundary in the feature space, so that data points outside the boundary are considered anomalies;
the anomaly detector is a single-class support vector machine, and the boundary of the anomaly detector is a hyperplane and is used for separating normal data points in the characteristic space from an original point, so that the boundary of the hyperplane is as close to the normal data points as possible; during deployment, the anomaly detector checks whether the input test sample is within the boundaries constructed by the perception model training samples, and if not, considers the test sample to be an outlier, i.e., the underlying perception model's predictions of the test sample are incorrect.
According to the application, the method for realizing the prediction of the test sample by the anomaly detector for judging the underlying perception model comprises the following steps:
dividing a training data set into 10 equal parts, wherein 9 parts are used as training sets to train a wireless perception model, and then calculating the probability vectors and the statistical vectors of the remaining 1 part according to the wireless perception model, so that the probability vectors and the statistical vectors of all training data sets are obtained by repeating 10 times;
connecting the probability vector and the statistical vector into a vector serving as a feature training anomaly detector; training an anomaly detector for each candidate class;
connecting the obtained probability vector and the statistical vector into a vector, inputting the vector into an anomaly detector of a candidate class corresponding to the prediction label, and if the output result is 1, predicting correctly; otherwise the prediction is wrong.
Compared with the prior art, the wireless perception model robustness detection method based on probability and statistical evaluation has the following beneficial technical effects:
(1) The first method uses the combination of probability evaluation and statistical evaluation to detect the robustness of the wireless sensing system in the deployment stage;
(2) The first uses anomaly detector and classification reject strategy to improve the robustness of the machine learning-based wireless perception model;
(3) The method can be used on any machine learning-based wireless perception model to detect its robustness in the deployment phase; the method has good performance in 11 existing wireless perception models based on machine learning.
Drawings
Fig. 1 is a flowchart of the overall method for detecting robustness of a wireless perception model based on probability and statistical evaluation.
Fig. 2 is a diagram of a controllable environment setup.
Fig. 3 is a diagram of different environmental changes in a controlled environment.
Fig. 4 is a diagram of the environmental change in different positions and orientations.
Fig. 5 is the performance of the different models themselves in static and dynamic settings.
Fig. 6 is a graph of performance for detecting a drifting sample in different models.
The application is described in further detail below with reference to the drawings and specific examples.
Detailed Description
Aiming at the robustness problem of the wireless perception model in the deployment stage based on machine learning, firstly, the maximum difference between the robustness problem of the wireless perception model in the deployment stage and the robustness problem of the wireless perception model in the design stage are clarified. The robustness of the wireless perception model in the design stage mainly depends on the design of the feature extraction algorithm and the classification algorithm, and the robustness in the deployment stage mainly depends on the adaptability of the model to the environment, so that the robustness can be timely detected before the environment change has a great influence on the performance of the model. Thus, the applicant has proposed a new idea to detect the robustness of a machine learning based wireless sensor model in the deployment phase, i.e. to detect when the wireless sensor model will fail due to environmental changes in the deployment phase.
Generally, for a machine learning based wireless perception model, a predicted output is always given regardless of the input, but whether the output is correct is not guaranteed. Judging whether the prediction of the model is correct or not mainly according to the intermediate result of the calculation process of the machine learning algorithm and a probability and statistics evaluation method, and only outputting the correct prediction result. Thus, when a prediction of a model is determined to be a false prediction, the model is deemed to have failed in such an environment. Therefore, the method for detecting the robustness of the wireless perception model based on probability and statistical evaluation provided by the application complements the existing method for detecting the robustness of the design phase model, and when the occurrence of the failure of the deployment phase model is detected, the robustness of the design phase model can be enhanced by using a robustness enhancement scheme of the design phase model. However, for robustness of the deployment phase model, how to detect model failure remains the biggest challenge.
As shown in fig. 1, the present embodiment provides a method for detecting robustness of a wireless sensing model based on probability and statistical evaluation, which includes the following steps:
step 1, training a bottom layer perception model
Generally, for a wireless sensing task, a set of training samples are collected and labeled with class labels, then signal features of different classes are extracted, and finally a machine learning algorithm is selected to train a wireless sensing model based on machine learning. Since the objective is to detect the robustness of a trained perceptual model in the deployment process, it is assumed that a wireless perceptual model of n classes has been trained.
Step 2, calculating probability vector [ c ] in the process of predicting test samples by the wireless perception model 1 ,c 2 ,c 3 ...c n ]
When the trained wireless perception model is deployed in an actual environment, for monitoring the robustness of the perception model in real time, for each test sample, calculating a probability vector of the perception model in the process of predicting the test sample so as to evaluate the credibility of the prediction. The probability vector represents the probability that a test sample is attributed to each candidate class by the underlying wireless perception model, and the sum of the probabilities of all the classes is 1. If the probability of the selected prediction class is more remarkable than other classes, the reliability of the prediction is higher, and the reliability of the prediction can be represented by using a probability vector compared with the direct output of the prediction result.
The probability vector can be calculated by using a predictive_proba method in a scikit-learn packet in python, and the scikit-learn packet in python almost contains all machine learning algorithms, for example, a support vector machine, a random forest, a decision tree and the like, so that the requirements of the basic machine learning algorithm can be met.
Although the probability vector can represent the reliability of the prediction, since the sum of probabilities of all candidate classes must be 1, an abnormal probability vector sometimes occurs. For example, a classification algorithm has two candidate classes, and for a test sample that does not belong to any of the candidate classes, if the classification algorithm has a low probability of attributing the test sample to class 1, e.g., c 1 =0.0001, at this time, to satisfy the sum of all probabilities as 1, the test sample is assigned a probability c of class 2 2 =1-0.0001=0.9999, which results in a warped probability vector c 1 ,c 2 ]＝[0.0001,0.9999]. It is not enough to use only probability vectors to represent the reliability of the prediction.
Step 3, calculating a statistical vector [ p ] in the process of predicting test samples by the wireless perception model 1 ,p 2 ,p 3 ...p n ]
Since the sum of probabilities in the probability vector is limited to 1, a statistical vector is introduced by means of the conformal prediction theory to compensate the defect. If the probability vector is said to represent the probability that the test sample belongs to each class compared to the other candidate classes, then the statistical vector represents the probability that the test sample belongs to each class compared to the samples preceding each candidate class, so the statistical vector has no constraint of sum 1. The method specifically comprises the following substeps:
and 3.1, defining an inconsistency measurement function according to the inconsistency measurement theory of the conformal prediction theory and the machine learning algorithm in the step 1. The inconsistency measurement function evaluates how different a test sample is from a previous set of samples. The larger the inconsistency measure, the less similar the test sample is to the previous sample of that class;
step 3.2, defining a calibration data set
The calibration data set should be representative and reflect the situation of the training data set. Thus, the training data set is first divided into equal k shares, then k-1 of the k shares are used as the training set, the k remaining shares are used as the calibration data set, and the k times are repeated, and all samples in the training data set are subjected to one calibration data set. In this embodiment, k is set to 10.
Step 3.3, calculating an inconsistency measurement score
An inconsistency measure score for the calibration data set and the test data set is calculated using the inconsistency measure function defined in step 3.1. For calibration data sets, real tags are known, e.g., there are m calibration sets { [ x ] 1 ,y 1 ],[x 2 ,y 2 ],...,[x m ,y m ]An inconsistency measurement score of m 1 vector is obtained 11 ,A 22 ...A mm ]. For the test dataset, the real tags are not known, and an inconsistency measure score is calculated for each candidate class. For example, there is a test sample S, whose inconsistency measure scores [ A ] are calculated with respect to n candidate classes S1 ,A S2 ...A Sn ]。
Step 3.4, calculating a statistical vector
Firstly, sorting the inconsistency measurement scores of the calibration data sets from small to large according to categories, and if each category has t samples, the inconsistency measurement scores of the calibration data sets after sorting are shown as follows, { [ A ] 11 ,A 21 ...A t1 ],[A 12 ,A 22 ...A t2 ],....,[A 1n ,A 2n ...A tn ]And } wherein,[A 11 ,A 21 ...A t1 ]representing the non-uniformity measurement score of the first class of samples in the calibration dataset after ordering [ A ] 1n ,A 2n ...A tn ]Representing the non-uniformity measurement scores of the n-th class of samples in the calibration data set after ordering. For the test samples, calculating a ratio of the inconsistency measurement scores in the calibration set to at least the inconsistency measurement scores of the temporarily marked test samples. For example, test specimens are temporarily labeled as category 1, with an inconsistency measurement score of A S1 Suppose A S1 Inconsistent measurement scores [ A ] ranked in class 1 in calibration dataset 11 ,A 21 ...A t1 ]The statistical measure of the test sample belonging to class 1 is (t+1-k)/t, and is repeated n times to obtain a statistical measure of the test sample belonging to each class, i.e., a statistical vector [ p ] 1 ,p 2 ...p n ]。
Step 4, judging whether the prediction of the bottom layer perception model to the test sample is correct or not by using an anomaly detector
The anomaly detector primarily finds a tight boundary in the feature space, so that data points outside the boundary are considered anomalies. In the specific experimental example of the embodiment of the applicant, the feature space is defined by the probability vectors and the statistical vectors calculated in step 3 and step 4. In this embodiment, the anomaly detector is a single class support vector machine (one-class SVM) whose boundary is a hyperplane for separating normal (i.e., perceptual model training samples) data points in the feature space from the origin, such that the hyperplane boundary is as close as possible to the normal data points. During deployment, the anomaly detector checks whether the input test sample is within the boundaries constructed by the perceptual model training samples. If not, the test sample is considered an outlier, i.e., the underlying perceptual model predicts that the test sample is incorrect. The specific implementation steps are as follows:
step 4.1 training anomaly Detector
The training data set is divided into 10 equal parts, 9 parts are used as the training set to train the wireless perception model, and the surplus is calculated according to the wireless perception modelThe probability vector and the statistical vector of the next 1 part are repeated 10 times to obtain the probability vector and the statistical vector of all training data sets. Connecting the probability vector and the statistical vector into a vector [ c ] 1 ,c 2 ...c n ,p 1 ,p 2 ...p n ]The anomaly detector is trained as a feature. An anomaly detector is trained for each candidate class.
Step 4.2, calculating probability vector and statistical vector of the test sample
And (3) calculating probability vectors and statistical vectors of the test samples according to the step 2 and the step 3.
Step 4.3, judging whether the prediction of the bottom layer perception model to the test sample is correct
Connecting the probability vector and the statistical vector obtained in the step 4.2 into a vector, inputting the vector into an anomaly detector of a candidate class corresponding to the prediction label, and if the output is 1, the prediction is correct; otherwise the prediction is wrong.
Experiment design part:
in order to evaluate the wireless perception model robustness detection method based on probability and statistical evaluation provided by the above embodiment, the applicant applies the method to 11 existing wireless perception models based on machine learning, as shown in table 1, and the method comprises different perception tasks, different wireless signals, different signal characteristics and different machine learning algorithms.
Table 1: wireless sensing method for evaluation
1)WiG：He W，Wu K，Zou Y，et al.WiG:WiFi-Based Gesture Recognition System[C]//International Conference on Computer Communication&Networks.IEEE，2015。
2)WiAG：Virmani A,Shahzad M.Position and orientation agnostic gesture recognition using wifi[C]//Proceedings of the 15th Annual International Conference on Mobile Systems，Applications，and Services.2017：252-264。
3)TACT：Wang Y，Zheng Y.Modeling RFID signal reflection for contact-free activity recognition[J].Proceedings of the ACM on Interactive，Mobile，Wearable and Ubiquitous Technologies，2018，2(4):1-22。
4)AllSee：Kellogg B，Talla V，Gollakota S.Bringing gesture recognition to all devices[C]//11th{USENIX}Symposium on Networked Systems Design and Implementation({NSDI}14).2014：303-316。
5)EI：Jiang W，Miao C，Ma F，et al.Towards environment independent device free human activity recognition[C]//Proceedings of the 24th Annual International Conference on Mobile Computing and Networking.2018：289-304。
6)WiWho：Zeng Y，Pathak P H，Mohapatra P.WiWho：WiFi-based person identification in smart spaces[C]//2016 15th ACM/IEEE International Conference on Information Processing in Sensor Networks(IPSN).IEEE,2016：1-12。
7)WifiU：Wang W，Liu A X，Shahzad M.Gait recognition using wifi signals[C]//Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing.2016：363-373。
8)ibWrite：Liu J，Wang C，Chen Y，et al.VibWrite:Towards finger-input authentication on ubiquitous surfaces via physical vibration[C]//Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security.2017：73-87。
9)Taprint：Chen W：Chen L，Huang Y，et al.Taprint：Secure text input for commodity smart wristbands[C]//The 25th Annual International Conference on Mobile Computing and Networking.2019：1-16。
10)UDO-Free：Ustev Y E，Durmaz Incel O，Ersoy C.User,device and orientation independent human activity recognition on mobile phones：Challenges and a proposal[C]//Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication.2013：1427-1436。
11)M-Touch：Song Y，Cai Z，Zhang Z L.Multi-touch authentication using hand geometry and behavioral information[C]//2017IEEE symposium on security and privacy(SP).IEEE，2017：357-372。
1. Experimental scenario
In the experiment, two scenes, an ideal controllable environment and a daily practical environment are set. The controlled environment was a radio frequency darkroom of 3.5m (length) ×2.5m (width) ×2m (height), as shown in fig. 2. The wall, the ceiling and the floor in the darkroom can absorb the wireless signals, so that the influence of uncontrollable signal multipath superposition on experiments can be effectively reduced. The daily practical environment is in daily offices and outdoor environments.
2. Experimental setup
And setting by taking the perception task as a unit according to the experiment in the original paper.
Gesture recognition: in this case study, six representative gestures are considered, including "push and pull", "draw a circle", "throw", "slide", "sweep" and "draw zig" fonts, which are commonly supported by existing methods. In a controlled environment, wiFi, RFID, and ultrasonic signals are collected. For reproducibility, a programmable robotic arm is used to simulate gestures. There are five settings in the controlled environment, labeled S1 through S5 in fig. 3. For WiAG (a data enhancement method), wiFi signals are collected from the controlled environment and the everyday office environment, respectively. In addition to evaluating the WiAG under S1 to S5 settings in a controlled environment, two experimental variants, wiAG-C and WiAG-O, were created. Following the experimental setup described in the WiAG original paper, wiAG-C and WiAG-O represent the collection of signals under controlled and office environments, respectively, changing five different positions (positions at different distances and angles from the sender, labeled L1-L5), the specific positions being shown in fig. 4.
Gait recognition: in this case study, gait data for 15 volunteers (8 women) at the S1 to S5 settings were collected from the controlled environment using WiFi signals.
Activity recognition: in this case study, with reference to the settings of the original paper, vibWrite-R and Taprint-R are used to represent input identifications of VibWrite and Taprint, respectively, and VibWrite-A and Taprint-A are used to represent user identifications of VibWrite and Taprint, respectively. The user recognition method of VibWrite and Taprint can predict which target users have entered text. Using data collected from smartphone Inertial Measurement Unit (IMU) sensors, UDO-Free can identify human activities including "running", "walking", "cycling", "standing" and "sitting". M-Touch uses data collected from IMU sensors for user authentication. The participants were also those 15 users who identified gait.
3. Device arrangement
WiFi: two mini-PCs were used, each with an Intel 5300 Network Interface Card (NIC) as the transmitting and receiving end to collect WiFi signals. Channel State Information (CSI) measurements are collected using an open source CSI measurement tool. As is conventional in the prior art, the sender is configured to send 1000 packets per second to the receiver.
RFID: signals are transmitted and received using an H47 RFID tag and a directional antenna powered by an Impinj R420 RFID reader. The same number of tags as the original paper of the relevant method was used, 4 for the TACT experiment and 3 for the AllSee experiment.
Ultrasounds: a commercial speaker (JBL Jembe) was used to transmit a modulated 19kHz ultrasonic wave and a microphone (SAMSON Meteor mic) was used to collect the ultrasonic signal. The speaker and microphone are connected to the notebook computer for data processing.
And (3) carrying out the following steps: with reference to the setting of the Vibwrite experiment, a vibration motor and a piezoelectric sensor are connected to a notebook computer to transmit and receive vibration signals.
Sensor data: for pprint, 15 volunteers were asked to wear an LR G smartwatch on the left hand and tap the left hand's 12 knuckles 50 times each with the right hand. Sensor data is collected using an Inertial Measurement Unit (IMU) of the smart watch. The millet smart phone is also used to run a customized android application program to collect IMU sensor data for other tasks.
4. Environmental change
For each case, various changes to the environment were made by changing the location and manner of execution of the activity, or adding furniture to introduce additional multipaths, with specific changes being shown in table 2 below.
Table 2: environmental changes for each case study
5. Evaluation setting
Consider two training test scenarios: static settings and dynamic settings. In a static setting, the training and test samples are from the same environment, e.g., for gesture recognition, both the training set and test set samples are from the S1 environment. In a dynamic setting, data samples from different environments are mixed, including the environment used to collect the sensor model training samples and data from other environmental settings. Specifically, k-fold cross-validation is applied to data obtained from different environments, each set of data collected by each environment, the perceptual model is trained with data collected in four of the environments, and the trained model is then tested on data collected in the remaining environments.
6. Evaluation results
As shown in fig. 5, in the static setting, the prediction accuracy of all the sensing methods exceeds 93%. However, in dynamic settings, their performance may be affected, and the prediction accuracy is observed to drop 22% on average in this experiment. For wireless signals (such as WiFi and ultrasound), the effects of environmental changes are also apparent, and experiments find that the prediction accuracy is reduced by more than 40%. This is not surprising, as these signals are very sensitive to environmental changes.
Fig. 6 illustrates the performance of the wireless sensing model robustness detection method (hereinafter referred to as rim) based on probability and statistical evaluation according to the above embodiment to detect samples (i.e., drift samples) that are mispredicted by the underlying sensing model. For most perceptual methods, the sense gives an average accuracy of 92% (at least 89%). Meaning that it rarely filters out correctly predicted samples. For some perceptual approaches, such as Taprint-R, the average accuracy of RISE is 89%. This translates to a false positive rate of 3.7% (i.e. normal samples are erroneously predicted as drift samples), which means that the sense sometimes refuses the correct perceptual prediction. This is found to be limited by the underlying model, which performs poorly in dynamic environments, with its predictive probability vector becoming noisy, which in turn affects the RISE filtering of the drift samples. Similar trends in recall and F1-score were also observed. Overall, the average accuracy of the RISE was 94.5% and the recall was 92.3% in all case studies. The result showed that the false positive rate was 1.8%, and the false negative rate (i.e., omission) was 7.7%.
Claims (2)
1. The wireless perception model robustness detection method based on probability and statistical evaluation is characterized by comprising the following steps of:
step 1, training a bottom layer perception model
For a wireless sensing task, firstly, collecting a group of training samples and marking class labels, then extracting signal characteristics of different classes, and finally, selecting a machine learning algorithm to train a wireless sensing model based on machine learning;
step 2, calculating probability vectors in the process of predicting test samples by the wireless perception model
When the trained wireless perception model is deployed in an actual environment, for monitoring the robustness of the perception model in real time, for each test sample, calculating a probability vector of the perception model in the process of predicting the test sample so as to evaluate the credibility of the prediction; the probability vector is calculated by using the predictive_proba method in the scikit-learn packet in python;
step 3, calculating a statistical vector in the process of predicting test samples by the wireless perception model
Introducing a statistical vector, wherein the implementation step of the statistical vector comprises the following steps:
step 3.1, defining an inconsistency measurement function according to an inconsistency measurement theory of a conformal prediction theory and a machine learning algorithm in the step 1; the inconsistency measurement function evaluates that a test sample differs from a previous set of samples in a number, a larger inconsistency measurement indicating that the test sample is less similar to the previous set of samples;
step 3.2, defining a calibration data set
Firstly dividing a training data set into k equal parts, then taking k-1 parts of the training data set as a training set and the rest k parts as a calibration data set, repeating the steps k times, wherein k=10, and all samples in the training data set are subjected to one calibration data set;
step 3.3, calculating an inconsistency measurement score
Calculating an inconsistency measure score for the calibration data set and the test data set using the inconsistency measure function defined in step 3.1:
for the calibration dataset, knowing the true tag will result in an inconsistency measurement score for the calibration dataset vector; for the test dataset, calculating an inconsistency measure score with each candidate class for the unknown real label;
step 3.4, calculating a statistical vector
Firstly, sorting the inconsistent measurement scores of the calibration data set from small to large according to categories; calculating, for the test samples, a proportion of the inconsistency measurement scores in the calibration set that is at least the same as the inconsistency measurement scores of the temporarily marked test samples;
step 4, judging whether the prediction of the bottom layer perception model to the test sample is correct or not by using an anomaly detector
The anomaly detector primarily finds a tight boundary in the feature space, so that data points outside the boundary are considered anomalies;
the anomaly detector is a single-class support vector machine, and the boundary of the anomaly detector is a hyperplane and is used for separating normal data points in the characteristic space from an original point, so that the boundary of the hyperplane is as close to the normal data points as possible; during deployment, the anomaly detector checks whether the input test sample is within the boundaries constructed by the perception model training samples, and if not, considers the test sample to be an outlier, i.e., the underlying perception model's predictions of the test sample are incorrect.
2. The method of claim 1, wherein the method for the anomaly detector to determine the predictive implementation of the test sample by the underlying perceptual model is:
dividing a training data set into 10 equal parts, wherein 9 parts are used as training sets to train a wireless perception model, and then calculating the probability vectors and the statistical vectors of the remaining 1 part according to the wireless perception model, so that the probability vectors and the statistical vectors of all training data sets are obtained by repeating 10 times;
connecting the probability vector and the statistical vector into a vector serving as a feature training anomaly detector; training an anomaly detector for each candidate class;
connecting the obtained probability vector and the statistical vector into a vector, inputting the vector into an anomaly detector of a candidate class corresponding to the prediction label, and if the output result is 1, predicting correctly; otherwise the prediction is wrong.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111030938.4A CN113837393B (en) | 2021-09-03 | 2021-09-03 | Wireless perception model robustness detection method based on probability and statistical evaluation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111030938.4A CN113837393B (en) | 2021-09-03 | 2021-09-03 | Wireless perception model robustness detection method based on probability and statistical evaluation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113837393A CN113837393A (en) | 2021-12-24 |
| CN113837393B true CN113837393B (en) | 2023-10-24 |
Family
ID=78962221
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111030938.4A Active CN113837393B (en) | 2021-09-03 | 2021-09-03 | Wireless perception model robustness detection method based on probability and statistical evaluation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113837393B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108846412A (en) * | 2018-05-08 | 2018-11-20 | 复旦大学 | A kind of method of extensive zero sample learning |
| CN110717602A (en) * | 2019-09-29 | 2020-01-21 | 南京大学 | Machine learning model robustness assessment method based on noise data |
| CN111753986A (en) * | 2020-06-28 | 2020-10-09 | 浙江工业大学 | Dynamic testing method and device for deep learning model |
| CN111914919A (en) * | 2020-07-24 | 2020-11-10 | 天津大学 | Open set radiation source individual identification method based on deep learning |
| CN112036433A (en) * | 2020-07-10 | 2020-12-04 | 天津城建大学 | CNN-based Wi-Move behavior sensing method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA3085092A1 (en) * | 2019-06-27 | 2020-12-27 | Royal Bank Of Canada | System and method for detecting data drift |
-
2021
- 2021-09-03 CN CN202111030938.4A patent/CN113837393B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108846412A (en) * | 2018-05-08 | 2018-11-20 | 复旦大学 | A kind of method of extensive zero sample learning |
| CN110717602A (en) * | 2019-09-29 | 2020-01-21 | 南京大学 | Machine learning model robustness assessment method based on noise data |
| CN111753986A (en) * | 2020-06-28 | 2020-10-09 | 浙江工业大学 | Dynamic testing method and device for deep learning model |
| CN112036433A (en) * | 2020-07-10 | 2020-12-04 | 天津城建大学 | CNN-based Wi-Move behavior sensing method |
| CN111914919A (en) * | 2020-07-24 | 2020-11-10 | 天津大学 | Open set radiation source individual identification method based on deep learning |
Non-Patent Citations (1)
| Title |
|---|
| 人工智能对抗环境下的模型鲁棒性研究综述;王科迪;易平;;信息安全学报(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113837393A (en) | 2021-12-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zou et al. | Towards occupant activity driven smart buildings via WiFi-enabled IoT devices and deep learning | |
| US11763599B2 (en) | Model training method and apparatus, face recognition method and apparatus, device, and storage medium | |
| AU2016200905B2 (en) | A system and method for identifying and analyzing personal context of a user | |
| JP7265003B2 (en) | Target detection method, model training method, device, apparatus and computer program | |
| Abdelnasser et al. | A ubiquitous WiFi-based fine-grained gesture recognition system | |
| WO2020199926A1 (en) | Image recognition network model training method, image recognition method and device | |
| Zou et al. | Robust WiFi-enabled device-free gesture recognition via unsupervised adversarial domain adaptation | |
| US20200050745A1 (en) | Mobile terminal | |
| CN109918975A (en) | A kind of processing method of augmented reality, the method for Object identifying and terminal | |
| CN104160734A (en) | Measuring quality of experience associated with a mobile device | |
| Bu et al. | TransferSense: towards environment independent and one-shot wifi sensing | |
| Zou et al. | Joint adversarial domain adaptation for resilient WiFi-enabled device-free gesture recognition | |
| Hao et al. | CSI‐HC: A WiFi‐Based Indoor Complex Human Motion Recognition Method | |
| Ma et al. | RF-Siamese: Approaching accurate RFID gesture recognition with one sample | |
| CN113837393B (en) | Wireless perception model robustness detection method based on probability and statistical evaluation | |
| Ding et al. | Multimodal fusion-GMM based gesture recognition for smart home by WiFi sensing | |
| US20220262164A1 (en) | Apparatus, method and system for person detection and identification utilizing wireless signals and images | |
| US10496879B2 (en) | Multiple-detection gesture recognition | |
| Choujaa et al. | Activity recognition from mobile phone data: State of the art, prospects and open problems | |
| Ayinla et al. | SALLoc: An Accurate Target Localization In Wifi-Enabled Indoor Environments Via Sae-Alstm | |
| Lin et al. | Behavioral biometrics based on human-computer interaction devices | |
| Alkiek et al. | EarGest: Hand Gesture Recognition with Earables | |
| US20200314230A1 (en) | Mobile terminal | |
| Wang et al. | Contactless Continuous Activity Recognition based on Meta-Action Temporal Correlation in Mobile Environments | |
| Wei et al. | CATFSID: A few-shot human identification system based on cross-domain adversarial training |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |