CN117150229A - Data processing method, device, equipment and medium - Google Patents
Data processing method, device, equipment and medium Download PDFInfo
- Publication number
- CN117150229A CN117150229A CN202311108479.6A CN202311108479A CN117150229A CN 117150229 A CN117150229 A CN 117150229A CN 202311108479 A CN202311108479 A CN 202311108479A CN 117150229 A CN117150229 A CN 117150229A
- Authority
- CN
- China
- Prior art keywords
- data
- processed
- acquisition time
- current
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 30
- 230000002159 abnormal effect Effects 0.000 claims abstract description 98
- 238000012545 processing Methods 0.000 claims abstract description 46
- 238000000034 method Methods 0.000 claims abstract description 36
- 238000012937 correction Methods 0.000 claims abstract description 24
- 238000004590 computer program Methods 0.000 claims description 16
- 238000001514 detection method Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 description 15
- 238000004891 communication Methods 0.000 description 10
- 238000013480 data collection Methods 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 7
- 230000005856 abnormality Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
- G06F18/15—Statistical pre-processing, e.g. techniques for normalisation or restoring missing data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
- G06F18/232—Non-hierarchical techniques
- G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
-
- G—PHYSICS
- G07—CHECKING-DEVICES
- G07C—TIME OR ATTENDANCE REGISTERS; REGISTERING OR INDICATING THE WORKING OF MACHINES; GENERATING RANDOM NUMBERS; VOTING OR LOTTERY APPARATUS; ARRANGEMENTS, SYSTEMS OR APPARATUS FOR CHECKING NOT PROVIDED FOR ELSEWHERE
- G07C5/00—Registering or indicating the working of vehicles
- G07C5/008—Registering or indicating the working of vehicles communicating information to a remotely located station
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Testing And Monitoring For Control Systems (AREA)
Abstract
The invention discloses a data processing method, a device, equipment and a medium. The method comprises the following steps: acquiring a current data set to be processed corresponding to a vehicle; clustering each data in the current data set to be processed by a preset clustering algorithm, and detecting whether each data in the current data set to be processed is abnormal data or not according to a clustering result; after detecting that the target data is abnormal data, acquiring associated data corresponding to the target data from a current data set to be processed according to the data acquisition time of the target data, and correcting the target data according to the associated data; and storing the current data set to be processed after correction processing to a preset storage position. According to the embodiment of the invention, whether each data in the data set is abnormal data can be automatically detected according to the clustering result of the data in the data set, and the abnormal data in the data set can be automatically corrected based on the associated data corresponding to the abnormal data.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a data processing method, apparatus, device, and medium.
Background
In the running process of the vehicle, the vehicle-mounted remote information processor arranged on the vehicle can collect data related to the running state of the vehicle and respectively send the collected data sets to the server based on a wireless communication technology. The data in the data set sent to the server by the vehicle-mounted remote information processor is the vehicle networking data of the vehicle. The server is used for managing the vehicle. The server can determine the real state of the vehicle according to the data in the data set sent by the vehicle-mounted remote information processor, and provide accurate driving advice for the driver of the vehicle.
The vehicle-mounted remote information processor transmits more data, and abnormal data is difficult to avoid. After receiving the designated data set sent by the vehicle-mounted remote information processor, the data in the data set needs to be processed, and abnormal data in the designated data set is corrected. In the related art, a common data processing method scheme is as follows: processing the data in the data set according to the manual experience, detecting whether abnormal data exists in the data set, and correcting the abnormal data in the data set after detecting that the abnormal data exists in the data set. The data processing scheme in the related art depends on manual experience, the accuracy and the reliability of the data processing process are low, whether abnormal data exist in the data set or not cannot be accurately detected, and the abnormal data in the data set are corrected.
Disclosure of Invention
The invention provides a data processing method, a device, equipment and a medium, which are used for solving the problems that a data processing scheme in the related technology depends on manual experience, the accuracy and the reliability of a data processing process are low, whether abnormal data exist in a data set can not be accurately detected, and the abnormal data in the data set are corrected.
According to an aspect of the present invention, there is provided a data processing method including:
acquiring a current data set to be processed corresponding to a vehicle;
clustering each data in the current data set to be processed through a preset clustering algorithm, and detecting whether each data in the current data set to be processed is abnormal data or not according to a clustering result;
after detecting that the target data in the current data set to be processed is abnormal data, acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data, and correcting the target data according to the associated data;
and storing the current data set to be processed after correction processing to a preset storage position.
According to another aspect of the present invention, there is provided a data processing apparatus comprising:
The data set acquisition module is used for acquiring a current data set to be processed corresponding to the vehicle;
the data detection module is used for clustering each data in the current data set to be processed through a preset clustering algorithm, and detecting whether each data in the current data set to be processed is abnormal data or not according to a clustering result;
the data correction module is used for acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data after detecting that the target data in the current data set to be processed is abnormal data, and correcting the target data according to the associated data;
and the data set storage module is used for storing the current data set to be processed after correction processing to a preset storage position.
According to another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor;
and a memory communicatively coupled to the at least one processor;
wherein the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data processing method according to any one of the embodiments of the present invention.
According to another aspect of the present invention, there is provided a computer readable storage medium storing computer instructions for causing a processor to execute a data processing method according to any one of the embodiments of the present invention.
According to the technical scheme, the current data set to be processed corresponding to the vehicle is obtained; then clustering each data in the current data set to be processed through a preset clustering algorithm, and detecting whether each data in the current data set to be processed is abnormal data or not according to a clustering result; after detecting that the target data in the current data set to be processed is abnormal data, acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data, and correcting the target data according to the associated data; the method has the advantages that the current data set to be processed after correction processing is stored in the preset storage position, the problems that the accuracy and the reliability of a data processing scheme in the related technology are low, whether abnormal data exist in the data set or not cannot be accurately detected, the abnormal data in the data set are corrected are solved, whether each data in the data set are abnormal data can be automatically detected according to the clustering result of the data in the data set, the associated data corresponding to the abnormal data can be automatically determined according to the data acquisition time of the abnormal data in the data set, then the abnormal data in the data set are accurately corrected based on the associated data corresponding to the abnormal data, the accuracy and the reliability of the data processing process are improved, the actual state of a vehicle can be accurately determined based on the data in the data set after correction processing, and an accurate driving suggestion beneficial effect is provided for a driver of the vehicle.
It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the invention or to delineate the scope of the invention. Other features of the present invention will become apparent from the description that follows.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a data processing method according to a first embodiment of the present invention.
Fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention.
Fig. 3 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention.
Fig. 4 is a schematic structural diagram of an electronic device implementing a data processing method according to an embodiment of the present invention.
Detailed Description
In order that those skilled in the art will better understand the present invention, a technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in which it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
It should be noted that the terms "object," "first," "second," and the like in the description and the claims of the present invention and the above drawings are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
Example 1
Fig. 1 is a flowchart of a data processing method according to a first embodiment of the present invention. The embodiment is applicable to a case where after a data set sent by an on-board telematics processor of a vehicle is received, data in the data set is processed, and abnormal data in the data set is corrected. The method may be performed by a data processing apparatus. The data processing means may be implemented in the form of hardware and/or software. The data processing device can be configured in an electronic device. The electronic device may be a server for managing the vehicle. As shown in fig. 1, the method includes:
Step 101, acquiring a current data set to be processed corresponding to a vehicle.
Optionally, in the running process of the vehicle, a vehicle-mounted remote information processor arranged on the vehicle collects data related to the running state of the vehicle, and the collected data sets are respectively sent to the electronic equipment based on a wireless communication technology. The data in the data set sent to the electronic equipment by the vehicle-mounted remote information processor is the internet of vehicles data of the vehicle. Each data set sent to the electronic device by the vehicle-mounted telematics processor contains a specified type of data that the vehicle-mounted telematics processor regularly collects during a preset period of time. The specified type of data is related to the vehicle travel state and may be the speed, acceleration, or other type of data related to the vehicle travel state. Illustratively, a certain data set contains the speed of the vehicle that the on-board telematics processor periodically collects over a preset period of time. The preset duration is a preset duration in the vehicle-mounted telematics processor. The preset duration can be set according to the service requirement, and the embodiment of the invention is not particularly limited.
Optionally, after receiving each data set sent by the vehicle-mounted telematics processor, the electronic device needs to process data in each data set and correct abnormal data in each data set. The abnormal data in the data set may be data in which there is a large difference in the values in the data set from the values of other data. The current data set to be processed corresponding to the vehicle is the data set that the electronic device needs to process at the current moment.
Optionally, acquiring the current data set to be processed corresponding to the vehicle includes: and acquiring a current data set to be processed corresponding to the vehicle from the data set queue at regular time according to a preset time interval.
Optionally, the data set queue is a queue provided in the electronic device for storing respective data sets that the electronic device has not processed after receiving. After receiving each data set sent by the vehicle-mounted remote information processor, the electronic equipment stores each received data set into a data set queue. The preset time interval is a preset time interval in the electronic device. The electronic equipment executes the operation of acquiring the current data set to be processed corresponding to the vehicle from the data set queue once at preset time intervals. The preset time interval can be set according to the service requirement, and the embodiment of the invention is not particularly limited.
Optionally, the obtaining the current data set to be processed corresponding to the vehicle from the data set queue includes: acquiring a data set with earliest receiving time in each data set in a data set queue; and determining the data set with the earliest receiving time as the current data set to be processed corresponding to the vehicle. The time of receipt of the data set is the time the electronic device received the data set. The electronic device has stored therein the times of receipt of the respective data sets. The electronic device may determine, according to the receiving time of each data set in the data set queue, a data set with the earliest receiving time in each data set in the data set queue, obtain a data set with the earliest receiving time in each data set in the data set queue, and then determine the data set with the earliest receiving time in each data set in the data set queue as a current data set to be processed corresponding to the vehicle.
Step 102, clustering each data in the current data set to be processed through a preset clustering algorithm, and detecting whether each data in the current data set to be processed is abnormal data or not according to a clustering result.
Optionally, the preset clustering algorithm is a clustering algorithm preset in the electronic device and used for clustering each data in the current data set to be processed. Preset clustering algorithms include, but are not limited to, density-based clustering algorithms with noise (Density-Based Spatial Clustering of Applications with Noise, DBSCAN).
Optionally, clustering each data in the current data set to be processed by a preset clustering algorithm, and detecting whether each data in the current data set to be processed is abnormal data according to a clustering result, including: clustering each data in the current data set to be processed by a clustering algorithm based on density with noise to obtain a cluster and noise points corresponding to the current data set to be processed; and detecting whether each data in the current data set to be processed is abnormal data or not according to the noise point.
Optionally, the parameters of the density-based clustering algorithm with noise include a scan radius and a minimum inclusion point number. The parameters of the density-based clustering algorithm with noise can be preset and can also be determined according to the data characteristics of the current data set to be processed.
Optionally, clustering each data in the current data set to be processed by a density-based clustering algorithm with noise to obtain a cluster and a noise point corresponding to the current data set to be processed, including: selecting one piece of unaccessed data from each piece of data in the current data set to be processed, and finding out all other pieces of data with the distance within a scanning radius from the data in the current data set to be processed; forming a cluster comprising said data and all other data within a scanning radius from said data if the number of other data within a scanning radius from said data is greater than a minimum inclusion point, and marking said data as accessed; if the number of all other data within the scanning radius from the data is less than the minimum inclusion point number, marking the data as noise points and marking the data as accessed; if the number of all other data within the scanning radius from the data is equal to the minimum inclusion point number, marking the data as a boundary point and marking the data as accessed; and processing all the data which are not marked as accessed in the current data set to be processed by the method to obtain a cluster and a noise point corresponding to the current data set to be processed.
Alternatively, the data that is not accessed is data that is not marked as accessed. In the initial state, all data in the current data set to be processed are not marked as accessed. The clusters corresponding to the current data set to be processed are all formed clusters. The noise points corresponding to the current set of data to be processed are all data in the current set of data to be processed that are marked as noise points.
Optionally, detecting whether each data in the current to-be-processed data set is abnormal data according to the noise point includes: the following operations are performed for each data in the current set of pending data: judging whether the noise point has the same data as the data or not; if the noise point has the same data as the data, determining that the data is abnormal data; and if the data which is the same as the data does not exist in the noise point, determining that the data is not abnormal data.
Optionally, the noise points corresponding to the data set are all data in the data set marked as noise points. The abnormal data in the data set is data in which there is a large difference between the numerical value in the data set and the numerical value of other data. Typically, after clustering each data in the data set by a density-based clustering algorithm with noise, data in which a value in the data set is greatly different from that of other data will be marked as noise points, i.e. abnormal data in the data set will be marked as noise points. Therefore, if the data marked as noise points in the current data set to be processed is abnormal data in the current data set to be processed.
Optionally, for each data in the current data set to be processed, it is determined whether the same data as the data exists in the noise point corresponding to the current data set to be processed. If the noise point has the same data as the data, the data is the data marked as the noise point in the current data set to be processed, the numerical value of the data is greatly different from the numerical values of other data in the current data set to be processed, the data is abnormal data in the current data set to be processed, and the data can be determined to be the abnormal data. If the data which is the same as the data does not exist in the noise point, the data is not the data marked as the noise point in the current data set to be processed, the numerical value of the data is not greatly different from the numerical values of other data in the current data set to be processed, the data is not the abnormal data in the current data set to be processed, and the data can be determined to be not the abnormal data.
Optionally, the parameters of the density-based clustering algorithm with noise are determined based on the data characteristics of the current set of data to be processed. Specifically, the minimum contains points=2×the data dimension-1 of the current data set to be processed. The data dimension of the current set of data to be processed is the data dimension that affects the change in data in the current set of data to be processed. The data dimension of the current set of data to be processed may be sent to the electronic device by the vehicle-mounted telematics processor. Each data in the current data set to be processed is ordered according to the sequence of the data acquisition time. The data acquisition time of the data is the time when the vehicle-mounted remote information processor acquires the data. After the minimum containing point number is determined, determining the data with the sequencing serial number equal to the minimum containing point number in the current data set to be processed. And respectively calculating the distance between each other data and the data with the sequence number equal to the minimum containing point number. The distance between the data may be a euclidean distance between the data. Each other data is a respective data other than the data having the sorting order number equal to the minimum containing point number in the current data set to be processed. And sorting the distances according to the values from small to large, and determining the distance with the ratio of the sorting sequence number to the total number of the distances equal to 0.95 as the scanning radius. If no distance exists for which the ratio of the sequence number to the total number of distances is equal to 0.95, the distance for which the ratio of the sequence number to the total number of distances is closest to 0.95 is determined as the scan radius. The total number of distances is the total number of calculated distances. Illustratively, the data dimension of the current set of data to be processed is 2. And determining that the minimum inclusion point number is 3 according to the data dimension of the current data set to be processed. The distance between each other data and the data with the sequence number equal to 3 is calculated separately. And sorting the distances according to the values from small to large, and determining the distance with the ratio of the sorting sequence number to the total number of the distances equal to 0.95 as the scanning radius. If no distance exists for which the ratio of the sequence number to the total number of distances is equal to 0.95, the distance for which the ratio of the sequence number to the total number of distances is closest to 0.95 is determined as the scan radius.
Step 103, after detecting that the target data in the current data set to be processed is abnormal data, acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data, and correcting the target data according to the associated data.
Optionally, the target data is data of which any one of the current set of data to be processed is determined to be abnormal data. After detecting that the target data in the current data set to be processed is abnormal data, acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data, and correcting the target data according to the associated data. Thereby, correction is made for the data determined as the abnormal data for each of the current set of data to be processed.
Optionally, each data in the current data set to be processed is ordered according to the sequence of the data acquisition time. The data acquisition time of the data is the time when the vehicle-mounted remote information processor acquires the data. The data acquisition time of the target data is the time when the vehicle-mounted remote information processor acquires the target data. The vehicle-mounted remote information processor sends the current data set to be processed to the electronic equipment, and simultaneously sends the data acquisition time of each data in the current data set to be processed to the electronic equipment.
Optionally, the associated data corresponding to the target data includes first data acquired at a data acquisition time preceding the data acquisition time of the target data and second data acquired at a data acquisition time following the data acquisition time of the target data; acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data, correcting the target data according to the associated data, and comprising the following steps: detecting whether the current data set to be processed contains first data acquired at the data acquisition time before the data acquisition time of the target data and second data acquired at the data acquisition time after the data acquisition time of the target data according to the data acquisition time of each data in the current data set to be processed; and if the current data set to be processed contains first data acquired at the data acquisition time before the data acquisition time of the target data and second data acquired at the data acquisition time after the data acquisition time of the target data, determining that the abnormal cause of the target data is a data error, and correcting the target data according to the first data and the second data.
Optionally, the current data set to be processed includes a specified type of data that is periodically collected by the vehicle-mounted telematics processor within a preset duration of time at a target time interval. The target time interval is a preset time interval in the on-board telematics processor. The first data acquired at a data acquisition time preceding the data acquisition time of the target data is data of which the data acquisition time is equal to the data acquisition time preceding the data acquisition time of the target data. The second data acquired at the data acquisition time subsequent to the data acquisition time of the target data is data of which the data acquisition time is equal to the data acquisition time subsequent to the data acquisition time of the target data. The previous data acquisition time of the target data is a time that is equal to the target time interval and is earlier than the data acquisition time of the target data by a time difference from the data acquisition time of the target data. The latter data acquisition time of the target data is a time that is equal to the target time interval and later than the data acquisition time of the target data.
Alternatively, in general, if there is no missing data in the current data set to be processed, the current data set to be processed should include the first data acquired at the data acquisition time preceding the data acquisition time of the target data and/or the second data acquired at the data acquisition time following the data acquisition time of the target data.
Optionally, detecting, according to the data acquisition time of each data in the current data set to be processed, whether the current data set to be processed includes first data acquired at a data acquisition time previous to the data acquisition time of the target data and second data acquired at a data acquisition time subsequent to the data acquisition time of the target data, includes: inquiring data of which the data acquisition time is equal to the previous data acquisition time of the target data in the current data set to be processed according to the data acquisition time of each data in the current data set to be processed; if the data of the previous data acquisition time with the data acquisition time equal to the data acquisition time of the target data is queried, querying the data of the next data acquisition time with the data acquisition time equal to the data acquisition time of the target data in the current data set to be processed according to the data acquisition time of each data in the current data set to be processed; if the data of the data acquisition time which is equal to the data acquisition time of the target data and the data of the data acquisition time which is the next data acquisition time of the target data are queried, determining the first data which is acquired in the current data set to be processed and is included in the data acquisition time which is the previous data acquisition time of the target data and the second data which is acquired in the data acquisition time which is the next data acquisition time of the target data, and acquiring the first data and the second data from the current data set to be processed.
Alternatively, the abnormal cause of the data is a cause that causes a large difference in the value of the data from that of other data. Anomalies in data are either data errors or data deletions. The abnormality of data is due to data errors, which means that the data itself is erroneous data, and thus the value of data is greatly different from that of other data. The abnormality of the data is due to the data missing, which means that the data collected before and after the data collection time of the data is missing in the data set, so that the numerical value of the data is greatly different from that of other data.
Alternatively, in general, if there is a large difference between the value of the data and the value of the other data because the data itself is erroneous data, the data set may include first data acquired at a data acquisition time preceding the data acquisition time of the data and/or second data acquired at a data acquisition time following the data acquisition time of the data. If the data collection lacks data collected before and after the data collection time of the data, a large difference exists between the value of the data and the value of other data, and the data collection does not contain the first data collected at the data collection time before the data collection time of the data and the second data collected at the data collection time after the data collection time of the data. Thus, for abnormal data in the data set, if it is detected that the data set contains first data acquired at a data acquisition time preceding the data acquisition time of the data and/or second data acquired at a data acquisition time following the data acquisition time of the data, it may be determined that the abnormality of the data is due to a data error; if it is detected that the data set does not include the first data acquired at the data acquisition time preceding the data acquisition time of the data and the second data acquired at the data acquisition time following the data acquisition time of the data, it can be determined that the abnormality of the data is due to the data absence.
Optionally, if the current data set to be processed includes first data acquired at a data acquisition time previous to the data acquisition time of the target data and second data acquired at a data acquisition time subsequent to the data acquisition time of the target data, determining that an abnormal cause of the target data is a data error, and correcting the target data according to the first data and the second data.
Optionally, correcting the target data according to the first data and the second data includes: calculating an average value of the first data and the second data; and replacing the target data in the current data set to be processed with the average value.
Optionally, the first data and the second data are data acquired before and after a data acquisition time of the target data. Typically, the average of the first data and the second data is very close to the correct data actually acquired at the data acquisition time of the target data. Thus, the average of the first data and the second data may be regarded as correct data acquired at the data acquisition time of the target data. And calculating the average value of the first data and the second data, and replacing the target data in the current data set to be processed with the average value, so that the target data which has a large difference between the value of the data and the value of other data because the data is the wrong data is corrected, and the target data which is the wrong data in the current data set to be processed is corrected to be correct data.
Optionally, after querying, in the current to-be-processed data set, data of a data acquisition time equal to a data acquisition time of the target data and a data acquisition time previous to the data acquisition time of the target data according to the data acquisition time of each data in the current to-be-processed data set, the method further includes: if no data of the previous data acquisition time with the data acquisition time equal to the data acquisition time of the target data is queried, querying the current data set to be processed for data of the next data acquisition time with the data acquisition time equal to the data acquisition time of the target data according to the data acquisition time of each data in the current data set to be processed; and if no data with the data acquisition time equal to the data acquisition time of the target data is queried, determining that the current data set to be processed does not contain the first data acquired at the data acquisition time before the data acquisition time of the target data and the second data acquired at the data acquisition time after the data acquisition time of the target data.
Optionally, after detecting, according to the data acquisition time of each data in the current data set to be processed, whether the current data set to be processed includes the first data acquired at the data acquisition time before the data acquisition time of the target data and the second data acquired at the data acquisition time after the data acquisition time of the target data, the method further includes: if the current data set to be processed does not contain the first data acquired at the data acquisition time before the data acquisition time of the target data and the second data acquired at the data acquisition time after the data acquisition time of the target data, determining that the abnormal cause of the target data is data missing, and performing polynomial fitting on the data acquisition time of each data in the current data set to be processed and each data in the current data set to be processed to obtain a polynomial reflecting the functional relation between the data acquisition time and the data; determining first prediction data corresponding to a data acquisition time before the data acquisition time of the target data and second prediction data corresponding to a data acquisition time before the data acquisition time of the target data according to the polynomial; the first prediction data and the second prediction data are added to the current set of pending data.
Optionally, performing polynomial fitting on the data acquisition time of each data in the current data set to be processed and each data in the current data set to be processed to obtain a polynomial reflecting a functional relationship between the data acquisition time and the data, including: and performing polynomial fitting on the data acquisition time of each data in the current data set to be processed and each data in the current data set to be processed through a preset polynomial fitting algorithm to obtain a polynomial reflecting the functional relationship between the data acquisition time and the data. The data acquisition time is an independent variable of the polynomial, and the data acquired at the data acquisition time is an independent variable of the polynomial. Therefore, the data acquired at the data acquisition time can be obtained by substituting a certain data acquisition time into the polynomial.
Optionally, determining, according to the polynomial, first predicted data corresponding to a data acquisition time previous to the data acquisition time of the target data and second predicted data corresponding to a data acquisition time previous to the data acquisition time of the target data includes: substituting the previous data acquisition time of the target data into the polynomial to obtain first prediction data corresponding to the previous data acquisition time of the target data; substituting the previous data acquisition time of the target data into the polynomial to obtain second prediction data corresponding to the previous data acquisition time of the target data. The first predicted data corresponding to the data acquisition time preceding the data acquisition time of the target data is data acquired at the data acquisition time preceding the data acquisition time of the target data calculated from a polynomial. The second predicted data corresponding to the data acquisition time preceding the data acquisition time of the target data is data acquired at the data acquisition time following the data acquisition time of the target data calculated from the polynomial.
Optionally, the first predicted data and the second predicted data are added to the current data set to be processed, so that the target data, which has a large difference between the value of the data and the value of other data because the data acquired before and after the data acquisition time of the data is missing in the data set, is corrected, and the data acquired before and after the data acquisition time of the target data, which is missing in the current data set to be processed, is filled into the current data set to be processed.
Optionally, after querying, in the current to-be-processed data set, data of a data acquisition time equal to a data acquisition time of the target data and a data acquisition time previous to the data acquisition time of the target data according to the data acquisition time of each data in the current to-be-processed data set, the method further includes: if the data of the previous data acquisition time with the data acquisition time equal to the data acquisition time of the target data is queried, querying the data of the next data acquisition time with the data acquisition time equal to the data acquisition time of the target data in the current data set to be processed according to the data acquisition time of each data in the current data set to be processed; and if no data of the data acquisition time which is equal to the data acquisition time of the target data and is the data of the data acquisition time which is the next data acquisition time of the target data is queried, determining that the current data set to be processed only comprises the first data acquired at the data acquisition time which is the previous data acquisition time of the target data.
Optionally, if the current data set to be processed only includes the first data acquired at the data acquisition time before the data acquisition time of the target data, determining that the cause of the abnormality of the target data is a data error, and correcting the target data according to the first data.
Optionally, correcting the target data according to the first data includes: and replacing the target data in the current data set to be processed with the first data. The first data is data acquired before a data acquisition time of the target data. Typically, the first data is very close to the correct data actually acquired at the data acquisition time of the target data. Thus, the first data may be regarded as correct data acquired at the data acquisition time of the target data. And replacing the target data in the current data set to be processed with the first data, so as to correct the target data which has a large difference between the value of the data and the value of other data because the data is the wrong data, and correcting the target data which is the wrong data in the current data set to be processed into correct data.
Optionally, after querying, in the current to-be-processed data set, data of a data acquisition time equal to a data acquisition time of the target data and a data acquisition time previous to the data acquisition time of the target data according to the data acquisition time of each data in the current to-be-processed data set, the method further includes: if no data of the previous data acquisition time with the data acquisition time equal to the data acquisition time of the target data is queried, querying the current data set to be processed for data of the next data acquisition time with the data acquisition time equal to the data acquisition time of the target data according to the data acquisition time of each data in the current data set to be processed; and if the data of the data acquisition time which is equal to the data acquisition time of the target data and is the data of the data acquisition time which is the next data acquisition time of the target data are inquired, determining that the second data which is only contained in the current data set to be processed and is acquired at the data acquisition time which is the next data acquisition time of the target data.
Optionally, if the current data set to be processed only includes second data acquired at a data acquisition time subsequent to the data acquisition time of the target data, determining that an abnormal cause of the target data is a data error, and correcting the target data according to the second data.
Optionally, correcting the target data according to the second data includes: and replacing the target data in the current data set to be processed with the second data. The second data is data acquired after a data acquisition time of the target data. Typically, the second data is very close to the correct data actually acquired at the data acquisition time of the target data. Thus, the second data may be regarded as correct data acquired at the data acquisition time of the target data. And replacing the target data in the current data set to be processed with the second data, so as to correct the target data which has a large difference between the value of the data and the value of other data because the data is the wrong data, and correcting the target data which is the wrong data in the current data set to be processed into correct data.
Step 104, storing the current data set to be processed after correction processing to a preset storage position.
Optionally, after correcting the data determined as the abnormal data for each of the current data sets to be processed, a corrected current data set to be processed is obtained. And storing the current data set to be processed after correction processing to a preset storage position, so that the follow-up data in the current data set to be processed after correction processing can accurately determine the real state of the vehicle, and accurate driving advice can be provided for a driver of the vehicle.
Optionally, the preset storage location is a storage location provided in the electronic device for storing the processed data set. The preset storage location may be a data storage file. The data storage file is a file provided in the electronic device for storing the data set for which the processing is completed. And after correcting the data which is determined to be the abnormal data in each of the current data sets to be processed, obtaining a corrected current data set to be processed, and storing the corrected current data set to a data storage file.
Optionally, after storing the current data set to be processed after the correction processing in the preset storage position, confirming that the data processing process is finished, and waiting for the next data processing process to start.
According to the technical scheme, the current data set to be processed corresponding to the vehicle is obtained; then clustering each data in the current data set to be processed through a preset clustering algorithm, and detecting whether each data in the current data set to be processed is abnormal data or not according to a clustering result; after detecting that the target data in the current data set to be processed is abnormal data, acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data, and correcting the target data according to the associated data; the method has the advantages that the current data set to be processed after correction processing is stored in the preset storage position, the problems that the accuracy and the reliability of a data processing scheme in the related technology are low, whether abnormal data exist in the data set or not cannot be accurately detected, the abnormal data in the data set are corrected are solved, whether each data in the data set are abnormal data can be automatically detected according to the clustering result of the data in the data set, the associated data corresponding to the abnormal data can be automatically determined according to the data acquisition time of the abnormal data in the data set, then the abnormal data in the data set are accurately corrected based on the associated data corresponding to the abnormal data, the accuracy and the reliability of the data processing process are improved, the actual state of a vehicle can be accurately determined based on the data in the data set after correction processing, and an accurate driving suggestion beneficial effect is provided for a driver of the vehicle.
Example two
Fig. 2 is a flowchart of a data processing method according to a second embodiment of the present invention. Embodiments of the invention may be combined with various alternatives to one or more of the embodiments described above. As shown in fig. 2, the method includes:
step 201, a current data set to be processed corresponding to a vehicle is obtained.
Step 202, clustering each data in the current data set to be processed by a clustering algorithm based on density with noise to obtain a cluster and noise points corresponding to the current data set to be processed.
And 203, detecting whether each data in the current data set to be processed is abnormal data or not according to the noise points.
Step 204, after detecting that the target data in the current data set to be processed is abnormal data, acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data, and correcting the target data according to the associated data.
Step 205, storing the current data set to be processed after the correction processing to a preset storage position.
According to the technical scheme, the association data corresponding to the abnormal data can be automatically determined according to the data acquisition time of the abnormal data in the data set, then the abnormal data in the data set is accurately corrected based on the association data corresponding to the abnormal data, the accuracy and the reliability of a data processing process are improved, the follow-up accurate determination of the real state of the vehicle based on the data in the corrected data set is facilitated, and accurate driving advice is provided for a driver of the vehicle.
Example III
Fig. 3 is a schematic structural diagram of a data processing apparatus according to a third embodiment of the present invention. The apparatus may be configured in an electronic device. As shown in fig. 3, the apparatus includes: a data set acquisition module 301, a data detection module 302, a data correction module 303, and a data set storage module 304.
The data set obtaining module 301 is configured to obtain a current data set to be processed corresponding to the vehicle; the data detection module 302 is configured to cluster each data in the current data set to be processed by using a preset clustering algorithm, and detect whether each data in the current data set to be processed is abnormal data according to a clustering result; the data correction module 303 is configured to, after detecting that target data in the current data set to be processed is abnormal data, obtain associated data corresponding to the target data from the current data set to be processed according to a data acquisition time of the target data, and correct the target data according to the associated data; the data set storage module 304 is configured to store the corrected current data set to be processed to a preset storage location.
According to the technical scheme, the current data set to be processed corresponding to the vehicle is obtained; then clustering each data in the current data set to be processed through a preset clustering algorithm, and detecting whether each data in the current data set to be processed is abnormal data or not according to a clustering result; after detecting that the target data in the current data set to be processed is abnormal data, acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data, and correcting the target data according to the associated data; the method has the advantages that the current data set to be processed after correction processing is stored in the preset storage position, the problems that the accuracy and the reliability of a data processing scheme in the related technology are low, whether abnormal data exist in the data set or not cannot be accurately detected, the abnormal data in the data set are corrected are solved, whether each data in the data set are abnormal data can be automatically detected according to the clustering result of the data in the data set, the associated data corresponding to the abnormal data can be automatically determined according to the data acquisition time of the abnormal data in the data set, then the abnormal data in the data set are accurately corrected based on the associated data corresponding to the abnormal data, the accuracy and the reliability of the data processing process are improved, the actual state of a vehicle can be accurately determined based on the data in the data set after correction processing, and an accurate driving suggestion beneficial effect is provided for a driver of the vehicle.
In an optional implementation manner of the embodiment of the present invention, optionally, the data detection module 302 is specifically configured to: clustering each data in the current data set to be processed by a clustering algorithm based on density with noise to obtain a cluster and noise points corresponding to the current data set to be processed; and detecting whether each data in the current data set to be processed is abnormal data or not according to the noise point.
In an optional implementation manner of the embodiment of the present invention, optionally, when performing the operation of detecting whether each data in the current data set to be processed is abnormal data according to the noise point, the data detection module 302 is specifically configured to: the following operations are performed for each data in the current set of pending data: judging whether the noise point has the same data as the data or not; if the noise point has the same data as the data, determining that the data is abnormal data; and if the data which is the same as the data does not exist in the noise point, determining that the data is not abnormal data.
In an optional implementation manner of the embodiment of the present invention, optionally, the associated data corresponding to the target data includes first data acquired at a data acquisition time before a data acquisition time of the target data and second data acquired at a data acquisition time after the data acquisition time of the target data; the data correction module 303 is specifically configured to, when performing an operation of obtaining, according to the data collection time of the target data, associated data corresponding to the target data from the current data set to be processed, and correcting the target data according to the associated data: detecting whether the current data set to be processed contains first data acquired at the data acquisition time before the data acquisition time of the target data and second data acquired at the data acquisition time after the data acquisition time of the target data according to the data acquisition time of each data in the current data set to be processed; and if the current data set to be processed contains first data acquired at the data acquisition time before the data acquisition time of the target data and second data acquired at the data acquisition time after the data acquisition time of the target data, determining that the abnormal cause of the target data is a data error, and correcting the target data according to the first data and the second data.
In an optional implementation manner of the embodiment of the present invention, optionally, the data correction module 303 is specifically configured to, when performing an operation of correcting the target data according to the first data and the second data: calculating an average value of the first data and the second data; and replacing the target data in the current data set to be processed with the average value.
In an optional implementation of the embodiment of the present invention, optionally, the data correction module 303 is further configured to: if the current data set to be processed does not contain the first data acquired at the data acquisition time before the data acquisition time of the target data and the second data acquired at the data acquisition time after the data acquisition time of the target data, determining that the abnormal cause of the target data is data missing, and performing polynomial fitting on the data acquisition time of each data in the current data set to be processed and each data in the current data set to be processed to obtain a polynomial reflecting the functional relation between the data acquisition time and the data; determining first prediction data corresponding to a data acquisition time before the data acquisition time of the target data and second prediction data corresponding to a data acquisition time before the data acquisition time of the target data according to the polynomial; the first prediction data and the second prediction data are added to the current set of pending data.
In an optional implementation manner of the embodiment of the present invention, optionally, the data set acquisition module 301 is specifically configured to: and acquiring a current data set to be processed corresponding to the vehicle from the data set queue at regular time according to a preset time interval.
The specific manner in which the various modules perform the operations in the apparatus of the above embodiments have been described in detail in connection with the embodiments of the method, and will not be described in detail herein.
The data processing device can execute the data processing method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the data processing method.
Example IV
Fig. 4 shows a schematic diagram of an electronic device 10 that may be used to implement the data processing method of an embodiment of the invention. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.
As shown in fig. 4, the electronic device 10 includes at least one processor 11, and a memory, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, etc., communicatively connected to the at least one processor 11, in which the memory stores a computer program executable by the at least one processor, and the processor 11 may perform various appropriate actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program built into the Random Access Memory (RAM) 13 from the storage unit 18. In the RAM 13, various programs and data required for the operation of the electronic device 10 may also be stored. The processor 11, the ROM 12 and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.
Various components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, etc.; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
The processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, digital Signal Processors (DSPs), and any suitable processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as data processing methods.
In some embodiments, the data processing method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When a computer program is built into RAM 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data processing method in any other suitable way (e.g. by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
A computer program for carrying out data processing methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be implemented. The computer program may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.
The computing system may include clients and servers. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service are overcome.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present invention may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution of the present invention are achieved, and the present invention is not limited herein.
The above embodiments do not limit the scope of the present invention. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of the present invention.
Claims (10)
1. A method of data processing, comprising:
acquiring a current data set to be processed corresponding to a vehicle;
clustering each data in the current data set to be processed through a preset clustering algorithm, and detecting whether each data in the current data set to be processed is abnormal data or not according to a clustering result;
after detecting that the target data in the current data set to be processed is abnormal data, acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data, and correcting the target data according to the associated data;
And storing the current data set to be processed after correction processing to a preset storage position.
2. The data processing method according to claim 1, wherein clustering each data in the current data set to be processed by a preset clustering algorithm, detecting whether each data in the current data set to be processed is abnormal data according to a clustering result, comprises:
clustering each data in the current data set to be processed by a clustering algorithm based on density with noise to obtain a cluster and noise points corresponding to the current data set to be processed;
and detecting whether each data in the current data set to be processed is abnormal data or not according to the noise point.
3. The data processing method according to claim 2, wherein detecting whether each data in the current data set to be processed is abnormal data according to the noise point comprises:
the following operations are performed for each data in the current set of pending data:
judging whether the noise point has the same data as the data or not;
if the noise point has the same data as the data, determining that the data is abnormal data;
And if the data which is the same as the data does not exist in the noise point, determining that the data is not abnormal data.
4. The data processing method according to claim 1, wherein the associated data corresponding to the target data includes first data acquired at a data acquisition time preceding a data acquisition time of the target data and second data acquired at a data acquisition time following the data acquisition time of the target data;
acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data, correcting the target data according to the associated data, and comprising the following steps:
detecting whether the current data set to be processed contains first data acquired at the data acquisition time before the data acquisition time of the target data and second data acquired at the data acquisition time after the data acquisition time of the target data according to the data acquisition time of each data in the current data set to be processed;
and if the current data set to be processed contains first data acquired at the data acquisition time before the data acquisition time of the target data and second data acquired at the data acquisition time after the data acquisition time of the target data, determining that the abnormal cause of the target data is a data error, and correcting the target data according to the first data and the second data.
5. The data processing method according to claim 4, wherein correcting the target data based on the first data and the second data comprises:
calculating an average value of the first data and the second data;
and replacing the target data in the current data set to be processed with the average value.
6. The data processing method according to claim 4, wherein detecting, after the first data acquired at a data acquisition time preceding the data acquisition time of the target data and the second data acquired at a data acquisition time following the data acquisition time of the target data included in the current set of data to be processed according to the data acquisition time of each data in the current set of data to be processed, further comprises:
if the current data set to be processed does not contain the first data acquired at the data acquisition time before the data acquisition time of the target data and the second data acquired at the data acquisition time after the data acquisition time of the target data, determining that the abnormal cause of the target data is data missing, and performing polynomial fitting on the data acquisition time of each data in the current data set to be processed and each data in the current data set to be processed to obtain a polynomial reflecting the functional relation between the data acquisition time and the data;
Determining first prediction data corresponding to a data acquisition time before the data acquisition time of the target data and second prediction data corresponding to a data acquisition time before the data acquisition time of the target data according to the polynomial;
the first prediction data and the second prediction data are added to the current set of pending data.
7. The data processing method according to claim 1, wherein acquiring a current set of data to be processed corresponding to a vehicle includes:
and acquiring a current data set to be processed corresponding to the vehicle from the data set queue at regular time according to a preset time interval.
8. A data processing apparatus, comprising:
the data set acquisition module is used for acquiring a current data set to be processed corresponding to the vehicle;
the data detection module is used for clustering each data in the current data set to be processed through a preset clustering algorithm, and detecting whether each data in the current data set to be processed is abnormal data or not according to a clustering result;
the data correction module is used for acquiring associated data corresponding to the target data from the current data set to be processed according to the data acquisition time of the target data after detecting that the target data in the current data set to be processed is abnormal data, and correcting the target data according to the associated data;
And the data set storage module is used for storing the current data set to be processed after correction processing to a preset storage position.
9. An electronic device, the electronic device comprising:
at least one processor;
and a memory communicatively coupled to the at least one processor network;
wherein the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the data processing method of any one of claims 1-7.
10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions for causing a processor to implement the data processing method of any one of claims 1-7 when executed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311108479.6A CN117150229A (en) | 2023-08-30 | 2023-08-30 | Data processing method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311108479.6A CN117150229A (en) | 2023-08-30 | 2023-08-30 | Data processing method, device, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117150229A true CN117150229A (en) | 2023-12-01 |
Family
ID=88902020
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311108479.6A Pending CN117150229A (en) | 2023-08-30 | 2023-08-30 | Data processing method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117150229A (en) |
-
2023
- 2023-08-30 CN CN202311108479.6A patent/CN117150229A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210390802A1 (en) | Method, Computer Program And Device For Processing Signals | |
| CN114884813B (en) | Network architecture determining method and device, electronic equipment and storage medium | |
| CN117150229A (en) | Data processing method, device, equipment and medium | |
| CN115086384B (en) | Remote control test method, device, equipment and storage medium | |
| CN115219086A (en) | Pressure sensor identification method, device, equipment and storage medium | |
| CN114157486A (en) | Communication flow data abnormity detection method and device, electronic equipment and storage medium | |
| CN117492822B (en) | Change contrast method, device, electronic equipment and storage medium | |
| CN111176931A (en) | Operation monitoring method, operation monitoring device, server and storage medium | |
| CN115361308B (en) | Industrial control network data risk determination method, device, equipment and storage medium | |
| CN116991684B (en) | Alarm information processing method, device, equipment and medium | |
| CN116449810B (en) | Fault detection method and device, electronic equipment and storage medium | |
| CN117632294A (en) | Application program configuration method, device, equipment and medium | |
| CN117292843B (en) | Electrical signal data processing method, apparatus, device and storage medium | |
| CN114049615B (en) | Traffic object fusion association method and device in driving environment and edge computing equipment | |
| CN115791205A (en) | Vehicle detection method, device, equipment and storage medium | |
| CN116258769B (en) | Positioning verification method and device, electronic equipment and storage medium | |
| CN114694138B (en) | Road surface detection method, device and equipment applied to intelligent driving | |
| CN118243137A (en) | Vehicle mileage rollback processing method and device, electronic equipment and storage medium | |
| CN116089499A (en) | Data statistics method, device and medium based on kafka data volume | |
| CN116087894A (en) | Radar attitude detection method and device, electronic equipment and storage medium | |
| CN117194018A (en) | Processing method and device of system temperature control algorithm in multi-core and multi-chip environment | |
| CN115685171A (en) | Radar test method, apparatus, electronic device, medium, and computer program product | |
| CN117609801A (en) | Data comparison method, device, equipment and medium | |
| CN116534037A (en) | Fault detection method, device, equipment and storage medium | |
| CN117075132A (en) | Barrier tracking method, device, equipment and medium based on laser radar blind area |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |