CN116932631B - Big data-based detection data visual management system and method - Google Patents
Big data-based detection data visual management system and method Download PDFInfo
- Publication number
- CN116932631B CN116932631B CN202310881604.0A CN202310881604A CN116932631B CN 116932631 B CN116932631 B CN 116932631B CN 202310881604 A CN202310881604 A CN 202310881604A CN 116932631 B CN116932631 B CN 116932631B
- Authority
- CN
- China
- Prior art keywords
- data
- data point
- checked
- embedded computer
- marking
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 53
- 238000001514 detection method Methods 0.000 title claims abstract description 43
- 238000000034 method Methods 0.000 title claims abstract description 16
- 238000012545 processing Methods 0.000 claims abstract description 56
- 238000012937 correction Methods 0.000 claims abstract description 18
- 230000001502 supplementing effect Effects 0.000 claims abstract description 11
- 238000012800 visualization Methods 0.000 claims description 62
- 238000004364 calculation method Methods 0.000 claims description 34
- 238000007726 management method Methods 0.000 claims description 19
- 239000013589 supplement Substances 0.000 claims description 10
- 230000001174 ascending effect Effects 0.000 claims description 7
- 230000014759 maintenance of location Effects 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 6
- 238000013079 data visualisation Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 4
- 238000007689 inspection Methods 0.000 claims description 3
- 230000008439 repair process Effects 0.000 claims description 3
- 238000012163 sequencing technique Methods 0.000 description 6
- 230000009471 action Effects 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000013523 data management Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000009776 industrial production Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/215—Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- General Factory Administration (AREA)
Abstract
The invention relates to the technical field of big data, in particular to a detection data visual management system and method based on big data, comprising the following steps: the intelligent data acquisition system comprises a data detection module, a rough data output module, an auxiliary checking module, a data correction module and a visual supplementing module, wherein the data detection module is used for calculating the original data quantity and the processing capacity of a computer, the rough data output module is used for calculating the minimum data point quantity, outputting a rough image, the auxiliary checking module is used for detecting idle equipment, checking the data points by the computer of the idle equipment, the data correction module is used for carrying out data correction on the data points, and the visual supplementing module is used for repairing image details and outputting a complete chart.
Description
Technical Field
The invention relates to the technical field of big data, in particular to a detection data visual management system and method based on big data.
Background
Along with the development of information technology, the visual tool is increasingly used for the management of detection data with the characteristics of intuitiveness and convenience, and plays an important role in the aspects of industrial production, hazard investigation, medical detection and the like.
The visual processing flow of the detection information is as follows: the method comprises the steps of providing original data by a sensor in the device, calculating data points by an embedded computer in the device, delivering the processed data points to a visualization tool by the computer, and generating a chart after the visualization tool receives the data points.
However, when the equipment is busy, the condition that the data amount of the original data given by the sensor in the equipment is overlarge can occur, and the calculation capability is limited because the embedded computer is small in size, and although the calculation result can still be given, the accuracy of the result can not be ensured, so that the visual chart data is distorted, and the user is puzzled;
In addition, in order to ensure the continuity of the visual chart on the time axis, the computer must constantly process data, but when the data volume of the equipment is too large and the processing speed of the computer cannot keep up with the generating speed of the chart, the situation that the data is lost in a certain time period occurs in the visual chart, so that the chart is partially incomplete, and the visual effect is affected.
Disclosure of Invention
The invention aims to provide a system and a method for visual management of detection data based on big data, so as to solve the problems in the background technology.
In order to solve the technical problems, the invention provides the following technical scheme: a big data based detection data visualization management system, comprising: the system comprises a data detection module, a rough data output module, an auxiliary checking module, a data correction module and a visual supplementing module;
the data detection module is used for receiving the original data transmitted by the equipment sensor, calculating the data processing capacity of the corresponding embedded computer, and marking the equipment as a busy state after the data quantity exceeds the processing capacity of the computer;
The rough data output module is used for calculating the minimum data point number required for enabling the chart to be continuous according to the frequency information of the visual chart, calculating the acquisition characteristic value of the data points processed by the computer according to the history data of the equipment, sequencing the data points according to the acquisition characteristic value, sequentially delivering the data points to the visual tool according to the sequencing until the delivered data point number reaches the calculated minimum data point number, enabling the visual tool to output a rough image, storing all the data points, and marking the data points as data points to be checked;
The auxiliary checking module is used for detecting the working condition of other equipment connected with the system, finding out an idle computer in the auxiliary checking module, delivering stored original data to be checked to the idle computer for calculation, marking the calculated data points as auxiliary data points, and transmitting the auxiliary data points to the original computer;
The data correction module is used for comparing the auxiliary data point with the data point to be checked after the auxiliary data point is received by the original computer, correcting the data of the auxiliary data point according to the data point to be checked, and marking the corrected data point as a final data point;
The visualization supplementing module is used for marking the final data point with a time mark and then sending the data point to the visualization tool, the visualization tool adds the data point to the rough image at the corresponding time according to the time mark of the data point, repairs the details of the image, replaces the original rough image and outputs a complete visualization chart;
further, the data detection module includes: a data amount detection unit, a processing capability calculation unit and a state marking unit;
the data quantity detection unit is used for collecting the original data acquired by the equipment sensor and calculating the data quantity of the original data;
The processing capacity calculating unit is used for sending the test data packet, calculating the highest data volume which can be processed by the computer in unit time according to the processing speed of the computer, and recording the highest data volume as the processing capacity of the computer;
the state marking unit is used for comparing the data volume of the original data with the computer processing capacity, if the data volume of the original data is smaller than or equal to the computer processing capacity, the computer processing result is normally delivered to the visualization tool, and if the data volume of the original data is larger than the computer processing capacity, the equipment is marked as busy equipment and enters the management flow;
The invention can intelligently detect the working condition of the equipment and judge the working condition of the equipment;
further, the coarse data generating module includes: the device comprises a data point calculation unit, a data point generation unit, a credibility arrangement unit and a coarse image generation unit;
The data point calculation unit is used for calculating the minimum data point quantity required for making the chart continuous according to the historical fluctuation condition of the visual chart;
The data point generating unit is used for controlling the computer to calculate the original data, outputting a calculation result and marking all the calculated data points as data points to be checked;
The credibility arrangement unit is used for analyzing the historical data of the equipment, judging the credibility of each data point to be checked according to the historical operation condition of the equipment, and arranging the data points in ascending order according to the credibility to form a data point selection sequence;
The rough image generation unit is used for sequentially delivering the data points to the visualization tool according to the sequence of the data point selection until the quantity of the delivered data points reaches the minimum quantity of the data points, so that the visualization tool generates a rough image;
the method can calculate the minimum data point number generated by the visual chart, only upload partial data meeting the conditions to the visual tool, and output a rough chart so as to ensure that a user can see the chart at any time;
further, the auxiliary checking module comprises: an idle detection unit and an inspection unit;
The idle detection unit is used for detecting the working condition of other equipment, and when one piece of other equipment is idle, the embedded computer of the equipment is marked as an idle computer;
the checking calculation unit is used for delivering the original data to the idle computer, outputting a calculation result by the idle computer, and transmitting the calculated data to the system to be recorded as auxiliary data points;
The invention can intelligently judge the working states of other devices, send the original data to other idle computers, and carry out checking calculation by an external computer;
further, the data correction module includes: the data point comparison unit and the data point adjustment unit;
The data point comparison unit is used for calculating the acquisition characteristic value of each auxiliary data point, if the acquisition characteristic value of one auxiliary data point is higher than the corresponding data point to be checked, a correction command of the data point is sent out, and if the acquisition characteristic value of the auxiliary data point is lower than or equal to the corresponding data point to be checked, a retention command of the data point is sent out;
The data point adjusting unit is used for deleting the data point to be checked after receiving a correction command of one data point, marking the auxiliary data point at the corresponding position as a final data point, deleting the auxiliary data point after receiving a retention command of one data point, and directly marking the data point to be checked as the final data point;
the invention can compare the checking result with the stored data, correct the original data according to the checking result, and ensure the accuracy of the data;
further, the visual supplement module includes: a time marking unit and a coarse image perfecting unit;
The time marking unit is used for marking the final data point with a time mark, wherein the time mark comprises the number of the time period where the data point is located and the number of the data point in the time period;
The rough image perfecting unit is used for transmitting the final data point to the visualization tool, the visualization tool supplements and replaces the data point on the rough image at the corresponding time according to the time mark of the data point, perfects the detail of the rough image, and outputs the visualized chart after perfection;
according to the invention, after the data checking calculation is finished, the adjusted data is uploaded and refilled to the previous time point, so that the original rough chart is perfected, and the accuracy requirement of the visual image is met;
a visual management method of detection data based on big data comprises the following steps:
S100, after receiving the original data from the device sensor, calculating the data quantity of the original data and the data processing capacity of the embedded computer, and when the data quantity of the original data in a unit time period exceeds the data processing capacity of the computer, marking the device as a busy device in the current time period, and turning to step S200;
S200, processing the original data of the busy equipment in the step S100 by a computer to generate data points to be checked; calculating the minimum data point quantity required by the chart to be continuous according to the history generating record of the visual chart, calculating the letter collecting characteristic value of each data point to be checked according to the history record of the equipment, and sequencing the data points to be checked according to the letter collecting characteristic values; sequentially delivering the data points to be checked into the visualization tool until the number of the delivered data points reaches the minimum number of the data points, so that the visualization tool generates a rough image;
S300, detecting working conditions of other equipment, and marking an embedded computer of the idle equipment as an idle computer when the other equipment is idle; transmitting the original data to be checked stored in the step S200 to an idle computer for calculation, recording the calculated data points as auxiliary data points, and transmitting all the auxiliary data points to an embedded computer of the corresponding busy equipment;
S400, after receiving the auxiliary data points calculated in the step S300, the original computer calculates the acquisition characteristic value of each auxiliary data point, marks the auxiliary data point as a final data point if the acquisition characteristic value of one auxiliary data point is higher than the corresponding data point to be checked, deletes the data point to be checked at the corresponding position, and marks the data point to be checked at the corresponding position as the final data point if the acquisition characteristic value of the auxiliary data point is lower than or equal to the corresponding data point to be checked;
S500, marking the final data point obtained in the step S400 with a time mark, then sending the final data point to a visualization tool, supplementing and replacing the data point on the rough image in the corresponding time period according to the time mark of the final data point, perfecting the detail of the rough image, and outputting a complete visualization chart;
Further, step S100 includes:
S101, after equipment starts to work, collecting original data acquired by an equipment sensor in a unit time T, and calculating the data quantity S1 of the original data in the current time period;
step S102, the system sends a test data packet with the data quantity of C0 to an embedded computer of the equipment, the processing time T0 of the computer is detected, the processing capacity S2 of the embedded computer is calculated, and S2=C0 is T/T0;
Step S103, comparing the data quantity S1 of the original data in the current time period with the computer processing capacity S2, if S1< S2 or S1=S2, normally delivering the computer processing result to the visualization tool, if S1> S2, marking the device as a busy device in the current time period, and entering a data management flow;
further, step S200 includes:
step S201, a control computer calculates original data, data points output by the computer are marked as data points to be checked, a set of all data point values to be checked is marked as { P1, P2, …, pa }, wherein a represents the number of the data point in the current time period, and Pa represents the value of the data point to be checked with the number a;
S202, the system extracts a history record of a visual chart within a period of time Tn from the current time, wherein Tn represents a history record viewing range preset by the system, the number of time periods contained in the Tn time is recorded as n, and the total number of data points contained in all the time periods in the Tn time is recorded as R1; subtracting the values of every two adjacent data points in the same time period, if the subtracted values of the two adjacent data points exceed the range set by the system, marking the time period containing the two data points as a peak time period, marking the number of all peak time periods contained in Tn time as m, and marking the total number of the data points contained in all peak time periods as R2;
calculating a minimum number of data points R required to continue the chart, the Wherein R1> R2>0, n > m >0;
S203, collecting the value of each data point in Tn time in a historical record, and calculating the average value of the data points in each time period according to the number of the unit time period in the historical record, wherein the set of all the average values is { Q1, Q2, …, qn }, and Qn represents the average value of the data in the nth time period; calculating a sampling characteristic value K of each data point to be checked, and calculating according to the following formula:
Wherein i represents the number of the unit time period in Tn time;
s204, arranging the data points to be checked according to the ascending order of the acquired characteristic values to form a data point selection sequence, sequentially delivering the data points to the visualization tool in sequence until the number of the delivered data points reaches the minimum number of the data points, and enabling the visualization tool to generate a rough image;
further, step S300 includes:
Step S301, detecting the working condition of other equipment by the system, if the other equipment is idle, recording an embedded computer of the equipment as an idle computer, and if the other equipment is not idle, waiting until the equipment is idle;
step S302, transmitting the original data to an idle computer, outputting a calculation result after the idle computer calculates, recording the calculated data points as auxiliary data points, wherein the set of all the auxiliary data points is { F1, F2, …, fa }, wherein Fa represents the value of the auxiliary data point with the number of a, and transmitting all the auxiliary data points to the original computer;
Further, in step S400, the acquisition characteristic values K of all the auxiliary data points are calculated according to the formula in step S203;
Comparing the magnitude of the characteristic value of the data point to be checked and the auxiliary data point on each number, if the characteristic value of the auxiliary data point is higher than the data point to be checked on the same number, deleting the data point to be checked, and marking the auxiliary data point on the corresponding position as a final data point;
If the acquired characteristic value of one auxiliary data point is lower than the data point to be checked on the corresponding number, deleting the auxiliary data point, and marking the data point to be checked on the corresponding position as a final data point;
Further, in step S500, the system marks the final data point with a time stamp, where the time stamp includes the number of the time period in which the data point is located and the number of the data point in the time period, sends the marked final data point to the visualization tool, and the visualization tool supplements and replaces the data point on the rough image in the corresponding time period according to the time stamp of the data point, perfects the detail of the rough image, and outputs the visualization chart after being finished.
Compared with the prior art, the invention has the following beneficial effects:
1. The invention can intelligently detect the working condition of the equipment, judge the working state of the equipment, when the original data volume is overlarge and exceeds the data processing capacity of a computer, calculate the minimum data point quantity generated by a visualized chart according to the characteristic that the image fluctuates in a time period and the data points in the time period are more according to the condition of a history record, only upload partial data meeting the condition to the visualized chart, and output a rough chart so as to ensure that a user can see the chart at any time.
2. The invention can intelligently judge the working states of other devices, send the original data to other idle computers, carry out checking calculation by an external computer, compare the checking calculation result with stored data, correct the original data according to the checking calculation result and ensure the accuracy of the data.
3. According to the invention, after the data checking calculation is finished, the adjusted data can be uploaded and refilled to the previous time point, the original rough chart is perfected, the missing data of the visual image is complemented, and the details of the chart are enriched so as to meet the accuracy requirement of the visual image.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the invention, serve to explain the invention. In the drawings:
FIG. 1 is a schematic diagram of a visual management system for detection data based on big data;
FIG. 2 is a schematic diagram of steps of a method for visual management of detection data based on big data according to the present invention;
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, the present invention provides the following technical solutions: a big data based detection data visualization management system, comprising: the system comprises a data detection module, a rough data output module, an auxiliary checking module, a data correction module and a visual supplementing module;
the data detection module is used for receiving the original data transmitted by the equipment sensor, calculating the data processing capacity of the corresponding embedded computer, and marking the equipment as a busy state after the data quantity exceeds the processing capacity of the computer;
the data quantity detection unit is used for collecting the original data acquired by the equipment sensor and calculating the data quantity of the original data;
The processing capacity calculating unit is used for sending the test data packet, calculating the highest data volume which can be processed by the computer in unit time according to the processing speed of the computer, and recording the highest data volume as the processing capacity of the computer;
the state marking unit is used for comparing the data volume of the original data with the computer processing capacity, if the data volume of the original data is smaller than or equal to the computer processing capacity, the computer processing result is normally delivered to the visualization tool, and if the data volume of the original data is larger than the computer processing capacity, the equipment is marked as busy equipment and enters the management flow;
The rough data output module is used for calculating the minimum data point number required for enabling the chart to be continuous according to the frequency information of the visual chart, calculating the acquisition characteristic value of the data points processed by the computer according to the history data of the equipment, sequencing the data points according to the acquisition characteristic value, sequentially delivering the data points to the visual tool according to the sequencing until the delivered data point number reaches the calculated minimum data point number, enabling the visual tool to output a rough image, storing all the data points, and marking the data points as data points to be checked;
The rough data generation module includes: the device comprises a data point calculation unit, a data point generation unit, a credibility arrangement unit and a coarse image generation unit;
The data point calculation unit is used for calculating the minimum data point quantity required for making the chart continuous according to the historical fluctuation condition of the visual chart;
The data point generating unit is used for controlling the computer to calculate the original data, outputting a calculation result and marking all the calculated data points as data points to be checked;
The credibility arrangement unit is used for analyzing the historical data of the equipment, judging the credibility of each data point to be checked according to the historical operation condition of the equipment, and arranging the data points in ascending order according to the credibility to form a data point selection sequence;
The rough image generation unit is used for sequentially delivering the data points to the visualization tool according to the sequence of the data point selection until the quantity of the delivered data points reaches the minimum quantity of the data points, so that the visualization tool generates a rough image;
The auxiliary checking module is used for detecting the working condition of other equipment connected with the system, finding out an idle computer in the auxiliary checking module, delivering stored original data to be checked to the idle computer for calculation, marking the calculated data points as auxiliary data points, and transmitting the auxiliary data points to the original computer;
The auxiliary checking module comprises: an idle detection unit and an inspection unit;
The idle detection unit is used for detecting the working condition of other equipment, and when one piece of other equipment is idle, the embedded computer of the equipment is marked as an idle computer;
the checking calculation unit is used for delivering the original data to the idle computer, outputting a calculation result by the idle computer, and transmitting the calculated data to the system to be recorded as auxiliary data points;
The data correction module is used for comparing the auxiliary data point with the data point to be checked after the auxiliary data point is received by the original computer, correcting the data of the auxiliary data point according to the data point to be checked, and marking the corrected data point as a final data point;
the data correction module comprises: the data point comparison unit and the data point adjustment unit;
The data point comparison unit is used for calculating the acquisition characteristic value of each auxiliary data point, if the acquisition characteristic value of one auxiliary data point is higher than the corresponding data point to be checked, a correction command of the data point is sent out, and if the acquisition characteristic value of the auxiliary data point is lower than or equal to the corresponding data point to be checked, a retention command of the data point is sent out;
The data point adjusting unit is used for deleting the data point to be checked after receiving a correction command of one data point, marking the auxiliary data point at the corresponding position as a final data point, deleting the auxiliary data point after receiving a retention command of one data point, and directly marking the data point to be checked as the final data point;
The visualization supplementing module is used for marking the final data point with a time mark and then sending the data point to the visualization tool, the visualization tool adds the data point to the rough image at the corresponding time according to the time mark of the data point, repairs the details of the image, replaces the original rough image and outputs a complete visualization chart; further, the data detection module includes: a data amount detection unit, a processing capability calculation unit and a state marking unit;
the visual supplement module comprises: a time marking unit and a coarse image perfecting unit;
The time marking unit is used for marking the final data point with a time mark, wherein the time mark comprises the number of the time period where the data point is located and the number of the data point in the time period;
The rough image perfecting unit is used for transmitting the final data point to the visualization tool, the visualization tool supplements and replaces the data point on the rough image at the corresponding time according to the time mark of the data point, perfects the detail of the rough image, and outputs the visualized chart after perfection;
As shown in fig. 2, a method for visual management of detection data based on big data includes the following steps:
S100, after receiving the original data from the device sensor, calculating the data quantity of the original data and the data processing capacity of the embedded computer, and when the data quantity of the original data in a unit time period exceeds the data processing capacity of the computer, marking the device as a busy device in the current time period, and turning to step S200;
The step S100 includes:
S101, after equipment starts to work, collecting original data acquired by an equipment sensor in a unit time T, and calculating the data quantity S1 of the original data in the current time period;
step S102, the system sends a test data packet with the data quantity of C0 to an embedded computer of the equipment, the processing time T0 of the computer is detected, the processing capacity S2 of the embedded computer is calculated, and S2=C0 is T/T0;
Step S103, comparing the data quantity S1 of the original data in the current time period with the computer processing capacity S2, if S1< S2 or S1=S2, normally delivering the computer processing result to the visualization tool, if S1> S2, marking the device as a busy device in the current time period, and entering a data management flow;
S200, processing the original data of the busy equipment in the step S100 by a computer to generate data points to be checked; calculating the minimum data point quantity required by the chart to be continuous according to the history generating record of the visual chart, calculating the letter collecting characteristic value of each data point to be checked according to the history record of the equipment, and sequencing the data points to be checked according to the letter collecting characteristic values; sequentially delivering the data points to be checked into the visualization tool until the number of the delivered data points reaches the minimum number of the data points, so that the visualization tool generates a rough image;
Step S200 includes:
step S201, a control computer calculates original data, data points output by the computer are marked as data points to be checked, a set of all data point values to be checked is marked as { P1, P2, …, pa }, wherein a represents the number of the data point in the current time period, and Pa represents the value of the data point to be checked with the number a;
S202, the system extracts a history record of a visual chart within a period of time Tn from the current time, wherein Tn represents a history record viewing range preset by the system, the number of time periods contained in the Tn time is recorded as n, and the total number of data points contained in all the time periods in the Tn time is recorded as R1; subtracting the values of every two adjacent data points in the same time period, if the subtracted values of the two adjacent data points exceed the range set by the system, marking the time period containing the two data points as a peak time period, marking the number of all peak time periods contained in Tn time as m, and marking the total number of the data points contained in all peak time periods as R2;
calculating a minimum number of data points R required to continue the chart, the Wherein R1> R2>0, n > m >0;
S203, collecting the value of each data point in Tn time in a historical record, and calculating the average value of the data points in each time period according to the number of the unit time period in the historical record, wherein the set of all the average values is { Q1, Q2, …, qn }, and Qn represents the average value of the data in the nth time period; calculating a sampling characteristic value K of each data point to be checked, and calculating according to the following formula:
Wherein i represents the number of the unit time period in Tn time;
s204, arranging the data points to be checked according to the ascending order of the acquired characteristic values to form a data point selection sequence, sequentially delivering the data points to the visualization tool in sequence until the number of the delivered data points reaches the minimum number of the data points, and enabling the visualization tool to generate a rough image;
S300, detecting working conditions of other equipment, and marking an embedded computer of the idle equipment as an idle computer when the other equipment is idle; transmitting the original data to be checked stored in the step S200 to an idle computer for calculation, recording the calculated data points as auxiliary data points, and transmitting all the auxiliary data points to an embedded computer of the corresponding busy equipment;
step S300 includes:
Step S301, detecting the working condition of other equipment by the system, if the other equipment is idle, recording an embedded computer of the equipment as an idle computer, and if the other equipment is not idle, waiting until the equipment is idle;
step S302, transmitting the original data to an idle computer, outputting a calculation result after the idle computer calculates, recording the calculated data points as auxiliary data points, wherein the set of all the auxiliary data points is { F1, F2, …, fa }, wherein Fa represents the value of the auxiliary data point with the number of a, and transmitting all the auxiliary data points to the original computer;
S400, after receiving the auxiliary data points calculated in the step S300, the original computer calculates the acquisition characteristic value of each auxiliary data point, marks the auxiliary data point as a final data point if the acquisition characteristic value of one auxiliary data point is higher than the corresponding data point to be checked, deletes the data point to be checked at the corresponding position, and marks the data point to be checked at the corresponding position as the final data point if the acquisition characteristic value of the auxiliary data point is lower than or equal to the corresponding data point to be checked;
In step S400, calculating the acquisition characteristic values K of all the auxiliary data points according to the formula in step S203;
Comparing the magnitude of the characteristic value of the data point to be checked and the auxiliary data point on each number, if the characteristic value of the auxiliary data point is higher than the data point to be checked on the same number, deleting the data point to be checked, and marking the auxiliary data point on the corresponding position as a final data point;
If the acquired characteristic value of one auxiliary data point is lower than the data point to be checked on the corresponding number, deleting the auxiliary data point, and marking the data point to be checked on the corresponding position as a final data point;
S500, marking the final data point obtained in the step S400 with a time mark, then sending the final data point to a visualization tool, supplementing and replacing the data point on the rough image in the corresponding time period according to the time mark of the final data point, perfecting the detail of the rough image, and outputting a complete visualization chart;
In step S500, the system marks the final data point with a time stamp, where the time stamp includes the number of the time period in which the data point is located and the number of the data point in the time period, sends the marked final data point to the visualization tool, and the visualization tool supplements and replaces the data point on the rough image in the corresponding time period according to the time stamp of the data point, perfects the detail of the rough image, and outputs the completed visualization chart.
Examples:
the system sets a unit time T=1 minute, records a viewing range Tn=10 minutes, receives the original data from the equipment sensor every 1 minute, calculates the data quantity S1=20MB of the original data, and marks the equipment as busy because of S1> S2 due to the data processing capability S2=10MB of the embedded computer in 1 minute;
The embedded computer calculates the original data, calculates 5 data points to be checked, the numbers are 1, 2, 3,4 and 5 respectively, and the values of the data points to be checked are respectively: 5. 10, 30, 5, 15;
The system extracts a history record within 10 minutes from the current time, analyzes the history record, obtains 10 unit time periods in the history record, wherein the number of all data points in the 10 unit time periods is R1=40, 2 peak time periods are obtained in the history record within 10 minutes, the number of the data points in the peak time period is R2=16, and the minimum number of the data points is obtained A plurality of;
the system uses the extracted history record Calculating the characteristic values of data acquired from each number, wherein the characteristic values of data points with the numbers of 1,2, 3, 4 and 5 are 8, 5, 20, 8 and 10 respectively, and the system delivers only 3 data points with the numbers of 1,2 and 4 to the visualization tool and outputs a rough chart;
When the system waits for other equipment to be idle, the original data is sent to an embedded computer of the equipment to calculate 5 auxiliary data points, the calculated result is sent to the original computer, after the value of the data points is corrected, the corrected data points are marked with time marks and uploaded to a visualization tool, the visualization tool supplements and replaces the data points on the rough image in the corresponding time period according to the time marks of the data points, the details of the rough image are perfected, and a visualization chart after completion is output.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus.
Finally, it should be noted that: the foregoing description is only a preferred embodiment of the present invention, and the present invention is not limited thereto, but it is to be understood that modifications and equivalents of some of the technical features described in the foregoing embodiments may be made by those skilled in the art, although the present invention has been described in detail with reference to the foregoing embodiments. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (6)
1. A method for visual management of detection data based on big data, the method comprising the steps of:
s100, after receiving the original data from the device sensor, calculating the data quantity of the original data and the data processing capacity of the embedded computer, and when detecting that the data quantity of the original data exceeds the data processing capacity of the embedded computer in the current time period, marking the device as a busy device in the current time period, and turning to step S200;
S200, processing the original data of the busy equipment in the step S100 by the embedded computer to generate a data point to be checked; calculating the minimum data point number required for making the chart continuous according to the history generating record of the visual chart, calculating the letter picking characteristic value of each data point to be checked according to the history record of the equipment, sorting the data points to be checked in ascending order according to the size of the letter picking characteristic value, sequentially delivering the data points to be checked to the visual tool until the delivered data point number reaches the minimum data point number, and enabling the visual tool to generate a rough image;
S300, detecting working conditions of other equipment, and marking an embedded computer of the idle equipment as an idle embedded computer when the other equipment is idle; sending the original data in the step S200 to an idle embedded computer for calculation, recording the calculated data points as auxiliary data points, and sending all the auxiliary data points to the embedded computers of the corresponding busy devices;
S400, after receiving the auxiliary data points calculated in the step S300, the embedded computer of the corresponding busy equipment calculates the acquisition characteristic value of each auxiliary data point, marks the auxiliary data point as a final data point if the acquisition characteristic value of one auxiliary data point is higher than the corresponding data point to be checked, deletes the data point to be checked in the corresponding position, and marks the data point to be checked in the corresponding position as the final data point if the acquisition characteristic value of the auxiliary data point is lower than or equal to the corresponding data point to be checked;
S500, marking the final data point obtained in the step S400 with a time mark, then sending the final data point to a visualization tool, supplementing and replacing the data point on the rough image in the corresponding time period according to the time mark of the final data point, perfecting the detail of the rough image, and outputting a complete visualization chart;
Step S200 includes:
Step S201, controlling an embedded computer to calculate original data, marking data points output by the embedded computer as data points to be checked, and marking a set of all data point values to be checked as { P1, P2, …, pa }, wherein a represents the number of the data point in the current time period, and Pa represents the value of the data point to be checked with the number of a;
S202, extracting a history record of a visual chart in a time Tn from the current time, wherein Tn represents a preset history record viewing range, the number of time periods contained in the Tn is recorded as n, and the total number of data points contained in all the time periods in the Tn is recorded as R1; subtracting the values of every two adjacent data points in the same time period, if the subtracted values of the two adjacent data points exceed a set range, marking the time period containing the two data points as a peak time period, marking the number of all peak time periods contained in Tn time as m, and marking the total number of the data points contained in all peak time periods as R2;
calculating a minimum number of data points R required to continue the chart, the Wherein R1> R2>0, n > m >0;
S203, collecting the value of each data point in Tn time in a historical record, and calculating the average value of the data points in each time period according to the number of the unit time period in the historical record, wherein the set of all the average values is { Q1, Q2, …, qn }, and Qn represents the average value of the data in the nth time period; calculating a sampling characteristic value K of each data point to be checked, and calculating according to the following formula:
Wherein i represents the number of the unit time period in Tn time;
S204, arranging the data points to be checked according to the ascending order of the acquired characteristic values to form a data point selection sequence, sequentially delivering the data points to the visualization tool until the number of the delivered data points reaches the minimum number of the data points, and enabling the visualization tool to generate a rough image.
2. The visual management method for detection data based on big data according to claim 1, wherein:
The step S100 includes:
s100, after equipment starts to work, collecting original data acquired by an equipment sensor in a unit time T, and calculating the data quantity S1 of the original data in the current time period;
Step S102, sending a test data packet with the data quantity of C0 to an embedded computer of the equipment, detecting the processing time T0 of the embedded computer, and calculating the processing capacity S2 of the embedded computer, wherein S2=C0 is T/T0;
step S103, comparing the data quantity S1 of the original data in the current time period with the processing capacity S2 of the embedded computer, if S1< S2 or S1=S2, normally delivering the processing result of the embedded computer to the visualization tool, and if S1> S2, marking the device as a busy device in the current time period, and turning to step S200.
3. The visual management method for detection data based on big data according to claim 1, wherein: step S300 includes:
Step S301, detecting working conditions of other equipment, if the other equipment is idle, marking an embedded computer of the equipment as an idle embedded computer, and if the other equipment is not idle, waiting until the equipment is idle;
Step S302, transmitting the original data to an idle embedded computer, outputting a calculation result after the idle embedded computer calculates, recording the calculated data points as auxiliary data points, wherein the set of all the auxiliary data points is { F1, F2, …, fa }, wherein Fa represents the value of the auxiliary data point with the number of a, and transmitting all the auxiliary data points to the original embedded computer.
4. The visual management method for detection data based on big data according to claim 1, wherein:
In step S400, calculating the acquisition characteristic values K of all the auxiliary data points according to the formula in step S203;
Comparing the magnitude of the characteristic value of the data point to be checked and the auxiliary data point on each number, if the characteristic value of the auxiliary data point is higher than the data point to be checked on the same number, deleting the data point to be checked, and marking the auxiliary data point on the corresponding position as a final data point;
In step S500, the final data point is marked with a time stamp, where the time stamp includes the number of the time period in which the data point is located and the number of the data point in the time period, the final data point after the marking is sent to the visualization tool, the visualization tool supplements and replaces the data point on the rough image in the corresponding time period according to the time stamp of the data point, the details of the rough image are perfected, and the visualization chart after the completion is output.
5. A big data based detection data visualization management system, the system comprising the following modules: the system comprises a data detection module, a rough data output module, an auxiliary checking module, a data correction module and a visual supplementing module;
The data detection module is used for receiving the original data transmitted by the equipment sensor, calculating the data processing capacity of the corresponding embedded computer, and marking the equipment as a busy state after the data quantity exceeds the processing capacity of the embedded computer;
The rough data output module is used for generating a record according to the history of the visual chart, calculating the minimum data point number required for enabling the chart to be continuous, calculating the characteristic value of the acquired data point processed by the embedded computer according to the history data of the equipment, sorting the data points according to the characteristic value of the acquired data point, sequentially delivering the data points to the visual tool according to the sorting until the delivered data point number reaches the calculated minimum data point number, enabling the visual tool to output a rough image, storing all the data points, and marking the data points as data points to be checked;
the auxiliary checking module is used for detecting the working condition of other equipment connected with the system, finding out an idle embedded computer in the auxiliary checking module, delivering stored original data to be checked to the idle embedded computer for calculation, marking the calculated data points as auxiliary data points, and sending the auxiliary data points to the original embedded computer;
the data correction module is used for comparing the auxiliary data point with the data point to be checked after the auxiliary data point is received by the original embedded computer, correcting the data of the auxiliary data point according to the data point to be checked, and marking the corrected data point as a final data point;
The visualization supplementing module is used for marking the final data point with a time mark and then sending the data point to the visualization tool, the visualization tool adds the data point to the rough image at the corresponding time according to the time mark of the data point, repairs the details of the image, replaces the original rough image and outputs a complete visualization chart;
The coarse data generation module includes: the device comprises a data point calculation unit, a data point generation unit, a credibility arrangement unit and a coarse image generation unit;
The data point calculation unit is used for calculating the minimum data point quantity required for making the chart continuous according to the historical fluctuation condition of the visual chart;
The data point generating unit is used for controlling the embedded computer to calculate the original data, marking the data points output by the embedded computer as data points to be checked, and marking a set of all data point values to be checked as { P1, P2, …, pa }, wherein a represents the number of the data point in the current time period, and Pa represents the value of the data point to be checked with the number of a;
The credibility arrangement unit is used for extracting the history record of the visual chart within the time Tn from the current time, wherein Tn represents a preset history record viewing range, the number of time periods contained in the Tn time is recorded as n, and the total number of data points contained in all the time periods in the Tn time is recorded as R1; subtracting the values of every two adjacent data points in the same time period, if the subtracted values of the two adjacent data points exceed a set range, marking the time period containing the two data points as a peak time period, marking the number of all peak time periods contained in Tn time as m, and marking the total number of the data points contained in all peak time periods as R2;
calculating a minimum number of data points R required to continue the graph, wherein R1> R2>0, n > m >0;
Collecting the value of each data point in Tn time in a historical record, and calculating the average value of the data points in each time period according to the number of the unit time period in the historical record, wherein the set of all the average values is { Q1, Q2, …, qn }, and Qn represents the average value of the data in the nth time period; calculating a sampling characteristic value K of each data point to be checked, and calculating according to the following formula:
Wherein i represents the number of the unit time period in Tn time;
the rough image generation unit is used for arranging the data points to be checked according to the ascending order of the acquired characteristic values to form a data point selection sequence, sequentially delivering the data points to the visualization tool according to the sequence until the quantity of the delivered data points reaches the minimum quantity of the data points, and enabling the visualization tool to generate a rough image;
the auxiliary checking module comprises: an idle detection unit and an inspection unit;
The idle detection unit is used for detecting the working condition of other equipment, and when one piece of other equipment is idle, the embedded computer of the equipment is marked as an idle embedded computer;
The checking and calculating unit is used for delivering the original data to the idle embedded computer, outputting a calculation result by the idle embedded computer, and sending the calculated data to the system to be recorded as auxiliary data points;
the data correction module comprises: the data point comparison unit and the data point adjustment unit;
The data point comparison unit is used for calculating the acquisition characteristic value of each auxiliary data point, if the acquisition characteristic value of one auxiliary data point is higher than the corresponding data point to be checked, a correction command of the data point is sent out, and if the acquisition characteristic value of the auxiliary data point is lower than or equal to the corresponding data point to be checked, a retention command of the data point is sent out;
The data point adjusting unit is used for deleting the data point to be checked after receiving a correction command of one data point, marking the auxiliary data point at the corresponding position as a final data point, deleting the auxiliary data point after receiving a retention command of one data point, and directly marking the data point to be checked as the final data point;
the visual supplement module comprises: a time marking unit and a coarse image perfecting unit;
The time marking unit is used for marking the final data point with a time mark, wherein the time mark comprises the number of the time period where the data point is located and the number of the data point in the time period;
The rough image perfecting unit is used for transmitting the final data point to the visualization tool, the visualization tool supplements and replaces the data point on the rough image at the corresponding time according to the time mark of the data point, perfects the detail of the rough image, and outputs the finished visualization chart.
6. The big data based detection data visualization management system of claim 5, wherein: the data detection module comprises: a data amount detection unit, a processing capability calculation unit and a state marking unit;
the data quantity detection unit is used for collecting the original data acquired by the equipment sensor and calculating the data quantity of the original data;
the processing capacity calculation unit is used for sending a test data packet, calculating the highest data volume which can be processed by the embedded computer in unit time according to the processing speed of the embedded computer, and recording the highest data volume as the processing capacity of the embedded computer;
The state marking unit is used for comparing the data volume of the original data with the processing capacity of the embedded computer, if the data volume of the original data is smaller than or equal to the processing capacity of the embedded computer, the processing result of the embedded computer is normally delivered to the visualization tool, and if the data volume of the original data is larger than the processing capacity of the embedded computer, the device is marked as a busy device and enters the management flow.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310881604.0A CN116932631B (en) | 2023-07-18 | 2023-07-18 | Big data-based detection data visual management system and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310881604.0A CN116932631B (en) | 2023-07-18 | 2023-07-18 | Big data-based detection data visual management system and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116932631A CN116932631A (en) | 2023-10-24 |
| CN116932631B true CN116932631B (en) | 2024-07-09 |
Family
ID=88393614
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310881604.0A Active CN116932631B (en) | 2023-07-18 | 2023-07-18 | Big data-based detection data visual management system and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116932631B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107248892A (en) * | 2017-07-20 | 2017-10-13 | 中国检验检疫科学研究院 | A kind of monitoring system of rays safety detection apparatus |
| CN109309726A (en) * | 2018-10-25 | 2019-02-05 | 平安科技（深圳）有限公司 | Document generating method and system based on mass data |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112259168B (en) * | 2020-10-22 | 2023-03-28 | 深圳华大基因科技服务有限公司 | Gene sequencing data processing method and gene sequencing data processing device |
| CN115550263A (en) * | 2021-06-28 | 2022-12-30 | 华为技术有限公司 | Active queue management method of network equipment and network equipment |
| CN114567471B (en) * | 2022-02-22 | 2022-10-28 | 珠海市鸿瑞信息技术股份有限公司 | Electric power communication network safety detection system and method based on 5G |
| CN115061825B (en) * | 2022-08-09 | 2022-11-18 | 深圳致星科技有限公司 | Heterogeneous computing system and method for private computing, private data and federal learning |
-
2023
- 2023-07-18 CN CN202310881604.0A patent/CN116932631B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107248892A (en) * | 2017-07-20 | 2017-10-13 | 中国检验检疫科学研究院 | A kind of monitoring system of rays safety detection apparatus |
| CN109309726A (en) * | 2018-10-25 | 2019-02-05 | 平安科技（深圳）有限公司 | Document generating method and system based on mass data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116932631A (en) | 2023-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111283474B (en) | Numerical control automation equipment fault detection system based on big data | |
| US20200209830A1 (en) | Machining apparatus error correction system and method | |
| CN114531618B (en) | Water meter collector data acquisition method, system, storage medium and intelligent terminal | |
| CN116932631B (en) | Big data-based detection data visual management system and method | |
| CN114266944A (en) | Rapid model training result checking system | |
| CN113835417A (en) | Fault detection and diagnosis method based on 5G communication network | |
| CN117394337A (en) | Power grid load early warning method and system thereof | |
| JP6956347B2 (en) | Trace data acquisition system, trace data acquisition method, and program | |
| WO2022121409A1 (en) | Communication control method and apparatus for air conditioner, and communication system and readable storage medium | |
| CN115829300B (en) | Data processing method and device suitable for monitoring state of wire and cable joint | |
| CN106780435A (en) | A kind of object count method and device | |
| CN113934536A (en) | Data acquisition method facing edge calculation | |
| TWI623840B (en) | System and method for analyzing dynamic rate of adaptive machine | |
| CN108279641A (en) | A kind of CNC tune machine method and its system | |
| CN103728904A (en) | Encoder correcting system and method | |
| CN116307342A (en) | Production control method and control device of touch display screen | |
| CN210378426U (en) | Automatic burning device for TFT flash control | |
| CN113902408A (en) | Method, device and equipment for managing abnormity of production process of printed circuit board and storage medium | |
| CN114690835A (en) | Power supply control system of robot encoder | |
| CN116908533B (en) | Power consumer electricity consumption information acquisition equipment with metering function | |
| CN113742326B (en) | Power optimizer and power missing value filling method and device thereof | |
| CN110569996B (en) | Vehicle overhaul data processing method and system | |
| CN118214703B (en) | Digital twin data stream tracking and tracing system | |
| CN115167327B (en) | Control method and system for new energy automobile part production equipment | |
| CN117236651B (en) | Comprehensive management method and system for safe production |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20240617 Address after: No. 118 Minjian Street, Gangnan District, Guigang City, Guangxi Zhuang Autonomous Region, 537110 Applicant after: Mo Xiaobo Country or region after: China Address before: Room 101-25, Science and Technology Innovation City (No. 2305, Chuangxin 1st Road), High tech Industrial Development Zone, Harbin City, Heilongjiang Province, 150036 Applicant before: Harbin Chenwen Technology Development Co.,Ltd. Country or region before: China |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |