A kind of data processing method and device
Technical field
This application involves technical field of data processing, more particularly to a kind of data processing method and device.
Background technology
In various engineering tests or simulation process, it generally can will all test or emulation data preserve, facilitate the later stage
It plays back and checks.
For some experiments or simulating scenes, run time is very long, it is necessary to the data volume kept records of is very big, can generally reach
To GB ranks.If all data are all recorded in a data file, data file is not split, when data volume very
When big, the data file will very big or even individual data file size will check the limit beyond software, cause existing
Data check the data that software can not be all directly viewable in the data file, and browsing is caused to fail.Therefore, the prior art is recording
During data, big data file generally can be all split as to multiple small data files, different sections of data are distinguished
It is preserved, reduces the size of data file, played back convenient for software with checking.
But at least there are the following problems in the prior art:It is literary when big data file is split as multiple small data
During part, the data in the small data file currently selected can only be checked, global browsing can not be carried out to mass data, particularly
When there are many small data file split into, browsing will be very inconvenient.
The content of the invention
In view of this, the application provides a kind of data processing method and device, to realize that the overall situation to mass data browses.
To achieve these goals, technical solution provided by the embodiments of the present application is as follows:
A kind of data processing method, including:
Pending data is obtained, judges whether the data volume of the pending data is more than predetermined threshold value;
If the data volume of the pending data is more than predetermined threshold value, the pending data is divided into multiple data
Section, and the data volume in each data segment is no more than the predetermined threshold value;
At least one sample data is selected from the multiple data segment, utilizes at least one sample data structure selected
It builds for the data subset of overall situation browsing, the data volume in the data subset is not more than the predetermined threshold value.
Preferably, it is described that the pending data is divided into multiple data segments, including:
Obtain the first index information of the pending data;
The pending data is divided into according to first index information by multiple data segments, first index information
For positioning the data in each data segment.
Preferably, it is described that at least one sample data is selected from the multiple data segment, including:
At least one sample data in the pending data is determined according to the predetermined sampling interval, and according to described
One index information determines the data segment where each sample data;
The sample data is selected from the data segment where each sample data, and records each sample data and place
Data segment correspondence.
Preferably, which is characterized in that further include:
The selection operation to the sample data in the data subset is obtained, determines the selected sample of the selection operation
Data;
According to each sample data and the correspondence of the data segment at place, extract and show selected by the selection operation
The corresponding data segment of sample data in data.
Preferably, further include:
Obtain the second index information of each data segment;
Each data segment is divided into according to second index information by multiple subdata sections, second index information is used
Data in each subdata section of positioning;
The second selection operation to the data in the subdata section is obtained, according to second index information, extraction is simultaneously
Show the data in the corresponding subdata section of the selected data of the second selection operation.
The application also provides a kind of data processing equipment, including:
First acquisition module for obtaining pending data, judges whether the data volume of the pending data is more than in advance
If threshold value;
First division module, will be described pending if being more than predetermined threshold value for the data volume of the pending data
Data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value;
Module is built, for selecting at least one sample data from the multiple data segment, using selecting at least
One sample data, which is built, is used for the global data subset browsed, and the data volume in the data subset is not more than the default threshold
Value.
Preferably, first division module, including:
Acquiring unit, for obtaining the first index information of the pending data;
Division unit, for the pending data to be divided into multiple data segments, institute according to first index information
The first index information is stated for positioning the data in each data segment.
Preferably, the structure module, including:
Sample unit, for determining at least one sample number in the pending data according to the predetermined sampling interval
According to, and determine according to first index information data segment where each sample data;
Selecting unit for selecting the sample data from the data segment where each sample data, and records each
The correspondence of the data segment at sample data and place.
Preferably, which is characterized in that further include:
Determining module for obtaining the selection operation to the sample data in the data subset, determines the selection behaviour
Make selected sample data;
Extraction module for the correspondence according to each sample data and the data segment at place, is extracted and shown described
Data in the corresponding data segment of the selected sample data of selection operation.
Preferably, further include:
Second acquisition module, for obtaining the second index information of each data segment;
Second division module, for each data segment to be divided into multiple subdata sections according to second index information,
Second index information is used to position the data in each subdata section;
Second extraction module, for obtaining to the second selection operation of the data in the subdata section, according to described the
Two index informations extract and show the data in the corresponding subdata section of the selected data of the second selection operation.
The data processing method and device provided by more than the application obtain pending data, judge the pending number
According to data volume whether be more than predetermined threshold value;If the data volume of the pending data is more than predetermined threshold value, wait to locate by described
Reason data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value;From the multiple
At least one sample data is selected in data segment, using at least one sample data structure selected for the number of global browsing
According to subset, the data volume in the data subset is not more than the predetermined threshold value.In this way, the pending data of magnanimity can be drawn
It is divided into multiple small data segments, multiple sample data composition data subsets, data segment sum number is then selected from multiple data segments
According to the data volume in subset no more than predetermined threshold value, check software browsing for existing data and check, so as to by clear
The global browsing of mass data can be realized in the data subset of structure look at.
Description of the drawings
It in order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, below will be to embodiment or existing
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments described in application, for those of ordinary skill in the art, without creative efforts,
It can also be obtained according to these attached drawings other attached drawings.
Fig. 1 is the flow chart for the data processing method that the embodiment of the present application one provides;
Fig. 2 is the flow chart for the data processing method that the embodiment of the present application two provides;
Fig. 3 is the flow chart for the data processing method that the embodiment of the present application three provides;
Fig. 4 is the flow chart for the data processing method that the embodiment of the present application four provides;
Fig. 5 is the flow chart for the data processing method that the embodiment of the present application five provides;
Fig. 6 is a kind of structure diagram for data processing equipment that the application provides;
Fig. 7 is the structure diagram for another data processing equipment that the application provides;
Fig. 8 is the structure diagram for another data processing equipment that the application provides;
Fig. 9 is the structure diagram for another data processing equipment that the application provides;
Figure 10 is the structure diagram for another data processing equipment that the application provides.
Specific embodiment
In order to make those skilled in the art better understand the technical solutions in the application, below in conjunction with attached drawing, it is right
The technical solution of the application is clearly and completely described, it is clear that described embodiment is only that the application part is implemented
Example, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art are not making creativeness
All other embodiments obtained under the premise of work should all belong to the scope of the application protection.
In order to which those skilled in the art is made to more fully understand application scheme, the application is made below in conjunction with the accompanying drawings into one
The detailed description of step:
Embodiment one:
Fig. 1 is the flow chart for the data processing method that the embodiment of the present application one provides.
With reference to shown in Fig. 1, data processing method provided by the embodiments of the present application, including:
Step S11:Pending data is obtained, judges whether the data volume of the pending data is more than predetermined threshold value.
In the embodiment of the present application, after pending data is got, the data of pending data can further be obtained
Information, can be by operating system including data name, storing path, data type, data format and data volume size etc.
The function carried of uniting directly acquires.
Since pending data may be the data of magnanimity, it is difficult to be read by software after saving as a data file, because
This can judge pending data after the data message of pending data is got according to the data message of pending data
Whether data volume is more than predetermined threshold value, and predetermined threshold value here can be the maximum for the data volume that popular software can be read.
Step S12:If the data volume of the pending data is more than predetermined threshold value, the pending data is divided into
Multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value.
In the embodiment of the present application, when the data volume of pending data is more than the data of predetermined threshold value, i.e. pending data
When amount is more than the maximum for the data volume that software can be read, pending data can be divided into multiple data segments, made each
Data volume in data segment is no more than the predetermined threshold value, so that each data segment can be read by software.
Step S13:At least one sample data is selected from the multiple data segment, utilizes at least one sample selected
Notebook data, which is built, is used for the global data subset browsed, and the data volume in the data subset is not more than the predetermined threshold value.
In the embodiment of the present application, at least one sample data is selected from multiple data segments that step S12 is divided, it can be with
A sample data is selected from each data segment, multiple sample datas can also be selected from any one data segment, may be used also
To select the number of a sample data namely sample data not necessarily consistent with the number of data segment from multiple data segments,
It is preferred that the number of sample data is The more the better, the number of the data segment at place is The more the better, then utilizes at least one selected
A sample data structure equally makes the data volume in the data subset no more than described pre- for the data subset of global browsing
If threshold value, you can the data subset for global browsing of structure, the global situation of general view pending data are opened by software.
By above data processing method provided by the embodiments of the present application, pending data is obtained, judges the pending number
According to data volume whether be more than predetermined threshold value;If the data volume of the pending data is more than predetermined threshold value, wait to locate by described
Reason data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value;From the multiple
At least one sample data is selected in data segment, using at least one sample data structure selected for the number of global browsing
According to subset, the data volume in the data subset is not more than the predetermined threshold value.In this way, the pending data of magnanimity can be drawn
It is divided into multiple small data segments, multiple sample data composition data subsets, data segment sum number is then selected from multiple data segments
According to the data volume in subset no more than predetermined threshold value, check software browsing for existing data and check, so as to by clear
The global browsing of mass data can be realized in the data subset of structure look at.
Embodiment two:
Fig. 2 is the flow chart for the data processing method that the embodiment of the present application two provides.
With reference to shown in Fig. 2, data processing method provided by the embodiments of the present application, including:
Step S21:Pending data is obtained, judges whether the data volume of the pending data is more than predetermined threshold value.
Step S22:If the data volume of the pending data is more than predetermined threshold value, the of the pending data is obtained
One index information.
In the embodiment of the present application, if the data volume of the pending data is more than predetermined threshold value, data are carried out
Index process obtains the first index information of the pending data.It if, can for example, comprising temporal information in pending data
Therefrom extracting time information, as the first index information, the first index information here is the foundation of follow-up data processing, at this time
If only having data message not have temporal information in pending data, additional temporal information can be assigned automatically, together with number
It is believed that breath is stored in jointly in an index file.
Step S23:The pending data is divided into according to first index information by multiple data segments, described first
Index information is used to position the data in each data segment, and the data volume in each data segment is no more than the default threshold
Value.
In the embodiment of the present application, the first index information is used to position the data in each data segment, refers to the first index
Information can uniquely position in pending data some or certain group data information, the first index information here may be certain
A data name, data value or combination, such as:" data when test period is 0.1 second ", here, " during experiment
Between " it is the first index information, be not in multiple 0.1 seconds because " 0.1 second " this time value is unique.Opposite,
" data when speed is 0.1m/s ", then cannot be uniquely determined a data or one group of data, can be with because speed can change
Repeatedly reach 0.1m/s, therefore " speed " in this example is not the first index information.
According to the first index information of selection, the first index information of setting is corresponding with the data segment of pending data to close
System, it is possible to the pending data is divided into according to the first index information by multiple data segments according to the correspondence, and can
This correspondence to be stored in together with obtained data message before in an index file jointly.
By taking a size is the pending data test.txt of 3.29GB as an example, data format refers to following table 1.It treats
When each column in processing data preserves a variable data, the line number of pending data is more than 20,000,000 rows, and utilization is common
Data scan tool can not all browse.
And Time variables are unique in test.txt, therefore can be using this variable as the first index information, for convenience
It searches, it is necessary to the value of the first index information, that is, Time is mapped with the line number of pending data, it therefore, can be according to Time
The corresponding pending data of value line number, pending data is divided into different data segments according to line number.For example, this waits to locate
Managing in the index file of data needs the information preserved to include:
The data name of pending data:test.txt
The data volume size of pending data:3.29GB
The position of pending data:C:(example)
The data format of pending data:TXT
First index information of pending data:
Time (time) |
Line (line number) |
0 |
2~20001 |
100 |
20002~40001 |
200 |
40002~60001 |
…… |
…… |
Table 1
Wherein, corresponding data segment is 2~20001 rows when the first index information Time is 0, and the first index information Time is
Corresponding data segment is 20002~40001 rows when 100, and corresponding data segment is 40002 when the first index information Time is 200
~60001 rows, and so on.
Step S24:At least one sample data is selected from the multiple data segment, utilizes at least one sample selected
Notebook data, which is built, is used for the global data subset browsed, and the data volume in the data subset is not more than the predetermined threshold value.
By above data processing method provided by the embodiments of the present application, pending data is obtained, judges the pending number
According to data volume whether be more than predetermined threshold value;If the data volume of the pending data is more than predetermined threshold value, according to acquisition
The pending data is divided into multiple data segments by the first index information of the pending data, and in each data segment
Data volume is no more than the predetermined threshold value;At least one sample data is selected from the multiple data segment, using selecting
At least one sample data structure for the data subset of global browsing, the data volume in the data subset is no more than described
Predetermined threshold value.In this way, can the pending data of magnanimity be divided by multiple small data segments according to the first index information, then
Select multiple sample data composition data subsets from multiple data segments, the data volume in data segment and data subset no more than
Predetermined threshold value is checked software browsing for existing data and is checked, so as to can be realized by the data subset for browsing structure
The global browsing of mass data.
Embodiment three:
Fig. 3 is the flow chart for the data processing method that the embodiment of the present application three provides.
With reference to shown in Fig. 3, data processing method provided by the embodiments of the present application, including:
Step S31:Pending data is obtained, judges whether the data volume of the pending data is more than predetermined threshold value.
Step S32:If the data volume of the pending data is more than predetermined threshold value, the of the pending data is obtained
One index information.
In the embodiment of the present application, if the data volume of the pending data is more than predetermined threshold value, data are carried out
Index process obtains the first index information of the pending data.It if, can for example, comprising temporal information in pending data
Therefrom extracting time information, as the first index information, the first index information here is the foundation of follow-up data processing, at this time
If only having data message not have temporal information in pending data, additional temporal information can be assigned automatically, together with number
It is believed that breath is stored in jointly in an index file.
Step S33:The pending data is divided into according to first index information by multiple data segments, described first
Index information is used to position the data in each data segment, and the data volume in each data segment is no more than the default threshold
Value.
In the embodiment of the present application, the first index information is used to position the data in each data segment, refers to the first index
Information can uniquely position in pending data some or certain group data information, the first index information here may be certain
A data name, data value or combination, such as:" data when test period is 0.1 second ", here, " during experiment
Between " it is the first index information, be not in multiple 0.1 seconds because " 0.1 second " this time value is unique.Opposite,
" data when speed is 0.1m/s ", then cannot be uniquely determined a data or one group of data, can be with because speed can change
Repeatedly reach 0.1m/s, therefore " speed " in this example is not the first index information.
According to the first index information of selection, the first index information of setting is corresponding with the data segment of pending data to close
System, it is possible to the pending data is divided into according to the first index information by multiple data segments according to the correspondence, and can
This correspondence to be stored in together with obtained data message before in an index file jointly.
Step S34:Determine at least one sample data in the pending data according to the predetermined sampling interval, and according to
The data segment where each sample data is determined according to first index information.
In the embodiment of the present application, the first index information that the predetermined sampling interval can be in index file determines,
Can also sets itself as desired, sampling interval here can data segment corresponding with the first index information number one
It causes, can not also be consistent, preferred unanimous circumstances in the embodiment of the present application.
In the embodiment of the present application, after the sampling interval for determining the pending data, it is possible to according between the sampling
Every determining sample data to be extracted from pending data, moreover, because the first index information is corresponding with data segment, determine
After going out each sample data, it is possible to the data segment where each sample data is determined according to first index information.
Step S35:The sample data is selected from the data segment where each sample data, and records each sample number
According to the correspondence of the data segment with place, using at least one sample data structure selected for the data of global browsing
Subset, the data volume in the data subset are not more than the predetermined threshold value.
In the embodiment of the present application, after determining the first index information, pending data can be sampled as needed,
A data subset is obtained, the data volume of this data subset checks that software is directly handled suitable for existing data.This data
Collection can have the function that preview overall situation trend, by reading and showing that this data set reaches the preview effect to global data.
For example, pending data test.txt for the example above, it can be according to the first index information Time and data
Section is the correspondence of line number Line, one group of sample data is extracted at interval of 20000 rows, in this way, the data volume of data subset is then
Only 1000 rows or so, existing data scan tool can be browsed easily completely.
By above data processing method provided by the embodiments of the present application, pending data is obtained, judges the pending number
According to data volume whether be more than predetermined threshold value;If the data volume of the pending data is more than predetermined threshold value, according to acquisition
The pending data is divided into multiple data segments by the first index information of the pending data, and in each data segment
Data volume is no more than the predetermined threshold value;At least one sample data is selected from the multiple data segment, using selecting
At least one sample data structure for the data subset of global browsing, the data volume in the data subset is no more than described
Predetermined threshold value.In this way, can the pending data of magnanimity be divided by multiple small data segments according to the first index information, then
Select multiple sample data composition data subsets from multiple data segments, the data volume in data segment and data subset no more than
Predetermined threshold value is checked software browsing for existing data and is checked, so as to can be realized by the data subset for browsing structure
The global browsing of mass data.
Example IV:
Fig. 4 is the flow chart for the data processing method that the embodiment of the present application four provides.
With reference to shown in Fig. 4, data processing method provided by the embodiments of the present application, including:
Step S41:Pending data is obtained, judges whether the data volume of the pending data is more than predetermined threshold value.
Step S42:If the data volume of the pending data is more than predetermined threshold value, the pending data is divided into
Multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value.
Step S43:At least one sample data is selected from the multiple data segment, utilizes at least one sample selected
Notebook data, which is built, is used for the global data subset browsed, and the data volume in the data subset is not more than the predetermined threshold value.
Step S44:It obtains and the first choice of the sample data in the data subset is operated, determine the first choice
Operate selected sample data.
In the embodiment of the present application, user is after according to the global trend for browsing entire pending data of data subset, also
Can be as needed, the interested sample data in the data subset is made choice.
Step S45:According to each sample data and the correspondence of the data segment at place, extract and show first choosing
Select the data operated in the corresponding data segment of selected sample data.
In the embodiment of the present application, after the first choice for determining user operates selected sample data, it may be determined that
Data segment where the sample data of user's selection, and the total data of the data segment is extracted from pending data, into
Row is checked in detail.
Wherein, when including the first index information in pending data, it can determine that user selects according to the first index information
Data segment where the sample data selected when not including the first index information in pending data, can also directly basis be drawn
The data segment divided determines the data segment where the sample data of user's selection, and the application is not limited in any way this.
Continue taking the above example as an example, if the sample data of the 20000th row is checked in user's selection in detail, can pass through
The data segment of division directly determines that the sample data of the 20000th row, can also be according to the in the 2nd~20001 data segment
The corresponding first index information Time of sample data of 20000 rows is the 0 corresponding data segment of sample data for determining the 20000th row
For 2~20001 rows, and then all data in the 2nd~20001 data segment can be extracted and shown, checked in detail for user.
By above data processing method provided by the embodiments of the present application, pending data is obtained, judges the pending number
According to data volume whether be more than predetermined threshold value;If the data volume of the pending data is more than predetermined threshold value, wait to locate by described
Reason data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value;From the multiple
At least one sample data is selected in data segment, using at least one sample data structure selected for the number of global browsing
According to subset, the data volume in the data subset is not more than the predetermined threshold value;It obtains to the sample number in the data subset
According to first choice operate, determine that the first choice operates selected sample data;According to each sample data and place
Data segment correspondence, extract and show that the first choice is operated in the corresponding data segment of selected sample data
Data.In this way, the pending data of magnanimity can be divided into multiple small data segments, then selected from multiple data segments more
A sample data composition data subset, the data volume in data segment and data subset is no more than predetermined threshold value, for existing
Data are checked software browsing and are checked, so as to which the overall situation that mass data can be realized by browsing the data subset of structure browses,
Then interested sample data can also be selected from data subset, checks the complete of data segment where the sample data in detail
Portion's data can be the browsing scope of user's rapid drop data after the overall situation browses, and provide browsing part number interested in detail
According to experience.
Embodiment five:
Fig. 5 is the flow chart for the data processing method that the embodiment of the present application five provides.
With reference to shown in Fig. 5, data processing method provided by the embodiments of the present application, including:
Step S51:Pending data is obtained, judges whether the data volume of the pending data is more than predetermined threshold value.
Step S52:If the data volume of the pending data is more than predetermined threshold value, the of the pending data is obtained
One index information.
Step S53:The pending data is divided into according to first index information by multiple data segments, described first
Index information is used to position the data in each data segment, and the data volume in each data segment is no more than the default threshold
Value.
In the embodiment of the present application, the first index information is used to position the data in each data segment, refers to the first index
Information can uniquely position in pending data some or certain group data information, the first index information here may be certain
A data name, data value or combination, such as:" data when test period is 0.1 second ", here, " during experiment
Between " it is the first index information, be not in multiple 0.1 seconds because " 0.1 second " this time value is unique.Opposite,
" data when speed is 0.1m/s ", then cannot be uniquely determined a data or one group of data, can be with because speed can change
Repeatedly reach 0.1m/s, therefore " speed " in this example is not the first index information.
According to the first index information of selection, the first index information of setting is corresponding with the data segment of pending data to close
System, it is possible to the pending data is divided into according to the first index information by multiple data segments according to the correspondence, and can
This correspondence to be stored in together with obtained data message before in an index file jointly.
Step S54:At least one sample data is selected from the multiple data segment, utilizes at least one sample selected
Notebook data, which is built, is used for the global data subset browsed, and the data volume in the data subset is not more than the predetermined threshold value.
Step S55:Obtain the second index information of each data segment.
In the embodiment of the present application, the information preserved in order to prevent in single index information is too many, influences processing speed, also
Two-stage index can be established, i.e., outside the first index information of pending data, the second index letter is established for each data segment
Breath.
Step S56:Each data segment is divided into according to second index information by multiple subdata sections, second rope
Fuse ceases to position the data in each subdata section.
In the embodiment of the present application, since the second index information is the second index information of each data segment, so second
Index information can be further divided into each data segment multiple subdata sections.
By taking a size is the pending data test.txt of 3.29GB as an example, data format refers to following table 2.It treats
When each column in processing data preserves a variable data, the line number of pending data is more than 20,000,000 rows, and utilization is common
Data scan tool can not all browse.
And Time variables are unique in test.txt, therefore can be using this variable as the first index information, for convenience
It searches, it is necessary to the value of the first index information, that is, Time is mapped with the line number of pending data, it therefore, can be according to Time
The corresponding pending data of value line number, pending data is divided into different data segments according to line number.For example, this waits to locate
Managing in the index file of data needs the information preserved to include:
The data name of pending data:test.txt
The data volume size of pending data:3.29GB
The position of pending data:C:(example)
The data format of pending data:TXT
First index information of pending data:
Time (time) |
Line (line number) |
Secondary index |
0 |
2~20001 |
|
100 |
20002~40001 |
1.index |
200 |
40002~60001 |
2.index |
…… |
…… |
…… |
Table 2
Wherein, corresponding data segment is 2~20001 rows when the first index information Time is 0, and the first index information Time is
Corresponding data segment is 20002~40001 rows when 100, and corresponding data segment is 40002 when the first index information Time is 200
~60001 rows, and so on.
And since the index span in upper table 2 is bigger, therefore " secondary index " i.e. the second index information is added, convenient pair
After data Primary Location, reading position is reduced again.For example, the first index information Time is saved in 1.index from 0 to 100
Corresponding Line be 2~20001 row data segments the second index information, content such as the following table 3:
Table 3
Wherein, corresponding subdata section is 2~2001 rows when the second index information Time is 0, the second index information Time
For 10 when corresponding subdata section be 2002~4001 rows, the second index information Time when being 20 corresponding subdata section be 4002
~6001 rows, and so on.
Compared by table 2 and table 3 as it can be seen that Time from 5 to 15 between data, in the first index information Time from 0 to
Between 100 in corresponding data segment, i.e., between the 2 to 20002nd row of pending data.Due to there are the second index information, therefore
According to the second index information 1.index, can further location data position, Time from 5 to 15 between data second index
It in subdata section between Time is from 0 to 20 in information, positions in this way, data area can be reduced 1000 times.Treat
Between the 2 to 4002nd row for handling data, common data scan tool can show the data of this data volume.
Step S57:The second selection operation to the data in the subdata section is obtained, according to the described second index letter
Breath, extracts and shows the data in the corresponding subdata section of the selected data of the second selection operation.
In the embodiment of the present application, user, can also basis after the total data in viewing interested data segment
It needs, interested data in the data segment is made choice.
In the embodiment of the present application, after the second selection operation for determining user selected data, it may be determined that user
Subdata section in data segment where the data of selection, and the total data of the subdata section is extracted, it is checked in detail.
Continue taking the above example as an example, if the sample data of the 20000th row is checked in user's selection in detail, according to the
The corresponding first index information Time of sample data of 20000 rows is the 0 corresponding data segment of sample data for determining the 20000th row
For 2~20001 rows, extract all data in the 2nd~20001 data segment and show, after being checked in detail for user, if with
Family further selects to check the data of the 2000th row, then can be according to the corresponding second index information Time of data of the 2000th row
The corresponding subdata section of data for 0 definite 2000th row is 2~2001 rows, and then is extracted in the 2nd~2001 subdata section
All data and show, checked in detail for user.
Moreover, after all data during user reads the 2nd~2001 subdata section that Time is 0 using software,
Due to data area very little, more small-scale data can also be voluntarily searched in software inhouse, carry out the display of next step
With checking.
In the embodiment of the present application, the grade quantity of index information is not limited, therefore except first in above-described embodiment
Outside index information and the second index information, the 3rd index information and the 4th index information etc. can also be set, and so on, with
Just smaller diminution data area.
By above data processing method provided by the embodiments of the present application, pending data is obtained, judges the pending number
According to data volume whether be more than predetermined threshold value;If the data volume of the pending data is more than predetermined threshold value, wait to locate by described
Reason data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value;From the multiple
At least one sample data is selected in data segment, using at least one sample data structure selected for the number of global browsing
According to subset, the data volume in the data subset is not more than the predetermined threshold value;It obtains to the sample number in the data subset
According to first choice operate, determine that the first choice operates selected sample data;According to each sample data and place
Data segment correspondence, extract and show that the first choice is operated in the corresponding data segment of selected sample data
Data.In this way, the pending data of magnanimity can be divided into multiple small data segments, then selected from multiple data segments more
A sample data composition data subset, the data volume in data segment and data subset is no more than predetermined threshold value, for existing
Data are checked software browsing and are checked, so as to which the overall situation that mass data can be realized by browsing the data subset of structure browses,
Then interested sample data can also be selected from data subset, which is checked according to the first index information in detail
The total data of the data segment at place, in the subdata section where checking the data in data segment in detail according to the second index information
Total data, can the overall situation browse after, be user's rapid drop data browsing scope, provide in detail browsing part feel emerging
The experience of interesting data.
It is understood that for foregoing each embodiment, if it is judged that the data volume of the pending data is little
In predetermined threshold value, you can directly open and check pending data by data scan tool, without subdivided data segment and
Carry out subsequent processing.
For foregoing each method embodiment, in order to be briefly described, therefore it is all expressed as to a series of combination of actions, but
It is that those skilled in the art should know, the present invention and from the limitation of described sequence of movement, because according to the present invention, certain
A little steps may be employed other orders or be carried out at the same time.
Present invention data processing method disclosed above, correspondingly, the invention also discloses apply above-mentioned data processing side
The device of method, which, which is used to implement, browses the overall situation of mass data.
Fig. 6 is a kind of structure diagram for data processing equipment that the application provides.
With reference to shown in Fig. 6, data processing equipment provided by the embodiments of the present application, including:
First acquisition module 1 for obtaining pending data, judges whether the data volume of the pending data is more than in advance
If threshold value.
First division module 2, will be described pending if being more than predetermined threshold value for the data volume of the pending data
Data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value.
Module 3 is built, for selecting at least one sample data from the multiple data segment, using selecting at least
One sample data, which is built, is used for the global data subset browsed, and the data volume in the data subset is not more than the default threshold
Value.
The data processing side in above method embodiment may be employed in data processing equipment provided by the embodiments of the present application
Method, details are not described herein again.
Fig. 7 is the structure diagram for another data processing equipment that the application provides.
With reference to shown in Fig. 7, data processing equipment provided by the embodiments of the present application, including:
First acquisition module 1 for obtaining pending data, judges whether the data volume of the pending data is more than in advance
If threshold value.
First division module 2, will be described pending if being more than predetermined threshold value for the data volume of the pending data
Data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value.
First division module 2 specifically includes:Acquiring unit 21, if the data volume for the pending data is big
In predetermined threshold value, the first index information of the pending data is obtained.Division unit 22, for being believed according to the described first index
The pending data is divided into multiple data segments by breath, and first index information is used to position the number in each data segment
According to, and the data volume in each data segment is no more than the predetermined threshold value.
Module 3 is built, for selecting at least one sample data from the multiple data segment, using selecting at least
One sample data, which is built, is used for the global data subset browsed, and the data volume in the data subset is not more than the default threshold
Value.
The data processing side in above method embodiment may be employed in data processing equipment provided by the embodiments of the present application
Method, details are not described herein again.
Fig. 8 is the structure diagram for another data processing equipment that the application provides.
With reference to shown in Fig. 8, data processing equipment provided by the embodiments of the present application, including:
First acquisition module 1 for obtaining pending data, judges whether the data volume of the pending data is more than in advance
If threshold value.
First division module 2, will be described pending if being more than predetermined threshold value for the data volume of the pending data
Data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value.
First division module 2 specifically includes:Acquiring unit 21, if the data volume for the pending data is big
In predetermined threshold value, the first index information of the pending data is obtained.Division unit 22, for being believed according to the described first index
The pending data is divided into multiple data segments by breath, and first index information is used to position the number in each data segment
According to, and the data volume in each data segment is no more than the predetermined threshold value.
Module 3 is built, for selecting at least one sample data from the multiple data segment, using selecting at least
One sample data, which is built, is used for the global data subset browsed, and the data volume in the data subset is not more than the default threshold
Value.
The structure module 3 specifically includes:Sample unit 31, it is described pending for being determined according to the predetermined sampling interval
At least one sample data in data, and determine according to first index information data segment where each sample data.
Selecting unit 32 for selecting the sample data from the data segment where each sample data, and records each sample number
According to the correspondence of the data segment with place, using at least one sample data structure selected for the data of global browsing
Subset, the data volume in the data subset are not more than the predetermined threshold value.
The data processing side in above method embodiment may be employed in data processing equipment provided by the embodiments of the present application
Method, details are not described herein again.
Fig. 9 is the structure diagram for another data processing equipment that the application provides.
With reference to shown in Fig. 9, data processing equipment provided by the embodiments of the present application, including:
First acquisition module 1 for obtaining pending data, judges whether the data volume of the pending data is more than in advance
If threshold value.
First division module 2, will be described pending if being more than predetermined threshold value for the data volume of the pending data
Data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value.
Module 3 is built, for selecting at least one sample data from the multiple data segment, using selecting at least
One sample data, which is built, is used for the global data subset browsed, and the data volume in the data subset is not more than the default threshold
Value.
Determining module 4 for obtaining the selection operation to the sample data in the data subset, determines the selection behaviour
Make selected sample data.
First extraction module 5 for the correspondence according to each sample data and the data segment at place, is extracted and shown
Data in the corresponding data segment of the selected sample data of selection operation.
The data processing side in above method embodiment may be employed in data processing equipment provided by the embodiments of the present application
Method, details are not described herein again.
Figure 10 is a kind of structure diagram for data processing equipment that the application provides.
With reference to shown in Figure 10, data processing equipment provided by the embodiments of the present application, including:
First acquisition module 1 for obtaining pending data, judges whether the data volume of the pending data is more than in advance
If threshold value.
First division module 2, will be described pending if being more than predetermined threshold value for the data volume of the pending data
Data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value.
First division module 2 specifically includes:Acquiring unit 21, if the data volume for the pending data is big
In predetermined threshold value, the first index information of the pending data is obtained.Division unit 22, for being believed according to the described first index
The pending data is divided into multiple data segments by breath, and first index information is used to position the number in each data segment
According to, and the data volume in each data segment is no more than the predetermined threshold value.
Module 3 is built, for selecting at least one sample data from the multiple data segment, using selecting at least
One sample data, which is built, is used for the global data subset browsed, and the data volume in the data subset is not more than the default threshold
Value.
The structure module 3 specifically includes:Sample unit 31, it is described pending for being determined according to the predetermined sampling interval
At least one sample data in data, and determine according to first index information data segment where each sample data.
Selecting unit 32 for selecting the sample data from the data segment where each sample data, and records each sample number
According to the correspondence of the data segment with place, using at least one sample data structure selected for the data of global browsing
Subset, the data volume in the data subset are not more than the predetermined threshold value.
Determining module 4 for obtaining the selection operation to the sample data in the data subset, determines the selection behaviour
Make selected sample data.
First extraction module 5 for the correspondence according to each sample data and the data segment at place, is extracted and shown
Data in the corresponding data segment of the selected sample data of selection operation.
Second acquisition module 6, for obtaining the second index information of each data segment.
Second division module 7, for each data segment to be divided into multiple subdata sections according to second index information,
Second index information is used to position the data in each subdata section.
Second extraction module 8, for obtaining to the second selection operation of the data in the subdata section, according to described the
Two index informations extract and show the data in the corresponding subdata section of the selected data of the second selection operation.
The data processing side in above method embodiment may be employed in data processing equipment provided by the embodiments of the present application
Method, details are not described herein again.
The data processing method and device provided by more than the application obtain pending data, judge the pending number
According to data volume whether be more than predetermined threshold value;If the data volume of the pending data is more than predetermined threshold value, wait to locate by described
Reason data are divided into multiple data segments, and the data volume in each data segment is no more than the predetermined threshold value;From the multiple
At least one sample data is selected in data segment, using at least one sample data structure selected for the number of global browsing
According to subset, the data volume in the data subset is not more than the predetermined threshold value.In this way, the pending data of magnanimity can be drawn
It is divided into multiple small data segments, multiple sample data composition data subsets, data segment sum number is then selected from multiple data segments
According to the data volume in subset no more than predetermined threshold value, check software browsing for existing data and check, so as to by clear
The global browsing of mass data can be realized in the data subset of structure look at.
For convenience of description, it is divided into various units during description apparatus above with function to describe respectively.Certainly, this is being implemented
The function of each unit is realized can in the same or multiple software and or hardware during application.
Each embodiment in this specification is described by the way of progressive, identical similar portion between each embodiment
Point just to refer each other, and the highlights of each of the examples are difference from other examples.Especially for device or
For system embodiment, since it is substantially similar to embodiment of the method, so describing fairly simple, related part is referring to method
The part explanation of embodiment.Apparatus and system embodiment described above is only schematical, wherein the conduct
The unit that separating component illustrates may or may not be it is physically separate, the component shown as unit can be or
Person may not be physical location, you can be located at a place or can also be distributed in multiple network element.It can root
Factually border needs to select some or all of module therein realize the purpose of this embodiment scheme.Ordinary skill
Personnel are without creative efforts, you can to understand and implement.
Professional further appreciates that, with reference to each exemplary unit of the embodiments described herein description
And algorithm steps, can be realized with the combination of electronic hardware, computer software or the two, in order to clearly demonstrate hardware and
The interchangeability of software generally describes each exemplary composition and step according to function in the above description.These
Function is performed actually with hardware or software mode, specific application and design constraint depending on technical solution.Specialty
Technical staff can realize described function to each specific application using distinct methods, but this realization should not
Think beyond the scope of this invention.
It can directly be held with reference to the step of method or algorithm that the embodiments described herein describes with hardware, processor
The combination of capable software module or the two is implemented.Software module can be placed in random access memory (RAM), memory, read-only deposit
Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology
In any other form of storage medium well known in field.
The foregoing description of the disclosed embodiments enables professional and technical personnel in the field to realize or use the present invention.
A variety of modifications of these embodiments will be apparent for those skilled in the art, it is as defined herein
General Principle can be realized in other embodiments without departing from the spirit or scope of the present invention.Therefore, it is of the invention
The embodiments shown herein is not intended to be limited to, and is to fit to and the principles and novel features disclosed herein phase one
The most wide scope caused.