CN106296498A - Data processing method and device - Google Patents
Data processing method and device Download PDFInfo
- Publication number
- CN106296498A CN106296498A CN201510263271.0A CN201510263271A CN106296498A CN 106296498 A CN106296498 A CN 106296498A CN 201510263271 A CN201510263271 A CN 201510263271A CN 106296498 A CN106296498 A CN 106296498A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- granularity
- collect
- initial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 10
- 238000000034 method Methods 0.000 claims abstract description 37
- 238000013499 data model Methods 0.000 claims abstract description 30
- 238000007596 consolidation process Methods 0.000 claims abstract description 9
- 238000012545 processing Methods 0.000 claims description 19
- 230000008859 change Effects 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 8
- 230000001788 irregular Effects 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 4
- 230000008569 process Effects 0.000 description 13
- 238000007726 management method Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 7
- 238000006243 chemical reaction Methods 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 230000005540 biological transmission Effects 0.000 description 3
- 238000004140 cleaning Methods 0.000 description 3
- 238000013500 data storage Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 241001269238 Data Species 0.000 description 1
- 235000012364 Peperomia pellucida Nutrition 0.000 description 1
- 240000007711 Peperomia pellucida Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 239000012141 concentrate Substances 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000000875 corresponding effect Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/20—Education
Landscapes
- Business, Economics & Management (AREA)
- Tourism & Hospitality (AREA)
- Engineering & Computer Science (AREA)
- Human Resources & Organizations (AREA)
- Primary Health Care (AREA)
- Health & Medical Sciences (AREA)
- Economics (AREA)
- General Health & Medical Sciences (AREA)
- Educational Administration (AREA)
- Marketing (AREA)
- Educational Technology (AREA)
- Strategic Management (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a kind of data processing method and device, wherein, the method includes: gather the initial data from data source；Handover raw data is the first data meeting target data model, and wherein, the first data include the feature of at least one of: consolidation form encodes, unified data type, unified data form；Store the first data.By the present invention, solve the problem that data-handling efficiency that big Stored Data Type disunity causes is low, improve treatment effeciency.
Description
Technical field
The present invention relates to data processing field, in particular to a kind of data processing method and device.
Background technology
At present, all parts of the country are all advancing IT application in education sector work.Set up the Informatization Service common platform of education, open
The experimental work of exhibition Digital Campus, sets up various " Digital Learning " Experiment site school, develops " micro-course ", opens
Exhibition " upset classroom " teaching research, man-to-man " E classroom " teaching practice, to really promote education water product, the heaviest
Want be Top-layer Design Method and theory advanced.
Big data, refer to involved data quantity huge to passing through current main software instrument, rationally
Reach in time to capture, manage, process, and arrange and become information for the purpose of help enterprise management decision-making.Big data with
Traditional data are compared, and have data volume big (Volume), the source of data and form various (Variety), data to increase
The feature such as long quickly (Velocity), value density low (Value), complexity big (Complexity).
In education sector, how to introduce big data technique, utilize people (student, the head of a family, teacher), school, Bureau of Education,
And other with education correlate data, it is achieved the design of educational environment, the layout of educational experimentation scene, education
The change of space-time, the study change of scene, the collection of educational management data and decision-making etc. are urgently studied at present.How to utilize
Advanced information technology, the data supporting of big data, change the decision mode adding experience in the past by bat head or theory inspiration,
It it is the hot issue of research at present.
Inventor finds in research process, the hugest along with data scale, data type and form increasingly sophisticated,
Efficiently substantial amounts of data cannot be applied and have become as the new problem that big data age faces.
The problem that the data-handling efficiency that causes for Stored Data Type disunity big in correlation technique is low, the most not yet carries
Go out effective solution.
Summary of the invention
In order to solve above-mentioned technical problem, the invention provides a kind of data processing method and device.
According to an aspect of the invention, it is provided a kind of data processing method, including: gather from data source is original
Data；Changing described initial data is the first data meeting target data model, wherein, described first data include with
At least one lower feature: consolidation form encodes, unified data type, unified data form；Store described first
Data.
Preferably, gather the described initial data from described data source to include: the cycle gathers the institute from described data source
State initial data；Or the acquisition condition instantaneous acquiring according to setting is from the described initial data of described data source.
Preferably, change described initial data into described first data before, described method also includes: according to default plan
Slightly, reject the irregular data in described initial data and/or do not meet factual data.
Preferably, change described initial data into described first data after, described method also includes: to described first
Data carry out data summarization, and wherein, described data summarization includes at least one of: collects time granularity, collect network element
Granularity, collect spatial granularity, collect business granularity.
Preferably, store described first data to include: use the mode of redundant storage to store described first data.
Preferably, after storing described first data, described method also includes: obtain the data model that user sets up；
Data needed for data model described in described first extracting data；Export the result of calculation of described data model.
According to another aspect of the present invention, additionally provide a kind of data processing equipment, including: acquisition module, it is used for adopting
Collect the initial data from data source；Modular converter, is meet target data model for changing described initial data
One data, wherein, described first data include the feature of at least one of: consolidation form encodes, unified data class
Type, unified data form；Memory module, is used for storing described first data.
Preferably, described acquisition module is used for: the cycle gathers the described initial data from described data source；Or according to
The acquisition condition instantaneous acquiring set is from the described initial data of described data source.
Preferably, described device also includes: reject module, for according to preset strategy, rejects in described initial data
Irregular data and/or do not meet factual data.
Preferably, described device also includes: summarizing module, for described first data being carried out data summarization, wherein,
Described data summarization includes at least one of: collects time granularity, collect network element granularity, collect spatial granularity, collect
Business granularity.
Preferably, described memory module, for using the mode of redundant storage to store described first data.
Preferably, described device also includes: acquisition module, for obtaining the data model that user sets up；Extraction module,
For the data needed for data model described in described first extracting data；Output module, is used for exporting described data mould
The result of calculation of type.
By the present invention, use and gather the initial data from data source；Handover raw data is for meeting target data model
The first data, wherein, the first data include the feature of at least one of: consolidation form encodes, unified data class
Type, unified data form；Store the mode of the first data, solve the data that big Stored Data Type disunity causes
The problem that treatment effeciency is low, improves treatment effeciency.
Accompanying drawing explanation
Accompanying drawing described herein is used for providing a further understanding of the present invention, constitutes the part of the application, this
Bright schematic description and description is used for explaining the present invention, is not intended that inappropriate limitation of the present invention.At accompanying drawing
In:
Fig. 1 is the flow chart of data processing method according to embodiments of the present invention；
Fig. 2 is the structural representation of data processing equipment according to embodiments of the present invention；
Fig. 3 is the preferred structure schematic diagram one of data processing equipment according to embodiments of the present invention；
Fig. 4 is the preferred structure schematic diagram two of data processing equipment according to embodiments of the present invention；
Fig. 5 is the preferred structure schematic diagram three of data processing equipment according to embodiments of the present invention；
Fig. 6 is the structural representation educating big Data application system according to the preferred embodiment of the invention；
Fig. 7 is the schematic flow sheet educating big data application method according to the preferred embodiment of the invention.
Detailed description of the invention
Below with reference to accompanying drawing and describe the present invention in detail in conjunction with the embodiments.It should be noted that do not conflicting
In the case of, the embodiment in the application and the feature in embodiment can be mutually combined.
Other features and advantages of the present invention will illustrate in the following description, and, partly become from description
Obtain it is clear that or understand by implementing the present invention.The purpose of the present invention and other advantages can be by being write
Structure specifically noted in description, claims and accompanying drawing realizes and obtains.
In order to make those skilled in the art be more fully understood that the present invention program, below in conjunction with in the embodiment of the present invention
Accompanying drawing, is clearly and completely described the technical scheme in the embodiment of the present invention, it is clear that described embodiment
It is only the embodiment of a present invention part rather than whole embodiments.Based on the embodiment in the present invention, ability
The every other embodiment that territory those of ordinary skill is obtained under not making creative work premise, all should belong to
The scope of protection of the invention.
Embodiments providing a kind of data processing method, Fig. 1 is data processing method according to embodiments of the present invention
Flow chart, as it is shown in figure 1, this flow process comprises the steps:
Step S102, gathers the initial data from data source；
Step S104, handover raw data is the first data meeting target data model, and wherein, the first data include
The feature of at least one of: consolidation form encodes, unified data type, unified data form；
Step S106, stores the first data.
By above-mentioned steps, in data processing data are uniformly processed into and meet the data of target data model and carry out
Storage, makes data obtain unified storage.Visible, use above-mentioned steps, the data of bulky complex can be made to be unified
Process, solve the problem that data-handling efficiency that big Stored Data Type disunity causes is low, improve data and process
Efficiency.
Preferably, above-mentioned data source includes at least one of: information-based Classroom System, examination system, school's logistics
Management system.
Preferably, in above-mentioned steps S102, the mode gathering initial data can be in the way of taking cycle collection, also
Can be in the way of taking instantaneous acquiring.Preferably, the cycle that the cycle gathers can be set according to the demand of user.
Preferably, big data are the hugest and numerous and jumbled due to data, are wherein contaminated with all kinds of effective or invalid data；
In order to save memory space, it is to avoid unnecessary resource consumption, and realize the conversion of efficient data, after collecting the data,
The irregular data in initial data can also be weeded out according to preset strategy and/or do not meets factual data.The most again
To rejecting irregular data and/or not meeting the initial data of factual data and store.
Preferably, after above-mentioned steps S104, the method also includes: the first data carry out data summarization, wherein,
Data summarization includes at least one of: collects time granularity, collect network element granularity, collect spatial granularity, collect business
Granularity.Data after collecting are conducive to promoting access efficiency.
The data volume of the first data owing to collecting may be the hugest, in order to promote access performance, it is preferable that in step
The first data can be stored, such as, by the first data in the way of using redundant storage when rapid S106 stores the first data
Carry out after piecemeal is copied into many parts, being stored in distributed storage network.
Preferably, after the first data are stored, in order to realize the application to data, user can according to demand,
Set up corresponding data model.In this case, the present embodiment is after above-mentioned steps S106, it is also possible to obtains and uses
The data model that family is set up；Data needed for the first extracting data data model；The result of calculation of output data model.
Preferably, it is also possible to according to result of calculation and preset strategy, the result of decision is exported.
Additionally provide a kind of data processing equipment in the present embodiment, be used for realizing above-described embodiment and preferred implementation,
Carried out repeating no more of explanation, below the module related in this device had been illustrated.Use as following
, term " module " can realize the software of predetermined function and/or the combination of hardware.Although described by following example
Device preferably realizes with software, but hardware, or the realization of the combination of software and hardware be also possible and by structure
Think.
Fig. 2 is the structural representation of data processing equipment according to embodiments of the present invention, as in figure 2 it is shown, this device includes:
Acquisition module 22, modular converter 24, memory module 26, wherein, acquisition module 22, for gathering from data source
Initial data；Modular converter 24, coupled to acquisition module 22, for handover raw data for meeting target data model
The first data, wherein, the first data include the feature of at least one of: consolidation form encodes, unified data class
Type, unified data form；Memory module 26, coupled to modular converter 24, for storage the first data.
Preferably, above-mentioned acquisition module 22 gathers the initial data from data source for the cycle；Or according to set
Acquisition condition instantaneous acquiring is from the initial data of data source.
Fig. 3 is the preferred structure schematic diagram one of data processing equipment according to embodiments of the present invention, as it is shown on figure 3, preferably
Ground, said apparatus also includes: rejects module 32, coupled between acquisition module 22 and modular converter 24, for basis
Preset strategy, rejects the irregular data in initial data and/or does not meets factual data.
Fig. 4 is the preferred structure schematic diagram two of data processing equipment according to embodiments of the present invention, as shown in Figure 4, preferably
Ground, said apparatus also includes: summarizing module 42, coupled between modular converter 24 and memory module 26, for the
One data carry out data summarization, and wherein, data summarization includes at least one of: collects time granularity, collect network element grain
Spend, collect spatial granularity, collect business granularity.
Preferably, above-mentioned memory module 26 is for using the mode of redundant storage to store the first data.
Fig. 5 is the preferred structure schematic diagram three of data processing equipment according to embodiments of the present invention, as it is shown in figure 5, preferably
Ground, said apparatus also includes: acquisition module 52, for obtaining the data model that user sets up；Extraction module 54, coupling
It is bonded to memory module 26 and acquisition module 52, for the data needed for the first extracting data data model；Output mould
Block 56, coupled to extraction module 54, for exporting the result of calculation of data model.
It addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, it is also possible to
It is that unit is individually physically present, it is also possible to two or more unit are integrated in a unit.Above-mentioned integrated
Unit both can realize to use the form of hardware, it would however also be possible to employ the form of SFU software functional unit realizes.
In order to the description making the embodiment of the present invention is clearer, it is described below in conjunction with preferred embodiment and illustrates.
The preferred embodiment of the present invention provides the big data application method of a kind of education, to realize adopting of magnanimity education related data
Collect, store, manage, analyze, inquire about, represent, it is therefore intended that final help student formulates study plan, promotes into
Achievement；Help teacher's accurate perception student's situation, teach students in accordance with their aptitude；School leaders is helped to improve management, intelligent decision；Help
Education related industry summary responses turn of the market, precision marketing.
For achieving the above object, the preferred embodiment of the present invention provides the big Data application system of a kind of education, including:
1 data acquisition module: the function of this module can be according to the interface type specified and characteristic requirements, from different numbers
According to obtaining initial data at source.Wherein it is possible to adopted by modes such as file interface, data base interface, message interfaces
Collection.Two ways is generally supported in data acquisition: the cycle gathers and instantaneous acquiring.Cycle collection refers to according to different data
Content, according to data pick-up cycle, the mode extracted data within the time specified.Instantaneous acquiring is system root
The most disposably operate according to the acquisition condition set, this action after having operated, is not repeated.Preferably, immediately adopt
The data in historical data and Resurvey applied by collection.
2 data processing modules: this module be mainly responsible for data cleaning, change, load, the merit such as regulation management and transmission
Energy.Data cleansing can complete the rejecting to " dirty data ", eliminates the inconsistent of data." dirty data " includes not advising
Then data, do not meet factual data.Data conversion mainly include being converted into consolidation form coding, unified data type,
Unified data form.Exception, data are changed and are also supported the most frequently used data summarization, such as: collect time granularity, remittance
Total network element granularity, collect spatial granularity, collect business granularity etc.；Load through over cleaning and conversion after meet target data
The data of model, or without " totally " data additionally processed.
3 data memory modules: this module is as the carrier of data, it is provided that mass data storage and the confession of stability and high efficiency are upper
The data-interface that layer accesses.Data include real time data and non-real-time data；Including structural data and unstructured data.
The mode using redundant storage can ensure that the reliability of storage data, is same number according to storing multiple copies, all
Mass data use distributed storage mode be stored in different nodes, the mode simultaneously using redundant storage is all right
The height providing high-throughput and high transmission rates concurrently accesses service.
4 market demand modules: be used for data analysis and excavated, generated final result data.Such as, according to concrete
Data are analyzed and process by business demand, including data modeling and externally provide service ability.Market demand mould
Block provides visual modeling tool and application development tool, supports that various components encapsulates and is integrated in developing instrument, to upper
Layer application provides unified application programming interfaces (Application Programming Interface, referred to as API), supplies
Application call, realizes details to application shielding bottom complexity, promotes application and development efficiency.
For achieving the above object, the preferred embodiment of the present invention additionally provides the big data application method of a kind of education, including as follows
Step:
Step 1: the rule that data acquisition module consults according to the application system that prior and each education is relevant, from each data source
Obtain data.Include but not limited to obtain the achievement of student from student examination system, wrong topic is analyzed, test time distribution
Information etc.；Obtain the data such as interaction of student's raising one's hand in information-based classroom, answer and teacher；Obtain going out of student
Diligent rate；Obtain various lives and the consumption data of student, including library, dining room, electronization classroom, supermarket etc..
Step 2: data processing module is according to the rule defined, and be carried out data and conversion etc. processes, and makes data
Become the data meeting target data model.
Step 3: data after treatment are stored in data memory module.
Step 4: be modeled in market demand module, utilizes various data to carry out COMPREHENSIVE CALCULATING, and intelligent analysis obtains
To various result datas and decision-making.The data obtained, for concrete educational applications, are finally reached the mesh promoted that gives an impulse to education
, it is achieved wisdom education.Include but not limited to predict the total marks of the examination of student；The prediction proportion of students entering schools of a higher grade；Provide how student changes
Enter the suggestion of study；Provide how teacher improves the suggestion of teaching level；Provide how school improves management and service water product
Suggestion etc..
Fig. 6 is the structural representation educating big Data application system according to the preferred embodiment of the invention, and Fig. 6 is Fig. 5
A kind of variant.As shown in Figure 6, this system includes: data acquisition module, data processing module, data storage mould
Block, market demand module, wherein:
1) data acquisition module: for according to the interface type specified and characteristic requirements, obtain at different data sources
Initial data.
2) data processing module: be responsible for data cleaning, change, load, the function such as regulation management and transmission.To adopt
Collect to source data be changed into and meet the data of target data model.
3) data memory module: be used for realizing mass data storage.
4) market demand module: for data mining analysis, and provide intelligent decision for end user.
Fig. 7 is the schematic flow sheet educating big data application method according to the preferred embodiment of the invention, as it is shown in fig. 7,
This flow process comprises the steps:
Step S701: application system (such as, information-based Classroom System, the examination that data acquisition module is correlated with from education
System, logistic system, teaching and administrative staff's performance management etc.) middle collection data.Between data acquisition module and each educational applications system
Interface include but not limited to file transfer protocol (FTP) (FTP), HTML (Hypertext Markup Language) (HTTP) etc..
Step S702: data processing module is according to the rule defined, and be carried out data and conversion etc. processes, and makes
Data become the data meeting target data model, to meet the requirement of follow-up storage and application.
Step S703: data after treatment are stored in data memory module.Wherein, data memory module is permissible
Use cloud storage technology, including distributed document storage, distributed data library storage etc..
Step S704: the modeling tool that application developer (i.e. user) uses market demand module to provide is modeled,
The process of modeling is exactly design calculation formula, and indicates which data of employing substitute into formula and calculate.Application developer makes
The application development tool provided by market demand module develops concrete educational applications, uses the number that formula calculates in application
According to, finally give analysis result.
Step S705: be that student, teacher, school, the head of a family are relevant with other education according to the intellectual analysis result obtained
User service, include but not limited to: prediction student total marks of the examination；The prediction proportion of students entering schools of a higher grade；Provide how student improves
The suggestion practised；Provide how teacher improves the suggestion of teaching level；Provide how school improves building of management and service water product
View etc..It is finally reached the purpose promoted that gives an impulse to education, it is achieved wisdom education.
In sum, by the above embodiment of the present invention and preferred embodiment, use big data technique, it is possible to achieve intelligence
Intelligent education.Such as, above preferred embodiment provide the big Data application system of education and method, run through " religion ", " learning ",
" manage " whole process, many-sided demand of school, teacher, the head of a family, student can be met simultaneously.
In another embodiment, additionally providing a kind of software, this software is used for performing above-described embodiment and the most real
Execute the technical scheme described in mode.
In another embodiment, additionally providing a kind of storage medium, in this storage medium, storage has above-mentioned software,
This storage medium includes but not limited to: CD, floppy disk, hard disk, scratch pad memory etc..
It should be noted that term " first " in description and claims of this specification and above-mentioned accompanying drawing, " second "
Etc. being for distinguishing similar object, without being used for describing specific order or precedence.Should be appreciated that and so make
Object can exchange in the appropriate case, in order to embodiments of the invention described herein can be with except here
Order beyond those of diagram or description is implemented.Additionally, term " includes " and " having " and their any deformation,
Be intended to cover non-exclusive comprising, such as, contain series of steps or the process of unit, method, system,
Product or equipment are not necessarily limited to those steps or the unit clearly listed, but can include the most clearly listing or
Other step intrinsic for these processes, method, product or equipment or unit.
Obviously, those skilled in the art should be understood that each module of the above-mentioned present invention or each step can be with general
Calculating device realize, they can concentrate on single calculating device, or is distributed in multiple calculating device institute
On the network of composition, alternatively, they can realize with calculating the executable program code of device, it is thus possible to
It is stored in storing in device and is performed by calculating device, and in some cases, can be to be different from herein
Order perform shown or described by step, or they are fabricated to respectively each integrated circuit modules, or will
Multiple modules or step in them are fabricated to single integrated circuit module and realize.So, the present invention is not restricted to appoint
What specific hardware and software combines.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for the skill of this area
For art personnel, the present invention can have various modifications and variations.All within the spirit and principles in the present invention, made
Any modification, equivalent substitution and improvement etc., should be included within the scope of the present invention.
Claims (12)
1. a data processing method, it is characterised in that including:
Gather the initial data from data source；
Changing described initial data is the first data meeting target data model, and wherein, described first data include
The feature of at least one of: consolidation form encodes, unified data type, unified data form；
Store described first data.
Method the most according to claim 1, it is characterised in that gather the described raw data packets from described data source
Include:
Cycle gathers the described initial data from described data source；Or
Acquisition condition instantaneous acquiring according to setting is from the described initial data of described data source.
Method the most according to claim 1, it is characterised in that change described initial data into described first data it
Before, described method also includes:
According to preset strategy, reject the irregular data in described initial data and/or do not meet factual data.
Method the most according to claim 1, it is characterised in that change described initial data into described first data it
After, described method also includes:
Described first data carry out data summarization, and wherein, described data summarization includes at least one of: collect
Time granularity, collect network element granularity, collect spatial granularity, collect business granularity.
Method the most according to claim 1, it is characterised in that store described first data and include:
The mode using redundant storage stores described first data.
Method the most according to any one of claim 1 to 5, it is characterised in that after storing described first data,
Described method also includes:
Obtain the data model that user sets up；
Data needed for data model described in described first extracting data；
Export the result of calculation of described data model.
7. a data processing equipment, it is characterised in that including:
Acquisition module, for gathering the initial data from data source；
Modular converter, is the first data meeting target data model, wherein, institute for changing described initial data
State the first data and include the feature of at least one of: consolidation form encodes, unified data type, unified number
According to form；
Memory module, is used for storing described first data.
Device the most according to claim 7, it is characterised in that described acquisition module is used for:
Cycle gathers the described initial data from described data source；Or
Acquisition condition instantaneous acquiring according to setting is from the described initial data of described data source.
Device the most according to claim 7, it is characterised in that described device also includes:
Reject module, for according to preset strategy, reject the irregular data in described initial data and/or be not inconsistent
Close factual data.
Device the most according to claim 7, it is characterised in that described device also includes:
Summarizing module, for described first data are carried out data summarization, wherein, described data summarization includes following
At least one: collect time granularity, collect network element granularity, collect spatial granularity, collect business granularity.
11. devices according to claim 7, it is characterised in that
Described memory module, for using the mode of redundant storage to store described first data.
12. according to the device according to any one of claim 7 to 11, it is characterised in that described device also includes:
Acquisition module, for obtaining the data model that user sets up；
Extraction module, for the data needed for data model described in described first extracting data；
Output module, for exporting the result of calculation of described data model.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510263271.0A CN106296498A (en) | 2015-05-21 | 2015-05-21 | Data processing method and device |
| PCT/CN2016/073956 WO2016184192A1 (en) | 2015-05-21 | 2016-02-17 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510263271.0A CN106296498A (en) | 2015-05-21 | 2015-05-21 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106296498A true CN106296498A (en) | 2017-01-04 |
Family
ID=57319343
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510263271.0A Pending CN106296498A (en) | 2015-05-21 | 2015-05-21 | Data processing method and device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN106296498A (en) |
| WO (1) | WO2016184192A1 (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108121508A (en) * | 2017-12-15 | 2018-06-05 | 华中师范大学 | Multi-source heterogeneous data collecting system and processing method based on education big data |
| CN108268645A (en) * | 2018-01-23 | 2018-07-10 | 广州南方人才资讯科技有限公司 | Big data processing method and system |
| CN108416506A (en) * | 2018-02-07 | 2018-08-17 | 平安科技（深圳）有限公司 | Customer risk level management approach, server and computer readable storage medium |
| CN108921747A (en) * | 2018-07-06 | 2018-11-30 | 重庆和贯科技有限公司 | Make the wisdom education system of student's feeling of immersion |
| CN109558400A (en) * | 2018-11-28 | 2019-04-02 | 北京锐安科技有限公司 | Data processing method, device, equipment and storage medium |
| CN109597846A (en) * | 2018-10-22 | 2019-04-09 | 平安科技（深圳）有限公司 | Big data platform data warehouse data processing method, device and computer equipment |
| CN110069553A (en) * | 2019-04-28 | 2019-07-30 | 中国疾病预防控制中心 | A kind of the data acquisition and processing method, equipment of public health emergency |
| CN112947263A (en) * | 2021-04-20 | 2021-06-11 | 南京云玑信息科技有限公司 | Management control system based on data acquisition and coding |
| CN113190608A (en) * | 2021-05-28 | 2021-07-30 | 北京红山信息科技研究院有限公司 | Data standardized acquisition method, device, equipment and storage medium |
| CN117407381A (en) * | 2023-09-26 | 2024-01-16 | 陕西小保当矿业有限公司 | Real-time processing method and device for big data of mine industry |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111427946A (en) * | 2020-04-16 | 2020-07-17 | 北京搜狐互联网信息服务有限公司 | Data processing method and device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663659A (en) * | 2012-03-27 | 2012-09-12 | 上海爱友科技有限公司 | Education system based on academic achievement development index |
| CN103473719A (en) * | 2013-09-26 | 2013-12-25 | 杭州意能软件有限公司 | Data acquisition method, device and system |
| CN103676798A (en) * | 2012-09-10 | 2014-03-26 | 任伟 | United monitoring platform |
| CN104134100A (en) * | 2014-07-22 | 2014-11-05 | 香港佳能通节能科技有限公司 | Energy-saving management system based on cloud computing |
| CN104462604A (en) * | 2014-12-31 | 2015-03-25 | 成都市卓睿科技有限公司 | Data processing method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9286595B2 (en) * | 2006-08-02 | 2016-03-15 | Emc Corporation | System and method for collecting and normalizing entitlement data within an enterprise |
-
2015
- 2015-05-21 CN CN201510263271.0A patent/CN106296498A/en active Pending
-
2016
- 2016-02-17 WO PCT/CN2016/073956 patent/WO2016184192A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663659A (en) * | 2012-03-27 | 2012-09-12 | 上海爱友科技有限公司 | Education system based on academic achievement development index |
| CN103676798A (en) * | 2012-09-10 | 2014-03-26 | 任伟 | United monitoring platform |
| CN103473719A (en) * | 2013-09-26 | 2013-12-25 | 杭州意能软件有限公司 | Data acquisition method, device and system |
| CN104134100A (en) * | 2014-07-22 | 2014-11-05 | 香港佳能通节能科技有限公司 | Energy-saving management system based on cloud computing |
| CN104462604A (en) * | 2014-12-31 | 2015-03-25 | 成都市卓睿科技有限公司 | Data processing method and system |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108121508A (en) * | 2017-12-15 | 2018-06-05 | 华中师范大学 | Multi-source heterogeneous data collecting system and processing method based on education big data |
| CN108268645A (en) * | 2018-01-23 | 2018-07-10 | 广州南方人才资讯科技有限公司 | Big data processing method and system |
| CN108416506B (en) * | 2018-02-07 | 2022-08-02 | 平安科技（深圳）有限公司 | Client risk level management method, server and computer readable storage medium |
| CN108416506A (en) * | 2018-02-07 | 2018-08-17 | 平安科技（深圳）有限公司 | Customer risk level management approach, server and computer readable storage medium |
| WO2019153598A1 (en) * | 2018-02-07 | 2019-08-15 | 平安科技（深圳）有限公司 | Customer risk level management method, server and computer readable storage medium |
| CN108921747A (en) * | 2018-07-06 | 2018-11-30 | 重庆和贯科技有限公司 | Make the wisdom education system of student's feeling of immersion |
| CN109597846A (en) * | 2018-10-22 | 2019-04-09 | 平安科技（深圳）有限公司 | Big data platform data warehouse data processing method, device and computer equipment |
| CN109597846B (en) * | 2018-10-22 | 2024-05-07 | 平安科技（深圳）有限公司 | Data processing method, device and computer equipment for large data platform data warehouse |
| CN109558400A (en) * | 2018-11-28 | 2019-04-02 | 北京锐安科技有限公司 | Data processing method, device, equipment and storage medium |
| CN110069553A (en) * | 2019-04-28 | 2019-07-30 | 中国疾病预防控制中心 | A kind of the data acquisition and processing method, equipment of public health emergency |
| CN112947263A (en) * | 2021-04-20 | 2021-06-11 | 南京云玑信息科技有限公司 | Management control system based on data acquisition and coding |
| CN113190608A (en) * | 2021-05-28 | 2021-07-30 | 北京红山信息科技研究院有限公司 | Data standardized acquisition method, device, equipment and storage medium |
| CN117407381A (en) * | 2023-09-26 | 2024-01-16 | 陕西小保当矿业有限公司 | Real-time processing method and device for big data of mine industry |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2016184192A1 (en) | 2016-11-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106296498A (en) | Data processing method and device | |
| Cheng et al. | Using multiple criteria decision analysis for supporting decisions of solid waste management | |
| Geertman et al. | Planning support systems best practice and new methods | |
| Hu et al. | A non-dominated sorting genetic algorithm for the location and districting planning of earthquake shelters | |
| CN110428890A (en) | A kind of medical treatment hospital guide method and system | |
| Wray et al. | Key challenges and potential urban modelling opportunities in South Africa, with specific reference to the Gauteng City-Region | |
| Geertman et al. | Planning support systems: content, issues and trends | |
| Forsman et al. | Considering student retention as a complex system: a possible way forward for enhancing student retention | |
| CN110737776A (en) | path learning planning system based on knowledge graph and target ontology | |
| CN111160857A (en) | Hospital standard culture clinical teaching management system based on internet platform | |
| Tucker et al. | Thermal simulation outputs: exploring the concept of patterns in design decision-making | |
| He | Efficient computational evaluation tools to accelerate the planning of vulnerability, resilience, and sustainability of the social-environmental Systems in the City of Nashville | |
| Sarran et al. | Untold Stories from the Field: a Novel Platform for Collecting Practical Learnings on Human Building Interactions | |
| Howick et al. | A hybrid simulation modelling framework for combining system dynamics and agent-based models | |
| Grigg et al. | Information technology in civil engineering curriculum | |
| Abisuga et al. | Towards conceptualization of SocioBIM-based post-occupancy evaluation framework for learning spaces | |
| Delmas et al. | Towards context & climate sensitive urban design: An integrated simulation and parametric design approach | |
| Corsi et al. | Technology Transfer, Anthropotechnology and Sustainable Development: How Do the Themes Relate? | |
| Beevers et al. | A hazard-agnostic model for unpacking systemic impacts in urban systems | |
| Schimpf et al. | Work in progress: Visualizing design team Analytics for representing and understanding design teams’ process | |
| Seifert et al. | Urban strategy playground: Rethinking the urban planner’s toolbox | |
| Liu et al. | Information resource sharing based on university dance big data management and utilization system design | |
| Stojanova et al. | Agent-based solution of caregiver scheduling problem in home-care context | |
| Lavy et al. | A Framework for Key Performance Indicators for a Holistic Facility Performance Assessment | |
| Feher | Recommendations, best practices and key factors of smart city and smart citizenship |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20170104 |
|
| WD01 | Invention patent application deemed withdrawn after publication |