CN106469211B - The memory loading method and device of big data - Google Patents
The memory loading method and device of big data Download PDFInfo
- Publication number
- CN106469211B CN106469211B CN201610801380.8A CN201610801380A CN106469211B CN 106469211 B CN106469211 B CN 106469211B CN 201610801380 A CN201610801380 A CN 201610801380A CN 106469211 B CN106469211 B CN 106469211B
- Authority
- CN
- China
- Prior art keywords
- data
- snapshot
- memory
- divided
- relationships
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
- G06F16/2219—Large Object storage; Management thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/282—Hierarchical databases, e.g. IMS, LDAP data stores or Lotus Notes
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of memory loading methods of big data, comprising steps of data are divided and merged according to class, form multiclass object data, and be stored in object database；Multiclass object data ordered implicative is formed into multiple tree-shaped relational networks according to the association sequence in relationship；Since top, the overlying relation of each tree-shaped relational network is recorded, multiple snapshots is formed, is stored in snapshot database；When needing to load data, all relationships in snapshot are obtained, and all relationships are loaded into memory.The invention also discloses a kind of memory loading devices of big data.Data treatment effeciency loaded into memory can be improved in the present invention.
Description
Technical field
The present invention relates to data load more particularly to the memory loading methods and device of a kind of big data.
Background technique
Information management system of enterprise, such as PDM system, ERP system, MES system, SRM system etc., these systems point
Not Fu Ze product design, production plan and the inventory of enterprise, financial management, manufacture execute with apparatus management/control, supplier management etc.,
Related data category has product, components, drawing, technique, set of books, buying order, work to enable list, material requistion, equipment, maintenance etc.
Have following features Deng, these data: amount is big, and type is more, and relationship is numerous and jumbled, the difference described additionally, due to information system to data
Property, leading to a kind of data, there are multiple descriptions, repeat, intersect so that data exist.The number of such many kinds of, magnanimity, intertexture
According to big data is formed, when these big datas are loaded into information system progress service logic operation, information system needs were screened
These data are filtered, causing information system to load, data are slow, logical process is difficult.
Summary of the invention
Goal of the invention: the present invention in view of the problems of the existing technology, provide a kind of big data memory loading method and
Data treatment effeciency loaded into memory can be improved in device, the present invention.
The present invention provides a kind of memory loading methods of big data, comprising the following steps:
Data are divided and merged according to class, form multiclass object data, and be stored in object database；
Multiclass object data ordered implicative is formed into multiple tree-shaped relational networks according to the association sequence in relationship；
Since top, the overlying relation of each tree-shaped relational network is recorded, multiple snapshots is formed, is stored in
In snapshot database；
When needing to load data, all relationships in snapshot are obtained, and all relationships are loaded into memory.
Further, described that data are divided and merged according to class, it specifically includes:
Of a sort object data will be belonged to and be divided into a kind of data；
In same class data, the identical different object datas of determinant attribute are merged, thus by a kind of number of objects
According to organizing together.
The present invention also provides a kind of memory loading devices of big data, comprising:
Data division module forms multiclass object data, and be stored in for data to be divided and merged according to class
In object database；
Data association module, for forming multiclass object data ordered implicative multiple according to the association sequence in relationship
Tree-shaped relational network；
Snapshot generation module, for the overlying relation of each tree-shaped relational network being recorded, shape since top
At multiple snapshots, it is stored in snapshot database；
Snapshot loading module, for obtaining all relationships in snapshot, and all relationships are added when needing to load data
It is downloaded in memory.
Further, the data division module specifically includes:
Data dividing unit is divided into a kind of data for that will belong to of a sort object data；
Data combination unit, in same class data, the identical different object datas of determinant attribute to be merged,
To which a kind of object data be organized together.
The utility model has the advantages that compared with prior art, the present invention its remarkable advantage is:
1, it is disposably loaded using snapshot, memory loading velocity is faster；
2, the relationship in snapshot is definable, by obtaining this definition, then uses join sentence in the database
It carries out SQL to piece together, original layer-by-layer search tree can be become to once (or several times) search tree, database access number and subtracted significantly
It is few, so improving efficiency；
3, data object is all in object database, if a data object is created, first according to this class object
It finds in matching rule to object database, if there is identical, then be used directly, so improve Data duplication utilization rate.
Detailed description of the invention
Fig. 1 is the flow diagram of one embodiment of the present of invention；
Fig. 2 is that data merge exemplary diagram；
Fig. 3 is tree-shaped relational network exemplary diagram；
Fig. 4 is the first sample BOM structure chart of supplementary material needed for each road manufacturing procedure of product；
Fig. 5 is the lower list BOM structure chart of supplementary material needed for each road manufacturing procedure of product.
Specific embodiment
As shown in Figure 1, the memory loading method of the big data of the present embodiment the following steps are included:
S1, data are divided and is merged according to class, form multiclass object data, and be stored in object database.
Wherein, divide according to class and merging specifically includes step: (1) of a sort object data will be belonged to and be divided into
A kind of data；For example, there are a variety of different product classes in data, only it is responsible for the relevant characteristic of description product in product class, such as: name
Title, model, function etc. have different part classes under product class, therefore, product class can be divided into a kind of object data,
Part class is divided into a kind of object data.(2) in same class data, by the identical different object datas of determinant attribute into
Row merges, so that a kind of object data be organized together；For example, such as components, each components have one it is unique
COM code, the identical components of COM code be exactly it is same, therefore, can be merged into one；As shown in Fig. 2, number of objects
According to A1 and object data A2, there are attribute B and C in object data A1, has attribute E and C in object data A2, determinant attribute C will
After object A1 and object A2 merges, object data A, attribute E, B, C are obtained.
S2, multiclass object data ordered implicative is formed into multiple tree-shaped relational networks according to the association sequence in relationship.
For example, product is assemblied by components according to assembly relation, each components have one or multiple drawings, often
A drawing may have one or more version, material is thus formed product-components-drawing-drawing version object network,
As shown in figure 3, product 1 includes components 1, components 1 are made of sub- components 2 and sub- components 3, have one under sub- components 2
Drawing 1 is opened, to constitute a tree-shaped relational network.
S3, since top, the overlying relation of each tree-shaped relational network is recorded, multiple snapshots is formed, deposits
It is stored in snapshot database.
For example, overlying relation is recorded for tree network shown in Fig. 3 since top, obtain snapshot such as
Shown in table 1.
1 snapshot structure table of table
| Product 1 | Components 1 | Snapshot 1 |
| Components 1 | Sub- components 1 | Snapshot 1 |
| Components 1 | Sub- components 3 | Snapshot 1 |
| Sub- components 2 | Drawing 1 | Snapshot 1 |
In table 1, the first row indicates there are components 1 under product 1, and the second row indicates there is sub- components 2, third under components 1
Row indicates there are sub- components 3 under components 1, and the third line indicates there is a drawing 1 under sub- components 2, and product 1 and components have one
A constituent relation, components 1 and sub- components 2, sub- components 3 have an assembly relation, and sub- components 2 and drawing 1 have one
Holding relationship, these relationships are all recorded in snapshot 1.
S4, when needing to load data, all relationships in snapshot are obtained, and all relationships are loaded into memory.
The memory loading device and methods described above of the big data of the present embodiment correspond, and specifically include:
Data division module forms multiclass object data, and be stored in for data to be divided and merged according to class
In object database；
Data association module, for forming multiclass object data ordered implicative multiple according to the association sequence in relationship
Tree-shaped relational network；
Snapshot generation module, for the overlying relation of each tree-shaped relational network being recorded, shape since top
At multiple snapshots, it is stored in snapshot database；
Snapshot loading module, for obtaining all relationships in snapshot, and all relationships are added when needing to load data
It is downloaded in memory.
Wherein, the data division module specifically includes:
Data dividing unit is divided into a kind of data for that will belong to of a sort object data；
Data combination unit, in same class data, the identical different object datas of determinant attribute to be merged,
To which a kind of object data be organized together.
When the present embodiment is applied in family's spinning system, as shown in Figure 4 and Figure 5, product ' Anna and state are had recorded respectively
The first sample BOM and lower list BOM structure chart of supplementary material needed for king ' each road manufacturing procedure, object data classification involved in the figure have
Product, semi-finished product, supplementary material, process, relationship include product-semi-finished product, semi-finished product-supplementary material, semi-finished product-process, in product
Two snapshots are had recorded in class, snapshot title is first sample BOM respectively and lower list BOM, first sample BOM are exactly the composition of product sample
BOM, lower list BOM are exactly the composition BOM of formal product, and all objects and relationship in two snapshots are loaded using the invention method
Data three times or more are improved in speed than existing method.
Above disclosed is only a preferred embodiment of the present invention, and the right model of the present invention cannot be limited with this
It encloses, therefore equivalent changes made in accordance with the claims of the present invention, is still within the scope of the present invention.
Claims (2)
1. a kind of memory loading method of big data, it is characterised in that this method comprises:
Data are divided and merged according to class, form multiclass object data, and be stored in object database, wherein institute
It states and data is divided and merged according to class method particularly includes: of a sort object data will be belonged to be divided into a kind of number
According to；In same class data, the identical different object datas of determinant attribute are merged, thus by a kind of object data tissue
Together；
Multiclass object data ordered implicative is formed into multiple tree-shaped relational networks according to the association sequence in relationship；
Since top, the overlying relation of each tree-shaped relational network is recorded, multiple snapshots is formed, is stored in snapshot
In database；
When needing to load data, all relationships in snapshot are obtained, and all relationships are loaded into memory.
2. a kind of memory loading device of big data, it is characterised in that the device includes:
Data division module forms multiclass object data, and be stored in object for data to be divided and merged according to class
In database；The data division module specifically includes:
Data dividing unit is divided into a kind of data for that will belong to of a sort object data；
Data combination unit, in same class data, the identical different object datas of determinant attribute to be merged, thus
A kind of object data is organized together；
Data association module, for forming multiclass object data ordered implicative multiple tree-shaped according to the association sequence in relationship
Relational network；
Snapshot generation module is formed more for since top, the overlying relation of each tree-shaped relational network to be recorded
A snapshot, is stored in snapshot database；
Snapshot loading module, for obtaining all relationships in snapshot, and all relationships are loaded into when needing to load data
In memory.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610801380.8A CN106469211B (en) | 2016-09-05 | 2016-09-05 | The memory loading method and device of big data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610801380.8A CN106469211B (en) | 2016-09-05 | 2016-09-05 | The memory loading method and device of big data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106469211A CN106469211A (en) | 2017-03-01 |
| CN106469211B true CN106469211B (en) | 2019-10-25 |
Family
ID=58230090
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610801380.8A Active CN106469211B (en) | 2016-09-05 | 2016-09-05 | The memory loading method and device of big data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106469211B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101232538A (en) * | 2007-12-28 | 2008-07-30 | 华为技术有限公司 | Apparatus and method for merging business data |
| CN101661507A (en) * | 2009-09-25 | 2010-03-03 | 金蝶软件（中国）有限公司 | Method for merging data and system thereof |
| CN102012947A (en) * | 2010-12-16 | 2011-04-13 | 创新科存储技术有限公司 | Method and system for online backup of database |
| CN103268270A (en) * | 2013-05-10 | 2013-08-28 | 曙光信息产业(北京)有限公司 | Method and device for managing snapshot |
-
2016
- 2016-09-05 CN CN201610801380.8A patent/CN106469211B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101232538A (en) * | 2007-12-28 | 2008-07-30 | 华为技术有限公司 | Apparatus and method for merging business data |
| CN101661507A (en) * | 2009-09-25 | 2010-03-03 | 金蝶软件（中国）有限公司 | Method for merging data and system thereof |
| CN102012947A (en) * | 2010-12-16 | 2011-04-13 | 创新科存储技术有限公司 | Method and system for online backup of database |
| CN103268270A (en) * | 2013-05-10 | 2013-08-28 | 曙光信息产业(北京)有限公司 | Method and device for managing snapshot |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106469211A (en) | 2017-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6901243B2 (en) | Systems and methods for optimizing control systems for process environments | |
| GB2581861A (en) | Industrial internet-based informatization management system for integrated circuit testing | |
| CN104598376A (en) | Data driving layered automation test system and method | |
| EP2321742A1 (en) | A method of generating an analytical data set for input into an analytical model | |
| CN109508355A (en) | A kind of data pick-up method, system and terminal device | |
| US10706077B2 (en) | Performance of distributed databases and database-dependent software applications | |
| CN104462169A (en) | Method and a system for merging data into a database table | |
| CN111177847B (en) | Civil aircraft configuration type configuration management method and device | |
| CN109947741B (en) | Method for modeling and storing attribute parameters of items | |
| CN116171453A (en) | Method and apparatus for generating and utilizing knowledge patterns for manufacturing simulation models | |
| US20220334572A1 (en) | Method for Generating a Digital Twin of a System or Device | |
| CN109635267A (en) | A kind of fishing expedition report-generating method and device | |
| US10664149B2 (en) | Method for configuring a MOM data warehouse and providing a UI to the MOM data warehouse configuration and computer readable medium | |
| Lugaresi et al. | Automated digital twin generation of manufacturing systems with complex material flows: graph model completion | |
| EP1510935A1 (en) | Mapping a data from a data warehouse to a data mart | |
| CN106469211B (en) | The memory loading method and device of big data | |
| US20220342372A1 (en) | Method for Generating Self-Description Data Modules | |
| CN105354671A (en) | Universal configuration management system and enterprise-level information system comprising same | |
| CN108062395A (en) | A kind of track traffic big data analysis method and system | |
| CN104778253B (en) | A kind of method and apparatus that data are provided | |
| Zhang et al. | Product manufacturing process similarity measure based on attributed graph matching | |
| CN105989060A (en) | Data management method and device | |
| Hu et al. | A new product family mining method based on PLM database | |
| Hossain et al. | Integrated modeling and application of standardized data schema | |
| Savino et al. | A structured approach to implement product traceability for ISO 9001: 2015 based on unified modeling language |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |