CN108304549A - A kind of big data Intelligent processing system - Google Patents
A kind of big data Intelligent processing system Download PDFInfo
- Publication number
- CN108304549A CN108304549A CN201810100200.2A CN201810100200A CN108304549A CN 108304549 A CN108304549 A CN 108304549A CN 201810100200 A CN201810100200 A CN 201810100200A CN 108304549 A CN108304549 A CN 108304549A
- Authority
- CN
- China
- Prior art keywords
- big data
- analysis
- module
- intelligent processing
- processing system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/22—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides a kind of big data Intelligent processing system, the system comprises：Big data platform, for storing the big data for receiving and having acquired；Big data preprocessing module, for being pre-processed to the big data acquired；The pretreatment is for realizing load balancing, resource virtualizing and Distributed Storage management；Intelligent analysis module for carrying out the analysis of the big data according to the instruction received, and comprehensively utilizes the data of big data preprocessing module offer, carries out event causality analysis；Application service for obtaining user is asked；It determines that corresponding Processing Algorithm is asked in the application service, and asks the high dimension vector for the event causality analysis that corresponding Processing Algorithm and the representative learning module obtain using the application service, the high dimension vector is handled using neural network.Big data Intelligent processing system is provided using the present invention, solves the problems, such as that traditional data processing method real-time, high efficiency and interactivity are poor.
Description
Technical field
The present invention relates to electronic technology field more particularly to a kind of big data Intelligent processing systems.
Background technology
With society's industrialization, the continuous improvement of the level of IT application, nowadays data, which have replaced, is calculated as information calculating
Center, cloud computing, big data are becoming a kind of trend and trend, including memory capacity, availability, I/O performances, data peace
All various aspects such as Quan Xing, scalability.Big data is the very huge and complicated data set of scale.Big data has 4V：Volume
(a large amount of), data volume increases continuously and healthily；Velocity (high speed), data I/O speed are faster；Variety (various), data class
Type and source diversification；Value (value), there are the usable values of various aspects.How from the extracting data of magnanimity, obtain
Desired knowledge or interested information, this is the requirement made good use of big data, and then preferably serve social development.Cause
This, data digging method comes into being.
Data mining is born in as a subject in the 1980s, being exactly from large amount of complex from the point of view of technology
, obtain information implicit, that people do not realize in advance, having potential value in irregular, random, fuzzy data
With the process of knowledge.In big data application field, user group can be often divided into according to the various actions feature of user
If Ganlei, in order to carry out accurate formula, personalized service for the feature of user group.Cluster is divided to user group
A kind of mode.Cluster is so that the object in same class is had very high similarity the classified process of data object, and make
Object height in inhomogeneity is different.Distinctiveness ratio is measured usually using distance.
But the effect that user group is divided for user behavior characteristics in cluster operation largely according to
The quality in basic data, the existing user group based on clustering algorithm is relied to divide and often can not reflect user's well
There is cluster inaccuracy in behavioural characteristic, it is difficult to carry out accurate formula, personalized clothes to user group using cluster result
Business.
Invention content
The purpose of the present invention is to provide a kind of big data Intelligent processing systems, and it is real to solve traditional data processing method
Shi Xing, high efficiency and the poor problem of interactivity, help user to perceive conditions of the enterprise in real time, to improve enterprise management efficiency
With business processing level.
In order to achieve the above object, the invention discloses a kind of big data Intelligent processing system, the system comprises：
Big data platform, for storing the big data for receiving and having acquired；
Big data preprocessing module, for being pre-processed to the big data acquired；The pretreatment is for realizing negative
Carry balanced, resource virtualizing and Distributed Storage management；
Intelligent analysis module, the analysis for carrying out the big data according to the instruction received, and it is pre- to comprehensively utilize big data
The data that processing module provides carry out event causality analysis；
Intelligent processing module, the application service for obtaining user are asked；Determine that the application service asks corresponding processing to be calculated
Method, and ask the event cause and effect that corresponding Processing Algorithm and the representative learning module obtain using the application service
The high dimension vector of analysis is handled the high dimension vector using neural network.
As a preferred technical solution of the present invention, the big data platform operation interface includes at least in following functions
It is one or more：With Visual Chart, analysis report, content retrieval and message push/subscription.
As a preferred technical solution of the present invention, the big data preprocessing module passes through distributed system foundation frame
The big data acquired described in structure Hadoop storages.
As a preferred technical solution of the present invention, the big data preprocessing module is for realizing load balancing, money
Source virtualization, Distributed Storage management and application programming interface api interface function.
As a preferred technical solution of the present invention, the analysis module is looked into for realizing extemporaneous inquiry/combination condition
The analytic functions such as inquiry, multidimensional OLAP, KPI indexs, MDX inquiries, while the data minings work(such as realize classification, cluster, correlation rule
Can, and flexible parameter configuration function；
As a preferred technical solution of the present invention, the intelligent processing module is used to be carried out according to preset data information real-time
Company's situation is assessed in real time；The preset data information includes：Manpower, finance, substance and business.
Compared with prior art, the present invention has the following advantages：
1, processing speed is fast：System architecture scheme using big data technology to calculating, store tasks carry out rational management, can be with
Give full play to the operational capability of each clustered node in system；It, can be conveniently by addition cluster when business demand increases
Node comes expansion system scale, lifting system performance.
2, better user experience：System supports multiple terminal operation, supports the real-time visual of feelings indexs in school at different levels, carries
For the interactive mode of simple, intuitive；
3, flexibility ratio is high：It can flexibly be created according to the actual conditions of this enterprise, Allocation Analysis model；System is set using layering
Meter is easy to deployment implementation, upgrade maintenance.
Description of the drawings
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below
There is attached drawing needed in technology description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with
Obtain other attached drawings according to these attached drawings.
Fig. 1 is the structural schematic diagram of the intelligent analysis system of big data provided in an embodiment of the present invention.
Specific implementation mode
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation describes, it is clear that described embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.It is based on
Embodiment in the present invention, it is obtained by those of ordinary skill in the art without making creative efforts every other
Embodiment shall fall within the protection scope of the present invention.
Fig. 1 is the flow diagram of the intelligent analysis system of big data provided in an embodiment of the present invention, and this method includes step
Suddenly：
Big data platform 11, for storing the big data for receiving and having acquired；
Big data preprocessing module 12, for being pre-processed to the big data acquired；It is described pretreatment for realizing
Load balancing, resource virtualizing and Distributed Storage management；
Intelligent analysis module 13 for carrying out the analysis of the big data according to the instruction received, and comprehensively utilizes big data
The data that preprocessing module provides carry out event causality analysis；
Intelligent processing module 14, the application service for obtaining user are asked；Determine that corresponding processing is asked in the application service
Algorithm, and ask the event that corresponding Processing Algorithm and the representative learning module obtain using the application service
The high dimension vector of causality analysis is handled the high dimension vector using neural network.
In the present embodiment, entire analysis platform is to realize cloud platform, cluster hardware configuration based on the Hadoop framework increased income
It is as follows：The memory of the GB of the CPU, 64 or 128 of 16 core, 32 thread, multiple default turns be directly connected to by the mainboard controller of CPU
The hard disk (always storing up to 24TB) of speed, and cluster is built using gigabit Ethernet.Wherein, the quantity of hard disk and rotating speed can roots
According to needing to be configured, if quantity is 20, rotating speed is 3600r/s etc..
Wherein, there are four types of basic task roles for Hadoop clusters：Name node (including alternative name node), work chase after
Track node, task execution node and back end.Name node is responsible for coordinating the data storage on cluster；Job trace node
It is responsible for coordination data and handles task；Task execution node is responsible for carrying out the tasks such as data acquisition, data processing；Back end is negative
Duty storage data.Most node needs while being used as back end and task execution node in cluster.
On the basis of Hadoop clusters, the support handled distributed parallel task is realized by Map/Reduce.
Map/Reduce is a programming model for being used for big data quantity parallel computation, while being also a kind of efficient task scheduling mould
Type, one big task is divided into many more fine-grained subtasks by it, by carrying out subtask between idle processing node
Scheduling, the node for avoiding processing speed slow extend the deadline of entire task.
In the present invention, entire intellectualized analysis platform includes three parts, such as big data preprocessing module 11, intellectual analysis mould
Block 12 and intelligent processing module 14；Wherein, big data warehouse has been built in big data preprocessing module 11, for storing ETL mistakes
Crucial initial data in the data source that journey extracts builds cube (Cube) on the basis of big data warehouse, is
Network analysis, displaying provide data and support.
Big data preprocessing module 12 is additionally operable to realize by cloud platform management module equal to the load of bottom layer node equipment
The functions such as weighing apparatus, resource virtualizing, Distributed Storage management, fault-tolerant strategy management and offer api interface, realize big number
According to processing and management.
Above-mentioned data source is each business department of enterprise independently all operation system and database, including HR Office's number
According to, Finance Department's data, research and development department's data, Finance Department's data, market department's data, data at assessment, interconnection wet end data and comprehensive
Close management data；Wherein, above-mentioned part may segment many small departments, if market department may further include secretary's group, market
Group etc., data at this time are included in market department.Big data preprocessing module passes through distributed system architecture Hadoop
The big data that has acquired described in storage, and for realizing load balancing, resource virtualizing, Distributed Storage management and
Application programming interface api interface function.
The intelligent analysis module 13, for realizing extemporaneous inquiry/combination condition query, multidimensional OLAP, KPI indexs, MDX
The analytic functions such as inquiry, while realizing the data mining capabilities such as classification, cluster, correlation rule, and flexible parameter configuration work(
Energy；Index evaluation module is assessed in real time for carrying out real-time company's situation according to preset data information；The present count it is believed that
Breath includes：Manpower, finance, substance and business.
Above-mentioned big data warehouse is based on HDFS and Hive and realizes, each phase of enterprise has been concentrated using distributed storage mode
Mass data in mutual independent operation system, data are provided for cube.Further, the number in big data warehouse
Dimension and the storage of the form of true table according to this, dimension here are the attributes of data, indicate that the angle of analysis data, type have
General dimension, time dimension and gradual change dimension；True table is the main table for storing the data to be analyzed, only includes major key, external key
And metric.
Operation interface includes at least one or more in following functions：With Visual Chart, analysis report, content inspection
Rope and message push/subscription.
In the embodiment of the present invention, intelligent processing module 14 has built the system users of B/S patterns using Javaweb technologies
End, realizes single-sign-on control, provides displaying, operation interface to the user；It is created using the ExtJS frames increased income abundant
Business information analysis assessment result, all kinds of details data hierarchy grades are carried out synthesis, showed in real time by figure and chart；System
System client has the function of Visual Chart, analysis report, content retrieval, message push/subscription etc., can pass through browser
Operate in mobile terminal, the ends PC.
It should be noted that herein, relational terms such as first and second and the like are used merely to a reality
Body or operation are distinguished with another entity or operation, are deposited without necessarily requiring or implying between these entities or operation
In any actual relationship or order or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to
Non-exclusive inclusion, so that the process, method, article or equipment including a series of elements is not only wanted including those
Element, but also include other elements that are not explicitly listed, or further include for this process, method, article or equipment
Intrinsic element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that
There is also other identical elements in process, method, article or equipment including the element.
The foregoing is merely illustrative of the preferred embodiments of the present invention, is not intended to limit the scope of the present invention.It is all
Any modification, equivalent replacement, improvement and so within the spirit and principles in the present invention, are all contained in protection scope of the present invention
It is interior.
Claims (7)
1. a kind of big data Intelligent processing system, which is characterized in that the system comprises：
Big data platform, for storing the big data for receiving and having acquired；
Big data preprocessing module, for being pre-processed to the big data acquired；The pretreatment is for realizing negative
Carry balanced, resource virtualizing and Distributed Storage management；
Intelligent analysis module, the analysis for carrying out the big data according to the instruction received, and it is pre- to comprehensively utilize big data
The data that processing module provides carry out event causality analysis；
Intelligent processing module, the application service for obtaining user are asked；Determine that the application service asks corresponding processing to be calculated
Method, and ask the event cause and effect that corresponding Processing Algorithm and the representative learning module obtain using the application service
The high dimension vector of analysis is handled the high dimension vector using neural network.
2. a kind of intelligent processing system of big data according to claim 1, which is characterized in that the big data platform behaviour
Make interface including at least one or more in following functions：It is pushed away with Visual Chart, analysis report, content retrieval and message
It send/subscribes to.
3. a kind of intelligent processing system of big data according to claim 1, which is characterized in that the big data pretreatment
Module passes through the big data acquired described in the Hadoop storages of distributed system architecture.
4. a kind of intelligent processing system of big data according to claim 1, which is characterized in that the big data pretreatment
Module is for realizing load balancing, resource virtualizing, Distributed Storage management and application programming interface api interface
Function.
5. a kind of intelligent processing system of big data according to claim 1, which is characterized in that the analysis module is used
In analytic functions such as the extemporaneous inquiry/combination condition query of realization, multidimensional OLAP, KPI indexs, MDX inquiries.
6. a kind of intelligent processing system of big data according to claim 1, which is characterized in that the analysis module also may be used
With realizing the data mining capabilities such as classification, cluster, correlation rule, and flexible parameter configuration function.
7. a kind of intelligent processing system of big data according to claim 1, which is characterized in that the intelligent processing module
It is assessed in real time for carrying out real-time company's situation according to preset data information；The preset data information includes：Manpower, finance,
Substance and business.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810100200.2A CN108304549A (en) | 2018-02-01 | 2018-02-01 | A kind of big data Intelligent processing system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810100200.2A CN108304549A (en) | 2018-02-01 | 2018-02-01 | A kind of big data Intelligent processing system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108304549A true CN108304549A (en) | 2018-07-20 |
Family
ID=62850651
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810100200.2A Withdrawn CN108304549A (en) | 2018-02-01 | 2018-02-01 | A kind of big data Intelligent processing system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108304549A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109067690A (en) * | 2018-08-07 | 2018-12-21 | 腾讯科技（深圳）有限公司 | The method for pushing and device of off-line calculation result data |
| CN109583712A (en) * | 2018-11-13 | 2019-04-05 | 咪咕文化科技有限公司 | Data index analysis method and device and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106445988A (en) * | 2016-06-01 | 2017-02-22 | 上海坤士合生信息科技有限公司 | Intelligent big data processing method and system |
| CN107590181A (en) * | 2017-08-01 | 2018-01-16 | 佛山市深研信息技术有限公司 | A kind of intelligent analysis system of big data |
-
2018
- 2018-02-01 CN CN201810100200.2A patent/CN108304549A/en not_active Withdrawn
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106445988A (en) * | 2016-06-01 | 2017-02-22 | 上海坤士合生信息科技有限公司 | Intelligent big data processing method and system |
| CN107590181A (en) * | 2017-08-01 | 2018-01-16 | 佛山市深研信息技术有限公司 | A kind of intelligent analysis system of big data |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109067690A (en) * | 2018-08-07 | 2018-12-21 | 腾讯科技（深圳）有限公司 | The method for pushing and device of off-line calculation result data |
| CN109583712A (en) * | 2018-11-13 | 2019-04-05 | 咪咕文化科技有限公司 | Data index analysis method and device and storage medium |
| CN109583712B (en) * | 2018-11-13 | 2021-06-29 | 咪咕文化科技有限公司 | Data index analysis method and device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107590181A (en) | A kind of intelligent analysis system of big data | |
| CN110199273B (en) | System and method for loading, aggregating and bulk computing in one scan in a multidimensional database environment | |
| CN104573071A (en) | Intelligent school situation analysis system and method based on megadata technology | |
| CN1956457B (en) | Method and apparatus for arranging mesh work in mesh computing system | |
| Chen et al. | How does the workload look like in production cloud? analysis and clustering of workloads on alibaba cluster trace | |
| CN104915793A (en) | Public information intelligent analysis platform based on big data analysis and mining | |
| CN107766402A (en) | A kind of building dictionary cloud source of houses big data platform | |
| CN107193967A (en) | A kind of multi-source heterogeneous industry field big data handles full link solution | |
| CN108322548A (en) | A kind of industrial process data analyzing platform based on cloud computing | |
| CN104951425A (en) | Cloud service performance adaptive action type selection method based on deep learning | |
| CN108038239A (en) | A kind of heterogeneous data source method of standardization management, device and server | |
| CN109075988A (en) | Task schedule and resource delivery system and method | |
| CA2587698A1 (en) | Performance monitoring within an enterprise software system | |
| CN107291539B (en) | Cluster program scheduler method based on resource significance level | |
| US20150271023A1 (en) | Cloud estimator tool | |
| CN114416855A (en) | Visualization platform and method based on electric power big data | |
| CN110928740A (en) | Centralized visualization method and system for operation and maintenance data of cloud computing center | |
| Chen et al. | Development and application of big data platform for garlic industry chain | |
| CN108399208A (en) | A kind of information display system of big data | |
| CN112632025A (en) | Power grid enterprise management decision support application system based on PAAS platform | |
| Parygin et al. | A convergent model for distributed processing of Big Sensor Data in urban engineering networks | |
| CN108304549A (en) | A kind of big data Intelligent processing system | |
| CN108363756A (en) | A kind of intelligent transportation big data processing system | |
| CN106575296A (en) | Dynamic N-dimensional cubes for hosted analytics | |
| CN108306916A (en) | Big data multi-internet integration scientific research all-in-one machine stage apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20180720 |
|
| WW01 | Invention patent application withdrawn after publication |