CN107526832A - A kind of method for building the big data business model that technology is pulled based on the page - Google Patents
A kind of method for building the big data business model that technology is pulled based on the page Download PDFInfo
- Publication number
- CN107526832A CN107526832A CN201710788729.3A CN201710788729A CN107526832A CN 107526832 A CN107526832 A CN 107526832A CN 201710788729 A CN201710788729 A CN 201710788729A CN 107526832 A CN107526832 A CN 107526832A
- Authority
- CN
- China
- Prior art keywords
- model
- big data
- page
- business
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a kind of method for the big data business model for building and technology being pulled based on the page, this method realizes the visual setting of each node parameter in business model by page dragging technology, and the relation established between node, Work flow model is formed, it is specific as follows：Establish the executor of big data platform user, i.e. business model；Disposition data source；Model is established, complete business model is formed by pulling each Service Component；Model trial operation；Issuance model, model state are to call；Module scheduling is set, and timing performs business model；Model running, the operation that the time point set according to scheduling performs model operate.The visual workflow that the present invention can change into big data business component nodes is set, and simplifies the development of big data developer, allows business personnel and the workflow automation of Data Analyst participation big data business to configure.
Description
Technical field
The invention belongs to computer software areas of information technology, and technology is pulled based on the page more particularly to a kind of build
The method of big data business model.
Background technology
With the fast development of big data technology, major incorporated business, especially Internet enterprises, all from all angles
Gathered data, data storage, processing data, sharing data, retrieval data, analyze data, display data and mining data are behind
Commercial value, and by making one-stop big data analysis platform, the problem of neutralizing full-service chain big data analysis.With industry
Business application becomes more meticulous, and reduces the focus on research direction that data analysis difficulty has become each major company.With big data platform
Increasingly ripe, each business department also payes attention to all the more the research and application of big data problem.
But the structure of platform and the landing of application still have a certain distance, as big data business applied analysis is patrolled
The foundation of volume workflow, can not realize multiplexing, otherwise for big data analysis characteristic, it is necessary to the cooperation of many-sided technical staff,
Substantial amounts of manpower and time are expended, under the present situation of talents reserve deficiency and knowledge fusion shortcoming, in order to reduce big data application
Development difficulty, the big data business modelling system for building a set of data-oriented analyst and business expert seems and very must
Will.
For big data platform mainly based on Hadoop platform, Hadoop, which is one, realizes MapReduce patterns at present
The software frame of distributed treatment can be carried out to mass data, is handled in a manner of one kind is reliable, efficient, telescopic
's.As a kind of workflow schedule engine Oozie of Hadoop platform, it is mainly used in running map-reduce task workflows.
The content of the invention
In order to solve to be led in the prior art because software development needs special technical staff to learn a large amount of relevant knowledges
Cause to waste learning time and cost, the task for needing to perform repeatedly or regularly, if skill can be caused without encapsulating execution
A large amount of mechanically duplications of labour of art personnel while business personnel and Data Analyst can not can be visually seen back-end data processing
Flow, the problem of big data business handles the correctness of logic can not be distinguished.It is an object of the invention to provide one kind structure to be based on
The method that the page pulls the big data business model of technology, big data business can be changed into the visual work of component nodes
Stream is set, and simplifies the development of big data developer, allows business personnel and Data Analyst participates in the work of big data business
Make stream automatic configuration.
In order to solve the above technical problems, the present invention uses following technical proposals：
A kind of method for building the big data business model that technology is pulled based on the page, it is characterised in that：This method passes through the page
Dragging technology realizes the visual setting of each node parameter in business model, and the relation established between node, forms work
Flow model, it is specific as follows：
1）Establish the executor of big data platform user, i.e. business model；
2）Disposition data source；
3）Model is established, complete business model is formed by pulling each Service Component；
4）Model trial operation, trial operation operation is carried out to model, if trial operation success, into 5）, otherwise model is repaiied
Change；
5）Issuance model, model state are to call；
6）Module scheduling is set, and timing performs business model；
7）Model running, the operation that the time point set according to scheduling performs model operate.
Further, in the step 1）In, described establishes Hadoop user, including based on Hadoop platform, establishes
Hadoop user, chosen when being established for model.
In the step 2）In, the configuration of the data source, including the data source such as database, HDFS texts, HIVE interfaces.
In the step 3）In, the foundation of the model, including the setting of input data module, data processing module are set
Put, the setting of data statistics module, the setting of output data module, Hadoop user, the associated configuration of data output paths.
In the step 4）In, described model trial operation, including trial operation behaviour is carried out to constructed Visualization Model
Make, perform each node of flow in order, it is ensured that workflow is unimpeded.
In the step 5）In, described Issuance model, including the state of model is configured, after model issue, shape
State is starting state, subsequently can directly be dispatched and perform by module scheduling setup module.
In the step 6）In, described module scheduling set, including Work flow model run at the beginning of between, at the end of
Between and interval time configuration；
In the step 7）In, described model running, the point between dispatching at the beginning of setting, Work flow model is performed
Operation, data processing node is broken into automatically executable jar bags and is published to Hadoop platform, given birth to automatically according to node sequence
The configuration file needed into workflow engine Oozie, is parsed by Oozie and performed.
The present invention pulls technology based on the page, pilot process is defined as into node, dependence is defined as line, utilizes figure
Change Row control means, " point-line " is connected, complete business mould will be formed after one or more data handling component series connection
Type；Model breaks into data processing node executable jar bags automatically, and is published to big data platform simultaneously, while according to section
Dot sequency automatically generates the configuration file of workflow engine Oozie needs；Model system is performed by collocation task scheduling, timing
Business model.
The visual workflow that the present invention can change into big data business component nodes is set, and simplifies big data exploitation
The development of personnel, business personnel and the workflow automation of Data Analyst participation big data business is allowed to configure.The present invention
The visual design function is provided, establishes the business model of correlation, can at least save the work of 50% big data business modelling
Amount.
Brief description of the drawings
Fig. 1 is the flow chart for building the big data business model based on page dragging technology.
Fig. 2 is the flow chart that Visualization Model is established in the present invention.
Embodiment
Below in conjunction with the accompanying drawings to the embodiment of hadoop distributed environments HDFS data service models in the present invention
Further illustrate.
As shown in figure 1, a kind of method for building the big data business model that technology is pulled based on the page, this method pass through page
The Orly technology of dragging realizes the visual setting of each node parameter in business model, and the relation established between node, forms work
Make flow model, it is specific as follows：
1）Establish the executor of big data platform user, i.e. business model；User is established based on hadoop platforms, at the same time set
The outgoing route and storage folder naming rule of fixed number evidence；It is described to establish Hadoop user, including based on Hadoop platform,
Hadoop user is established, is chosen when being established for model.
2）Disposition data source；Configuration including hadoop user names, HDFS file paths, and the connection in test data source
Property, it is ensured that state is to connect；The configuration of the data source, including the data source such as database, HDFS texts, HIVE interfaces.
3）Model is established, complete business model is formed by pulling each Service Component；Then Issuance model so that structure
Work flow model can call；The foundation of the model, including the setting of input data module, the setting of data processing module,
The setting of data statistics module, the setting of output data module, Hadoop user, the associated configuration of data output paths.
4）Model trial operation, trial operation operation is carried out to model, if trial operation success, into 5）, otherwise model is entered
Row modification；Described model trial operation, including trial operation operation is carried out to constructed Visualization Model, flow is performed in order
Each node, it is ensured that workflow is unimpeded.
5）Issuance model, model state are to call；Described Issuance model, including the state of model is configured,
After model issue, state is starting state, subsequently can directly be dispatched and perform by module scheduling setup module.
6）Module scheduling is set, and timing performs business model；Configuration work flow model run at the beginning of between, the end time
And interval time；
7）Model running, the operation that the time point set according to scheduling performs model operate.
As shown in Fig. 2 the flow chart that Visualization Model is established：
（1）Data input, the selection of HDFS data sources, the selection of hadoop user, data output paths, HDFS input datas
Set.
（2）Data processing, data filtering, data filling, data correlation, data fractionation or data can be selected to merge,
Related configuration is carried out for corresponding operation simultaneously, such as data filtering, filter condition etc. is set.
（3）Data statistics, specifically include algorithms selection（Summation, average value, counting, maximum, minimum value etc.）, operation row
Selection（Select calculative row）, and the selection of data output
（4）Data output, i.e. HDFS export, disposition data source, separator and data out field.
Claims (8)
- A kind of 1. method for building the big data business model that technology is pulled based on the page, it is characterised in that：This method passes through page The Orly technology of dragging realizes the visual setting of each node parameter in business model, and the relation established between node, forms work Make flow model, it is specific as follows：1）Establish the executor of big data platform user, i.e. business model；2）Disposition data source；3）Model is established, complete business model is formed by pulling each Service Component；4）Model trial operation, trial operation operation is carried out to model, if trial operation success, into 5）, otherwise model is repaiied Change；5）Issuance model, model state are to call；6）Module scheduling is set, and timing performs business model；7）Model running, the operation that the time point set according to scheduling performs model operate.
- 2. the method that structure according to claim 1 pulls the big data business model of technology based on the page, its feature exist In：In the step 1）In, it is described to establish big data platform user, including based on Hadoop platform, Hadoop user is established, Chosen when being established for model.
- 3. the method that structure according to claim 1 pulls the big data business model of technology based on the page, its feature exist In：In the step 2）In, the data source includes database, HDFS texts, HIVE interfaces.
- 4. the method that structure according to claim 1 pulls the big data business model of technology based on the page, its feature exist In：In the step 3）In, the foundation of the model, including the setting of input data module, the setting of data processing module, number The setting of module, the setting of output data module, Hadoop user, the associated configuration of data output paths according to statistics.
- 5. the method that structure according to claim 1 pulls the big data business model of technology based on the page, its feature exist In：In the step 4）In, described model trial operation, including trial operation operation is carried out to constructed Visualization Model, press Order performs each node of flow, it is ensured that workflow is unimpeded.
- 6. the method that structure according to claim 1 pulls the big data business model of technology based on the page, its feature exist In：In the step 5）In, described Issuance model, including the state of model is configured, after model issue, state is to open Dynamic state, it subsequently can directly be dispatched and perform by module scheduling setup module.
- 7. the method that structure according to claim 1 pulls the big data business model of technology based on the page, its feature exist In：In the step 6）In, described module scheduling set, including Work flow model run at the beginning of between, the end time and The configuration of interval time.
- 8. the method that structure according to claim 1 pulls the big data business model of technology based on the page, its feature exist In：In the step 7）In, described model running, the point between dispatching at the beginning of setting, Work flow model is performed Operation, data processing node is broken into automatically executable jar bags and is published to Hadoop platform, given birth to automatically according to node sequence The configuration file needed into workflow engine Oozie, is parsed by Oozie and performed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710788729.3A CN107526832A (en) | 2017-09-05 | 2017-09-05 | A kind of method for building the big data business model that technology is pulled based on the page |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710788729.3A CN107526832A (en) | 2017-09-05 | 2017-09-05 | A kind of method for building the big data business model that technology is pulled based on the page |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107526832A true CN107526832A (en) | 2017-12-29 |
Family
ID=60683413
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710788729.3A Pending CN107526832A (en) | 2017-09-05 | 2017-09-05 | A kind of method for building the big data business model that technology is pulled based on the page |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107526832A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108446853A (en) * | 2018-03-23 | 2018-08-24 | 肖娟 | A kind of Business Process Analysis system and method |
| CN108460077A (en) * | 2018-01-03 | 2018-08-28 | 北京字节跳动网络技术有限公司 | A kind of index analysis method, system and computer readable storage medium |
| CN109558392A (en) * | 2018-11-20 | 2019-04-02 | 南京数睿数据科技有限公司 | A kind of mass data moving apparatus that cross-platform multi engine is supported |
| CN109918465A (en) * | 2019-03-01 | 2019-06-21 | 北京超图软件股份有限公司 | A kind of Geoprocessing method and device |
| CN110187875A (en) * | 2019-05-28 | 2019-08-30 | 深圳市智慧郎数码科技有限公司 | A kind of component visual melts forwarding method |
| CN110427398A (en) * | 2018-04-28 | 2019-11-08 | 北京资采信息技术有限公司 | A kind of model management tool based on data mining and analysis |
| CN110727729A (en) * | 2018-06-29 | 2020-01-24 | 贵州白山云科技股份有限公司 | Method and device for realizing intelligent operation |
| CN111309315A (en) * | 2018-12-12 | 2020-06-19 | 中国科学院沈阳自动化研究所 | Automatic configuration method based on industrial Internet of things data and business modeling |
| CN112069243A (en) * | 2020-08-18 | 2020-12-11 | 福建博思软件股份有限公司 | Method for generating index analysis model based on visual page and storage device |
| CN112417704A (en) * | 2020-12-04 | 2021-02-26 | 重庆忽米网络科技有限公司 | Configurable dragging type digital industrial model construction method |
| CN112486475A (en) * | 2020-12-03 | 2021-03-12 | 成都大数据产业技术研究院有限公司 | Visual business modeling and model management system based on big data platform |
| CN114896003A (en) * | 2022-04-13 | 2022-08-12 | 青岛海尔科技有限公司 | Page configuration method and device, storage medium and electronic device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104572929A (en) * | 2014-12-26 | 2015-04-29 | 深圳市科漫达智能管理科技有限公司 | Data mining method and device |
| CN105550268A (en) * | 2015-12-10 | 2016-05-04 | 江苏曙光信息技术有限公司 | Big data process modeling analysis engine |
| CN106951534A (en) * | 2017-03-22 | 2017-07-14 | 北京数猎天下科技有限公司 | A kind of big data visualizes the graphic processing method and device of data correlation relation |
| CN107103050A (en) * | 2017-03-31 | 2017-08-29 | 海通安恒（大连）大数据科技有限公司 | A kind of big data Modeling Platform and method |
-
2017
- 2017-09-05 CN CN201710788729.3A patent/CN107526832A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104572929A (en) * | 2014-12-26 | 2015-04-29 | 深圳市科漫达智能管理科技有限公司 | Data mining method and device |
| CN105550268A (en) * | 2015-12-10 | 2016-05-04 | 江苏曙光信息技术有限公司 | Big data process modeling analysis engine |
| CN106951534A (en) * | 2017-03-22 | 2017-07-14 | 北京数猎天下科技有限公司 | A kind of big data visualizes the graphic processing method and device of data correlation relation |
| CN107103050A (en) * | 2017-03-31 | 2017-08-29 | 海通安恒（大连）大数据科技有限公司 | A kind of big data Modeling Platform and method |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108460077A (en) * | 2018-01-03 | 2018-08-28 | 北京字节跳动网络技术有限公司 | A kind of index analysis method, system and computer readable storage medium |
| CN108446853A (en) * | 2018-03-23 | 2018-08-24 | 肖娟 | A kind of Business Process Analysis system and method |
| CN110427398A (en) * | 2018-04-28 | 2019-11-08 | 北京资采信息技术有限公司 | A kind of model management tool based on data mining and analysis |
| CN110727729A (en) * | 2018-06-29 | 2020-01-24 | 贵州白山云科技股份有限公司 | Method and device for realizing intelligent operation |
| CN109558392A (en) * | 2018-11-20 | 2019-04-02 | 南京数睿数据科技有限公司 | A kind of mass data moving apparatus that cross-platform multi engine is supported |
| CN111309315A (en) * | 2018-12-12 | 2020-06-19 | 中国科学院沈阳自动化研究所 | Automatic configuration method based on industrial Internet of things data and business modeling |
| CN111309315B (en) * | 2018-12-12 | 2024-03-29 | 中国科学院沈阳自动化研究所 | Automatic configuration method based on industrial Internet of things data and business modeling |
| CN109918465A (en) * | 2019-03-01 | 2019-06-21 | 北京超图软件股份有限公司 | A kind of Geoprocessing method and device |
| CN110187875A (en) * | 2019-05-28 | 2019-08-30 | 深圳市智慧郎数码科技有限公司 | A kind of component visual melts forwarding method |
| CN112069243A (en) * | 2020-08-18 | 2020-12-11 | 福建博思软件股份有限公司 | Method for generating index analysis model based on visual page and storage device |
| CN112486475A (en) * | 2020-12-03 | 2021-03-12 | 成都大数据产业技术研究院有限公司 | Visual business modeling and model management system based on big data platform |
| CN112417704A (en) * | 2020-12-04 | 2021-02-26 | 重庆忽米网络科技有限公司 | Configurable dragging type digital industrial model construction method |
| CN112417704B (en) * | 2020-12-04 | 2022-07-08 | 重庆忽米网络科技有限公司 | Configurable dragging type digital industrial model construction method |
| CN114896003A (en) * | 2022-04-13 | 2022-08-12 | 青岛海尔科技有限公司 | Page configuration method and device, storage medium and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107526832A (en) | A kind of method for building the big data business model that technology is pulled based on the page | |
| CN112394922B (en) | Decision configuration method, business decision method and decision engine system | |
| CN104407977B (en) | Based on the automatization uniting and adjustment testing method of the task system stage by stage of model inspection | |
| CN103886203B (en) | Automatic modeling system and method based on index prediction | |
| CN111181773B (en) | Delay prediction method for multi-component application of heterogeneous border cloud collaborative intelligent system | |
| CN105278960A (en) | Process automation method and system in remote sensing application | |
| CN103530757A (en) | Network based multi-mode intelligent order following management method and intelligent management system | |
| Hasegan et al. | Predicting performance–a dynamic capability view | |
| CN106156115A (en) | A kind of resource regulating method and device | |
| CN111079997A (en) | Modeling and collaborative optimization method | |
| CN109784758A (en) | Construction quality supervision early warning system and method based on BIM model | |
| CN113656021A (en) | Oil-gas big data analysis system and method for business scene | |
| CN108681598A (en) | Task runs method, system, computer equipment and storage medium again automatically | |
| CN111930956A (en) | Integrated system for recommending and stream-driving multiple innovation methods by adopting knowledge graph | |
| CN106127365A (en) | Quantitative remote sensing On-line Product interactive mode autonomous production method | |
| CN103164774A (en) | Automobile complete vehicle development system based on workflow | |
| CN110442766A (en) | Webpage data acquiring method, device, equipment and storage medium | |
| CN112948353A (en) | Data analysis method, system and storage medium applied to DAstudio | |
| CN104123317A (en) | Service organization assessing and analyzing method based on knowledge base | |
| CN110766163B (en) | System for implementing machine learning process | |
| CN104484230B (en) | More satellite data central task stream dispatching algorithms based on nearly data calculating principle | |
| CN103824162B (en) | Reliability and performance integrated flexible workflow implementing method based on instruction chain | |
| CN105787141A (en) | Collaborative simulation method and system for complex weapon system operation process | |
| CN104123585A (en) | Service organization optimization analysis method based on service simulation | |
| CN104123584A (en) | Organization optimization method based on information system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20171229 |