CN106777142A - Service layer's system and method based on mobile Internet mass data - Google Patents
Service layer's system and method based on mobile Internet mass data Download PDFInfo
- Publication number
- CN106777142A CN106777142A CN201611177947.5A CN201611177947A CN106777142A CN 106777142 A CN106777142 A CN 106777142A CN 201611177947 A CN201611177947 A CN 201611177947A CN 106777142 A CN106777142 A CN 106777142A
- Authority
- CN
- China
- Prior art keywords
- data
- operation system
- service
- interface
- center
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/20—Software design
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Transfer Between Computers (AREA)
- Computer And Data Communications (AREA)
Abstract
The invention discloses a kind of service layer's system and method based on mobile Internet mass data, belong to big data field.The system includes the ETL being sequentially connected（100）, data center（200）, service layer's platform（300）And operation system（400）；Described data center（200）It is mass data storage and the core component for processing, including data storage center（210）Center is calculated with data（220）；Described service layer's platform（300）It is data center（200）The window of service, including bottom data access service module are externally provided（310）, business interface module（320）And control module（330）；This method is：1. operation system registration；2. operation system inquiry.The present invention can carry out unified query interface to the mass data of all mobile Internets, be that each application system unified query service providing technique is supported.
Description
Technical field
The invention belongs to big data field, more particularly to a kind of service layer system based on mobile Internet mass data and
Its method.
Background technology
In recent years, with the continuous lifting and the continuous decline of fees and price of mobile Internet networking speed, mobile interchange
How network data stores these data, how required for rapid extraction to user in these data into volatile growth
Information has turned into a heat subject.The data source that mobile Internet possesses is rich and varied, including various cell-phone customer terminals are produced
Structuring and non-structured some data such as business datum, webpage, voice data, video file and picture.For difference
The storage of species data also occurs in that all kinds of SQL and NoSQL databases.
In internet industry, business marketing personnel if necessary to finding useful information in mass data, it is necessary to
Run, in each business department, to inquire about each set system, leverage operating efficiency.
The content of the invention
It is an object of the invention to overcome shortcoming and defect present in prior art, there is provided one kind is based on mobile Internet
Service layer's system and method for mass data, realize following functions：1. various heterogeneous databases are unified（SQL/NoSQL）Visit
Ask mode；2. tables of data is self-defined（User Defined table view）.
Realizing the technical scheme of the object of the invention is：
First, the service system based on mobile Internet mass data（Abbreviation system）
The system includes the ETL, data center, service layer's platform and the operation system that are sequentially connected.
2nd, the method for servicing based on mobile Internet mass data（Abbreviation method）
This method comprises the following steps：
1. operation system registration
Operation system access service layer platform 300 in service layer's platform 300, it is necessary to first register, unique number of generation system.
2. operation system inquiry.
The present invention has following advantages and good effect：
1. general service layer's access platform is provided, according to http post and get the method call interface methods of standard, is carried
Json formatted datas for standard are returned；
2. processing speed is fast, and with fault-tolerance；
3. by integrating database interface and the services such as Hadoop distributed file systems, Solr text retrieval systems, ibatis
The technologies such as layer fusion of platforms, for the modulation conformity and unified operation interface between other big data technologies provide reference.
In a word, the present invention can carry out unified query interface to the mass data of all mobile Internets, be each application system
System unified query service providing technique is supported.
Brief description of the drawings
Fig. 1 is the block diagram of the system；
Fig. 2 is the block diagram of data center；
Fig. 3 is the block diagram of service centre；
Fig. 4 is system registry flow chart；
Fig. 5 is system queries request flow chart.
Wherein：
100—ETL；
200-data center,
210-data storage center,
201—Linux（Local file system）, 202-HBase（High-volume database）,
203—Solr（Full-text database）, 204-MySQL（Service Database）,
205—Hadoop；
220-data calculate center,
211—SQL（Enquiry module）, 212-Hive（Statistical analysis module）,
213—Mahout（Data-mining module）, 214-Spark（Data processing module）；
300-service layer platform,
310-bottom data access service module,
320-business interface module,
321-general service interface, 322-original service interface, 323-basic business interface；
330-control module,
331-daily record, 332-authorize；
400-operation system,
410-the 1 operation system,
420-the 2 operation system ... ...
4N0-the N operation systems, wherein N≤100.
English to Chinese
1、Storm：Be one freely increase income, fault-tolerant real time computation system distributed, high.Storm makes continual flowmeter
Calculation becomes easy, compensate for the unappeasable requirement of real time of Hadoop batch processings institute；
2、Kafka：It is a kind of distributed post subscription message system of high-throughput；
3、HDFS：The distributed file system of Hadoop；
4、Flume：A High Availabitity that Cloudera is provided, it is highly reliable, the collection of distributed massive logs, polymerization and
The system of transmission, Flume supports to customize Various types of data sender in log system, for collecting data；Meanwhile, Flume is carried
For data are carried out with simple process, and write various data receivings（It is customizable）Ability.
5、Linux：A kind of operating system；
6、Hadoop：A kind of software frame that distributed treatment can be carried out to mass data；
7、MySQL：MySQL is a Relational DBMS；
8、HBase：It is a kind of distributed, towards the Hadoop databases of row；
9、Solr：A kind of search server based on Lucene；
10、Hive：It is a kind of Tool for Data Warehouse based on Hadoop, the data file of structuring can be mapped as a number
According to storehouse table, and simple sql query functions are provided, sql sentences can be converted to MapReduce tasks and be run；
11、Spark：A kind of quick big data distributed process engines；
12、Mahout：A kind of open source projects under ASF, there is provided the reality of some expansible machine learning field classic algorithms
It is existing, it is intended to help developer to create intelligent application more conveniently；
13、Zookeeper：It is a kind of distributed, distributed application program coordination service of open source code, it is one to divide
Cloth application provides the software of Consistency service.
Specific embodiment
Described in detail below in conjunction with drawings and Examples：
First, system
1st, it is overall
Such as Fig. 1, the system includes the ETL100, data center 200, service layer's platform 300 and the operation system 400 that are sequentially connected.
2nd, functional part
1）ETL100
ETL100 refers to the general name of switch technology after all kinds of mobile Internet data sources are extracted and cleaned.
2）Data center 200
Such as Fig. 2, data center 200 is mass data storage and the core component for processing, including data storage center 210 and data
Calculating center 220；
Data storage center 210 includes Linux-211, HBase-212, Solr-213, MySQL-214 and Hadoop-215；
Data calculate center 220 includes SQL-221, Hive-222, Mahout-223 and Spark-224.
3）Service layer's platform 300
Service layer's platform 300 be data center 200 externally provide service window, including bottom data access service module 310,
Business interface module 320 and control module 330；
Bottom data access service module 310 obtains Various types of data according to the general API of data center 200；
Business interface module 320 includes general service interface 321, original service interface 322 and basic business interface 323；It is general
Business interface 321 and original service interface 322 are interacted with basic business interface 323 respectively.
Original service interface 322, basic business interface 323 is provided and general service interface 321 supplies upper layer application system
400 call access.
Control module 330 is used for Access Control, and the only operation system by authorizing 332 could provide the clothes of access interface
Business, and the action record daily record 331 to operating；Service can not be otherwise provided.
4）Operation system 400
Such as Fig. 1, operation system 400 include the 1st operation system 410, the 2nd operation system 420 ... N operation system 4N0, N are
Natural number, N≤100.
2nd, working mechanism
The system is received after the raw data file of mobile Internet, and ETL100 is carried according to specified standard logarithmic evidence
Take, clean, being deposited into data center 200；Data center 200 provides various data algorithm models and data mining model, right
Data carry out depth excavation and statistical analysis；Service layer's platform 300 is that upper-layer service system 400 provides infrastructure service, upper strata
The general service interface 321, original service that application system 400 calls service layer's platform 300 by HTTP GET/POST on demand connects
Mouth 322, basic business interface 323 obtains the Various types of data of data center 200.
2nd, method
* step is 1.：Operation system is registered
Such as Fig. 4, operation system register flow path is as follows：
A, reception operation system registration request -10；
B, the incoming application name of operation system is judged with the presence or absence of -11, after being then request failure -13
Step D is jumped to, otherwise into step C；
Unique number of C, distribution system gives operation system -12
D, end -14.
* step is 2.：Operation system is inquired about
Such as Fig. 5, the workflow of operation system inquiry is as follows：
A, receive operation system request -20；
B, judge whether operation system registers -21, be then to enter step c, otherwise jump to step j after request failure -29；
C, judge operation system parameter whether legal -22, be then to enter step d, otherwise i.e. after request failure -29 after jump to step
Rapid j；
D, judge AppKey, session, method version number and request method whether legal -23, be then to enter step e, otherwise i.e. please
Step j is jumped to after asking failure -29；
E, judge service parameter checking process whether legal -24, be then to enter step f, otherwise jumped to after request failure -29
Step j；
F, judge verification business whether limited accass, session whether exceed the frequency, upload file size whether exceed limitation -25,
It is then to enter step g, otherwise jumps to step j after request failure -29；
G, requested service layer interface -26；
H, request DAO layer interfaces -27；
I, ask successfully -28；
J, end -30.
Claims (2)
1. a kind of service layer system based on mobile Internet mass data, it is characterised in that：
Including the ETL being sequentially connected（100）, data center（200）, service layer's platform（300）And operation system（400）；
Described ETL（100）It refer to the general name of switch technology after all kinds of mobile Internet data sources are extracted and cleaned；
Described data center（200）It is mass data storage and the core component for processing, including data storage center（210）With
Data calculate center（220）；
Data storage center（210）Including Linux（211）、HBase（212）、Solr（213）、MySQL（214）And Hadoop
（215）；
Data calculate center（220）Including SQL（221）、Hive（222）、Mahout（223）And Spark（224）；
Described service layer's platform（300）It is data center（200）The window of service is externally provided, including bottom data accesses clothes
Business module（310）, business interface module（320）And control module（330）；
Bottom data access service module（310）According to data center（200）General API obtains Various types of data；
Business interface module（320）Including general service interface（321）, original service interface（322）With basic business interface
（323）；General service interface（321）With original service interface（322）Respectively with basic business interface（323）Interaction；
Original service interface is provided（322）, basic business interface（323）And general service interface（321）For upper layer application system
System（400）Call access；
Control module（330）For Access Control, only by authorizing（332）Operation system the clothes of access interface could be provided
Business, and the action record daily record to operating（331）；Service can not be otherwise provided；
Described operation system（400）Including the 1st operation system（410）, the 2nd operation system（420）... N operation systems
（4N0）, N is natural number, N≤100.
2. the method for servicing of system described in claim 1 is based on, it is characterised in that comprised the following steps：
1. operation system registration
Operation system access service layer platform（300）, it is necessary to first in service layer's platform（300）Middle registration, generates the unique of system
Number；
2. operation system inquiry；
The flow of described operation system registration is as follows：
A, reception operation system registration request（10）；
B, judge that the incoming application name of operation system whether there is（11）, it is then request failure（13）Afterwards
Step D is jumped to, otherwise into step C；
Unique number of C, distribution system gives operation system（12）；
D, end（14）；
The workflow of the operation system inquiry is as follows：
A, receive operation system request（20）；
B, judge whether operation system is registered（21）, it is then to enter step c, otherwise i.e. request failure（29）After jump to step j；
C, judge whether operation system parameter is legal（22）, it is then to enter step d, otherwise i.e. request failure（29）After jump to step
Rapid j；
D, judge whether AppKey, session, method version number and request method are legal（23）, it is then to enter step e, be otherwise
Request failure（29）After jump to step J；
E, judge whether service parameter checking process is legal（24）, it is then to enter step f, otherwise i.e. request failure（29）After redirect
To step j；
F, judge verification business whether limited accass, session whether exceed the frequency, upload file size whether exceed limitation（25）,
It is then to enter step g, otherwise i.e. request failure（29）After jump to step j；
G, requested service layer interface（26）；
H, request DAO layer interfaces（27）；
I, ask successfully（28）；
J, end（30）.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611177947.5A CN106777142A (en) | 2016-12-19 | 2016-12-19 | Service layer's system and method based on mobile Internet mass data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611177947.5A CN106777142A (en) | 2016-12-19 | 2016-12-19 | Service layer's system and method based on mobile Internet mass data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN106777142A true CN106777142A (en) | 2017-05-31 |
Family
ID=58890320
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611177947.5A Withdrawn CN106777142A (en) | 2016-12-19 | 2016-12-19 | Service layer's system and method based on mobile Internet mass data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106777142A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107193573A (en) * | 2017-08-02 | 2017-09-22 | 广州云徙科技有限公司 | Intranet open platform based on data-driven |
| CN108052574A (en) * | 2017-12-08 | 2018-05-18 | 南京中新赛克科技有限责任公司 | Slave ftp server based on Kafka technologies imports the ETL system and implementation method of mass data |
| CN108959952A (en) * | 2017-05-23 | 2018-12-07 | ***通信集团重庆有限公司 | data platform authority control method, device and equipment |
| CN109086296A (en) * | 2018-06-13 | 2018-12-25 | 广州市信景技术有限公司 | A kind of e-commerce system based on browser and server structure |
| CN109344186A (en) * | 2018-08-23 | 2019-02-15 | 成都四方伟业软件股份有限公司 | A kind of BI system Various database is across the inter-library emerging system in source and fusion method |
| CN109446219A (en) * | 2018-10-10 | 2019-03-08 | 新华三大数据技术有限公司 | Right management method and device |
| CN109559138A (en) * | 2017-09-25 | 2019-04-02 | 北京京东尚科信息技术有限公司 | Dodge purchase activity sales volume prediction technique and device, storage medium, electronic equipment |
| CN111625218A (en) * | 2020-05-14 | 2020-09-04 | 中电工业互联网有限公司 | Big data processing method and system for custom library development |
| CN115757552A (en) * | 2022-11-29 | 2023-03-07 | 广发银行股份有限公司 | Bank historical data management system based on distributed micro-service |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104008135A (en) * | 2014-05-07 | 2014-08-27 | 南京邮电大学 | Multi-source heterogeneous database fusion system and data query method thereof |
| CN105117393A (en) * | 2014-11-04 | 2015-12-02 | 合肥轩明信息科技有限公司 | Big data based application mode in industry application |
| CN105787064A (en) * | 2016-03-01 | 2016-07-20 | 广州铭诚计算机科技有限公司 | Mining platform establishment method based on big data |
-
2016
- 2016-12-19 CN CN201611177947.5A patent/CN106777142A/en not_active Withdrawn
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104008135A (en) * | 2014-05-07 | 2014-08-27 | 南京邮电大学 | Multi-source heterogeneous database fusion system and data query method thereof |
| CN105117393A (en) * | 2014-11-04 | 2015-12-02 | 合肥轩明信息科技有限公司 | Big data based application mode in industry application |
| CN105787064A (en) * | 2016-03-01 | 2016-07-20 | 广州铭诚计算机科技有限公司 | Mining platform establishment method based on big data |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108959952A (en) * | 2017-05-23 | 2018-12-07 | ***通信集团重庆有限公司 | data platform authority control method, device and equipment |
| CN108959952B (en) * | 2017-05-23 | 2020-10-30 | ***通信集团重庆有限公司 | Data platform authority control method, device and equipment |
| CN107193573A (en) * | 2017-08-02 | 2017-09-22 | 广州云徙科技有限公司 | Intranet open platform based on data-driven |
| CN109559138A (en) * | 2017-09-25 | 2019-04-02 | 北京京东尚科信息技术有限公司 | Dodge purchase activity sales volume prediction technique and device, storage medium, electronic equipment |
| CN108052574A (en) * | 2017-12-08 | 2018-05-18 | 南京中新赛克科技有限责任公司 | Slave ftp server based on Kafka technologies imports the ETL system and implementation method of mass data |
| CN109086296A (en) * | 2018-06-13 | 2018-12-25 | 广州市信景技术有限公司 | A kind of e-commerce system based on browser and server structure |
| CN109086296B (en) * | 2018-06-13 | 2021-12-07 | 广州市信景技术有限公司 | Electronic commerce system based on browser and server structure |
| CN109344186A (en) * | 2018-08-23 | 2019-02-15 | 成都四方伟业软件股份有限公司 | A kind of BI system Various database is across the inter-library emerging system in source and fusion method |
| CN109446219A (en) * | 2018-10-10 | 2019-03-08 | 新华三大数据技术有限公司 | Right management method and device |
| CN109446219B (en) * | 2018-10-10 | 2021-06-29 | 新华三大数据技术有限公司 | Authority management method and device |
| CN111625218A (en) * | 2020-05-14 | 2020-09-04 | 中电工业互联网有限公司 | Big data processing method and system for custom library development |
| CN111625218B (en) * | 2020-05-14 | 2024-01-09 | 中电工业互联网有限公司 | Big data processing method and system for custom library development |
| CN115757552A (en) * | 2022-11-29 | 2023-03-07 | 广发银行股份有限公司 | Bank historical data management system based on distributed micro-service |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106777142A (en) | Service layer's system and method based on mobile Internet mass data | |
| JP7170779B2 (en) | Methods and systems for automatic intent mining, classification, and placement | |
| JP7273045B2 (en) | Dimensional Context Propagation Techniques for Optimizing SQL Query Plans | |
| US11941016B2 (en) | Using specified performance attributes to configure machine learning pipepline stages for an ETL job | |
| CA3140340C (en) | Systems and methods for chatbot generation | |
| US11093501B2 (en) | Searching in a database | |
| CN103620601B (en) | Joining tables in a mapreduce procedure | |
| US11899681B2 (en) | Knowledge graph building method, electronic apparatus and non-transitory computer readable storage medium | |
| US10970284B2 (en) | Dynamic self-reconfiguration of nodes in a processing pipeline | |
| CN106776780B (en) | Data exchange and sharing method and system in cloud environment | |
| CN109997126A (en) | Event-driven is extracted, transformation, loads (ETL) processing | |
| CN109684330A (en) | User's portrait base construction method, device, computer equipment and storage medium | |
| US10812551B1 (en) | Dynamic detection of data correlations based on realtime data | |
| CN104298771A (en) | Massive web log data query and analysis method | |
| WO2017166944A1 (en) | Method and device for providing service access | |
| US9594853B2 (en) | Combined deterministic and probabilistic matching for data management | |
| JP7084691B2 (en) | How to process and present real-time social data on a map | |
| US20180165367A1 (en) | Hyperdata generation in the cloud | |
| CN113887941B (en) | Business process generation method, device, electronic equipment and medium | |
| US11640416B2 (en) | Computer-based systems configured for efficient entity resolution for database merging and reconciliation | |
| CN105786941B (en) | Information mining method and device | |
| US10033737B2 (en) | System and method for cross-cloud identity matching | |
| CN113051389B (en) | Knowledge pushing method and device | |
| CN112506887B (en) | Vehicle terminal CAN bus data processing method and device | |
| US9602575B2 (en) | Monitoring social media for specific issues |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20170531 |