CN114003646A - High-concurrency real-time multi-attribute aggregated map cluster service system - Google Patents
High-concurrency real-time multi-attribute aggregated map cluster service system Download PDFInfo
- Publication number
- CN114003646A CN114003646A CN202111638401.6A CN202111638401A CN114003646A CN 114003646 A CN114003646 A CN 114003646A CN 202111638401 A CN202111638401 A CN 202111638401A CN 114003646 A CN114003646 A CN 114003646A
- Authority
- CN
- China
- Prior art keywords
- service
- module
- service module
- map cluster
- attribute
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004220 aggregation Methods 0.000 claims abstract description 25
- 230000002776 aggregation Effects 0.000 claims abstract description 25
- 238000012423 maintenance Methods 0.000 claims abstract description 12
- 230000033772 system development Effects 0.000 claims abstract description 12
- 238000000034 method Methods 0.000 abstract description 3
- 230000002688 persistence Effects 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 230000003203 everyday effect Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2455—Query execution
- G06F16/24553—Query execution of query operations
- G06F16/24554—Unary operations; Data partitioning operations
- G06F16/24556—Aggregation; Duplicate elimination
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/70—Software maintenance or management
- G06F8/71—Version control; Configuration management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2216/00—Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
- G06F2216/03—Data mining
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Remote Sensing (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computer Security & Cryptography (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a high-concurrency real-time multi-attribute aggregated map cluster service system, which comprises a user front-end service module, a Hadoop-based aggregated map cluster service module, a geographic metadata service module, a system development operation and maintenance module and a code version library module, wherein the user front-end service module consists of a first party client, a direct service access module, an open platform access module, a cooperative enterprise service module and an open platform authority authentication module, and the Hadoop-based aggregated map cluster service module consists of a user rear-end micro-service module, an attribute aggregation and reasoning module and a Hadoop basic service module, and has the advantages that: the advantageous effects of the present invention are specifically explained. The method solves the time-sensitive geographic attribute task, can excavate hidden semantic expression of the geographic attribute, establishes the relation between space and time and semantic high latitude between geographic entities, and provides the bottom support of AI for the geographic related front-end application.
Description
Technical Field
The invention relates to the technical field of geographic information data processing, in particular to a high-concurrency real-time multi-attribute aggregated map cluster service system.
Background
With the development of the internet and big data, huge amounts of geographic data are generated every day. Multiple attributes contained in the mining system are mined, and a plurality of services can be driven by combining geographic information, including: the map navigation service, the peripheral place recommendation service, the traffic flow prediction and dispersion service, the urban construction planning service and the like. Limited by the requirements of data size and effectiveness, a system capable of processing mass data in real time is needed to provide underlying driving for the above services. The system provides high-throughput real-time service response by adopting a Hadoop technology, and simultaneously, multiple attributes in the system are mined by combining a self-adaptive attribute clustering reasoning module to provide support for upper-layer application.
Disclosure of Invention
The invention aims to provide a high-concurrency real-time multi-attribute aggregation map cluster service system to solve the problems in the background technology.
In order to achieve the purpose, the invention provides the following technical scheme: a high-concurrency real-time multi-attribute aggregated map cluster service system comprises a user front-end service module, a Hadoop-based aggregated map cluster service module, a geographic metadata service module, a system development operation and maintenance module and a code version library module, the user front-end service module consists of a first party client, a direct service access module, an open platform access module, a cooperative enterprise service module and an open platform authority authentication module, the aggregated map cluster service module based on Hadoop consists of a user back-end micro-service module, an attribute aggregation and reasoning module and a Hadoop basic service module, the geographic metadata service module consists of a basic service module, a message center module, a relational data cluster backup module, a static resource document persistence service module and a service configuration and discovery module, the system development operation and maintenance module is composed of an automation service module and a DevOps module.
As a further scheme of the invention: the Hadoop-based aggregated map cluster service module provides real-time service for the front-end service module through a server, the user back-end micro-service module and the open platform access module respectively comprise Restful API, service layer driving service and a database, and the database consists of NoSQL and RDB.
As a further scheme of the invention: the basic service module is used for providing task scheduling service, log service, workflow service and message pushing service.
As a further scheme of the invention: the service configuration and discovery module provides configuration center service by using Apollo, provides service discovery tasks by using Eureka, the system development and operation and maintenance module provides automation service by using a DevOps mode, and the automation service specifically realizes construction automation by using Travis and deployment automation by using Docker.
As a further scheme of the invention: the code version library module manages code versions and branch iterations specifically by using git services.
As a further scheme of the invention: the user front-end service module directly drives a first party client through a direct service access module, and the first party client is specifically one or a combination of any one of an iOS/Android terminal, a PC terminal, a WeChat public number and a mobile web terminal.
As a further scheme of the invention: the aggregated map cluster service module based on Hadoop can provide services for a third party through the open platform access module.
As a further scheme of the invention: the open platform access module controls the request from the first party client side direct service access module through the api gateway and the security policy at the same time.
Compared with the prior art, the invention has the beneficial effects that:
1. hidden layer attributes in real-time geographic text data can be mined;
2. an instant geographic information API response can be provided;
3. providing PaaS support based on artificial intelligence for geographic information service.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a system architecture diagram of the present invention;
fig. 2 is a schematic diagram of the operation of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1-2, in an embodiment of the present invention, a high-concurrency real-time multi-attribute aggregated map cluster service system includes a user front-end service module, a Hadoop-based aggregated map cluster service module, a geographic metadata service module, a system development operation and maintenance module, and a code version library module, where the user front-end service module includes a first-party client, a direct service access module, an open platform access module, a cooperative enterprise service module, and an open platform authority authentication module, the Hadoop-based aggregated map cluster service module includes a user back-end micro service module, an attribute aggregation and inference module, and a Hadoop base service module, the geographic metadata service module includes a base service module, a message center module, a relational data cluster backup module, a static resource document persistence service module, and a service configuration and discovery module, the system development operation and maintenance module is composed of an automatic service module and a DevOps module, the user front-end service module directly drives a first party client through a direct service access module, the first party client is specifically one or a combination of any one of an iOS/Android terminal, a PC terminal, a WeChat public number and a mobile web terminal, the Hadoop-based aggregated map cluster service module can provide services for a third party through an open platform access module, and when the Hadoop-based aggregated map cluster service module is used as an open platform, the open platform access module is required to carry out authentication.
It is worth noting that the Hadoop-based aggregated map cluster service module provides real-time service for the front-end service module through a server, the user back-end micro-service module and the open platform access module both comprise Restful API, service layer driving service and a database, and the database is composed of NoSQL and RDB.
The basic service module is used for providing task scheduling service, log service, workflow service and message pushing service, the attribute aggregation and reasoning module inputs attribute representation and multiple attribute aggregation in the module, aggregation attributes and aggregation semantics are output through a self-adaptive aggregation model, and the self-adaptive aggregation model specifically adopts an R tree and neural network method to mine the aggregation characteristics of geographic positions and multi-dimensional attributes.
It is noted that the service configuration and discovery module provides configuration center services by using Apollo, provides service discovery tasks by using Eureka, and the system development and operation and maintenance module provides automation services by using DevOps, wherein the automation services are specifically to realize building automation by using Travis and realize deployment automation by using Docker.
It is noted that the code version library module manages code versions and branch iterations specifically using git services, and the open platform access module controls requests from the first party client direct service access module through api gateway and security policies simultaneously.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned.
Example (b):
and the user front-end service module is accessed to the first party service and directly drives the clients such as the iOS/Android terminal, the PC terminal, the WeChat public account, the mobile web and the like. The module can also serve as an open platform to provide services for third parties. When the service is provided as an open platform, authentication is required to be realized through the open platform OAuth2.0, a first party service module, the open platform module, an optional security policy, an API Gateway for controlling access authentication simultaneously, the OAuth2.0 module, an optional Restful API and a service layer driving service are required, and a database consists of NoSQL and RDB;
the system comprises a Hadoop-based aggregated map cluster service module, a front-end service module, a database and a server, wherein the aggregated map cluster service module provides real-time service for the front-end service module through a service route, and comprises a user rear-end micro-service module, an attribute aggregation and reasoning module and a Hadoop basic service module, the user rear-end micro-service module, an optional Restful API and a business layer driving service, the database consists of NoSQL and RDB, the attribute aggregation and reasoning module inputs attribute representation and multiple attribute aggregation in the module, and outputs aggregation attribute and aggregation semantics through a self-adaptive aggregation model, the self-adaptive aggregation model specifically adopts a method of an R tree and a neural network to mine the aggregation characteristics of geographic positions and multi-dimensional attributes, and the Hadoop basic service module adopts a distributed architecture of MapReduce, uses an HDFS distributed storage system and adopts a YARN resource management system;
the geographic metadata service module comprises a basic service module, a message center module based on a RabbitMQ, a relational data cluster backup module, a static resource document persistence service module based on CDN service, and a service configuration and discovery module, wherein the basic service module provides task scheduling service, log service, workflow service and message pushing service, the service configuration and discovery module provides configuration center service by using Apollo, provides service discovery task by using Eureka, and the system development and operation and maintenance service module drives system development and operation and maintenance by using a DevOps mode and provides automation service. The automation service uses Travis to realize building automation and uses Docker to realize deployment automation.
Furthermore, it should be understood that although the present description refers to embodiments, not every embodiment may contain only a single embodiment, and such description is for clarity only, and those skilled in the art should integrate the description, and the embodiments may be combined as appropriate to form other embodiments understood by those skilled in the art.
Claims (5)
1. The utility model provides a high concurrent real-time multiattribute aggregation map cluster service system, includes user's front end service module, aggregation map cluster service module based on Hadoop, geographical metadata service module, system development operation and maintenance module and code version library module, its characterized in that: the system development operation and maintenance module consists of an automation service module and a DevOps module, the Hadoop-based aggregated map cluster service module provides real-time service for the front-end service module through a server, and the user rear-end micro-service module and the open platform access module respectively comprise a Restful API (application program interface), a business layer driving service and a database, the database consists of NoSQL and RDB, the basic service module is used for providing task scheduling service, log service, workflow service and message pushing service, the service configuration and discovery module provides configuration center service by using Apollo and provides service discovery task by using Eureka, the system development and operation and maintenance module provides automation service by using DevOps mode, the automation service is realized by using Travis to construct automation, and Docker is used to realize deployment automation.
2. The high-concurrency real-time multi-attribute aggregation map cluster service system according to claim 1, wherein: the code version library module manages code versions and branch iterations specifically by using git services.
3. The high-concurrency real-time multi-attribute aggregation map cluster service system according to claim 1, wherein: the user front-end service module directly drives a first party client through a direct service access module, and the first party client is specifically one or a combination of any one of an iOS/Android terminal, a PC terminal, a WeChat public number and a mobile web terminal.
4. The high-concurrency real-time multi-attribute aggregation map cluster service system according to claim 1, wherein: the aggregated map cluster service module based on Hadoop can provide services for a third party through the open platform access module.
5. The high-concurrency real-time multi-attribute aggregation map cluster service system according to claim 1, wherein: the open platform access module controls the request from the first party client side direct service access module through the api gateway and the security policy at the same time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111638401.6A CN114003646A (en) | 2021-12-30 | 2021-12-30 | High-concurrency real-time multi-attribute aggregated map cluster service system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111638401.6A CN114003646A (en) | 2021-12-30 | 2021-12-30 | High-concurrency real-time multi-attribute aggregated map cluster service system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114003646A true CN114003646A (en) | 2022-02-01 |
Family
ID=79932476
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111638401.6A Pending CN114003646A (en) | 2021-12-30 | 2021-12-30 | High-concurrency real-time multi-attribute aggregated map cluster service system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114003646A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114741378A (en) * | 2022-04-06 | 2022-07-12 | 广西师范大学 | Data center system and method for multi-data-source tourist destination |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101388904A (en) * | 2008-07-29 | 2009-03-18 | 北京超图软件股份有限公司 | GIS service aggregating method, device and system |
| CN103036954A (en) * | 2012-12-03 | 2013-04-10 | 北京邮电大学 | Mobile information aggregation system based on geographic information system (GIS) and mobile information aggregation method based on GIS |
| CN112001704A (en) * | 2020-08-27 | 2020-11-27 | 中犹(南京)智慧城市创新研究院有限公司 | Provincial level traffic construction intelligent construction site management platform based on micro-service framework |
| CN112328794A (en) * | 2020-11-10 | 2021-02-05 | 南京师范大学 | Typhoon event information aggregation method |
-
2021
- 2021-12-30 CN CN202111638401.6A patent/CN114003646A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101388904A (en) * | 2008-07-29 | 2009-03-18 | 北京超图软件股份有限公司 | GIS service aggregating method, device and system |
| CN103036954A (en) * | 2012-12-03 | 2013-04-10 | 北京邮电大学 | Mobile information aggregation system based on geographic information system (GIS) and mobile information aggregation method based on GIS |
| CN112001704A (en) * | 2020-08-27 | 2020-11-27 | 中犹(南京)智慧城市创新研究院有限公司 | Provincial level traffic construction intelligent construction site management platform based on micro-service framework |
| CN112328794A (en) * | 2020-11-10 | 2021-02-05 | 南京师范大学 | Typhoon event information aggregation method |
Non-Patent Citations (3)
| Title |
|---|
| 山金孝等: "《OPENSHIFT云原生架构原理与实践＝CLOUD NATIVE ARCHITECTURE BASED ON OPENSHIFT PRINCIPLE AND PRACTICE》", 30 April 2020, 机械工业出版社 * |
| 袁波等: "《云应用***开发技术》", 29 February 2020, 西安电子科技大学出版社 * |
| 赵仲恺: "银行开放平台分布式应用架构设计与实现", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114741378A (en) * | 2022-04-06 | 2022-07-12 | 广西师范大学 | Data center system and method for multi-data-source tourist destination |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Bedi et al. | Beginning with big data simplified | |
| EP2327024A2 (en) | Techniques for resource location and migration across data centers | |
| CN104794150A (en) | Cloud storage model and management method based on space knowledge cloud environment | |
| CN103473320A (en) | Method for service combination facing cloud-spanning platform | |
| CN108268569A (en) | The acquisition of water resource monitoring data and analysis system and method based on big data technology | |
| CN111177292A (en) | WebGIS collaborative plotting method and system for emergency incident | |
| CN102207978A (en) | Database access method and system | |
| Hillier | The fourth sustainability, creativity: statistical associations and credible mechanisms | |
| CN111126852A (en) | BI application system based on big data modeling | |
| CN114003646A (en) | High-concurrency real-time multi-attribute aggregated map cluster service system | |
| CN116166191A (en) | Integrated system of lake and storehouse | |
| US10650353B2 (en) | Context oriented assessment for travel companionship | |
| Nam et al. | Building a big data oriented architecture for enterprise integration | |
| US11625364B2 (en) | Cognitive synchronization of digital files | |
| CN115510116A (en) | Data directory construction method, device, medium and equipment | |
| US9754228B2 (en) | Integrating software solutions to execute business applications | |
| US8990836B2 (en) | Integrating software solution units | |
| Marrero et al. | Performance analysis in NoSQL databases, relational databases and NoSQL databases as a service in the cloud | |
| CN111563832A (en) | Cloud-based multi-citizen service fusion platform | |
| Vojinović et al. | The development of location based services for fleet management | |
| US20230176913A1 (en) | Cross-domain cabin computing system and method based on data resource distribution | |
| Mishra et al. | The role of grid technologies: a next level combat with big data | |
| Li et al. | Architecture Design of Cryptographic Data Management Platform Based on Hadoop | |
| CN115063275A (en) | Comprehensive micro-service system | |
| Begum et al. | A SURVEY ON CLOUD DBMS ENABLED DATA TRANSACTIONS AND DATA STORAGE |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20220201 |
|
| RJ01 | Rejection of invention patent application after publication |