CN105956006A - Structured query system for key value pair data storage - Google Patents
Structured query system for key value pair data storage Download PDFInfo
- Publication number
- CN105956006A CN105956006A CN201610249392.4A CN201610249392A CN105956006A CN 105956006 A CN105956006 A CN 105956006A CN 201610249392 A CN201610249392 A CN 201610249392A CN 105956006 A CN105956006 A CN 105956006A
- Authority
- CN
- China
- Prior art keywords
- alarm
- memory database
- module
- functional module
- redis
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
- G06F16/24534—Query rewriting; Transformation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/242—Query formulation
- G06F16/2433—Query languages
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a structured query system for key value pair data storage. The system comprises an alarm application end function module, a JDBC driving function module, and a redis memory database. The alarm application end function module comprises an alarm centralized showing assembly, an alarm acquisition assembly, a knowledge enriching assembly, and an alarm bottom layer acquisition assembly. The alarm application end function module is connected with the JDBC driving module. The JDBC driving function module is used as a bridge of the alarm application end function module to read the redis memory database, and is connected with the redis memory database. The redis memory database is used as a supporting body of alarm data, and a front end queries the memory database through the JDBC driving function module, so as to realize correlation analysis of real-time alarm information. The system solves technical problems in the prior art that system development and maintenance costs are relatively high since a memory database is used to support large amount of data of an alarm system, but mainstream memory databases at present mostly use self-defined API interfaces to integrate with an application system, not following a SQL interface, and the mainstream memory databases are not beneficial for integrating with external system.
Description
Technical field
The invention belongs to database development and alarming processing field, particularly relate to the structuring of a kind of key-value pair data storage
Inquiry system.
Background technology
In traditional data base, owing to the physical arrangement of disk limits, its access and recall precision and parallel computation
Ability seems the most weak in the environment of big data quantity, although such issues that have a lot of way to solve, such as application program
Caching, distributed etc., but cost can be the highest.And along with the development of hardware these years, the memory size that present server has by
Cumulative greatly, additionally due to the maturation of NUMA architecture, also eliminate multi-CPU and access the bottleneck problem of internal memory, therefore memory database
Applied, by the mass data of memory database carrying warning system, more can be played the advantage of memory database.But at present
The api interface of memory database many employings self-defining of main flow carries out integrated with application system, does not follow SQL interface, and this will
Cause exploitation and operation maintenance personnel to put into a large amount of manpowers and time and go to relearn NOSQL grammer and api interface so that system
Development and maintenance cost is higher, and is unfavorable for the outside system integration.
Summary of the invention
The technical problem to be solved in the present invention is: provide the structuralized query system that a kind of key-value pair data stores, to solve
The certainly prior art mass data of memory database carrying warning system, but the many employings of memory database of main flow are certainly at present
The api interface of row definition carries out integrated with application system, does not follow SQL interface, and it is big that this will cause exploitation and operation maintenance personnel to put into
Measure manpower and the time goes to relearn NOSQL grammer and api interface so that the development and maintenance cost of system is higher, and not
It is beneficial to technical problems such as the outside system integrations.
Technical solution of the present invention:
A kind of structuralized query system of key-value pair data storage, it includes:
Alarm application end functional module, it includes that alerting concentration represents assembly, alarm collection assembly, instructive assembly and alarm
Bottom acquisition component, drives module to be connected with JDBC；
JDBC drive functional module, reads the bridge of redis memory database, with redis as alarm application end functional module
Memory database connects；
Redis memory database, as the supporting body of alarm data, JDBC drive functional module is passed through to memory database in front end
Inquire about, it is achieved the association analysis of Real-time Alarm information.
It also includes memory manager functional module, by graphic interface, managing internal memory data.Memory manager function
Module is connected into memory database by j JDBC drive functional module, passes through on the administration interface of memory manager functional module
Data base is directly operated by SQL statement, and the data that alarm application end functional module sends are also by memory manager function mould
Block management is safeguarded.
Described JDBC drive functional module includes that JDBC drives module, database connection module and SQL-JEDIS grammer to turn
Die change block, JDBC drive module as alarm application end functional module all component bridge, by JDBC drive module with
Redis memory database realizes data interaction；Database connection module is for arranging the IP address of data base, username and password
Configuration information, and preserve be encapsulated in JDBC drive functional module；SQL-JEDIS syntax conversion module, will alarm application end merit
The SQL statement that energy module sends is changed by SQL-JEDIS syntax conversion module, and this module uses
The SQL statement of input is resolved by CCJSqlParserManager.
The SQL statement of input is resolved by described use CCJSqlParserManager, and its analytic method includes:
For query statement, first parse table name and the condition of inquiry, then resolve to the grammer of Jedis: by redis internal memory
Database connection object gets the connection of redis memory database, then uses the inquiry of BinaryJedis.keys method
Redis memory database；
For updating statement, including update and delete, the most first parse table name and the condition field comprised, then solve
Analyse into the grammer of Jedis: got the connection of redis memory database by redis memory database connecting object, re-use
BinaryJedis.hset operates redis memory database.
The method have the benefit that
The whole application end program of the present invention is represented assembly, alarm collection assembly, instructive assembly and alarm by alerting concentration
Bottom sampling instrument forms, and alarm bottom collection produces mass data therewith by the real-time status gathering each test side, these
Data are aggregated entrance alarm component classification and process, and connect entrance data base by JDBC drive functional module, are saved in
In redis memory database, the data of redis memory database are loaded directly into internal memory, and driver accesses the most all at internal memory
The inside operation, is just saved in disk when persistence when, will be at least to the I/O operation efficiency of disk file than traditional database
High an order of magnitude, enters ip address that redis memory database configured, user name, password write into Databasce all in advance even
In connection module, if there is changing redis memory database situation, then only need to be again by new redis internal storage data configuration information
Write into Databasce link block, alarm represents end and sends SQL statement by SQL-JEDIS syntax conversion module and include inserting
Enter, update, the mass data storehouse of redis memory database can be presented to user by Alerting requirements, separately by the common statement such as deletion
The relevant knowledge of alarm that user can be increased newly by outer instructive assembly is inserted into data base and classifies preservation, solves prior art and uses
The mass data of memory database carrying warning system, but the API of memory database many employings self-defining of current main flow
Interface carries out integrated with application system, does not follow SQL interface, and this will cause exploitation and operation maintenance personnel to put into a large amount of manpowers and time
Go to relearn NOSQL grammer and api interface so that the development and maintenance cost of system is higher, and be unfavorable for outside system
The technical problem such as unite integrated.
Accompanying drawing explanation
Fig. 1 is modular structure schematic diagram of the present invention；
Fig. 2 is database connection module configuration diagram of the present invention；
Fig. 3 is overall process flow schematic diagram of the present invention；
Fig. 4 is memory manager functional module framework figure of the present invention；
Fig. 5 is that JDBC drive functional module of the present invention forms structural representation.
Detailed description of the invention
A kind of structuralized query system of key-value pair data storage, it includes:
Alarm application end functional module, it includes that alerting concentration represents assembly, alarm collection assembly, instructive assembly and alarm
Bottom acquisition component, drives module to be connected with JDBC；
Alarm application end functional module all component all uses java language development to complete, and enters in redis so wanting to connect
Warehousing have to apply energy normal and in Redis by JDBC driving as bridge by using JEDIS client to realize alarm
Deposit data Kuku carries out data interaction.
JDBC drive functional module, reads the bridge of redis memory database as alarm application end functional module, with
Redis memory database connects；
Described JDBC drive functional module includes that JDBC drives module, database connection module and SQL-JEDIS syntax conversion mould
Block.
JDBC drive module as alarm application end functional module all component bridge, by JDBC drive module with
Redis memory database realizes data interaction, and JDBC drives module to use JEDIS client to realize.
Database connection module is for arranging the IP address of data base, username and password configuration information, and preserves encapsulation
In JDBC drive functional module；
Alarm application end wants to connect redis memory database and have to arrange the IP address of data base, user name, password etc. and join
Confidence ceases, and this configuration information needs artificial write into Databasce link block, then this module is encapsulated into JDBC and drives function mould
In block.Alarm application end drives module to be successfully connected data base by calling the JDBC of NOSQL, and it is for all of data base
Operation is all to carry out on this connects basis, and the present invention defines a series of class, it is achieved JDBC standard interface, in being used for obtaining
The connection deposited, as follows:
Definition drives class RedisDriver, it is achieved java.sql.Driver interface；
Definition data base connects class RedisConnection, it is achieved java.sql.Connection interface；
Use development interface open for Jedis, set IP, port, obtain data base after user cipher and connect.This connection is also wrapped
Include other attributes, than such as whether closedown etc.；
The work process of RedisConnection object:
1. by IP, user name, password write configuration file, to prepare to log in.
2. the API calling Jedis can operate Redis.
3. by defined good JDBC drive connection data base.
SQL-JEDIS syntax conversion module, passes through SQL-JEDIS by the SQL statement that alarm application end functional module sends
Syntax conversion module is changed, and this module uses CCJSqlParserManager to resolve the SQL statement of input.
The SQL statement of input is resolved by described use CCJSqlParserManager, and its analytic method includes:
For query statement, first parse table name and the condition of inquiry, then resolve to the grammer of Jedis: by redis internal memory
Database connection object gets the connection of redis memory database, then uses BinaryJedis.keys (" table name:
Key ") method inquiry redis memory database；
For updating statement, including update and delete, the most first parse table name and the condition field comprised, then solve
Analyse into the grammer of Jedis: got the connection of redis memory database by redis memory database connecting object, re-use
BinaryJedis.hset (table name, field, value) operates redis memory database.
Redis memory database, as the supporting body of alarm data, the internal poke of JDBC drive functional module is passed through in front end
Inquire about according to storehouse, it is achieved the association analysis of Real-time Alarm information.
Memory database Redis, it is a high-performance key-value(key-value pair increased income) data base, it supports to deposit
The value type of storage is the most more, including String (character string), List (chained list), set (set), Zset (sorted set
--ordered set) and Hash(Hash type), on this basis, Redis supports the sequence of various different modes, in order to ensure effect
Rate, data are all to be buffered in internal memory.In whole implementation process we using memory database as high-volume alarm data
Supporting body, all alarm application end assemblies are all connected with redis data base, and front end drives internal poke by the JDBC of NOSQL
Inquire about according to storehouse, rely on memory database High Performance Data Query can allow access this warning system all monitor supervision platforms, it is achieved
The association analysis of Real-time Alarm information.
It also includes memory manager functional module, by graphic interface, managing internal memory data.Memory manager function
Module is connected into memory database by j JDBC drive functional module, passes through on the administration interface of memory manager functional module
Data base is directly operated by SQL statement, and the data that alarm application end functional module sends are also by memory manager function mould
Block management is safeguarded.
Memory manager functional module is an add-on module of whole warning system, owing to record alert database is interior poke
According to storehouse, safeguard for convenience of its management, one memory manager functional module of special addition, by its graphic interface, can be more directly perceived
Managing internal memory data.After module can drive module to be connected into memory database by JDBC, the management of module can be driven at JDBC
Directly being operated data base by SQL statement on interface, the alarm high-volume data that send of application end also can thus module
Management is safeguarded.
Present invention achieves the NOSQL database-driven supporting SQL query, driven by this JDBC and allow alarm application end
Can be connected into memory database, and rely on high performance computation and the storage of memory database, make what memory bank carried to alert number in a large number
Secondary operations is carried out, it is possible to be supplied to each monitoring system timely and be analyzed processing according to the SQL statement sent by user.
Claims (4)
1. a structuralized query system for key-value pair data storage, it includes:
Alarm application end functional module, it includes that alerting concentration represents assembly, alarm collection assembly, instructive assembly and alarm
Bottom acquisition component, drives module to be connected with JDBC；
JDBC drive functional module, reads the bridge of redis memory database, with redis as alarm application end functional module
Memory database connects；
Redis memory database, as the supporting body of alarm data, JDBC drive functional module is passed through to memory database in front end
Inquire about, it is achieved the association analysis of Real-time Alarm information.
The structuralized query system of a kind of key-value pair data the most according to claim 1 storage, it is characterised in that: it also wraps
Include memory manager functional module, by graphic interface, managing internal memory data；Memory manager functional module passes through j JDBC
Drive functional module is connected into memory database, by SQL statement to data on the administration interface of memory manager functional module
Storehouse directly operates, and the data that alarm application end functional module sends also are safeguarded by memory manager managing functional module.
The structuralized query system of a kind of key-value pair data the most according to claim 1 storage, it is characterised in that: described
JDBC drive functional module includes that JDBC drives module, database connection module and SQL-JEDIS syntax conversion module, and JDBC drives
Dynamic model block, as the bridge of alarm application end functional module all component, drives module and redis memory database by JDBC
Realize data interaction；Database connection module is for arranging the IP address of data base, username and password configuration information, and preserves
It is encapsulated in JDBC drive functional module；SQL-JEDIS syntax conversion module, the SQL that alarm application end functional module is sent
Statement is changed by SQL-JEDIS syntax conversion module, and this module uses CCJSqlParserManager to input
SQL statement resolves.
The structuralized query system of a kind of key-value pair data the most according to claim 3 storage, it is characterised in that make described in:
Resolving the SQL statement of input with CCJSqlParserManager, its analytic method includes:
For query statement, first parse table name and the condition of inquiry, then resolve to the grammer of Jedis: by redis internal memory
Database connection object gets the connection of redis memory database, then uses the inquiry of BinaryJedis.keys method
Redis memory database；
For updating statement, including update and delete, the most first parse table name and the condition field comprised, then solve
Analyse into the grammer of Jedis: got the connection of redis memory database by redis memory database connecting object, re-use
BinaryJedis.hset operates redis memory database.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610249392.4A CN105956006A (en) | 2016-04-21 | 2016-04-21 | Structured query system for key value pair data storage |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610249392.4A CN105956006A (en) | 2016-04-21 | 2016-04-21 | Structured query system for key value pair data storage |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105956006A true CN105956006A (en) | 2016-09-21 |
Family
ID=56917871
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610249392.4A Pending CN105956006A (en) | 2016-04-21 | 2016-04-21 | Structured query system for key value pair data storage |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105956006A (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106597968A (en) * | 2016-12-24 | 2017-04-26 | 大连尚能科技发展有限公司 | Converter high-speed real-time monitoring system and method based on Redis |
| CN107480210A (en) * | 2017-07-27 | 2017-12-15 | 北京北信源软件股份有限公司 | Internal storage data operating method and device |
| CN107656971A (en) * | 2017-09-02 | 2018-02-02 | 国网辽宁省电力有限公司 | A kind of intelligent grid collection Monitoring Data storage method based on Redis |
| CN108108429A (en) * | 2017-12-01 | 2018-06-01 | 深圳市小牛在线互联网信息咨询有限公司 | Date storage method, device, storage medium and the computer equipment of terminal device |
| CN108256896A (en) * | 2017-12-15 | 2018-07-06 | 微梦创科网络科技（中国）有限公司 | A kind of method and device of business Alerts aggregation |
| CN108664507A (en) * | 2017-03-31 | 2018-10-16 | 武汉斗鱼网络科技有限公司 | User information treating method and apparatus |
| CN114928550A (en) * | 2022-04-29 | 2022-08-19 | 杭州默安科技有限公司 | Database interaction account security detection method and system |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101833620A (en) * | 2010-04-28 | 2010-09-15 | 国网电力科学研究院 | Custom security JDBC driver-based database protective method |
| CN103955492A (en) * | 2014-04-17 | 2014-07-30 | 国网宁夏电力公司 | Method for storing data in electrical power system |
| CN104270432A (en) * | 2014-09-22 | 2015-01-07 | 苏州耐克斯特能源开采技术有限公司 | Real-time data service system and data interaction method based on drilling industry |
| CN104944240A (en) * | 2015-05-19 | 2015-09-30 | 重庆大学 | Elevator equipment state monitoring system based on large data technology |
-
2016
- 2016-04-21 CN CN201610249392.4A patent/CN105956006A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101833620A (en) * | 2010-04-28 | 2010-09-15 | 国网电力科学研究院 | Custom security JDBC driver-based database protective method |
| CN103955492A (en) * | 2014-04-17 | 2014-07-30 | 国网宁夏电力公司 | Method for storing data in electrical power system |
| CN104270432A (en) * | 2014-09-22 | 2015-01-07 | 苏州耐克斯特能源开采技术有限公司 | Real-time data service system and data interaction method based on drilling industry |
| CN104944240A (en) * | 2015-05-19 | 2015-09-30 | 重庆大学 | Elevator equipment state monitoring system based on large data technology |
Non-Patent Citations (1)
| Title |
|---|
| 唐诚: "Redis数据库在微博***中的实践", 《厦门城市职业学院学报》 * |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106597968A (en) * | 2016-12-24 | 2017-04-26 | 大连尚能科技发展有限公司 | Converter high-speed real-time monitoring system and method based on Redis |
| CN108664507A (en) * | 2017-03-31 | 2018-10-16 | 武汉斗鱼网络科技有限公司 | User information treating method and apparatus |
| CN108664507B (en) * | 2017-03-31 | 2022-03-22 | 武汉斗鱼网络科技有限公司 | User information processing method and device |
| CN107480210A (en) * | 2017-07-27 | 2017-12-15 | 北京北信源软件股份有限公司 | Internal storage data operating method and device |
| CN107480210B (en) * | 2017-07-27 | 2020-07-14 | 北京北信源软件股份有限公司 | Memory data operation method and device |
| CN107656971A (en) * | 2017-09-02 | 2018-02-02 | 国网辽宁省电力有限公司 | A kind of intelligent grid collection Monitoring Data storage method based on Redis |
| CN108108429A (en) * | 2017-12-01 | 2018-06-01 | 深圳市小牛在线互联网信息咨询有限公司 | Date storage method, device, storage medium and the computer equipment of terminal device |
| CN108108429B (en) * | 2017-12-01 | 2020-08-18 | 深圳市小牛在线互联网信息咨询有限公司 | Data storage method and device of terminal equipment, storage medium and computer equipment |
| CN108256896A (en) * | 2017-12-15 | 2018-07-06 | 微梦创科网络科技（中国）有限公司 | A kind of method and device of business Alerts aggregation |
| CN114928550A (en) * | 2022-04-29 | 2022-08-19 | 杭州默安科技有限公司 | Database interaction account security detection method and system |
| CN114928550B (en) * | 2022-04-29 | 2024-05-03 | 杭州默安科技有限公司 | Database interaction account security detection method and system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105956006A (en) | Structured query system for key value pair data storage | |
| Gupta et al. | Cloud computing and big data analytics: what is new from databases perspective? | |
| US11544239B2 (en) | Low-latency database analysis using external data sources | |
| US7917463B2 (en) | System and method for data warehousing and analytics on a distributed file system | |
| US10572494B2 (en) | Bootstrapping the data lake and glossaries with ‘dataset joins’ metadata from existing application patterns | |
| JP4814628B2 (en) | Data access layer class generator | |
| CN102968374B (en) | A kind of data warehouse method of testing | |
| CN107038162A (en) | Real time data querying method and system based on database journal | |
| CN105160012B (en) | A kind of management method and system of heterogeneous database | |
| CN102081661A (en) | Data integration method and system of heterogeneous relational database based on XML (Extensive Makeup Language) | |
| Ramzan et al. | Intelligent data engineering for migration to NoSQL based secure environments | |
| EP3654198A1 (en) | Conversational database analysis | |
| CN102184222A (en) | Quick searching method in large data volume storage | |
| US11429607B2 (en) | Machine language query management for low-latency database analysis system | |
| CN112965979B (en) | User behavior analysis method and device and electronic equipment | |
| CN114218218A (en) | Data processing method, device and equipment based on data warehouse and storage medium | |
| CN112347071A (en) | Power distribution network cloud platform data fusion method and power distribution network cloud platform | |
| Ranawade et al. | Online analytical processing on hadoop using apache kylin | |
| CN106372135A (en) | Structured query language SQL execution efficiency tracking method and device | |
| CN111125045B (en) | Lightweight ETL processing platform | |
| CN116594982A (en) | Flow number bin construction method based on rule engine and Clickhouse | |
| US9058344B2 (en) | Supporting flexible types in a database | |
| US11803543B2 (en) | Lossless switching between search grammars | |
| Marinov et al. | NoSQL approach for sensor data storage and retrieval | |
| CN113010483A (en) | Mass log management method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20160921 |