CN108920614A - The method, apparatus and system of online query data - Google Patents
The method, apparatus and system of online query data Download PDFInfo
- Publication number
- CN108920614A CN108920614A CN201810688094.4A CN201810688094A CN108920614A CN 108920614 A CN108920614 A CN 108920614A CN 201810688094 A CN201810688094 A CN 201810688094A CN 108920614 A CN108920614 A CN 108920614A
- Authority
- CN
- China
- Prior art keywords
- data
- inquiry
- tree
- query
- web platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention provides the method, apparatus and system of a kind of online query data, this method includes：Receive the inquiry request that analysis personnel are issued using the web platform pre-established；Syntax parsing is carried out to the inquiry request received and obtains corresponding syntax tree, rule searching corresponding with inquiry request is customized based on syntax tree；Corresponding inquiry data are inquired from initialized data base according to rule searching, by the inquiry data feedback inquired to web platform.Thus, the mode of the online query data of the embodiment of the present invention can support customized a variety of query grammars, rule searching corresponding with inquiry request flexibly can be customized based on syntax tree, and inquire data required for inquiring from database according to rule searching.Also, this programme also effectively improves the search efficiency of online query data.
Description
Technical field
The present invention relates to field of computer technology, more particularly to the method, apparatus and system of a kind of online query data.
Background technique
With the continuous development of big data technology, data query function Internet-based has obtained more and more extensive answer
With.In the prior art, it when the OA operation analysis personnel of profession inquire data from the database for be stored with mass data, usually needs
The inquiry request of user is first parsed, that is, understands the query intention of user, and then realize the inquiry of data.
It, cannot be flexible by the way of the request of specific syntax analyzing user queries but at present during data query
The customized query grammar of support, the diversity and requirement on flexibility of inquiry can not be met well.Therefore, how more
Efficiently and the flexible inquiry for realizing data is the important technical problem faced at present.
Summary of the invention
In view of the above problems, it proposes on the present invention overcomes the above problem or at least be partially solved in order to provide one kind
State the method, apparatus and system of the online query data of problem.
One side according to the present invention provides a kind of method of online query data, including：
It receives analysis personnel and passes through the inquiry request that the web platform pre-established issues；
Syntax parsing is carried out to the inquiry request received and obtains corresponding syntax tree, based on syntax tree customization and institute
State the corresponding rule searching of inquiry request；
Corresponding inquiry data are inquired from initialized data base according to the rule searching, the inquiry data inquired are anti-
It is fed to the web platform.
Optionally, analysis personnel are received and pass through the inquiry request that the web platform pre-established issues, including：Based on http
Agreement receives analysis personnel and passes through the http inquiry request that the web platform pre-established issues.
Optionally, rule searching corresponding with the inquiry request is customized based on the syntax tree, including：Using optimizer
The syntax tree is optimized；
Corresponding query plan tree is generated according to the syntax tree after optimization, wherein includes multiple in the query plan tree
Tree node, the tree node correspond to the query steps of the rule searching.
Optionally, corresponding inquiry data are inquired from initialized data base according to the rule searching, are looked into what is inquired
Data feedback is ask to the web platform, including：
According to the sequencing of the corresponding query steps of tree node in the query plan tree, successively from the preset data
Corresponding inquiry data are inquired in library；
After whole tree nodes inquires data according to the query plan tree, the inquiry number inquired is integrated
According to by the inquiry data feedback after integration to the web platform.
Optionally, corresponding inquiry data are inquired from initialized data base according to the rule searching, are looked into what is inquired
Data feedback is ask to the web platform, including：
According to the sequencing of the corresponding query steps of tree node in the query plan tree, successively from the preset data
Corresponding inquiry data are inquired in library；
After inquiring corresponding inquiry data according to the corresponding query steps of any tree node, by current queries to look into
Data feedback is ask to the web platform.
Optionally, corresponding inquiry data are inquired from initialized data base according to the rule searching, are looked into what is inquired
Data feedback is ask to the web platform, including：
Multiple tree nodes are chosen from the query plan tree；
From the initialized data base inquiry corresponding inquiry data parallel according to the multiple tree nodes chosen；
After whole tree nodes inquires data according to the query plan tree, the inquiry number inquired is integrated
According to by the inquiry data feedback after integration to the web platform.
Optionally, multiple tree nodes are chosen from the query plan tree, including：
Multiple tree nodes are chosen according to the sequencing of the corresponding query steps of tree node from the query plan tree；
And/or
Multiple tree nodes are randomly selected from the query plan tree.
Optionally, further include：According to the rule searching after inquiring corresponding inquiry data in initialized data base,
The inquiry data and corresponding inquiry request that caching query arrives；
When receiving the same queries request that analysis personnel are issued by the web platform pre-established again, directly obtain
Take the inquiry data corresponding with the inquiry request of caching.
Optionally, by the inquiry data feedback inquired to the web platform, including：
The inquiry data inquired are formatted, the inquiry data of uniform format are obtained；
By the inquiry data feedback after uniform format to the web platform.
Another aspect according to the present invention additionally provides a kind of device of online query data, including
Receiving module passes through the inquiry request that the web platform pre-established issues suitable for receiving analysis personnel；
Parsing module obtains corresponding syntax tree suitable for carrying out syntax parsing to the inquiry request received, based on described
Syntax tree customizes rule searching corresponding with the inquiry request；
Enquiry module is suitable for inquiring corresponding inquiry data from initialized data base according to the rule searching, will inquire
The inquiry data feedback arrived is to the web platform.
Optionally, the receiving module is further adapted for：It is flat by the web pre-established that analysis personnel are received based on http agreement
The http inquiry request that platform issues.
Optionally, the parsing module is further adapted for：The syntax tree is optimized using optimizer；
Corresponding query plan tree is generated according to the syntax tree after optimization, wherein includes multiple in the query plan tree
Tree node, the tree node correspond to the query steps of the rule searching.
Optionally, the enquiry module is further adapted for：According to the corresponding query steps of tree node in the query plan tree
Sequencing successively inquires corresponding inquiry data from the initialized data base；
After whole tree nodes inquires data according to the query plan tree, the inquiry number inquired is integrated
According to by the inquiry data feedback after integration to the web platform.
Optionally, the enquiry module is further adapted for：According to the corresponding query steps of tree node in the query plan tree
Sequencing successively inquires corresponding inquiry data from the initialized data base；
After inquiring corresponding inquiry data according to the corresponding query steps of any tree node, by current queries to look into
Data feedback is ask to the web platform.
Optionally, the enquiry module is further adapted for：Multiple tree nodes are chosen from the query plan tree；
From the initialized data base inquiry corresponding inquiry data parallel according to the multiple tree nodes chosen；
After whole tree nodes inquires data according to the query plan tree, the inquiry number inquired is integrated
According to by the inquiry data feedback after integration to the web platform.
Optionally, the enquiry module is further adapted for：According to the corresponding query steps of tree node from the query plan tree
Sequencing choose multiple tree nodes；And/or multiple tree nodes are randomly selected from the query plan tree.
Optionally, described device further includes：Cache module, suitable for being looked into from initialized data base according to the rule searching
After asking corresponding inquiry data, inquiry data and corresponding inquiry request that caching query arrives；
It is asked when the receiving module receives analysis personnel again by the same queries that the web platform pre-established issues
When asking, the inquiry data corresponding with the inquiry request of caching are directly acquired.
Optionally, the enquiry module is further adapted for：
The inquiry data inquired are formatted, the inquiry data of uniform format are obtained；
By the inquiry data feedback after uniform format to the web platform.
According to the present invention in another aspect, additionally provide a kind of system of online query data, including web platform, above
The device and initialized data base of online query data described in any embodiment, wherein
The web platform is received by the inquiry request of analysis personnel's input, and by the inquiry request be sent to it is described
The device of line inquiry data；
The device of the online query data receives the inquiry request from the web platform, carries out to the inquiry request
Syntax parsing obtains corresponding syntax tree, customizes rule searching corresponding with the inquiry request based on the syntax tree；
The device of the online query data inquires corresponding inquiry number according to the rule searching from initialized data base
According to, and by the inquiry data feedback inquired to the web platform.
Another aspect according to the present invention, additionally provides a kind of computer storage medium, and the computer storage medium is deposited
Computer program code is contained, when the computer program code is run on the computing device, the calculating equipment is caused to be held
The method of online query data described in row any embodiment above.
According to the present invention in another aspect, additionally provide a kind of calculating equipment, including：Processor；It is stored with computer journey
The memory of sequence code；When the computer program code is run by the processor, the calculating equipment is caused to execute
The method of online query data described in literary any embodiment.
In embodiments of the present invention, when the inquiry request for receiving the web platform sending that analysis personnel utilization pre-establishes
Afterwards, syntax parsing first is carried out to the inquiry request received and obtains corresponding syntax tree, be then based on syntax tree customization and inquiry
Corresponding rule searching is requested, and then inquires corresponding inquiry data from initialized data base according to rule searching, will be inquired
The inquiry data feedback arrived is to web platform.The mode of the online query data of the embodiment of the present invention passes through to inquiry request as a result,
It carries out syntax parsing and obtains corresponding syntax tree, to support customized a variety of query grammars, that is, realize own inquiry language
Method, and then corresponding with inquiry request rule searching flexibly can be customized based on syntax tree, and according to rule searching from data
Data are inquired required for inquiring in library.Further, this programme also effectively improves the inquiry effect of online query data
Rate.
The above description is only an overview of the technical scheme of the present invention, in order to better understand the technical means of the present invention,
And it can be implemented in accordance with the contents of the specification, and in order to allow above and other objects of the present invention, feature and advantage can
It is clearer and more comprehensible, the followings are specific embodiments of the present invention.
According to the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will be brighter
The above and other objects, advantages and features of the present invention.
Detailed description of the invention
By reading the following detailed description of the preferred embodiment, various other advantages and benefits are common for this field
Technical staff will become clear.The drawings are only for the purpose of illustrating a preferred embodiment, and is not considered as to the present invention
Limitation.And throughout the drawings, the same reference numbers will be used to refer to the same parts.In the accompanying drawings：
Fig. 1 shows the flow diagram of the method for online query data according to an embodiment of the invention；
Fig. 2 shows the search engine design framework schematic diagrames of online query data according to an embodiment of the invention；
Fig. 3 shows the structural schematic diagram of the device of online query data according to an embodiment of the invention；
Fig. 4 shows the structural schematic diagram of the device of online query data in accordance with another embodiment of the present invention；And
Fig. 5 shows the structural schematic diagram of the system of online query data according to an embodiment of the invention.
Specific embodiment
Exemplary embodiments of the present disclosure are described in more detail below with reference to accompanying drawings.Although showing the disclosure in attached drawing
Exemplary embodiment, it being understood, however, that may be realized in various forms the disclosure without should be by embodiments set forth here
It is limited.On the contrary, these embodiments are provided to facilitate a more thoroughly understanding of the present invention, and can be by the scope of the present disclosure
It is fully disclosed to those skilled in the art.
In order to solve the above technical problems, the embodiment of the invention provides a kind of methods of online query data.Fig. 1 is shown
The flow diagram of the method for online query data according to an embodiment of the invention.Referring to Fig. 1, this method includes at least step
Rapid S102 to step S106.
Step S102 receives analysis personnel and passes through the inquiry request that the web platform pre-established issues.
In this step, the type of inquiry request can be http inquiry request, and therefore, the present invention program can be based on
Http agreement receives analysis personnel and passes through the http inquiry request that the web platform pre-established issues.Certainly, inquiry request may be used also
To be other types, for different types of inquiry request, the transport protocol of use is also different, and the embodiment of the present invention is to inquiry
The type of request and the transport protocol type of use do not do specific restriction.
In addition, the inquiry request in the present invention can be the inquiry request about inquiry log of analysis personnel sending.
Step S104 carries out syntax parsing to the inquiry request received and obtains corresponding syntax tree, fixed based on syntax tree
Make rule searching corresponding with inquiry request.
Rule searching in the step is actually based on the inquiry plan corresponding with inquiry request of syntax tree customization, i.e.,
Corresponding query steps are planned out according to inquiry request, which step first inquired, then which step inquired, to effectively improve inquiry effect
Rate.
Step S106 inquires corresponding inquiry data, the inquiry that will be inquired according to rule searching from initialized data base
Data feedback is to web platform.
In this step, after inquiring data feedback to web platform, can carry out in the display equipment in web platform can
It is shown depending on changing, analysis personnel can see the inquiry data of feedback by the web platform, to be analyzed and used accordingly.
The mode of the online query data of the embodiment of the present invention is corresponding by obtaining to inquiry request progress syntax parsing
Syntax tree realizes own query grammar, and then can flexibly be based on language to support customized a variety of query grammars
Method tree customizes rule searching corresponding with inquiry request, and inquires number required for inquiring from database according to rule searching
According to.Further, this programme also effectively improves the search efficiency of online query data.
Step S104 is seen above, in an embodiment of the present invention, look into corresponding with inquiry request is customized based on syntax tree
When asking rule, in order to guarantee the accuracy of syntax tree, syntax tree can also be optimized using corresponding optimizer, into
And corresponding query plan tree is generated according to the syntax tree after optimization, it include multiple tree nodes, tree node pair in query plan tree
It should be in the query steps of rule searching.Query plan tree can effectively understand inquiry plan, it is succinct embody, help
In efficiently and accurately realizing data query.In the embodiment, the optimizer optimized to syntax tree can be excellent using plain
Change device, other kinds of optimizer can also be used, the embodiment of the present invention does not do specific restriction to this.
Step S104 and S106 are seen above, after generating corresponding query plan tree according to the syntax tree after optimization, due to
Include multiple tree nodes in query plan tree, therefore is inquiring corresponding inquiry from initialized data base according to query plan tree
During data, data query can be carried out using various ways.
Mode one
According to the sequencing of the corresponding query steps of tree node in query plan tree, successively inquired from initialized data base
Corresponding inquiry data.In turn, it after inquiring data according to whole tree nodes in query plan tree, integrates and inquires
Data are inquired, by the inquiry data feedback after integration to web platform.
For example, include three tree nodes in the query plan tree generated, then, the corresponding query steps 1 of tree node 1, burl
The corresponding query steps 2 of point 2, the corresponding query steps 3 of tree node 3, according to query steps 1, step 2, the sequence of step 3, i.e. basis
The sequence of tree node 1 to 3 inquires corresponding inquiry data from initialized data base.When three tree nodes realize the inquiry of data
Later, the data inquired are integrated, to be integrated into data content corresponding with inquiry request, and by the inquiry after integration
Data feedback is into web platform.
Mode two
According to the sequencing of the corresponding query steps of tree node in query plan tree, successively inquired from initialized data base
Corresponding inquiry data.In turn, after inquiring corresponding inquiry data according to the corresponding query steps of any tree node, will work as
Before the inquiry data feedback that inquires to web platform.
Which two and mode one the difference is that, mode two is not to be according to tree whole in query plan tree
Inquiry data are returned after querying node data, but after inquiring data according to any one tree node, it will be current
The data inquired directly feed back to web platform, can also effectively improve the data query experience of analysis personnel in this way, help
Inquiry data are viewed in time in analysis personnel.
Mode three
Firstly, choosing multiple tree nodes from query plan tree.Then, according to the multiple tree nodes chosen it is parallel from pre-
It sets and inquires corresponding inquiry data in database.Finally, being finished according to tree node inquiry data whole in query plan tree
Afterwards, the inquiry data inquired are integrated, by the inquiry data feedback after integration to web platform.
In this approach, it can be from query plan tree according to tree node from choosing multiple tree nodes in query plan tree
The sequencing of corresponding query steps chooses multiple tree nodes, for example, the sequencing according to query steps chooses 2 or 3
A or 4 equal number of tree nodes, for the parallel carry out data query of multiple tree nodes of selection.It is, of course, also possible to from looking into
Multiple tree nodes are randomly selected in inquiry plan tree.Furthermore it is also possible to the sequencing choosing according to query steps in query plan tree
The tree node of specified quantity is taken, and randomly selects some tree nodes.The embodiment of the present invention does not do the selection quantity of tree node
It is specific to limit.
Above-described embodiment is only schematical, inquires corresponding inquiry number from initialized data base according to query plan tree
According to mode can also be other modes.
In embodiments of the present invention, initialized data base can actually include one, also may include multiple, and every number
Different types of data can be stored respectively according to library.
For example, initialized data base may include Poseidon database, quick_table database, tidb database,
Mysql database is below introduced the storage object in each database.
Poseidon database, main to provide tera-scale data, data are precisely retrieved according to participle, pass through the data
Library can go out the original log in tera-scale data according to the index quick-searching built up in advance, and index gets up also relatively more flexible.
When the inquiry request that business analyst issues is that log query is requested, can be inquired from the database in relevant log
Hold.
Quick_table database, the data more frequently inquired provide quick-searching, such as sample (tera-scale)
Daily queries first appears the time etc., the grade retrieval of design object second.
The database is actually a complementation to Poseidon database, although poseidon database has index
Flexibly, the advantages that storage data quantity is big equal, but its dependence QPS (Query Per Second, query rate per second) is very low
Hadoop component usually requires to look into many layer indexs to retrieve a data, when the data and sample for needing to inquire
MD5 (Message-Digest Algorithm 5, message digest algorithm 5) and sha1 (Secure Hash Algorithm,
Secure Hash Algorithm) it is related when, essence relates only to simple kv (Key-Value) query demand, without so clever
Index living.For example, some users (such as white list user) need PV (Page View, page browsing amount), UV
When (UniqueVisitor, independent access number of users) interface meets the automatic usage scenario obtained with batch (every time 1000), adopt
The application can not be currently supported with Poseidon database.Therefore, introducing quick table database to a certain extent can be with
Accelerate the index speed of some Field Inquiries.
Tidb database is the distributed mysql of an open source, mainly stores the data of some real-time updates.For example, root
The all properties information of the sample of the sample set, nearest a period of time (such as 1 hour) that are filtered out according to certain specified conditions.
Mysql database mainly stores the metadata information of some data.
In order to more clearly embody this programme, now with a specific embodiment to the reality of online query data of the invention
Existing process and each step in the process are introduced, which can be based on a class SQL (Structured Query
Language, structured query language) search engine realize, the search engine reception asked from the inquiry that Web platform is sent
It asks, and is handled inquiry request and obtained corresponding inquiry data.Referring to fig. 2, search engine may include data Layer
Smart, driving layer GDO, Model layer.
Step 1, the inquiry request (such as http request) that user (i.e. analysis personnel) is sent by Web platform, according to inquiry
The correlation parameters of request carry out morphology parsing to inquiry request with the corresponding querying condition of determination, the inquiry item such as parsed
Part is：Needing to inquire md5 is xxx, and path is that the sample of xxx comes out.
Step 2, querying condition is transferred to data Layer smart, data Layer carries out syntax parsing to querying condition and obtains language
Method tree, and inquiry plan (rule searching i.e. hereinbefore) corresponding with inquiry request is customized according to syntax tree, inquiry plan can
By be by the query plan tree of physics in the form of embody.Herein, grammer solution can be carried out to querying condition using grammar parser
Analysis, and syntax tree is optimized using the optimizer of the type such as plain optimizer.The step generates data directory.
In this step, data Layer smart can also realize concurrent Exec, i.e., looked by concurrently accessing database to improve
Ask efficiency, as explained above to data query mode three in slave query plan tree in choose multiple tree nodes, according to choosing
The parallel slave initialized data base of multiple tree nodes in the corresponding inquiry data of inquiry process.
In addition, caching is also a kind of effective means for improving search efficiency, pass through the data buffer storage that will be inquired before
Get up, do not have to access database again when accessing next time, but directly acquire the data of caching, so as to accelerate inquiry speed
Degree.Specifically, can be arrived after inquiring corresponding inquiry data from initialized data base according to rule searching with caching query
Inquire data and corresponding inquiry request.In turn, when analysis personnel are again by identical the looking into of the web platform sending pre-established
When asking request, the inquiry data corresponding with the inquiry request cached before can directly acquiring, without from initialized data base
Data are inquired in middle acquisition, and the workload and Internet resources of inquiry data is greatly saved.
Step 3, driving layer GDO is obtained accordingly based on the query plan tree that data Layer smart is generated from initialized data base
Inquiry data, when get inquiry data after, the inquiry data that can be will acquire are transmitted to Model layers, right by Model layers
The format of the inquiry data received carries out unification.
Driving layer GDO obtains data can be by carrying out data query from Poseidon database and obtaining, can be with base
URL interface polls are carried out according to its URL carried in http inquiry request, Protobuf (Protocol can also be based on
Buffers it) inquires and obtains data, wherein Protobuf is a kind of data description language, can be by structured data sequence
Change, can be used for data storage, communication protocol etc..Drive layer GDO by obtaining inquiry data from different databases,
Different data Layers can be docked.
In this step, if packing forms of the SDK as database, then when driving layer GDO is obtained from initialized data base
It, can be by calling SDK (Software Development Kit, software outside database when taking corresponding inquiry data
Development kit) inquire data.
Step 4, Model layers can be to receiving the uniform format of the inquiry data from driving layer GDO into specified lattice
Formula is such as unified into JSON format, and the inquiry data of specified format is exported to data Layer smart, will be inquired by data Layer
Data feed back to user again.
In the step, Model layers can also carry out field filter before carrying out uniform format to inquiry data, such as from pre-
The field being first stored in needed for filtering out output in the information table of database and calculating.Information table includes Proc Chain chain of processes
Information table, Network network sample information table, Proc Behavior process behavior table, Basic sample Basic Information Table,
Cloud Info sample cloud look into information table (including sample cloud looks into relevant information, such as file path, history rank etc.),
Dropped Files file release information table, ScanLog scanning information table, Upload file upload information table etc. of tracing to the source.Its
In, it include sample key message in Basic sample Basic Information Table, such as historical query amount first appears time, rank, the table
The significance level that a sample can be quickly understood helps to realize the quick search of data, under normal conditions this partial data
From quick table database.Information table can also include other kinds of information table, it is not limited to shown in Figure 2
Each information table.
In addition, Model layers to inquiry data carry out uniform format after, can also using corresponding algorithm to format unite
Data after one are filtered or merge, and then the data after filtering or merging are exported to data Layer.It is corresponding herein to calculate
Method can be using operation methods such as group polymerization, join connections.
Although it should be noted that illustrate only Poseidon database and Stored database in the database of Fig. 2,
It can also include actually the other kinds of database being described above, not do specific restriction herein.
In an embodiment of the present invention, a kind of data processing system is additionally provided, which may be implemented offline to magnanimity
Data processing and online data processing, and data are stored into storage assembly in corresponding initialized data base by treated.
Wherein, what off-line data and online data referred to can be daily record data.
It is extracted in process in off-line data, firstly, being adjusted from file system (such as hdfs, S3) using distributed scheduler
The log of magnanimity is spent, and extracts the metadata of log using Spark engine based on MapReduce model.Then, at using data
Reason frame (such as patent frame) carries out polymerization to the log of scheduling and log metadata and specific format (such as json lattice is calculated
Formula) log (i.e. the intermediate data of Fig. 3), and the log of specific format is sorted out to merge and generates corresponding Virtual table, and
Metadata is counted to obtain statistical information.Finally, Virtual table is stored in poseidon database, and statistical information is saved
In mysql database.Virtual table herein corresponds to information table mentioned hereinabove, can also include specimen_
Detail, specimen details table, specimen_cloud_detail, specimen cloud look into static attribute information table,
Scan_info scanning information table, file_relations document relationship table, specimen have collected sample information table, pe_info
Executable information table (including the relevant table of information can be performed in sample) etc..
It is extracted in process in real time data, when user carries out query service (as antivirus, sample upload, url inquiry, dns
When the services such as (Domain Name System, domain name system) inquiry), it can use real-time logs and handle cluster from query service
Query result in extract the log of generation in real time, and it is (real-time distributed to disappear that the log of extraction is sent to the nsq being pre-created
Breath transmitting platform) in message queue, and then feature extraction cluster consumes log from nsq message queue, to parse and extract
The log feature data of the log of extraction and extraction are collectively stored in tidb database by the characteristic of log in turn.Its
In, the characteristic of log may include the metadata information of log.
In this embodiment, in storage assembly can also include builder cluster, the builder cluster can read by
The log of specific format is converted to the conversion instruction of extended formatting log, and then according to the conversion instruction read to specific format
Log format, finally, the log after format is converted stores in the initialized data base into storage assembly.Example
Such as, data processing equipment offline logs polymerization is calculated the log of json format, and builder cluster is being received json
After the log of format is converted to the instruction of extended formatting, the log of json format is formatted, it is specified to be converted to
Extended formatting, and the log of extended formatting is stored into quick_table database.
Based on the same inventive concept, the embodiment of the invention also provides a kind of device of online query data, Fig. 3 is shown
The structural schematic diagram of the device of online query data according to an embodiment of the invention.Referring to Fig. 3, the dress of online query data
Setting 300 includes receiving module 310, parsing module 320 and enquiry module 330.
Now introduce each composition of the device 300 based on online query data of the embodiment of the present invention or the function of device and
Connection relationship between each section：
Receiving module 310 passes through the inquiry request that the web platform pre-established issues suitable for receiving analysis personnel；
Parsing module 320 is coupled with receiving module 310, is obtained suitable for carrying out syntax parsing to the inquiry request received
Corresponding syntax tree customizes rule searching corresponding with inquiry request based on syntax tree；
Enquiry module 330 is coupled with parsing module 320, is suitable for being inquired from initialized data base according to rule searching corresponding
Inquiry data, by the inquiry data feedback inquired to web platform.
In an embodiment of the present invention, receiving module 310 is further adapted for, and receives analysis personnel based on http agreement and passes through in advance
The http inquiry request that the web platform of foundation issues.
In an embodiment of the present invention, parsing module 320 is further adapted for, and is optimized using optimizer to syntax tree, and root
Corresponding query plan tree is generated according to the syntax tree after optimization, wherein includes multiple tree nodes, tree node pair in query plan tree
It should be in the query steps of rule searching.
In an embodiment of the present invention, enquiry module 330 is further adapted for, according to the corresponding inquiry of tree node in query plan tree
The sequencing of step successively inquires corresponding inquiry data from initialized data base, in turn, according to complete in query plan tree
After the tree node inquiry data in portion, the inquiry data inquired are integrated, the inquiry data feedback after integration to web is put down
Platform.
In an embodiment of the present invention, enquiry module 330 is further adapted for, according to the corresponding inquiry of tree node in query plan tree
The sequencing of step successively inquires corresponding inquiry data from initialized data base, in turn, corresponding according to any tree node
Query steps inquire corresponding inquiry data after, inquiry data feedback that current queries are arrived to web platform.
In an embodiment of the present invention, enquiry module 330 is further adapted for, and multiple tree nodes are chosen from query plan tree, according to
According to the corresponding inquiry data of inquiry in the parallel slave initialized data base of multiple tree nodes of selection, according to complete in query plan tree
After the tree node inquiry data in portion, the inquiry data inquired are integrated, the inquiry data feedback after integration to web is put down
Platform.
In an embodiment of the present invention, enquiry module 330 is further adapted for, and looks into from query plan tree according to tree node is corresponding
The sequencing for asking step chooses multiple tree nodes；And/or multiple tree nodes are randomly selected from query plan tree.
In an embodiment of the present invention, enquiry module 330 is further adapted for, and formats to the inquiry data inquired,
Obtain the inquiry data of uniform format, and by the inquiry data feedback after uniform format to web platform.
The embodiment of the invention also provides the devices of another online query data, referring to fig. 4, the dress of online query data
300 are set other than including receiving module 310, parsing module 320 and enquiry module 330, further includes cache module 340.
Cache module 340 is coupled with enquiry module 330, suitable for being inquired from initialized data base according to rule searching
After corresponding inquiry data, inquiry data and corresponding inquiry request that caching query arrives.When receiving module 310 receives again
When analysis personnel pass through the same queries request that the web platform that pre-establishes issues, directly acquire caching with the inquiry request
Corresponding inquiry data.
Based on the same inventive concept, the present invention also provides a kind of systems of online query data to look into online referring to Fig. 5
The system 500 for asking data includes web platform 510, the device 300 of online query data in any embodiment above and preset
Database 520, wherein web platform 510 is received by the inquiry request of analysis personnel's input, and inquiry request is sent to online
Inquire the device 300 of data；After the device 300 of online query data receives the inquiry request from web platform, which is asked
It asks progress syntax parsing to obtain corresponding syntax tree, rule searching corresponding with inquiry request is customized based on syntax tree.And then
The device 300 of line inquiry data inquires corresponding inquiry data according to rule searching from initialized data base 520, and will inquire
Inquiry data feedback to web platform 510.Wherein, specific online query data procedures may refer to foregoing embodiments, herein
It does not do and specifically repeats.
The present invention also provides a kind of computer storage medium is additionally provided, computer storage medium is stored with computer journey
Sequence code causes calculating equipment to execute in any embodiment above when computer program code is run on the computing device
The method of online query data.
In addition, the present invention also provides a kind of calculating equipment, including：Processor；It is stored with depositing for computer program code
Reservoir；When computer program code is run by processor, calculating equipment is caused to execute looking into any embodiment above online
The method for asking data.
According to the combination of any one above-mentioned preferred embodiment or multiple preferred embodiments, the embodiment of the present invention can reach
Following beneficial effect：
In embodiments of the present invention, when the inquiry request for receiving the web platform sending that analysis personnel utilization pre-establishes
Afterwards, syntax parsing first is carried out to the inquiry request received and obtains corresponding syntax tree, be then based on syntax tree customization and inquiry
Corresponding rule searching is requested, and then inquires corresponding inquiry data from initialized data base according to rule searching, will be inquired
The inquiry data feedback arrived is to web platform.The mode of the online query data of the embodiment of the present invention passes through to inquiry request as a result,
It carries out syntax parsing and obtains corresponding syntax tree, to support customized a variety of query grammars, that is, realize own inquiry language
Method, and then corresponding with inquiry request rule searching flexibly can be customized based on syntax tree, and according to rule searching from data
Data are inquired required for inquiring in library.Further, this programme also effectively improves the inquiry effect of online query data
Rate.
It is apparent to those skilled in the art that the specific work of the system of foregoing description, device and unit
Make process, can refer to corresponding processes in the foregoing method embodiment, for brevity, does not repeat separately herein.
In addition, each functional unit in each embodiment of the present invention can be physically independent, can also two or
More than two functional units integrate, and can be all integrated in a processing unit with all functional units.It is above-mentioned integrated
Functional unit both can take the form of hardware realization, can also be realized in the form of software or firmware.
Those of ordinary skill in the art will appreciate that：If the integrated functional unit is realized and is made in the form of software
It is independent product when selling or using, can store in a computer readable storage medium.Based on this understanding,
Technical solution of the present invention is substantially or all or part of the technical solution can be embodied in the form of software products,
The computer software product is stored in a storage medium comprising some instructions, with so that calculating equipment (such as
Personal computer, server or network equipment etc.) various embodiments of the present invention the method is executed when running described instruction
All or part of the steps.And storage medium above-mentioned includes：USB flash disk, mobile hard disk, read-only memory (ROM), random access memory
Device (RAM), the various media that can store program code such as magnetic or disk.
Alternatively, realizing that all or part of the steps of preceding method embodiment can be (all by the relevant hardware of program instruction
Such as personal computer, the calculating equipment of server or network equipment etc.) it completes, described program instruction can store in one
In computer-readable storage medium, when described program instruction is executed by the processor of calculating equipment, the calculating equipment is held
The all or part of the steps of row various embodiments of the present invention the method.
Finally it should be noted that：The above embodiments are only used to illustrate the technical solution of the present invention., rather than its limitations；To the greatest extent
Present invention has been described in detail with reference to the aforementioned embodiments for pipe, those skilled in the art should understand that：At this
Within the spirit and principle of invention, it is still possible to modify the technical solutions described in the foregoing embodiments or right
Some or all of the technical features are equivalently replaced；And these are modified or replaceed, and do not make corresponding technical solution de-
From protection scope of the present invention.
The embodiment of the invention provides A1, a kind of method of online query data, including：
It receives analysis personnel and passes through the inquiry request that the web platform pre-established issues；
Syntax parsing is carried out to the inquiry request received and obtains corresponding syntax tree, based on syntax tree customization and institute
State the corresponding rule searching of inquiry request；
Corresponding inquiry data are inquired from initialized data base according to the rule searching, the inquiry data inquired are anti-
It is fed to the web platform.
A2, method according to a1, wherein receive analysis personnel and pass through the inquiry that the web platform pre-established issues
Request, including：
Analysis personnel, which are received, based on http agreement passes through the http inquiry request that the web platform pre-established issues.
A3, method according to a1 or a2, wherein look into corresponding with the inquiry request is customized based on the syntax tree
Rule is ask, including：
The syntax tree is optimized using optimizer；
Corresponding query plan tree is generated according to the syntax tree after optimization, wherein includes multiple in the query plan tree
Tree node, the tree node correspond to the query steps of the rule searching.
A4, method according to a3, wherein inquire corresponding inquiry from initialized data base according to the rule searching
Data, by the inquiry data feedback inquired to the web platform, including：
According to the sequencing of the corresponding query steps of tree node in the query plan tree, successively from the preset data
Corresponding inquiry data are inquired in library；
After whole tree nodes inquires data according to the query plan tree, the inquiry number inquired is integrated
According to by the inquiry data feedback after integration to the web platform.
A5, method according to a3, wherein inquire corresponding inquiry from initialized data base according to the rule searching
Data, by the inquiry data feedback inquired to the web platform, including：
According to the sequencing of the corresponding query steps of tree node in the query plan tree, successively from the preset data
Corresponding inquiry data are inquired in library；
After inquiring corresponding inquiry data according to the corresponding query steps of any tree node, by current queries to look into
Data feedback is ask to the web platform.
A6, method according to a3, wherein inquire corresponding inquiry from initialized data base according to the rule searching
Data, by the inquiry data feedback inquired to the web platform, including：
Multiple tree nodes are chosen from the query plan tree；
From the initialized data base inquiry corresponding inquiry data parallel according to the multiple tree nodes chosen；
After whole tree nodes inquires data according to the query plan tree, the inquiry number inquired is integrated
According to by the inquiry data feedback after integration to the web platform.
A7, the method according to A6, wherein multiple tree nodes are chosen from the query plan tree, including：
Multiple tree nodes are chosen according to the sequencing of the corresponding query steps of tree node from the query plan tree；
And/or
Multiple tree nodes are randomly selected from the query plan tree.
A8, method according to a1 or a2, wherein further include：
According to the rule searching after inquiring corresponding inquiry data in initialized data base, caching query to look into
Ask data and corresponding inquiry request；
When receiving the same queries request that analysis personnel are issued by the web platform pre-established again, directly obtain
Take the inquiry data corresponding with the inquiry request of caching.
A9, method according to a1 or a2, wherein by the inquiry data feedback inquired to the web platform, packet
It includes：
The inquiry data inquired are formatted, the inquiry data of uniform format are obtained；
By the inquiry data feedback after uniform format to the web platform.
B10, a kind of device of online query data, including
Receiving module passes through the inquiry request that the web platform pre-established issues suitable for receiving analysis personnel；
Parsing module obtains corresponding syntax tree suitable for carrying out syntax parsing to the inquiry request received, based on described
Syntax tree customizes rule searching corresponding with the inquiry request；
Enquiry module is suitable for inquiring corresponding inquiry data from initialized data base according to the rule searching, will inquire
The inquiry data feedback arrived is to the web platform.
B11, device according to b10, wherein the receiving module is further adapted for：
Analysis personnel, which are received, based on http agreement passes through the http inquiry request that the web platform pre-established issues.
B12, device according to b10 or b11, wherein the parsing module is further adapted for：
The syntax tree is optimized using optimizer；
Corresponding query plan tree is generated according to the syntax tree after optimization, wherein includes multiple in the query plan tree
Tree node, the tree node correspond to the query steps of the rule searching.
B13, device according to b12, wherein the enquiry module is further adapted for：
According to the sequencing of the corresponding query steps of tree node in the query plan tree, successively from the preset data
Corresponding inquiry data are inquired in library；
After whole tree nodes inquires data according to the query plan tree, the inquiry number inquired is integrated
According to by the inquiry data feedback after integration to the web platform.
B14, device according to b12, wherein the enquiry module is further adapted for：
According to the sequencing of the corresponding query steps of tree node in the query plan tree, successively from the preset data
Corresponding inquiry data are inquired in library；
After inquiring corresponding inquiry data according to the corresponding query steps of any tree node, by current queries to look into
Data feedback is ask to the web platform.
B15, device according to b12, wherein the enquiry module is further adapted for：
Multiple tree nodes are chosen from the query plan tree；
From the initialized data base inquiry corresponding inquiry data parallel according to the multiple tree nodes chosen；
After whole tree nodes inquires data according to the query plan tree, the inquiry number inquired is integrated
According to by the inquiry data feedback after integration to the web platform.
B16, the device according to B15, wherein the enquiry module is further adapted for：
Multiple tree nodes are chosen according to the sequencing of the corresponding query steps of tree node from the query plan tree；
And/or
Multiple tree nodes are randomly selected from the query plan tree.
B17, device according to b10 or b11, wherein further include：
Cache module, suitable for according to the rule searching after inquiring corresponding inquiry data in initialized data base,
The inquiry data and corresponding inquiry request that caching query arrives；
It is asked when the receiving module receives analysis personnel again by the same queries that the web platform pre-established issues
When asking, the inquiry data corresponding with the inquiry request of caching are directly acquired.
B18, device according to b10 or b11, wherein the enquiry module is further adapted for：
The inquiry data inquired are formatted, the inquiry data of uniform format are obtained；
By the inquiry data feedback after uniform format to the web platform.
C19, a kind of system of online query data, including the described in any item online query numbers of web platform, B10-B18
According to device and initialized data base, wherein
The web platform is received by the inquiry request of analysis personnel's input, and by the inquiry request be sent to it is described
The device of line inquiry data；
The device of the online query data receives the inquiry request from the web platform, carries out to the inquiry request
Syntax parsing obtains corresponding syntax tree, customizes rule searching corresponding with the inquiry request based on the syntax tree；
The device of the online query data inquires corresponding inquiry number according to the rule searching from initialized data base
According to, and by the inquiry data feedback inquired to the web platform.
D20, a kind of computer storage medium, the computer storage medium is stored with computer program code, when described
When computer program code is run on the computing device, causing the calculating equipment to execute, A1-A9 is described in any item to be looked into online
The method for asking data.
E21, a kind of calculating equipment, including：Processor；It is stored with the memory of computer program code；When the calculating
When machine program code is run by the processor, the calculating equipment is caused to execute the described in any item online query numbers of A1-A9
According to method.
Claims (10)
1. a kind of method of online query data, including：
It receives analysis personnel and passes through the inquiry request that the web platform pre-established issues；
Syntax parsing is carried out to the inquiry request received and obtains corresponding syntax tree, is customized based on the syntax tree and is looked into described
It askes and requests corresponding rule searching；
Corresponding inquiry data are inquired from initialized data base according to the rule searching, extremely by the inquiry data feedback inquired
The web platform.
2. according to the method described in claim 1, wherein, receiving analysis personnel and passing through looking into for the web platform sending pre-established
Request is ask, including：
Analysis personnel, which are received, based on http agreement passes through the http inquiry request that the web platform pre-established issues.
3. method according to claim 1 or 2, wherein customized based on the syntax tree corresponding with the inquiry request
Rule searching, including：
The syntax tree is optimized using optimizer；
Corresponding query plan tree is generated according to the syntax tree after optimization, wherein includes multiple burls in the query plan tree
Point, the tree node correspond to the query steps of the rule searching.
4. according to the method described in claim 3, wherein, being inquired from initialized data base according to the rule searching and being looked into accordingly
Data are ask, by the inquiry data feedback inquired to the web platform, including：
According to the sequencing of the corresponding query steps of tree node in the query plan tree, successively from the initialized data base
The corresponding inquiry data of inquiry；
After whole tree nodes inquires data according to the query plan tree, the inquiry data inquired are integrated, it will
Inquiry data feedback after integration is to the web platform.
5. according to the method described in claim 3, wherein, being inquired from initialized data base according to the rule searching and being looked into accordingly
Data are ask, by the inquiry data feedback inquired to the web platform, including：
According to the sequencing of the corresponding query steps of tree node in the query plan tree, successively from the initialized data base
The corresponding inquiry data of inquiry；
After inquiring corresponding inquiry data according to the corresponding query steps of any tree node, inquiry number that current queries are arrived
According to feeding back to the web platform.
6. according to the method described in claim 3, wherein, being inquired from initialized data base according to the rule searching and being looked into accordingly
Data are ask, by the inquiry data feedback inquired to the web platform, including：
Multiple tree nodes are chosen from the query plan tree；
From the initialized data base inquiry corresponding inquiry data parallel according to the multiple tree nodes chosen；
After whole tree nodes inquires data according to the query plan tree, the inquiry data inquired are integrated, it will
Inquiry data feedback after integration is to the web platform.
7. a kind of device of online query data, including
Receiving module passes through the inquiry request that the web platform pre-established issues suitable for receiving analysis personnel；
Parsing module obtains corresponding syntax tree suitable for carrying out syntax parsing to the inquiry request received, is based on the grammer
Tree customizes rule searching corresponding with the inquiry request；
Enquiry module is suitable for inquiring corresponding inquiry data from initialized data base according to the rule searching, by what is inquired
Data feedback is inquired to the web platform.
8. a kind of system of online query data, the device including web platform, online query data as claimed in claim 7 with
And initialized data base, wherein
The web platform receives by the inquiry request of analysis personnel's input, and the inquiry request is sent to and described is looked into online
Ask the device of data；
The device of the online query data receives the inquiry request from the web platform, carries out grammer to the inquiry request
Parsing obtains corresponding syntax tree, customizes rule searching corresponding with the inquiry request based on the syntax tree；
The device of the online query data inquires corresponding inquiry data according to the rule searching from initialized data base,
And by the inquiry data feedback inquired to the web platform.
9. a kind of computer storage medium, the computer storage medium is stored with computer program code, when the computer
When program code is run on the computing device, causing the calculating equipment perform claim to require, 1-6 is described in any item to be looked into online
The method for asking data.
10. a kind of calculating equipment, including：Processor；It is stored with the memory of computer program code；When the computer program
When code is run by the processor, the calculating equipment perform claim is caused to require the described in any item online query numbers of 1-6
According to method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810688094.4A CN108920614B (en) | 2018-06-28 | 2018-06-28 | Method, device and system for inquiring data online |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810688094.4A CN108920614B (en) | 2018-06-28 | 2018-06-28 | Method, device and system for inquiring data online |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108920614A true CN108920614A (en) | 2018-11-30 |
| CN108920614B CN108920614B (en) | 2021-08-20 |
Family
ID=64421943
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810688094.4A Expired - Fee Related CN108920614B (en) | 2018-06-28 | 2018-06-28 | Method, device and system for inquiring data online |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108920614B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710641A (en) * | 2018-12-17 | 2019-05-03 | 浩云科技股份有限公司 | A kind of inquiry processing method and system of mass data |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2010050675A2 (en) * | 2008-10-29 | 2010-05-06 | 한국과학기술원 | Method for automatically extracting relation triplets through a dependency grammar parse tree |
| CN103761080A (en) * | 2013-12-25 | 2014-04-30 | 中国农业大学 | Structured query language (SQL) based MapReduce operation generating method and system |
| US20170168785A1 (en) * | 2014-09-06 | 2017-06-15 | Aquameta LLC | Computer programming system and method |
| CN107832391A (en) * | 2017-10-31 | 2018-03-23 | 长城计算机软件与***有限公司 | A kind of data query method and system |
| CN107943952A (en) * | 2017-11-24 | 2018-04-20 | 北京赛思信安技术股份有限公司 | A kind of implementation method that full-text search is carried out based on Spark frames |
-
2018
- 2018-06-28 CN CN201810688094.4A patent/CN108920614B/en not_active Expired - Fee Related
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2010050675A2 (en) * | 2008-10-29 | 2010-05-06 | 한국과학기술원 | Method for automatically extracting relation triplets through a dependency grammar parse tree |
| CN103761080A (en) * | 2013-12-25 | 2014-04-30 | 中国农业大学 | Structured query language (SQL) based MapReduce operation generating method and system |
| US20170168785A1 (en) * | 2014-09-06 | 2017-06-15 | Aquameta LLC | Computer programming system and method |
| CN107832391A (en) * | 2017-10-31 | 2018-03-23 | 长城计算机软件与***有限公司 | A kind of data query method and system |
| CN107943952A (en) * | 2017-11-24 | 2018-04-20 | 北京赛思信安技术股份有限公司 | A kind of implementation method that full-text search is carried out based on Spark frames |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710641A (en) * | 2018-12-17 | 2019-05-03 | 浩云科技股份有限公司 | A kind of inquiry processing method and system of mass data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108920614B (en) | 2021-08-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9361344B2 (en) | System and method for distributed database query engines | |
| US10430480B2 (en) | Enterprise data processing | |
| US10839038B2 (en) | Generating configuration information for obtaining web resources | |
| CN110431545A (en) | Inquiry is executed for structural data and unstructured data | |
| Lith et al. | Investigating storage solutions for large data-A comparison of well performing and scalable data storage solutions for real time extraction and batch insertion of data | |
| CN106294757B (en) | A kind of distributed data base and its clustered partition method divided based on hypergraph | |
| CN106776783B (en) | Unstructured data memory management method and system | |
| CN103248645A (en) | BT (Bit Torrent) off-line data downloading system and method | |
| WO2020087082A1 (en) | Trace and span sampling and analysis for instrumented software | |
| GB2511935A (en) | Low Latency query engine for apache hadoop | |
| CN112732466B (en) | Service calling method, device and system | |
| CN108985981A (en) | Data processing system and method | |
| Van Herwegen et al. | Query execution optimization for clients of triple pattern fragments | |
| CN111858760B (en) | Data processing method and device for heterogeneous database | |
| CN102054000A (en) | Data querying method, device and system | |
| CN110390739A (en) | A kind of vehicle data processing method and vehicle data processing system | |
| CN103034650B (en) | A kind of data handling system and method | |
| Um et al. | Distributed RDF store for efficient searching billions of triples based on Hadoop | |
| US20170293626A1 (en) | Managing persistent database result sets | |
| CN108920614A (en) | The method, apparatus and system of online query data | |
| CN108804502A (en) | Big data inquiry system, method, computer equipment and storage medium | |
| CN106599244B (en) | General original log cleaning device and method | |
| CN111786801B (en) | Method and device for charging based on data flow | |
| CN113779017A (en) | Method and apparatus for data asset management | |
| CN111078975A (en) | Multi-node incremental data acquisition system and acquisition method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210820 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |