CN105912587A - Data acquisition method and system - Google Patents
Data acquisition method and system Download PDFInfo
- Publication number
- CN105912587A CN105912587A CN201610202878.2A CN201610202878A CN105912587A CN 105912587 A CN105912587 A CN 105912587A CN 201610202878 A CN201610202878 A CN 201610202878A CN 105912587 A CN105912587 A CN 105912587A
- Authority
- CN
- China
- Prior art keywords
- data acquisition
- data
- log information
- information
- business
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L9/00—Cryptographic mechanisms or cryptographic arrangements for secret or secure communications; Network security protocols
- H04L9/40—Network security protocols
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Debugging And Monitoring (AREA)
Abstract
The embodiment of the application discloses a data acquisition method and system. The method concretely comprises following steps: adopting a distributed storage mode to store log information of at least one business; setting a data acquisition task rule based on a corresponding business requirement, wherein the data acquisition task rule corresponds to at least one feature information; acquiring target data from the stored log information based on the data acquisition task rule; and storing a data acquisition result of target data. The embodiment of the invention can enrich a data center platform while providing user-friendly services based on data acquisition results. Therefore, the business platform can be improved.
Description
Technical field
It relates to data processing field, particularly relate to a kind of collecting method and system.
Background technology
Along with the development of modern science and technology, data acquisition technology has penetrated into all trades and professions and various
In technical field.
Daily record is the thing being log that the network equipment, system and service routine etc. operationally produce
Part record；Every a line daily record all recites retouching of the associative operations such as date, time, user and action
State information.The log recording life cycle of system, by consulting daily record, it can be realized that system exists
State in which sometime；By the analysis to daily record, collect useful data, can be used
The use information at family and acess control, provide for the optimization of service system and network security problem prevention etc.
Foundation.
But existing collecting method only can carry out structuring from data base and network file
The collection of data, but have ignored destructuring in the log information for containing a large number of users behavioral data
The collection of data, the data about user behavior therefore gathered are the abundantest.
Summary of the invention
Disclosure embodiment provides a kind of collecting method and system, in order to solve available data collection side
The problem that the data about user behavior of method collection are enriched not, it is possible to abundant data center platform, with
Time can provide the user more humane service according to data acquisition results, and business of more improving is put down
Platform.
Disclosure embodiment provides a kind of collecting method, including:
Distributed storage mode is used to store the log information of at least one business；
According to corresponding service demand configuration data acquisition session rule；Wherein, described data acquisition session rule
Then at least one characteristic information corresponding；
From the described log information of described storage, number of targets is gathered according to described data acquisition session rule
According to；
Store described target data collection result.
Disclosure embodiment provides a kind of data collecting system, including: log information memory module, data
Acquisition tasks configuration module, log data acquisition module, and target data memory module；
Wherein, described log information memory module, it is used for using distributed storage mode to store at least one
The log information of business；
Described data acquisition session configuration module, for configuring data acquisition session according to corresponding service demand
Rule；Wherein, described data acquisition session rule at least one characteristic information corresponding；
Described log data acquisition module, for believing from described daily record according to described data acquisition session rule
Breath memory module gathers target data in the described log information of storage；And
Described target data memory module, for storing the described mesh of described log data acquisition module output
Mark data.
Disclosure embodiment provides a kind of collecting method and system, on the one hand, use distributed storage
Mode stores log information, can effectively reduce the performance requirement to unit CPU and resource, reduce
The cost of data acquisition；On the other hand, disclosure embodiment can be by resolving log information, thus root
Data acquisition session rule according to configuration analytically carries out the collection of target data in result, due to substantial amounts of
Log information contains substantial amounts of user data, user operation behavioral data and business datum, therefore adopts
Collection result not only enriches data center's platform, simultaneously so that according to data acquisition results to user's row
Effectively analyze, to know more about the demand of user such that it is able to for user for custom and business operation
More humane service is provided, and more improves business platform.
Accompanying drawing explanation
In order to be illustrated more clearly that disclosure embodiment or technical scheme of the prior art, below will be to reality
Execute the required accompanying drawing used in example or description of the prior art to be briefly described, it should be apparent that under,
Accompanying drawing in the description of face is some embodiments of the disclosure, for those of ordinary skill in the art,
On the premise of not paying creative work, it is also possible to obtain other accompanying drawing according to these accompanying drawings.
Fig. 1 is the flow chart of steps of a kind of collecting method embodiment one of the disclosure；
Fig. 2 is a kind of User Activity collection of illustrative plates schematic diagram of the disclosure；
Fig. 3 is the flow chart of steps of a kind of collecting method embodiment two of the disclosure；
Fig. 4 is the structural representation of a kind of data collecting system embodiment one of the disclosure；
Fig. 5 is the structural representation of a kind of data collecting system embodiment two of the disclosure；And
Fig. 6 is the structural representation of a kind of data collecting system embodiment three of the disclosure.
Detailed description of the invention
For making the purpose of disclosure embodiment, technical scheme and advantage clearer, below in conjunction with these public affairs
Open the accompanying drawing in embodiment, the technical scheme in disclosure embodiment be clearly and completely described,
Obviously, described embodiment is a part of embodiment of the disclosure rather than whole embodiments.Based on
Embodiment in the disclosure, those of ordinary skill in the art are obtained under not making creative work premise
The every other embodiment obtained, broadly falls into the scope of disclosure protection.
Embodiment of the method one
With reference to Fig. 1, it is shown that the flow chart of steps of a kind of collecting method embodiment one of the disclosure,
Specifically may include that
Step 101, employing distributed storage mode store the log information of at least one business；
In disclosure embodiment, the log information of business can use distributed storage mode to store,
Also will be evenly distributed on multiple data server store, to these data servers by journal file
Resource carry out unified management and distribution, and provide a user with file system access interface, use distribution
The resource that the substantial amounts of log information file that the mode of formula storage can effectively solve takies is too much, and it is right to cause
The problem that unit storage many requirements such as resource and cpu performance are higher, can effectively solve daily record letter
Breath file size, log information quantity of documents, open the restricted problem of log information number of files etc..
In disclosure embodiment, log information specifically can include in disclosure embodiment, and log information has
Body can include user behavior custom and business information data, such as, includes from transcoding log information
When starting download time, context initialization time, film source detection time, key frame sweep time, section
Between, the file transcoding time, Audio Processing, captions process the data such as time, finished product detection time；Again
As: in the media file log information (MediaInfo file) being stored in distributed file system, tool
Body may include that code check, frame per second, size, video duration, audio frequency duration, video format, audio frequency lattice
The data messages such as formula, channel number, video code rate, audio code rate.
It is appreciated that above-mentioned transcoding log information and media file log information are only used as the disclosure and implement
A kind of example of the log information of example, and it is not understood to the restriction of log information in disclosure embodiment, real
On border, log information specifically can include the user's correlation log information produced by user operation, Yi Jiye
The log information that the business produced in business processing procedure is correlated with, log information is not done by disclosure embodiment to be had
Body limits.
In disclosure embodiment, log information is carried out the process of distributed storage and distributed deposits with reference to existing
The process of storage fileinfo, this is not specifically limited by disclosure embodiment.
Step 102, according to corresponding service demand configuration data acquisition session rule；Wherein, described data
Acquisition tasks rule at least one characteristic information corresponding；
In disclosure embodiment, above-mentioned data acquisition session rule can pre-establish according to demand, permissible
Target data and target data characteristic of correspondence information according to pre-acquired formulate corresponding data acquisition session
Rule, owing to the target data of pre-acquired may relate to one or more characteristic information, the most accordingly
Data acquisition session rule one or more characteristic information corresponding, such as: target data is nearest one
Within Yue, Shanghai node video uploads successful video total amount, and the characteristic information being directed to specifically wraps
Including: regional feature information is: Shanghai, time range characteristic information is: nearest one month (such as:
2015/10/01 to 2015/11/01), video is uploaded state characteristic information and is: the most totally three features letters
Breath, then data acquisition session rule accordingly to should three characteristic informations, corresponding data acquisition session
Rule can be thought: statistically characteristic of field information be Shanghai and video uplink time 2015/10/01 to
Between 2015/11/01 and video uploads the quantity that state is successful video；
In disclosure embodiment, user can carry out data acquisition session rule on User Interface
Configuration, such as: within service needed adds up nearest one month, Shanghai node video uploads successful video
Total amount, namely within target data is nearest one month, it is total that Shanghai node video uploads successful video
Amount, then configuring above-mentioned data acquisition session rule on User Interface for statistically characteristic of field information is
Shanghai and video uplink time are between 2015/10/01 to 2015/11/01 and video uploads state for becoming
The quantity of the video of merit, accordingly, can be with configuration feature information: regional feature information is: Shanghai, time
Between scoped features information be: nearest one month (such as: 2015/10/01 to 2015/11/01), on video
Biography state characteristic information is: success.
It is appreciated that the configuration carrying out data acquisition session regular above by User Interface is only used as
Disclosure embodiment configures a kind of mode of data acquisition session rule, and being not understood to is to the disclosure
Embodiment configures a kind of restriction of data acquisition session rule, actually can also be by people in the art
Member directly writes configuration file according to business demand, thus realizes the configuration of data acquisition session rule；This
The configuration mode of above-mentioned data acquisition session rule is not specifically limited by open embodiment at this.
Step 103, according to described data acquisition session rule adopt from the described log information of described storage
Collection target data；
In a kind of alternative embodiment of the disclosure, described regular from described according to described data acquisition session
The described log information of storage gathers the step 103 of target data, specifically may include that
Step A1, from the log information of described storage, obtain the target journaling information of corresponding business；
Step A2, described target journaling information is resolved, to advise according to described data acquisition session
Result the most analytically gathers described target data.
Owing to, in disclosure embodiment, log information storage server can store the daily record of multiple business
Information, such as: upload task, downloading task, store tasks and transcoding task etc., accordingly, it would be desirable to
The target journaling information of corresponding business is extracted, such as: type of service is for uploading, then according to class of business
Log information storage server uploads log information corresponding to task and is target journaling information.
In disclosure embodiment, can carry by carrying out log information resolving obtaining in log information
Data, such as: of acquisition corresponding to uploading the log information of business is: 2015-10-20
10:00:30user=1001upload a file IP=10.80.25.32success, then resolve this log information and obtain
To analysis result be: upload the date: 2015-10-20；Uplink time: 10:00:30；User:
1001；IP:10.80.25.32；Upload state: success；And according to data acquisition session rule from solution
Analysis result gathers described target data, namely analysis result is carried out point according to data task collection rule
Analysis, to gather target data, such as: data acquisition session rule is upper for statistically characteristic of field information
Sea and video upload the date between 2015/10/01 to 2015/11/01 and video to upload state be successfully
The quantity of video, then the analysis result of the log information obtained can be carried out traversal and analyze, statistics is full
Foot correspondence Shanghai, IP address, and upload the date between 2015/10/01 to 2015/11/01, and upload state
For the quantity of the log information of success, it is the collection result of final goal data.
It is appreciated that above-mentioned data acquisition session rule is Shanghai and video for statistically characteristic of field information
Between 2015/10/01 to 2015/11/01 and video uploads the number that state is successful video to upload the date
Amount is only used as a kind of example of data acquisition session rule in disclosure embodiment, and being not understood to is to this
A kind of restriction of data acquisition session rule in open embodiment, it practice, data acquisition session rule can
To be set according to business demand by those skilled in the art, such as: service needed counting user A is in the time
Upload the number of videos of failure in section B, then corresponding data acquisition session rule is: counting user feature is believed
Breath is user A, and time range characteristic information is time period B, and state of uploading is failed daily record letter
Breath sum；Data acquisition session rule is not defined by disclosure embodiment at this.
Step 104, store described target data collection result.
In a kind of alternative embodiment of the disclosure, disclosure embodiment specifically can also include:
Receive the log information that at least one business is uploaded；And/or,
Log information is read from least one business described.
That is, disclosure embodiment can by the business diary information storage medium of access service, and from
Business diary information storage medium reads described log information；Described business can also be received and pass through API
The described log information that interface is uploaded, disclosure embodiment at this acquisition mode for log information do not do
Concrete restriction.
Disclosure embodiment can carry out from log information the collection of data, concrete in its collection result
User, content, production process and the data of research and development index can be contained, and then above-mentioned data are carried out
Analyzing, can obtain the finest data analysis, as obtained User Activity atlas analysis, user is fine
Change operation etc., such as: with reference to Fig. 2, it is shown that in disclosure embodiment, a kind of User Activity collection of illustrative plates shows
Being intended to, its basis data to gathering are that a certain user uses all days produced by a certain application program
Will information carries out data acquisition and analysis obtains.
To sum up, a kind of collecting method provided in disclosure embodiment, on the one hand, use distributed
Storage mode storage log information, can effectively reduce the performance requirement to unit CPU and resource, fall
The low cost of data acquisition；On the other hand, disclosure embodiment can by resolve log information, from
And carry out the collection of target data according to the data acquisition session rule of configuration, due to substantial amounts of log information
In contain substantial amounts of user data, user operation behavioral data and business datum, therefore collection result is not
But enrich data center's platform, simultaneously so that according to data acquisition results to user behavior custom and
Business operation is effectively analyzed, to know more about the demand of user such that it is able to provide the user more people
The service of property, and more improve business platform.
Embodiment of the method two
With reference to Fig. 3, it is shown that the flow chart of steps of the disclosure a kind of collecting method embodiment two, tool
Body may include that
Step 301, employing distributed storage mode store the log information of at least one business；
Step 302, according to corresponding service demand configuration data acquisition session rule；Described data acquisition is appointed
Business rule at least one characteristic information corresponding；Wherein, described data acquisition session also includes: data acquisition
Collection interface message；
Step 303, according to described data acquisition session rule adopt from the described log information of described storage
Collection target data；
Step 304, according to described data acquisition session rule from described data base gather target data；
And/or
From described text, target data is gathered according to described data acquisition session rule；
Step 305, store described target data collection result.
Relative to embodiment of the method one, disclosure embodiment adds step 304, in this step 304
The collection of target data, namely disclosure embodiment can be carried out from data base, and/or text
In can carry out the collection of data, data acquisition results more horn of plenty based on multiple data sources.
In disclosure embodiment, above-mentioned data acquisition interface information specifically may include that the name of data source
Title, data storage method and data memory format, wherein, above-mentioned data storage method specifically can wrap
Include: the storage class of data and storage position, storage class specifically may include that type of database, example
As: My sql, Oracle etc., text type, such as: txt, syslog etc., log information, such as: web
Daily record, operating system daily record etc.；For the data of type of database, concrete in data acquisition interface information
Also need to indicate the host IP address at data base place, database-name, user name, password；For literary composition
The data of this type, specifically also need to the store path of specified document in data acquisition interface information；Data
Storage format refers to the form of data itself, and for database data, above-mentioned database purchase form includes
Data table name to be read, field name, major key information；For text type, above-mentioned database purchase lattice
Formula mainly includes the title of file, and keyword message.
Below by way of concrete example in disclosure embodiment, from data base, gather data be illustrated:
Such as, service needed gathers film source from Production database and uploads data, i.e. data acquisition interface letter
Breath specifically may include that data source: Production database, Stored Data Type: My sql, data are deposited
Storage space is put: upload the information such as task list, to gather film source in task list from uploading of above-mentioned Production database
Uploading data, described data specifically may include that file size, file name, upload user, upload
Client ip, start uplink time, upload deadline, memory node etc..
Disclosure embodiment can determine the number of Current data acquisition task according to data acquisition interface information
It is data base according to source, or text, or log information.
In a kind of alternative embodiment of the disclosure, said method specifically can also include:
Distributed storage mode is used to store the database data information of at least one business；And/or
Distributed storage mode is used to store the text file information of at least one business.
That is, in disclosure embodiment, in data base, data message and the text of storage can also
Use distributed storage mode to store, enable to reduce the cpu to unit and the performance of resource
Requirement, and then save data acquisition cost.
It should be noted that for embodiment of the method, in order to be briefly described, therefore it is all expressed as one it be
The combination of actions of row, but those skilled in the art should know, and the embodiment of the present application is not by described
The restriction of sequence of movement because according to the embodiment of the present application, some step can use other orders or
Person is carried out simultaneously.Secondly, those skilled in the art also should know, embodiment described in this description
Belong to preferred embodiment, necessary to involved action not necessarily the embodiment of the present application.
Device embodiment one
With reference to Fig. 4, it is shown that the structural representation of a kind of data collecting system embodiment one of the disclosure,
Specifically may include that log information memory module 401, data acquisition session configuration module 402, daily record
Data acquisition module 403, and target data memory module 404；
Wherein, described log information memory module 401, may be used for using distributed storage mode to store
The log information of at least one business；
Described data acquisition session configuration module 402, may be used for configuring data according to corresponding service demand
Acquisition tasks rule；Wherein, described data acquisition session rule at least one characteristic information corresponding；
Described log data acquisition module 403, may be used for according to described data acquisition session rule from institute
State and log information memory module 401 gathers in the described log information of storage target data；And
Described target data memory module 304, may be used for storing the output of described log data acquisition module
Described target data.
Device embodiment two
With reference to Fig. 5, it is shown that the structural representation of a kind of data collecting system embodiment one of the disclosure,
Specifically may include that log information memory module 501, data acquisition session configuration module 502, daily record
Data acquisition module 503, and target data memory module 504；
Wherein, described log information memory module 501, may be used for using distributed storage mode to store
The log information of at least one business；
Described data acquisition session configuration module 502, may be used for configuring data according to corresponding service demand
Acquisition tasks rule；Wherein, described data acquisition session rule at least one characteristic information corresponding；
Described log data acquisition module 503, may be used for according to described data acquisition session rule from institute
State and log information memory module 501 gathers in the described log information of storage target data；And
Described target data memory module 504, may be used for storing the output of described log data acquisition module
Described target data.
Wherein, above-mentioned log data acquisition module 503, specifically may include that log information obtains submodule
Block 4031 and log information analyzing sub-module 5032；Wherein,
Described log information obtains submodule 5031, may be used for obtaining from described log information memory module
Take the target journaling information of corresponding business；
Described log information analyzing sub-module 5032, may be used for that described log information is obtained submodule and obtains
The described target journaling information taken resolves, with according to described data acquisition session rule analytically result
The described target data of middle collection.
Device embodiment three
With reference to Fig. 6, it is shown that the structural representation of a kind of data collecting system embodiment three of the disclosure,
Specifically may include that log information memory module 601, data acquisition session configure module 602, daily record
Data acquisition module 603, database data acquisition module 604, text data acquisition module 605 and
Target data memory module 606,
Wherein, described log information memory module 601, may be used for using distributed storage mode to store
The log information of at least one business；
Described data acquisition session configuration module 602, may be used for configuring data according to corresponding service demand
Acquisition tasks rule；Wherein, described data acquisition session rule at least one characteristic information corresponding；
Described log data acquisition module 603, may be used for according to described data acquisition session rule from institute
State and log information memory module 601 gathers in the described log information of storage target data；
Described target data memory module 606, may be used for storing the output of described log data acquisition module
Described target data；
Described database data acquisition module 604, may be used for according to described data acquisition session rule from
Described data base gathers target data；And
Described text data acquisition module 605, may be used for according to described data acquisition session rule
Target data is gathered from described text；
The most described data memory module 606, it is also possible to be used for storing described database data acquisition module
604, and the described target data of described text data acquisition module 605 output.
In a kind of alternative embodiment of the disclosure, disclosure embodiment specifically can also include:
Database data memory module, may be used for using distributed storage mode to store at least one business
Database data information；And/or
Text data memory module, may be used for using distributed storage mode to store at least one industry
The text file information of business.
For device embodiment, due to itself and embodiment of the method basic simlarity, so the comparison described
Simply, relevant part sees the part of embodiment of the method and illustrates.
Device embodiment described above is only schematically, wherein said illustrates as separating component
Unit can be or may not be physically separate, the parts shown as unit can be or
Person may not be physical location, i.e. may be located at a place, or can also be distributed to multiple network
On unit.Some or all of module therein can be selected according to the actual needs to realize the present embodiment
The purpose of scheme.Those of ordinary skill in the art are not in the case of paying performing creative labour, the most permissible
Understand and implement.
Through the above description of the embodiments, those skilled in the art is it can be understood that arrive each reality
The mode of executing can add the mode of required general hardware platform by software and realize, naturally it is also possible to by firmly
Part.Based on such understanding, the portion that prior art is contributed by technique scheme the most in other words
Dividing and can embody with the form of software product, this computer software product can be stored in computer can
Read in storage medium, such as ROM/RAM, magnetic disc, CD etc., including some instructions with so that one
Computer equipment (can be personal computer, server, or the network equipment etc.) performs each to be implemented
The method described in some part of example or embodiment.
Last it is noted that above example is only in order to illustrate the technical scheme of the disclosure, rather than to it
Limit；Although the disclosure being described in detail with reference to previous embodiment, the ordinary skill of this area
Personnel it is understood that the technical scheme described in foregoing embodiments still can be modified by it, or
Person carries out equivalent to wherein portion of techniques feature；And these amendments or replacement, do not make corresponding skill
The essence of art scheme departs from the spirit and scope of the disclosure each embodiment technical scheme.
Claims (10)
1. a collecting method, described method includes:
Distributed storage mode is used to store the log information of at least one business；
According to corresponding service demand configuration data acquisition session rule；Wherein, described data acquisition session rule
Then at least one characteristic information corresponding；
From the described log information of described storage, number of targets is gathered according to described data acquisition session rule
According to；
Store described target data collection result.
Method the most according to claim 1, wherein, described advises according to described data acquisition session
From the described log information of described storage, then gather target data include:
The target journaling information of corresponding business is obtained from the log information of described storage；
Described target journaling information is resolved, analytically to tie according to described data acquisition session rule
Described target data is gathered in Guo.
Method the most according to claim 1, wherein, also includes in described data acquisition session:
Data acquisition interface information；The most described method also includes:
From described data base, target data is gathered according to described data acquisition session rule；And/or
From described text, target data is gathered according to described data acquisition session rule.
Method the most according to claim 3, wherein, described method also includes:
Distributed storage mode is used to store the database data information of at least one business；And/or
Distributed storage mode is used to store the text file information of at least one business.
Method the most according to claim 1, wherein, described method also includes:
Receive the log information that at least one business is uploaded；And/or,
Log information is read from least one business described.
6. a data collecting system, wherein, including: log information memory module, data acquisition is appointed
Business configuration module, log data acquisition module, and target data memory module；
Wherein, described log information memory module, it is used for using distributed storage mode to store at least one
The log information of business；
Described data acquisition session configuration module, for configuring data acquisition session according to corresponding service demand
Rule；Wherein, described data acquisition session rule at least one characteristic information corresponding；
Described log data acquisition module, for believing from described daily record according to described data acquisition session rule
Breath memory module gathers target data in the described log information of storage；And
Described target data memory module, for storing the described mesh of described log data acquisition module output
Mark data.
System the most according to claim 6, wherein, described log data acquisition module, bag
Include: log information obtains submodule and log information analyzing sub-module；Wherein,
Described log information obtains submodule, for obtaining corresponding industry from described log information memory module
The target journaling information of business；
Described log information analyzing sub-module, for obtaining described in submodule acquisition described log information
Target journaling information resolves, with according to gathering institute in described data acquisition session rule analytically result
State target data.
System the most according to claim 6, wherein, also includes in described data acquisition session:
Data acquisition interface information；The most described system also includes: database data acquisition module, and/or text
Data collector file module；Wherein,
Described database data acquisition module, for regular from described data according to described data acquisition session
Storehouse gathers target data；And/or
Described text data acquisition module, for regular from described literary composition according to described data acquisition session
Presents gathers target data；
The most described data memory module, is additionally operable to store described database data acquisition module, and/or institute
State the described target data of text data acquisition module output.
System the most according to claim 8, wherein, described system also includes:
Database data memory module, for using distributed storage mode to store the number of at least one business
According to database data information；And/or
Text data memory module, for using distributed storage mode to store at least one business
Text file information.
System the most according to claim 6, wherein, described device also includes:
Receive log information module, for receiving the log information that at least one business is uploaded；And/or,
Read log information module, for reading log information from least one business described.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610202878.2A CN105912587A (en) | 2016-03-31 | 2016-03-31 | Data acquisition method and system |
| PCT/CN2016/096968 WO2017166644A1 (en) | 2016-03-31 | 2016-08-26 | Data acquisition method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610202878.2A CN105912587A (en) | 2016-03-31 | 2016-03-31 | Data acquisition method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN105912587A true CN105912587A (en) | 2016-08-31 |
Family
ID=56745348
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610202878.2A Pending CN105912587A (en) | 2016-03-31 | 2016-03-31 | Data acquisition method and system |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN105912587A (en) |
| WO (1) | WO2017166644A1 (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109271531A (en) * | 2018-11-16 | 2019-01-25 | 苏州友教习亦教育科技有限公司 | Control data corporation based on O&M knowledge mapping |
| CN109327351A (en) * | 2018-09-12 | 2019-02-12 | 拉扎斯网络科技（上海）有限公司 | Real-time collecting method, device, electronic equipment and the storage medium of daily record data |
| CN109918048A (en) * | 2018-12-27 | 2019-06-21 | 北京奇艺世纪科技有限公司 | Target object extracting method, device, system and computer readable storage medium |
| CN110932918A (en) * | 2019-12-26 | 2020-03-27 | 远景智能国际私人投资有限公司 | Log data acquisition method and device and storage medium |
| CN113126562A (en) * | 2020-01-16 | 2021-07-16 | 智能云科信息科技有限公司 | Data acquisition method, device and system and computer readable storage medium |
Families Citing this family (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109710605B (en) * | 2017-10-25 | 2022-12-16 | 卓望数码技术（深圳）有限公司 | Automatic equipment information acquisition device and method |
| CN109634929A (en) * | 2018-09-30 | 2019-04-16 | 阿里巴巴集团控股有限公司 | Acquisition method, device and the server of business datum |
| CN110968561B (en) * | 2018-09-30 | 2024-02-13 | 北京国双科技有限公司 | Log storage method and distributed system |
| CN110413496B (en) * | 2019-07-29 | 2022-08-19 | 福建南威软件有限公司 | Method for realizing componentized collection of electronic license operation data |
| CN110502514B (en) * | 2019-08-15 | 2023-06-27 | 中国平安财产保险股份有限公司 | Data acquisition method, device, equipment and computer readable storage medium |
| CN110781248A (en) * | 2019-09-27 | 2020-02-11 | 浙江省北大信息技术高等研究院 | Multi-source heterogeneous data acquisition method and device |
| CN113377848A (en) * | 2020-02-25 | 2021-09-10 | 北京数聚鑫云信息技术有限公司 | Data processing method, device, equipment and storage medium |
| CN111343190A (en) * | 2020-03-05 | 2020-06-26 | 贵州宝智达网络科技有限公司 | Remote wireless data tamper-proof acquisition equipment and system |
| CN111352903A (en) * | 2020-03-13 | 2020-06-30 | 京东方科技集团股份有限公司 | Log management platform, log management method, medium, and electronic device |
| CN111611207B (en) * | 2020-05-21 | 2023-06-23 | 四川虹美智能科技有限公司 | State data processing method and device and computer equipment |
| CN112347180B (en) * | 2020-12-04 | 2023-08-01 | 航天信息股份有限公司企业服务分公司 | Data pushing method and electronic equipment |
| CN112667728B (en) * | 2021-01-06 | 2023-11-21 | 上海振华重工(集团)股份有限公司 | Visual single machine data acquisition method in wharf efficiency analysis |
| CN113010240B (en) * | 2021-03-29 | 2024-02-02 | 北京金山云网络技术有限公司 | Data acquisition method, system, electronic equipment and storage medium |
| CN112948504B (en) * | 2021-03-30 | 2022-12-02 | 苏宁易购集团股份有限公司 | Data acquisition method and device, computer equipment and storage medium |
| CN113791946A (en) * | 2021-08-31 | 2021-12-14 | 北京达佳互联信息技术有限公司 | Log processing method and device, electronic equipment and storage medium |
| CN114328076B (en) * | 2021-09-18 | 2024-04-30 | 腾讯科技（深圳）有限公司 | Log information extraction method, device, computer equipment and storage medium |
| CN114168509A (en) * | 2021-10-22 | 2022-03-11 | 中科苏州微电子产业技术研究院 | Expansion control method and system of data acquisition chip |
| CN114189367A (en) * | 2021-11-30 | 2022-03-15 | 南京理工大学 | Safety log analysis system based on knowledge graph |
| CN114461490B (en) * | 2021-12-31 | 2023-05-30 | 广东航宇卫星科技有限公司 | Fortune dimension aggregation system |
| CN114720761A (en) * | 2022-04-08 | 2022-07-08 | 北京汇能精电科技股份有限公司 | Configurable civil hybrid energy storage power supply data acquisition method and device |
| CN115278562A (en) * | 2022-06-24 | 2022-11-01 | 北京思特奇信息技术股份有限公司 | Method and system for managing and controlling short message reminding based on flow configuration, electronic device and storage medium |
| CN114840488B (en) * | 2022-07-04 | 2023-05-02 | 柏科数据技术（深圳）股份有限公司 | Distributed storage method, system and storage medium based on super fusion structure |
| CN115102972A (en) * | 2022-07-15 | 2022-09-23 | 济南浪潮数据技术有限公司 | Method, device, equipment and medium for storing NFS (network file system) file |
| CN117061165A (en) * | 2023-08-10 | 2023-11-14 | 江苏瀚天智能科技股份有限公司 | Safety protection system based on space-time data lake technology of monitoring and control system |
| CN117194179B (en) * | 2023-11-08 | 2024-04-16 | 杭州星锐网讯科技有限公司 | Index determination method and device, electronic equipment and storage medium |
| CN117251499B (en) * | 2023-11-15 | 2024-02-06 | 山东光合云谷大数据有限公司 | Data acquisition system |
| CN117290190B (en) * | 2023-11-27 | 2024-02-13 | 博为科技有限公司 | Remote serial port log acquisition method, device and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104869022A (en) * | 2015-05-27 | 2015-08-26 | 北京京东尚科信息技术有限公司 | Log acquisition method and system |
| CN104883365A (en) * | 2015-05-14 | 2015-09-02 | 浪潮电子信息产业股份有限公司 | Method and device for storing and reading security logs and security control system |
| CN105099764A (en) * | 2015-06-29 | 2015-11-25 | 百度在线网络技术（北京）有限公司 | Log processing method and log processing device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020178146A1 (en) * | 2001-05-24 | 2002-11-28 | International Business Machines Corporation | System and method for selective object history retention |
| CN101610174B (en) * | 2009-07-24 | 2011-08-24 | 深圳市永达电子股份有限公司 | Log correlation analysis system and method |
-
2016
- 2016-03-31 CN CN201610202878.2A patent/CN105912587A/en active Pending
- 2016-08-26 WO PCT/CN2016/096968 patent/WO2017166644A1/en active Application Filing
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104883365A (en) * | 2015-05-14 | 2015-09-02 | 浪潮电子信息产业股份有限公司 | Method and device for storing and reading security logs and security control system |
| CN104869022A (en) * | 2015-05-27 | 2015-08-26 | 北京京东尚科信息技术有限公司 | Log acquisition method and system |
| CN105099764A (en) * | 2015-06-29 | 2015-11-25 | 百度在线网络技术（北京）有限公司 | Log processing method and log processing device |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109327351A (en) * | 2018-09-12 | 2019-02-12 | 拉扎斯网络科技（上海）有限公司 | Real-time collecting method, device, electronic equipment and the storage medium of daily record data |
| CN109327351B (en) * | 2018-09-12 | 2020-11-20 | 拉扎斯网络科技（上海）有限公司 | Method and device for collecting log data in real time, electronic equipment and storage medium |
| CN109271531A (en) * | 2018-11-16 | 2019-01-25 | 苏州友教习亦教育科技有限公司 | Control data corporation based on O&M knowledge mapping |
| CN109271531B (en) * | 2018-11-16 | 2023-04-18 | 苏州友教习亦教育科技有限公司 | Data management center based on operation and maintenance knowledge graph |
| CN109918048A (en) * | 2018-12-27 | 2019-06-21 | 北京奇艺世纪科技有限公司 | Target object extracting method, device, system and computer readable storage medium |
| CN109918048B (en) * | 2018-12-27 | 2022-09-06 | 北京奇艺世纪科技有限公司 | Target object extraction method, device and system and computer readable storage medium |
| CN110932918A (en) * | 2019-12-26 | 2020-03-27 | 远景智能国际私人投资有限公司 | Log data acquisition method and device and storage medium |
| CN110932918B (en) * | 2019-12-26 | 2023-01-10 | 远景智能国际私人投资有限公司 | Log data acquisition method and device and storage medium |
| CN113126562A (en) * | 2020-01-16 | 2021-07-16 | 智能云科信息科技有限公司 | Data acquisition method, device and system and computer readable storage medium |
| CN113126562B (en) * | 2020-01-16 | 2023-03-10 | 智能云科信息科技有限公司 | Data acquisition method, device and system and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2017166644A1 (en) | 2017-10-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105912587A (en) | Data acquisition method and system | |
| US20230177008A1 (en) | Session-Based Processing Method and System | |
| US11381592B2 (en) | System and method for identifying cybersecurity threats | |
| CN107818150B (en) | Log auditing method and device | |
| CN105490854B (en) | Real-time logs collection method, system and application server cluster | |
| US20160301732A1 (en) | Systems and Methods for Recording and Replaying of Web Transactions | |
| CN107800591B (en) | Unified log data analysis method | |
| CN105930363B (en) | HTML5 webpage-based user behavior analysis method and device | |
| US20170147615A1 (en) | Systems and methods for pruning data by sampling | |
| CN108509326B (en) | Service state statistical method and system based on nginx log | |
| CN101409690A (en) | Method and system for obtaining internet user behaviors | |
| CN103178982A (en) | Method and device for analyzing log | |
| CN102546668B (en) | Method, device and system for counting unique visitors | |
| CN109144836B (en) | Method and device for processing operation log and electronic equipment | |
| CN109255093A (en) | Behavioral data processing method, device, electronic equipment and computer-readable medium | |
| Sanjappa et al. | Analysis of logs by using logstash | |
| CN106559498A (en) | Air control data collection platform and its collection method | |
| US20160188676A1 (en) | Collaboration system for network management | |
| CN106250397A (en) | A kind of analysis method and device of user behavior feature | |
| CN113158118A (en) | Page buried point data acquisition method, device and system | |
| CN108011721A (en) | A kind of data leak method for early warning and system based on restoring files | |
| CN116028192A (en) | Multi-source heterogeneous data acquisition method, device and storage medium | |
| CN105550264A (en) | User journal collecting and processing system and method | |
| CN112929237B (en) | Analysis method, system, equipment and medium for website subdivision flow | |
| CN114817754A (en) | VR learning system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20160831 |
|
| WD01 | Invention patent application deemed withdrawn after publication |