CN109729130A - Information analysis method, service server, storage medium and device - Google Patents
Information analysis method, service server, storage medium and device Download PDFInfo
- Publication number
- CN109729130A CN109729130A CN201810295703.XA CN201810295703A CN109729130A CN 109729130 A CN109729130 A CN 109729130A CN 201810295703 A CN201810295703 A CN 201810295703A CN 109729130 A CN109729130 A CN 109729130A
- Authority
- CN
- China
- Prior art keywords
- information
- component
- equipment
- information analysis
- user
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Landscapes
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses information analysis method, service server, storage medium and devices.In the present invention in the push instruction for receiving the transmission of Producer component, push instruction is based on by Broker component and obtains corresponding user information to be pulled, user information to be pulled is stored to default storage region, receiving when pulling instruction of Consumer component transmission, user information is obtained from default storage region according to instruction is pulled by Broker component, the user information pushing that will acquire is to the second equipment, so that the Consumer component in the second equipment analyzes the user information got in real time.In the present invention by the way that real time data is transferred to Broker equipment, and pulling data and data analysis is completed in real time from Broker equipment, solves the technical issues of cannot preferably collecting real time data in the prior art and be analyzed existing for operation system.
Description
Technical field
The present invention relates to field of communication technology more particularly to information analysis method, service server, storage medium and dresses
It sets.
Background technique
With the continuous development of Communications And Computer technology, provide the company of Internet service in operation system constantly into
Going with service feature is the update being oriented to.For example, video website company need to provide for user Online Video viewing with
The service such as video download, in order to more optimize user experience, video website company needs to design more for the operation system of itself
Optimization is efficient, can handle the video-see request of high concurrent amount and is enable to respond quickly the network architecture of user's request.
And continuing to optimize with the network architecture, the various technologies based on data center's O&M are constantly suggested, and are caused each
The operation system of the various types of network services of offer in a field is constantly enhanced on service feature really, though
The service feature and operational efficiency of right operation system are all enhanced, however, existing operation system is mostly in real time data
Existing defects in analysis are played or are downloaded for example, the existing business system of video website company can preferably handle user
The request of video still can not collect these users request in time to analyze the usage trend and operation system sheet of user
The real-time running state of body causes these users request not analyzed effectively, requests in these users by business system
After system is responded and handled, invalid data just it has been considered as.
Even if in addition, there are the analysis means of some data for existing operation system, and periodically carrying out data
Statistical data is periodically obtained while storage, and data analysis can not be carried out in real time while getting real time data.
So existing operation system there is technical issues that preferably collect real time data and carry out.
Above content is only used to facilitate the understanding of the technical scheme, and is not represented and is recognized that above content is existing skill
Art.
Summary of the invention
The main purpose of the present invention is to provide information analysis method, service server, storage medium and devices, it is intended to solve
The technical issues of certainly cannot preferably collecting real time data existing for operation system in the prior art and be analyzed.
To achieve the above object, the present invention provides a kind of information analysis method, and the information analysis method includes following step
It is rapid:
When receiving the push instruction that the Producer component in the first equipment is sent, it is based on by Broker component described
Push instruction obtains corresponding user information to be pulled, and the user information to be pulled is stored to default storage region；
Receiving that Consumer component in the second equipment sends when pulling instruction, by the Broker component according to
It is described pull instruction user information, the Producer component, the Broker component are obtained from the default storage region
Kafka message components are belonged to the Consumer component；
The user information pushing that will acquire is to second equipment, so that the Consumer in second equipment
Component analyzes the user information got in real time.
Preferably, described to receive when the pulling instruction of Consumer component transmission in the second equipment, by described
Broker component pulls instruction before obtaining user information in the default storage region according to, the information analysis side
Method further include:
In the information analysis request for receiving the transmission of third equipment, target is extracted from information analysis request and is used
Family information；
Target user's information is stored to the first temporary storage area by Flume component；
When monitoring data conversion storage instruction, target user's letter is extracted from first temporary storage area
Breath, target user's information is stored to the default storage region.
It is preferably, described to be stored target user's information to the first temporary storage area by Flume component, comprising:
Target user's information is stored to the first temporary storage area by first agent's component of Flume component；
Correspondingly, described when monitoring data conversion storage instruction, it is extracted from first temporary storage area described
Target user's information stores target user's information to the default storage region, comprising:
When monitoring data conversion storage instruction, target user's letter is extracted from first temporary storage area
Breath；
The description components order of connection is read, the description components order of connection is used to record the company of the Agent components of each equipment
Connect sequence；
Next Agent components are determined from the description components order of connection based on first agent's component, by the mesh
Mark user information is sent to the 4th equipment locating for next Agent components, so that the 4th equipment is by next agency
Component stores target user's information to the default storage region.
Preferably, described determined from the description components order of connection based on first agent's component next acts on behalf of group
Target user's information is sent to the 4th equipment locating for next Agent components by part so that the 4th equipment by
Next Agent components store target user's information to the default storage region, comprising:
Each next agency of preset quantity is determined from the description components order of connection based on first agent's component
Component, the preset quantity are more than or equal to 2；
Determining each target device corresponding with the next Agent components；
Target user's information will be sent to each target device so that the target device by with the target device pair
The Agent components answered store target user's information to the default storage region.
Preferably, described in the information analysis request for receiving the transmission of third equipment, from information analysis request
Extract target user's information, comprising:
In the information analysis request for receiving the transmission of third equipment, the information analysis is asked by OpenResty component
It asks and is responded, to extract target user's information from information analysis request.
Preferably, described in the information analysis request for receiving the transmission of third equipment, by OpenResty component to described
Information analysis request is responded, to extract target user's information from information analysis request, comprising:
In the information analysis request for receiving the transmission of third equipment, found according to information analysis request
Default Nginx module in OpenResty component, by the information analysis request be sent to the default Nginx module, with
Information analysis request is responded in the default Nginx module；
When responding to information analysis request, target user's letter is extracted from information analysis request
Breath.
Preferably, described when receiving the push instruction that the Producer component in the first equipment is sent, by Broker
Component is based on push instruction and obtains corresponding user information to be pulled, and the user information to be pulled is stored to default and is deposited
After storage area domain, the information analysis method further include:
Broker component into the 5th equipment sends information checking instruction, so that whether the 5th equipment verification stores
Have and is sent as first equipment and user information to be pulled described in saving into the 5th equipment, when verifying successfully,
It generates and feeds back storage successful information；
When monitoring the storage successful information, execution is described to receive the Consumer component hair in the second equipment
Send when pulling instruction, as the Broker component pulled according to instruction from the default storage region obtain user letter
The step of breath.
In addition, to achieve the above object, the present invention also proposes that a kind of service server, the service server include storage
Device, processor and it is stored in the information analysis program that can be run on the memory and on the processor, the information point
The step of analysis program is arranged for carrying out information analysis method as described above.
In addition, to achieve the above object, the present invention also proposes a kind of storage medium, information is stored on the storage medium
The step of analysis program, the information analysis program realizes information analysis method as described above when being executed by processor.
In addition, to achieve the above object, the present invention also proposes a kind of information analysis apparatus, the information analysis apparatus packet
It includes: info push module, information exploitation module and information analysis module；
The info push module, in the push instruction for receiving the Producer component transmission in the first equipment
When, push instruction is based on by Broker component and obtains corresponding user information to be pulled, by the user information to be pulled
It stores to default storage region；
The information exploitation module, for pulling instruction receive that Consumer component in the second equipment sends
When, instruction is pulled according to as the Broker component obtains user information from the default storage region, it is described
Producer component, the Broker component and the Consumer component belong to Kafka message components；
The information analysis module, the user information pushing for will acquire is to second equipment, so that described
The Consumer component in two equipment analyzes the user information got in real time.
In the present invention by the way that real time data is transferred to Broker equipment, when user needs using the real time data,
From Broker equipment in real time pulling data and complete data analysis, thus realize for real time data it is timely collect with
And the consumption of asynchronization multiterminal, it solves preferably collect real time data existing for operation system in the prior art and be analyzed
The technical issues of.
Detailed description of the invention
Fig. 1 is the service server structural schematic diagram for the hardware running environment that the embodiment of the present invention is related to；
Fig. 2 is the flow diagram of information analysis method first embodiment of the present invention；
Fig. 3 is the deployment configuration diagram of Kafka component；
Fig. 4 is the flow diagram of information analysis method second embodiment of the present invention；
Fig. 5 is the first schematic diagram of deployment framework of Flume component；
Fig. 6 is the flow diagram of information analysis method 3rd embodiment of the present invention；
Fig. 7 is the second schematic diagram of deployment framework of Flume component；
Fig. 8 is the flow diagram of information analysis method fourth embodiment of the present invention.
Fig. 9 is the structural block diagram of information analysis apparatus first embodiment of the present invention.
The embodiments will be further described with reference to the accompanying drawings for the realization, the function and the advantages of the object of the present invention.
Specific embodiment
It should be appreciated that the specific embodiments described herein are merely illustrative of the present invention, it is not intended to limit the present invention.
Referring to Fig.1, Fig. 1 is the service server structural representation for the hardware running environment that the embodiment of the present invention is related to
Figure.
As shown in Figure 1, the service server may include: processor 1001, such as CPU, communication bus 1002, Yong Hujie
Mouth 1003, network interface 1004, memory 1005.Wherein, communication bus 1002 is logical for realizing the connection between these components
Letter.User interface 1003 may include display screen (Display), and optional user interface 1003 can also include that the wired of standard connects
Mouth, wireless interface, the wireline interface for user interface 1003 can be USB interface in the present invention.Network interface 1004 is optional
May include standard wireline interface and wireless interface (such as WI-FI interface).Memory 1005 can be high speed RAM memory,
It is also possible to stable memory (non-volatile memory), such as magnetic disk storage.Memory 1005 optionally may be used also
To be independently of the storage device of aforementioned processor 1001.
It, can be with it will be understood by those skilled in the art that structure shown in Fig. 1 does not constitute the restriction to service server
Including perhaps combining certain components or different component layouts than illustrating more or fewer components.
As shown in Figure 1, as may include that operating system, network are logical in a kind of memory 1005 of computer storage medium
Believe module, Subscriber Interface Module SIM and information analysis program.
In service server shown in Fig. 1, network interface 1004 is mainly used for connecting background server, with the backstage
Server carries out data communication；User interface 1003 is mainly used for connecting peripheral hardware；The service server passes through processor 1001
The information analysis program stored in memory 1005 is called, and executes information analysis method provided in an embodiment of the present invention.
Based on above-mentioned hardware configuration, the embodiment of information analysis method of the present invention is proposed.
It is the flow diagram of information analysis method first embodiment of the present invention referring to Fig. 2, Fig. 2.
In the first embodiment, the information analysis method the following steps are included:
Step S10: when receiving the push instruction that the Producer component in the first equipment is sent, by Broker component
Corresponding user information to be pulled is obtained based on push instruction, the user information to be pulled is stored to default memory block
Domain；
It is understood that the executing subject of the present embodiment is service server.Although in view of existing operation system
The relevant function of business can be preferably executed, for example, service server may provide the relevant function of taking out insurances, industry
Although the taking out insurances website of business server operation can better meet the browsing demand that user reads various insurance kinds in webpage
And the user's operation that user insures online is executed, still, not well to the browse operation of user and operation etc. of insuring
Operational order is collected, and then completes the analysis for user's operation.So lost largely during the circulation of information
The data such as the operation and instruction of generation waste the value of the partial data, do not analyze the partial data.Cause
For, if can be analyzed in real time the partial data, and analysis result is fed back in operation system in operating status,
The optimization of adaptability can be carried out to operation system well.
It in the concrete realization, can be by Kafka in order to overcome defect of the existing operation system in the real-time analysis of information
Component is combined into operation system, to realize a kind of deployment of new business system.The existing operation system of ratio, existing business system
System, even if having the system function of data analysis, can not also carry out the collection of data since performance is weaker in real time, because, it is real
When data be different from conventional data, real time data has the characteristics that formation speed is fast and the scale of construction is big, so, storage and processing are real
When data be all a kind of challenge for system performance for operation system.And in the present embodiment, by by Kafka
Component is combined into operation system, can be quickly accomplished the timely collection for real time data, and can asynchronization will be real
When data be supplied to the equipment with analysis demand, and then the processing of real time data can be completed well.
Wherein, Kafka component is that a kind of distributed post of high-throughput subscribes to message system, can refer to Fig. 3, Fig. 3 is
The deployment configuration diagram of Kafka component, there are Three roles and corresponding component, Producer to disappear in Kafka component
Breath and data producer, can give out information and data；Broker, caching agent can buffered message and data；Consumer disappears
Breath and data consumer, can pull and consume message and data.
It should be understood that for example, reference can be made to Fig. 3, equipment A1, equipment A2 and equipment A3 are deployed with Producer component
And executive role is Producer, equipment B1, equipment B2 and equipment B3 are deployed with Consumer component and executive role is
Consumer, Broker component is deployed on executing subject, that is, service server and executive role is Broker.So when
After Producer produces outbound message, message can be stored to Broker；When Consumer need to carry out message analysis or
When using the message, message can be grabbed from Broker to get message in time.
In the concrete realization, equipment A1 can be set as to the first equipment, equipment B1 is set as the second equipment, equipment A1 and equipment
B1 can be the equipment such as cell phone, PC and server.So when user is intended in operation equipment A1 in taking out insurances page
Face is insured, and can be acquired clicking operation and the information in input page that user clicks the taking out insurances page, be will click on behaviour
Make and various types of information such as input information are used as user information to be pulled, to remain the later period in point for carrying out real time data
It is used when analysis.
It is understood that in order to enable operation system can quickly and effectively carry out the analysis of real time data,
Equipment A1 is generated after pulling user information, automatically and can wait pulling by this in time based on the Producer component on equipment A1
User information pushing is timely stored real time data with realizing in the Broker into operation system.In addition, can set
After standby A1 automatically generates push instruction, user information to be pulled is sent to the default storage region in service server and is deposited
Storage；It can also be that user inputs push instruction " Push " on equipment A1, it, will be wait draw when monitoring " Push " of user's input
It takes user information to be sent to the default storage region in service server to be stored.
It should be understood that the default storage region can be empty for the storage of the service server local in operation system
Between, it can also be the memory space of other server locals in operation system.Wherein, operation system, which refers to, provides access service
The hardware and software systems that a plurality of types of network equipments are built.In view of Broker may be simultaneously present multiple equipment, Fig. 3
In only have recorded a service server, still, it is contemplated that the scale of construction of real time data is larger, and the equipment for playing the role of Broker can
Can exist simultaneously it is multiple, can will be wait draw for example, Broker may be simultaneously present service server, equipment C1 and equipment C2 etc.
User information is taken any of to store to these equipment, or integrality and safety in order to guarantee storing data, it can
User information to be pulled is stored in multiple into these equipment simultaneously.It is deposited so storing the default of user information to be pulled
Storage area domain can also be the memory space of other server locals in operation system.
Further, described when receiving the push instruction that the Producer component in the first equipment is sent, by
Broker component is based on the corresponding user information to be pulled of push instruction acquisition, by the user information to be pulled store to
After default storage region, the information analysis method further include:
Broker component into the 5th equipment sends information checking instruction, so that whether the 5th equipment verification stores
Have and is sent as first equipment and user information to be pulled described in saving into the 5th equipment, when verifying successfully,
It generates and feeds back storage successful information；When monitoring the storage successful information, execute described in receiving the second equipment
Consumer component send when pulling instruction, pulled and instructed from the default storage according to as the Broker component
The step of user information is obtained in region.
In the concrete realization, for example, may be simultaneously present two equipment for playing the role of Broker, service server and
Five equipment are also deployed with Broker component in the 5th equipment.In order to improve the reliability of information transmitting, when the first equipment, that is, equipment
When the new real time data generated at A1, identical real time data can be transmitted respectively to service server and the 5th equipment simultaneously,
Real time data is stored to two positions simultaneously with realizing, is asked so as to evade falling storage mistake and the network of individual node
Topic guarantees that data can succeed and is stored.Also, after service server completes storing step, it can send and believe to the 5th equipment
Checking command is ceased, to guarantee that both sides store success；After assert that both sides store successfully, the data pull of next step is just carried out
Process.
In addition, the mode of real time data is obtained for the 5th equipment, in addition to can directly be sent out from the first equipment to the 5th equipment
User information to be pulled is sent, is locally saved outside user information to realize in the 5th equipment, can also be taken in the first equipment to business
Business device is sent after pulling user information, and the user information to be pulled being locally stored actively is copied to the 5th by service server
In equipment, that is, using the synchronization between data of the framework to complete distinct device of master and servant's mode (Master-S lave).
Step S20: when the pulling instruction of Consumer component transmission in the second equipment is being received, by the Broker
Component pulls instruction according to and obtains user information from the default storage region, the Producer component, described
Broker component and the Consumer component belong to Kafka message components；
It is understood that storing user information to default storage region, in order to neatly to real time data into
Row is pulled and is analyzed, and Consumer group can be disposed on the equipment B1, equipment B2 and equipment B3 that need to use these real time datas
Part, so that equipment B1 can be sent to service server pulls instruction, so as to find user's letter from default storage region
Breath.Certainly, for pulling the mode of user information to be pulled, it can also be inputted on equipment B1 for user and pull instruction " Pull ",
When monitoring " Pull " of user's input, corresponding user information is found from default storage region, and will use be pulled
Family information feeds back to corresponding equipment B1.
Step S30: the user information pushing that will acquire to second equipment, so that described in second equipment
Consumer component analyzes the user information got in real time.
It should be understood that equipment B1 after getting user information, can carry out timely data to the user information
Analysis, in order to avoid abandoned during operation system operation using these real time datas as invalid information, to reduce
The use value of real time data.
It is understood that can promptly be retained due to by the way that real time data to be transferred to Broker equipment in time
Real time data, to prevent as these real time datas are abandoned or carried out as invalid data in existing operation system
Meaningless log storage；Since real time data is transferred to scheduled memory space, thus by the storage and drawing of real time data
Cancellation fee is separated, and the acquisition of data and application have just also been carried out asynchronous process, and user can need to be counted in real time
When according to analysis, the real time data is obtained from the equipment for being arbitrarily deployed with Consumer component and completes disappearing for real time data
Take, is also achieved that the asynchronization multiterminal consumption of real time data.Have benefited from the characteristic of Kafka component, real time data can be by equipment
The consumption of B1-B3 multiterminal, can be handled real time data during stream compression, to make full use of the actual effect of data
Sexual valence value.
In the present embodiment by the way that real time data is transferred to Broker equipment, when user needs to use the real time data
When, from Broker equipment in real time pulling data and complete data analysis, to realize the timely receipts for real time data
Collection and the consumption of asynchronization multiterminal, solve preferably collect real time data existing for operation system in the prior art and carry out
The technical issues of analysis.
It is the flow diagram of information analysis method second embodiment of the present invention referring to Fig. 4, Fig. 4, based on shown in above-mentioned Fig. 2
First embodiment, propose the second embodiment of information analysis method of the present invention.
In a second embodiment, before the step S20, the information analysis method further include:
Step S101: it in the information analysis request for receiving the transmission of third equipment, is mentioned from information analysis request
Take out target user's information；
It is understood that the scale of construction in view of real time data is larger, deposited to save real-time data synchronization to default
Storage area domain, synchronization time take a long time, and synchronization time may be up to several hours, and then also just influence whether subsequent for real-time
The analytic process of data.Also, the common method of synchronization is to be synchronized by way of batch processing, it will be apparent that, this kind of mode
It synchronizes and takes a long time.So real time data can be acquired using the asynchronization of Flume component to shorten synchronous duration, in turn,
Real time data that asynchronous collecting arrives can be carried out it is centrally stored, it is centrally stored due to having carried out real time data, be also just not necessarily to
Additional data synchronization process is carried out, to save synchronization time.
It should be understood that the Flume component be one kind can distributed result collection system, can with collector journal,
The data such as event, and the biggish data progress of the scale of construction being collected into is centrally stored.In short, Flume component collects data
And the data being collected into are carried out centrally stored.It is the first schematic diagram of deployment framework of Flume component, Flume referring to Fig. 5, Fig. 5
Component includes Agent components Agent, includes again Channel component etc. in Agent components Agent, Channel component can be deposited temporarily
Store up data.Referring to Fig. 5, the component type of first agent's component is Agent components Agent, the component class of the first temporary storage area
Type is Channel component.
Step S102: target user's information is stored to the first temporary storage area by Flume component；
It in the concrete realization, can will be newborn by Flume component after collecting third equipment newly-generated real time data
At real time data temporarily stored nearby into the first temporary storage area.The step is different from conventional synchronous step, conventional same
Data can be stored suddenly to the memory space in fixed storage path to complete to synchronize step by step.
It should be noted that multiple equipment for being deployed with Flume component will be existed simultaneously in the present embodiment, so, it deposits
In multiple temporary storage areas, newly-generated real time data is stored in temporary storage area actually, finally temporarily deposits all
The real time data collection stored in storage area domain converges to a fixed storage region.
Step S103: when monitoring data conversion storage instruction, the mesh is extracted from first temporary storage area
User information is marked, target user's information is stored to the default storage region.
It should be understood that after by real-time data memory to temporary storage area, in order to guarantee that Consumer equipment can
Required real time data is extracted from Broker equipment, real time data collection can be transferred to default storage region, so that
Consumer equipment can find real time data from default storage region, to both ensure that the nearest of real time data was deposited in time
Storage, also ensures that Consumer equipment can finally get real time data.Wherein, default storage region can be service server
Local storage space can also be extra large dupp distributed file system (Hadoop Distributed File System, HDFS)
In memory space.
In the present embodiment by storing target user's information to temporary storage area, and finally by target user's information
It is transferred to default storage region, thus ensure that real time data can be stored in time, meanwhile, save synchronization time.
It is the flow diagram of information analysis method 3rd embodiment of the present invention referring to Fig. 6, Fig. 6, based on shown in above-mentioned Fig. 4
Second embodiment, propose the 3rd embodiment of information analysis method of the present invention.
In the third embodiment, the step S102 may include:
Step S102': target user's information is stored to first by first agent's component of Flume component and is temporarily deposited
Storage area domain；
Correspondingly, step S103 may include:
Step S1031: when monitoring data conversion storage instruction, the mesh is extracted from first temporary storage area
Mark user information；
It is understood that in view of more equipment may be deployed in operation system, with reference to Fig. 5, in operation system
The next node of middle first agent's component is directly not necessarily equipment locating for default storage region, may be equipment D1 or equipment
D2 etc., such as Fig. 7, Fig. 7 are the second schematic diagram of deployment framework of Flume component, and the next node of first agent's component is second
The next node of Agent components, second agent's component is third Agent components, and the next node of third Agent components is just default
Equipment locating for storage region.
It should be understood that in order to realize that target user's information by each temporary storage area is finally stored to default storage
Region can determine next node based on the description components order of connection, referring to Fig. 7, second agent's component is deployed in equipment D1,
It include the second temporary storage area in second agent's component；The third generation is disposed in equipment D2 and manages component, is wrapped in third Agent components
Include third temporary storage area.
Step S1032: reading the description components order of connection, and the description components order of connection is used to record the generation of each equipment
Manage the order of connection of component；
It is understood that description components connect after proposing the description components order of connection from the first temporary storage area
Next component that connecing journal has first agent's component be second agent's component, second agent's component next component be third
Next component of Agent components and third Agent components is default storage region.
Step S1033: it is determined from the description components order of connection based on first agent's component and next acts on behalf of group
Target user's information is sent to the 4th equipment locating for next Agent components by part so that the 4th equipment by
Next Agent components store target user's information to the default storage region.
In the concrete realization, target user's information is extracted from the first temporary storage area by elder generation, then by mesh
Mark user information is transferred in the second temporary storage area of next node equipment D1, and then, finding next node is equipment
After D2, according to the lookup principle of above-mentioned lookup next node, finally, default storage region is found and successfully by target user
Information is stored into default storage region.
Further, the step S1033 can also include: based on first agent's component from the description components
Each next Agent components of preset quantity are determined in the order of connection, the preset quantity is more than or equal to 2；The determining and next generation
Manage the corresponding each target device of component；Target user's information will be sent to each target device, so that the target is set
It is standby to be stored target user's information to the default storage region by Agent components corresponding with the target device.
It in the concrete realization, is all one-to-one single line connection between each Agent components in Fig. 7, in order to guarantee data
Integrality and safety, and then the loss of real time data is prevented, the connection relationship between each Agent components can be expanded to " one
To more " connection relationship, for example, one-to-one connection relationship in Fig. 7 is that " next component of first agent's component is the second generation
Component is managed, next component of second agent's component is third Agent components, and the next node of third Agent components is default storage
Region ", can also be designed as " next component of first agent's component be second agent's component or third Agent components, second agent
The next node of component and third Agent components is default storage region ".By by next agency of first agent's component
Component is set as the connection relationship that multiple components form " one-to-many ", prevents one in second agent's component or third Agent components
When a failure, target user's information can be successfully transferred to default storage region by another Agent components.
In addition, can also arrange more temporary storage areas by the connection relationship for forming " one-to-many ", with more and
When collect the real time data of different location.
The description components order of connection is read in the present embodiment, and target user's information is finally transmitted to by many places equipment
Default storage region, is stored the real time data collection of many places equipment to default storage region with realizing.
It is the flow diagram of information analysis method fourth embodiment of the present invention referring to Fig. 8, Fig. 8, based on shown in above-mentioned Fig. 4
Second embodiment, propose the fourth embodiment of information analysis method of the present invention.
In the fourth embodiment, the step S101 can also include:
Step S101': in the information analysis request for receiving the transmission of third equipment, by OpenResty component to described
Information analysis request is responded, to extract target user's information from information analysis request.
It is understood that in view of existing operation system is in processing real time data and in the acquisition of real time data
Runnability is insufficient, and reason is that the design criteria of operation system is with the processing speed of service request and treatment effect for mark
Quasi- progress adaptive optimization, and the characteristic that the real time data that operation system generates at runtime has the scale of construction big, so, it is existing
Operation system may not be able to support acquisition and processing behavior for real time data.
It should be understood that considering the above-mentioned performance deficiency of operation system, can come rapidly using OpenResty component
It collects real time data and handles the real time data.Wherein, OpenResty component is the high-performance based on Nginx and Lua
Web platform, for building web application and the Web service etc. that are capable of handling high concurrent.Nginx is one high performance
Reverse Proxy, Lua are a kind of scripting languages.
In the concrete realization, handled by calling OpenResty component a large amount of information analysis request with it is a large amount of real
When data, may be implemented high concurrent request quick response and real time data rapidly extracting.
Further, the step S101' can also include: in the information analysis request for receiving the transmission of third equipment
When, the default Nginx module in OpenResty component is found according to information analysis request, the information analysis is asked
It asks and is sent to the default Nginx module, to be responded in the default Nginx module to information analysis request；
When responding to information analysis request, target user's information is extracted from information analysis request.
It is understood that passing through the response process for allowing information analysis to request during application OpenResty component
It is internal directly to betide Nginx service, that is, betides in default Nginx module, high performance request response may be implemented.This is
Because Nginx server has the I/O model of non-obstruction, so, it, can when the response made requests in Nginx server
With more flexible and quickly handle the request.So, it is believed that, receiving information analysis request and target user's information
When, the response that OpenResty component carries out data, a kind of realization of the OpenResty component as server architecture can be first passed through
Mode can be realized quickly and request of data is responded by neatly using Nginx module.
In the present embodiment, handled by calling OpenResty component a large amount of information analysis request with it is a large amount of real
When data, may be implemented high concurrent request quick response and real time data rapidly extracting.
In addition, the embodiment of the present invention also proposes a kind of storage medium, it is stored with information analysis program on the storage medium,
The information analysis program realizes the step of information analysis method as described above when being executed by processor.
In addition, the embodiment of the present invention also proposes that a kind of information analysis apparatus, the information analysis apparatus include: referring to Fig. 9
Info push module 10, information exploitation module 20 and information analysis module 30；
The info push module 10, in the push instruction for receiving the Producer component transmission in the first equipment
When, push instruction is based on by Broker component and obtains corresponding user information to be pulled, by the user information to be pulled
It stores to default storage region；
The information exploitation module 20, for pulling instruction receive that Consumer component in the second equipment sends
When, instruction is pulled according to as the Broker component obtains user information from the default storage region, it is described
Producer component, the Broker component and the Consumer component belong to Kafka message components；
The information analysis module 30, the user information pushing for will acquire is to second equipment, so that described
The Consumer component in second equipment analyzes the user information got in real time.
In the present embodiment by the way that real time data is transferred to Broker equipment, when user needs to use the real time data
When, from Broker equipment in real time pulling data and complete data analysis, to realize the timely receipts for real time data
Collection and the consumption of asynchronization multiterminal, solve preferably collect real time data existing for operation system in the prior art and carry out
The technical issues of analysis.
It should be noted that, in this document, the terms "include", "comprise" or its any other variant are intended to non-row
His property includes, so that the process, method, article or the system that include a series of elements not only include those elements, and
And further include other elements that are not explicitly listed, or further include for this process, method, article or system institute it is intrinsic
Element.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including being somebody's turn to do
There is also other identical elements in the process, method of element, article or system.
The serial number of the above embodiments of the invention is only for description, does not represent the advantages or disadvantages of the embodiments.
In the unit claims listing several devices, several in these devices can be by same hard
Part item embodies.The use of word first, second, and third does not indicate any sequence, can be by these word explanations
For title.
Through the above description of the embodiments, those skilled in the art can be understood that above-described embodiment side
Method can be realized by means of software and necessary general hardware platform, naturally it is also possible to by hardware, but in many cases
The former is more preferably embodiment.Based on this understanding, technical solution of the present invention substantially in other words does the prior art
The part contributed out can be embodied in the form of software products, which is stored in a storage medium
In (such as ROM/RAM, magnetic disk, CD), including some instructions are used so that a terminal device (can be mobile phone, computer, clothes
Business device, air conditioner or the network equipment etc.) execute method described in each embodiment of the present invention.
The above is only a preferred embodiment of the present invention, is not intended to limit the scope of the invention, all to utilize this hair
Equivalent structure or equivalent flow shift made by bright specification and accompanying drawing content is applied directly or indirectly in other relevant skills
Art field, is included within the scope of the present invention.
Claims (10)
1. a kind of information analysis method, which is characterized in that the information analysis method the following steps are included:
When receiving the push instruction that the Producer component in the first equipment is sent, the push is based on by Broker component
Instruction obtains corresponding user information to be pulled, and the user information to be pulled is stored to default storage region；
When the pulling instruction of Consumer component transmission in the second equipment is being received, as the Broker component according to
It pulls instruction and obtains user information, the Producer component, the Broker component and institute from the default storage region
It states Consumer component and belongs to Kafka message components；
The user information pushing that will acquire is to second equipment, so that the Consumer component in second equipment
The user information got is analyzed in real time.
2. information analysis method as described in claim 1, which is characterized in that described in receiving the second equipment
When pulling instruction of Consumer component transmission pulls instruction from the default memory block as the Broker component according to
Before obtaining user information in domain, the information analysis method further include:
In the information analysis request for receiving the transmission of third equipment, target user's letter is extracted from information analysis request
Breath；
Target user's information is stored to the first temporary storage area by Flume component；
When monitoring data conversion storage instruction, target user's information is extracted from first temporary storage area, it will
Target user's information is stored to the default storage region.
3. information analysis method as claimed in claim 2, which is characterized in that it is described by Flume component by the target user
Information is stored to the first temporary storage area, comprising:
Target user's information is stored to the first temporary storage area by first agent's component of Flume component；
Correspondingly, described when monitoring data conversion storage instruction, the target is extracted from first temporary storage area
User information stores target user's information to the default storage region, comprising:
When monitoring data conversion storage instruction, target user's information is extracted from first temporary storage area；
The description components order of connection is read, the connection for the Agent components that the description components order of connection is used to record each equipment is suitable
Sequence；
Next Agent components are determined from the description components order of connection based on first agent's component, and the target is used
Family information is sent to the 4th equipment locating for next Agent components, so that the 4th equipment is by next Agent components
Target user's information is stored to the default storage region.
4. information analysis method as claimed in claim 3, which is characterized in that it is described based on first agent's component from described
Next Agent components are determined in the description components order of connection, and target user's information is sent to next Agent components institute
4th equipment at place, so that the 4th equipment is stored target user's information to described pre- by next Agent components
If storage region, comprising:
Each next Agent components of preset quantity are determined from the description components order of connection based on first agent's component,
The preset quantity is more than or equal to 2；
Determining each target device corresponding with the next Agent components；
Target user's information will be sent to each target device, so that the target device is by corresponding with the target device
Agent components store target user's information to the default storage region.
5. information analysis method as claimed in claim 2, which is characterized in that described in the information for receiving the transmission of third equipment
When analysis request, target user's information is extracted from information analysis request, comprising:
Receive third equipment transmission information analysis request when, by OpenResty component to the information analysis request into
Row response, to extract target user's information from information analysis request.
6. information analysis method as claimed in claim 5, which is characterized in that described in the information for receiving the transmission of third equipment
When analysis request, information analysis request is responded by OpenResty component, to be mentioned from information analysis request
Take out target user's information, comprising:
In the information analysis request for receiving the transmission of third equipment, OpenResty is found according to information analysis request
Information analysis request is sent to the default Nginx module, described default by the default Nginx module in component
Information analysis request is responded in Nginx module；
When responding to information analysis request, target user's information is extracted from information analysis request.
7. such as information analysis method described in any one of claims 1 to 6, which is characterized in that described to set receiving first
When the push instruction that the Producer component in is sent, it is corresponding wait draw that push instruction acquisition is based on by Broker component
User information is taken, the user information to be pulled is stored to default storage region, the information analysis method further include:
Broker component into the 5th equipment sends information checking instruction so that the 5th equipment verification whether be stored with by
First equipment is sent and user information to be pulled described in saving into the 5th equipment, when verifying successfully, generates
And feed back storage successful information；
When monitoring the storage successful information, execution is described to receive what the Consumer component in the second equipment was sent
When pulling instruction, instruction is pulled according to as the Broker component and obtains user information from the default storage region
Step.
8. a kind of service server, which is characterized in that the service server includes: memory, processor and is stored in described
On memory and program can be analyzed by operation information on the processor, when the information analysis program is executed by the processor
The step of realizing the information analysis method as described in any one of claims 1 to 7.
9. a kind of storage medium, which is characterized in that be stored with information analysis program, the information analysis journey on the storage medium
The step of information analysis method as described in any one of claims 1 to 7 is realized when sequence is executed by processor.
10. a kind of information analysis apparatus, which is characterized in that the information analysis apparatus includes: info push module, information exploitation
Module and information analysis module；
The info push module, when for being instructed in the push for receiving the Producer component transmission in the first equipment, by
Broker component is based on the corresponding user information to be pulled of push instruction acquisition, by the user information to be pulled store to
Default storage region；
The information exploitation module, for receiving when the pulling instruction of Consumer component transmission in the second equipment, by
The Broker component pulls instruction according to and obtains user information, the Producer group from the default storage region
Part, the Broker component and the Consumer component belong to Kafka message components；
The information analysis module, the user information pushing for will acquire is to second equipment, so that described second sets
The Consumer component in standby analyzes the user information got in real time.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810295703.XA CN109729130A (en) | 2018-04-04 | 2018-04-04 | Information analysis method, service server, storage medium and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810295703.XA CN109729130A (en) | 2018-04-04 | 2018-04-04 | Information analysis method, service server, storage medium and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109729130A true CN109729130A (en) | 2019-05-07 |
Family
ID=66293812
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810295703.XA Pending CN109729130A (en) | 2018-04-04 | 2018-04-04 | Information analysis method, service server, storage medium and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109729130A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110809050A (en) * | 2019-11-08 | 2020-02-18 | 智者四海（北京）技术有限公司 | Personalized push system and method based on streaming computing |
| CN111211964A (en) * | 2019-12-26 | 2020-05-29 | 北京大米未来科技有限公司 | Instruction transmission method, device, storage medium and server |
| CN113572854A (en) * | 2021-08-10 | 2021-10-29 | 北京无线电测量研究所 | Kafka component-based data transmission method and system |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105608188A (en) * | 2015-12-23 | 2016-05-25 | 北京奇虎科技有限公司 | Data processing method and data processing device |
| CN105893628A (en) * | 2016-05-17 | 2016-08-24 | 中国农业银行股份有限公司 | Real-time data collection system and method |
| CN106095965A (en) * | 2016-06-17 | 2016-11-09 | 上海智臻智能网络科技股份有限公司 | A kind of data processing method and device |
| US20170060622A1 (en) * | 2015-02-27 | 2017-03-02 | Savi Technology, Inc. | System and method for a data processing architecture |
| CN106559498A (en) * | 2016-12-02 | 2017-04-05 | 上海钱柳电子商务有限公司 | Air control data collection platform and its collection method |
| CN107332719A (en) * | 2017-08-16 | 2017-11-07 | 北京云端智度科技有限公司 | A kind of method that daily record is analyzed in real time in CDN system |
-
2018
- 2018-04-04 CN CN201810295703.XA patent/CN109729130A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170060622A1 (en) * | 2015-02-27 | 2017-03-02 | Savi Technology, Inc. | System and method for a data processing architecture |
| CN105608188A (en) * | 2015-12-23 | 2016-05-25 | 北京奇虎科技有限公司 | Data processing method and data processing device |
| CN105893628A (en) * | 2016-05-17 | 2016-08-24 | 中国农业银行股份有限公司 | Real-time data collection system and method |
| CN106095965A (en) * | 2016-06-17 | 2016-11-09 | 上海智臻智能网络科技股份有限公司 | A kind of data processing method and device |
| CN106559498A (en) * | 2016-12-02 | 2017-04-05 | 上海钱柳电子商务有限公司 | Air control data collection platform and its collection method |
| CN107332719A (en) * | 2017-08-16 | 2017-11-07 | 北京云端智度科技有限公司 | A kind of method that daily record is analyzed in real time in CDN system |
Non-Patent Citations (2)
| Title |
|---|
| 费秀宏: ""基于Kafka的日志处理平台的研究"", 《中国优秀硕士学位论文全文数据库》 * |
| 陈付梅等: ""大数据环境下的分布式数据流处理关键技术探析"", 《计算机应用》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110809050A (en) * | 2019-11-08 | 2020-02-18 | 智者四海（北京）技术有限公司 | Personalized push system and method based on streaming computing |
| CN110809050B (en) * | 2019-11-08 | 2022-11-29 | 智者四海（北京）技术有限公司 | Personalized pushing system and method based on streaming computing |
| CN111211964A (en) * | 2019-12-26 | 2020-05-29 | 北京大米未来科技有限公司 | Instruction transmission method, device, storage medium and server |
| CN113572854A (en) * | 2021-08-10 | 2021-10-29 | 北京无线电测量研究所 | Kafka component-based data transmission method and system |
| CN113572854B (en) * | 2021-08-10 | 2023-11-14 | 北京无线电测量研究所 | Data transmission method and system based on Kafka component |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110908879B (en) | Reporting method, reporting device, reporting terminal and recording medium of buried point data | |
| CN104699736B (en) | A kind of distributed larger scale data acquisition system and method based on movable equipment | |
| CN109684370A (en) | Daily record data processing method, system, equipment and storage medium | |
| CN109918349A (en) | Log processing method, device, storage medium and electronic device | |
| CN110457281A (en) | Data processing method, device, equipment and medium | |
| CN104935655A (en) | Data presentation method and device | |
| CN109729130A (en) | Information analysis method, service server, storage medium and device | |
| CN107122457A (en) | Record the method and its device, computer-readable medium of networks congestion control data | |
| CN111131379A (en) | Distributed flow acquisition system and edge calculation method | |
| CN110704400A (en) | Real-time data synchronization method and device and server | |
| CN104394211A (en) | Hadoop-based user behavior analysis system design and implementation method | |
| CN113010565B (en) | Server real-time data processing method and system based on server cluster | |
| CN103200272B (en) | Streaming media storage system and storage method | |
| CN108280023A (en) | Task executing method, device and server | |
| CN104660633A (en) | New media public service platform | |
| CN110502583A (en) | Distributed Data Synchronization method, apparatus, equipment and readable storage medium storing program for executing | |
| CN113282611B (en) | Method, device, computer equipment and storage medium for synchronizing stream data | |
| CN110716848A (en) | Data collection method and device, electronic equipment and storage medium | |
| CN113656673A (en) | Master-slave distributed content crawling robot for advertisement delivery | |
| CN115357497A (en) | Service fault analysis method, device, medium and computer equipment | |
| CN103034655A (en) | Collection method and system of user behavior information and related equipment | |
| CN114153609A (en) | Resource control method and device, electronic equipment and computer readable storage medium | |
| CN111444412B (en) | Method and device for scheduling web crawler tasks | |
| CN109542638A (en) | A kind of document handling method and device based on educational system | |
| CN110297904B (en) | Event name generation method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190507 |