CN107179953B - Index file generation method, device and system - Google Patents
Index file generation method, device and system Download PDFInfo
- Publication number
- CN107179953B CN107179953B CN201710207991.4A CN201710207991A CN107179953B CN 107179953 B CN107179953 B CN 107179953B CN 201710207991 A CN201710207991 A CN 201710207991A CN 107179953 B CN107179953 B CN 107179953B
- Authority
- CN
- China
- Prior art keywords
- message
- index file
- log files
- service
- log
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/546—Message passing systems or structures, e.g. queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9027—Trees
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The embodiment of the invention provides an index file generation method, which is applied to main equipment in an index file generation system, wherein the index file generation system also comprises at least two service devices, and the method comprises the following steps: acquiring all local log files for storing messages; grouping the log files according to a preset rule and then sending the log files to each service device so that each service device synchronously sends the received messages in the log files back to the master device; and receiving and storing the messages sent by each service device, and generating an index file of the messages sent by each service device. By implementing the scheme provided by the embodiment of the invention, the real-time performance of the ActiveMQ service can be ensured.
Description
Technical Field
The present invention relates to the field of information retrieval, and in particular, to a method, an apparatus, and a system for generating an index file.
Background
In the large background of the information age, message transmission is often required between multiple applications to coordinate tasks. For example, a formula is inserted into a word document, and the word document is required to send a message for MathType to a formula editor MathType, so that the MathType is matched with the word document to create the formula when receiving the message. However, when the number and the types of the application programs are large, the number of the messages is also greatly increased, and when the number of the messages exceeds the maximum number limited by a message caching mechanism of the message receiving program, the messages are lost, so that the application programs cannot cooperate to complete the tasks.
In known methods, technicians typically use a message bus ActiveMQ (open source message bus, by Apache) service to solve the above-mentioned problems. The specific process is as follows: the ActiveMQ of the local device acquires all messages sent by all local message sending programs and stores the messages into a plurality of log files created by the ActiveMQ; then, the ActiveMQ scans the messages in each log file one by one, and establishes index files of all the obtained messages one by one, wherein the index files comprise the storage addresses of all the messages; when the ActiveMQ detects that the application program to which any one of the acquired messages is directed is in an idle state, the position of the message in the log file is acquired through the index file, the message is taken out from the log file, and the message is sent to the application program to which the message is directed.
However, in the above method, when the number of messages acquired by the ActiveMQ is huge, the size and the number of the log files are increased, and the process of scanning the log files becomes slow and long, so that the establishment of the index file becomes tedious and time-consuming, which affects the normal receiving and sending of the messages, and further, the real-time performance of the ActiveMQ service cannot be ensured.
Disclosure of Invention
The embodiment of the invention aims to provide an index file generation method, device and system to ensure the instantaneity of ActiveMQ service. The specific technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides an index file generation method, where the method is applied to a master device in an index file generation system, where the index file generation system further includes at least two service devices, and the method includes:
acquiring all local log files for storing messages;
grouping the log files according to a preset rule and then sending the log files to each service device so that each service device synchronously sends the received messages in the log files back to the master device;
and receiving and storing the messages sent by each service device, and generating an index file of the messages sent by each service device.
Optionally, the step of grouping the log files according to a preset rule and then sending the grouped log files to each service device includes:
according to the number of the service devices, averagely dividing the log file into a corresponding number of log groups;
and respectively sending each group of log files to each service device.
Optionally, the step of grouping the log files according to a preset rule and then sending the grouped log files to each service device includes:
acquiring the current load capacity of each service device, and determining the number of log files to be sent to each service device according to the acquired current load capacity of each service device and the total number of the log files;
grouping the log files according to the determined number of the log files to be sent to each service device;
and sending the log packets containing the corresponding number of log files to the corresponding service equipment.
Optionally, the step of receiving and storing the message sent by each service device, and generating an index file of the message sent by each service device includes:
receiving and storing messages sent by each service device;
parallelly acquiring offset addresses of all messages, and establishing a B-tree according to the acquired offset addresses;
and storing the B tree into a newly-built file, and determining the newly-built file as an index file.
In a second aspect, an embodiment of the present invention provides an index file generation method, where the method is applied to any service device in an index file generation system, where the index file generation system further includes a master device, and the method includes:
receiving a log file sent by the master device, wherein the log file is sent to the service device after the master device groups all local log files for storing messages according to a preset rule;
receiving a message migration instruction, and sending a message to be migrated to the primary device, to which the message migration instruction in the received log file is directed, so that the primary device stores the message to be migrated.
Optionally, the receiving a message migration instruction, and sending the to-be-migrated message to the primary device, where the message migration instruction is specific to the received log file, so that the primary device stores the to-be-migrated message includes:
receiving a message migration instruction, and acquiring a producer and a consumer of the message to be migrated, wherein the message to be migrated is a message in the received log file, the consumer is an application program which acquires the message and sends the acquired message to the producer, and the producer is an application program which receives the message sent by the consumer, produces the message, and sends the produced message to the primary device;
each producer and consumer is operated to have each consumer send the message for which it is intended to the producer of that message, so that the producer produces the message and sends the produced message to the master device.
Optionally, the number of producers and the number of consumers is the same as the number of destination objects of the message to be migrated, where the destination object is an object for which the message to be migrated is directed, and multiple messages to be migrated are directed to one destination object.
In a third aspect, an embodiment of the present invention provides an index file generation apparatus, where the apparatus is applied to a master device in an index file generation system, where the index file generation system further includes at least two service devices, and the apparatus includes:
the obtaining module is used for obtaining all local log files for storing messages;
the sending module is used for grouping the log files according to a preset rule and then sending the log files to each service device so that each service device can synchronously send the received messages in the log files back to the main device;
and the receiving module is used for receiving and storing the messages sent by each service device and generating an index file of the messages sent by each service device.
Optionally, the sending module includes:
the first dividing module is used for averagely dividing the log files into log groups with corresponding number according to the number of the service devices;
and the first sending submodule is used for respectively sending each group of log files to each service device.
Optionally, the sending module includes:
the first determining submodule is used for acquiring the current load of each service device and determining the number of log files to be sent to each service device according to the acquired current load of each service device and the total number of the log files;
the second division submodule is used for grouping the log files according to the determined number of the log files to be sent to each service device;
and the second sending submodule is used for sending the log packets containing the corresponding number of log files to the corresponding service equipment.
Optionally, the receiving module includes:
the receiving submodule is used for receiving and storing messages sent by each service device;
the building submodule is used for parallelly obtaining the offset address of each message and building a B tree according to the obtained offset address;
and the second determining submodule is used for storing the B tree into a new file and determining the new file as an index file.
In a fourth aspect, an embodiment of the present invention provides an index file generation apparatus, where the apparatus is applied to any service device in an index file generation system, where the index file generation system further includes a master device, and the apparatus includes:
the receiving module is used for receiving the log file sent by the main equipment, wherein the log file is sent to the service equipment after the main equipment groups all local log files for storing messages according to a preset rule;
the generating module is used for receiving a message migration instruction, and sending a message to be migrated to the main device, which is in the received log file and is aimed at by the message migration instruction, so that the main device stores the message to be migrated.
Optionally, the generating module includes:
the obtaining submodule is used for receiving a message migration instruction, obtaining a producer and a consumer of the message to be migrated, wherein the message to be migrated is a received message in the log file, the consumer is an application program which obtains the message and sends the obtained message to the producer, and the producer is an application program which receives the message sent by the consumer, produces the message and sends the produced message to the main device;
and the production module is used for operating each producer and each consumer, so that each consumer sends the message aimed at to the producer of the message, and the producer produces the message and sends the produced message to the master equipment.
Optionally, the number of the producers is the same as the number of the destination objects of the message to be migrated, where the destination object is an object for which the message to be migrated is directed, and multiple messages to be migrated are directed to one destination object.
In a fifth aspect, an embodiment of the present invention provides an index file generation system, where the system includes a main device and at least two service devices;
the main device is configured to execute the index file generation method according to the embodiment of the invention in the first aspect;
each of the service devices is configured to execute the index file generation method according to the embodiment of the invention in the second aspect.
The embodiment of the invention provides an index file generation method, an index file generation device and an index file generation system, wherein the method comprises the following steps: the log file used for storing the messages on the main device is distributed to at least two service devices in the index file generation system, so that each service device synchronously sends the messages stored in the log file which is received by the service device and grouped by the main device back to the main device. Compared with the prior art that the messages in the log file are scanned one by one and the index of the messages is established one by one, the method provided by the embodiment of the invention can accelerate the generation speed of the index file, shorten the time for generating the index file, avoid the problem that the normal receiving and sending of the messages are influenced because the ActiveMQ takes a long time to establish the index file in the prior art, and ensure the real-time performance of the ActiveMQ service.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a schematic flowchart illustrating steps of a method for generating an index file according to an embodiment of the present invention;
fig. 2 is a schematic flow chart illustrating another step of an index file generation method according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a B-tree structure of an index file generation method according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of an index file generating apparatus according to an embodiment of the present invention;
fig. 5 is another schematic structural diagram of an index file generating apparatus according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an index file generation system according to an embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The embodiment of the invention provides an index file generation method, an index file generation device and an index file generation system.
In a first aspect, the index file generation method provided in the embodiment of the present invention is applied to a master device in an index file generation system, where the index file generation system further includes at least two service devices. The main device and the service device may be servers of a server side, and may also be electronic devices of a client side. From another perspective, the main device and the service device may be a physical machine that implements a certain function and is installed with an embedded system, or may be a general-purpose machine.
It should be noted that the index file generation method provided in the embodiment of the present invention is applied to a main device in an index file generation system, and in particular, may be applied to an index file generation apparatus in the main device. When the main device is an electronic device of a client, the index file generation device can be special message management software or a functional plug-in of related software; when the main device is a server, the index file generation device may be a system providing a message management service, or may be a system plug-in of a related system.
As shown in fig. 1, an index file generation method provided in an embodiment of the present invention is applied to a master device in an index file generation system, and includes the following steps:
s101: acquiring all local log files for storing messages;
when a plurality of application programs need to cooperate to complete a certain function, message transmission is carried out among the application programs, and if the transmitted message cannot be processed in time, the transmitted message is stored in a log file. In the embodiment of the invention, when the index file of the message needs to be generated, the master device can obtain the log file stored in the local disk.
Wherein the obtained log file may be an original file of the storage directory. For example, the log file may be db.log file under KahaDB (persistent storage means by default for ActiveMQ); of course, the files in the backed-up directory may be backed up after the storage directory is backed up. Log files in the backed up KahaDB directory, for example.
The message may be contents of mutual communication between applications on the same device, or may be contents of mutual communication between applications on different devices.
S102: grouping the log files according to a preset rule and then sending the log files to each service device so that each service device synchronously sends the received messages in the log files back to the master device;
optionally, the preset rule may be related to the number of the service devices, may also be related to the operating state of the service devices, and is specifically preset by a worker according to an actual requirement.
In one implementation, the step of grouping the log files according to a preset rule may include: according to the number of the service devices, averagely dividing the log files into a corresponding number of log groups; and respectively sending each group of log files to each service device. For example, the number of log files is 3000, the number of service devices is 3, and the log files are respectively device a, device B, and device C, so that the log files can be averagely divided into 3 groups, which are respectively 1000 log files of the first group, 1000 log files of the second group, and 1000 log files of the third group. Then, a first set of 1000 log files may be sent to device a, a second set of 1000 log files may be sent to device B, and a third set of 1000 log files may be sent to device C.
In practical applications, there are cases where the log files cannot be evenly grouped, for example, 3100 log files are evenly grouped into 3 groups. In this case, the log file that is redundant after the average grouping can be added to any one group. For example, after 3100 log files are equally divided into 3 groups (each group includes 1033 log files), 1 log file remains. In this case, the remaining 1 log file may be added to any group, such that the group includes 1034 log files.
In another implementation, the step of grouping the log files according to a preset rule may include: acquiring the current load capacity of each service device, and determining the number of log files to be sent to each service device according to the acquired current load capacity of each service device and the total number of the log files; grouping the log files according to the determined number of the log files to be sent to each service device; and sending the log packets containing the corresponding number of log files to the corresponding service equipment.
The load of each server may be represented as a CPU (Central processing unit) utilization rate of each service device. The method for grouping the log files can be as follows: when the current load capacity of a certain service device exceeds a certain value, a smaller number of logs can be sent for it. For example, when the usage rate of device a is higher than 90%, the determined number of log files in the group sent to device a is divided into a first group and a second group, the log files of the first group are sent to device a, and all (average groups) of the log files of the second group are sent to some device (corresponding to the number of groups) other than a.
The method for grouping the log files may further comprise: and setting the proportion of the residual load of each service device to be consistent with the quantity proportion of the files sent to each device. For example, A, B, C, D, E service devices are included in the index file, and the current remaining load is a: 40%, B: 90%, C: 30%, D: 50%, E: 20%, adding the current remaining load values of the service devices, and calculating the sum value to be 230%, so that the proportion of the remaining load of the device a in all five service devices is
Namely, it is
The number of files allocated to device a accounts for all log files of the master device
The proportion of the residual load of the equipment B in all five service equipment is
Namely, it is
Then the device B is allocated the number of files occupying all log files of the master device
Similarly, device C is assigned a file count that accounts for all log files of the master device
Allocating the number of files to all log files of the master device for the device D
Allocating the number of files to the equipment E in all log files of the master equipment
After the log files are grouped, the main device sends the log files in each group to the corresponding service device, and the process of sending the log files by the main device is a file moving process, and specifically, the log files can be moved by means of qpress (a tool for file backup and migration in a computer system).
Each service device receives the log file sent by the master device and synchronously sends the message in the received log file back to the master device. Specifically, the service device may send the received message in the log file back to the master device by: storing the received message in the log file, and generating an index file of the message in the log file; when a message migration instruction is received, a message is sent back to the master device by the producer. The producer may be an application that may be used to produce messages and communicate with the master device to send the produced messages to the master device.
The embodiment of the invention does not limit the sending equipment of the message migration instruction, and only needs to ensure that each service equipment receives the message migration instruction at the same time, thereby further ensuring the synchronism of the messages sent by each service equipment.
S103: and receiving and storing the messages sent by each service device, and generating an index file of the messages sent by each service device.
It is to be understood that since the steps in S102 cause the service apparatuses to synchronously transmit messages to the master apparatus, the master apparatus receives a plurality of messages at the same time. For example, if 4 service devices synchronously send messages, the master device receives 4 messages at the same time, and thus may generate indexes of 4 messages at the same time, and further obtain an index file.
Optionally, the manner of generating the index file may be: receiving and storing messages sent by each service device; parallelly acquiring offset addresses of all messages, and establishing a B-tree according to the acquired offset addresses; and storing the B-tree into the new file, and determining the new file as an index file. The B-tree is a nonlinear data structure in computer science and is used for indexing data.
The method for building the B-tree according to the offset address may be: and grouping the messages according to the size of the offset address value, and storing the messages in each group to the bottommost nodes of different branches of the B tree. For example, the offset address of the retrieved message is shown in table 1.
TABLE 1
| Message | Offset address |
| Message a | 1 |
| Message b | 2 |
| Message c | 3 |
| |
4 |
| Message e | 5 |
| Message f | 6 |
| Message g | 7 |
| Message h | 8 |
When the division criterion of the value 4 is preset, the messages a, b, c and d with the offset addresses not greater than 4 can be divided into a first group, and the messages e, f, g and h with the offset addresses greater than 4 can be divided into a second group. Storing the two groups of messages to the lowest node of the B-tree to obtain the B-tree shown in FIG. 3. Thus, when indexing a message, assuming that the offset address 2 of the message B is obtained, when indexing the message B according to the B-tree, comparing 2 with the top node value 4 yields 2<4, traversing the message B in the left node (looking for the message B from the 4 messages).
And after the B tree is established, storing the B tree into a file to obtain an index file.
The method for generating the index file provided by the embodiment of the invention comprises the following steps: the log file used for storing the messages on the main device is distributed to at least two service devices in the index file generation system, so that each service device synchronously sends the messages stored in the log file which is received by the service device and grouped by the main device back to the main device, and because each service device synchronously sends the messages to the main device, the main device receives the messages with the same number (or more than the service devices) as the service devices in the same time, and the main device can simultaneously establish indexes of the messages with the same number after receiving and storing the messages with the number. Compared with the prior art that the messages in the day files are scanned one by one and the indexes are established one by one, the method provided by the embodiment of the invention can accelerate the generation speed of the index file, shorten the time for generating the index file, avoid the problem that the normal receiving and sending of the messages are influenced because the ActiveMQ spends a long time to establish the index file in the prior art, and ensure the real-time property of the ActiveMQ service.
In a second aspect, an embodiment of the present invention further provides an index file generation method, where the method is applied to any service device in an index file generation system, and the index file generation system further includes a master device. For the introduction of the master device and the service device, please refer to the introduction in the first aspect.
It should be noted that the index file generation method provided in the embodiment of the present invention is applied to a service device in an index file generation system, and in particular, may be applied to an index file generation apparatus in a host device. When the service device is an electronic device of a client, the index file generation device may be a special message management software or a functional plug-in of related software; when the service device is a server, the index file generation device may be a system providing a message management service, or may be a system plug-in of a related system.
As shown in fig. 2, an index file generation method provided in an embodiment of the present invention is applied to a service device in an index file generation system, and includes the following steps:
s201: receiving a log file sent by a main device in an index file generation system, wherein the log file is sent to the service device after the main device groups all local log files for storing messages according to a preset rule;
when the index file is required to be generated, the main device in the index file generation system sends the log files storing the messages to each service device after grouping, so that each service device synchronously sends the messages in the received log files back to the main device. Therefore, each service device in the above system receives the log file to send the message in the received log file back to the master device, and the master device can then restore the message.
The received log file may be an original file in a storage directory on the host device, for example, the log file may be a db.log file in a KahaDB (persistent storage mode default to ActiveMQ) directory stored by the host device; of course, the files in the backed-up directory, for example, db.log files in the backed-up KahaDB directory, may also be the files in the backed-up directory after the primary device backs up the storage directory.
The message may be contents of mutual communication between applications on the same device, or may be contents of mutual communication between applications on different devices.
The preset rules according to which the master device groups may be related to the number of the service devices, and may also be related to the operating states of the service devices, and are preset by the staff according to actual requirements.
S202: receiving a message migration instruction, and sending a message to be migrated to the primary device, to which the message migration instruction in the received log file is directed, so that the primary device stores the message to be migrated.
Each service device receives the message migration instruction, and sends the message to be migrated to the main device, wherein the message is aimed at by the message migration instruction. The method specifically comprises the following steps: receiving a message migration instruction, and acquiring a producer and a consumer of a message to be migrated to which the message migration instruction is directed, wherein the message to be migrated is a message in a received log file, the consumer is an application program which acquires the message and sends the acquired message to the producer, and the producer is an application program which receives the message sent by the consumer, produces the message and sends the produced message to a main device; each producer and consumer is operated such that each producer produces a message and sends the produced message to the master device.
The producer and the consumer may be applications on the service device, or may be applications on other devices communicating with the service device. The process of deleting (consuming) the message by the consumer and producing the message by the producer is the prior art, and the embodiment of the invention is not limited.
The message migration instruction may be for a certain message or for a plurality of messages.
In the index file generation method provided by the embodiment of the invention, after receiving the message migration instruction, the service device in the index file generation system sends the message to be migrated in the received log file, to which the message migration instruction is directed, back to the main device. Compared with the prior art that the messages in the log file are scanned one by one and the index of the messages is established one by one, the method provided by the embodiment of the invention can accelerate the generation speed of the index file, shorten the time for generating the index file, avoid the problem that the normal receiving and sending of the messages are influenced because the ActiveMQ takes a long time to establish the index file in the prior art, and ensure the real-time performance of the ActiveMQ service.
In order to avoid that the service device repeatedly sends the message to be migrated (omits to send the message to be migrated), in the embodiment of the present invention, after sending the message to the master device, the method further includes: and receiving a migration message deleting instruction, and deleting the target message according to the migration message deleting instruction, wherein the target message is a message successfully sent to the main equipment.
The migration message deletion instruction may be a deletion instruction for a certain message, or may be a deletion instruction for one or more messages. For example, a certain service device in the index file generation system stores a message a, a message b, and a message c of the master device, and when the received message migration instruction is a message migration instruction for the message a, the producer deletes the message a according to a deletion instruction for the message a after producing the message a to the master device; and when the received message migration instruction is a message migration instruction aiming at the message b, after the producer produces the message b to the main equipment, deleting the message b according to a deletion instruction aiming at the message b, and circulating the steps. For another example, when the received message migration instruction is a message migration instruction for all messages (including message a, message b, and message c), the producer deletes all messages according to the deletion instruction for all messages after producing all messages to the master device.
Alternatively, the migration message deletion instruction may be sent by the consumer (the application program for deleting the corresponding message on the service device). The number of consumers may correspond to the number of producers producing the message.
The consumer and the producer for producing messages may be applications on a service device that performs the producing and deleting steps, or may be applications on devices other than the service device.
Furthermore, in order to reduce the time delay generated by establishing connection between each producer and the service device and speed up the message sending speed of the service device to the main device, the number of producers and consumers in the embodiment of the invention can be the same as the number of destination objects of the message to be migrated, and then the consumers and the producers corresponding to the destination objects can send the message back to the main device in parallel, that is, one service device can send a plurality of messages to the main device at the same time, thereby further shortening the time occupied by the index file and further ensuring the real-time of the ActiveMQ service. The destination object is an object for which the messages to be migrated are directed, and a plurality of messages to be migrated are directed to one destination object. That is, messages of the same destination object share one producer.
Corresponding to the method embodiment of the first aspect, an embodiment of the present invention further provides an index file generating apparatus, as shown in fig. 4, where the apparatus is applied to a master device in an index file generating system, the index file generating system further includes at least two service devices, and the apparatus includes:
an obtaining module 410, configured to obtain all local log files for storing messages;
a sending module 420, configured to group the log files according to a preset rule and send the group to each service device, so that each service device synchronously sends a message in the received log file back to the master device;
the receiving module 430 is configured to receive and store the message sent by each service device, and generate an index file of the message sent by each service device.
The index file generation apparatus provided in an embodiment of the present invention allocates the log file for storing messages on the master device to at least two service devices in the index file generation system, so that each service device synchronously sends back the messages stored in the log file, which is received by each service device and is grouped by the master device, to the master device again. Compared with the prior art that the messages in the day files are scanned one by one and the indexes are established one by one, the device provided by the embodiment of the invention can accelerate the generation speed of the index file, shorten the time for generating the index file, avoid the problem that the normal receiving and sending of the messages are influenced because the ActiveMQ spends a long time to establish the index file in the prior art, and ensure the real-time property of the ActiveMQ service.
Optionally, the sending module 420 includes:
the first dividing module is used for averagely dividing the log files into log groups with corresponding number according to the number of the service devices;
and the first sending submodule is used for respectively sending each group of log files to each service device.
Optionally, the sending module 420 includes:
the first determining submodule is used for acquiring the current load of each service device and determining the number of log files to be sent to each service device according to the acquired current load of each service device and the total number of the log files;
the second division submodule is used for grouping the log files according to the determined number of the log files to be sent to each service device;
and the second sending submodule is used for sending the log packets containing the corresponding number of log files to the corresponding service equipment.
Optionally, the receiving module 430 includes:
the receiving submodule is used for receiving and storing messages sent by each service device;
the building submodule is used for parallelly obtaining the offset address of each message and building a B tree according to the obtained offset address;
and the second determining submodule is used for storing the B tree into a new file and determining the new file as an index file.
Corresponding to the method embodiment of the second aspect, an embodiment of the present invention further provides an index file generating apparatus, as shown in fig. 5, where the apparatus is applied to any service device in an index file generating system, the index file generating system further includes a master device, and the apparatus includes:
a receiving module 510, configured to receive a log file sent by the master device, where the log file is sent to the service device after the master device groups all local log files for storing messages according to a preset rule;
the generating module 520 receives a message migration instruction, and sends the message to be migrated to the main device, to which the message migration instruction in the received log file is directed, so that the main device stores the message to be migrated.
In the index file generation apparatus provided in the embodiment of the present invention, after receiving the message migration instruction, the service device in the index file generation system sends the message to be migrated in the received log file, to which the message migration instruction is directed, back to the master device. Compared with the prior art that messages in the log file are scanned one by one and the index of the messages is established one by one, the device provided by the embodiment of the invention can accelerate the generation speed of the index file, shorten the time for generating the index file, avoid the problem that the normal receiving and sending of the messages are influenced because the ActiveMQ spends a long time to establish the index file in the prior art, and ensure the real-time property of the ActiveMQ service.
Optionally, the generating module 520 includes:
the obtaining submodule receives a message migration instruction and obtains a producer and a consumer of the message to be migrated, wherein the message to be migrated is a received message in the log file, the consumer is an application program which obtains the message and sends the obtained message to the producer, and the producer is an application program which receives the message sent by the consumer, produces the message and sends the produced message to the main device;
and the production module is used for operating each producer and each consumer, so that each consumer sends the message aimed at to the producer of the message, and the producer produces the message and sends the produced message to the master equipment.
Optionally, the number of the producers is the same as the number of the destination objects of the message to be migrated, where the destination object is an object for which the message to be migrated is directed, and multiple messages to be migrated are directed to one destination object.
An embodiment of the present invention further provides an index file generating system, as shown in fig. 6, where the system includes a main device 610 and at least two service devices 620;
the main device 610 is configured to execute an index file generation method corresponding to each inventive embodiment of the first aspect;
each of the service devices 620 is configured to execute an index file generation method corresponding to each inventive embodiment of the second aspect.
The index file generation system provided by the embodiment of the invention allocates the log file for storing the message on the main device to at least two service devices in the index file generation system, causing each service device to resend the message in the assigned log file back to the master device, since the service devices simultaneously transmit messages to the master device, the master device simultaneously receives and stores at least two messages, therefore, the index of at least two messages can be established in parallel, compared with the prior art that the messages in the log file are scanned one by one and the index of the messages is established one by one, the system provided by the embodiment of the invention can accelerate the generation speed of the index file, shorten the time for generating the index file, avoid that the ActiveMQ in the prior art takes a long time to establish the index file, the problem of influencing the normal receiving and sending of the message is solved, and the real-time performance of the ActiveMQ service is ensured.
For the apparatus/system embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and reference may be made to some descriptions of the method embodiment for relevant points.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments.
The above description is only for the preferred embodiment of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention shall fall within the protection scope of the present invention.
Claims (15)
1. An index file generation method is applied to a main device in an index file generation system, wherein the index file generation system further comprises at least two service devices, and the method comprises the following steps:
acquiring all local log files for storing messages;
grouping the log files according to a preset rule and then sending the log files to each service device so that each service device synchronously sends the received messages in the log files back to the master device;
receiving and storing messages sent by each service device, and generating an index file of the messages sent by each service device; the message comprises: content communicated between applications on the same device or content communicated between applications on different devices.
2. The method according to claim 1, wherein the step of grouping the log files according to a preset rule and then sending the grouped log files to each service device comprises:
according to the number of the service devices, averagely dividing the log file into a corresponding number of log groups;
and respectively sending each group of log files to each service device.
3. The method according to claim 1, wherein the step of grouping the log files according to a preset rule and then sending the grouped log files to each service device comprises:
acquiring the current load capacity of each service device, and determining the number of log files to be sent to each service device according to the acquired current load capacity of each service device and the total number of the log files;
grouping the log files according to the determined number of the log files to be sent to each service device;
and sending the log packets containing the corresponding number of log files to the corresponding service equipment.
4. The method according to any one of claims 1 to 3, wherein the step of receiving and storing the messages sent by each service device and generating the index file of the messages sent by each service device comprises:
receiving and storing messages sent by each service device;
parallelly acquiring offset addresses of all messages, and establishing a B-tree according to the acquired offset addresses;
and storing the B tree into a newly-built file, and determining the newly-built file as an index file.
5. An index file generation method is applied to any service device in an index file generation system, the index file generation system further comprises a main device, and the method comprises the following steps:
receiving a log file sent by the master device, wherein the log file is sent to the service device after the master device groups all local log files for storing messages according to a preset rule;
receiving a message migration instruction, and sending a message to be migrated to the primary device, to which the message migration instruction in the received log file is directed, so that the primary device stores the message to be migrated; the message comprises: content communicated between applications on the same device or content communicated between applications on different devices.
6. The method according to claim 5, wherein the step of receiving a message migration instruction, and sending the message to be migrated to the primary device in the received log file, to which the message migration instruction is directed, so that the primary device stores the message to be migrated includes:
receiving a message migration instruction, and acquiring a producer and a consumer of the message to be migrated, wherein the message to be migrated is a message in the received log file, the consumer is an application program which acquires the message and sends the acquired message to the producer, and the producer is an application program which receives the message sent by the consumer, produces the message, and sends the produced message to the primary device;
each producer and consumer is operated to have each consumer send the message for which it is intended to the producer of that message, so that the producer produces the message and sends the produced message to the master device.
7. The method according to claim 6, wherein the number of producers and the number of consumers are the same as the number of destination objects of the message to be migrated, wherein the destination objects are objects for which the message to be migrated is directed, and a plurality of messages to be migrated are directed to one destination object.
8. An index file generation device, which is applied to a master device in an index file generation system, wherein the index file generation system further includes at least two service devices, and the index file generation device includes:
the obtaining module is used for obtaining all local log files for storing messages;
the sending module is used for grouping the log files according to a preset rule and then sending the log files to each service device so that each service device can synchronously send the received messages in the log files back to the main device;
and the receiving module is used for receiving and storing the messages sent by each service device and generating an index file of the messages sent by each service device.
9. The apparatus of claim 8, wherein the sending module comprises:
the first dividing module is used for averagely dividing the log files into log groups with corresponding number according to the number of the service devices;
and the first sending submodule is used for respectively sending each group of log files to each service device.
10. The apparatus of claim 8, wherein the sending module comprises:
the first determining submodule is used for acquiring the current load of each service device and determining the number of log files to be sent to each service device according to the acquired current load of each service device and the total number of the log files;
the second division submodule is used for grouping the log files according to the determined number of the log files to be sent to each service device;
and the second sending submodule is used for sending the log packets containing the corresponding number of log files to the corresponding service equipment.
11. The apparatus according to any one of claims 8-10, wherein the receiving module comprises:
the receiving submodule is used for receiving and storing messages sent by each service device;
the building submodule is used for parallelly obtaining the offset address of each message and building a B tree according to the obtained offset address;
and the second determining submodule is used for storing the B tree into a new file and determining the new file as an index file.
12. An index file generation device, which is applied to any service device in an index file generation system, wherein the index file generation system further includes a master device, and the index file generation device includes:
the receiving module is used for receiving the log file sent by the main equipment, wherein the log file is sent to the service equipment after the main equipment groups all local log files for storing messages according to a preset rule;
the generating module is used for receiving a message migration instruction, and sending a message to be migrated to the main device, which is in the received log file and is aimed at by the message migration instruction, so that the main device stores the message to be migrated.
13. The apparatus of claim 12, wherein the generating module comprises:
the obtaining submodule is used for receiving a message migration instruction, obtaining a producer and a consumer of the message to be migrated, wherein the message to be migrated is a received message in the log file, the consumer is an application program which obtains the message and sends the obtained message to the producer, and the producer is an application program which receives the message sent by the consumer, produces the message and sends the produced message to the main device;
and the production module is used for operating each producer and each consumer, so that each consumer sends the message aimed at to the producer of the message, and the producer produces the message and sends the produced message to the master equipment.
14. The apparatus of claim 13, wherein the number of producers is the same as the number of destination objects of the message to be migrated, and wherein the destination objects are objects for which the message to be migrated is intended, and wherein a plurality of messages to be migrated are intended for one destination object.
15. An index file generation system is characterized by comprising a main device and at least two service devices;
the master device, configured to execute an index file generation method according to any one of claims 1 to 4;
each of the service devices is configured to execute an index file generation method according to any one of claims 5 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710207991.4A CN107179953B (en) | 2017-03-31 | 2017-03-31 | Index file generation method, device and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710207991.4A CN107179953B (en) | 2017-03-31 | 2017-03-31 | Index file generation method, device and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107179953A CN107179953A (en) | 2017-09-19 |
| CN107179953B true CN107179953B (en) | 2020-04-03 |
Family
ID=59829825
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710207991.4A Active CN107179953B (en) | 2017-03-31 | 2017-03-31 | Index file generation method, device and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107179953B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1877582A (en) * | 2006-07-12 | 2006-12-13 | 百度在线网络技术（北京）有限公司 | Advertisement information retrieval system and method therefor |
| CN102768845B (en) * | 2011-05-03 | 2015-03-11 | ***通信集团公司 | Video index method and system |
| CN105988996A (en) * | 2015-01-27 | 2016-10-05 | 腾讯科技（深圳）有限公司 | Index file generation method and device |
-
2017
- 2017-03-31 CN CN201710207991.4A patent/CN107179953B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1877582A (en) * | 2006-07-12 | 2006-12-13 | 百度在线网络技术（北京）有限公司 | Advertisement information retrieval system and method therefor |
| CN102768845B (en) * | 2011-05-03 | 2015-03-11 | ***通信集团公司 | Video index method and system |
| CN105988996A (en) * | 2015-01-27 | 2016-10-05 | 腾讯科技（深圳）有限公司 | Index file generation method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107179953A (en) | 2017-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103067525B (en) | A kind of cloud storing data backup method of feature based code | |
| CN110737658A (en) | Data fragment storage method, device, terminal and readable storage medium | |
| CN102938784A (en) | Method and system used for data storage and used in distributed storage system | |
| CN102419770A (en) | File sharing system, method for realizing file sharing, and file index service equipment | |
| CN105069111A (en) | Similarity based data-block-grade data duplication removal method for cloud storage | |
| CN107220271B (en) | Method and system for storage processing and management of distributed digital resources | |
| CN104881466A (en) | Method and device for processing data fragments and deleting garbage files | |
| CN105975493A (en) | File combination method and apparatus | |
| US10700934B2 (en) | Communication control device, communication control method, and computer program product | |
| CN111447265A (en) | File storage method, file downloading method, file processing method and related components | |
| CN104956340A (en) | Scalable data deduplication | |
| US20220129333A1 (en) | Message Processing Method and Apparatus in Distributed System | |
| CN103744882A (en) | Catalogue fragment expressing method and device based on key value pair | |
| CN110109751B (en) | Distribution method and device of distributed graph cutting tasks and distributed graph cutting system | |
| CN103905512A (en) | Data processing method and equipment | |
| CN107179953B (en) | Index file generation method, device and system | |
| CN111444148A (en) | Data transmission method and device based on MapReduce | |
| CN110347654B (en) | Method and device for online cluster characteristics | |
| EP3519985A1 (en) | Flexible in-memory column store placement | |
| CN109992575B (en) | Distributed storage system for big data | |
| EP2765517B1 (en) | Data stream splitting for low-latency data access | |
| CN112115206A (en) | Method and device for processing object storage metadata | |
| CN109144991B (en) | Method and device for dynamic sub-metering, electronic equipment and computer-storable medium | |
| CN116089527A (en) | Data verification method, storage medium and device | |
| CN112073395B (en) | File distribution method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |