CN110109751B - Distribution method and device of distributed graph cutting tasks and distributed graph cutting system - Google Patents
Distribution method and device of distributed graph cutting tasks and distributed graph cutting system Download PDFInfo
- Publication number
- CN110109751B CN110109751B CN201910267201.0A CN201910267201A CN110109751B CN 110109751 B CN110109751 B CN 110109751B CN 201910267201 A CN201910267201 A CN 201910267201A CN 110109751 B CN110109751 B CN 110109751B
- Authority
- CN
- China
- Prior art keywords
- map
- tile
- task execution
- distributed
- cutting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/48—Program initiating; Program switching, e.g. by interrupt
- G06F9/4806—Task transfer initiation or dispatching
- G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
- G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to the field of image processing, in particular to a distributed map cutting task allocation method, a distributed map cutting task allocation device and a distributed map cutting system, wherein the method comprises the following steps: the method comprises the steps of obtaining space distribution information of all map-width files within a to-be-mapped range, coding the map-width files by adopting a space filling curve according to the space distribution information, and distributing the map-width files to task execution ends in sequence according to the codes. The embodiment of the invention is suitable for a distributed graph cutting system and task allocation of the distributed graph cutting, can effectively reduce the generation quantity of tile fragments, and improves the fragment merging efficiency, thereby improving the graph cutting efficiency.
Description
Technical Field
The invention relates to the field of image processing, in particular to a distributed map cutting task allocation method, a distributed map cutting task allocation device and a distributed map cutting system.
Background
The spatial image data mainly refers to: and various types of image data are obtained in modes of airplane aerial photography, satellite remote sensing and the like. Such data is typically organized in units of picture frames in production. Generally, the range of the frame data is large, and the framed image data needs to be cut into smaller rectangular pictures, hereinafter collectively referred to as tiles, according to a certain rule. The segmented tiles are more suitable for image data online browsing, caching and concurrent data processing operations. The space image data generally has a very large data volume and a relatively fast data updating speed, so the time effectiveness requirement of the space image data mapping on the technology is very high. The traditional single-machine graph switching scheme is limited by the hardware resource bottleneck of the single machine. Meanwhile, the cost is very high due to the fact that the computing resources of a single machine are longitudinally lifted. Therefore, the single graph cutting scheme is difficult to improve the graph cutting efficiency. Since the space image cutout requires a large amount of computing resources, it is necessary to provide a cutout method with cheap computing resources, simple computing power expansion and high computing performance. The following two solutions are currently used:
a single map-switching scheme. The scheme uses a single machine to start a plurality of map cutting processes, and performs map cutting in a single-machine multi-process mode by taking a map sheet as a unit. The main drawbacks of this solution are: the hardware resource of a single machine becomes a bottleneck, and the cost for improving the computing performance of the single machine is too high.
The traditional multi-machine parallel graph cutting scheme. Compared with the first single machine scheme, the scheme overcomes the bottleneck of single machine computing resources and improves the graph cutting efficiency. The main drawbacks of this solution are: in the distributed processing, the time consumption for splicing the fragments is very high, and the overall graph cutting efficiency is influenced finally. The traditional distributed scheme does not perform special processing on fragments generated in the graph cutting, so that the fragment processing becomes a bottleneck, and the efficiency of the whole graph cutting task is restricted.
The distributed graph cutting scheme can present tile fragment handling challenges. During the drawing cutting process, each drawing frame file generates a large amount of tile fragments at the edge part of the drawing frame, and the tile fragments need to be spliced with the fragments on other drawing frames to form a complete tile. In the distributed scheme, it is the splicing of these fragmented data that limits the efficiency of graph cutting. In the traditional mode, fragments are stored in a database, and after all the image files are processed, the splicing operation of the fragments is uniformly carried out. During this time, a large amount of network transmission occurs, and a large amount of time is taken for the fragmentation to be reassembled. The efficiency of the map cutting is relatively low.
Disclosure of Invention
The invention aims to provide a distribution method of a distributed map cutting task, which is characterized in that by utilizing the characteristic of spatial proximity of spatial image data, map files which are close to each other in space are preferentially distributed to the same server for cutting, and at least the problem of large generation amount of tile fragments in the map cutting process is solved.
In order to achieve the above object, an embodiment of a first aspect of the present invention provides a method for allocating distributed graph cutting tasks, including: acquiring the spatial distribution information of all the map width files in the range of the to-be-cut map, and further comprising the following steps of:
and coding the image frame file by adopting a space filling curve according to the space distribution information, and distributing the image frame file to a task execution end in sequence according to the coding.
Optionally, the space-filling curve is a hilbert space-filling curve, and the encoding is hilbert encoding.
Optionally, the encoding the map-frame file by using a space filling curve according to the spatial distribution information includes:
obtaining the map space distribution according to the space distribution information;
mapping the map sheet space distribution to a two-dimensional coordinate system to form a map sheet file two-dimensional code;
and reducing the dimension of the two-dimensional code of the map-frame file into a one-dimensional sequence by adopting a Hilbert space filling curve to obtain the code.
Optionally, the sequentially allocating the chart file to the task execution end according to the codes includes:
calculating the number of fragment bytes to be generated by each map frame file;
and sequentially distributing the map frame files according to the memory information of each task execution end.
Optionally, sequentially allocating the map frame files according to the memory information of each task execution end includes:
a1, comparing whether the number of fragment bytes of the current map frame file to be distributed is less than the memory allowance of the current task execution end;
if the current to-be-distributed map file is smaller than the preset value, distributing the current to-be-distributed map file to a current task execution end, and turning A2;
if not, selecting the next task execution end as the current task execution end, and turning to A1;
a2, judging whether the map file is distributed completely, if not, selecting the next map file as the current map file to be distributed, and turning to A1.
Optionally, before the step a1, obtaining a memory margin of the current task execution end.
Optionally, the method further includes: and acquiring the current online state of the task execution end according to the heartbeat information generated by the task execution end.
The embodiment of the second aspect of the present invention provides an apparatus for distributing distributed map cutting tasks, where the apparatus is configured to distribute distributed map cutting tasks according to the foregoing method.
Optionally, the device is a server.
The embodiment of the third aspect of the invention provides a distributed graph cutting system, which includes the distribution device as described above, and at least two task execution terminals, where the distribution device is connected to the task execution terminals in communication.
Optionally, the distribution device and the task execution end are both servers, the distribution device is a distribution server, and the task execution end is a graph cutting server.
Optionally, the graph cutting server reports heartbeat information and the memory margin to the distribution server.
Optionally, the map cutting server performs tile fragment merging after the map cutting of the map-width file is completed, where the tile fragment merging includes: and merging the tile fragments belonging to the same tile ID in the memory.
Optionally, the graph cutting server is further configured to, after performing tile fragment merging, if tile fragments still exist, perform tile fragment mapping, where the tile fragment mapping includes: and mapping the tile IDs of the tile fragments to the corresponding map cutting server IDs in batches, and then respectively transmitting the tile fragments to the map cutting servers corresponding to the map cutting server IDs mapped by the tile IDs of the tile fragments.
Optionally, the mapping tile IDs to which the tile fragments belong in batch to the respective corresponding map cutting server IDs includes: and HASH is carried out on the tile IDs to the set formed by the cutting server IDs.
Optionally, after the respectively transmitting the tile fragments to the mapped graph cutting servers, the method further includes: the graph cutting server carries out tile fragment combination again; and waiting for the next tile fragment mapping and/or tile fragment merging of the tile fragments still remaining after the merging.
A fourth aspect of the present invention provides a storage medium, which stores instructions that, when executed on a computer, cause the computer to execute the method for distributing distributed graph cutting tasks as described above.
A fourth aspect of the present invention provides a storage medium having stored therein instructions that, when run on a computer, cause the computer to perform an audio isolation detection method as described above.
Through the technical scheme, the invention provides the distribution method and the distribution device for the distributed graph cutting task and the distributed graph cutting system, which not only can fully utilize the computing resources of a plurality of machines to complete the parallel graph cutting task, but also can effectively reduce the generation of tile fragments and the transmission quantity of the tile fragments in a network, and simultaneously can utilize the characteristic of quick memory read-write to improve the efficiency of fragment combination.
Drawings
FIG. 1 is a flow chart of a method for distributing a distributed graph cutting task according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of the generation and consolidation of tile fragments according to an alternative embodiment of the present invention;
FIG. 3 is a schematic diagram of an encoding scheme of a frame file according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a distributed graph cutting system according to an embodiment of the present invention.
Detailed Description
The following detailed description of embodiments of the invention refers to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the present invention, are given by way of illustration and explanation only, not limitation.
Fig. 1 is a schematic flow chart of a distribution method of a distributed graph cutting task according to an embodiment of the present invention. As shown in fig. 1, the method comprises the steps of: acquiring the spatial distribution information of all the map-width files in the range of the to-be-cut map (this step is not shown), and further comprising: and coding the image frame file by adopting a space filling curve according to the space distribution information, and distributing the image frame file to a task execution end in sequence according to the coding.
Therefore, the image map files are sequenced and encoded by utilizing the characteristic of spatial proximity of the image map files, the map files close to each other in space are ensured to be processed in a centralized manner, and the performance consumption of tile fragments in circulation among a plurality of machines is greatly reduced.
Specifically, fig. 2 is a schematic diagram of the generation and combination of tile fragments for map cutting provided by an alternative embodiment of the present invention, as shown in fig. 2, during the map cutting process, each map frame is not necessarily cut into an integral number of tiles. The tiles at the boundaries of the respective drawings are incomplete during the drawing cutting process because the portions of the tiles are located in different drawings, and are referred to as tile fragments, i.e., the vertical line shadow filling part and the horizontal line shadow filling part in fig. 2. Therefore, after one-time drawing cutting is completed, all the parts of the tile fragments need to be spliced into a complete tile. In the distributed mapping process, the four image data (map files 1-4) are likely to be distributed on different machines, and in this case, the fragment merging needs to be performed by collecting the fragments to be merged into the same machine.
If the adjacent map sheets are allocated to the same map cutting server for map cutting under the condition of permission, the generation of tile fragments is greatly reduced. Also for example, in FIG. 2, if FIG. 1 and FIG. 2 are assigned to the same graph-cut server, tile fragments generated by adjacent partitions (i.e., the vertical line shadow filled portions) will be merged in the local memory, and need not be circulated among multiple machines, causing additional performance consumption.
Further, the space-filling curve is a hilbert space-filling curve, and the encoding is hilbert encoding. Space-filling curves are curves that are one-dimensional curves to include a function curve of the entire two-dimensional or even multi-dimensional space. And different space filling curves can be obtained according to different arrangement rules. Common space-filling curves are the Z-order curve (i.e., the curve taken by the geohash), or the Peano curve, and the Hilbert curve preferred herein. Both the Hilbert curve and its discrete approximation method are very practical because the method of transforming a multidimensional space into a one-dimensional space preserves spatial proximity well. Because of this proximity, space-filling curves are widely used in computer science, and in multidimensional database indexing, Z-order curves are often replaced with Hilbert curves.
Further, the encoding the map-frame file by using a space filling curve includes: obtaining the map space distribution according to the space distribution information; mapping the map sheet space distribution to a two-dimensional coordinate system to form a map sheet file two-dimensional code; and reducing the dimension of the two-dimensional code of the map-frame file into a one-dimensional sequence by adopting a Hilbert space filling curve to obtain the code.
Fig. 3 is a schematic diagram of a coding scheme of a map-frame file according to an embodiment of the present invention, as shown in fig. 3, the map-frame spatial distribution is mapped into a two-dimensional coordinate system, if the map-frame file 1 at the lower left corner is coded as (0, 0), the map-frame code at the right side thereof is shifted by 1 bit toward the X axis, the obtained code is (0, 1), and then the map-frame file 2 is coded as (1, 1) with respect to the map-frame file 1, and similarly, the two-dimensional codes of the map-frame file 3 and the map-frame file 4 can be obtained. And then reducing the dimension of the two-dimensional code of the map-frame file into a one-dimensional sequence by using a Hilbert space filling curve, so as to obtain the rightmost Hilbert code (Hilbert code) in the figure 3.
In the method provided by an embodiment of the present invention, the sequentially allocating the chart files to task execution ends according to the codes includes: calculating the number of fragment bytes to be generated by each map frame file; and sequentially distributing the map frame files according to the memory information of each task execution end.
Specifically, the number of bytes of fragments generated by each image file is calculated according to the size information (length TileLong, width tilelode) of each image file, the tile size information (length pielong, width pielong), and the tile pixel information (PixelLong, PixelWide), that is, the maximum inscribed rectangle formed by N tiles is subtracted according to the image file area, so that the remaining area is obtained; and multiplying the area of the residual area by the tile pixel information (the number of bytes corresponding to the unit area of the tile) to obtain the number of bytes corresponding to the area of the residual area. And then sequentially distributing the map files according to the memory information of each task execution end, namely ensuring that fragments generated by each batch of data (namely one or more map files) do not exceed the memory size of the map cutting server. The task scheduling is carried out based on the memory allowance of each single server in the server cluster, each batch of distributed tasks is guaranteed, all tile fragments can be cached in the memory, the tile fragments are prevented from falling to the disk, slow retrieval and loading of the disk are reduced, high-speed reading or retrieval based on the memory is guaranteed all the time, and therefore the efficiency of fragment combination is improved.
Further, the sequentially allocating the map files according to the memory information of each task execution end includes:
a1, comparing whether the number of fragment bytes of the current map frame file to be distributed is less than the memory allowance of the current task execution end;
if the current to-be-distributed map file is smaller than the preset value, distributing the current to-be-distributed map file to a current task execution end, and turning A2;
if not, selecting the next task execution end as the current task execution end, and turning to A1;
a2, judging whether the map file is distributed completely, if not, selecting the next map file as the current map file to be distributed, and turning to A1.
Specifically, whether the number of fragment bytes of the current to-be-allocated map file is smaller than the memory margin of the current task execution end is compared, and if so, the current to-be-allocated map file is allocated to the current task execution end. After the assignment, the current task execution end will have a smaller memory margin due to the new loading of a frame file. And then allocating the next chart file to be allocated by the same method until the number of fragment bytes of the chart file to be allocated is larger than or equal to the memory allowance of the current task execution end, at the moment, the chart file is not allocated to the task execution end, the next task execution end is allocated, and the previous steps are repeated until all the chart files are allocated. Because the number of fragment bytes is generally not too large, the situation that the number of fragment bytes cannot be matched with the proper task execution end does not occur, even if the fragment bytes cannot be matched with the proper task execution end, when the graph cut of the task execution end is completed, the memory is released, the memory margin is increased, and the proper task execution end is easily matched at the moment.
Further, before the step a1, the memory margin of the current task execution end is obtained. In a specific embodiment, the task allocation server acquires the memory margin information of all task execution terminals (specifically, the graph cutting servers), and maintains the mapping relationship between the ID of each graph cutting server and the memory margin. The method further comprises the following steps: and acquiring the current online state of the task execution end according to the heartbeat information generated by the task execution end.
The embodiment of the invention also provides a distribution device of the distributed map cutting task, which comprises an acquisition module, a mapping module and a mapping module, wherein the acquisition module is used for acquiring the spatial distribution information of all map width files within the range to be mapped; further comprising:
the encoding module is used for encoding the image frame file by adopting a space filling curve according to the space distribution information; and
and the distribution module is used for sequentially distributing the chart file to the task execution end according to the codes.
Wherein the space-filling curve is a Hilbert space-filling curve, and the code is Hilbert code.
The encoding module encodes the map-frame file by adopting a space filling curve according to the space distribution information, and comprises the following steps: obtaining the map space distribution according to the space distribution information; mapping the map sheet space distribution to a two-dimensional coordinate system to form a map sheet file two-dimensional code; and reducing the dimension of the two-dimensional code of the map-frame file into a one-dimensional sequence by adopting a Hilbert space filling curve to obtain the code.
The distribution module distributes the chart file to the task execution end according to the codes in sequence, and the distribution module comprises: calculating the number of fragment bytes to be generated by each map frame file; and sequentially distributing the map frame files according to the memory information of each task execution end.
The distribution module distributes the map files in sequence according to the memory information of each task execution end, and the distribution module comprises:
a1, comparing whether the number of fragment bytes of the current map frame file to be distributed is less than the memory allowance of the current task execution end;
if the current to-be-distributed map file is smaller than the preset value, distributing the current to-be-distributed map file to a current task execution end, and turning A2;
if not, selecting the next task execution end as the current task execution end, and turning to A1;
a2, judging whether the map file is distributed completely, if not, selecting the next map file as the current map file to be distributed, and turning to A1.
Before a1, obtaining the memory margin of the current task execution end.
The device further comprises: and the state acquisition module is used for acquiring the current online state of the task execution end according to the heartbeat information generated by the task execution end.
The embodiment of the invention also provides a distributed map cutting task distribution device, which is configured to distribute the distributed map cutting tasks according to the method. The apparatus herein is a hardware entity for executing the foregoing method, and further, the apparatus is a server. Servers are widely used due to their stable performance and powerful processing power.
The embodiment of the invention also provides a distributed graph cutting system. The task execution device comprises the distribution device and at least two task execution terminals, wherein the distribution device is in communication connection with the task execution terminals. The distributed graph cutting scheme can fully utilize the computing resources of a plurality of machines and complete the graph cutting task in parallel. Meanwhile, according to the task amount and the timeliness requirement, computing resources can be dynamically increased, and the hardware bottleneck is avoided.
In another optional embodiment of the present invention, the distribution device and the task execution end are both servers, the distribution device is a distribution server, and the task execution end is a graph cutting server. Fig. 4 is a schematic structural diagram of a distributed graph cutting system according to an embodiment of the present invention. As shown in fig. 4, the distribution server and at least two map cutting servers are connected through a network and cooperate to execute the map cutting task. In the situation that the stability and the specialty of the system are not required, the processing equipment can be a PC, so that the system is more convenient to deploy and implement and is lower in cost.
Further, the graph cutting server reports heartbeat information and the memory margin to the distribution server. The map cutting server enables the distribution server to know whether the distribution server is online or not and whether the operation is normal or not by transmitting heartbeat packets. And simultaneously reporting the memory margin information of the distribution server, so that when the distribution server distributes the map file, fragments generated by the distributed map file can be ensured to be smaller than the memory margin of the map cutting server. At this time, the allocation server may maintain the information such as the status or address of each online graph cutting server and the memory margin information of the online graph cutting server by establishing a configuration table.
In an embodiment of the present invention, the graph cutting server performs tile fragment merging after the graph cutting of the map-width file is completed, where the tile fragment merging includes: and merging the tile fragments belonging to the same tile ID in the memory. The merging is mainly merging within the graph cutting server, more specifically, merging is performed in the memory of the server, and the merging is based on the same tile ID. The map tiles are organized in a pyramid structure, and each tile can be uniquely marked by a level, a row and a column number, and the unique mark is used as a tile ID.
In the last step, after merging inside a single graph cutting server, if the merging is followed by complete fragmentation, the tiled tiles are directly stored in the storage medium, but there is a high possibility that tile fragmentation still exists because another part of the tile fragmentation is on another graph cutting server, where the tile fragmentation needs to be reallocated after being aggregated and allocated to a pending graph cutting server for further merging. Therefore, we need to go to the next step, namely: the map cut server is further configured to, after performing tile fragment merging, if tile fragments also exist, perform a tile fragment map comprising: and mapping the tile IDs of the tile fragments to the corresponding map cutting server IDs in batches, and then respectively transmitting the tile fragments to the map cutting servers corresponding to the map cutting server IDs mapped by the tile IDs of the tile fragments. The tile ID batches and the tile server IDs are mapped so that each tile server can be assigned to a tile fragment, making full use of multiple servers for parallel processing for load balancing and balancing processing power among the servers.
The aforementioned tile fragment mapping may also be optimized as: the mapping the tile IDs of the tile fragments to the corresponding cutting server IDs in batches comprises the following steps: and HASH is carried out on the tile IDs to the set formed by the cutting server IDs. The set of cutmap server IDs here includes the IDs of all the cutmap servers that are currently online. Hash, which is generally translated as a Hash, or transliteration, is a process of converting an input of arbitrary length (also called pre-map image) into an output of fixed length by a hashing algorithm, where the output is a Hash value. The hash value has a deterministic result. This transformation is a kind of compression mapping, i.e. the space of hash values is usually much smaller than the space of inputs, different inputs may hash to the same output, so it is not possible to determine a unique input value from a hash value. In short, it is a function of compressing a message of an arbitrary length to a message digest of a certain fixed length. The mapping is carried out after the Hash, so that the size of mapping data can be reduced, tile fragments can be more balanced when being mapped and distributed on the graph cutting server, and meanwhile, all tile IDs can be mapped to one graph cutting server ID.
Further, after the tile fragments are respectively transmitted to the mapped cutting servers, the method further includes: the graph cutting server carries out tile fragment combination again; and waiting for the next tile fragment mapping and/or tile fragment merging of the tile fragments still remaining after the merging.
After the tile fragments are transmitted in the previous step, the tile fragments in the memory of each graph cutting server are updated and further collected, and the tile fragments are preferably combined once after the updating and the collection are finished; the tile fragment combination can be performed again after the next graph cutting task is completed without performing the tile fragment combination. Tile fragment merging here can further reduce tile fragments in time. Whether the tile fragment combination is performed or not, the tile fragments still exist in the memory (the ideal situation that no tile fragment exists is almost avoided), and the tile fragments wait for the next tile fragment mapping and/or tile fragment combination. The merging of the tile fragments is carried out in the memory, and the merging speed and efficiency of the tile fragments are obviously improved.
Embodiments of the present invention also provide a storage medium having stored therein instructions, which when run on a computer, cause the computer to execute the distribution method of the distributed graph cutting task as described above.
Therefore, the embodiment of the invention can effectively reduce the generation quantity of the tile fragments and improve the fragment merging efficiency, thereby improving the graph cutting efficiency. The method is applied to real graph cutting tasks at present, the efficiency of the method is improved by 20 times compared with a single graph cutting scheme, and the performance of the method is improved by 200% -300% compared with the traditional distributed graph cutting scheme.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In a typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). The memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, a computer readable medium does not include a transitory computer readable medium such as a modulated data signal and a carrier wave.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in the process, method, article, or apparatus that comprises the element.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The above are merely examples of the present application and are not intended to limit the present application. Various modifications and changes may occur to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims (17)
1. A distribution method of distributed graph cutting tasks is applied to a task distribution end in a distributed graph cutting system, wherein the distributed graph cutting system comprises the task distribution end and at least two task execution ends; wherein the task execution end is configured to: merging the tile fragments formed after the map of the map-frame file is cut; when the tile fragments needing to be combined are not at the same task execution end, exchanging the tile fragments; the method comprises the following steps: the method for obtaining the space distribution information of all the map width files in the range of the to-be-cut map is characterized by further comprising the following steps:
and coding the image frame file by adopting a space filling curve according to the space distribution information, and distributing the image frame file to a task execution end in sequence according to the coding.
2. The method of claim 1, wherein the space-filling curve is a hilbert space-filling curve and the encoding is hilbert encoding.
3. The method of claim 2, wherein encoding the map-file with a space-filling curve according to the spatial distribution information comprises:
obtaining the map space distribution according to the space distribution information;
mapping the map sheet space distribution to a two-dimensional coordinate system to form a map sheet file two-dimensional code;
and reducing the dimension of the two-dimensional code of the map-frame file into a one-dimensional sequence by adopting a Hilbert space filling curve to obtain the code.
4. The method of claim 1, wherein said sequentially distributing said banner files to task execution ends according to said encoding comprises:
calculating the number of fragment bytes to be generated by each map frame file;
and sequentially distributing the map frame files according to the memory information of each task execution end.
5. The method according to claim 4, wherein said sequentially allocating the map files according to the memory information of each task execution end comprises:
a1, comparing whether the number of fragment bytes of the current map frame file to be distributed is less than the memory allowance of the current task execution end;
if the current to-be-distributed map file is smaller than the preset value, distributing the current to-be-distributed map file to a current task execution end, and turning A2;
if not, selecting the next task execution end as the current task execution end, and turning to A1;
a2, judging whether the map file is distributed completely, if not, selecting the next map file as the current map file to be distributed, and turning to A1.
6. The method according to claim 5, wherein before A1, the method further comprises obtaining the memory margin of the current task execution end.
7. The method of claim 1, further comprising: and acquiring the current online state of the task execution end according to the heartbeat information generated by the task execution end.
8. An apparatus for distributing distributed cut tasks, the apparatus being configured to distribute distributed cut tasks according to the method of any of claims 1-7.
9. The apparatus of claim 8, wherein the apparatus is a server.
10. A distributed graph cutting system, characterized in that the system comprises a distribution device as claimed in claim 8, and at least two task execution terminals, said distribution device being communicatively connected to said task execution terminals.
11. The system according to claim 10, wherein the distribution device and the task execution end are both servers, the distribution device is a distribution server, and the task execution end is a graph cutting server.
12. The system of claim 11, wherein the graph cutting server reports heartbeat information and memory margin to the distribution server.
13. The system of claim 12, wherein the graph cut server performs tile fragment merging after the graph-width file has been cut, the tile fragment merging comprising: and merging the tile fragments belonging to the same tile ID in the memory.
14. The system of claim 13, wherein the graph cut server is further configured to, after performing tile fragment merging, if tile fragments still exist, perform tile fragment mapping comprising: and mapping the tile IDs of the tile fragments to the corresponding map cutting server IDs in batches, and then respectively transmitting the tile fragments to the map cutting servers corresponding to the map cutting server IDs mapped by the tile IDs of the tile fragments.
15. The system of claim 14, wherein the batch mapping tile IDs to which tile fragments belong to respective corresponding cut server IDs comprises: and HASH is carried out on the tile IDs to the set formed by the cutting server IDs.
16. The system according to claim 14 or 15, wherein after the transmitting the tile fragments to the mapped graph cutting servers respectively, further comprises: the graph cutting server carries out tile fragment combination again; and waiting for the next tile fragment mapping and/or tile fragment merging of the tile fragments still remaining after the merging.
17. A storage medium having stored therein instructions that, when run on a computer, cause the computer to perform the method of distributing distributed graph cutting tasks of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910267201.0A CN110109751B (en) | 2019-04-03 | 2019-04-03 | Distribution method and device of distributed graph cutting tasks and distributed graph cutting system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910267201.0A CN110109751B (en) | 2019-04-03 | 2019-04-03 | Distribution method and device of distributed graph cutting tasks and distributed graph cutting system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110109751A CN110109751A (en) | 2019-08-09 |
| CN110109751B true CN110109751B (en) | 2022-04-05 |
Family
ID=67485119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910267201.0A Active CN110109751B (en) | 2019-04-03 | 2019-04-03 | Distribution method and device of distributed graph cutting tasks and distributed graph cutting system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110109751B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112579722B (en) * | 2020-12-22 | 2024-06-11 | 北京航天世景信息技术有限公司 | Automatic rapid image cutting method for high-customization remote sensing image |
| CN114461739A (en) * | 2021-12-29 | 2022-05-10 | 广州极飞科技股份有限公司 | Map tile processing method, device, equipment and medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663399A (en) * | 2012-04-16 | 2012-09-12 | 北京博研新创数码科技有限公司 | Image local feature extracting method on basis of Hilbert curve and LBP (length between perpendiculars) |
| CN103631878A (en) * | 2013-11-08 | 2014-03-12 | 中国科学院深圳先进技术研究院 | Method, device and system for processing massive data of graph structure |
| CN104091301A (en) * | 2014-06-30 | 2014-10-08 | 江苏华大天益电力科技有限公司 | Tile pyramid parallel construction method based on MapReduce |
| CN105224246A (en) * | 2015-09-25 | 2016-01-06 | 联想(北京)有限公司 | A kind of information and internal memory configuring method and device |
| CN105808346A (en) * | 2014-12-30 | 2016-07-27 | 华为技术有限公司 | Task scheduling method and device |
| CN105956165A (en) * | 2016-05-20 | 2016-09-21 | 江苏得得空间信息科技有限公司 | Tile type large file organization storage method of three-dimensional model data |
| CN106021567A (en) * | 2016-05-31 | 2016-10-12 | 中国农业大学 | Mass vector data partition method and system based on Hadoop |
| US9787321B1 (en) * | 2016-11-17 | 2017-10-10 | Google Inc. | Point cloud data compression using a space-filling curve |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3863775B2 (en) * | 2001-12-25 | 2006-12-27 | 株式会社九州エレクトロニクスシステム | Image information compression method, image information compression apparatus, and image information compression program |
| US7965902B1 (en) * | 2006-05-19 | 2011-06-21 | Google Inc. | Large-scale image processing using mass parallelization techniques |
| CN102254312A (en) * | 2011-07-08 | 2011-11-23 | 国电南瑞科技股份有限公司 | Method for splicing geographical tile graphs |
| US9899007B2 (en) * | 2012-12-28 | 2018-02-20 | Think Silicon Sa | Adaptive lossy framebuffer compression with controllable error rate |
| CN103455624B (en) * | 2013-09-16 | 2017-07-25 | 湖北文理学院 | A kind of global multidimensional remote sensing image Map Service of Network implementation method of lightweight |
| CN103793496B (en) * | 2014-01-22 | 2017-08-22 | 百度在线网络技术（北京）有限公司 | A kind of processing method and system of tile figure |
| CN104182966B (en) * | 2014-07-16 | 2017-03-29 | 江苏大学 | A kind of regular shredded paper method for automatically split-jointing |
-
2019
- 2019-04-03 CN CN201910267201.0A patent/CN110109751B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663399A (en) * | 2012-04-16 | 2012-09-12 | 北京博研新创数码科技有限公司 | Image local feature extracting method on basis of Hilbert curve and LBP (length between perpendiculars) |
| CN103631878A (en) * | 2013-11-08 | 2014-03-12 | 中国科学院深圳先进技术研究院 | Method, device and system for processing massive data of graph structure |
| CN104091301A (en) * | 2014-06-30 | 2014-10-08 | 江苏华大天益电力科技有限公司 | Tile pyramid parallel construction method based on MapReduce |
| CN105808346A (en) * | 2014-12-30 | 2016-07-27 | 华为技术有限公司 | Task scheduling method and device |
| CN105224246A (en) * | 2015-09-25 | 2016-01-06 | 联想(北京)有限公司 | A kind of information and internal memory configuring method and device |
| CN105956165A (en) * | 2016-05-20 | 2016-09-21 | 江苏得得空间信息科技有限公司 | Tile type large file organization storage method of three-dimensional model data |
| CN106021567A (en) * | 2016-05-31 | 2016-10-12 | 中国农业大学 | Mass vector data partition method and system based on Hadoop |
| US9787321B1 (en) * | 2016-11-17 | 2017-10-10 | Google Inc. | Point cloud data compression using a space-filling curve |
Non-Patent Citations (2)
| Title |
|---|
| "A novel Geo-Spatial image storage method based on Hilbert Space Filling Curves";Fei Li 等;《2010 18th International Conference on Geoinformatics》;20100909;第1-4页 * |
| 基于信息瓶颈的图像分割算法;吴楠;《中国优秀硕士学位论文全文数据库信息科技辑》;20120415;第2012年卷(第4期);I138-1702 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110109751A (en) | 2019-08-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11762697B2 (en) | Method and apparatus for scheduling resource for deep learning framework | |
| CN107391629B (en) | Method, system, server and computer storage medium for data migration between clusters | |
| JP6542909B2 (en) | File operation method and apparatus | |
| CN102129425B (en) | The access method of big object set table and device in data warehouse | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| CN103473121A (en) | Mass image parallel processing method based on cloud computing platform | |
| CN103077197A (en) | Data storing method and device | |
| US9836516B2 (en) | Parallel scanners for log based replication | |
| CN111966631A (en) | Mirror image file generation method, system, equipment and medium capable of being rapidly distributed | |
| CN111399764B (en) | Data storage method, data reading device, data storage equipment and data storage medium | |
| CN110109751B (en) | Distribution method and device of distributed graph cutting tasks and distributed graph cutting system | |
| CN104881466A (en) | Method and device for processing data fragments and deleting garbage files | |
| CN105677904A (en) | Distributed file system based small file storage method and device | |
| CN111552555A (en) | Point cloud visualization method and device | |
| EP3963853B1 (en) | Optimizing storage and retrieval of compressed data | |
| CN104951442A (en) | Method and device for determining result vector | |
| CN116842012A (en) | Method, device, equipment and storage medium for storing Redis cluster in fragments | |
| US9537941B2 (en) | Method and system for verifying quality of server | |
| Wang et al. | A Cloud‐Computing‐Based Data Placement Strategy in High‐Speed Railway | |
| CN109902067B (en) | File processing method and device, storage medium and computer equipment | |
| US9697036B2 (en) | Method and apparatus for generating unique identifier for distributed computing environment | |
| CN107179953B (en) | Index file generation method, device and system | |
| CN110955733A (en) | Data equalization method and system for distributed system | |
| CN113656411B (en) | Method and device for storing graph data | |
| CN117992422A (en) | Data transmission method, device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |