US10628628B2 - Method, system, and computer program product for arranging content in a plurality of regions to obtain a content-aware layout - Google Patents
Method, system, and computer program product for arranging content in a plurality of regions to obtain a content-aware layout Download PDFInfo
- Publication number
- US10628628B2 US10628628B2 US15/158,663 US201615158663A US10628628B2 US 10628628 B2 US10628628 B2 US 10628628B2 US 201615158663 A US201615158663 A US 201615158663A US 10628628 B2 US10628628 B2 US 10628628B2
- Authority
- US
- United States
- Prior art keywords
- objects
- layout
- region
- minimum
- regions
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related, expires
Links
- 238000000034 method Methods 0.000 title claims abstract description 161
- 238000004590 computer program Methods 0.000 title abstract description 7
- 238000012545 processing Methods 0.000 claims abstract 25
- 238000012966 insertion method Methods 0.000 claims description 6
- 238000012805 post-processing Methods 0.000 claims description 6
- 238000005259 measurement Methods 0.000 claims 3
- 238000003780 insertion Methods 0.000 description 11
- 230000037431 insertion Effects 0.000 description 11
- 238000013459 approach Methods 0.000 description 8
- 230000008901 benefit Effects 0.000 description 8
- 238000004519 manufacturing process Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000005457 optimization Methods 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 2
- 230000015654 memory Effects 0.000 description 2
- 230000006399 behavior Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000013341 scale-up Methods 0.000 description 1
- 238000013077 scoring method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/901—Indexing; Data structures therefor; Storage structures
- G06F16/9027—Trees
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/103—Formatting, i.e. changing of presentation of documents
- G06F40/106—Display of layout of documents; Previewing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/10—Text processing
- G06F40/12—Use of codes for handling textual entities
- G06F40/14—Tree-structured documents
Definitions
- This invention generally relates to methods, systems and computer program products for arranging content in a plurality of regions and, in particular, to methods, systems and computer program products which automatically arrange content, such as text and images, into multiple regions, such as pages in a multi-page object or distinct surfaces in a multi-object arrangement.
- Multi-region content layout is an important and challenging problem for publishing and graphic design.
- Content including text and images, must be carefully arranged to produce a desirable layout.
- text and images must be at least a minimum size such that a viewer can easily read the text or see the image content.
- large portions of images may have important content that should not be cropped, which constrains its aspect ratio inside the region.
- Text can have similar aspect ratio constraints based on desired line length and number of line breaks.
- the overall ordering and placement of content inside of regions may depend on the nature of content itself and relationships between pieces of content. For example, text content may describe particular image content and therefore needs to be placed adjacently in the same region, or the overall ordering of content may be chronological and, therefore, the content needs to be placed in a linear progression of regions, such as pages in a book.
- the first such type of method is employed by the Mosaic mobile application [mosaic]. It involves selecting one content object (an image) for each region (page) in the multi-region layout. Once the content has been selected, the software will insert one content object per region in the center of the region. While this method arguably creates a multi-region content layout, it does it in the most na ⁇ ve and simplistic way possible. It does not pay any regard to the dimensions of the content (in this case, images) with respect to the region size. As a result, the visual quality of the resulting layout is very poor, with large gaps on the sides of images having an aspect ratio different than the page aspect ratio, and potentially blurry images if the original resolution is insufficient for clear printing at their size on the page.
- This simplistic slot-based template method is very limited and often leads to undesirable layouts. Because the template is pre-generated without consideration of the actual content, if the content doesn't exactly match the expected content for the slot-based template, then the content may be cropped or have unsightly gaps. Furthermore, the number of slot-based templates available can severely limit how many content objects that can be placed on the page.
- Atkins for producing a layout in a single region are also limited compared to the methods described herein.
- the first method described by Atkins [atkins03a] outlines a method for evaluating multiple tree layouts based on a single property: the amount of unoccupied space in the region with respect to the generated layout. This is a simplistic one-dimensional measure of layout quality.
- Atkins [atkins03a] also doesn't teach an effective way to generate trees. Atkins describes randomly assigning images to leaf nodes of a binary tree, and randomly selecting a horizontal or vertical cut/layout for branch nodes in that tree. While this method will generate a tree layout, the chances of it being desirable are very low because it is random. An effective method of intelligently and efficiently generating candidate tree structures for evaluation is not obvious. The method taught by Atkins [atkins03a] is inadequate in this regard.
- Atkins also teaches a single-pass, tree-based, layout generation method [atkins03b]. This method involves iteratively inserting a content object (images) into a tree layout. This method further involves attempting to insert each new image at every node in the tree, scoring the result, and proceeding to the next iteration with the insertion that produced the best score. This method typically does not generate all possible tree layouts.
- Atkins describes two different methods for layout generation of an album of photos [atkins03a, atkins03b]. Borders outlines a method for generating a layout of photos that resembles one or more shapes [borders13]. However, laying out content into multiple regions presents a number of major challenges without obvious solutions that are not present when dealing with a single region.
- the number of content objects to place in the region is pre-determined by user input.
- a multi-region layout method must determine how many content objects to place in each region to balance layout desirability across all of the regions. The obvious method for doing this—arbitrarily dividing up the content into groups of equal size for each region—leads to undesirable results. It may not be possible to generate a desirable layout for an arbitrary fixed group of content.
- An object of at least one embodiment of the present invention is to provide a method, system and computer program product for automatically arranging content in multiple regions.
- a goal of at least one embodiment of the invention is to aid in the layout of content, such as text and images, into multiple regions. These regions are most commonly distinct pages in a multi-page object, or distinct surfaces in a multi-object arrangement.
- By automatically determining many aspects of the multi-region layout at least one embodiment of the invention helps a user to easily produce a layout that is visually desirable and satisfies constraints of the content and of the regions, such as sizes, positions, aspect ratios, and alignment.
- At least one embodiment of the invention described herein provides a method, system, and computer program product for arranging content in a plurality of regions.
- An algorithm automatically arranges a plurality of content objects into a plurality of regions.
- the automatic layout algorithm employs a tree structure for each region to achieve a layout that closely satisfies constraints that produce a desirable content layout.
- the method for producing a layout with a fixed set of content inside of a single region has been partially described in prior art patents [atkins03a, atkins03b], and in [borders13].
- the algorithm component of at least one embodiment of the present invention is producing an assignment of a plurality of content objects into a plurality of regions such that: (1) the content layouts are all desirable, (2) each of the content objects is included in a region, and (3) each of the regions contains content in a desirable layout.
- Properly assigning content a specified set of content to a particular number of regions or a range of numbers of regions in a manner that satisfies all of these three constraints is not obvious based on a method for producing a desirable layout within one region given a fixed set of content, and at least one embodiment of the current invention employs novel and non-obvious techniques to achieve these goals.
- At least one embodiment of the invention creates an intelligent layout with multiple content objects per region using a sophisticated method for producing region layouts.
- At least embodiment of the invention uses a dynamic tree-based layout for each region. This provides maximum flexibility for arranging any number of content objects of arbitrary dimensions into a region of arbitrary dimensions. This flexibility makes it much easier to generate a professional-looking layout with content closely fitting the region with only negligible gaps or cropping and provides substantial advantages over previous work.
- At least one embodiment of the invention provides a more advanced method whereby a variable number of content objects are placed into each region such that a layout of those content objects closely matches desired properties of the region (e.g., aspect ratio). It further teaches a two-pass minimal/target fitting method where a limited number of content objects are inserted into a minimum number of regions to produce layouts for the regions such that each meets a minimum level of desirability. This allows the at least one embodiment of the invention to produce higher-quality layouts with fewer images per region.
- At least one embodiment of the invention describes a number of additional quality measures for a layout, including the relative size of objects, the position of objects, and the absolute size of objects. Weighting multiple properties together into a score leads to a layout that is much more desirable. Conversely, using a simple uncovered-space metric may lead to highly undesirable layouts (e.g., with some objects being much too small or large).
- FIG. 1 shows a tree structure representing a layout of multiple content objects in a region, as well as the corresponding layout for that tree structure with arrows indicating which areas of the layout correspond to which nodes in the tree; H and V indicate that these nodes have their children arranged horizontally or vertically;
- FIG. 2 illustrates the node insert process where a node N is inserted into an existing layout at node B; new parent node P is created with B and N as children, which replaced B in the layout;
- FIG. 3 shows a layout of multiple content objects into four different regions; this shows a minimal layout where fewer content objects (in this case 1) may be inserted into a region if those objects have an aspect ratio closely matching the region;
- FIG. 4 shows a layout of multiple content objects into four different regions; unlike FIG. 3 , the layout in this figure uses more content objects per region to produce an ideal target layout that prefers using more than a few images per region; and
- FIG. 5 shows the process of removing a content object from a region having a minimal layout, adding it to the back of the object queue, and generating a new TARGET layout for the region using all the objects in the object queue; in a multi-pass procedure, this is how initial minimal layouts are converted to target layouts.
- One goal of at least one embodiment of the method, system, and computer program product described herein is to arrange content into a plurality of regions.
- the unit of content that the invention arranges is an object.
- An object can represent an image, text, or any other content.
- Each object has a rectangular bounding box that largely encapsulates the content.
- the algorithms described here will position and scale objects based on their bounding boxes.
- Each object may have a single fixed aspect ratio (the ratio of height to width). It may also have a range of aspect ratios that are acceptable (e.g., allowing an image to be cropped or skewed to some degree, or for a text area to have different height:width ratios).
- a key property of objects is their ability to scale up or down in size while maintaining the aspect ratio constraints.
- the methods described in the sections that follow may apply a scale to one or more objects to produce a desirable layout.
- objects may also have constraints on the degree to which they should be scaled with respect to their containing region. For example, if an image has an original source resolution of 800 ⁇ 600 pixels, that image object may have a constraint that it is undesirable to scale to larger than 2 ⁇ its original size (1600 ⁇ 1200 pixels) or smaller than half its size (400 ⁇ 300 pixels) when being placed into a region of a given pixel size.
- the scoring method described later may assess a penalty so that the layout generation algorithm will avoid scaling the image object beyond its desired size.
- Objects may also have desired locations with respect to their containing regions. For example, if it is desirable for a particular object to appear in the lower-right corner of a layout for whatever reason, this preference may be expressed by evaluating the position of the object relative to its desired position during the scoring procedure described later.
- FIG. 1 illustrates how this tree structure works.
- Each leaf node in the tree represents an object.
- Sibling nodes are scaled such that they either have the same height (if the parent node specifies a horizontal layout, as indicated by parent node H in the figure) or the same width (if the parent node specifies a vertical layout, as indicated by parent node V in the figure). Then, they are arranged adjacent to one another to form the area covered by the parent node.
- the process for computing the node layout is as follows. Iterate through the nodes in the tree, starting with the lowest-level leaf nodes and working back up to the root of the tree. For each parent node, calculate the range of possible aspect ratios of the parent node bounding box by either:
- the next step is to set an absolute size and position of the whole tree based in some coordinate space. (It can be pixels, points, or any other units. This invention does not depend on any particular coordinate space.) This will depend on the size and desired covering of the region. Two common ways to fill a region are to completely cover it and allow parts of objects to fall outside the region, or to completely contain the layout inside of the region so that no objects fall outside. At this point, the exact aspect ratio should also be determined. The most common way to do this is to choose a value in the allowable range that is closest to the aspect ratio of the region so that the layout fits the layout of the region as closely as possible.
- the tree is meant to fit completely inside of a 1000 ⁇ 1000 pixel region, be centered with respect to the center of the region, exactly touch two of the sides, and have a height:width aspect ratio of exactly 4:3, then the upper-left corner of the tree should be at coordinate [125, 0] and the lower-right corner should be at [875, 1000].
- This assumes a y-negative coordinate space like the one used by web browsers. The remainder of the examples will assume a y-negative coordinate space. In a y-positive coordinate space the y-values would be flipped).
- the final step in producing a layout is to iterate top-down through the tree and assign positions and sizes to each of the individual nodes. If there is a range of aspect ratios ⁇ A, B ⁇ for the tree and an actual chosen aspect ratio C between A and B, then let the constant k be equal to (C ⁇ A)/(B ⁇ A). Starting at the root node and given the starting point (e.g., [125, 0]) and ending point (e.g., [875, 1000]) or size (e.g., [750, 1000]), for each child node:
- step 4 For each child node, either proceed back to step 1 if it is a branch node, or output its position and size if it is a leaf node.
- the procedure described in the previous section can generate a layout for a given object tree structure. However, it does not proscribe a method for efficiently generating a desirable tree structure in the first place, which is a much more difficult and non-obvious problem due to the exponential number of possible tree structures for a tree with n leaf nodes—O(nn). Even if it is computationally feasible to try every possible tree structure (for low values of n), it is also not obvious how to create a procedure for selecting and evaluating candidate trees that leads to the most visually desirable tree, because there are different competing attributes that affect desirability in various ways. This section describes methods for generating candidate trees. (A method for scoring and choosing the best one is discussed later.)
- All of the generation methods described here will iteratively insert object leaf nodes into a tree until all of the input objects have been inserted. Inserting the very first node when the tree is empty entails making that node the one and only root node. After that, given an existing tree, a new object leaf node to insert N, and an existing node B (which could be a branch or leaf node) at which to insert the new node, inserting a node involves creating a new parent branch node P for nodes N and B, and then setting P's parent to existing node B's original parent. This process is illustrated in FIG. 2 . The left illustration shows the tree before, and the right illustration shows the tree after inserting node N at node B by replacing B with a new branch node B, and making B and N children of P.
- the new parent branch node must be assigned a layout direction—either horizontal (H) or vertical (V). This will impact whether the two child nodes will be arranged horizontally from left to right, or vertically from top to bottom.
- the relative order of the new node N and existing node B can be specified such that either N or B comes first in the layout (on the left or top). The node order only affects the node position and has no impact on the node size or overall layout aspect ratio (the horizontal/vertical layout direction does), so the generation algorithm may just choose one node ordering if the node position is not a parameter referenced by the scoring procedure.
- branch nodes may have a child that is another branch node with the same layout direction.
- This structure is equivalent to a non-binary tree in which the child branch node with the same layout is removed and its children are added to the original parent, creating a tree where some branch nodes have three or more children, making the tree non-binary.
- This flattening operation may be done at any time, and the methods described here do not require the layout tree to be a binary tree.
- the first method of generating candidate tree structures from a given set of objects is to produce every possible tree structure. This involves iteratively inserting each new object at every possible position into the current tree with both possible layouts (horizontal or vertical) for the newly created parent node, and, optionally, both orderings of the newly inserted node with respect to its sibling (in FIG. 2 , B before N or N before B).
- This procedure which is called BRUTE_FORCE_INSERT, involves inserting the first object O as the root of the tree T, and then proceeding with each subsequent object O as follows:
- the method described in the previous section has exponential time complexity of O(nn). This means it gets to be very expensive (generating a large number of trees) very quickly as n—the number of objects—increases. As such, it may be necessary to have a method for constructing candidate trees for larger node counts that selectively explores the space of possible tree layouts in a more efficient way. Fortunately, having more objects and more potential layouts also means that there are more options for generating a layout that satisfies the original constraints and is thus desirable.
- the general approach for selectively inserting nodes into a tree is to choose a particular location at which to insert each node that maintains properties of the layout at desirable levels.
- Two common important layout properties are the overall layout aspect ratio (for fitting the layout into a region of a given aspect ratio), and the relative size of object leaf nodes with respect to one another (so that no nodes end up substantially larger or smaller than the other nodes in the layout).
- An example of a SELECTIVE_INSERT procedure that tries to maintain both overall aspect ratio fit and relative leaf node size is as follows. Note that this specific choice of criteria is meant to be illustrative and not limiting; other criteria may be used to select the insertion point (such as its position) if there are other desirable properties to maintain during SELECTIVE_INSERT:
- leaf node A Given the next object leaf node O to insert into tree T, locate the leaf node A such that A has the largest relative size (either total area, width, or height) of any leaf node in the tree T, then:
- the selective insert algorithm can be adjusted in step 1 to select an insertion node based on a weighted scoring of possible insertion points.
- the method described above effectively just uses a one-dimensional score of the node's relative size and chooses the largest node.
- the node selection process can be updated to use a weighted combination of the relative node size and the distance of the center of the new node after insertion from its desired position. The weighting can be done, for example, by multiplying the two numbers together, and then choosing the option with the lowest weight.
- At least one embodiment of the invention does not depend on the particular method for weighting the selection parameters, so long as it leads to selection choices that cause the SELECTIVE_INSERT method to produce a tree layout that approaches the desired parameters.
- the post-processing method for a finished tree T involves the following steps:
- the methods described above involve inserting nodes to generate new tree layouts.
- the first optimization is caching the computed aspect ratios for each node in the tree.
- a node's aspect ratio will not change unless one of the nodes below it changes.
- Another optimization is caching the relative size of the largest and smallest leaf nodes underneath a branch node. Similarly, this will not change unless a node underneath the branch node changes. This again makes the overhead of recomputing the largest and smallest size of nodes in the tree following an insertion log 2(m). Furthermore, it reduces the time to search for the largest node in the tree—an operation used in step 1 of SELECTIVE_INSERT—from m to log 2(m).
- the effective cost per generated tree is O(log 2(m)) where m is the current size of the tree instead of O(m) in the na ⁇ ve implementation.
- the methods described in the previous section teach a method for generating candidate tree structures with desirable values of properties (like aspect ratio, the ratio of the largest to smallest leaf node, etc.).
- This section provides a procedure for scoring a generated tree layout to determine if it is more desirable than other layouts. The way that these methods typically work together is that the scoring procedure is applied to each tree as it is generated. If the score for a tree is the best observed score so far, then that layout is copied and saved along with its score, replacing the previous best layout. Once all layout possibilities have been generated and scored, the layout with the best score will remain.
- the SCORE procedure itself is designed to consider the values of a plurality of layout properties, and produce a scalar output score that makes it possible to select one tree with a better score than another. Without a scoring function, it would be unclear how to select one layout with a better property value along one dimension (e.g., aspect ratio), but a worse property value along another (e.g., ratio of smallest to largest node).
- the exact scoring procedure that is used is not integral to the proper functioning of this invention. The only requirement is that it produces a score that makes comparison of two trees possible, and that the score generally indicates a more desirable layout.
- a penalty of 1 indicates that the property has a fully desirable value and should not penalize the current layout. Any value greater than 1 indicates that there is a degree of undesirability for the current property value and another tree with a lower penalty would be favorable.
- the penalty score can be MAX(actual ⁇ target ⁇ threshold+1, 1) n so that differences less than threshold do not incur any penalty.
- the procedures described in the previous sections provide methods for generating and scoring tree layouts. These procedures can be used to produce the most desirable layout for a fixed number of objects, or for a variable number of objects.
- the idea behind using a variable number of objects is that it will provide greater flexibility and lead to more desirable layouts when adding content to multiple regions; by using a number of objects that produces the best layout in one region, unused objects can be placed into other regions.
- a score threshold STOP_SCORE when generating layouts. If a layout is found with a better score than STOP_SCORE, then execution can halt, and that layout can be returned instead of running until it finds the best possible option. This helps to reduce the average execution time required to find a suitable result, because the procedure will stop running if it finds a layout with a good enough score, even if a slightly better layout exists.
- REGION_LAYOUT ties together the procedures described in the earlier section to produce a desirable tree layout with between MIN_OBJECTS and MAX_OBJECTS objects in the layout.
- BRUTE_FORCE_THRESHOLD the number of iterations for which to use the BRUTE_FORCE_INSERT method at the end—based on MAX_OBJECTS.
- the previous section described how to generate a single desirable tree layout for one or more content objects. This section utilizes the REGION_LAYOUT procedure from the previous section to generate a plurality of layouts that cover a plurality of regions.
- the regions are pages, and the book may have a minimum and maximum number of desirable pages (MIN_REGIONS and MAX_REGIONS).
- the minimum number of pages for a book can be based on physical constraints if the book is being printed onto paper.
- the maximum number of pages can be set based on practical considerations for printing.
- An example of MIN_REGIONS and MAX_REGIONS for a book might be 20 and 200.
- MIN_MIN_REGIONS and MAX_REGIONS may both be the same (e.g., 12), indicating that the multi-region layout procedure described here should generate layouts for exactly 12 regions.
- region parameter specification function REGION_PARAMS(region number) to provide parameters for each region based on its number. These parameters may include the region size in some unit space, a region aspect ratio, or a range of potential sizes/aspect ratios. It also may include a size and position of the region if the region is a sub-region of a larger region (for example, having multiple regions for multiple layouts at different places on a single page).
- the region parameters may also contain any other settings that will impact the behavior of the REGION_LAYOUT procedure for the given region, including values of STOP_SCORE, or weight constants to use in the SCORE procedure (for example, if image size ratios are more important than aspect ratios for the region, e.g., due to its absolute physical size).
- This MULTI_LAYOUT method will repeatedly apply the REGION_LAYOUT procedure to produce desirable layouts for each region.
- the key challenge in this method is choosing suitable values for CURMIN and CURMAX that use an appropriate number of objects and also provide enough flexibility to generate desirable layouts.
- CURMIN and CURMAX computation are to make both values equal to the number of remaining objects in OQUEUE divided by the number of remaining regions to be filled to reach MIN_REGIONS (either rounded up or down—the last iteration will use all the remaining objects). This will use all of the objects in the queue and spread them evenly over MIN_REGIONS.
- This approach is effective for situations with a fixed number of regions (i.e., MIN_REGIONS and MAX_REGIONS are the same), like calendars. However, it does not provide an effective way to produce content layouts with a variable number of regions because it does not afford the REGION_LAYOUT procedure any flexibility with regard to using a variable number of objects to produce the best layout for an individual region.
- CURMIN and CURMAX values in the MULTI_LAYOUT procedure Another way to compute the CURMIN and CURMAX values in the MULTI_LAYOUT procedure is to set CURMIN to 1 or some other low number such that each call to REGION_LAYOUT will use the fewest number of objects that it can to produce a desirable layout. This should be coupled with loosening STOP_SCORE or the weighting parameters of the SCORE procedure so that they will stop and choose layouts having this minimum number of objects if they meet other parameters of acceptability (e.g., aspect ratio).
- CURMIN, CURMAX can be set to an arbitrary maximum number of objects to use in each region for the minimal layout, or a lower number based on the number of remaining objects in OQUEUE and the number of remaining regions to be filled to reach MIN_REGIONS.
- FIG. 3 shows an example of a minimal layout for four regions where two of the regions contain a single object, while the other two require multiple objects (four and five) to closely match their aspect ratio because the original object aspect ratios are substantially different from those of the regions. (In this figure, there is some padding between the edges of the layout and the edge of each page/region, which is denoted by a dotted line.)
- This minimal layout generation approach is effective if the number of available objects is small, but it is not ideal in the general case because layouts with only one or a few objects may be undesirable compared to layouts with more objects. For example, in FIG. 3 , the layouts on the left side with four and five objects are visually preferable to just placing a single object in a region. However, if all regions where assigned a larger number of objects from the outset, then the MULTI_LAYOUT procedure could run out of extra objects and be forced to revert to layouts with only one object that do not match other region parameters (e.g., aspect ratio) very well.
- region parameters e.g., aspect ratio
- the goal of the minimal layout approach is to trade off some degree of layout desirability in earlier iterations so that more objects are available in later iterations to produce more desirable layouts at that point. This is helpful when there are a limited number of objects in OQUEUE relative to MIN_REGIONS.
- the STOP_SCORE and weight parameters of the SCORE algorithm should be set to give preference to overall layout desirability irrespective of accepting layouts with fewer objects. As such, it may also be appropriate to set a default value of CURMIN to a higher number like 3 or 4 to prevent layouts with fewer objects altogether.
- the SCORE procedure could be configured to assess a large penalty to layouts with fewer objects, which would effectively lead to choosing layouts with more objects unless they are very undesirable in some other way.
- CURMAX should ideally be set to some number at least a few objects greater than CURMIN to allow flexibility in REGION_LAYOUT choosing an appropriate number of objects to produce a better layout.
- CURMIN and CURMAX may be reduced to ensure that each region has at least one or a few objects available for its layout.
- FIG. 4 shows the result of TARGET layout generation. Notice compared to FIG. 3 that some region layouts may end up being the same as in a minimal layout generation if the minimal layout is the same as the ideal target layout (the left two regions in FIG. 3 and FIG. 4 ). But, it may produce layouts having more content objects for regions where a minimal layout generation method chose a less desirable layout having fewer objects (the right two regions in FIG. 3 and FIG. 4 ).
- the method described above will use the TARGET mode of MULTI_LAYOUT on as many regions as possible, but also ensure that all of the MIN_REGIONS at least have a layout from the MINIMAL mode of MULTI_LAYOUT. This is better than the na ⁇ ve single-pass implementation described in the previous section, which would just fill up the remaining regions with the absolute minimum number of objects each (e.g., 1 object) and produce undesirable layouts after running out of objects. Doing multiple passes leads to layouts that at least meet a minimum level of desirability.
- FIG. 5 illustrates an iteration of this multi-pass procedure where the object(s) from the initial MINIMAL pass are re-added to the end of OQUEUE, and then the region layout is re-generated using an ITERATION of the MULTI_LAYOUT procedure in TARGET, which produces a more desirable layout having more objects.
- the single-pass method outlined by Atkins is somewhat similar to the SELECTIVE_INSERT method described in the present invention, which is designed for larger numbers of content objects where using BRUTE_FORCE_INSERT for all of the objects is too expensive.
- the SELECTIVE_INSERT method described here has a number of substantial advantages over Atkins' single-pass method.
- single-pass requires visiting every node in the tree, and then computing the score that would result from inserting a new node at that point. Altogether, the overhead of this operation would be at least O(m*log(m)) to visit m nodes and then compute a score for each.
- SELECTIVE_INSERT searches for a single insertion point, which can just be the largest node. If properties are effectively cached, then this search process only requires visiting O(log(m)) nodes. Then, it inserts the new object at that point and proceeds to the next iteration of the procedure. The overhead of this method is also only O(log(m)).
- the SELECTIVE_INSERT method described herein has an advantage over single-pass in that it can use BRUTE_FORCE_INSERT for the final iterations of the procedure, thus trying out more possibilities during the critical final iterations to achieve a result that has the best score possible. Because single-pass only inserts one object at a time, it does not explore as many possibilities and so is less likely to produce as good of a layout.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Databases & Information Systems (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Processing Or Creating Images (AREA)
- Editing Of Facsimile Originals (AREA)
Abstract
Description
- [atkins03a] C. Brian Atkins; Automatic photo album page layout. U.S. Pat. No. 7,743,322;
- [atkins03b] C. Brian Atkins; Single pass automatic photo album page layout. U.S. Pat. No. 8,065,627;
- [borders13] Kevin Borders et al. U.S. Pat. Nos. 9,275,479 and 8,976,198;
- [montage] Interactive Memories, Inc. Montage interactive web application. https://www.montagebook.com; 2015;
- [mosaic] Interactive Memories, Inc. Mosiac mobile application. http://heymosaic.com; 2015; and
- [photostory] Shutterfly, Inc. Shutterfly Photo Story for iPad. https://www.shutterflv.com/photo-book-ipad-app/; 2015.
-
- a. If the node layout is vertical, then set the child width to be the input width for the current parent node, and the child x-value to be the parent node x-value.
- b. If the node layout is horizontal, then set the child height to be the input height for the current parent node, and the child y-value to be the parent y-value.
-
- a. Set its left/top position to P and the opposite-dimension position/size based on the fixed values in
step 1. - b. Compute the remaining width (if horizontal) or height (if vertical) value V of the node and assign the all four values [left, top, width, height] to the child node.
- c. Add value V to the current position P and continue back to (a) for the next node until all the nodes have been visited.
- a. Set its left/top position to P and the opposite-dimension position/size based on the fixed values in
-
- a. Insert O as a sibling of A with new parent P having a horizontal layout to produce a new tree T2 and ordering [A, O].
- i. Recursively proceed back to
step 1 for the next node, or GENERATE the current tree if the number of objects in it are within the range of objects to use for the tree (between MIN_OBJECTS and MAX_OBJECTS, described later).
- i. Recursively proceed back to
- b. Insert O as a sibling of A with new parent P having a vertical layout to produce a new tree T3 and ordering [A, O].
- i. Recursively proceed back to
step 1 for the next node, or GENERATE the current tree if the number of objects in it are within the range of objects to use for the tree (between MIN_OBJECTS and MAX_OBJECTS, described later).
- i. Recursively proceed back to
- c. Optionally, if node position matters, repeat steps (a) and (b) to produce and recurse with new trees T4 and T5 having the reverse ordering [O, A].
- a. Insert O as a sibling of A with new parent P having a horizontal layout to produce a new tree T2 and ordering [A, O].
-
- a. Insert O as a sibling of A with new parent P having ordering [A, O] to create a new tree T2.
- b. If the original aspect ratio of T (or the middle of the aspect ratio range) is less than the aspect target aspect ratio for the region (i.e., height:width is less), then set the layout of P to vertical (thus increasing the aspect ratio of T2), otherwise set it to horizontal.
- c. If the number of objects in the tree is within the range of objects to use (describe later, between MIN_OBJECTS and MAX_OBJECTS), then GENERATE the tree.
- d. Recursively proceed back to
step 1 for the next node, or if there are less than BRUTE_FORCE_THRESHOLD nodes remaining, proceed usingstep 1 of BRUTE_FORCE_INSERT, or if no objects remain, stop.
-
- a. Remove node A from its parent node P. If the parent node P now only has one child B, then also remove P and replace it with B. This creates an intermediate tree layout T2 with node A removed.
- b. Re-insert node A into a tree layout T2 using an iteration of the SELECTIVE_INSERT method, which will find a new location at which to insert A and produce a new tree layout T3.
- c. Continue to step 1 until no nodes remain with undesirable properties, or until a maximum iteration limit has occurred (to prevent an infinite loop if this method doesn't arrive at a layout where no nodes have undesirable properties).
-
- a. Apply SCORE(L) to compute the score of the layout S. If the score S is the best score seen so far, then save S and L as Sbest and Lbest.
- b. If S is better than STOP_SCORE, then stop the layout generation loop.
-
- a. Compute region parameters R for the current region by using the function REGION_PARAMS(iteration number).
- b. Compute CURMIN—the current minimum number of objects from OQUEUE to use in the region layout.
- i. (The method for doing this will depend on the operating mode of the procedure.)
- c. Compute CURMAX—the current maximum number of objects from OQUEUE to use in the region layout.
- i. (The method for doing this will depend on the operating mode of the procedure.)
- d. Execute procedure REGION_LAYOUT with region parameters R, queue of objects OQUEUE, object minimum CURMIN, and object maximum CURMAX to produce a desirable layout L for the current region.
- e. Remove the objects in L from OQUEUE.
- f. Append the layout L to an output list OLIST.
-
- a. In this case, STOP and output the list of region layouts
-
- a. Remove the objects from the layout generated during the MINIMAL first pass, and add them back to the end of OQUEUE. This step is illustrated in the left side of
FIG. 5 . - b. Using the objects that remain in OQUEUE, execute an iteration of MULTI_LAYOUT in the TARGET mode to use the deal number of objects for the region. This step is illustrated in the right side of
FIG. 5 .
- a. Remove the objects from the layout generated during the MINIMAL first pass, and add them back to the end of OQUEUE. This step is illustrated in the left side of
Claims (45)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/158,663 US10628628B2 (en) | 2015-07-24 | 2016-05-19 | Method, system, and computer program product for arranging content in a plurality of regions to obtain a content-aware layout |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201562196348P | 2015-07-24 | 2015-07-24 | |
US15/158,663 US10628628B2 (en) | 2015-07-24 | 2016-05-19 | Method, system, and computer program product for arranging content in a plurality of regions to obtain a content-aware layout |
Publications (2)
Publication Number | Publication Date |
---|---|
US20170024362A1 US20170024362A1 (en) | 2017-01-26 |
US10628628B2 true US10628628B2 (en) | 2020-04-21 |
Family
ID=57837718
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/158,663 Expired - Fee Related US10628628B2 (en) | 2015-07-24 | 2016-05-19 | Method, system, and computer program product for arranging content in a plurality of regions to obtain a content-aware layout |
Country Status (1)
Country | Link |
---|---|
US (1) | US10628628B2 (en) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10902656B2 (en) * | 2016-02-29 | 2021-01-26 | Fujifilm North America Corporation | System and method for generating a digital image collage |
US10817911B2 (en) * | 2016-09-30 | 2020-10-27 | Criteo Sa | Programmatic generation of graphical display source code for a computerized graphical advertisement display |
US10902479B2 (en) | 2017-10-17 | 2021-01-26 | Criteo Sa | Programmatic generation and optimization of images for a computerized graphical advertisement display |
CN111353115B (en) * | 2018-12-24 | 2023-10-27 | 中移(杭州)信息技术有限公司 | Method and device for generating snowplow map |
US11120215B2 (en) | 2019-04-24 | 2021-09-14 | International Business Machines Corporation | Identifying spans using visual recognition |
US11580295B2 (en) * | 2020-06-19 | 2023-02-14 | Adobe Inc. | Systems for generating layouts of text objects |
DE112021001955T5 (en) * | 2021-03-01 | 2023-01-26 | Boe Technology Group Co., Ltd. | INTERFACE DISPLAY METHOD AND APPARATUS, COMPUTER READABLE STORAGE MEDIUM AND ELECTRONIC DEVICE |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050071783A1 (en) * | 2003-09-30 | 2005-03-31 | Atkins C. Brian | Automatic photo album page layout |
US20070186154A1 (en) * | 2006-02-06 | 2007-08-09 | Microsoft Corporation | Smart arrangement and cropping for photo views |
US20090022424A1 (en) * | 2005-07-07 | 2009-01-22 | Eugene Chen | Systems and methods for creating photobooks |
US20100275152A1 (en) * | 2009-04-23 | 2010-10-28 | Atkins C Brian | Arranging graphic objects on a page with text |
US20110029635A1 (en) * | 2009-07-30 | 2011-02-03 | Shkurko Eugene I | Image capture device with artistic template design |
US8065627B2 (en) | 2003-09-30 | 2011-11-22 | Hewlett-Packard Development Company, L.P. | Single pass automatic photo album page layout |
US20130298013A1 (en) * | 2011-01-25 | 2013-11-07 | Andrew Hunter | System and Method for Generation of a Layout of Document Elements |
US20130294709A1 (en) * | 2012-03-30 | 2013-11-07 | Intellectual Ventures Fund 83 Llc | Known good layout |
US8976198B2 (en) | 2009-10-22 | 2015-03-10 | Collage.Com, Llc | Method, system and computer program product for creating shape collages |
US20150189107A1 (en) * | 2012-09-03 | 2015-07-02 | Sony Corporation | Information processing device, information processing method, and program |
US9275479B2 (en) | 2009-10-22 | 2016-03-01 | Collage.Com, Llc | Method, system and computer program product for creating collages that visually resemble a particular shape or group of shapes |
-
2016
- 2016-05-19 US US15/158,663 patent/US10628628B2/en not_active Expired - Fee Related
Patent Citations (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20050071783A1 (en) * | 2003-09-30 | 2005-03-31 | Atkins C. Brian | Automatic photo album page layout |
US7743322B2 (en) | 2003-09-30 | 2010-06-22 | Hewlett-Packard Development Company, L.P. | Automatic photo album page layout |
US8065627B2 (en) | 2003-09-30 | 2011-11-22 | Hewlett-Packard Development Company, L.P. | Single pass automatic photo album page layout |
US20090022424A1 (en) * | 2005-07-07 | 2009-01-22 | Eugene Chen | Systems and methods for creating photobooks |
US20070186154A1 (en) * | 2006-02-06 | 2007-08-09 | Microsoft Corporation | Smart arrangement and cropping for photo views |
US20100275152A1 (en) * | 2009-04-23 | 2010-10-28 | Atkins C Brian | Arranging graphic objects on a page with text |
US20110029635A1 (en) * | 2009-07-30 | 2011-02-03 | Shkurko Eugene I | Image capture device with artistic template design |
US8976198B2 (en) | 2009-10-22 | 2015-03-10 | Collage.Com, Llc | Method, system and computer program product for creating shape collages |
US9275479B2 (en) | 2009-10-22 | 2016-03-01 | Collage.Com, Llc | Method, system and computer program product for creating collages that visually resemble a particular shape or group of shapes |
US20130298013A1 (en) * | 2011-01-25 | 2013-11-07 | Andrew Hunter | System and Method for Generation of a Layout of Document Elements |
US20130294709A1 (en) * | 2012-03-30 | 2013-11-07 | Intellectual Ventures Fund 83 Llc | Known good layout |
US20150189107A1 (en) * | 2012-09-03 | 2015-07-02 | Sony Corporation | Information processing device, information processing method, and program |
Non-Patent Citations (3)
Title |
---|
<http://heymosaic.com>; retrieved on May 19, 2016. |
<https://www.montagebook.com>; retrieved on May 19, 2016. |
<https://www.shuttertly.com/photo-book-ipad-app/>; retrieved on May 19, 2016. |
Also Published As
Publication number | Publication date |
---|---|
US20170024362A1 (en) | 2017-01-26 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10628628B2 (en) | Method, system, and computer program product for arranging content in a plurality of regions to obtain a content-aware layout | |
JP5043163B2 (en) | A method for positioning an object and a method for positioning an image in a predetermined space | |
JP4751047B2 (en) | How to place a set of objects in an area | |
US7634723B2 (en) | Constrained document layout | |
US8237716B2 (en) | Algorithm for drawing directed acyclic graphs | |
US8291314B2 (en) | Arranging graphic objects on a page | |
AU776510B2 (en) | Method for organizing digital images on a page | |
AU2014319964B2 (en) | System and method for automated conversion of interactive sites and applications to support mobile and other display environments | |
Kaser et al. | Tag-cloud drawing: Algorithms for cloud visualization | |
US9983760B2 (en) | Apparatus, method and computer readable recording medium for arranging a plurality of items automatically in a canvas | |
US11461943B1 (en) | Mosaic display systems and methods for intelligent media search | |
US9183185B2 (en) | Inserting content and exclusion zone(s) into a page | |
US8122351B2 (en) | Document edit device and storage medium | |
US9454520B2 (en) | Generating a representation of a layout having partitions for exclusion zones | |
AU2015216600B2 (en) | A system for comparison and merging of versions in edited websites and interactive applications | |
JP4666071B2 (en) | Document editing apparatus, program, and storage medium | |
JP4911240B2 (en) | Document editing apparatus, program, and storage medium | |
CN114816186A (en) | Report setting method and device based on mobile terminal, electronic equipment and storage medium | |
CN116740461A (en) | Training data generation method and device, electronic equipment and storage medium | |
KR20140060035A (en) | Method for input question in digital textbook |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: COLLAGE.COM, INC., MICHIGAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:BORDERS, KEVIN R.;REEL/FRAME:038644/0764 Effective date: 20160517 |
|
AS | Assignment |
Owner name: FUJIFILM NORTH AMERICA CORPORATION, NEW YORK Free format text: REDACTED SECURITY AGREEMENT;ASSIGNOR:COLLAGE.COM;REEL/FRAME:041519/0595 Effective date: 20161107 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: COLLAGE.COM, MICHIGAN Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:FUJIFILM NORTH AMERICA CORPORATION;REEL/FRAME:055534/0643 Effective date: 20210308 |
|
AS | Assignment |
Owner name: ALLY BANK, CALIFORNIA Free format text: SECURITY INTEREST;ASSIGNOR:COLLAGE.COM, INC.;REEL/FRAME:055549/0644 Effective date: 20210310 |
|
AS | Assignment |
Owner name: PHOTO FUNDING I L.P., AS SUCCESSOR ADMINISTRATIVE AGENT, MASSACHUSETTS Free format text: ASSIGNMENT OF PATENT SECURITY AGREEMENT;ASSIGNOR:ALLY BANK, AS RESIGNING ADMINISTRATIVE AGENT;REEL/FRAME:062104/0739 Effective date: 20221209 |
|
AS | Assignment |
Owner name: COLLAGE.COM, INC., MICHIGAN Free format text: TERMINATION AND RELEASE OF PATENT SECURITY AGREEMENT;ASSIGNOR:PHOTO FUNDING I L.P.;REEL/FRAME:064605/0678 Effective date: 20230814 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20240421 |