CN115100280A - Article placement method and apparatus - Google Patents

Article placement method and apparatus Download PDF

Info

Publication number
CN115100280A
CN115100280A CN202210664032.6A CN202210664032A CN115100280A CN 115100280 A CN115100280 A CN 115100280A CN 202210664032 A CN202210664032 A CN 202210664032A CN 115100280 A CN115100280 A CN 115100280A
Authority
CN
China
Prior art keywords
article
placement
determining
area
depth map
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210664032.6A
Other languages
Chinese (zh)
Inventor
沈蕾
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Jingdong Qianshi Technology Co Ltd
Original Assignee
Beijing Jingdong Qianshi Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Jingdong Qianshi Technology Co Ltd filed Critical Beijing Jingdong Qianshi Technology Co Ltd
Priority to CN202210664032.6A priority Critical patent/CN115100280A/en
Publication of CN115100280A publication Critical patent/CN115100280A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
    • G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • Human Resources & Organizations (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Strategic Management (AREA)
  • Tourism & Hospitality (AREA)
  • Development Economics (AREA)
  • General Business, Economics & Management (AREA)
  • Finance (AREA)
  • Accounting & Taxation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the disclosure discloses an article placement method and device. One embodiment of the method comprises: acquiring a depth map of an article placement area, and acquiring the size and the position of an article to be placed; determining a depth threshold value according to the size, and performing binarization processing on the depth map by using the depth threshold value to obtain a binarization depth map, wherein the depth threshold value is greater than the distance between the position to be placed and an article placement area; determining an overlapped edge included in a projection area of an article to be placed, wherein the projection area is formed by projecting the article to be placed to the article placement area when the article to be placed is located at a position to be placed, corresponding edges of the overlapped edge in the binary depth map include target pixel points, and the target pixel points are pixel points with values larger than a depth threshold value; determining a placing path of the article to be placed according to the overlapped edges, and placing the article to be placed in the article placing area according to the placing path and the position to be placed. This embodiment enables a convenient determination of the placement path.

Description

Article placement method and apparatus
Technical Field
The embodiment of the disclosure relates to the technical field of computers, in particular to an article placement method and device.
Background
In the fields of logistics storage and the like, sorting, stacking and the like are one of time-consuming processes. With the increasing size of logistics and warehousing, sorting and palletizing of a large number of goods usually requires various automated devices to assist in completion, so as to reduce labor cost. In the existing automatic palletizing process, generally, preset palletizing algorithms and the like are used to calculate the coordinates of the placement position and the placement path of the goods (such as the rotation direction and the entering path relative to the destination pallet), and then the goods and the like are dragged from the source or the conveyor belt to the placement position of the destination tractor or the conveyor belt one by one according to the placement path.
A relatively simple placement path is straight-up and straight-down, i.e. stacking the goods from a higher position to a designated position along a vertical direction. But this approach requires no occlusion in the placement path. Another more flexible placing path is the oblique insertion type, and the goods can be flexibly placed along various directions.
Disclosure of Invention
The embodiment of the disclosure provides an article placing method and device.
In a first aspect, embodiments of the present disclosure provide an article placement method, including: acquiring a depth map of an article placement area, and acquiring the size and the position of an article to be placed; determining a depth threshold value according to the size, and carrying out binarization processing on the depth map by using the depth threshold value to obtain a binarization depth map, wherein the depth threshold value is larger than the distance between the position to be placed and an article placing area; determining an overlapped edge included in a projection area of an article to be placed, wherein the projection area is formed by projecting the article to be placed to the article placement area when the article to be placed is located at a position to be placed, corresponding edges of the overlapped edge in the binary depth map include target pixel points, and the target pixel points are pixel points with values larger than a depth threshold value; determining a placing path of the article to be placed according to the overlapped edges, and placing the article to be placed in the article placing area according to the placing path and the position to be placed.
In a second aspect, embodiments of the present disclosure provide an article placement device, the device comprising: an acquisition unit configured to acquire a depth map of an article placement area, and acquire a size and a position of an article to be placed; the image processing unit is configured to determine a depth threshold value according to the size, and carry out binarization processing on the depth map by using the depth threshold value to obtain a binarization depth map, wherein the depth threshold value is larger than the distance between the position to be placed and the article placement area; the overlap edge determining unit is configured to determine an overlap edge included in a projection area of the article to be placed, wherein the projection area is formed by projecting the article to be placed to the article placement area when the article to be placed is located at a position to be placed, a corresponding edge of the overlap edge in the binarized depth map includes target pixel points, and the target pixel points are pixel points with values larger than a depth threshold value; the placing unit is configured to determine a placing path of the article to be placed according to the overlapped edges, and place the article to be placed in the article placing area according to the placing path and the placing position.
In a third aspect, an embodiment of the present disclosure provides an electronic device, including: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any implementation of the first aspect.
In a fourth aspect, embodiments of the present disclosure provide a computer-readable medium on which a computer program is stored, which computer program, when executed by a processor, implements the method as described in any of the implementations of the first aspect.
According to the article placement method and device provided by the embodiment of the disclosure, the depth map of the article placement area is used, binarization is performed on the depth map based on the size of the article to be placed, then it is determined that the projection area formed by projecting the article to be placed to the article placement area corresponds to the overlapped edge comprising the pixel points with larger values in the binarized depth map, then the placement path of the article to be placed is determined according to the overlapped edge, and then the article to be placed can be placed at the corresponding position in the article placement area according to the position to be placed of the article to be placed and the determined placement path, so that the placement path of the article to be placed can be conveniently determined on the basis of the known article placement area and the position to be placed of the article to be placed, and the method and device can be suitable for determining the placement paths in various scenes and have higher applicability.
Drawings
Other features, objects and advantages of the present disclosure will become more apparent upon reading of the detailed description of non-limiting embodiments made with reference to the following drawings:
FIG. 1 is an exemplary system architecture diagram in which one embodiment of the present disclosure may be applied;
FIG. 2 is a flow chart of one embodiment of an item placement method according to the present disclosure;
3a-3c are schematic diagrams of one embodiment of generating a depth map of an item placement area;
FIG. 4 is a schematic diagram of one embodiment of determining overlapping edges;
FIG. 5 is a schematic view of one embodiment of the perpendicular orientation of the sides of the projection area;
FIG. 6 is a schematic structural diagram of one embodiment of an article placement device according to the present disclosure;
FIG. 7 is a schematic structural diagram of an electronic device suitable for use in implementing embodiments of the present disclosure.
Detailed Description
The present disclosure is described in further detail below with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and are not to be construed as limiting the invention. It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that, in the present disclosure, the embodiments and features of the embodiments may be combined with each other without conflict. The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.
Fig. 1 illustrates an exemplary architecture 100 to which embodiments of the item placement method or the item placement device of the present disclosure may be applied.
As shown in fig. 1, the system architecture 100 may include a tool 101 for performing item placement and a server 102. The tool 101 for performing the placement of the object and the server 102 may be communicatively connected by various connections, such as wire, wireless communication link, or fiber optic cable.
The tool 101 for performing item placement may be a variety of tools capable of performing item placement. Such as robots, robotic arms, palletizers, and the like. The server 102 may be a server that provides various services. For example, server 102 may be a server that provides management and control for item placement. The server 102 may determine a placement path of the article to be placed, and then transmit the placement path to the tool 101 for performing article placement, so as to control the tool 101 for performing article placement to complete the placement of the article to be placed according to the received placement path.
It should be noted that the article placement method provided by the embodiment of the present disclosure is generally executed by the server 102, and accordingly, the article placement device is generally disposed in the server 102.
It should be noted that the tool 101 for performing the placement of the article may also be provided with a control center such as a processor, and the control center may determine the placement path of the article to be placed and control the tool 101 for performing the placement of the article to complete the placement of the article to be placed according to the received placement path. At this time, the article placing method provided by the embodiment of the present disclosure may also be performed by the tool 101 for performing article placing, and accordingly, the article placing device may also be provided in the tool 101 for finally performing article placing. At this point, the exemplary system architecture 100 may not have a server 102.
The server 102 may be hardware or software. When the server 102 is hardware, it may be implemented as a distributed server cluster composed of a plurality of servers, or may be implemented as a single server. When the server 102 is software, it may be implemented as multiple pieces of software or software modules (e.g., multiple pieces of software or software modules used to provide distributed services), or as a single piece of software or software module. And is not particularly limited herein.
It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.
With continued reference to fig. 2, a flow 200 of one embodiment of an item placement method according to the present disclosure is shown. The article placement method comprises the following steps:
step 201, obtaining a depth map of an article placement area, and obtaining the size and the placement position of an article to be placed.
In this embodiment, the article placement area may be used for placing articles. The article can be various articles, and can be specifically set according to an actual application scenario. As an example, the item may be a variety of goods, such as goods contained in an order for an e-commerce platform. The article placement area can be flexibly set according to the actual application scene. For example, the article placement area may be a pre-designated piece of space, such as a storage space of a designated container.
The depth map of the item placement area may be used to represent the height of each location of the item placement area. Specifically, the value of a pixel point in the depth map may represent the height of a position corresponding to the pixel point in the article placement area. It should be noted that the article placement region may be a three-dimensional space in general, and therefore, the height may refer to a height in a direction perpendicular to a horizontal plane.
The article to be placed may be any article to be placed in the article placement area. The size of the article to be placed may be used to describe the volume size of the article to be placed. The to-be-placed position where the to-be-placed object is to be placed may refer to a position where the to-be-placed object is to be placed in the object placement area. The size of the article to be placed and the position to be placed can adopt various representation methods according to specific application requirements. For example, the size of the item to be placed may be expressed using the length, width, and height of the item to be placed. The position to be placed where the article to be placed is placed can be represented using the coordinates of the vertex (e.g., the coordinates of the vertex in the lower left corner) when the article to be placed is placed in the article placement area.
An execution subject of the item placement method (e.g., server 102 shown in fig. 1, etc.) may obtain a depth map of the item placement area, the size of the item to be placed, and the location to be placed from a local and/or other storage device, etc. The depth map of the article placement area, the size of the article to be placed, and the position to be placed may be determined in advance by various methods.
For example, the depth map of the article placement area may be obtained by capturing an image of the article placement area using a depth imaging device (e.g., a depth camera). The size of the article to be placed can be obtained by measuring the article to be placed. The position to be placed of the article to be placed can be specified in advance or calculated by a preset position to be placed determining algorithm, and can be determined according to an actual application scene.
Step 202, determining a depth threshold value according to the size of the article to be placed, and performing binarization processing on the depth map of the article placement area by using the depth threshold value. And obtaining a binary depth map.
In this embodiment, the depth threshold may be greater than a distance between the to-be-placed position where the article is to be placed and the article placement area. The distance between the position to be placed of the article to be placed and the article placement area can be flexibly determined by various methods according to actual application scenes. For example, a distance between a coordinate of a certain point on the article to be placed (such as the lowest point) and a coordinate of a certain point on the article placement region (such as the highest point) may be determined as a distance between the position to be placed of the article to be placed and the article placement region.
Specifically, the depth threshold may be determined by various methods according to an actual application scenario. For example, the corresponding relationship between the size and the depth threshold may be preset, and at this time, the depth threshold corresponding to the size of the article to be placed may be directly queried.
After the depth threshold is obtained, the depth map of the article placement area may be subjected to binarization processing using the depth threshold, and the result of the binarization processing may be determined as a binarized depth map. Generally, each pixel point in the depth map can be divided into two types of pixel points based on the relationship between the value of each pixel point in the depth map and the depth threshold, so as to implement the binarization processing of the depth map. For example, the value of one type of pixel point is greater than the depth threshold, the value of the one type of pixel point may be updated to "1", the value of the other type of pixel point is not greater than the depth threshold, and the value of the one type of pixel point may be updated to "0", which is specifically shown as the following formula:
Figure BDA0003690990350000061
the binary _ img (i, j) represents the value of the pixel point (i, j) in the binary depth map, and the depth _ img (i, j) represents the value of the pixel point (i, j) in the depth map before the binary depth map is not generated. "thresh" represents the depth threshold.
Step 203, determining the overlapped edges included in the projection area of the article to be placed.
In this embodiment, the projection area where the article is to be placed may be formed by projecting the article placement area when the article is located at the position where the article is to be placed. The corresponding edge of the overlapped edge in the binarized depth map may include a target pixel point. The target pixel point may refer to a pixel point whose value is greater than the depth threshold. Since the projection area is projected onto the article placement area, the projection area may have a corresponding area in the binarized depth map, and the corresponding area and the projection area correspond to the same position in the article placement area.
Specifically, for each edge included in the projection area of the article to be placed, a corresponding edge of the edge in the binarized depth map is determined first, and then whether the corresponding edge includes the target pixel point is determined, and if yes, the edge may be determined to be an overlapped edge.
And 204, determining a placing path of the article to be placed according to the overlapped edges, placing the article to be placed in the article placing area according to the placing path and the position to be placed.
In this embodiment, the placement path may be used to complete the placement of the item to be placed. Because the overlap edge is an edge including the target pixel point, and the height of the article placement area indicated by the target pixel point is greater than the depth threshold, the overlap edge may hinder the placement of the article to be placed. Based on this, after the overlapped edges are determined, various methods can be adopted to determine the placing path of the article to be placed according to the overlapped edges so as to avoid the obstruction of the placing process of the article to be placed by the overlapped edges. For example, several placing paths may be preset, and at this time, a placing path that is not in contact with the overlapped edge may be selected from the preset placing paths as a placing path of the article to be placed.
After the placement path is determined, the to-be-placed item may be placed at the to-be-placed position in the item placement area in accordance with the placement path. The object to be placed can be placed by the execution main body, and can also be placed by other objects controlled by the execution main body.
For example, after determining the placement path, the performer main body may send the determined placement path to a tool (such as 101 shown in fig. 1) for performing article placement, so as to control the tool for performing article placement to complete placement of the article to be placed according to the placement path and the position to be placed, that is, to place the article to be placed at the position to be placed in the article placement area.
It should be noted that the determined placement path may be applied specifically in combination with the actual application scenario. For example, in some application scenarios, the size of the to-be-placed object may be obtained by using a device such as a camera, and then the to-be-placed position of each to-be-placed object in the object placement area is calculated by using a preset palletizing algorithm, so that the placement path of the to-be-placed object may be determined by the above-mentioned method provided in this embodiment of the present application. The mechanical arm and other devices can grab the article to be placed according to the grabbing position of the article to be placed determined by the camera and other devices, and then place the grabbed article to be placed at the corresponding position to be placed according to the determined placing path.
The depth map of the article placement area is binarized by using the depth threshold determined based on the size of the article to be placed, the corresponding area of the projection area formed by the projection of the article to be placed to the article placement area on the binarized depth map is determined, the edge corresponding to the pixel point with the larger value in the projection area is determined as the overlapping edge, the current article placement condition of the article placement area can be known based on the determined overlapping edge and the non-overlapping edge, the placement path of the article to be placed can be reasonably planned based on the overlapping edges, the placement path of the article to be placed is adapted to the current article placement condition of the article placement area, and the reasonable placement of the article to be placed is realized. In addition, the method only relates to simpler operations such as binarization image processing and image projection processing, is simpler, has lower cost, has smaller limitation on specific application scenes for placing articles, and has higher applicability.
In some alternative implementations of the present embodiment, the item placement area may include an item placement tool. Wherein the article placement tool may be used to place an article. For example, the article placement tools include pallets, magazines, pallets, and the like. At this time, the article placing region may be formed by a region capable of placing the article provided by the article placing tool.
In this case, the depth map of the article placement area may be acquired by:
step one, determining a depth map of an article placement tool.
In this step, the depth map of the article placement tool may be used to represent the height of each position of the article placement tool. Specifically, the value of a pixel point in the depth map may represent the height of a position corresponding to the pixel point in the article placement tool. The depth map of the article placement tool may be determined in advance according to the size of the article placement tool.
And step two, in response to determining that no article is placed on the article placement tool, determining the depth map of the article placement tool as the depth map of the article placement area.
In this step, it may be determined whether an article is currently placed on the article placing tool. If no article is currently placed on the article placement tool, the depth map of the article placement tool can be directly determined as the depth map of the article placement area.
And thirdly, responding to the fact that the articles are placed on the article placing tool, obtaining the size and the placing positions of the articles placed on the article placing tool, and generating a depth map of the article placing area according to the size and the placing positions of the articles placed on the article placing tool.
In this step, if an article is placed on the article placing tool, the size and the placing position of each placed article on the article placing tool may be acquired, respectively. Wherein, the placing position may refer to a position where the placed article is placed on the article placing tool. Specifically, the size and placement position of the article placed on the article placement tool may be obtained from a local or other storage device or the like. The size and placement position of the placed article may be determined in advance by various methods. For example, the dimensions of the placed items may be obtained by measuring the placed items. The placement location of the placed item may be recorded at the time the item is placed.
After the size and the placement position of the placed article are obtained, the depth map of the article placement tool may be further updated to generate a depth map of the article placement area. Specifically, for a pixel point at a position in the depth map corresponding to the position of the placed article, the value of the pixel point may be updated, so that the updated value of the pixel point may represent the height of the article placement tool plus the article placed thereon.
By way of example, further reference is made to fig. 3a-3c, which show schematic diagrams of one embodiment of generating a depth map of an item placement area. Referring first to fig. 3a, assuming that the length of the article placement area (e.g., the carrying area of the tray) is "l" and the width is "w", all pixels in the depth map of the article placement area may be initialized to "0". Specifically, the length direction of the article placement area is an "x" axis, the width direction of the article placement area is a "y" axis, the initialized depth map of the article placement area is "l" pixels in the direction of the "x" axis, and "w" pixels in the direction of the "y" axis, that is, the depth map matches with the article placement area.
Assume that "n" items have been placed on the item placement area. As shown in FIG. 3b, when the height direction of the article placement area is taken as the "z" axis, the lower left corner coordinate of the article can be used as (x) for the "i" th placed article i ，y i ，z i ) Indicates the placement position of the article, and has a length of "l" using the article i ", width is" w i ", and a height of" h i "indicates the size of the article.
Then can be used for each alreadyThe placed article updates the initialized depth map by the following method in sequence: as shown in FIG. 3c, the "y" th i "to" y i +w i Row and x i "to" x i +l i The value of the pixel point in the region formed by the column is updated to be z i +h i ", wherein" z i "is the current value of the pixel point during updating.
The depth map of the article placement area is generated by utilizing the size of the article placement area and the size and placement position of articles placed on the article placement area, and devices such as a depth camera and the like do not need to be installed to obtain the depth map of the article placement area, so that the determination of the placement path based on the depth map can be realized under the scene that the devices such as the depth camera and the like are inconvenient to install.
In some optional implementations of the present embodiment, the depth threshold may be determined according to a sum of a distance between the article to be placed and the article placement area and a preset offset. Wherein the preset offset may be predetermined according to the tool used to perform the placement of the item (e.g., 101 shown in fig. 1, etc.). In particular, the offset may be preset by a technician based on the tool used to perform the placement of the item to assist or facilitate the tool used to perform the placement of the item to complete the placement of the item. It can be seen that the depth thresholds corresponding to different articles to be placed may be the same or different. The preset offset corresponding to different articles to be placed can be the same.
Determining the depth threshold value according to the sum of the offset amount predetermined by the tool for performing the article placement and the distance between the article to be placed and the article placement area can enable binarization based on the depth threshold value to more accurately distinguish whether the tool for performing the article placement is convenient for placing the article or not, so that the finally determined placement path can be matched with the tool for performing the article placement.
In some optional implementations of the present embodiment, the overlapped edges included in the projection area where the article is to be placed may be determined by:
step one, for each edge of the projection area, determining the number of target pixel points included by the corresponding edge of the edge in the binary depth map.
In this step, the number of target image points included in each edge included in the projection region corresponding to the corresponding edge in the binarized depth map may be counted first.
And step two, in response to the fact that the number of the target pixel points included in the corresponding edge is larger than a preset number threshold, determining the edge as an overlapped edge.
In this step, the preset number threshold may be preset by a technician according to an actual application scenario. Specifically, it may be determined whether the number of target pixel points included in the corresponding edge of each edge is greater than a preset number threshold. If so, the edge may be determined to be an overlapping edge.
And step three, in response to the fact that the number of the target pixel points included in the corresponding edge is not larger than a preset number threshold, determining that the edge is not an overlapped edge, namely a non-overlapped edge.
Whether each edge of the projection area is an overlapped edge is determined by setting a preset number threshold value and judging whether the total number of target pixel points included by the corresponding edge in the binary depth map of each edge of the projection area is larger than the preset number threshold value or not, the preset number threshold value can be flexibly set according to influences of various factors such as actual application scenes and possible errors, the judgment scale of the overlapped edge can be flexibly adjusted, and the flexibility and the applicability of the placing path determining method are further improved.
Alternatively, the projection region may be generated by:
step one, determining a projection area corresponding to the article to be placed as an initial projection area according to the size and the position of the article to be placed.
In this step, the initial projection area is a projection area formed by projecting the object placement area when the object to be placed is located at the position to be placed.
And step two, expanding the initial projection area to obtain an expanded projection area, and determining the expanded projection area as the projection area of the article to be placed.
In this step, the projection area may be expanded to a certain extent, and the specific expansion range may be flexibly set by a technician according to the actual application scenario. For example, the expanded range may be determined based on historical item placement experience. For another example, the expanded range may be determined according to the size of the article to be placed or/and the size of the article placement area.
Because a certain error may exist in an actual application scene, the accuracy of the placement path determined based on the overlap edge is improved by expanding the projection area and then determining the overlap edge based on the expanded projection area, and the influence of the error on the determined placement path is reduced.
Alternatively, the preset number threshold may be determined by various methods according to the expansion range of the initial projection area. For example, the correspondence between the expanded range and the number threshold may be set in advance by a technician. At this time, the number threshold corresponding to the expanded range of the initial projection area may be queried as the preset number threshold. For another example, the preset number threshold may be set to be greater than the number of pixels corresponding to the enlarged projection area.
The preset number threshold is flexibly set according to the expansion range of the projection area so as to determine the overlapped edges, and the influence of the expansion of the projection area on the determined placing path can be reduced while the influence of the error on the determined placing path is reduced.
By way of example, further reference is made to FIG. 4, which illustrates a schematic diagram of one embodiment of determining overlapping edges. As shown in fig. 4, it is assumed that a pixel in a white region in the binarized depth map is a target pixel. In the figure, a quadrilateral region composed of four sides "a", "b", "c", and "d" is assumed as a projection region where a number of pixels are enlarged.
Then, a boolean variable may be set for each of the four edges "a", "b", "c", and "d", and the initial value set to "false" indicates that the edge is a non-overlapping edge. For the edge "a", the total number of target pixels belonging to each pixel covered by the edge "a" in the binarized depth map can be counted, and if the counted total number is greater than a preset number threshold, the boolean value of the edge "a" is updated to be "true", which means that the edge is an overlapped edge, that is, the edge of the original projection region before the edge "a" is not expanded is an overlapped edge. Correspondingly, if the counted total number is not greater than the preset number threshold, the boolean value of the edge "a" is kept as "false". The method for determining whether the other edges "b", "c", and "d" are overlapped edges is the same as the edge "a", and will not be described herein.
In some alternative implementations of this embodiment, the direction of the placement path of the item to be placed may be different from the perpendicular direction of the overlapping edges comprised by the projection area. Wherein, the vertical direction of the overlapped side may refer to the vertical direction of the direction in which the overlapped side is located. Specifically, the direction of the placement path of the article to be placed can be flexibly determined. For example, an included angle between the direction of the placing path of the article to be placed and the vertical direction of the overlapped edge may be set to be a preset angle, so that the determined direction of the placing path can avoid the vertical direction of the overlapped edge.
The direction of the placing path of the article to be placed is controlled to be different from the vertical direction of the overlapped edge, so that the placing path of the article to be placed is prevented from being influenced by the overlapped edge, and the placing efficiency and the success rate of the article to be placed are improved.
Alternatively, the placement path of the article to be placed may be determined by:
step one, determining the sum of unit vectors respectively corresponding to each overlapped edge included in a projection area.
In this step, the unit vector may represent a vertical direction of the corresponding edge. Specifically, the unit vectors corresponding to the overlapping edges included in the projection area may be accumulated to obtain a sum. Where the resulting sum is also typically a vector.
As an example, as shown in fig. 5, a schematic diagram of one embodiment of a perpendicular direction of each side of the projection area is shown. The vertical direction of the side "a" may be represented by a unit vector va (-1, 0), the vertical direction of the side "b" may be represented by a unit vector vb (0, 1), the vertical direction of the side "c" may be represented by a unit vector vc (1, 0), and the vertical direction of the side "b" may be represented by a unit vector vd (0, -1).
And step two, responding to the fact that the obtained sum is not a zero vector, unitizing the obtained sum to obtain a unitization result, and determining the direction represented by the unitization result as the direction of the placing path of the article to be placed.
In this step, if the obtained sum is not a zero vector, the obtained sum may be unitized first to obtain a unit vector corresponding to the sum, and then a direction indicated by the unit vector corresponding to the sum may be determined as a direction of a placement path of the article to be placed.
Specifically, various situations that the sum obtained by subdivision is not a zero vector can be further subdivided to further determine the placing path of the article to be placed in detail, and the placing path can be flexibly set according to the actual application scene.
Taking fig. 5 as an example, if there is only one overlapping side in the projection area, if the side "a" is an overlapping side and the other sides are non-overlapping sides, the direction of the placing path can be determined as the direction denoted by va ═ 1, 0.
If there are two overlapping sides in the projection area, for example, the sides "a" and "b" are overlapping sides, and the other sides are non-overlapping sides, then the direction of the placing path can be determined to be a forty-five degree angle direction with va ═ 1, 0 and vb ═ 0, 1.
If there are three overlapping sides in the projection area, if the sides "a", "b", and "c" are overlapping sides and the side "c" is a non-overlapping side, the direction of the placement path can be determined as the direction represented by (0, 1) where vb is equal to the direction represented by (0, 1).
If the obtained sum is a zero vector, the direction of the placing path of the article to be placed can be determined according to the specific distribution condition of the overlapping edges in the vertical direction, so as to avoid the influence of the overlapping edges on the placing of the article to be placed.
Still taking fig. 5 as an example, if "a", "b", "c", and "d" are all overlapping edges, then the direction of the placement path may be determined to be straight up or straight down. If the edges "a" and "c" are overlapping edges and the other edges are non-overlapping edges, then the placement path can be oriented parallel to the "y" axis. If sides "b" and "d" are overlapping sides and the other sides are non-overlapping sides, then the placement path can be oriented parallel to the "x" axis.
It should be noted that fig. 5 shows only one example in which the projection area is a quadrangle, and in practical applications, the shape of the object may be various, and therefore, the shape of the projection area is not limited to a quadrangle, and may be various shapes, which is specifically determined according to practical application scenarios.
The method is characterized in that the sum of all the edges of the projection area is calculated by using a unit vector which represents the vertical direction of all the edges, the direction of the placement path is flexibly set based on various conditions that the sum is zero or non-zero, a simpler method for determining the direction of the placement path is provided, and the influence of too complex calculation on real-time performance and cost is avoided.
With further reference to fig. 6, as an implementation of the methods illustrated in the above figures, the present disclosure provides an embodiment of an article placement device, which corresponds to the method embodiment illustrated in fig. 2, and which is particularly applicable in various electronic devices.
As shown in fig. 6, the article placement device 600 provided by the present embodiment includes an acquisition unit 601, an image processing unit 602, an overlap edge determination unit 603, and a placement unit 604. Wherein the acquisition unit 601 is configured to acquire a depth map of an article placement area, and acquire a size and a placement position of an article to be placed; the image processing unit 602 is configured to determine a depth threshold value according to the size, and perform binarization processing on the depth map by using the depth threshold value to obtain a binarized depth map, wherein the depth threshold value is greater than the distance between the position to be placed and the article placement area; the overlap edge determining unit 603 is configured to determine an overlap edge included in a projection area of the to-be-placed article, wherein the projection area is formed by projecting the to-be-placed article to the article placement area when the to-be-placed article is located at the to-be-placed position, a corresponding edge of the overlap edge in the binarized depth map includes target pixel points, and the target pixel points are pixel points whose values are greater than the depth threshold; the placement unit 604 is configured to determine a placement path for the article to be placed based on the overlapping edges, and place the article to be placed on the article placement area according to the placement path and the placement position.
In the present embodiment, in the article placement device 600: the specific processing of the obtaining unit 601, the image processing unit 602, the overlapping edge determining unit 603, and the placing unit 604 and the technical effects thereof can refer to the related descriptions of step 201, step 202, step 203, and step 204 in the corresponding embodiment of fig. 2, which are not described herein again.
In some optional implementations of this embodiment, the item placement area includes an item placement tool; and the acquisition unit 601 is further configured to: determining a depth map of an item placement tool; in response to determining that no item is placed on the item placement tool, determining a depth map of the item placement tool as a depth map of the item placement area; in response to determining that an item is placed on the item placement tool, acquiring a size and a placement position of the item already placed on the item placement tool; and updating the depth map of the article placement tool according to the size and the placement position to obtain a depth map of the article placement area.
In some optional implementations of this embodiment, the depth threshold is determined from a sum of the distance and a preset offset, wherein the preset offset is determined from a tool used to perform the placement of the item.
In some optional implementations of this embodiment, the overlap edge determining unit 603 is further configured to: for each edge of the projection area, determining the number of target pixel points included by the corresponding edge of the edge in the binary depth map; in response to the fact that the number of the target pixel points included in the corresponding edge is larger than a preset number threshold value, determining the edge as an overlapped edge; in response to determining that the number of target pixel points included by the corresponding edge is not greater than a preset number threshold, determining that the edge is not an overlapping edge.
In some optional implementations of this embodiment, the projection region is generated by: determining a projection area corresponding to the article to be placed as an initial projection area according to the size and the position of the article to be placed; expanding the initial projection area, and determining the expanded projection area as the projection area of the article to be placed; and the preset number threshold is determined according to the expansion range of the initial projection area.
In some alternative implementations of this embodiment, the direction of the placement path of the item to be placed is different from the perpendicular direction of the overlapping edges comprised by the projection area.
In some optional implementations of the present embodiment, the placing unit 604 is further configured to: determining the sum of unit vectors respectively corresponding to all overlapped edges included in a projection area, wherein the unit vectors represent the vertical direction of the corresponding edges; in response to determining that the sum is not a zero vector, unitizing the sum, and determining a direction represented by the unitizing result as a direction of a placement path in which the item is to be placed.
According to the device provided by the above embodiment of the disclosure, the depth map of the article placement area is obtained through the obtaining unit, and the size and the position of the article to be placed are obtained; the image processing unit determines a depth threshold value according to the size, and performs binarization processing on the depth map by using the depth threshold value to obtain a binarization depth map, wherein the depth threshold value is larger than the distance between the position to be placed and the article placement area; the overlap edge determining unit determines overlap edges included in a projection area of the article to be placed, wherein the projection area is formed by projecting the article to be placed to the article placing area when the article to be placed is located at a position to be placed, corresponding edges of the overlap edges in the binaryzation depth map include target pixel points, and the target pixel points are pixel points with values larger than a depth threshold value; the placing unit determines a placing path of the article to be placed according to the overlapped edges, and places the article to be placed in the article placing area according to the placing path and the placing position. Therefore, the current article placing condition of the article placing area can be known through the determined overlapped edges and the non-overlapped edges, so that the placing path of the article to be placed can be reasonably planned based on the overlapped edges, the placing path of the article to be placed is adaptive to the current article placing condition of the article placing area, and the reasonable placing of the article to be placed is realized. In addition, the method only relates to simpler operations such as binarization image processing, image projection processing and the like, is simpler and has lower cost, less limitation on specific application scenes for placing articles and higher applicability.
Referring now to FIG. 7, a block diagram of an electronic device (e.g., the server of FIG. 1) 700 suitable for use in implementing embodiments of the present disclosure is shown. The server shown in fig. 7 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 7, electronic device 700 may include a processing means (e.g., central processing unit, graphics processor, etc.) 701 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM)702 or a program loaded from storage 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data necessary for the operation of the electronic apparatus 700 are also stored. The processing device 701, the ROM 702, and the RAM703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Generally, the following devices may be connected to the I/O interface 705: input devices 706 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 707 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 708 including, for example, magnetic tape, hard disk, etc.; and a communication device 709. The communication means 709 may allow the electronic device 700 to communicate wirelessly or by wire with other devices to exchange data. While fig. 7 illustrates an electronic device 700 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 7 may represent one device or a plurality of devices as necessary.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program comprising program code for performing the method illustrated by the flow chart. In such embodiments, the computer program may be downloaded and installed from a network via the communication means 709, or may be installed from the storage means 708, or may be installed from the ROM 702. The computer program, when executed by the processing device 701, performs the above-described functions defined in the methods of embodiments of the present disclosure.
It should be noted that the computer readable medium described in the embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In embodiments of the present disclosure, however, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a depth map of an article placement area, and acquiring the size and the position of an article to be placed; determining a depth threshold value according to the size, and performing binarization processing on the depth map by using the depth threshold value to obtain a binarization depth map, wherein the depth threshold value is greater than the distance between the position to be placed and an article placement area; determining overlapped edges included in a projection area of an article to be placed, wherein the projection area is formed by projecting the article to be placed to the article placing area when the article to be placed is located at a position to be placed, corresponding edges of the overlapped edges in the binary depth map comprise target pixel points, and the target pixel points are pixel points with values larger than a depth threshold value; determining a placing path of the article to be placed according to the overlapped edges, and placing the article to be placed in the article placing area according to the placing path and the position to be placed.
Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, and may be described as: a processor includes an acquisition unit, an image processing unit, an overlap edge determination unit, and a placement unit. Here, the names of these units do not constitute a limitation to the unit itself in some cases, and for example, the acquisition unit may also be described as "a unit that acquires a depth map of an article placement area, and acquires the size of an article to be placed and a position to be placed".
The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as set forth above. For example, the above features and (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.

Claims (10)

1. An article placement method comprising:
acquiring a depth map of an article placement area, and acquiring the size and the position of an article to be placed;
determining a depth threshold value according to the size, and performing binarization processing on the depth map by using the depth threshold value to obtain a binarized depth map, wherein the depth threshold value is larger than the distance between the position to be placed and the article placement area;
determining an overlapped edge included in a projection area of the article to be placed, wherein the projection area is formed by projecting the article to be placed to the article placement area when the article to be placed is located at the position to be placed, corresponding edges of the overlapped edge in the binaryzation depth map include target pixel points, and the target pixel points are pixel points with values larger than the depth threshold value;
determining a placing path of the article to be placed according to the overlapped edges, and placing the article to be placed in the article placing area according to the placing path and the position to be placed.
2. The method of claim 1, wherein the item placement area comprises an item placement tool; and
the obtaining of the depth map of the article placement area includes:
determining a depth map of the item placement tool;
in response to determining that no item is placed on the item placement tool, determining a depth map of the item placement tool as a depth map of the item placement area;
in response to determining that an item is placed on the item placement tool, obtaining a size and a placement position of the item already placed on the item placement tool; and updating the depth map of the article placement tool according to the size and the placement position to obtain the depth map of the article placement area.
3. The method of claim 1, wherein the depth threshold is determined from a sum of the distance and a preset offset, wherein the preset offset is determined from a tool used to perform the placement of the item.
4. The method of claim 1, wherein the determining the overlap edges included in the projected area of the item to be placed comprises:
for each edge of the projection region, determining the number of target pixel points included by the corresponding edge of the edge in the binarized depth map;
in response to determining that the number of target pixel points included in the corresponding edge is greater than a preset number threshold, determining that the edge is an overlapping edge;
and in response to determining that the number of target pixel points included by the corresponding edge is not greater than a preset number threshold, determining that the edge is not an overlapping edge.
5. The method of claim 4, wherein the projection region is generated by:
determining a projection area corresponding to the article to be placed as an initial projection area according to the size and the position of the article to be placed;
expanding the initial projection area, and determining the expanded projection area as the projection area of the article to be placed; and
the preset number threshold is determined according to the expansion range of the initial projection area.
6. Method according to one of claims 1 to 5, wherein the direction of the placement path of the item to be placed is different from the vertical direction of the overlapping edges comprised by the projection area.
7. The method of claim 6, wherein said determining a placement path for said item to be placed from said overlapping edges comprises:
determining the sum of unit vectors respectively corresponding to each overlapped side included in the projection area, wherein the unit vectors represent the vertical direction of the corresponding side;
in response to determining that the sum is not a zero vector, unitizing the sum, and determining a direction represented by a result of the unitizing as a direction of a placement path of the item to be placed.
8. An article placement device, wherein the device comprises:
an acquisition unit configured to acquire a depth map of an article placement area, and acquire a size and a position of an article to be placed;
an image processing unit configured to determine a depth threshold according to the size, and perform binarization processing on the depth map by using the depth threshold to obtain a binarized depth map, wherein the depth threshold is larger than the distance between the position to be placed and the article placement area;
an overlap determining unit configured to determine an overlap included in a projection area of the article to be placed, wherein the projection area is formed by projecting the article to be placed to the article placement area when the article to be placed is located at the position to be placed, a corresponding edge of the overlap in the binarized depth map includes target pixel points, and the target pixel points are pixel points whose values are greater than the depth threshold;
the placing unit is configured to determine a placing path of the article to be placed according to the overlapped edges, and place the article to be placed in the article placing area according to the placing path and the placing position.
9. An electronic device, comprising:
one or more processors;
a storage device having one or more programs stored thereon;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.
10. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.
CN202210664032.6A 2022-06-13 2022-06-13 Article placement method and apparatus Pending CN115100280A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210664032.6A CN115100280A (en) 2022-06-13 2022-06-13 Article placement method and apparatus

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210664032.6A CN115100280A (en) 2022-06-13 2022-06-13 Article placement method and apparatus

Publications (1)

Publication Number Publication Date
CN115100280A true CN115100280A (en) 2022-09-23

Family

ID=83291858

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210664032.6A Pending CN115100280A (en) 2022-06-13 2022-06-13 Article placement method and apparatus

Country Status (1)

Country Link
CN (1) CN115100280A (en)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160016311A1 (en) * 2014-07-16 2016-01-21 Google Inc. Real-Time Determination of Object Metrics for Trajectory Planning
CN110458173A (en) * 2019-08-16 2019-11-15 京东数字科技控股有限公司 Method and apparatus for generating article color value
EP3683721A1 (en) * 2019-01-18 2020-07-22 Intelligrated Headquarters LLC A material handling method, apparatus, and system for identification of a region-of-interest
CN113345015A (en) * 2021-08-05 2021-09-03 浙江华睿科技股份有限公司 Package position detection method, device and equipment and readable storage medium
CN113763458A (en) * 2020-08-21 2021-12-07 北京京东乾石科技有限公司 Method and device for determining target object placing surface
US20220114868A1 (en) * 2020-10-13 2022-04-14 Trax Technology Solutions Pte Ltd. Updating shopping list based on analysis of images

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160016311A1 (en) * 2014-07-16 2016-01-21 Google Inc. Real-Time Determination of Object Metrics for Trajectory Planning
EP3683721A1 (en) * 2019-01-18 2020-07-22 Intelligrated Headquarters LLC A material handling method, apparatus, and system for identification of a region-of-interest
CN110458173A (en) * 2019-08-16 2019-11-15 京东数字科技控股有限公司 Method and apparatus for generating article color value
CN113763458A (en) * 2020-08-21 2021-12-07 北京京东乾石科技有限公司 Method and device for determining target object placing surface
US20220114868A1 (en) * 2020-10-13 2022-04-14 Trax Technology Solutions Pte Ltd. Updating shopping list based on analysis of images
CN113345015A (en) * 2021-08-05 2021-09-03 浙江华睿科技股份有限公司 Package position detection method, device and equipment and readable storage medium

Similar Documents

Publication Publication Date Title
US10679379B1 (en) Robotic system with dynamic packing mechanism
CN111325796B (en) Method and apparatus for determining pose of vision equipment
CN111079619B (en) Method and apparatus for detecting target object in image
CN111209978B (en) Three-dimensional visual repositioning method and device, computing equipment and storage medium
CN106813568A (en) object measuring method and device
CN110197615A (en) For generating the method and device of map
US20220215567A1 (en) Depth estimation device, depth estimation model learning device, depth estimation method, depth estimation model learning method, and depth estimation program
CN113607185B (en) Lane line information display method, lane line information display device, electronic device, and computer-readable medium
CN113255619B (en) Lane line recognition and positioning method, electronic device, and computer-readable medium
CN111857114A (en) Robot formation moving method, system, equipment and storage medium
CN116079697A (en) Monocular vision servo method, device, equipment and medium based on image
CN114638846A (en) Pickup pose information determination method, pickup pose information determination device, pickup pose information determination equipment and computer readable medium
CN114454168A (en) Dynamic vision mechanical arm grabbing method and system and electronic equipment
CN115100280A (en) Article placement method and apparatus
CN113345023B (en) Box positioning method and device, medium and electronic equipment
CN111113405A (en) Method for robot to obtain position service and robot
CN114633979A (en) Goods stacking method and device, electronic equipment and computer readable medium
CN110634159A (en) Target detection method and device
CN111784218A (en) Method and apparatus for processing information
CN113688704A (en) Item sorting method, item sorting device, electronic device, and computer-readable medium
CN110068834B (en) Road edge detection method and device
CN112880675A (en) Pose smoothing method and device for visual positioning, terminal and mobile robot
CN111410045A (en) Container handling method and device
CN113160324A (en) Bounding box generation method and device, electronic equipment and computer readable medium
CN116342858B (en) Object detection method, device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination