CN113393480A - Method for projecting notes in real time based on book positions - Google Patents
Method for projecting notes in real time based on book positions Download PDFInfo
- Publication number
- CN113393480A CN113393480A CN202110645802.8A CN202110645802A CN113393480A CN 113393480 A CN113393480 A CN 113393480A CN 202110645802 A CN202110645802 A CN 202110645802A CN 113393480 A CN113393480 A CN 113393480A
- Authority
- CN
- China
- Prior art keywords
- point
- book
- coordinates
- projector
- camera
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 230000006870 function Effects 0.000 claims abstract description 15
- 238000012545 processing Methods 0.000 claims abstract description 12
- 239000003550 marker Substances 0.000 claims description 20
- 239000011159 matrix material Substances 0.000 claims description 11
- 230000003287 optical effect Effects 0.000 claims description 10
- 230000000007 visual effect Effects 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 9
- 101100534231 Xenopus laevis src-b gene Proteins 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000004519 manufacturing process Methods 0.000 claims description 2
- 239000000463 material Substances 0.000 claims description 2
- 230000001172 regenerating effect Effects 0.000 claims description 2
- 238000003708 edge detection Methods 0.000 abstract description 2
- 238000012937 correction Methods 0.000 description 7
- 238000003384 imaging method Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000008602 contraction Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B29/00—Combinations of cameras, projectors or photographic printing apparatus with non-photographic non-optical apparatus, e.g. clocks or weapons; Cameras having the shape of other objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4007—Scaling of whole images or parts thereof, e.g. expanding or contracting based on interpolation, e.g. bilinear interpolation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/181—Segmentation; Edge detection involving edge growing; involving edge linking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method for projecting notes in real time based on book positions, which adopts an identification system with a camera, a projector and an image processing identification module, and comprises the following steps: acquiring a relation between coordinates of a projector and a camera; reading book coordinates through functions in a CV2 library; step three, judging whether the book outline is extracted, if so, carrying out the next step, and otherwise, returning to the step two; and step four, carrying out actual projection according to the projection coordinates. The invention can project notes in real time and assist students in learning; the book outline is extracted through edge detection, so that the inaccuracy of image recognition caused by ambient light can be well avoided, and the universality is wider compared with color recognition; the edge of the book is supplemented through the shape of the book, and inaccurate identification caused by hand shielding is prevented.
Description
Technical Field
The invention relates to the technical field of projection display, in particular to a method for projecting notes in real time based on book positions.
Background
At present, online education is more and more taken attention by people, but the traditional online education can only acquire knowledge unidirectionally from a screen by students, the learning effect of the students is not good in a plurality of teaching scenes, and the students are likely to be distracted or only listen and do not take notes. And online education can be performed by adopting a new mode, and the mode is that the notes are directly projected on books, so that students can write notes while reading the notes on the books, and the learning impression is deepened. The method relates to how to position the book, convert the position of the book into the projection coordinates of the camera, and convert the way by which the note pictures are finally projected onto the book. Meanwhile, in order to improve the robustness of the technology, the interference of the hand action on the book positioning is also considered. The problem is solved, and the method becomes an important condition for realizing novel interactive teaching equipment.
In the existing patent, information is acquired through a camera, and then the projector outputs specific data after operation to form interaction. However, the patent does not relate to how to accurately project the camera acquisition position, and does not clearly process the position relationship between the camera position and the projector position. And thus is different from the present application. The telescopic deformation of the book on the plane can be adapted by perspective transformation and can be projected to the right position basically, but if the book has telescopic distortion in the three-dimensional direction, the book is difficult to project the note to the right position.
Disclosure of Invention
In order to achieve the technical purpose, the invention relates to image positioning, coordinate conversion and projection, and has the functions of reading book coordinates, processing coordinates, removing hand interference, converting into projection coordinates and projecting notes.
The invention is realized by at least one of the following technical schemes.
A method for projecting notes in real time based on book positions adopts an identification system with a camera, a projector and an image processing identification module, and comprises the following steps:
acquiring a relation between a projector and a camera coordinate by using a projection relation;
reading book coordinates through functions in a CV2 library;
step three, extracting the book outline, judging whether the book outline is extracted or not, if so, carrying out the next step, and otherwise, returning to the step two;
and fourthly, performing actual projection according to the projection coordinates and the perspective transformation matrix.
Preferably, the obtaining of the relationship between the coordinates of the projector and the camera includes the following steps:
(1) placing two markers on a base, wherein the two markers are respectively marked as a first marker and a second marker;
(2) acquiring an image with a first marker and a second marker through a camera;
(3) manufacturing a black picture occupying a projector screen according to the projection size of the projector, marking a red point on the black picture, recording the position coordinates of the red point as (alpha 0, beta 0), and projecting the black picture to the projector for projection, wherein the black picture is named as src;
(4) marking a first marker in an image acquired by the camera and acquiring pixel coordinates (x11, y11) of the first marker; if the red point of the black picture src does not belong to the surface of the first marker, regenerating a new red point, shifting the position of the regenerated red point, putting the black base picture with the regenerated new red point coordinates to a projector for projection, and continuously performing the step (4) until the red point falls on the surface of the first marker, and at the moment, reading the pixel coordinates of the red point in the black picture src and recording as (x21, y 21);
(5) acquiring pixel coordinates (x12, y12) of a second marker from a camera picture, repeating the steps (3) to (4), projecting the picture by using a projector, enabling a red point to fall on the second marker, obtaining the pixel coordinates of the red point in a black picture src, and recording the pixel coordinates as (x22, y 22);
(6) the degrees of expansion k1 and k2 of the image read by the camera and the projected image in the X direction and the Y direction and the offsets b1 and b2 of the central point of the optical axis of the camera and the central point of the projection optical axis of the projector in the X direction and the Y direction are obtained according to the following formulas and are expressed as a formula I:
x11=k1*x21+b1
x12=k1*x22+b1
y11=k2*y21+b2
y12=k2*y22+b2
the coordinates of a certain pixel point shot by the camera are recorded as (x1, y1), the coordinates projected by the projector are recorded as (x2, y2), and the relationship between the two points is recorded as a formula two:
x1=k1*x2+b1
y1=k2*y2+b2
and (4) obtaining the relation between the coordinates of the projector and the camera through the steps (1) to (7), and applying the relation to subsequent coordinate transformation.
Preferably, reading book coordinates comprises the steps of:
21) placing a book on the base, and adjusting the visual field range of the camera to ensure that the visual field range is the base;
22) placing a book above the base within the visual field range of the camera, and actually measuring to obtain the width w and the height h of the book;
23) reading the book picture through a camera, and compressing the obtained book picture;
24) converting the compressed image into a gray scale image;
25) making the gray level image into Gaussian blur;
26) performing edge extraction on the image after the Gaussian blur;
27) and (3) performing polygon fitting processing on edges including hands and books by using an approxPolyDP function in a CV2 library, and changing a curve into a straight line polygon after fitting to obtain a set M of coordinates of each point of the polygon.
Preferably, the step of judging whether the book outline is extracted comprises the following steps:
31) recording corner coordinates of the upper left and the upper right of the book as an upper left point and an upper right point, wherein the upper left point and the upper right point are obtained through three points with the maximum absolute value of the y coordinate in the set M obtained in the step 7), calculating the distance between every two points, and if the distance between the two points is most consistent with the width w of the book, defining the coordinates of the two points as P1(x1, y1) and P2(x2, y 2);
32) comparing x1 with x2, if x1 is more than x2, P1 is the upper right point, P2 is the upper left point, otherwise P1 is the upper left point, and P2 is the upper right point;
33) traversing the remaining points in the set M, and taking the points with the following characteristics as the corner points at the lower right of the page:
the difference between the characteristic a and the x coordinate value of the upper right point is smaller than an allowable range;
the distance between the feature b and the upper right point is within the range of the page width w;
according to the above method, there are two cases: point P3 is obtained and defined as the bottom right point; defining the lower right point as null by obtaining points which do not meet the conditions;
based on the same method, the lower left point is found, and two situations also occur: point P4 is obtained and defined as the bottom left point; if the point that does not meet the above condition is defined as empty, there are four cases:
the first condition is as follows: only the lower left point is found, and the lower right point is empty;
case two: only the lower right point is found, and the lower left point is empty;
case three: simultaneously finding a left lower point and a right lower point;
case four: the lower left and lower right points are not found and are null.
34) If the situation is the first situation, only the lower left point is found, the lower right point is empty, and the lower left point P4(x4, y4) is translated to obtain (x4+ w, y4) which is marked as the lower right point;
if the situation is two, only the lower right point is found, the lower left point is empty, and the lower right point P3(x3, y3) is translated to obtain (x3-w, y3) which is marked as the lower left point;
if the situation is three, directly obtaining a left lower point and a right lower point;
if the situation is four, the book is not identified in the drawing;
35) in the first case, the coordinates of the upper left point, the upper right point, the lower left point and the lower right point are obtained according to the image or calculation in the third case, the book is considered to be recognized, and in the fourth case, the book is considered not to be recognized;
36) and if the book is identified, performing the fourth step, otherwise, returning to the second step.
Preferably, reading book coordinates comprises the steps of:
(41) according to the requirement of a formula II, taking the upper left point obtained in the step III as a camera to shoot the coordinate of a certain pixel point, substituting the coordinate into the formula II to obtain the actual coordinate projected by the projector, and marking as (x1, y 1);
(42) taking the coordinates of the upper right point, the lower left point and the lower right point obtained in the step three as the coordinates of a certain pixel point shot by a camera, substituting the coordinates into a formula II to obtain the coordinates projected by the projector, and recording the coordinates as (x2, y2), (x3, y3), (x4, y 4);
(43) cutting a note picture containing a note into pictures with the same width and height ratio as those of a book;
(44) acquiring a perspective transformation matrix M by using a getPerfectTransform in a CV2 library, putting four points (0,0), (0, h), (w, 0), (w, h) of the note picture and the four points obtained in the steps (41) and (42) into a getPerfectTransform function, and calculating to obtain a coordinate mapping matrix M between the note picture and the projection picture;
(45) converting the note picture into a picture src2 with the same size as the actual book and the same actual position relative to the projector by using warPeractive in a CV2 library according to the coordinate mapping matrix M;
(46) and (5) sending the picture src2 obtained in the step (45) to a projector for projection.
Preferably, the CV2 image processing library is used to convert the compressed image into a gray scale map.
Preferably, edge extraction is performed on the image after the Gaussian blur by using a canny operator, and a threshold value in the canny operator is set as an upper threshold value.
Preferably, the upper threshold is equal to three times the lower threshold.
Preferably, the image size is compressed by using a bilinear interpolation method.
Preferably, the base is a base made of black materials,
Compared with the prior art, the invention has the beneficial effects that
(1) The method has the whole process from the calibration of the camera and the projector to the realization of all functions, and the landing is completely feasible. In addition, the method starts from the principle, performs approximate processing on the premise of not influencing the effect, simplifies the implementation steps and the conditions required by implementation, and can project notes in real time to assist students in learning;
(2) the book outline is extracted through edge detection, so that the inaccuracy of image recognition caused by ambient light can be well avoided, and the universality is wider compared with color recognition;
(3) the edges of the books are supplemented through the shapes of the books, so that inaccurate identification caused by hand shielding is prevented;
(4) the projection position correction is only related to the positions of the camera and the projector, and the correction can be performed only once after the projection position correction is fixed, so that the use is convenient and the universality is considered;
(5) the problem of computer end note to the projection of object is solved.
Drawings
Fig. 1 is a flowchart of a method for projecting notes in real time based on book positions according to an embodiment of the present invention.
Detailed Description
The practice of the present invention will be further illustrated, but is not limited, by the following figures and examples. The purpose of the drawings is to supplement the description of the written part of the specification with figures so that a person can intuitively and visually understand each and every technical feature and the whole technical solution of the present invention, but it should not be construed as limiting the scope of the present invention.
In the description of the present invention, it should be understood that the orientation or positional relationship referred to in the description of the orientation, such as the upper, lower, front, rear, left, right, etc., is based on the orientation or positional relationship shown in the drawings, and is only for convenience of description and simplification of description, and thus, it should not be construed as limiting the present invention.
In the description of the present invention, the meaning of a plurality of means is one or more, the meaning of a plurality of means is two or more, and larger, smaller, larger, etc. are understood as excluding the number, and larger, smaller, inner, etc. are understood as including the number. If the first and second are described for the purpose of distinguishing technical features, they are not to be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.
As shown in fig. 1, a method for projecting notes in real time based on book positions in this example employs an identification system having a camera, a projector, and an image processing and identification module, and includes the following steps:
the method comprises the following steps of firstly, acquiring the relation between the coordinates of the projector and the camera, and specifically comprising the following steps:
(1) placing two small white chess pieces with the diameter of about 5mm on a base, and respectively marking the small white chess pieces as a first chess piece 1 and a second chess piece 2, wherein the two chess pieces are required to be not on the same horizontal line, and the spacing distance between the chess pieces is about 30cm in the transverse direction and 21cm in the longitudinal direction, so that the parameter values obtained in the subsequent correction step are more accurate;
(2) acquiring an image with a first chess piece 1 and a second chess piece 2 through a camera;
(3) making a black picture just occupying the projector screen according to the projector projection size and with a red dot at an arbitrary position (alpha 0, beta 0), and projecting the picture to the projector, wherein the picture is named src.
(4) The first pawn 1 is marked in the image acquired by the camera and the first pawn 1 pixel coordinates are acquired (x11, y 11). If the red point of the picture src is not on the surface of the chess piece 1, for example, the red point is on the left side of the chess piece 1, then the next red point is generated and shifted to the right, a black base picture with different red point coordinates is generated again and is placed on a projector for projection, the step is continuously carried out until the red point falls on the surface of the first chess piece, and at the moment, the pixel coordinates of the red point in the picture src are read and are marked as (x21, y 21);
(5) similarly, repeating the step (3) and the step (4), acquiring pixel coordinates (x12, y12) of the second chess piece 2 from the camera picture, projecting the picture src by the projector, changing the position of the red point in the src to enable the red point to fall at the second chess piece 2, and obtaining the pixel coordinates of the red point in the picture src and marking as (x22, y 22);
(6) the expansion degrees k1 and k2 of the image read by the projector and the projected image in the X direction and the Y direction and the offsets b1 and b2 of the central point of the optical axis of the camera and the central point of the projection optical axis of the projector in the X direction and the Y direction are obtained according to the following formulas and are represented as a formula I:
x11=k1*x21+h1
x12=k1*x22+b1
y11=k2*y21+b2
y12=k2*y22+b2
(7) the correction step is ended;
(8) the relation between the coordinates of the projector and the camera is determined by the imaging process and the relative position between the camera and the projector, and after the position between the projector and the camera is fixed, the relation between the coordinates of the position between the projector and the camera can be calculated by various parameters. However, the way of calculating the relationship between the position coordinates of the projector and the camera according to the actual position between the projector and the camera is too complicated, so the conversion relationship between the two sets of coordinates is obtained by selecting the actual correction way. As can be seen from the principle of camera imaging and the principle of projector projection, the image taken by the camera is centered on the optical axis of the camera, and the projection note of the projector is centered on the projection optical axis of the projector, so the offset between the two sets of points in the X direction and the Y direction is denoted as b1 and b 2. When the projector and the camera are fixed, the length and the width of the rectangle for imaging or projecting the projector and the camera are respectively parallel, so that the rotational offset between the two groups of points can be ignored. Finally, since the imaging and the projection are performed in two different sets, the projection size and the image size are often inconsistent, which causes the read image and the projected image to have expansion and contraction in the X direction and the Y direction, and therefore the expansion and contraction degrees are respectively denoted as k1 and k 2. After the relationship is obtained according to the principle, the relationship between shooting and projection is obtained, the coordinates of a certain pixel point shot by the camera are recorded as (x1, y1), the coordinates (namely, the actual coordinates) projected by the projector are recorded as (x2, y2), the relationship between two points can be obtained and recorded as a formula two:
x1=k1*x2+b1
y1=k2*y2+b2
therefore, after the above steps (1) to (7), k1, k2, b1 and b2 are obtained, the relationship between the coordinates of the projector and the camera is obtained and applied to the subsequent coordinate transformation.
As an example, parameters k1, k2, b1 and b2 are obtained by using a fixed device comprising a projector, a camera and a base, and according to the position relation between the projector and the camera, the method in the step one is used to obtain the relation parameters k 1-61/26, b 1-18760/13, k 2-192/85 and b 2-76471/85 between the camera and the projector.
Marking the position of a white chess piece in the camera, obtaining the pixel coordinates (100, 10) of the white chess piece in the picture of the camera, substituting the formula II to calculate, wherein X is k 1X 100+ b1, Y is k 2X 2+ b2, and obtaining X, Y values which are 1208 and 877 respectively, so the red point coordinates projected by the projector are (1208, 877).
The positions of the red points and the white chess pieces are found to be coincident, pixel coordinates in the camera image are obtained at any position in the middle point of the camera, the pixel coordinates are converted into projection coordinates, an image with the red points based on the projection coordinates is generated and is sent to a projector for projection, and the projector can project the red points to the actual positions.
Reading book coordinates;
(21) selecting a black base, and adjusting the visual field range of the camera to ensure that the base is black in the visual field range;
(22) books are placed above the base within the visual field range of the camera, and the width w and the height h of the books are obtained through actual measurement. The bottom plate of the step (21) and the bottom plate of the step (22) are different in color from the book, the outer contour of the book can be identified through strong color contrast between the black base and the book, in addition, the base in the visual field range can isolate the interference of the surrounding environment, and the robustness to the environment is improved.
(23) The method comprises the steps of reading a book picture through a camera, and compressing the image size of the obtained book picture by a bilinear interpolation method, and aims to improve the calculation speed.
(24) The resulting image was converted to a grayscale map using a CV2 image processing library.
(25) The gray scale map is blurred by gaussian. Over many trials, a larger convolution kernel was used in gaussian blur, in the example (9 × 9 size) for blur. The feasibility of using large convolution kernel gaussian filtering extracts only the edges of the book and does not process the content. The method has the advantages that the large convolution kernel can well keep the edge of the image and eliminate noise caused by camera shooting to cause the edge to be inaccurate.
(26) And (5) carrying out edge extraction on the image after the Gaussian blur by using a canny operator. And setting the threshold value in the canny operator as an upper threshold value, wherein the upper threshold value is equal to three times of a lower threshold value. The specific reason is that the canny operator needs to set an upper threshold and a lower threshold to judge the sensitivity degree to the edge so as to obtain an edge image, in the example, when the lower threshold is 26 through manual experiments, the extracted edge effect is optimal, then the upper threshold is set to be equal to three times of the lower threshold, and finally the upper threshold and the lower threshold are obtained.
(27) Using an approxplolydp function in a CV2 library, performing polygon fitting processing on edges including hands and books, selecting a fitting coefficient of 0.01 × peri (the perimeter of the edge), and after fitting, changing a curve into a straight line polygon to obtain a set M of coordinates of each point of the polygon.
(28) Recording corner coordinates on the left upper part and the right upper part of the book as a left upper point and a right upper point, acquiring the left upper point and the right upper point through three points with the maximum y coordinate value in the set M obtained in the step (27), and respectively calculating the distance between every two points of the three points, wherein if the distance between the two points is most consistent with the width w of the book, the three points are defined as P1(x1, y1) and P2(x2, y 2).
(29) Comparing the two points P1 and P2 obtained in the above step according to the sizes of x1 and x2, if x1 is more than x2, the P1 bit is the upper right point, P2 is the upper left point, otherwise, the P1 bit is the upper left point, and P2 is the upper right point.
(210) Traversing the remaining points in M, and taking the points with the following characteristics as the corner points at the bottom right of the page: the difference of the x value from the upper right point is less than the allowable range (taking the length of 15 pixels); and the upper right point is within a known width of the page.
According to the above method, there are two cases: point P3 is obtained and defined as the bottom right point; points which do not meet the above conditions are not obtained, and the lower right point is defined as null.
Based on the same method, the lower left point is found, and two situations also occur: point P4 is obtained and defined as the bottom left point; case one cannot be met: only the lower left point is found, and the lower right point is empty;
case two: only the lower right point is found, and the lower left point is empty;
case three: simultaneously finding a left lower point and a right lower point;
case four: the lower left and lower right points are not found and are null.
(211) If the situation is the first situation, only the lower left point is found, the lower right point is empty, and the lower left point P4(x4, y4) is translated to obtain (x4+ w, y4) which is marked as the lower right point;
if the situation is two, only the lower right point is found, the lower left point is empty, and the lower right point P3(x3, y3) is translated to obtain (x3-w, y3) which is marked as the lower left point;
if the situation is three, directly obtaining a left lower point and a right lower point;
if the situation is four, the book is not identified in the drawing;
(212) in the first case, the coordinates of the upper left point, the upper right point, the lower left point and the lower right point can be obtained according to the image or calculation, the book is considered to be recognized, and in the fourth case, the book is considered not to be recognized. And if the book is identified, performing the step three, otherwise, returning to the step (23) of the step two.
As an example, the coordinates of four points of the book are extracted in a laboratory environment, the above method can successfully track the exact position of the book after the book is moved, and the recognition of the book is not affected when the handle is placed on the book.
Step three, carrying out actual projection according to the projection coordinates;
(31) and (4) taking the upper left point obtained in the step two as the coordinate of a certain pixel point shot by the camera according to the requirement of the formula two, substituting the coordinate into the formula two to obtain the coordinate (namely the actual coordinate) projected by the projector, and recording the coordinate as (x1, y 1).
(32) Similarly, the coordinates of the upper right point, the lower left point and the lower right point obtained in the second step are taken as the coordinates of a certain pixel point photographed by the camera, and are substituted into the second formula to obtain the coordinates (i.e., actual coordinates) projected by the projector, and the coordinates are recorded as (x2, y2), (x3, y3), (x4, y 4).
(33) And taking a picture with a written note, wherein the width and height ratio of the note picture and the width and height ratio of the book are the same as w and h.
(34) Acquiring a perspective transformation matrix M by using a getPerspecfectTransform function in a CV2 library, inputting four points (0,0), (0, h), (w, 0), (w, h) of the note picture and four points such as (x1, y1) obtained in the step 31)32) into the getPerspecfecttransform function, and calculating to obtain a coordinate mapping matrix M between the note picture and the projection picture.
(35) The coordinate mapping matrix M in the above step converts the note picture into a picture src2 with the same size as the actual book and the same actual position relative to the projector by using a warpPerspective function in a CV2 library.
(35) And sending the converted picture src2 to a projector for projection, and completing the whole method for projecting notes in real time based on the book position.
As an example, a blank note picture with width and height equal to book w-440 and h-220 is defined, and notes are written on the picture by drawing software. The camera acquires coordinates of an upper left point, an upper right point, a lower left point and a lower right point of the book in the second step, wherein the coordinates are respectively (100 ), (100, 200), (300, 100) and (300, 200), the four points are respectively substituted into the second formula to be calculated, so that four points (1208, 1172), (1208, 2281), (3612, 1172) and (3612, 2281) are respectively obtained, then the four points are substituted into a getPtersctiveTransform function to calculate a projected picture, and the projected picture is given to a projector to be projected, so that the position of the note is found to be overlapped with the book. Then, notes drawn on the drawing software are continuously read and projected through the steps, and the notes can be projected in real time based on the book position.
Example two:
for reading the book coordinates in the step two, the following method can be adopted for extracting the book outline:
1) a picture containing the pages of a complete book is taken with a camera.
2) 5 to 7 points with representative colors in the page range of the book are extracted from the picture by Photoshop, the color RGB values of the points are read, and the RGB range of the page color of the book is determined according to the RGB values.
3) And according to the RGB range in the step, extracting the corresponding color range in the image acquired by the camera by using an inRange function in a CV2 library to obtain a binary image.
4) And (5) performing Gaussian blur on the binarized picture.
5) Then, step (26) is performed, and the subsequent steps are the same.
Example three:
for reading the book coordinates in the second step, the following method can be adopted for calculating the coordinates of four points of the book:
1) judging the polygon obtained in the step (27), if the number of corner points of the polygon is equal to 4 (namely, the polygon is extracted), then the book outline is considered to be extracted, and the coordinates of the four corner points of the polygon are (x1, y1), (x2, y2), (x3, y3), (x4, y 4).
2) Substituting the (x1, y1), (x2, y2), (x3, y3) and (x4, y4) obtained in the above step into the step (32), and the subsequent steps are the same.
Example four:
aiming at the relationship between the coordinates of the projector and the camera acquired in the step one, the following changes are carried out:
(1) a square hard paper sheet with a side length of 10cm is placed on the base. Hereinafter referred to as a sheet of paper, which serves to provide an accurate reference for subsequent calibration.
(2) An image with a square shape is acquired by a camera.
(3) A black picture which just occupies the projector screen is generated according to the projection size of the projector and is provided with a red square at any position, and the red square is recorded below. The middle point of the red square is located at (α 0, β 0), where the camera distortion is taken into account and therefore the width and height are not necessarily equal, noted as W0, H0, and the picture is put to the projector for projection, where the picture is named src.
(4) Marking the center point of the paper sheet in the image acquired by the camera and acquiring the pixel coordinates (x11, y11) of the center point of the paper sheet, wherein the width of the paper sheet is W and the height of the paper sheet is H.
(5) If the center point of the red square of the picture src does not coincide with the center point of the paper sheet, for example, the center point of the projected red square is on the left side of the center point of the paper sheet, then the coordinates of the center point of the red square in the black base picture generated again next time are shifted to the right and are projected by the projector, and the step is continuously performed until the center point of the red square coincides with the center point of the paper sheet, and at this time, the pixel coordinates of the center point of the red square in the picture src are read and are marked as (x21, y 21);
(6) after the center point of the red square is superposed with the center point of the paper sheet, adjusting the width of the red square in the generated black base picture, projecting the red square on a projector, continuously performing the step until the widths of the red square and the paper sheet are the same, reading the pixel width of the red square in the picture src at the moment, and recording the pixel width as W1;
(7) similarly, the height of the red square in the generated black base picture is adjusted and projected to the projector. This step is continued until the red square and the sheet are the same height, at which point the red square is read for pixel height in picture src and marked as H1;
(8) the expansion degrees k1 and k2 of the image read by the projector and the projected image in the X direction and the Y direction and the offsets b1 and b2 of the central point of the optical axis of the camera and the central point of the projection optical axis of the projector in the X direction and the Y direction are obtained according to the following formulas and are represented as a formula I:
h1=x11-x21
b2=y11-y21
k1=W/W1
k2=H/H1
(9) k1, k2, b1 and b2 are obtained, and the correction step is ended.
The preferred embodiments of the invention disclosed above are intended to be illustrative only. The preferred embodiments are not intended to be exhaustive or to limit the invention to the precise embodiments disclosed. Obviously, many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and the practical application, to thereby enable others skilled in the art to best utilize the invention. The invention is limited only by the claims and their full scope and equivalents.
Claims (10)
1. A method for projecting notes in real time based on book positions adopts an identification system with a camera, a projector and an image processing identification module, and is characterized by comprising the following steps:
acquiring a relation between a projector and a camera coordinate by using a projection relation;
reading book coordinates through functions in a CV2 library;
step three, extracting the book outline, judging whether the book outline is extracted or not, if so, carrying out the next step, and otherwise, returning to the step two;
and fourthly, performing actual projection according to the projection coordinates and the perspective transformation matrix.
2. The method of projecting notes in real time based on book position as claimed in claim 1, wherein obtaining the relationship between projector and camera coordinates comprises the steps of:
(1) placing two markers on a base, wherein the two markers are respectively marked as a first marker and a second marker;
(2) acquiring an image with a first marker and a second marker through a camera;
(3) manufacturing a black picture occupying a projector screen according to the projection size of the projector, marking a red point on the black picture, recording the position coordinates of the red point as (alpha 0, beta 0), and projecting the black picture to the projector for projection, wherein the black picture is named as src;
(4) marking a first marker in an image acquired by the camera and acquiring pixel coordinates (x11, y11) of the first marker; if the red point of the black picture src does not belong to the surface of the first marker, regenerating a new red point, shifting the position of the regenerated red point, putting the black base picture with the regenerated new red point coordinates to a projector for projection, and continuously performing the step (4) until the red point falls on the surface of the first marker, and at the moment, reading the pixel coordinates of the red point in the black picture src and recording as (x21, y 21);
(5) acquiring pixel coordinates (x12, y12) of a second marker from a camera picture, repeating the steps (3) to (4), projecting the picture by using a projector, enabling a red point to fall on the second marker, obtaining the pixel coordinates of the red point in a black picture src, and recording the pixel coordinates as (x22, y 22);
(6) the degrees of expansion k1 and k2 of the image read by the camera and the projected image in the X direction and the Y direction and the offsets b1 and b2 of the central point of the optical axis of the camera and the central point of the projection optical axis of the projector in the X direction and the Y direction are obtained according to the following formulas and are expressed as a formula I:
x11=k1*x21+b1
x12=k1*x22+b1
y11=k2*y21+b2
y12=k2*y22+b2
the coordinates of a certain pixel point shot by the camera are recorded as (x1, y1), the coordinates projected by the projector are recorded as (x2, y2), and the relationship between the two points is recorded as a formula two:
x1=k1*x2+b1
y1=k2*y2+b2
and (4) obtaining the relation between the coordinates of the projector and the camera through the steps (1) to (7), and applying the relation to subsequent coordinate transformation.
3. The method of projecting notes in real time based on book position of claim 2, wherein reading book coordinates, comprises the steps of:
21) placing a book on the base, and adjusting the visual field range of the camera to ensure that the visual field range is the base;
22) placing a book above the base within the visual field range of the camera, and actually measuring to obtain the width w and the height h of the book;
23) reading the book picture through a camera, and compressing the obtained book picture;
24) converting the compressed image into a gray scale image;
25) making the gray level image into Gaussian blur;
26) performing edge extraction on the image after the Gaussian blur;
27) and (3) performing polygon fitting processing on edges including hands and books by using an approxPolyDP function in a CV2 library, and changing a curve into a straight line polygon after fitting to obtain a set M of coordinates of each point of the polygon.
4. The method of claim 3, wherein determining whether to extract the book outline comprises:
31) recording corner coordinates of the upper left and the upper right of the book as an upper left point and an upper right point, wherein the upper left point and the upper right point are obtained through three points with the maximum absolute value of the y coordinate in the set M obtained in the step 7), calculating the distance between every two points, and if the distance between the two points is most consistent with the width w of the book, defining the coordinates of the two points as P1(x1, y1) and P2(x2, y 2);
32) comparing x1 with x2, if x1> x2, P1 is the upper right point, P2 is the upper left point, otherwise P1 is the upper left point, and P2 is the upper right point;
33) traversing the remaining points in the set M, and taking the points with the following characteristics as the corner points at the lower right of the page:
the difference between the characteristic a and the x coordinate value of the upper right point is smaller than an allowable range;
the distance between the feature b and the upper right point is within the range of the page width w;
according to the above method, there are two cases: point P3 is obtained and defined as the bottom right point; defining the lower right point as null by obtaining points which do not meet the conditions;
based on the same method, the lower left point is found, and two situations also occur: point P4 is obtained and defined as the bottom left point; if the point that does not meet the above condition is defined as empty, there are four cases:
the first condition is as follows: only the lower left point is found, and the lower right point is empty;
case two: only the lower right point is found, and the lower left point is empty;
case three: simultaneously finding a left lower point and a right lower point;
case four: the lower left point and the lower right point are not found and are null;
34) if the situation is the first situation, only the lower left point is found, the lower right point is empty, and the lower left point P4(x4, y4) is translated to obtain (x4+ w, y4) which is marked as the lower right point;
if the situation is two, only the lower right point is found, the lower left point is empty, and the lower right point P3(x3, y3) is translated to obtain (x3-w, y3) which is marked as the lower left point;
if the situation is three, directly obtaining a left lower point and a right lower point;
if the situation is four, the book is not identified in the drawing;
35) in the first case, the coordinates of the upper left point, the upper right point, the lower left point and the lower right point are obtained according to the image or calculation in the third case, the book is considered to be recognized, and in the fourth case, the book is considered not to be recognized;
36) and if the book is identified, performing the fourth step, otherwise, returning to the second step.
5. The method of projecting notes in real time based on book position of claim 4, wherein reading book coordinates, comprises the steps of:
(41) according to the requirement of a formula II, taking the upper left point obtained in the step III as a camera to shoot the coordinate of a certain pixel point, substituting the coordinate into the formula II to obtain the actual coordinate projected by the projector, and marking as (x1, y 1);
(42) taking the coordinates of the upper right point, the lower left point and the lower right point obtained in the step three as the coordinates of a certain pixel point shot by a camera, substituting the coordinates into a formula II to obtain the coordinates projected by the projector, and recording the coordinates as (x2, y2), (x3, y3), (x4, y 4);
(43) cutting a note picture containing a note into pictures with the same width and height ratio as those of a book;
(44) acquiring a perspective transformation matrix M by using a getPerfectTransform in a CV2 library, putting four points (0,0), (0, h), (w, 0), (w, h) of the note picture and the four points obtained in the steps (41) and (42) into a getPerfectTransform function, and calculating to obtain a coordinate mapping matrix M between the note picture and the projection picture;
(45) converting the note picture into a picture src2 with the same size as the actual book and the same actual position relative to the projector by using warPeractive in a CV2 library according to the coordinate mapping matrix M;
(46) and (5) sending the picture src2 obtained in the step (45) to a projector for projection.
6. The method for projecting notes in real time based on book positions as claimed in claim 5, wherein the compressed image is converted into a gray scale image using a CV2 image processing library.
7. The method of claim 5, wherein edge extraction is performed on the Gaussian-blurred image by using a canny operator, and a threshold value in the canny operator is set as an upper threshold value.
8. The method of claim 6, wherein the upper threshold is equal to three times the lower threshold.
9. The method of claim 8, wherein the image size is compressed by bilinear interpolation.
10. The method of claim 8, wherein the base is a black material base.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110645802.8A CN113393480B (en) | 2021-06-09 | 2021-06-09 | Method for projecting notes in real time based on book positions |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110645802.8A CN113393480B (en) | 2021-06-09 | 2021-06-09 | Method for projecting notes in real time based on book positions |
Publications (2)
Publication Number | Publication Date |
---|---|
CN113393480A true CN113393480A (en) | 2021-09-14 |
CN113393480B CN113393480B (en) | 2023-01-06 |
Family
ID=77620149
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110645802.8A Active CN113393480B (en) | 2021-06-09 | 2021-06-09 | Method for projecting notes in real time based on book positions |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN113393480B (en) |
Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104575120A (en) * | 2015-01-09 | 2015-04-29 | 代四广 | Display system for aided teaching |
CN108874187A (en) * | 2018-06-06 | 2018-11-23 | 哈尔滨工业大学 | A kind of projector Notes System |
CN109241244A (en) * | 2018-08-31 | 2019-01-18 | 广东小天才科技有限公司 | Interaction method, intelligent device and system for assisting user in solving problems |
CN109254663A (en) * | 2018-09-07 | 2019-01-22 | 许昌特博特科技有限公司 | A kind of application method of children's book equipped aid reading robot |
CN109493288A (en) * | 2018-10-23 | 2019-03-19 | 安徽慧视金瞳科技有限公司 | A kind of interactive mode Teaching System hot spot self organizing maps algorithm |
CN110781734A (en) * | 2019-09-18 | 2020-02-11 | 长安大学 | Children cognitive game system based on paper-pen interaction |
CN111625151A (en) * | 2020-06-02 | 2020-09-04 | 吕嘉昳 | Method and system for accurately identifying contact position in deformation projection based on touch method |
CN112614190A (en) * | 2020-12-14 | 2021-04-06 | 北京淳中科技股份有限公司 | Method and device for projecting map |
-
2021
- 2021-06-09 CN CN202110645802.8A patent/CN113393480B/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104575120A (en) * | 2015-01-09 | 2015-04-29 | 代四广 | Display system for aided teaching |
CN108874187A (en) * | 2018-06-06 | 2018-11-23 | 哈尔滨工业大学 | A kind of projector Notes System |
CN109241244A (en) * | 2018-08-31 | 2019-01-18 | 广东小天才科技有限公司 | Interaction method, intelligent device and system for assisting user in solving problems |
CN109254663A (en) * | 2018-09-07 | 2019-01-22 | 许昌特博特科技有限公司 | A kind of application method of children's book equipped aid reading robot |
CN109493288A (en) * | 2018-10-23 | 2019-03-19 | 安徽慧视金瞳科技有限公司 | A kind of interactive mode Teaching System hot spot self organizing maps algorithm |
CN110781734A (en) * | 2019-09-18 | 2020-02-11 | 长安大学 | Children cognitive game system based on paper-pen interaction |
CN111625151A (en) * | 2020-06-02 | 2020-09-04 | 吕嘉昳 | Method and system for accurately identifying contact position in deformation projection based on touch method |
CN112614190A (en) * | 2020-12-14 | 2021-04-06 | 北京淳中科技股份有限公司 | Method and device for projecting map |
Non-Patent Citations (2)
Title |
---|
LIANHAO XU ET AL.: "The Design and Implementation of Arrow Game Projection Interactive System Based on Deep Learning", 《2020 INTERNATIONAL SYMPOSIUM ON AUTONOMOUS SYSTEMS (ISAS)》 * |
陈安 等: "适用于精密电子装备的高分辨率图像采集***", 《仪表技术与传感器》 * |
Also Published As
Publication number | Publication date |
---|---|
CN113393480B (en) | 2023-01-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10713528B2 (en) | System for determining alignment of a user-marked document and method thereof | |
CN103745104B (en) | A kind of method of marking examination papers based on augmented reality | |
US7965904B2 (en) | Position and orientation measuring apparatus and position and orientation measuring method, mixed-reality system, and computer program | |
CN112132907B (en) | Camera calibration method and device, electronic equipment and storage medium | |
CN110458898A (en) | Camera calibration plate, nominal data acquisition method, distortion correction method and device | |
CN111507251A (en) | Method and device for positioning answer area in test question image and electronic equipment | |
CN111401266B (en) | Method, equipment, computer equipment and readable storage medium for positioning picture corner points | |
CN102521859B (en) | Reality augmenting method and device on basis of artificial targets | |
JP6188052B2 (en) | Information system and server | |
US11544875B2 (en) | Image processing apparatus, image processing method, and storage medium | |
CN113393480B (en) | Method for projecting notes in real time based on book positions | |
KR20200036353A (en) | Apparatus and method for estimating 3d pose | |
CN112712032A (en) | Semi-automatic object posture labeling scheme | |
CN104933430B (en) | A kind of Interactive Image Processing method and system for mobile terminal | |
CN116110069A (en) | Answer sheet identification method and device based on coding mark points and relevant medium thereof | |
CN116125489A (en) | Indoor object three-dimensional detection method, computer equipment and storage medium | |
KR101766787B1 (en) | Image correction method using deep-learning analysis bassed on gpu-unit | |
JP4444684B2 (en) | Processing method of captured image of object, image display system, program, and recording medium | |
CN114550176A (en) | Examination paper correcting method based on deep learning | |
CN110443847B (en) | Automatic vending machine holder positioning detection method based on camera | |
JP2011071746A (en) | Video output device, and video output method | |
JPH07146937A (en) | Pattern matching method | |
KR20220017705A (en) | System and method for measuring crack of structure and a recording medium having computer readable program for executing the method | |
CN113112546B (en) | Space target detection identification and pose tracking method based on three-X combined marker | |
CN112001966B (en) | Positioning and tracking method for display screen in flight training AR system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |