CN111898555A

CN111898555A - Book checking identification method, device, equipment and system based on images and texts

Info

Publication number: CN111898555A
Application number: CN202010762471.1A
Authority: CN
Inventors: 施晓华; 许敬一; 杨婉茹; 卢宏涛
Original assignee: Shanghai Jiaotong University; CERNET Corp
Current assignee: Shanghai Jiaotong University; CERNET Corp
Priority date: 2020-07-31
Filing date: 2020-07-31
Publication date: 2020-11-06
Anticipated expiration: 2040-07-31
Also published as: CN111898555B

Abstract

The invention provides a book checking identification method, a book checking identification device, book checking equipment and a book checking identification system based on images and texts, wherein the method comprises the following steps: the method comprises the following steps: collecting a bookshelf image, and cutting the bookshelf image into a plurality of spine images containing spines; identifying the spine images to obtain keywords corresponding to the spines; judging whether the similarity between the keywords corresponding to any spine and book information in a book database reaches a first similarity threshold value: if so, finishing checking the books corresponding to the book spine; if not, taking the key word corresponding to the spine and at least one key word corresponding to the spine around the spine as an integral key word, and continuously judging whether the similarity between the integral key word and the book information in the book database reaches a second similarity threshold value: if so, finishing checking books corresponding to the spines forming the whole keywords; if not, determining that the inventory fails. The invention can assist the automatic book checking of the intelligent library.

Description

Book checking identification method, device, equipment and system based on images and texts

Technical Field

The invention belongs to the technical field of intelligent libraries, particularly relates to the technical field of book management, and particularly relates to a book checking identification method, device, equipment and system based on images and texts.

Background

In recent years, the research and application of intelligent libraries attract the attention of many scholars and companies, and are still a difficult subject. The automatic identification based on the spine of the library book is the leading core research and development content of intelligent inventory. The visual analysis can acquire accurate spine text information of the book in real time, and has great prospect for improving book checking efficiency and reducing book checking cost by combining a high-efficiency text matching method.

The existing book identification technology of a library mainly depends on radio frequency identification technology (RFID) except manual work, and the method has high accuracy, but has the problems of high cost, privacy disclosure, information conflict, incapability of acquiring book sequencing and the like. The application of the technology in the field of efficient intelligent book checking of the library is greatly limited. In terms of current computer application algorithms, there are many published text detection and recognition algorithms, but few are used in library book inventory systems. Text recognition technology based on deep learning has been developed greatly in recent years, efficiency and accuracy are gradually improved, and a large amount of open source software and open APIs are already available. In consideration of the popularization of intelligent libraries in the future, the intelligent detection of books by using visual analysis, deep learning and text intelligent matching technologies is obviously more advantageous than the existing radio frequency identification technology.

With the development of image processing-based text recognition technology and the popularization of intelligent devices, the application field of visual analysis is becoming more and more extensive. Books, as the most common and extensive knowledge carriers in daily life, store most of the information in the form of texts. Meanwhile, books contain many potential visual information and are yet to be discovered and applied. One of them is character recognition of the back of a book stored on a bookshelf. The efficiency of the book checking system is one of the daily problems faced by the library, the accuracy rate is not high when the book checking is carried out through the RFID robot, and the actual sequence of the discharging of the tracing books cannot be effectively obtained.

Disclosure of Invention

In view of the above drawbacks of the prior art, an object of the present invention is to provide a method, an apparatus, a device and a system for book inventory identification based on images and texts, which are used to solve the problems of low book inventory efficiency and low book inventory accuracy in the prior art.

To achieve the above and other related objects, an embodiment of the present invention provides a book inventory identification method based on images and texts, including: collecting a bookshelf image, and cutting the bookshelf image into a plurality of spine images containing spines; identifying the spine images to obtain keywords corresponding to the spines; judging whether the similarity between the keywords corresponding to any spine and book information in a book database reaches a first similarity threshold value: if so, finishing checking the books corresponding to the book spine; if not, taking the key word corresponding to the spine and at least one key word corresponding to the spine around the spine as an integral key word, and continuously judging whether the similarity between the integral key word and the book information in the book database reaches a second similarity threshold value: if so, finishing checking books corresponding to the spines forming the whole keywords; if not, determining that the inventory fails.

In an embodiment of the present invention, an implementation manner of cutting the bookshelf image into a plurality of spine images including a spine includes: detecting the edge of the bookshelf image by using an edge detection algorithm; and identifying straight lines in the bookshelf image, and cutting the bookshelf image into a plurality of spine images containing the spines according to the identified straight lines.

In an embodiment of the present invention, the book inventory identification method based on images and texts further includes: and carrying out direction correction and/or definition correction on the spine image.

In an embodiment of the present invention, the OCR character model is used to identify the spine image, and obtain the keywords corresponding to each spine.

The embodiment of the invention also provides a book inventory identification device based on images and texts, which comprises: the acquisition processing module is used for acquiring bookshelf images and cutting the bookshelf images into a plurality of spine images containing spines; the recognition module is used for recognizing the spine images and acquiring keywords corresponding to the spines; the judging and matching module is used for judging whether the similarity between the keyword corresponding to any spine and the book information in the book database reaches a first similarity threshold value: if so, finishing checking the books corresponding to the book spine; if not, taking the key word corresponding to the spine and at least one key word corresponding to the spine around the spine as an integral key word, and continuously judging whether the similarity between the integral key word and the book information in the book database reaches a second similarity threshold value: if so, finishing checking books corresponding to the spines forming the whole keywords; if not, determining that the inventory fails.

In an embodiment of the present invention, the acquisition processing module includes: the image acquisition unit is used for acquiring a bookshelf image from the image shooting equipment; an edge detection unit for detecting an edge of the bookshelf image by using an edge detection algorithm; and the cutting unit is used for identifying straight lines in the bookshelf image and cutting the bookshelf image into a plurality of spine images containing the spine according to the identified straight lines.

In an embodiment of the present invention, the acquisition processing module further includes: and the correction unit is used for carrying out direction correction and/or definition correction on the spine image.

In an embodiment of the present invention, the recognition module recognizes the spine image by using an OCR character model, and obtains a keyword corresponding to each spine.

Embodiments of the present invention also provide an electronic device, comprising a processor and a memory, the memory storing program instructions; the processor runs the program instructions to realize the book inventory identification method based on the images and the texts.

The embodiment of the invention also provides a book checking identification system, which comprises: the image shooting equipment is specially arranged on a moving vehicle and is used for shooting bookshelf images; the electronic device as described above connected to the image pickup device.

As described above, the book inventory identification method, apparatus, device and system based on images and texts of the present invention have the following advantages:

the invention obtains the book information through the visual analysis of the camera and the matching of the text, provides basis and reference for the automatic book checking of the subsequent library, can assist the automatic book checking of the intelligent library, has stronger adaptability, wider application range, lower cost and better universality compared with the traditional schemes of radio frequency identification technology RFID and the like, does not depend on a large amount of manpower to carry out manual checking, and the like.

Drawings

Fig. 1 is a flow chart illustrating a book inventory identification method based on images and texts according to the present invention.

Fig. 2 is a flowchart illustrating an implementation manner of cutting a bookshelf image into a plurality of spine images including a spine according to the method for identifying book inventory based on images and texts.

Fig. 3 is a schematic diagram illustrating the overall implementation process of the book inventory identification method based on images and texts according to the present invention.

Fig. 4 is a schematic diagram showing the principle structure of the book inventory recognition device based on images and texts.

Fig. 5 is a schematic structural diagram illustrating the principle of the acquisition processing module in the book inventory identification device based on images and texts according to the present invention.

Fig. 6 is a schematic structural diagram of an electronic device in an embodiment of the present application.

Fig. 7 is a schematic structural diagram of a book inventory identification system in an embodiment of the present application.

Description of the element reference numerals

1 book checking identification system

10 electronic device

1101 processor

1102 memory

20 image pickup apparatus

100 book checking identification device based on image and text

110 acquisition processing module

111 image acquisition unit

112 edge detection unit

113 cutting unit

114 correction unit

120 identification module

130 judging and matching module

S100 to S800

S121 to S122

Detailed Description

The embodiments of the present invention are described below with reference to specific embodiments, and other advantages and effects of the present invention will be easily understood by those skilled in the art from the disclosure of the present specification. The invention is capable of other and different embodiments and of being practiced or of being carried out in various ways, and its several details are capable of modification in various respects, all without departing from the spirit and scope of the present invention. It is to be noted that the features in the following embodiments and examples may be combined with each other without conflict.

The embodiment aims to provide a book checking identification method, a book checking identification device, book checking equipment and a book checking identification system based on images and texts, which are independent of book types, high in reliability and low in cost, and are used for solving the problems of low book checking efficiency and low book checking accuracy in the prior art.

The book checking identification method is a method combining visual identification and text matching, and solves the problems that a library book identification system based on a radio frequency identification technology is high in cost, privacy is leaked, information conflicts cannot be obtained, book sequencing cannot be achieved, and the like.

The principles and embodiments of the image and text-based book inventory identification method, apparatus, device and system of the present invention will be described in detail below, so that those skilled in the art can understand the image and text-based book inventory identification method, apparatus, device and system without creative work.

Example 1

As shown in fig. 1, the present embodiment provides a book inventory identification method based on images and texts, including:

step S100: collecting a bookshelf image, and cutting the bookshelf image into a plurality of spine images containing spines;

step S200: identifying the spine images to obtain keywords corresponding to the spines;

step S300: judging whether the similarity between the keywords corresponding to any spine and book information in a book database reaches a first similarity threshold value: if not, continue to execute step S500, if yes, continue to execute step S400: finishing the inventory of the books corresponding to the book spine;

step S500: taking the key word corresponding to the spine and at least one key word corresponding to the spine around the spine as an integral key word;

step S600: continuously judging whether the similarity between the whole keywords and the book information in the book database reaches a second similarity threshold value: if yes, continue to execute step S700: finishing the inventory of books corresponding to the book spines forming the whole keywords; if not, continue to execute step S800: and determining that the inventory fails.

The following describes steps S100 to S800 of the book inventory identification method based on images and texts in this embodiment in detail.

Step S100: the method comprises the steps of collecting bookshelf images, and cutting the bookshelf images into a plurality of spine images containing spines.

Specifically, in this embodiment, as shown in fig. 2, one implementation manner of cutting the bookshelf image into a plurality of spine images including a spine includes:

step S110: detecting the edge of the bookshelf image by using an edge detection algorithm;

step S120: and identifying straight lines in the bookshelf image, and cutting the bookshelf image into a plurality of spine images containing the spines according to the identified straight lines.

The edge detection may be performed by using, but not limited to, an edge detection algorithm such as canny algorithm, Sobel algorithm, Laplacian algorithm, and the like. Edge detection of images is well known to those skilled in the art and will not be described in detail herein.

In this embodiment, the straight line recognition algorithm for recognizing the straight line in the bookshelf image may adopt an LSD straight line detection method and a hough transform method.

In other words, in this embodiment, the bookshelf image in the camera is acquired in real time, for example, an optimized convolution operator is used to perform convolution preprocessing on the image, and a point with a gradient amplitude larger than a threshold is marked as an edge, so that the edge of the bookshelf image is detected; then, straight lines in the edges of the bookshelf image are identified by Hough transform, and the bookshelf image is cut according to the straight lines, so that the bookshelf image is cut into a plurality of spine images containing the spine.

In this embodiment, the book inventory identification method based on images and texts further includes: and carrying out direction correction and/or definition correction on the spine image.

For example, the spine image is corrected by a function such as an affine function, so that characters displayed on the spine image are clear, and character recognition is performed on the corrected spine image.

Step S200: and identifying the spine image to obtain keywords corresponding to each spine.

In this embodiment, the OCR character model is used to identify the spine image, and obtain the keywords corresponding to each spine. OCR character recognition by an OCR character model is well known to those skilled in the art and will not be described in detail herein.

Step S300: judging whether the similarity between the keywords corresponding to any spine and book information in a book database reaches a first similarity threshold value: if not, continue to execute step S500, if yes, continue to execute step S400: and finishing the inventory of the book corresponding to the book spine.

Step S500: and taking the key word corresponding to the book spine and at least one key word corresponding to the book spine around the book spine as an integral key word.

Wherein the keywords and the overall keywords are book information. In this embodiment, the book information identified by the book is text-matched with the book information in the book database, and if the similarity between the keyword information identified by the book and the text of a certain book in the book database reaches a set threshold, the matching is considered to be successful; otherwise, putting the keywords extracted from the books arranged around the books into a book database for overall matching, judging the text similarity again, if the text similarity reaches a threshold value, successfully matching, and outputting the obtained result to a subsequent system after successful matching so as to obtain book information for book inventory.

As shown in fig. 3, the specific implementation process of the book inventory identification method based on images and texts in this embodiment is as follows:

firstly, shooting by a camera to obtain a bookshelf image, collecting the bookshelf image shot by the camera, detecting the edge of a book in the bookshelf image, cutting the bookshelf image into a plurality of spine images, and correcting characters on the spine images to be clear in the front; processing and recognizing the image containing the book spine by OCR recognition, extracting keywords, performing text matching on the recognized book information and the book information in the book database, and if the text similarity between the book and a certain book in the database reaches a set threshold, determining that the matching is successful; otherwise, the invention puts the key words extracted from the books arranged around the books into the database for integral matching, and then carries out text similarity judgment again, if the text similarity reaches the threshold value, the matching is successful, and if the matching is unsuccessful, the relevant personnel is informed to carry out special treatment on the books which can not be identified. Thereby obtaining book information and carrying out book identification and inventory.

According to the book inventory identification method based on the images and the texts, book identification is realized through comprehensive application of camera data collection, a character identification technology, a computer mode identification technology, a text matching technology and the like, so that automatic book inventory of an intelligent library is assisted, book identification is realized, and library inventory can be conveniently performed by subsequent equipment library managers. Compared with traditional schemes such as a radio frequency identification technology RFID and the like, the book checking identification method based on the images and the texts is stronger in adaptability, wider in application range, lower in cost, independent of expensive radio frequency identification bar codes, independent of a large amount of manpower for manual checking and good in universality.

And the book checking identification method based on the images and the texts can acquire the images of the book shelf, the characters on the spine of the book and the book information so as to sort and check the books, solve the problems that the books are mistakenly placed, the books are not placed, and the like, and facilitate observation and further operation and inspection of operators.

Example 2

As shown in fig. 4, the book inventory identification device 100 based on images and texts of the present embodiment includes: an acquisition processing module 110, an identification module 120 and a judgment matching module 130.

In this embodiment, the collecting and processing module 110 is configured to collect a bookshelf image and cut the bookshelf image into a plurality of spine images including a spine.

Specifically, as shown in fig. 5, in the present embodiment, the acquisition processing module 110 includes: an image acquisition unit 111, an edge detection unit 112, a cutting unit 113 and a correction unit 114.

In this embodiment, the image capturing unit 111 is configured to capture a bookshelf image from the image capturing apparatus 20; the edge detection unit 112 is configured to detect an edge of the bookshelf image by using an edge detection algorithm; the cutting unit 113 is configured to identify straight lines in the bookshelf image, and cut the bookshelf image into a plurality of spine images including a spine according to the identified straight lines; the correcting unit 114 is configured to perform direction correction and/or sharpness correction on the spine image.

In this embodiment, for example, the spine image is corrected by using a function such as an affine function, so that the front of the text display on the spine image is clear, and text recognition is performed on the corrected spine image.

In this embodiment, the recognition module 120 is configured to recognize the spine image and obtain keywords corresponding to each spine.

Specifically, in this embodiment, the recognition module 120 recognizes the spine image by using an OCR character model, and obtains keywords corresponding to each spine. OCR character recognition by an OCR character model is well known to those skilled in the art and will not be described in detail herein.

In this embodiment, the determining and matching module 130 is configured to determine whether the similarity between the keyword corresponding to any one of the book spines and the book information in the book database reaches a first similarity threshold: if so, finishing checking the books corresponding to the book spine; if not, taking the key word corresponding to the spine and at least one key word corresponding to the spine around the spine as an integral key word, and continuously judging whether the similarity between the integral key word and the book information in the book database reaches a second similarity threshold value: if so, finishing checking books corresponding to the spines forming the whole keywords; if not, determining that the inventory fails.

The technical features of the specific implementation of the book inventory recognition device 100 based on images and texts in this embodiment are substantially the same as the book inventory recognition method based on images and texts in embodiment 1, and the general technical contents between the embodiments are not repeated.

It should be noted that the division of the modules of the above apparatus is only a logical division, and the actual implementation may be wholly or partially integrated into one physical entity, or may be physically separated. And these modules can be realized in the form of software called by processing element; or may be implemented entirely in hardware; and part of the modules can be realized in the form of calling software by the processing element, and part of the modules can be realized in the form of hardware. For example, the identification module 120 may be a processing element separately set up, or may be integrated into a chip of an electronic terminal, or may be stored in a memory of the terminal in the form of program code, and the processing element of the terminal calls and executes the functions of the tracking calculation module. Other modules are implemented similarly. In addition, all or part of the modules can be integrated together or can be independently realized. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each module above may be implemented by an integrated logic circuit of hardware in a processor element or an instruction in the form of software.

For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as: one or more Application Specific Integrated Circuits (ASICs), or one or more microprocessors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs), among others. For another example, when one of the above modules is implemented in the form of a Processing element scheduler code, the Processing element may be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

Example 3

As shown in fig. 6, the present embodiment further provides an electronic device 10, where the electronic device 10 includes a processor 1101 and a memory 1102.

The electronic device 100 may be, for example, a fixed terminal such as a server, desktop, etc.; or a mobile terminal, such as a notebook computer, a smart phone, or a tablet computer.

The memory 1102 is connected to the processor 1101 through a system bus and is used for storing a computer program, the processor 1101 is coupled to the display 1003 and the memory 1002, and the processor 1101 is used for running the computer program, so that the electronic device 10 executes the book inventory recognition method based on images and texts as described in embodiment 1. The book inventory identification method based on images and texts has been described in detail in embodiment 1, and is not described herein again.

The image and text based book inventory identification method may be applied to many types of electronic devices 10. In an exemplary embodiment, the electronic device 10 may be implemented by one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors, cameras, or other electronic components for performing the above-described image and text based book inventory identification method.

It should be noted that the above-mentioned system bus may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The system bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 6, but this is not intended to represent only one bus or type of bus. The communication interface is used for realizing communication between the database access device and other equipment (such as a client, a read-write library and a read-only library). The Memory may include a Random Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory.

The Processor 1101 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component.

Those of ordinary skill in the art will understand that: all or part of the steps for implementing the above method embodiments may be performed by hardware associated with a computer program. The aforementioned computer program may be stored in a computer readable storage medium. When executed, the program performs steps comprising the method embodiments described above; and the aforementioned storage medium includes: various media that can store program codes, such as ROM, RAM, magnetic or optical disks.

Example 4

As shown in fig. 7, the present embodiment provides a book inventory identification system 1, the book inventory identification system 1 comprising: an image pickup apparatus 20 and an electronic apparatus 10 as described in embodiment 3.

The image capturing device 20 is specially disposed on a mobile vehicle, connected to the electronic device 10 through a wireless network, and configured to capture a bookshelf image and transmit the bookshelf image to the electronic device 10. The image pickup device 20 is preferably, but not limited to, a camera.

In conclusion, the invention obtains the book information through the visual analysis of the camera and the matching of the text, provides basis and reference for the automatic book checking of the subsequent library, can assist the automatic book checking of the intelligent library, has stronger adaptability, wider application range and lower cost compared with the traditional schemes such as the radio frequency identification technology RFID and the like, does not depend on a large amount of manpower to carry out manual checking, and has good universality. Therefore, the invention effectively overcomes various defects in the prior art and has high industrial utilization value.

The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.

Claims

1. A book checking identification method based on images and texts is characterized in that: the method comprises the following steps:

collecting a bookshelf image, and cutting the bookshelf image into a plurality of spine images containing spines;

identifying the spine images to obtain keywords corresponding to the spines;

judging whether the similarity between the keywords corresponding to any spine and book information in a book database reaches a first similarity threshold value:

if so, finishing checking the books corresponding to the book spine;

if not, taking the key word corresponding to the spine and at least one key word corresponding to the spine around the spine as an integral key word, and continuously judging whether the similarity between the integral key word and the book information in the book database reaches a second similarity threshold value:

if so, finishing checking books corresponding to the spines forming the whole keywords;

if not, determining that the inventory fails.

2. The image and text based book inventory identification method of claim 1, wherein: one implementation of cutting the bookshelf image into a plurality of spine images containing a spine includes:

detecting the edge of the bookshelf image by using an edge detection algorithm;

and identifying straight lines in the bookshelf image, and cutting the bookshelf image into a plurality of spine images containing the spines according to the identified straight lines.

3. The image and text based book inventory identification method according to claim 1 or 2, characterized in that: the book inventory identification method based on the images and the texts further comprises the following steps: and carrying out direction correction and/or definition correction on the spine image.

4. The image and text based book inventory identification method as recited in claim 1, wherein: and recognizing the spine image by using an OCR character model, and acquiring the keywords corresponding to the spines.

5. The book checking and identifying device based on the images and the texts is characterized in that: the method comprises the following steps:

the acquisition processing module is used for acquiring bookshelf images and cutting the bookshelf images into a plurality of spine images containing spines;

the recognition module is used for recognizing the spine images and acquiring keywords corresponding to the spines;

the judging and matching module is used for judging whether the similarity between the keyword corresponding to any spine and the book information in the book database reaches a first similarity threshold value: if so, finishing checking the books corresponding to the book spine; if not, taking the key word corresponding to the spine and at least one key word corresponding to the spine around the spine as an integral key word, and continuously judging whether the similarity between the integral key word and the book information in the book database reaches a second similarity threshold value: if so, finishing checking books corresponding to the spines forming the whole keywords; if not, determining that the inventory fails.

6. The image and text based book inventory identification device of claim 5, wherein: the acquisition processing module comprises:

the image acquisition unit is used for acquiring a bookshelf image from the image shooting equipment;

an edge detection unit for detecting an edge of the bookshelf image by using an edge detection algorithm;

and the cutting unit is used for identifying straight lines in the bookshelf image and cutting the bookshelf image into a plurality of spine images containing the spine according to the identified straight lines.

7. The image and text based book inventory identification device of claim 6, wherein: the acquisition processing module further comprises:

and the correction unit is used for carrying out direction correction and/or definition correction on the spine image.

8. The image and text based book inventory identification device of claim 5, wherein: and the recognition module recognizes the spine image by using an OCR character model to acquire keywords corresponding to each spine.

9. An electronic device, characterized in that: comprising a processor and a memory, said memory storing program instructions; the processor executes the program instructions to realize the book inventory recognition method based on the images and the texts as claimed in any one of claims 1 to 4.

10. A book inventory identification system, comprising:

the image shooting equipment is specially arranged on a moving vehicle and is used for shooting bookshelf images;

the electronic device of claim 9 connected to the image capture device.