CN110110122A - Image based on multilayer semanteme depth hash algorithm-text cross-module state retrieval - Google Patents
Image based on multilayer semanteme depth hash algorithm-text cross-module state retrieval Download PDFInfo
- Publication number
- CN110110122A CN110110122A CN201810649234.7A CN201810649234A CN110110122A CN 110110122 A CN110110122 A CN 110110122A CN 201810649234 A CN201810649234 A CN 201810649234A CN 110110122 A CN110110122 A CN 110110122A
- Authority
- CN
- China
- Prior art keywords
- multilayer
- data
- text
- module
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5846—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using extracted text
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/5866—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Library & Information Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Image Analysis (AREA)
Abstract
The present invention relates to the image-text cross-module state retrieval models for combining deep learning and hash method.In order to solve the problems, such as that traditional cross-module state hash method based on deep learning is directly converted into the limitation of single label when handling multi-tag data problem, a kind of depth cross-module state hash algorithm based on multilayer semanteme is proposed.The similarity between data is defined by the cooccurrence relation between multi-tag data, and in this, as the supervision message of network training.Design synthesis considers the loss function of multilayer semantic similarity and two-value similarity, is trained to network, so that feature extraction and Hash codes learning process are unified in a frame, realizes end-to-end study.The algorithm makes full use of the semantic dependency information between data, improves retrieval rate.
Description
Technical field
The present invention relates to cross-module state searching field more particularly to a kind of combination deep learning based on multilayer semanteme with
The image of hash method-text cross-module state searching algorithm.
Background technique
It is universal with the equipment such as smart phone, digital camera with the development of mobile internet, the multimedia on internet
Data are in explosive growth.In information retrieval field, the continuous growth of multimedia big data brings the retrieval application of cross-module state and needs
It asks.And at present mainstream search engine, such as Baidu, Google, must should, a kind of search result of mode is only provided.In addition, with
Deep learning obtains a series of breakthroughs in fields such as computer vision, natural language processings, by multimedia big data with
Artificial intelligence combines, and is the following common development trend in two fields.Therefore, it in conjunction with new technology and new demand, explores new
Cross-module state search modes become one of the challenge urgently to be resolved of current information searching field.
Traditional cross-module state retrieval generallys use the hand-designed feature for relying on domain knowledge, and " semantic gap " problem is still
The difficult point in the field.Deep learning is applied to cross-module state searching field, is not only to solve between different modalities heterogeneous data
" media wide gap " provides a large amount of feature learnings and indicates the advanced research achievement of aspect.However, not with multi-medium data
It is disconnected to increase, the challenge of memory space and recall precision is faced since dimension is excessive using the character representation of deep learning, is caused
It can not adapt to large scale multimedia data retrieval tasks.Meanwhile cross-module state search problem is also faced with truthful data there are multiple marks
The problem of label.Existing solution, which has largely been all made of, converts the relevant single label problem concerning study of two-value for problem, leads
It causes the model learnt that cannot be sufficiently reserved data in the incidence relation of former semantic space, influences final search result
Summary of the invention
It is an object of the invention to overcome the deficiencies in the prior art, the character representation based on deep learning will be combined, and same
When consider image, text both modalities which data two-value similitude and multilayer Semantic Similarity, pass through network using hash method
Training obtains the mapping of data to Hash codes, provides a kind of higher image of retrieval rate-text cross-module state search method.
To achieve the above object, technical solution provided by the present invention are as follows:
It is divided into three modules, respectively depth characteristic extraction module, similarity matrix generation module, Hash codes learn mould
Block；
Wherein, depth characteristic extraction module extracts image and text data feature using deep neural network.The module uses two
A sub-network extracts the structure of image and text modality data characteristics respectively, that is, includes two deep neural networks, one is used for
The feature of image data is extracted, one for extracting text data feature.Using depth convolutional neural networks CNN-F network structure
Carry out image characteristics extraction.The structure of CNN-F is made of 5 layers of convolutional layer and 3 layers of full articulamentum.In the Text character extraction stage,
Text data is modeled with bag of words (Bag-of-Words, BOW) vector first.Based on above-mentioned bag of words, Text character extraction
Network uses multi-layer perception (MLP) (Multi-Layer Perception, the MLP) network being made of three layers of full articulamentum to extract text
Eigen.
For similarity matrix generation module, is generated comprising two-value similarity matrix and multilayer semantic similarity matrix is raw
At.Each generates a cross-module state similarity matrix.For two-value similarity matrixWhen image i is similar to text j
When, matrix is correspondingValue is 1；As image i and text j dissmilarity, matrix is correspondingValue is 0.For multilayer
Semantic similarity matrixIts calculation method is designed according to label cooccurrence relation, so that the class label collection of two samples possesses
When more similar tags, the similarity of sample is bigger, when two tally sets are identical,Reach maximum value 1.When
When the label that two sample labels are concentrated is entirely different,It is minimized 0.
For Hash codes generation module, in order to make the Hash codes learnt retain two-value similarity matrixAnd multilayer language
Adopted similarity matrixIn semantic information, design object function:
Wherein,
By optimizing the objective function, learning network parameter obtains the mapping relations of data and Hash codes.
Compared with prior art, this programme principle and advantage are as follows:
This programme combination deep learning and hash method, overcome traditional-handwork design feature in character representation ability not
The shortcomings that foot and depth characteristic dimension are excessive, are unfavorable for data storage and calculate, and combine two-value similarity and multilayer semanteme phase
It like degree, fully considers across similarity relationship complicated between modal data, the Hash codes learnt is made to retain more semantic informations,
Improve retrieval rate.
Detailed description of the invention
Fig. 1 is that the present invention is based on the image of multilayer semanteme depth hash algorithm-text cross-module state retrieval general frame figures；
Specific embodiment
Below with reference to specific example, the invention will be further described:
It all discusses by taking image and text both modalities which as an example in the present invention.
The present invention provides a kind of image based on multilayer semanteme depth hash algorithm-text cross-module states to retrieve (Deep
Multi-Level Semantic Hashing for Cross-modal Retrieval, DMSH) method, wherein including three
Module: depth characteristic extraction module, similarity matrix generation module, Hash codes study module, as shown in Figure 1；
1 image characteristics extraction network structure of table
Depth characteristic extraction module extracts image and text data feature using deep neural network.Using depth convolution mind
Image characteristics extraction is carried out through network C NN-F network structure, network structure configuration is as shown in table 1.In the Text character extraction stage,
Text data is modeled with bag of words vector first.Based on bag of words, Text character extraction network is used by three layers of full articulamentum
The multi-layer perception (MLP) network of composition extracts text feature, network configuration as shown in table 2
Wherein, 4 step-length convolution are used for conv1 layers, conv2-conv5 layers are all made of 1 step-length convolution.Pad mends side
(Padding), step-length move mode is indicated.It usually shows image border and mends side, so that the picture size exported after convolution and original
Size is consistent.LRN indicates local acknowledgement's normalization (Local Response Normalization).Its mimic biology neuron
Lateral inhibition mechanism make to respond biggish value bigger to the activity creation competition mechanism of local neuron, and inhibit to feed back smaller
Neuron, enhance model generalization ability.The pond technology operated using MAX, takes the maximum value in a certain size of original image, from
And model parameter is effectively reduced, prevent over-fitting.And by Dropout Regularization Technique, lost by random during the training period
A certain number of neurons are abandoned, network over-fitting is prevented.
2 Text character extraction network of table
Wherein, first hidden layer of network is full articulamentum identical with input bag of words vector length, and the second layer is hidden
Layer is the full articulamentum of 4096 dimensions, and third layer is the full articulamentum that length is Hash code length.Output, that is, Text eigenvector of network.
Similarity matrix generation module includes that two-value similarity matrix generates and the generation of multilayer semantic similarity matrix.They
One cross-module state similarity matrix of each self-generatingFor two-value similarity matrixWhen image i is similar to text j, matrix
It is correspondingValue is 1；As image i and text j dissmilarity, matrix is correspondingValue is 0.Wherein, different modalities number
Similitude between is measured by class label.Even image i and text j has one group of common class label, it is considered that
They are similar；Otherwise it is assumed that they are dissimilar.It is defined as follows:
For multilayer semantic similarity matrixUsing a kind of based on the similarity matrix of class label cooccurrence relation
Calculation method；Specific generation method is described below.
For two class label ti,tj, define label similarity:
Wherein, d (ti, tj) indicate two labels semantic distance, be defined as follows:
Wherein,Respectively indicate t in training seti, tjThe number of appearance；Indicate ti, tjTime occurred jointly
Number；NcIndicate the number of all labels in training set.
By defining (2) it is found that s (ti, tj) ∈ [0,1], it indicates when the number that two labels occur jointly is more, they
Similarity it is bigger.According to label similitude s, the similitude between sample can define
For two sample Dm,Dn, define Sample Similarity
Wherein, tm, tmRespectively indicate sample Dm, DnClass label collection；|tm|,|tn| respectively indicate tm, tnNumber；That is Hash label.By definition it is found that when the class label collection of two samples possesses more similar tags, sample
Similarity it is bigger, as two tally set tm, tnWhen identical,Reach maximum value 1.Work as tmIn label and tn
In label it is all dissimilar when,It is minimized 0.Therefore, the semantic similarity matrix based on multi-tagIt can
Using the supervision message as Hash codes learning process.With two-value similarity matrixIt compares,By cross-module state similarity by from
Scattered { 0,1 } is extended to continuous [0,1] section value, remains the semanteme abundant more lain in data category label
Information.
Hash codes study module, withIndicate the sample D learntiCharacteristics of image, i.e. image
The output of feature extraction network；WithIndicate the sample D learntjCharacter features, i.e. character features
Extract the output of network.Respectively indicate the parameter of two depth networks.
In order to make the Hash codes learnt retain two-value similarity matrixSemantic information, using sigmoid cross entropy
Loss function:
Wherein,To guarantee the stability of training process and avoiding overflowing
Out, in implementation phase using the equivalent form of (3-5):
Based on above-mentioned two-value semantic information loss functionIt is further introduced into multilayer semanteme loss functionSo that study
The model arrived, which retains, is included in multilayer semantic similarity matrixIn semantic information more abundant.Here same to use
The equivalent form of sigmoid cross entropy loss function:
Therefore, the complete form of available objective function:
Wherein, F(g)、F(x)The feature vector of the image and text that learn is respectively indicated, they contain similarity matrixIn semantic information；C(g)、C(x)The Hash codes of image and text are respectively indicated, sign () indicates sign function,
Definition is such as formula (3-9).F(g)、F(x)In semantic information C is passed to by sign function(g)、C(x)；Indicate Fibonacci model
Number, E indicate that element value is all 1 vector；μ, ρ, τ are hyper parameter.
C(g)=sign (F(g)) (9)
C(x)=sign (F(x)) (10)
First two of objective function are the negative log-likelihood functions of cross-module state similarity, and by optimizing, this is certifiable to be worked as
When bigger, F (g)*iWith F(x) *jSimilarity it is bigger；It is smaller, F(g) *iWith F(x) *jSimilarity it is smaller.Therefore, optimize the 1st, 2
Item ensure that e-learning to image and the feature of text remain the cross-module state similitude of original semantic space.
The 3rd of objective functionIt is obtained for regularization term by optimizing this
To the Hash codes C of image and text(g)、C(x), and remain the feature F of network extraction(g) *iWith F(x) *jSimilitude.Due to F(g) *iWith F(x) *jThe cross-module state similitude of semantic space is maintained, therefore obtained Hash codes also remain the cross-module of semantic space
State similitude.
By the 4th of optimization object function, so that each of finally obtained Hash codes takes on entire training set
Value is that the number of " 1 " and " -1 " keeps balance, i.e., takes the number of " 1 " and " -1 " respectively to account for half on the same position of Hash codes.This
One constraint can guarantee that each information for including of Hash codes maximizes.
Experiment shows in the training process of network, enables image and text from same data point take identical
Hash codes can preferably promote the performance of network.Therefore, increase addition of constraints C on the basis of former objective function herein(g)=C(x)
=C, final objective function are as follows:
By optimizing the objective function, so that the network parameter that learning characteristic extracts simultaneously and Hash codes indicate, i.e., it will be special
Sign study and Hash codes learning process are unified in a deep learning frame, realize end-to-end study.
In test and application stage, the image or text data of arbitrary single mode are inputted, it can be by training
Network generate its corresponding two-value code vector, i.e. Hash codes.
Specifically, by data point DiImage modalities giNetwork is inputted, its Hash codes is produced by the propagated forward of network
It indicates, calculating process is as follows:
Similarly, to data point DjText modality xj, its corresponding Hash can be generated by the propagated forward of network
Code:
Therefore, the inquiry number of any one mode of given image or text may be implemented in DMSH retrieval model proposed in this paper
According to preceding k search result most like therewith in return different modalities database.In retrieving, inquiry data are calculated first
(Query) the distance between the Hash codes stored in Hash codes and database to be retrieved, then the nearest preceding k of layback is a
Hash codes, corresponding to k number according to be final search result.
Claims (5)
1. a kind of image based on multilayer semanteme depth hash algorithm-text cross-module state search method.It is characterized by: whole frame
Frame includes three modules: depth characteristic extraction module, similarity matrix generation module, Hash codes study module；It is respectively adopted two
A deep neural network extracts image and character features, and feature learning and Hash codes learning process are unified in a frame,
And entire training process is instructed by introducing the multi-level semantic supervision message based on label co-occurrence, the two-value code made is not only
Basic similar/dissimilar relationship in original sample space is remained, and the similarity degree between sample can be distinguished, greatly
Retain the high-level semantic between sample, improves retrieval rate；It is " similar in semantic space by applying to network in structure
Image and text have similar Hash codes in Hamming space " this constraint is trained, directly using Hash codes as network
Output, realizes end-to-end study, to guarantee that the feature learnt adapts to specific retrieval tasks.
2. a kind of image based on multilayer semanteme depth hash algorithm according to claim 1-text cross-module state retrieval side
Method, it is characterised in that: general frame is by depth characteristic extraction module, similarity matrix generation module, Hash codes study module three
A part is constituted, and passes through the two-value for being mapped as being made of in Hamming space "+the 1/-1 " of Unified Form by the data of luv space
Code vector reduces memory space, improves computational efficiency.
3. a kind of image based on multilayer semanteme depth hash algorithm according to claim 1-text cross-module state retrieval side
Method, it is characterised in that: depth characteristic extraction module image and text data is respectively adopted different deep neural networks, extracts
The semantic feature of both modalities which data, to image data, using improved CNN-F network, to text data, using Multilayer Perception
Machine network.
4. a kind of image based on multilayer semanteme depth hash algorithm according to claim 1-text cross-module state retrieval side
Method, it is characterised in that: similarity matrix generation module according to whether have between different modalities data common tag generate two-value phase
Like degree matrix, multilayer semantic similarity matrix is generated according to the similitude size of different modalities data label, retains more multi-tag
The implied meaning information of offer.
5. a kind of image based on multilayer semanteme depth hash algorithm according to claim 1-text cross-module state retrieval side
Method, it is characterised in that: Hash codes study module is by design while retaining data in the two-value similarity information of former semantic space
With the objective function of multilayer semantic similarity information, network is trained, mapping of the learning characteristic space to Hamming space.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810649234.7A CN110110122A (en) | 2018-06-22 | 2018-06-22 | Image based on multilayer semanteme depth hash algorithm-text cross-module state retrieval |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810649234.7A CN110110122A (en) | 2018-06-22 | 2018-06-22 | Image based on multilayer semanteme depth hash algorithm-text cross-module state retrieval |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110110122A true CN110110122A (en) | 2019-08-09 |
Family
ID=67483310
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810649234.7A Pending CN110110122A (en) | 2018-06-22 | 2018-06-22 | Image based on multilayer semanteme depth hash algorithm-text cross-module state retrieval |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110110122A (en) |
Cited By (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110597878A (en) * | 2019-09-16 | 2019-12-20 | 广东工业大学 | Cross-modal retrieval method, device, equipment and medium for multi-modal data |
| CN110765281A (en) * | 2019-11-04 | 2020-02-07 | 山东浪潮人工智能研究院有限公司 | Multi-semantic depth supervision cross-modal Hash retrieval method |
| CN110990597A (en) * | 2019-12-19 | 2020-04-10 | 中国电子科技集团公司信息科学研究院 | Cross-modal data retrieval system based on text semantic mapping and retrieval method thereof |
| CN111026887A (en) * | 2019-12-09 | 2020-04-17 | 武汉科技大学 | Cross-media retrieval method and system |
| CN111125457A (en) * | 2019-12-13 | 2020-05-08 | 山东浪潮人工智能研究院有限公司 | Deep cross-modal Hash retrieval method and device |
| CN111177421A (en) * | 2019-12-30 | 2020-05-19 | 论客科技（广州）有限公司 | Method and device for generating email historical event axis facing digital human |
| CN111221993A (en) * | 2020-01-09 | 2020-06-02 | 山东建筑大学 | Visual media retrieval method based on depth binary detail perception hash |
| CN111353076A (en) * | 2020-02-21 | 2020-06-30 | 华为技术有限公司 | Method for training cross-modal retrieval model, cross-modal retrieval method and related device |
| CN111368176A (en) * | 2020-03-02 | 2020-07-03 | 南京财经大学 | Cross-modal Hash retrieval method and system based on supervision semantic coupling consistency |
| CN111651660A (en) * | 2020-05-28 | 2020-09-11 | 拾音智能科技有限公司 | Method for cross-media retrieval of difficult samples |
| CN111813967A (en) * | 2020-07-14 | 2020-10-23 | 中国科学技术信息研究所 | Retrieval method, retrieval device, computer equipment and storage medium |
| CN111897909A (en) * | 2020-08-03 | 2020-11-06 | 兰州理工大学 | Ciphertext voice retrieval method and system based on deep perception Hash |
| CN111914156A (en) * | 2020-08-14 | 2020-11-10 | 中国科学院自动化研究所 | Cross-modal retrieval method and system for self-adaptive label perception graph convolution network |
| CN112035700A (en) * | 2020-08-31 | 2020-12-04 | 兰州理工大学 | Voice deep hash learning method and system based on CNN |
| CN112100413A (en) * | 2020-09-07 | 2020-12-18 | 济南浪潮高新科技投资发展有限公司 | Cross-modal Hash retrieval method |
| CN112199520A (en) * | 2020-09-19 | 2021-01-08 | 复旦大学 | Cross-modal Hash retrieval algorithm based on fine-grained similarity matrix |
| CN112613451A (en) * | 2020-12-29 | 2021-04-06 | 民生科技有限责任公司 | Modeling method of cross-modal text picture retrieval model |
| CN113095415A (en) * | 2021-04-15 | 2021-07-09 | 齐鲁工业大学 | Cross-modal hashing method and system based on multi-modal attention mechanism |
| CN113157739A (en) * | 2021-04-23 | 2021-07-23 | 平安科技（深圳）有限公司 | Cross-modal retrieval method and device, electronic equipment and storage medium |
| CN113177132A (en) * | 2021-06-30 | 2021-07-27 | 中国海洋大学 | Image retrieval method based on depth cross-modal hash of joint semantic matrix |
| CN113270199A (en) * | 2021-04-30 | 2021-08-17 | 贵州师范大学 | Medical cross-modal multi-scale fusion class guidance hash method and system thereof |
| CN113342922A (en) * | 2021-06-17 | 2021-09-03 | 北京邮电大学 | Cross-modal retrieval method based on fine-grained self-supervision of labels |
| CN113536067A (en) * | 2021-07-20 | 2021-10-22 | 南京邮电大学 | Cross-modal information retrieval method based on semantic fusion |
| CN113658683A (en) * | 2021-08-05 | 2021-11-16 | 重庆金山医疗技术研究院有限公司 | Disease diagnosis system and data recommendation method |
| CN113792207A (en) * | 2021-09-29 | 2021-12-14 | 嘉兴学院 | Cross-modal retrieval method based on multi-level feature representation alignment |
| CN113806580A (en) * | 2021-09-28 | 2021-12-17 | 西安电子科技大学 | Cross-modal Hash retrieval method based on hierarchical semantic structure |
| CN114239730A (en) * | 2021-12-20 | 2022-03-25 | 华侨大学 | Cross-modal retrieval method based on neighbor sorting relation |
| CN114780777A (en) * | 2022-04-06 | 2022-07-22 | 中国科学院上海高等研究院 | Semantic enhancement based cross-modal retrieval method and device, storage medium and terminal |
| CN116955675A (en) * | 2023-09-21 | 2023-10-27 | 中国海洋大学 | Hash image retrieval method and network based on fine-grained similarity relation contrast learning |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2004006128A2 (en) * | 2002-07-09 | 2004-01-15 | Koninklijke Philips Electronics N.V. | Method and apparatus for classification of a data object in a database |
| CN104166982A (en) * | 2014-06-30 | 2014-11-26 | 复旦大学 | Image optimization clustering method based on typical correlation analysis |
| CN104834748A (en) * | 2015-05-25 | 2015-08-12 | 中国科学院自动化研究所 | Image retrieval method utilizing deep semantic to rank hash codes |
| CN105760507A (en) * | 2016-02-23 | 2016-07-13 | 复旦大学 | Cross-modal subject correlation modeling method based on deep learning |
| CN107679580A (en) * | 2017-10-21 | 2018-02-09 | 桂林电子科技大学 | A kind of isomery shift image feeling polarities analysis method based on the potential association of multi-modal depth |
| CN107766555A (en) * | 2017-11-02 | 2018-03-06 | 电子科技大学 | Image search method based on the unsupervised type cross-module state Hash of soft-constraint |
| CN108170755A (en) * | 2017-12-22 | 2018-06-15 | 西安电子科技大学 | Cross-module state Hash search method based on triple depth network |
-
2018
- 2018-06-22 CN CN201810649234.7A patent/CN110110122A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2004006128A2 (en) * | 2002-07-09 | 2004-01-15 | Koninklijke Philips Electronics N.V. | Method and apparatus for classification of a data object in a database |
| CN104166982A (en) * | 2014-06-30 | 2014-11-26 | 复旦大学 | Image optimization clustering method based on typical correlation analysis |
| CN104834748A (en) * | 2015-05-25 | 2015-08-12 | 中国科学院自动化研究所 | Image retrieval method utilizing deep semantic to rank hash codes |
| CN105760507A (en) * | 2016-02-23 | 2016-07-13 | 复旦大学 | Cross-modal subject correlation modeling method based on deep learning |
| CN107679580A (en) * | 2017-10-21 | 2018-02-09 | 桂林电子科技大学 | A kind of isomery shift image feeling polarities analysis method based on the potential association of multi-modal depth |
| CN107766555A (en) * | 2017-11-02 | 2018-03-06 | 电子科技大学 | Image search method based on the unsupervised type cross-module state Hash of soft-constraint |
| CN108170755A (en) * | 2017-12-22 | 2018-06-15 | 西安电子科技大学 | Cross-module state Hash search method based on triple depth network |
Non-Patent Citations (4)
| Title |
|---|
| YUE CAO 等: "Deep Visual-Semantic Hashing for Cross-Modal Retrieval", 《PUBLICATION: KDD "16: PROCEEDINGS OF THE 22ND ACM SIGKDD INTERNATIONAL CONFERENCE ON KNOWLEDGE DISCOVERY AND DATA MINING》 * |
| ZHENYAN JI 等: "A Survey of Personalised Image Retrieval and Recommendation", 《THEORETICAL COMPUTER SCIENCE (2017)》 * |
| 姚伟娜: "基于深度哈希算法的图像—文本跨模态检索研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 张玉宏 等: "深度学习的方法论辨析", 《重庆理工大学学报(社会科学)》 * |
Cited By (48)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110597878A (en) * | 2019-09-16 | 2019-12-20 | 广东工业大学 | Cross-modal retrieval method, device, equipment and medium for multi-modal data |
| CN110597878B (en) * | 2019-09-16 | 2023-09-15 | 广东工业大学 | Cross-modal retrieval method, device, equipment and medium for multi-modal data |
| CN110765281A (en) * | 2019-11-04 | 2020-02-07 | 山东浪潮人工智能研究院有限公司 | Multi-semantic depth supervision cross-modal Hash retrieval method |
| CN111026887A (en) * | 2019-12-09 | 2020-04-17 | 武汉科技大学 | Cross-media retrieval method and system |
| CN111026887B (en) * | 2019-12-09 | 2023-05-23 | 武汉科技大学 | Cross-media retrieval method and system |
| CN111125457A (en) * | 2019-12-13 | 2020-05-08 | 山东浪潮人工智能研究院有限公司 | Deep cross-modal Hash retrieval method and device |
| CN110990597A (en) * | 2019-12-19 | 2020-04-10 | 中国电子科技集团公司信息科学研究院 | Cross-modal data retrieval system based on text semantic mapping and retrieval method thereof |
| CN110990597B (en) * | 2019-12-19 | 2022-11-25 | 中国电子科技集团公司信息科学研究院 | Cross-modal data retrieval system based on text semantic mapping and retrieval method thereof |
| CN111177421A (en) * | 2019-12-30 | 2020-05-19 | 论客科技（广州）有限公司 | Method and device for generating email historical event axis facing digital human |
| WO2021136318A1 (en) * | 2019-12-30 | 2021-07-08 | 论客科技（广州）有限公司 | Digital humanities-oriented email history eventline generating method and apparatus |
| CN111221993B (en) * | 2020-01-09 | 2023-07-07 | 山东建筑大学 | Visual media retrieval method based on depth binary detail perception hash |
| CN111221993A (en) * | 2020-01-09 | 2020-06-02 | 山东建筑大学 | Visual media retrieval method based on depth binary detail perception hash |
| CN111353076B (en) * | 2020-02-21 | 2023-10-10 | 华为云计算技术有限公司 | Method for training cross-modal retrieval model, cross-modal retrieval method and related device |
| CN111353076A (en) * | 2020-02-21 | 2020-06-30 | 华为技术有限公司 | Method for training cross-modal retrieval model, cross-modal retrieval method and related device |
| CN111368176A (en) * | 2020-03-02 | 2020-07-03 | 南京财经大学 | Cross-modal Hash retrieval method and system based on supervision semantic coupling consistency |
| CN111368176B (en) * | 2020-03-02 | 2023-08-18 | 南京财经大学 | Cross-modal hash retrieval method and system based on supervision semantic coupling consistency |
| CN111651660A (en) * | 2020-05-28 | 2020-09-11 | 拾音智能科技有限公司 | Method for cross-media retrieval of difficult samples |
| CN111651660B (en) * | 2020-05-28 | 2023-05-02 | 拾音智能科技有限公司 | Method for cross-media retrieval of difficult samples |
| CN111813967A (en) * | 2020-07-14 | 2020-10-23 | 中国科学技术信息研究所 | Retrieval method, retrieval device, computer equipment and storage medium |
| CN111813967B (en) * | 2020-07-14 | 2024-01-30 | 中国科学技术信息研究所 | Retrieval method, retrieval device, computer equipment and storage medium |
| CN111897909A (en) * | 2020-08-03 | 2020-11-06 | 兰州理工大学 | Ciphertext voice retrieval method and system based on deep perception Hash |
| CN111914156A (en) * | 2020-08-14 | 2020-11-10 | 中国科学院自动化研究所 | Cross-modal retrieval method and system for self-adaptive label perception graph convolution network |
| CN111914156B (en) * | 2020-08-14 | 2023-01-20 | 中国科学院自动化研究所 | Cross-modal retrieval method and system for self-adaptive label perception graph convolution network |
| CN112035700A (en) * | 2020-08-31 | 2020-12-04 | 兰州理工大学 | Voice deep hash learning method and system based on CNN |
| CN112100413A (en) * | 2020-09-07 | 2020-12-18 | 济南浪潮高新科技投资发展有限公司 | Cross-modal Hash retrieval method |
| CN112199520A (en) * | 2020-09-19 | 2021-01-08 | 复旦大学 | Cross-modal Hash retrieval algorithm based on fine-grained similarity matrix |
| CN112199520B (en) * | 2020-09-19 | 2022-07-22 | 复旦大学 | Cross-modal Hash retrieval algorithm based on fine-grained similarity matrix |
| CN112613451A (en) * | 2020-12-29 | 2021-04-06 | 民生科技有限责任公司 | Modeling method of cross-modal text picture retrieval model |
| CN113095415A (en) * | 2021-04-15 | 2021-07-09 | 齐鲁工业大学 | Cross-modal hashing method and system based on multi-modal attention mechanism |
| CN113095415B (en) * | 2021-04-15 | 2022-06-14 | 齐鲁工业大学 | Cross-modal hashing method and system based on multi-modal attention mechanism |
| CN113157739B (en) * | 2021-04-23 | 2024-01-09 | 平安科技（深圳）有限公司 | Cross-modal retrieval method and device, electronic equipment and storage medium |
| CN113157739A (en) * | 2021-04-23 | 2021-07-23 | 平安科技（深圳）有限公司 | Cross-modal retrieval method and device, electronic equipment and storage medium |
| CN113270199A (en) * | 2021-04-30 | 2021-08-17 | 贵州师范大学 | Medical cross-modal multi-scale fusion class guidance hash method and system thereof |
| CN113270199B (en) * | 2021-04-30 | 2024-04-26 | 贵州师范大学 | Medical cross-mode multi-scale fusion class guide hash method and system thereof |
| CN113342922A (en) * | 2021-06-17 | 2021-09-03 | 北京邮电大学 | Cross-modal retrieval method based on fine-grained self-supervision of labels |
| CN113177132A (en) * | 2021-06-30 | 2021-07-27 | 中国海洋大学 | Image retrieval method based on depth cross-modal hash of joint semantic matrix |
| CN113536067B (en) * | 2021-07-20 | 2024-01-05 | 南京邮电大学 | Cross-modal information retrieval method based on semantic fusion |
| CN113536067A (en) * | 2021-07-20 | 2021-10-22 | 南京邮电大学 | Cross-modal information retrieval method based on semantic fusion |
| CN113658683A (en) * | 2021-08-05 | 2021-11-16 | 重庆金山医疗技术研究院有限公司 | Disease diagnosis system and data recommendation method |
| CN113806580A (en) * | 2021-09-28 | 2021-12-17 | 西安电子科技大学 | Cross-modal Hash retrieval method based on hierarchical semantic structure |
| CN113806580B (en) * | 2021-09-28 | 2023-10-20 | 西安电子科技大学 | Cross-modal hash retrieval method based on hierarchical semantic structure |
| CN113792207B (en) * | 2021-09-29 | 2023-11-17 | 嘉兴学院 | Cross-modal retrieval method based on multi-level feature representation alignment |
| CN113792207A (en) * | 2021-09-29 | 2021-12-14 | 嘉兴学院 | Cross-modal retrieval method based on multi-level feature representation alignment |
| CN114239730A (en) * | 2021-12-20 | 2022-03-25 | 华侨大学 | Cross-modal retrieval method based on neighbor sorting relation |
| CN114780777B (en) * | 2022-04-06 | 2022-12-20 | 中国科学院上海高等研究院 | Cross-modal retrieval method and device based on semantic enhancement, storage medium and terminal |
| CN114780777A (en) * | 2022-04-06 | 2022-07-22 | 中国科学院上海高等研究院 | Semantic enhancement based cross-modal retrieval method and device, storage medium and terminal |
| CN116955675A (en) * | 2023-09-21 | 2023-10-27 | 中国海洋大学 | Hash image retrieval method and network based on fine-grained similarity relation contrast learning |
| CN116955675B (en) * | 2023-09-21 | 2023-12-12 | 中国海洋大学 | Hash image retrieval method and network based on fine-grained similarity relation contrast learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110110122A (en) | Image based on multilayer semanteme depth hash algorithm-text cross-module state retrieval | |
| Yan et al. | Cross-modality bridging and knowledge transferring for image understanding | |
| Tautkute et al. | Deepstyle: Multimodal search engine for fashion and interior design | |
| Piras et al. | Information fusion in content based image retrieval: A comprehensive overview | |
| Hoi et al. | A semi-supervised active learning framework for image retrieval | |
| Shi et al. | Deep adaptively-enhanced hashing with discriminative similarity guidance for unsupervised cross-modal retrieval | |
| Castellano et al. | Leveraging knowledge graphs and deep learning for automatic art analysis | |
| CN108959522B (en) | Migration retrieval method based on semi-supervised countermeasure generation network | |
| Lin et al. | Mask cross-modal hashing networks | |
| De Beul et al. | An ontology for video human movement representation based on benesh notation | |
| US11860932B2 (en) | Scene graph embeddings using relative similarity supervision | |
| CN110598022B (en) | Image retrieval system and method based on robust deep hash network | |
| CN112528136A (en) | Viewpoint label generation method and device, electronic equipment and storage medium | |
| Maheshwari et al. | Scene graph embeddings using relative similarity supervision | |
| Singhania et al. | Text-based image retrieval using deep learning | |
| Bouchakwa et al. | A review on visual content-based and users’ tags-based image annotation: methods and techniques | |
| Li et al. | Multimodal fusion with co-attention mechanism | |
| Zhu et al. | Cross-modal retrieval: a systematic review of methods and future directions | |
| Sharma et al. | Evolution of visual data captioning Methods, Datasets, and evaluation Metrics: A comprehensive survey | |
| Xue et al. | Learning semantic dependencies with channel correlation for multi-label classification | |
| CN109255098B (en) | Matrix decomposition hash method based on reconstruction constraint | |
| Zhou et al. | Disambiguating named entities with deep supervised learning via crowd labels | |
| Goyal et al. | A Review on Different Content Based Image Retrieval Techniques Using High Level Semantic Feature | |
| Ke et al. | Real web community based automatic image annotation | |
| An et al. | Pedestrian Reidentification Algorithm Based on Deconvolution Network Feature Extraction‐Multilayer Attention Mechanism Convolutional Neural Network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190809 |
|
| RJ01 | Rejection of invention patent application after publication |