US11681737B2 - Electronic media retrieval - Google Patents
Electronic media retrieval Download PDFInfo
- Publication number
- US11681737B2 US11681737B2 US16/843,218 US202016843218A US11681737B2 US 11681737 B2 US11681737 B2 US 11681737B2 US 202016843218 A US202016843218 A US 202016843218A US 11681737 B2 US11681737 B2 US 11681737B2
- Authority
- US
- United States
- Prior art keywords
- user
- cluster
- media
- query
- media items
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/55—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/43—Querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/43—Querying
- G06F16/438—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
- G06F16/45—Clustering; Classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/535—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/903—Querying
- G06F16/9035—Filtering based on additional data, e.g. user or group profiles
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
Definitions
- This disclosure relates to techniques for retrieving electronic media, and more particularly, to techniques for retrieving electronic media using a graph convolutional neural network (GCN).
- GCN graph convolutional neural network
- Content retrieval systems enable users to enter one or more keywords to retrieve desired electronic media such as an image, a movie, a computer program, a song, and the like.
- the content retrieval system might include a text field to enable the user to enter the keyword(s) and a pane that displays the media that best matches the entered keyword(s).
- Personalized content retrieval systems consider past user behavior when performing a search for related electronic media. For example, if a user frequently clicked on a landscape image, the user would be more likely to view similar landscape images in future queries.
- Embodiments of the present disclosure provide techniques for retrieving electronic media (e.g., images, movies, computer programs, songs, etc.) from user and item embeddings learned by leveraging user behavior data in a Graph Convolutional Neural Network (GCN) model.
- GCN Graph Convolutional Neural Network
- at least one of the exemplary embodiments provides a new framework referred to herein as a Cluster-Augmented Graph Convolutional Network (CA-GCN) that utilizes user-item interaction data to enhance both user and item embeddings in a GCN and also leverages item-item similarities in a clustering structure to augment sparse user behavior data.
- CA-GCN Cluster-Augmented Graph Convolutional Network
- a computer-implemented method for retrieving at least one media item from a set of media items in response to a given query from a given user of a set of users includes: generating a graph including first vertices representing the set of users, second vertices representing the set of media items, the first vertices connected to the second vertices via edges, each second vertex associated with one or more item embeddings of the media items and each edge associated with query embeddings of a query; performing a clustering on the set of media items to generate a plurality of clusters; generating cluster embeddings for each cluster from the item embeddings of the media items within the corresponding cluster; generating augmented query embeddings for each cluster from the cluster embedding of the corresponding cluster and the query embeddings of the edges that pass through the corresponding cluster; inputting the cluster embeddings and the augmented query embeddings to a first layer of a graph convolutional network (GCN)
- GCN graph convolutional network
- a system for retrieving at least one media item from a set of media items in response to a given query from a given user of a set of users includes: a client device including a graphical user interface configured to enable the given user to enter the given query and a computer program configured to output a request including a user identifier (ID) of the given user and the given query over a computer network; and a server configured to generate a graph from the set of users, the set of media items, and queries issued by the set of users for the set of media items, generate clusters from the set of media items, augment the graph to include the clusters, generate embeddings from the augmented graph, train a graph convolutional neural network (GCN) using the embeddings, apply the user ID and the given query to the GCN to generate a score for each of the media items, and output the media items to the client device having the score exceeding a threshold.
- a server configured to generate a graph from the set of users, the set of media items, and queries issued by the set of users
- a computer-implemented method for retrieving at least one media item from a set of media items in response to a given query from a given user of a set of users includes: generating a graph including first vertices representing the set of users, second vertices representing the set of media items, the first vertices connected to the second vertices via edges, each second vertex associated with one or more item embeddings of the media items and each edge associated with query embeddings of a query; performing a clustering on the set of media items to generate a plurality of clusters; generating cluster embeddings for each cluster from the item embeddings of the media items within the corresponding cluster; generating augmented query embeddings for each cluster from the cluster embedding of the corresponding cluster and the query embeddings of the edges that pass through the corresponding cluster; inputting the cluster embeddings and the augmented query embeddings to a layer of a graph convolutional network (GCN) to
- FIG. 1 is an illustration of an environment in an example implementation that is operable to employ techniques described herein.
- FIG. 2 illustrates a flow of the CA-GCN framework according to an exemplary embodiment of the disclosure.
- FIG. 3 illustrates an exemplary augmented graph generated as a result of clustering.
- FIG. 4 illustrates a method of generating additional embeddings for a user-item graph according to an exemplary embodiment of the disclosure.
- FIG. 5 is a graphical illustration of a neural network to implement an exemplary embodiment of the disclosure.
- FIG. 6 illustrates an example system including various components of an example device that can be implemented as any type of computing device as described and/or utilized with reference to FIGS. 1 - 6 to implement embodiments of the techniques described herein.
- search services and content repositories provide tools that allow users to search for content items having desired objects and/or visual characteristics.
- Conventional techniques for searching content find target items based on their matching with a given text query. Most of these methods aim to embed items and queries to the same space to facilitate matching between items and queries, but lead to sub-optimal performance since they ignore the personalized information.
- Conventional recommendation systems provide personalized results based on user history data, but are incapable of operating on user entered queries. Accordingly, such systems provide less relevant results.
- Other conventional recommendation systems make use of collaborative filtering. However due to data sparsity, such techniques may result in unreliable similarity information and poor recommendation quality.
- Some conventional techniques learn enhanced embeddings for recommendation by using a graph convolutional network (GCN).
- GCN graph convolutional network
- a GCN models users and items in a bipartite graph, propagates click information in a neural network, and captures the non-linear interactions.
- the similarities amongst items are further leveraged to augment the graph representation.
- the item similarities are incorporated by grouping similar items into clusters and then the clusters are represented as hidden super-nodes in the graph to address the sparsity issue of user behavior data.
- the information is propagated between users and item clusters to generate representative user embeddings and also the information may be propagated between item clusters and individual items to refine media item embeddings.
- This technique significantly improves retrieval performance and can be used to generate enhanced media item (e.g., image) embeddings that better reflect high-level semantics (e.g., emotion) and user preferences.
- the embodiment in prior history User-1 clicked a number of “alphabet”-related clip-art-styled images and User-2 clicked monochromatic color images, the embodiment returns with recommendation on searched objects, but in clip-art-styled images for user-1 and more monochromatic searched objects for User-2. According to at least one embodiment of the disclosure, a 38% improvement in terms of recall at the top 10% of images in an image retrieval task is achieved.
- Embodiments of the disclosure dynamically enhance the item embeddings by propagating click information between users and items to provide results that better capture high-level semantic concepts.
- conventional recommendation systems rely on fixed item embeddings, and are only capable of providing results that are visually similar to a query image, and sometimes even provide results semantically opposite to the query image.
- GCN Graph Convolutional Network
- V nodes or vertices of the Graph G
- E edges of the graph G that connect the nodes together.
- the GCN takes as an input feature matrix N ⁇ F 0 feature matrix X, where N is the number of nodes and F 0 is the number of input feature dimension for each node, and an N ⁇ N matrix representation of the graph structure such as the adjacency matrix A of G.
- the term “media item” may refer to electronic media such as electronic image (e.g., having formats such as JPEG, TIFF, GIF, BMP, PNG, etc.), an electronic movie (e.g., having formats such as AVI, WMV, MOV, MP4, FLV, etc.), and electronic song (e.g., having formats such as WAV, AIFF, AU, MPEG, etc.), computer software, etc.
- electronic image e.g., having formats such as JPEG, TIFF, GIF, BMP, PNG, etc.
- an electronic movie e.g., having formats such as AVI, WMV, MOV, MP4, FLV, etc.
- electronic song e.g., having formats such as WAV, AIFF, AU, MPEG, etc.
- computer software etc.
- media item embedding may refer to one or more textual keywords that are associated with a media item such as an image, a movie, a song, computer program, or song.
- user-item interaction data may refer to a set of (i j, l) tuples, where each tuple means that for each user j, the user j searches for a specific query l and then clicks on or purchases media items i from the retrieved results
- CA-GCN Cluster-Augmented Graph Convolutional Network
- CA-GCN may refer to a graph convolution neural network (GCN) generated from the user-item interaction data, but further augmented with clusters generated from the media items of the GCN.
- the users, media items, and clusters are all nodes in the CA-GCN and the queries are edges in the CA-CGN. If a user previously clicked on a media item displayed as a result of entering a query, the node representing the user is connected via an edge representing the query to a node representing a cluster including the media item and other related media items determining according to a clustering previously performed on the media items.
- user embedding may refer to one or more textual keywords that are associated with a particular user, which may include textual keywords associated with media items the user has an interest in or is predicted to have an interest in.
- cluster embedding may refer to one or more textual keywords that are associated with a group of related media items.
- query embedding may refer to one more textual keywords within a query of a user that relates to a media item.
- k-means clustering refers to an unsupervised machine learning algorithm that groups similar data points together and discovers underlying patterns.
- the algorithm looks for a fixed number (k) of clusters in a dataset, where each cluster is a collection of data points aggregated together because of certain similarities.
- the algorithm identifies k number of centroids (e.g., representing centers of the clusters), and then allocates every data point to the nearest cluster, while keeping the centroids as small as possible.
- the ‘means’ in the K-means refers to averaging of the data (i.e., finding the centroid).
- Example implementation details and procedures are then described which may be performed in the example environment as well as other environments. Consequently, performance of the example procedures is not limited to the example environment and the example environment is not limited to performance of the example procedures.
- FIG. 1 is an illustration of an environment 100 in an example implementation that is operable media (e.g., image, computer program, movie, song, etc.) retrieval as described herein.
- the illustrated environment 100 includes a service provider system 102 , a client device 104 , and a content repository system 106 that are communicatively coupled, one to another, via a network 108 .
- Computing devices that are usable to implement the service provider system 102 , the client device 104 , and the content repository system 106 may be configured in a variety of ways.
- a computing device may be configured as a desktop computer, a laptop computer, a mobile device (e.g., assuming a handheld configuration such as a tablet or mobile phone), and so forth.
- the computing device may range from full resource devices with substantial memory and processor resources (e.g., personal computers, game consoles) to a low-resource device with limited memory and/or processing resources (e.g., mobile devices).
- a computing device may be representative of a plurality of different devices, such as multiple servers utilized by a business to perform operations “over the cloud” as further described in relation to FIG. 6 .
- the service provider system 102 is illustrated as including a service manager module 110 that is representative of functionality to provide services accessible via the network 108 that are usable to make products or services available to consumers.
- the service manager module 110 may expose content or functionality that is accessible via the network 108 by an application 112 of the client device 104 .
- the application 112 may be configured as a browser, a network-enabled application, and so on, that obtains data from the service provider system 102 via the network 108 .
- This data can be employed by the application 112 to enable a user of the client device 104 to communicate with the service provider system 102 , such as to submit a media (e.g., image, computer program, movie, song, etc.) search query when the service provider system 102 provides functionality to perform media searches.
- a media e.g., image, computer program, movie, song, etc.
- the application 112 includes a media retrieval module 106 to enable a user to perform a media search.
- the user may enter a textual query into a graphical user interface (GUI) 150 presented by the image media retrieval module 114 so that the media retrieval module 114 can send a media request 116 to request that the media retrieval manager module 116 perform the media search using the entered query.
- GUI graphical user interface
- the content repository system 106 (e.g., a host server) is depicted receiving the media request 116 .
- the media request 116 may include the entered query and a user identifier (ID) that uniquely identifies the user that entered the query.
- the service provider system 102 may leverage the content repository system 106 .
- functionality of the content repository system 106 is illustrated as separate from the service provider system 102 , this functionality may also be incorporated as part of the service provider system 102 , further divided among other entities, and so forth.
- the content repository system 106 includes a media retrieval manager module 118 that is implemented at least partially in hardware of a computing device, in part, to perform searches of media content 120 , which is illustrated in storage 122 .
- the media content 120 may represent digital content configured as a variety of different media maintained by the content repository system 106 .
- the media can include images, movies, computer programs, or songs uploaded to the content repository system 106 for sharing, those indexed by the content repository system 106 in connection with crawling the web, and so forth.
- the media retrieval manager module 118 is configured to apply the entered query and a user ID of the user to a cluster-augmented graph convolutional network (CA-GCN) framework that can return scores for each of a plurality of media items.
- CA-GCN cluster-augmented graph convolutional network
- Media 126 including the media items having scores exceeding a certain item threshold can be output to the client device 104 for presentation on the graphical user interface 150 .
- FIG. 2 illustrates a method of creating the CA-GCN framework according to an exemplary embodiment of the disclosure.
- the CA-GCN framework is derived from a CA-GCN that is generated from existing user-item interaction data.
- the user-item interaction data is derived from a set of users and media items , and associated queries.
- the user-item interaction data is a set of (i j, l) tuples. Each tuple means that for each user j, the user j searches for a specific query l and then clicks on or purchases media items i from the retrieved results.
- the item features of a media item such as an image are image embeddings extracted from the image pixel values.
- the query features Q of a query are one or more textual keywords in query for a matching item. For example, if a user is searching for images of Ireland, the keywords of the corresponding query could include “Irish”, “landscape”, etc.
- x i represents the embedding for the media item i and q l is the embedding for the query l.
- Each item i has an initial input feature.
- the set contains edges that connect users and items through specific queries l and is derived from the user-item interaction data.
- Each edge is an (i j, l) tuple where the item i is clicked/purchased by the user j under the query l.
- a clustering is performed on the media items to address the sparsity in the user-item data (step 202 ).
- k-means clustering is performed to generate the clustering.
- each media item is assigned to the closest cluster center, and the cluster center is jointly updated until convergence.
- FIG. 2 shows an example where k-means clustering was performed to generate three clusters. Please note that three clusters are shown merely for ease of illustration as the items may be grouped in various numbers of clusters well exceeding three.
- the clustering is performed to add more potential connections between users and item since users are likely to be interested in other items that are similar to what they clicked or purchased.
- an augmented graph is generated from the user-item graph and the clusters (step 203 ).
- Hidden super-vertices are created that represent each of the clusters, which bridge users to items. For example, each edge that originally connected a user to an item in the user-item graph is replaced with a first edge that connects the user to a cluster of the item, and a second edge that connects the cluster to the item.
- the users are not directly connected to an item in the augmented graph, but to a cluster that includes the item.
- the augmented graph allows consideration of unclicked items. For example, if the user only clicked item 1, but item 2 is so similar that it is included in the same cluster as item 1, the framework essentially treats item 2 as if it had been clicked by the user.
- the set of edges between users and clusters k in the augmented graph are represented as , which contains a set of (j, k) pairs between users and clusters k.
- a matching between users, queries, and items is computed using embeddings generated from the augmented graph (step 204 ).
- the generated embeddings may include user embeddings, media item embeddings, and query embeddings (or user-specific query embeddings).
- FIG. 3 illustrates exemplary items clusters in a media item (e.g., image) retrieval task.
- items x 1 , x 2 , x 3 , and x 4 were grouped into a first cluster C 1 and items x 5 , x 6 , x 7 , and x 8 were grouped into a second cluster, where user u 1 is connected to the second cluster C 1 since user u 1 previously clicked one or more of the items in the second cluster C 2 , user u 2 is connected to the first cluster C 1 and the second cluster C 2 since user u 2 previously clicked one or more of the items in the first cluster C 1 and one or more items in the second cluster C 2 , and user u 3 is connected to the first cluster C 1 and the second third cluster C 2 since user u 3 previously clicked one or more of the items in the first cluster C 1 and one or more items in the second cluster C 2 .
- FIG. 4 illustrates a method according to an exemplary embodiment of the disclosure that can be applied to portions of the augmented graph to update embeddings of the items, create embedding of the clusters, and to create embeddings of the users.
- information is first propagated in the order of item ⁇ cluster ⁇ user (ICU-Flow), and then propagated in the opposite order of user ⁇ cluster ⁇ item (UCI-Flow).
- the method includes generating an embedding for each cluster k (corresponding to a hidden super-node in the augmented graph) by summarizing the embeddings of items of the cluster (step 401 ).
- the embedding of a cluster k may be represented by C k .
- the cluster embedding C k can be computed according to Equation 1 as follows:
- the method next includes generating aggregated query embeddings for each connected (j,k) pair (step 402 ). For example, a (j,k) pair is connected if it is an edge in .
- the aggregated query embeddings are an aggregated representation of which types of queries are searched for by the user j and finally lead to clicks/purchases of items in the cluster k.
- the method propagates information from clusters to users to generate user embeddings for each user j (step 403 ).
- the user embeddings u j for each user j may be generated according to Equation 3, as follows:
- u j ReLU ( W u ⁇ ⁇ ( j , k ) ⁇ ⁇ _ S j , k ⁇ P u [ c k , q ⁇ j , k ] + b u ) ( 3 )
- S j,k is a normalized click/purchase count from user j to cluster k.
- Equation 3 a user embedding of a user is generated by aggregating the information from clicked clusters via a ReLU function, and P u , W u , and b u are model parameters.
- Steps 401 - 403 represent the ICU-Flow.
- the method of FIG. 4 includes propagating the user information back to enhance the cluster embeddings (step 404 ).
- the enhanced cluster embeddings may be generated according to Equation 5, as follows: c k ⁇ ReLU( W c ( c k + ⁇ (j,k) ⁇ tilde over ( ⁇ ) ⁇ j P c u j )+ b c ), (5) where W c , P c , and b c are model parameters.
- the method of FIG. 4 includes refining the embedding of each item using information of user embeddings (step 405 ).
- media i e.g., an image
- its item embedding may be refined using Equation 6, as follows: x i ⁇ ReLU( W c ( x i + ⁇ (j,Cl(i)) ⁇ tilde over ( ⁇ ) ⁇ P c u j )+ b c ).
- the framework can apply a composite loss function to ensure that the query matches both the retrieved media item (e.g., an image) and the clusters.
- user-specific query (USQ) embeddings are generated because different users can refer to different meanings when using the same text query.
- a user-specific query embedding may be generated when an existing user enters a new query.
- the probability for retrieving cluster k can be computed according to equation 8, as follows:
- Equation 9 the probability for retrieving media i (e.g., an image) can be computed according to Equation 9, as follows:
- the media retrieval manager module 118 receives the media request 116 , determines a user specific query embedding for a user associated with the user ID from user embeddings of the augmented graph and query embeddings of the entered query, selects one of the available media items, determines item embeddings x i of the selected item from the augmented graph, and inputs the user specific query embeddings and item embeddings x i into equation 9 to determine a confidence score (e.g., a probability) for the selected item, and repeats this for each of the media items until it computes a confidence score for all the media items.
- the top-k items having a confidence score above a certain item threshold can be presented in the graphical user interface 150 .
- Equation 9 includes a softmax function, which takes as input a vector of K real numbers and normalizes the input vector into a probability distribution consisting of K probabilities proportional to the exponential of the input numbers.
- the peak value of the probability distribution may be used as the confidence score.
- a composite entropy-based loss function L is defined according to equation 10, as follows:
- the above-described framework may be applied to computer software.
- the user could enter a query in the graphical user interface for a certain type of computer game, where the available items are computer games.
- embeddings of such items could include a game price, a discounted price, a game metascore, an overall sentiment, and various other textual tags that describe the game or the genre of the game.
- the graph edges represent the purchase relationship between users and computer games.
- the query has been entered by a new user.
- the system behaves with no personalization (e.g., uses a standard query-based retrieval).
- media items e.g., images
- the system can gradually model the user preferences through the information propagation in the user-item graph. For example, if user u 3 of FIG. 3 is the new user, and the new user has made a few clicks in the inference/testing stage, such click information firstly propagates to the corresponding cluster c 2 , and then further passes to the user via the graph convolutional network.
- the parameters of the graph convolutional network and the embeddings of clusters for inference are obtained in the training stage.
- FIG. 5 illustrates a graphical illustration of a neural network to implement an exemplary embodiment of the disclosure.
- the graphical illustration includes a function 510 that operates on a cluster size (e.g.,
- equation 1 may be used to implement function 510 .
- the illustration further includes a first stage 520 of a neural network that operates on the cluster embeddings c k , the aggregated query embeddings ⁇ tilde over (q) ⁇ j,k , and the normalized click/purchase count S j,k to generate the user embeddings u j .
- the first stage 520 may be represented by Equation 3.
- the illustration further includes a second stage 530 of the neural network.
- the second stage 530 can be operated on the cluster embeddings c k and user embeddings u j generate updated cluster embeddings c k ′ or on the media item embeddings x i and the user embeddings u j to generate updated media item embeddings x i ′.
- the second stage 530 may be represented by Equation 5 and Equation 6.
- the illustration further includes a third stage 540 of the neural network that operates on user embeddings u j and the query embeddings q l to generate query-specific user embeddings q j,l u .
- the third stage 530 may be represented by Equation 7.
- FIG. 6 illustrates an example system generally that includes an example computing device 602 that is representative of one or more computing systems and/or devices that may implement the various techniques described herein. This is illustrated through inclusion of the media retrieval module 114 and the media retrieval manager module 118 .
- the computing device 602 may be, for example, a server of a service provider, a device associated with a client (e.g., a client device), an on-chip system, and/or any other suitable computing device or computing system.
- the processing system 604 is representative of functionality to perform one or more operations using hardware. Accordingly, the processing system 604 is illustrated as including hardware elements 610 that may be configured as processors, functional blocks, and so forth. This may include implementation in hardware as an application specific integrated circuit or other logic device formed using one or more semiconductors.
- the hardware elements 610 are not limited by the materials from which they are formed or the processing mechanisms employed therein.
- processors may be comprised of semiconductor(s) and/or transistors (e.g., electronic integrated circuits (ICs)).
- processor-executable instructions may be electronically-executable instructions.
- the computer-readable storage media 606 is illustrated as including memory/storage 612 .
- the memory/storage 612 represents memory/storage capacity associated with one or more computer-readable media.
- the memory/storage component 612 may include volatile media (such as random access memory (RAM)) and/or nonvolatile media (such as read only memory (ROM), Flash memory, optical disks, magnetic disks, and so forth).
- the memory/storage component 612 may include fixed media (e.g., RAM, ROM, a fixed hard drive, and so on) as well as removable media (e.g., Flash memory, a removable hard drive, an optical disc, and so forth).
- the computer-readable media 606 may be configured in a variety of other ways as further described below.
- Input/output interface(s) 608 are representative of functionality to allow a user to enter commands and information to computing device 602 , and also allow information to be presented to the user and/or other components or devices using various input/output devices.
- input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone, a scanner, touch functionality (e.g., capacitive or other sensors that are configured to detect physical touch), a camera (e.g., which may employ visible or non-visible wavelengths such as infrared frequencies to recognize movement as gestures that do not involve touch), and so forth.
- Examples of output devices include a display device (e.g., a monitor or projector), speakers, a printer, a network card, tactile-response device, and so forth.
- the computing device 602 may be configured in a variety of ways as further described below to support user interaction.
- modules include routines, programs, objects, elements, components, data structures, and so forth that perform particular tasks or implement particular abstract data types.
- module generally represent software, firmware, hardware, or a combination thereof.
- the features of the techniques described herein are platform-independent, meaning that the techniques may be implemented on a variety of commercial computing platforms having a variety of processors.
- Computer-readable media may include a variety of media that may be accessed by the computing device 602 .
- computer-readable media may include “computer-readable storage media” and “computer-readable signal media”.
- Computer-readable storage media may refer to media and/or devices that enable persistent and/or non-transitory storage of information in contrast to mere signal transmission, carrier waves, or signals per se. Thus, computer-readable storage media refers to non-signal bearing media.
- the computer-readable storage media includes hardware such as volatile and non-volatile, removable and non-removable media and/or storage devices implemented in a method or technology suitable for storage of information such as computer readable instructions, data structures, program modules, logic elements/circuits, or other data.
- Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, hard disks, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other storage device, tangible media, or article of manufacture suitable to store the desired information and which may be accessed by a computer.
- hardware elements 610 and computer-readable media 606 are representative of modules, programmable device logic and/or fixed device logic implemented in a hardware form that may be employed in some embodiments to implement at least some aspects of the techniques described herein, such as to perform one or more instructions.
- Hardware may include components of an integrated circuit or on-chip system, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), and other implementations in silicon or other hardware.
- ASIC application-specific integrated circuit
- FPGA field-programmable gate array
- CPLD complex programmable logic device
- hardware may operate as a processing device that performs program tasks defined by instructions and/or logic embodied by the hardware as well as a hardware utilized to store instructions for execution, e.g., the computer-readable storage media described previously.
- software, hardware, or executable modules may be implemented as one or more instructions and/or logic embodied on some form of computer-readable storage media and/or by one or more hardware elements 610 .
- the computing device 602 may be configured to implement particular instructions and/or functions corresponding to the software and/or hardware modules. Accordingly, implementation of a module that is executable by the computing device 602 as software may be achieved at least partially in hardware, e.g., through use of computer-readable storage media and/or hardware elements 610 of the processing system 604 .
- the instructions and/or functions may be executable/operable by one or more articles of manufacture (for example, one or more computing devices 602 and/or processing systems 604 ) to implement techniques, modules, and examples described herein.
- the techniques described herein may be supported by various configurations of the computing device 602 and are not limited to the specific examples of the techniques described herein. This functionality may also be implemented all or in part through use of a distributed system, such as over a “cloud” 614 via a platform 616 as described below.
- the cloud 614 includes and/or is representative of a platform 616 for resources 618 .
- the platform 616 abstracts underlying functionality of hardware (e.g., servers) and software resources of the cloud 614 .
- the resources 618 may include applications and/or data that can be utilized while computer processing is executed on servers that are remote from the computing device 602 .
- Resources 618 can also include services provided over the Internet and/or through a subscriber network, such as a cellular or Wi-Fi network.
- the platform 616 may abstract resources and functions to connect the computing device 602 with other computing devices.
- the platform 616 may also serve to abstract scaling of resources to provide a corresponding level of scale to encountered demand for the resources 618 that are implemented via the platform 616 .
- implementation of functionality described herein may be distributed throughout the system of FIG. 6 .
- the functionality may be implemented in part on the computing device 602 as well as via the platform 616 that abstracts the functionality of the cloud 614 .
- Embodiments of the proposed method can significantly improve retrieval performances.
- the framework can be used to generate enhanced item embeddings with richer semantic information.
- At least one embodiment of the method can be applied to image-based queries. For example, instead of the user entering a textual query into the user interface 150 , the user could enter an image that includes one or more item embeddings (tags).
- the framework is capable of finding nearest neighbor images of the entered image. For example, the nearest neighbors found by the framework have similar desired characteristics and more common tags with the query image. For example, the nearest images found by the framework could have similar components such as an emotion of a person.
- the proposed framework provides a retrieval method based on CA-GCN and sparse graph augmentation. This method effectively extracts personalized features for each user, which helps in targeting the right products to each query. Not only does the framework improve retrieval performance on real-world datasets, but it also can generate item embeddings with enhanced semantics.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Library & Information Science (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
where |Ck| represents the size of the cluster k.
{tilde over (q)} j,k = q l. (2)
where Sj,k is a normalized click/purchase count from user j to cluster k. The normalized click/purchase count Sj,k may be generated according to Equation 4, as follows:
S j,k =n j,k/Σk n j,k, (4)
where nj,k is the number of items clicked/purchased from user j to cluster k. In
c k←ReLU(W c(c k+μ(j,k)∈{tilde over (ε)} j P c u j)+b c), (5)
where Wc, Pc, and bc are model parameters.
x i←ReLU(W c(x i+μ(j,Cl(i))∈{tilde over (ε)} P c u j)+b c). (6)
q j,l u=ReLU(W q[u j ,q l]+b q), (7)
where Wq and bq are model parameters. A user-specific query embedding may be generated when an existing user enters a new query.
where * denotes an entry-wise product. Similarly, given a query l from user j, the probability for retrieving media i (e.g., an image) can be computed according to Equation 9, as follows:
where W1, W2, b1, and b2 are model parameters. When an existing user (e.g., a user already represented by the augmented graph) enters a query into the
where yi,j,l=1 if the user clicked on/purchased the item i under the query l, and yi,j,l=0 otherwise. The hyper-parameter λ controls the weight of the cluster level loss.
Claims (20)
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/843,218 US11681737B2 (en) | 2020-04-08 | 2020-04-08 | Electronic media retrieval |
GB2100958.4A GB2593963A (en) | 2020-04-08 | 2021-01-25 | Electronic media retrieval |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/843,218 US11681737B2 (en) | 2020-04-08 | 2020-04-08 | Electronic media retrieval |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210319056A1 US20210319056A1 (en) | 2021-10-14 |
US11681737B2 true US11681737B2 (en) | 2023-06-20 |
Family
ID=74858836
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/843,218 Active 2040-10-06 US11681737B2 (en) | 2020-04-08 | 2020-04-08 | Electronic media retrieval |
Country Status (2)
Country | Link |
---|---|
US (1) | US11681737B2 (en) |
GB (1) | GB2593963A (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11409821B1 (en) * | 2020-06-23 | 2022-08-09 | Pinterest, Inc. | Recommending content to subscribers |
US11588911B2 (en) * | 2021-01-14 | 2023-02-21 | International Business Machines Corporation | Automatic context aware composing and synchronizing of video and audio transcript |
EP4202725A1 (en) * | 2021-12-22 | 2023-06-28 | Naver Corporation | Joint personalized search and recommendation with hypergraph convolutional networks |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180032518A1 (en) * | 2016-05-20 | 2018-02-01 | Roman Czeslaw Kordasiewicz | Systems and methods for graphical exploration of forensic data |
US20190286752A1 (en) | 2018-03-13 | 2019-09-19 | Pinterest, Inc. | Efficient convolutional network for recommender systems |
CN110674407A (en) | 2019-09-30 | 2020-01-10 | 杭州电子科技大学 | Hybrid recommendation method based on graph convolution neural network |
-
2020
- 2020-04-08 US US16/843,218 patent/US11681737B2/en active Active
-
2021
- 2021-01-25 GB GB2100958.4A patent/GB2593963A/en not_active Withdrawn
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180032518A1 (en) * | 2016-05-20 | 2018-02-01 | Roman Czeslaw Kordasiewicz | Systems and methods for graphical exploration of forensic data |
US20190286752A1 (en) | 2018-03-13 | 2019-09-19 | Pinterest, Inc. | Efficient convolutional network for recommender systems |
CN110674407A (en) | 2019-09-30 | 2020-01-10 | 杭州电子科技大学 | Hybrid recommendation method based on graph convolution neural network |
Non-Patent Citations (12)
Title |
---|
Aggarwal et al., "Multitask Test-To-Visual Embedding With Titles and Clickthrough Data", arXiv:1905.13339v1 [cs.CV] May 30, 2019. |
Examination report dated Feb. 9, 2023 which was issued by the UK Intellectual Property Office in the corresponding UK Appln. No. GB2100958.4. |
Examination report dated Jun. 22, 2022 which was issued by the UK Intellectual Property Office in the corresponding UK Appln. No. GB2100958.4. |
Examination report dated Oct. 19, 2022 which was issued by the UK Intellectual Property Office in the corresponding UK Appln. No. GB2100958.4. |
Hamilton et al., "Inductive Representation Learning on Large Graphs", 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. |
He et al., "Deep Residual Learning for Image Recognition"; arXiv:1512.03385v1 [cs.CV] Dec. 10, 2015. |
Jones, Karen Sparck, "A Statistical Interpretation of Term Specificity and Its Application in Retrieval", Journal of Documentation, vol. 60, No. 5, 2004, pp. 493-502. |
Juan et al., "Graph-Rise: Graph-Regularized Image Semantic Embedding", ACM ISBN 978-1-4503-9999-9/18/06, Feb. 2019. |
Kanungo, et al., "An Efficient k-Means Clustering Algorithm: Analysis and Implementation", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, No. 7, Jul. 2002. |
Kingma, et al., "Adam: a Method for Stochastic Optimization", Published as a conference paper at ICLR 2015. |
Mikolov, et al., "Advances in Pre-Training Distributed Word Representations", arXIV:1712.09405v1 [cs.CL] Dec. 26, 2017. |
Search and Examination report dated Jun. 9, 2021 which was issued by the UK Intellectual Property Office in the corresponding UK Appln. No. GB2100958.4. |
Also Published As
Publication number | Publication date |
---|---|
US20210319056A1 (en) | 2021-10-14 |
GB2593963A (en) | 2021-10-13 |
GB202100958D0 (en) | 2021-03-10 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11347963B2 (en) | Systems and methods for identifying semantically and visually related content | |
US11216496B2 (en) | Visual interactive search | |
US20190188285A1 (en) | Image Search with Embedding-based Models on Online Social Networks | |
US20200089769A1 (en) | Consumer Insights Analysis Using Word Embeddings | |
US10685183B1 (en) | Consumer insights analysis using word embeddings | |
US11681737B2 (en) | Electronic media retrieval | |
US10606883B2 (en) | Selection of initial document collection for visual interactive search | |
JP6261547B2 (en) | Determination device, determination method, and determination program | |
US20170206416A1 (en) | Systems and Methods for Associating an Image with a Business Venue by using Visually-Relevant and Business-Aware Semantics | |
US20210406324A1 (en) | System and method for providing a content item based on computer vision processing of images | |
JP2005316999A (en) | Content propagation for enhanced document retrieval | |
US11636394B2 (en) | Differentiable user-item co-clustering | |
US10803248B1 (en) | Consumer insights analysis using word embeddings | |
US9767417B1 (en) | Category predictions for user behavior | |
Yan et al. | A unified video recommendation by cross-network user modeling | |
US11030539B1 (en) | Consumer insights analysis using word embeddings | |
US10474670B1 (en) | Category predictions with browse node probabilities | |
JP2017201535A (en) | Determination device, learning device, determination method, and determination program | |
US8756120B2 (en) | Hybrid context-sensitive matching algorithm for retrieving product catalogue information | |
US10579630B2 (en) | Content creation from extracted content | |
US10387934B1 (en) | Method medium and system for category prediction for a changed shopping mission | |
US11886809B1 (en) | Identifying templates based on fonts | |
Zeng et al. | Context-aware social media recommendation based on potential group | |
Vu et al. | Detecting emerging rumors by embedding propagation graphs | |
Wen | Development of personalized online systems for web search, recommendations, and e-commerce |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: ADOBE INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ZHAO, HANDONG;KALE, AJINKYA;JIA, XIAOWEI;AND OTHERS;SIGNING DATES FROM 20200331 TO 20200403;REEL/FRAME:052345/0149 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PRE-INTERVIEW COMMUNICATION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |