CN116089723A - Recommendation system recommendation method and device - Google Patents
Recommendation system recommendation method and device Download PDFInfo
- Publication number
- CN116089723A CN116089723A CN202310148085.7A CN202310148085A CN116089723A CN 116089723 A CN116089723 A CN 116089723A CN 202310148085 A CN202310148085 A CN 202310148085A CN 116089723 A CN116089723 A CN 116089723A
- Authority
- CN
- China
- Prior art keywords
- recommendation
- movie
- user
- data
- real
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 33
- 238000012545 processing Methods 0.000 claims abstract description 17
- 230000008520 organization Effects 0.000 claims abstract description 11
- 230000010365 information processing Effects 0.000 claims abstract description 8
- 238000004422 calculation algorithm Methods 0.000 claims description 36
- 230000006399 behavior Effects 0.000 claims description 22
- 238000004364 calculation method Methods 0.000 claims description 13
- 239000013598 vector Substances 0.000 claims description 9
- 238000012423 maintenance Methods 0.000 claims description 6
- 238000004891 communication Methods 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 4
- 238000010801 machine learning Methods 0.000 claims description 3
- 238000007619 statistical method Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 5
- 238000007726 management method Methods 0.000 description 16
- 230000006870 function Effects 0.000 description 8
- 230000008859 change Effects 0.000 description 5
- 239000011159 matrix material Substances 0.000 description 4
- 230000009471 action Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 101100481876 Danio rerio pbk gene Proteins 0.000 description 2
- 101100481878 Mus musculus Pbk gene Proteins 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 230000003247 decreasing effect Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 239000000047 product Substances 0.000 description 2
- 239000013589 supplement Substances 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000001960 triggered effect Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000002238 attenuated effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/73—Querying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/73—Querying
- G06F16/738—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9536—Search customisation based on social or collaborative filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9538—Presentation of query results
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Multimedia (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a recommendation system recommendation method and equipment thereof, and relates to the technical field of information processing; in order to solve the problem that users are not satisfied with the system recommending movies; the recommendation method comprises the following steps: constructing a recommendation system organization architecture; the new user registers and logs in a platform account number to enter a system foreground; selecting or searching more than z favorite movies and types thereof in a system foreground so as to generate recommendation data by the system; the user browses the watching information according to the recommended data; the recommendation system organization architecture comprises a storage layer, a data processing application layer, a display layer and a data loading module; the storage layer includes HDFS. The device of the recommendation method comprises an Ngnix server, a Tomcat server for deploying a back-end display page, a terminal, an information processing unit and a storage unit. The invention saves the searching time, ensures the recommending effect, provides personalized and accurate movie recommendation for different users, and improves the using sense of the users.
Description
Technical Field
The invention relates to the technical field of information processing, in particular to a recommendation system recommendation method and device.
Background
With the development of science and technology, internet users are faced with video feast with abundant and various contents and forms, so that lives of netizens are greatly enriched, and meanwhile, the internet users are impacted by a large amount of redundant and invalid information, so that the internet users are in a lost state which cannot be decided, and the internet users are in a negative influence-information overload-caused by the information big data age. The recommendation system is considered as one of the most effective tools for solving the information overload problem by providing personalized services for users, and has been widely applied in various fields of e-commerce websites, movie and television resource providers, social networks and the like, and by analyzing historical behavior data of users, a user interest model is established so as to predict the preference of the users and recommend content which is possibly interested to the users.
At present, on an intelligent television terminal and a movie or video website, a movie video recommendation system is often available, which can help users find movies interested in the users from an Internet massive video library, but the movie recommendation system used in the prior art usually recommends the movies to the users after grading the movies according to the grading conditions of each platform, and the situation that the users are not satisfied with the movies recommended by the system often occurs, and bad sense is brought to the users, so the defect that personalized and accurate recommendation cannot be performed according to the characteristics of different users is also present, and the recommendation effect is poor. To this end, we propose a recommendation system recommendation method and apparatus capable of providing accurate movie recommendation for users.
Disclosure of Invention
The invention aims to solve the defects in the prior art and provides a recommendation system recommendation method and device thereof.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
a recommendation system recommendation method, comprising the steps of:
s1: constructing a recommendation system organization architecture;
s2: the new user registers and logs in a platform account number to enter a system foreground;
s3: selecting or searching more than z favorite movies and types thereof in a system foreground so as to generate recommendation data by the system;
s4: the user browses the watching information according to the recommended data;
the recommendation system organization architecture comprises a storage layer, a data processing application layer, a display layer and a data loading module;
the storage layer comprises an HDFS, a Hive table, a Mysql block and a Redis block, wherein the Mysql block and the Redis block are used for storing a movie recommendation information table, and the Redis block is used for storing login information of a user, the latest user score and other behavior data information;
the data processing application layer comprises an offline recommendation module for calculating overall preference of the user by using all scoring data, a real-time recommendation module for triggering real-time calculation according to the latest single scoring behavior of the user, a statistical recommendation module for providing basic recommendation service for the user according to scoring information of the user, and a similar recommendation module for recommending other movies similar to the selected movie;
the display layer comprises a front-end display page and a rear-end display page;
the data loading module is used for writing movie information movie.csv, grading information formats.csv and tag information tags.csv into the Hive table through hivectalog.
Preferably: the movie recommendation information table consists of an offline recommendation list generated by an offline recommendation module, a real-time recommendation list generated by a real-time recommendation module, a statistical recommendation list generated by a statistical recommendation module and a similar recommendation list generated by a similar recommendation module.
Preferably: the generation mode of the offline recommendation list comprises the following contents:
a1: an AIink is used for a machine learning algorithm platform based on the Flink;
a2: an ALS collaborative filtering algorithm is selected by an offline recommendation algorithm;
a3: an Alink uses HiveCatalog to read a rate table storing user scoring data from the Hive table;
a4: and reading a user movie table in the Hive table to carry out predictive scoring, and finally generating an offline user recommendation list and storing the offline user recommendation list in a Mysql block.
Preferably: the generation mode of the real-time recommendation list comprises the following contents:
b1: receiving real-time log data;
b2: the log acquisition and transmission tool Flume transmits real-time log data to topics corresponding to the message queue Kafka;
b3: the Flink calculation engine consumes real-time log data collected from the topics corresponding to Kafka, queries the latest behavior data information of the user from the Redis block, and calculates to obtain a real-time recommendation list through a real-time recommendation algorithm;
b4: storing the real-time recommendation list into a Mysql block;
the real-time recommendation list includes a scored recommendation list and a non-scored recommendation list.
Preferably: the generation mode of the statistical recommendation list comprises the following contents:
c1: reading historical scoring data of movies without categories from the Hive table through the Flink SQL, and counting the movies with the largest scores in all the historical scoring data to generate a statistical recommendation list;
c2: the statistical recommendation list is stored in a Mysql block after being sorted from big to small;
the statistical recommendation list comprises a high-quality movie list, a Top list of movies of each class and a recent popular movie list.
Preferably: a statistical method for scoring a most rated movie in all historical scoring data, comprising the following:
1) According to scores of all users on films in the historical score data, periodically calculating average scores of each film;
2) According to all the provided movie categories, the 3-n movies with the highest scores in each category movie set are calculated respectively.
Preferably: the generation mode of the similar recommendation list comprises the following contents:
d1: establishing a dimension vector of each movie according to the movie type;
d2: calculating the similarity between each film and other films by using cosine similarity through the dimension vector;
d3: after obtaining the similarity through the Flink calculation, taking K movies with the highest similarity with each movie as a similar recommendation list of each movie;
d4: and storing the generated similar recommendation list and the corresponding movie id into the Mysql block.
Preferably: the front-end display page comprises a login block for user login and registration, a personal space block for user management and editing of personal information, a movie recommendation block for displaying movie recommendation information table data, a search window for information search, and an expansion block for displaying movie details and movie categories.
Preferably: the back-end display page comprises a login window which is convenient for a system authority manager to log in, and a management maintenance block which is used for managing and maintaining user data, a film recommendation information table and film details;
the rights manager comprises a super rights manager and a common rights manager with different service management rights;
the work content of the management maintenance block comprises the following aspects:
a1: maintaining an offline recommendation list and a statistical recommendation list;
a2: and receiving real-time recommendation and completing merging with the last real-time recommendation list.
The equipment for recommending the system recommending method comprises an Ngnix server, a Tomcat server for deploying a back-end display page, a terminal, an information processing unit and a storage unit, wherein the Ngnix server is deployed with an angular JS for realizing the front-end display page;
the terminal is in communication connection with the information processing unit and the storage unit.
The beneficial effects of the invention are as follows:
1. according to the invention, through setting the recommended system organization architecture, after a new user registers and logs in through a login block, a search window or an expansion block can be used for selecting a plurality of types of favorite movies so as to leave preference basis on a system platform and solve the problem of cold start; and after a new user enters the front-end display page of the system through the login block, the movie recommendation block is selected to display a real-time recommendation list, an offline recommendation list, a statistical recommendation list and a similar recommendation list for the user, so that the user is helped to select a favorite movie, the user is helped to select the favorite movie, the searching time is saved, the recommendation effect is ensured, personalized and accurate movie recommendation is provided for different users, and the use sense of the user is improved.
2. According to the invention, the front-end display page and the back-end display page are channels for effectively interacting with the whole complex recommendation system by a user, the user can clearly see the movie information recommended by each recommendation module through the front-end display page, various operations can be conveniently performed on personal and movie information, and the back-end display page is used for realizing various functions in a logic manner and processing various data including user data and recommendation data.
3. According to the invention, the computing engine Flink is adopted as a computing platform of the recommendation system, so that the performance is excellent, and good real-time performance can be well maintained when the data volume is increased continuously; an algorithm platform Alink based on the Flink is used in the offline recommendation module, so that the operation performance of an offline algorithm in a distributed system is effectively improved; the time weight is integrated into the real-time recommendation algorithm, so that the interest change of the user is more accurately represented; the user behavior weight is integrated into the real-time recommendation algorithm, unscored information of the user is fully utilized, a real-time recommendation list is generated, and the real-time recommendation effect is effectively improved.
4. According to the invention, all recommendation modules can be combined together, so that all modules work in a coordinated manner, the problem that compatibility of multiple modules of the system needs to be repeatedly tested and verified is solved, and movie data is efficiently processed, so that more accurate recommendation results are obtained. The invention is mainly applied to the recommendation service of movie or video websites, and provides recommendation services in all aspects for users through processing historical data and real-time data of the websites.
Drawings
FIG. 1 is a flow chart of a recommendation method of a recommendation system according to the present invention;
fig. 2 is a schematic diagram of a recommendation system organization structure of a recommendation method of a recommendation system according to the present invention.
Detailed Description
The technical scheme of the patent is further described in detail below with reference to the specific embodiments.
Embodiments of the present patent are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the drawings are exemplary only for explaining the present patent and are not to be construed as limiting the present patent.
Example 1:
1-2, a recommendation system recommendation method comprises the following steps:
s1: constructing a recommendation system organization architecture;
s2: the new user registers and logs in a platform account number to enter a system foreground;
s3: selecting or searching more than z favorite movies and types thereof in a system foreground so as to generate recommendation data by the system; preferably, z can be 5-10, the more choices, the more accurate the recommendation index.
S4: and the user browses the watching information according to the recommended data.
The recommendation system organization structure comprises a storage layer as a whole system data support, a data processing application layer for analyzing and processing movie recommendation information and a display layer for information display, wherein the display layer and the data processing application layer are respectively in communication connection with the storage layer, the data processing application layer is in communication connection with the display layer, the data processing application layer stores calculated data into the storage layer based on display layer foreground user operation data,
further, the storage layer comprises an HDFS (distributed file system), a Hive (data warehouse tool) table, a Mysql (relational database management system) block for storing movie recommendation related information and a dis (cache database) block for storing user related information, and preferably, the storage of the Hive table is realized by means of the HDFS (distributed file system); the MovieLens (recommendation system) data can be loaded into a table corresponding to Hive so as to facilitate subsequent use; the storage layer is used for storing all information of the film and the user so as to be used as a data support of the whole system, and the film and the user can be conveniently hung at any time.
Preferably, the Hive table is an intermediate table required in the calculation, such as a cartesian product table of a user and a movie, a model-scored user movie table, and the like, including movie_movie and user_movie, and the like.
Further, the Mysql block is configured to store a movie recommendation information table for query by a service system; preferably, the movie recommendation information table is composed of an offline recommendation list generated by the offline recommendation module, a real-time recommendation list generated by the real-time recommendation module, a statistical recommendation list generated by the statistical recommendation module and a similar recommendation list generated by the similar recommendation module.
Preferably, in the loading stage of the data, a movie recommendation information table is loaded into an elastic search, and fuzzy search of movies is realized through the elastic search.
Still further, the Redis block is used for storing login information of the user and recent scoring and other behavior data information of the user, preferably, the Redis block is in communication connection with the real-time recommendation module, and the information recorded by the Redis block is provided for the real-time recommendation module to use, so that the time delay of the real-time recommendation system is reduced, and the accuracy of real-time recommendation is ensured.
Preferably, the login information of the user includes a platform login management account and the like.
Further, the data processing application layer is a core part of the recommendation system and comprises an offline recommendation module for calculating overall preference of the user by using all scoring data, a real-time recommendation module for triggering real-time calculation according to the latest single scoring behavior of the user, a statistical recommendation module for providing basic recommendation service for the user according to the scoring information of the user and a similar recommendation module for recommending other movies similar to the selected movie; the overall movie preference of the user is evaluated through the offline recommendation module, the preference of the user is clear, and the recommendation accuracy is high; the real-time recommendation module estimates the latest movie preference of the user according to the latest behavior of the user to recommend, so that the user can change at any time along with the behavior change of the user, and the user has variability; the statistical recommendation module is convenient to provide basic recommendation service for the user according to all scoring contents participated by the user, and grasp the general direction of the user; and the similar recommending module is used for conveniently recommending movies similar to the preference categories for the user and providing diversified recommending services for the user.
Preferably, the scoring information of the user includes all the historical scoring data, and scoring movie contents participated by all the users such as movies with the largest historical scoring times.
Additionally, the way in which the offline recommendation module generates the offline recommendation list includes the following:
a1: an offline recommendation algorithm is enabled to be higher in efficiency in a distributed scene by using an Flink-based machine learning algorithm platform Alink;
a2: an ALS collaborative filtering algorithm is selected by an offline recommendation algorithm;
a3: the Alink uses HiveCatalog to read a rate table storing user scoring data from the Hive table for training a model;
a4: and reading a user movie table in the Hive table to carry out predictive scoring, and finally generating an offline user recommendation list and storing the offline user recommendation list in a Mysql block. The algorithm used by the offline recommendation module is an ALS collaborative filtering algorithm, the ALS algorithm is a recommendation algorithm for realizing collaborative filtering based on a model through a matrix decomposition method, the ALS is an alternating least square method, the basic idea is that a sparse matrix is subjected to model decomposition, and the value of a missing item is estimated, so that a basic training model is obtained; then according to the model, the new user and article data can be evaluated; ALS calculates the missing items by using an alternate least square method; the algorithm platform link based on the link can greatly improve the operation performance of the offline recommendation algorithm under the distributed platform.
Additionally, the manner in which the real-time recommendation module generates the real-time recommendation list comprises the following contents:
b1: receiving real-time log data; preferably, the real-time log data generated at the front-end embedded point and the background embedded point are stored in the Tomcat server;
preferably, the front-end embedded point comprises scoring, clicking, sharing, collecting, downloading data and the like of a user;
b2: the method comprises the steps that a log collection and transmission tool (data collection software) Flume transmits real-time log data to topics corresponding to a message queue Kafka;
b3: the Flink computing engine consumes real-time log data collected in the topics corresponding to Kafka, queries the latest behavior data information (such as the latest historical scores of the user) of the user from the Redis block at the same time, and then utilizes the data to calculate and obtain a real-time recommendation list through a real-time recommendation algorithm;
b4: and storing the real-time recommendation list into a streaming_recs table of the Mysql block for the service system to call. Real-time recommendation relies on real-time data and processing flow, the real-time data come from buried data at the front end, scoring, clicking, sharing, collecting and downloading data of a user are all collected in real time, then a log collecting and transmitting tool Flume collects log information into topics corresponding to Kafka, a link computing engine consumes the topics corresponding to Kafka, meanwhile, the latest historical scoring of the user is read from a cache database Redis, a real-time recommendation list is obtained through calculation of the data through a real-time recommendation algorithm, and finally the list is stored into a streaming_recs list of Mysql for a service system to call.
Preferably, a piece of real-time log data triggers a real-time recommendation module.
Further preferably, the latest real-time recommendation list is fused with the previous real-time recommendation list, and the fusion result is displayed to the user on the display layer.
Still further preferably, the real-time recommendation list includes a scored recommendation list and a non-scored recommendation list; the real-time recommendation module generates a recommendation list for the user by using recent scoring data, and reflects recent preferences of the user, such as: for recent user scoring data, if the user has a higher score for a movie, the user is likely to like a similar movie, and if the user has a lower score for a movie, the user is likely not to like a similar movie, and such a recommendation will be more consistent with the recent taste of the user. The real-time recommendation list is relatively quick in change, high surprise degree can be provided for users, and as the real-time recommendation can ensure good real-time performance, the calculated amount of a real-time recommendation algorithm is generally not too large, and the user experience is seriously reduced due to excessively complex calculation.
Additionally, the statistical recommendation module generates a statistical recommendation list in a manner comprising the following contents:
c1: reading historical scoring data of movies without categories from the Hive table through the Flink SQL, and counting the movies with the largest scores in all the historical scoring data to generate a statistical recommendation list;
preferably, the historical scoring data is read in months, i.e., the number of scores for a monthly movie is counted.
C2: and storing the statistical recommendation list into a rate_more_movie table in the Mysql block for the presentation layer to call after sorting from large to small. And reading the grading data set through the Flink SQL, modifying the grading data time into months through a UDF function, counting the grading number of the films per month, and storing the data into the Mysql block after counting.
Preferably, the statistical recommendation list comprises a high-quality movie list, a Top list of movies of each class, a recent popular movie list and the like.
Further, the statistical method for scoring the most movies in all the historical scoring data comprises the following steps:
1) According to scores of all users on films in the historical score data, periodically calculating average scores of each film; the method specifically comprises the steps of reading a scoring data set in a Hive table through a Flink SQL, and realizing average score statistics of a film through executing an SQL sentence;
2) According to all the provided movie categories, respectively calculating 3-n movies with highest scores in each category movie set; after calculating the average score of the whole film, the film set and the film type are subjected to Cartesian product, then the non-conforming items of the film type are filtered, and the data are stored in the genes_top_movies table of Mysql.
Additionally, the similar recommendation module generates a similar recommendation list in a manner comprising the following contents:
d1: establishing a dimension vector of each movie according to the movie type; preferably, each movie has 100 dimension features in a movie hidden feature matrix movie_factors, and the movie hidden feature matrix movie_factors are obtained through an offline recommendation module.
D2: calculating the similarity between each film and other films by using cosine similarity through the dimension vector; by calculating the cosine similarity of the two vectors, the similarity between movies can be obtained.
Preferably, since some popular type labels have a plurality of movies and have little contribution to similarity calculation, the weight of the popular labels is attenuated by adopting a word frequency-inverse document frequency (TF-IDF) method;
d3: after obtaining the similarity through the Flink calculation, taking K movies with the highest similarity with each movie as a similar recommendation list of each movie; preferably, the list of similar recommendations is ordered from high to low in similarity value.
D4: the generated similarity recommendation list and the corresponding movie id are stored in a similarity_movie_movie table of the Mysql block. And recommending the films similar to the selected films for the user through a similar recommending module, and displaying the films for the user on the detail page of the films.
Further, the display layer comprises a front-end display page and a rear-end display page, the front-end display page bears the functions of interacting with a user and displaying service data of each module of the recommendation system, and the rear-end display page timely and accurately provides required data information for the front-end display page.
The front-end display page comprises a login block for user login and registration, a personal space block for user management and editing of personal information, a movie recommendation block for displaying movie recommendation information table data, a search window for information search, and an expansion block for displaying movie details and movie categories; after a new user registers and logs in through the login block, a plurality of types of favorite movies can be selected through the search window or the expansion block so as to leave preference basis on a system platform and solve the problem of cold start; and after a new user enters the front-end display page of the system through the login block, the movie recommendation block is selected to display a real-time recommendation list, an offline recommendation list, a statistical recommendation list and a similar recommendation list to the user, so that the user is helped to select a favorite movie, the user is helped to select the favorite movie, the searching time is saved, the searching of movie information can be carried out through the searching window, the user can select any movie on the front-end display page, and movie details and movie categories of the movie can be displayed so as to be convenient for the user to know in detail.
Preferably, the personal information includes personal data, favorite movies and download movies, score movies, and the like.
Preferably, the movie details comprise clicking, collecting, downloading, sharing, similar movies and the like of the movies; the user can score one or more movies through the personal space block, click, collect, share and download the action data of the operation of the user can be recorded as a recommendation data source of a recommendation system, and the user can manage personal information of the user, modify interest preferences of the user and the like.
The back-end display page comprises a login window which is convenient for a system authority manager to log in, and a management maintenance block which is used for managing and maintaining user data, a film recommendation information table, film details and the like; preferably, the rights manager comprises a super rights manager and a common rights manager with different service management rights;
further, the service management authority of the super authority manager comprises the management of the common authority manager, such as increasing or decreasing the number, modifying the data, inquiring and the like of the common authority manager;
the service management authority of the super authority manager comprises the login and registration of a common authority manager and a user;
as a further supplement, the business management rights of the common rights manager comprise management users, such as increasing or decreasing the number of users, modifying data, inquiring and the like.
As a further supplement, the service management authorities of the super authority manager and the common authority manager also comprise film management, including direct management of a statistical recommendation list and editing of recently-mapped new film information. After a system authority manager logs in the system platform through a login window, corresponding services are managed according to the respective authorities.
Further, the work content of the management maintenance block includes the following aspects:
a1: maintaining an offline recommendation list and a statistical recommendation list;
a2: receiving real-time recommendation and completing merging with a last real-time recommendation list (comprising a scoring recommendation list and a non-scoring recommendation list); because the final list of the real-time recommendation list is not generated in the real-time recommendation algorithm, the final list needs to be combined with the last real-time recommendation list at the service back end.
When the embodiment is used, after a new user registers and logs in through the login block, a plurality of types of favorite movies can be selected through the search window or the expansion block so as to leave preference basis on a system platform and solve the problem of cold start; and after a new user enters the front-end display page of the system through the login block, the movie recommendation block is selected to display a real-time recommendation list, an offline recommendation list, a statistical recommendation list and a similar recommendation list for the user, so that the user is helped to select a favorite movie, the user is helped to select the favorite movie, the searching time is saved, the recommendation effect is ensured, personalized and accurate movie recommendation is provided for different users, and the use sense of the user is improved.
The front-end display page and the back-end display page are channels for effectively interacting with the whole complex recommendation system by a user, the user can clearly see the movie information recommended by each recommendation module through the front-end display page, various operations can be conveniently carried out on personal and movie information, the back-end display page is used for realizing various functions in a logic mode, and various data including user data and recommendation data are processed at the same time.
The invention can combine all the recommendation modules together, coordinate the modules to each other, and process the movie data efficiently, thereby obtaining more accurate recommendation results. The invention is mainly applied to the recommendation service of movie or video websites, and provides recommendation services in all aspects for users through processing historical data and real-time data of the websites.
Example 2:
1-2, the following additions are made on the basis of embodiment 1: the recommendation system organization architecture further comprises a data loading module which is used for writing movie information movie.csv, scoring information ratings.csv and tag information tags.csv into a Hive table of the data warehouse through hivectalog, wherein the data loading module builds a data transmission bridge for a storage layer and a display layer so as to call data for subsequent computing operation;
further, the data loading module is further configured to generate a large table user_movie corresponding to all the combinations of all the users and a large table movie_movie of all the movies and combinations between the movies;
preferably, user_movie is used when producing scores for all user movies in combination in the user offline recommendation;
preferably, movie_movie is used in calculating the similarity between each movie and all other movies in real-time calculation.
The real-time recommendation algorithm relies on implicit feature vectors of the articles obtained in the offline recommendation process, and the similarity between films can be obtained by using the vectors,the similarity between two films is judged by adopting a cosine similarity mode; when a user has a new score, the real-time recommendation list for the user needs to be updated, the latest m scored commodity list of the user u is recorded as Rm, the most similar commodity list of the user u is recorded as L, each commodity j in L belongs to L, and the recommendation priority of the candidate film j to the user u is recorded as P uj The update formula for priority is as follows:
when the similarity sim (j, r) between the movie j and the movie r is smaller than 0.6, the movie j and the movie r are considered to have no correlation and are ignored, wherein sum is the number of articles with similarity exceeding a threshold value, and Rr is the score of a user on the movie r in Rm; in the formula, the number of high-score articles with the similarity with the article j being greater than a threshold value in the recent m scores of the user is counted, and out is counted to the number of low-score articles with the similarity with the article being greater than the threshold value in the recent m scores of the user; one is an enhancement factor and one is an attenuation factor, in the enhancement factor represents the number of high-score movies similar to movie j in the historical scoring movies Rm of the user, and the larger the number is, the more the corresponding movie j should be recommended, so the enhancement factor can increase the recommended value of the movie j for the user u; the corresponding out in the decay factor indicates the number of low score movies in the user's history score movie Rm that are similar to movie j, the larger this number, the less the corresponding movie j should be recommended, so the decay will decrease the recommendation value of movie j for user u. Therefore, the recommended values of each film in the similar list for the user u are obtained by adding the enhancement factor and subtracting the attenuation factor, the top k of the values are ranked, and the values can be used as a real-time recommended list for the user u and stored in a Mysql block of a service database.
Because the primary scoring action of the user u triggers a real-time recommendation, a real-time recommendation list updaterec is produced, and the list is sent to a back-end display page and combined with a list Rec generated by the real-time recommendation triggered by the previous action of the user u, a new recommendation list NewRec is produced after the combination, and the list is used as a final result of the real-time recommendation of the user u.
NewRec＝topK(j∈updated Re c∪Re c,compare＝P uj )
Wherein topK is a function representing finding the largest k movies from the updated Rec and Rec lists, compare=p uj The recommendation priority P is indicated as the comparison object.
Some recent scoring data of the user are used in the real-time recommendation algorithm, the data are also time-sequential, the real-time recommendation algorithm is improved by adding a time decay function to historical data, namely, the manner of giving different time weights to the scoring data of different times is improved, the former offline recommendation algorithm and the real-time recommendation algorithm are mainly based on scoring of the user, preference information of the user can be intuitively obtained from scoring, but the behaviors of the user on a movie website are various, the behaviors of clicking, collecting, downloading, sharing and the like are also included, if the information of the behaviors is fully utilized, the quality of a recommendation result is further improved, better experience is brought to the user, and meanwhile, the preference information amount of the user is different, such as the preference of the user is more highlighted by comparing the collecting behaviors of the user with the clicking behaviors of the user, so that the non-scoring behaviors of the user are set with different weights, the non-scoring behaviors of the user are fused into the updating formula of the movie priority, and then the corresponding recommendation list is produced, and the non-scoring behaviors are triggered once again, the real-time recommendation algorithm is further improved.
When the embodiment is used, the computing engine Flink is adopted as a computing platform of the recommendation system, so that the performance is excellent, and the instantaneity is good; an algorithm platform Alink based on the Flink is used in the offline recommendation module, so that the operation performance of an offline algorithm in a distributed system is effectively improved; the time weight is integrated into the real-time recommendation algorithm, so that the interest change of the user is more accurately represented; the real-time recommendation algorithm is integrated with the user behavior weight, and the unscored information of the user is fully utilized to generate a real-time recommendation list.
Example 3:
the device of the recommendation system recommendation method of embodiment 1, the recommendation device includes an Ngnix server, an AngularJS (front end framework) deployed on the Ngnix server, a Tomcat server for deploying a back end presentation page, a terminal, an information processing unit for loading and implementing a data processing application layer, and a storage unit for implementing a function of storing data; the foreground recommendation service is conveniently realized for the user.
Preferably, the front-end display page is realized through an AngularJS;
preferably, the back-end display page is used for realizing the overall business logic of the Java layer, and is constructed by Spring to meet business requirements. The back-end display page is mainly used for processing data management and some business logics, and is required to process personal information of a user, detailed information of a movie and recommendation list information of each recommendation module.
When the embodiment is used, the Ngnix server and the Tomcat server are integrated through the terminal, so that foreground recommendation service for a user is conveniently realized, and a right manager is also conveniently subjected to page data management.
The foregoing is only a preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art, who is within the scope of the present invention, should make equivalent substitutions or modifications according to the technical scheme of the present invention and the inventive concept thereof, and should be covered by the scope of the present invention.
Claims (10)
1. A recommendation method of a recommendation system, comprising the steps of:
s1: constructing a recommendation system organization architecture;
s2: the new user registers and logs in a platform account number to enter a system foreground;
s3: selecting or searching more than z favorite movies and types thereof in a system foreground so as to generate recommendation data by the system;
s4: the user browses the watching information according to the recommended data;
the recommendation system organization architecture comprises a storage layer, a data processing application layer, a display layer and a data loading module;
the storage layer comprises an HDFS, a Hive table, a Mysql block and a Redis block, wherein the Mysql block and the Redis block are used for storing a movie recommendation information table, and the Redis block is used for storing login information of a user, the latest user score and other behavior data information;
the data processing application layer comprises an offline recommendation module for calculating overall preference of the user by using all scoring data, a real-time recommendation module for triggering real-time calculation according to the latest single scoring behavior of the user, a statistical recommendation module for providing basic recommendation service for the user according to scoring information of the user, and a similar recommendation module for recommending other movies similar to the selected movie;
the display layer comprises a front-end display page and a rear-end display page;
the data loading module is used for writing movie information movie.csv, grading information formats.csv and tag information tags.csv into the Hive table through hivectalog.
2. The recommendation system recommendation method according to claim 1, wherein the movie recommendation information table is composed of an offline recommendation list generated by an offline recommendation module, a real-time recommendation list generated by a real-time recommendation module, a statistical recommendation list generated by a statistical recommendation module, and a similar recommendation list generated by a similar recommendation module.
3. The recommendation system recommendation method according to claim 2, wherein the generation mode of the offline recommendation list comprises the following contents:
a1: an AIink is used for a machine learning algorithm platform based on the Flink;
a2: an ALS collaborative filtering algorithm is selected by an offline recommendation algorithm;
a3: an Alink uses HiveCatalog to read a rate table storing user scoring data from the Hive table;
a4: and reading a user movie table in the Hive table to carry out predictive scoring, and finally generating an offline user recommendation list and storing the offline user recommendation list in a Mysql block.
4. The recommendation system recommendation method according to claim 2, wherein the generation mode of the real-time recommendation list comprises the following contents:
b1: receiving real-time log data;
b2: the log acquisition and transmission tool Flume transmits real-time log data to topics corresponding to the message queue Kafka;
b3: the Flink calculation engine consumes real-time log data collected from the topics corresponding to Kafka, queries the latest behavior data information of the user from the Redis block, and calculates to obtain a real-time recommendation list through a real-time recommendation algorithm;
b4: storing the real-time recommendation list into a Mysql block;
the real-time recommendation list includes a scored recommendation list and a non-scored recommendation list.
5. The recommendation system recommendation method according to claim 2, wherein the generation mode of the statistical recommendation list comprises the following contents:
c1: reading historical scoring data of movies without categories from the Hive table through the Flink SQL, and counting the movies with the largest scores in all the historical scoring data to generate a statistical recommendation list;
c2: the statistical recommendation list is stored in a Mysql block after being sorted from big to small;
the statistical recommendation list comprises a high-quality movie list, a Top list of movies of each class and a recent popular movie list.
6. The recommendation system recommendation method according to claim 5, wherein the statistical method for scoring the most movies among all the historical scoring data comprises the following steps:
1) According to scores of all users on films in the historical score data, periodically calculating average scores of each film;
2) According to all the provided movie categories, the 3-n movies with the highest scores in each category movie set are calculated respectively.
7. The recommendation system recommendation method according to claim 2, wherein the generation mode of the similar recommendation list comprises the following contents:
d1: establishing a dimension vector of each movie according to the movie type;
d2: calculating the similarity between each film and other films by using cosine similarity through the dimension vector;
d3: after obtaining the similarity through the Flink calculation, taking K movies with the highest similarity with each movie as a similar recommendation list of each movie;
d4: and storing the generated similar recommendation list and the corresponding movie id into the Mysql block.
8. The recommendation system recommendation method according to claim 1, wherein the front-end presentation page comprises a login block for user login and registration, a personal space block for user management and editing of personal information, a movie recommendation block for presentation of movie recommendation information table data, a search window for information search, and an expansion block for displaying movie details and movie categories.
9. The recommendation system recommendation method according to claim 8, wherein the back-end presentation page includes a login window for facilitating login by a system rights manager, and a management maintenance block for managing and maintaining user data, movie recommendation information tables, and movie details;
the rights manager comprises a super rights manager and a common rights manager with different service management rights;
the work content of the management maintenance block comprises the following aspects:
a1: maintaining an offline recommendation list and a statistical recommendation list;
a2: and receiving real-time recommendation and completing merging with the last real-time recommendation list.
10. The equipment of the recommendation system recommendation method comprises an Ngnix server, a Tomcat server for deploying a back-end display page, a terminal, an information processing unit and a storage unit, and is characterized in that the Ngnix server is deployed with an AngularJS for realizing the front-end display page;
the terminal is in communication connection with the information processing unit and the storage unit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310148085.7A CN116089723A (en) | 2023-02-22 | 2023-02-22 | Recommendation system recommendation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310148085.7A CN116089723A (en) | 2023-02-22 | 2023-02-22 | Recommendation system recommendation method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116089723A true CN116089723A (en) | 2023-05-09 |
Family
ID=86211988
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310148085.7A Withdrawn CN116089723A (en) | 2023-02-22 | 2023-02-22 | Recommendation system recommendation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116089723A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117194804A (en) * | 2023-11-08 | 2023-12-08 | 上海银行股份有限公司 | Guiding recommendation method and system suitable for operation management system |
-
2023
- 2023-02-22 CN CN202310148085.7A patent/CN116089723A/en not_active Withdrawn
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117194804A (en) * | 2023-11-08 | 2023-12-08 | 上海银行股份有限公司 | Guiding recommendation method and system suitable for operation management system |
| CN117194804B (en) * | 2023-11-08 | 2024-01-26 | 上海银行股份有限公司 | Guiding recommendation method and system suitable for operation management system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11580104B2 (en) | Method, apparatus, device, and storage medium for intention recommendation | |
| US12001439B2 (en) | Information service for facts extracted from differing sources on a wide area network | |
| US9449271B2 (en) | Classifying resources using a deep network | |
| CN110717093B (en) | Movie recommendation system and method based on Spark | |
| US9087332B2 (en) | Adaptive targeting for finding look-alike users | |
| WO2018040069A1 (en) | Information recommendation system and method | |
| CN102982042B (en) | A kind of personalization content recommendation method, platform and system | |
| CN102054003B (en) | Methods and systems for recommending network information and creating network resource index | |
| CN112307762B (en) | Search result sorting method and device, storage medium and electronic device | |
| CN111475509A (en) | Big data-based user portrait and multidimensional analysis system | |
| CN102667761A (en) | Scalable cluster database | |
| EP2307983A1 (en) | Information processing with integrated semantic contexts | |
| EP2301192A1 (en) | Facilitating collaborative searching using semantic contexts associated with information | |
| CN112749330B (en) | Information pushing method, device, computer equipment and storage medium | |
| WO2021179481A1 (en) | Cold start method and apparatus for personalizing and pushing data content, device and storage medium | |
| CN111159341A (en) | Information recommendation method and device based on user investment and financing preference | |
| CN103034672A (en) | Social search system and social search method | |
| CN112632405A (en) | Recommendation method, device, equipment and storage medium | |
| CN112231593B (en) | Financial information intelligent recommendation system | |
| CN112000889A (en) | Information gathering and presenting system | |
| CN116089723A (en) | Recommendation system recommendation method and device | |
| CN110795613A (en) | Commodity searching method, device and system and electronic equipment | |
| US20230334314A1 (en) | Content recommendation method and apparatus, device, storage medium, and program product | |
| CN116823410A (en) | Data processing method, object processing method, recommending method and computing device | |
| CN109299368B (en) | Method and system for intelligent and personalized recommendation of environmental information resources AI |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20230509 |
|
| WW01 | Invention patent application withdrawn after publication |