Summary of the invention
The invention provides a kind of method and device thereof that the content of text audit of user's issue is handled, it can save a large amount of manual examination and verification time, has improved review efficiency.
Technical scheme of the present invention is: a kind of method that the content of text audit of user's issue is handled comprises step:
Receive the content of text of user's issue, according to list rule database judges information, said list rule database comprises blacklist, black rule, white list and white rule;
If said user profile neither belongs to white list or white rule, do not belong to blacklist or black rule yet, then the content of text to said user carries out format conversion, extracts the notional word in the said content of text;
Calculate the contrary document frequency weighted value of each notional word in the document database of setting up in advance that extracts, obtain first proper vector of forming by said contrary document frequency weighted value;
First similarity of second proper vector of the spam samples content of calculating said first proper vector and setting up in advance; Whether the content of text of judging said user's issue according to said first similarity is qualified content; If qualified content is then announced the content of text of said user's issue.
The invention also discloses a kind of device that the content of text audit of user's issue is handled; It comprises; Auditing module; Be used to receive the content of text of user's issue, according to list rule database judges information, said list rule database comprises blacklist, black rule, white list and white rule;
Modular converter is used for neither belonging to white list or white rule in said user profile, and when also not belonging to blacklist or black rule, the content of text that said user is issued carries out format conversion, extracts the notional word in the said content of text;
Computing module is used for calculating the contrary document frequency weighted value of each notional word of extraction at the document database of setting up in advance, obtains first proper vector of being made up of said contrary document frequency weighted value; First similarity of second proper vector of the spam samples content of calculating said first proper vector simultaneously and setting up in advance;
Judge module is used for judging according to said first similarity whether said user's content of text is qualified content, if qualified content is then announced the content of text of said user's issue.
The method and apparatus that the content of text audit of user's issue is handled of the present invention; Only to neither belonging to white list or white rule; The content of text that does not also belong to user's issue of blacklist or black rule is examined filtration treatment; Can and the underproof content of text of user's issue be sent to manual work with the content of text of the user's issue that belongs to black rule and blacklist and examine, the qualified content of text that content of text that the user who belongs to white rule and white list is issued and user issue is directly announced; Need not all examine user's information releasing like this, can save a large amount of manual examination and verification time, save human resources, also improve review efficiency accordingly via manual work.
Embodiment
The method and apparatus that the content of text audit of user's issue is handled of the present invention; Only to neither belonging to white list or white rule; The content of text that does not also belong to user's issue of blacklist or black rule is examined filtration treatment; To belong to black rule and blacklist user's issue content of text and the underproof content of text of user's issue sent to manual work examine, the qualified content of text that content of text that the user who belongs to white rule and white list is issued and user issue is directly announced; Need not all examine user's information releasing like this, can save a large amount of manual examination and verification time, save human resources, also improve review efficiency accordingly via manual work.
Below in conjunction with accompanying drawing and specific embodiment the present invention is done a detailed elaboration.
The method that the content of text audit of user issue is handled of the present invention can be applied in to be asked in the question and answer type services such as community, Baidu are known, Sina likes to ask.
The method that the content of text audit of user's issue is handled of the present invention comprises step, like Fig. 1,
The content of text of S100, reception user issue.S101, according to list rule database judges information; Said list rule database comprises blacklist, black rule, white list and white rule.In one embodiment, blacklist can be to have big probability that the user list of junk information is provided, and white list is to have big probability that the user list of proper information is provided; Black rule is to set according to user's grade or credit rating, and its expression user's lower grade is or credit rating is very low, and white rule also is to set according to user's grade or credit rating, and its grade of representing the user is than higher or credit rating is very high.
If the said user profile of S102 neither belongs to white list or white rule, do not belong to blacklist or black rule yet, then the content of text to said user's issue carries out format conversion, extracts the notional word in the said content of text.In one embodiment, format conversion can comprise that said content of text is carried out the traditional font to be changed, remove the conversion in unnecessary space etc. to half-angle to simplified conversion, full-shape, and notional word is the core word of content of text, and function word is not as core word.
Contrary document frequency (IDF) weighted value of each notional word in the document database of setting up in advance that S103, calculating are extracted obtains first proper vector of being made up of said contrary document frequency (IDF) weighted value.In one embodiment, the document database can be made up of the content of text of all user's issues.Contrary document frequency (IDF) weighted value of each notional word that calculate to extract in the document database of setting up in advance, specifically can for: according to formula
Calculate contrary document frequency (IDF) weighted value of each notional word; Wherein wgt is contrary document frequency (IDF) weighted value, t
fBe the frequency values that said notional word occurs in said user's content of text, U is the total number of documents in the said document database, and V is for the number of files of said notional word occurring.
S104, calculate said first proper vector and first similarity of second proper vector of the spam samples content set up in advance.Second proper vector of spam samples content can obtain in advance; It is the same with first proper vector that it obtains process; Take out a spam samples content,, extract notional word its format conversion; Calculate the contrary document frequency weighted value of each notional word in said document database then, form second proper vector by these weighted values.In one embodiment, calculate said first proper vector and first similarity of second proper vector of the spam samples content set up in advance, be specially: according to formula
Cos(X,Y)
Calculate said first similarity; Wherein represent said first similarity,
X={x
1,K,x
m},Y={y
1,K,y
n}
Represent said first proper vector and second proper vector respectively.
S105, judge according to said first similarity whether the content of text of said user issue is qualified content.This determination methods has a variety of modes, can set according to user's needs.In one embodiment, can set a predetermined threshold,, otherwise judge that the content of text of this user's issue is qualified content if the value of said first similarity, can judge then that the content of text of this user's issue is defective content greater than this threshold value.
If qualified content, then carry out the content of text that step S107 announces said user's issue, the content of text of said user's issue is sent to manual work examine otherwise can carry out step S106 in one embodiment.
In one embodiment, belong to blacklist or black rule, the content of text of said user's issue is sent to manual work examine if can also comprise step S102 user profile after the step S101.If the said user profile of S103 belongs to white list or white rule, with the content of text of announcing said user's issue.
In order to judge comprehensively accurately that further whether the content that the user issues is qualified content, reduces the probability of erroneous judgement.In one embodiment; Neither belong to white list or white rule in judges information; When not belonging to blacklist or black rule again; Can also comprise step, detect second similarity of content of text with the feature database of setting up in advance that comprises phone number format, webpage format and Mars word form etc. of said user issue, judge according to this second similarity and first similarity whether the content of text that said user issues is qualified content.When whether the content of text of judges issue is qualified content; Can distribute weights respectively for first similarity and second similarity; Whether detect the weights sum greater than a predetermined value; If greater than a predetermined value, can judge that the content of text of this user's issue is defective content, otherwise be qualified content.Whether the value that also can only detect this second similarity in addition greater than a predetermined value, if greater than could judge directly that the content of text of this user's issue is defective content.
In order to reach same purpose, judge comprehensively accurately further whether the content of user's issue is qualified content, reduce the probability of erroneous judgement.In one embodiment; Neither belong to white list or white rule in judges information; When not belonging to blacklist or black rule again; Can also comprise step, add up the number of characters of the content of text of said user's issue, judge according to this number of characters, first similarity and second similarity whether the content of text of said user's issue is qualified content.When whether the content of text of judges issue is qualified content; Can distribute weights respectively for number of characters, first similarity and second similarity; Whether detect the weights sum greater than a predetermined value; If greater than a predetermined value, can judge that the content of text of this user's issue is defective content, otherwise be qualified content.Also can set a predetermined value with regard to this number of characters separately in addition, if when detecting number of characters less than this predetermined value, content of text that directly can the judges issue is defective content.
In order to reach same purpose, judge comprehensively accurately further whether the content of user's issue is qualified content, reduce the probability of erroneous judgement.In one embodiment; Neither belong to white list or white rule in judges information; When not belonging to blacklist or black rule again; Can also comprise step; The third phase that detects the content of text of said user's issue and the data bank of setting up in advance that can not announce words (this data bank is to some special words and short sentence or the set of interior perhaps other settings of requirement shielding at no distant date) judges like degree, said number of characters, first similarity and second similarity whether the content of text that said user issues is qualified content according to this third phase like degree.When whether the content of text of judges issue is qualified content; Can distribute weights respectively like degree, number of characters, first similarity and second similarity for third phase; Whether detect the weights sum greater than predetermined value; If greater than a predetermined value, can judge that the content of text of this user's issue is defective content, otherwise be qualified content.Also can detect this third phase in addition separately and seemingly whether spend greater than a predetermined value, if greater than, can judge that then the content of text of this user's issue is defective content.
The present invention has also disclosed a kind of device that the content of text audit of user's issue is handled, and like Fig. 2, it comprises auditing module, modular converter, computing module and the judge module that connects successively;
Auditing module is used to receive the content of text that the user issues, and according to list rule database judges information, said list rule database comprises blacklist, black rule, white list and white rule.In one embodiment, blacklist can be to have big probability that the user list of junk information is provided, and white list is to have big probability that the user list of proper information is provided; Black rule is to set according to user's grade or credit rating, and its expression user's lower grade is or credit rating is very low, and white rule also is to set according to user's grade or credit rating, and its grade of representing the user is than higher or credit rating is very high.
Modular converter is used for neither belonging to white list or white rule in said user profile, and when also not belonging to blacklist or black rule, the content of text that said user is issued carries out format conversion, extracts the notional word in the said content of text.In one embodiment, format conversion can comprise that said content of text is carried out body to be changed, remove the conversion in unnecessary space etc. to half-angle to simplified conversion, full-shape, and notional word is the core word of content of text, and function word is not as core word.
Computing module is used for calculating contrary document frequency (IDF) weighted value of each notional word of extraction at the document database of setting up in advance, obtains first proper vector of being made up of said contrary document frequency (IDF) weighted value; First similarity of second proper vector of the spam samples content of calculating said first proper vector simultaneously and setting up in advance.In one embodiment, the document database can be made up of the content of text of all user's issues.Contrary document frequency (IDF) weighted value of each notional word that calculate to extract in the document database of setting up in advance, specifically can for: according to formula
Calculate contrary document frequency (IDF) weighted value of each notional word; Wherein wgt is contrary document frequency (IDF) weighted value, t
fBe the frequency values that said notional word occurs in said user's content of text, U is the total number of documents in the said document database, and V is for the number of files of said notional word occurring.Second proper vector of spam samples content can obtain in advance; It is the same with first proper vector that it obtains process; Take out a spam samples content,, extract notional word its format conversion; Calculate the contrary document frequency weighted value of each notional word in said document database then, form second proper vector by these weighted values.In one embodiment, calculate said first proper vector and first similarity of second proper vector of the spam samples content set up in advance, be specially: according to formula
Cos(X,Y)
Calculate said first similarity; Wherein represent said first similarity,
X={x
1,K,x
m},Y={y
1,K,y
n}
Represent said first proper vector and second proper vector respectively.
Judge module is used for judging according to said first similarity whether said user's content of text is qualified content, if qualified content is then announced the content of text of said user's issue.In one embodiment, be defective content if judge said user's content of text, the content of text that then said judge module is issued said user sends to manual work and examines.
In one embodiment, said auditing module belongs to blacklist or black rule in user profile, the content of text of said user's issue is sent to manual work examine; Belong to white list or white rule in said user profile, with the content of text of announcing said user's issue.
In order to judge comprehensively accurately that further whether the content that the user issues is qualified content, reduces the probability of erroneous judgement.Like Fig. 3; Between said auditing module and said judge module, also be connected with detection module; Be used for neither belonging to white list or white rule in user profile; When not belonging to blacklist or black rule again, detect the content of text of said user's issue and second similarity of the feature database that comprises phone number format, webpage format and Mars word form of foundation in advance; And/or detect said user's content of text and the third phase of the data bank that can not announce words set up in advance like degree; And said second similarity and/or third phase sent to said judge module like degree, said judge module judges like degree whether the content of text that said user issues is qualified content according to said first similarity, second similarity and/or third phase.When whether the content of text of judges issue is qualified content; Can distribute weights respectively like degree for first similarity, second similarity and/or third phase; Whether detect the weights sum greater than predetermined value; If greater than a predetermined value, can judge that the content of text of this user's issue is defective content, otherwise be qualified content.
In order to reach identical purpose, judge comprehensively accurately further whether the content of user's issue is qualified content, reduce the probability of erroneous judgement.Like Fig. 4; Between said auditing module and said judge module, also be connected with statistical module; Be used for neither belonging to white list or white rule, when not belonging to blacklist or black rule again, add up the number of characters of the content of text of said user's issue in user profile; And this number of characters sent to said judge module, said judge module judges like degree whether the content of text that said user issues is qualified content according to this number of characters, said first similarity, second similarity and/or third phase.When whether the content of text of judges issue is qualified content; Can distribute weights respectively like degree for number of characters, first similarity, second similarity and/or third phase; Whether detect the weights sum greater than predetermined value; If greater than a predetermined value, can judge that the content of text of this user's issue is defective content, otherwise be qualified content.
In sum; The method and apparatus that the content of text audit of user's issue is handled of the present invention; Can examine filtration treatment to the content of text of user profile and user's issue; To belong to black rule and blacklist user's issue content of text and the underproof content of text of user's issue sent to manual work examine, the qualified content of text that content of text that the user who belongs to white rule and white list is issued and user issue is directly announced; Need not all examine user's information releasing like this, can save a large amount of manual examination and verification time, save human resources, also improve review efficiency accordingly via manual work.
Above-described embodiment of the present invention does not constitute the qualification to protection domain of the present invention.Any modification of within spirit of the present invention and principle, being done, be equal to replacement and improvement etc., all should be included within the claim protection domain of the present invention.