CN101977360B - Junk short message filter method - Google Patents
Junk short message filter method Download PDFInfo
- Publication number
- CN101977360B CN101977360B CN201010504001.1A CN201010504001A CN101977360B CN 101977360 B CN101977360 B CN 101977360B CN 201010504001 A CN201010504001 A CN 201010504001A CN 101977360 B CN101977360 B CN 101977360B
- Authority
- CN
- China
- Prior art keywords
- note
- words
- short message
- word
- deletion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a junk short message (SM) filter method, which comprises the following steps: step 10, deleting words which are not related with the SM texts in the SMs; step 20, calculating file fingerprints of the SMs in which the words not related with the SM texts are deleted; and step 30, if the number of the file fingerprints of the SMs in which the words not related with the SM texts are deleted exceeds a first preset threshold value, judging that the SMs are junk SMs. The method is a filter method based on the file fingerprints, which refers to not only similarity of the SMs in texts, but also mode of transmission of the junk SMs. The junk short message filter method of the invention has the advantages that identification and filter of the junk SMs are fast, thereby being suitable for occasions with higher requirements on instantaneity; and the filter method is not affected by change of nonessential texts in the junk SMs, thereby being capable of effectively dealing with continuous changes of the junk SMs.
Description
Technical field
The present invention relates to the text information processing technical field, particularly a kind of filter method of refuse messages.
Background technology
Note is one of frequent information interchange mode of using of people, and meanwhile, refuse messages also begins progressively to spread unchecked.Statistics shows, in the huge note of quantity, about 30% belongs to refuse messages.For domestic consumer, refuse messages seriously disturbs daily life; For operator, refuse messages occupies a large amount of flows space, reduces information transfer efficiency.
The main contents of refuse messages comprise advertising message, pornographic information, false prize-winning information, fraud information and mischief etc., and are wherein common with pornographic information and false prize-winning information especially.
In the prior art, modal filter method comprises:
1, based on the filtering junk short messages of sensitive word
By arranging responsive vocabulary at server end and coming filtrating rubbish short message by the method for pattern matching.Method based on sensitive word and string coupling is the most general method for filtering spam short messages, and this method is often as the pre-treatment step of other method.Great advantage based on sensitive word and the filter method of string coupling is to carry out efficient than higher, and scanning gets final product for one time to input message.Usually use in the on-line system.The greatest drawback of this method is:
(1) it is higher " to manslaughter rate ", no matter is to carry out participle or do not carry out participle, all can cause no small " manslaughtering " based on the filtering junk short messages that mates.
(2) can not tackle the variation of refuse messages flexibly.The sender of refuse messages tends to constantly change possible sensitive word form, in the hope of walking around the sensitive word strobe utility.For example: Zhu Rongji, Zhu's solvent, Zhu Rong base, Zhu Rongji etc.
The mode of enumerating by sensitive word can't all sensitive word forms of limit, and this method lags behind the propagation of refuse messages forever.
2, based on the filtering junk short messages of disaggregated model
Method for filtering spam short messages based on disaggregated model mainly is based on content relatively.By note is divided into independent parts, and the relation of calculating between these parts and the classification judges whether to be the short breath of rubbish.Common classification generally comprises two classes: normal note and refuse messages.Generally, note is divided into word, word or phrase.
Common disaggregated model comprises model-naive Bayesian, vector space model, maximum entropy model, supporting vector machine model etc.General disaggregated model can be used for judging whether being the short breath of rubbish.
By introducing disaggregated model, can avoid the defective that sensitive word filters to a great extent, can judge whether message is refuse messages from whole content.
Although can be from the degree of reliability of content calculating message based on disaggregated model, there be the defective of self in this method:
(1) needs make up the training corpus of certain scale.
The corpus that the disaggregated model of main flow often needs to make up certain scale obtain the classifying parameter of usefulness, generally speaking, the training corpus scale is more big, and classification accuracy is more high.In order to make up the training corpus of certain scale, need the great cost of cost, and training corpus must be brought in constant renewal in, otherwise be difficult to catch up with the paces that rubbish short message changes.
(2) filtering junk short messages efficient is lower, is not suitable for the higher occasion of real-time.
Be that based on the lower reason of the filter efficiency of disaggregated model computation complexity is higher.In above-mentioned all kinds of disaggregated model, except Bayesian model, the parameter scale of other model is all bigger, and computation complexity is than higher.On the exigent filtering junk short messages to efficient, based on the method for disaggregated model and be not suitable for.
3, based on the filtering junk short messages of behavior pattern
Different with content-based rubbish filtering thinking, stress to utilize note sending mode and receiving mode to carry out filtering short message based on the method for filtering spam short messages of community network.
Sending to toward carrying out between the stranger of refuse messages, namely transmit leg and recipient's past were set up contact at voice, did not almost have communications records each other.
Message often can not replied by domestic consumer after receiving refuse messages, i.e. the reception of refuse messages does not often have answer.
Can find most refuse messages by the transmission of refuse messages and the characteristics of receive mode.The defect map of this method is present:
(1) the community network scale is too big, and modeling is difficulty relatively, and storage and computation complexity are all bigger.
(2) be not stranger's note all be refuse messages, be not yet all short messages of not replying all be refuse messages.
4, the filtering junk short messages of describing based on client
Different with the filtering short message of server end, can be in advance at the client deployment filtering module.The short breath of rubbish filtering module at client deployment often utilizes user's address book and further feature to filter rubbish message.This method has not only increased the weight of the calculating pressure of client, and as easy as rolling off a log filtration stranger's message.
In sum, method for filtering spam short messages of the prior art, or only filter based on the content of refuse messages, or only filter based on the circulation way of refuse messages, can not identify effectively and filtrating rubbish short message.
Summary of the invention
(1) technical problem that will solve
The technical problem to be solved in the present invention is: how a kind of method for filtering spam short messages is provided, when refuse messages is filtered, not only considered the content of refuse messages but also considered the circulation way of refuse messages, thereby can identify real-time and efficiently and filtrating rubbish short message.
(2) technical scheme
For solving the problems of the technologies described above, the invention provides a kind of method for filtering spam short messages, this method comprises:
The words that has nothing to do with short message content in step 10, the deletion note;
The file fingerprint of the note behind the words that step 20, calculating deletion and short message content have nothing to do;
If the number of the file fingerprint of the note behind the words that step 30 deletion and short message content have nothing to do has surpassed first predetermined threshold value, judge that then this note is refuse messages;
Preferably, between described step 10 and the step 20, this method further comprises:
Step 11, note is carried out Chinese word segmenting;
The words that has nothing to do with short message content in the note behind step 12, the deletion participle; Comprise with the irrelevant words of short message content in the note: one or more in control character, outline symbol and the invisible character;
Preferably, described step 11 specifically comprises:
Take the strategy of rollback one word, utilize the maximum entropy model at word one's own department or unit to judge ambiguity then; If in the cutting process, run into surname, judge whether to be name or other unregistered word according to surname and context thereof.
Preferably, described step 12 specifically comprises: weight is lower than the words of second predetermined threshold value in the note behind the deletion participle.
Preferably, the weight of described words is the contrary document frequencies IDF of words; The computing formula of described IDF is:
Wherein, w is the words in the note; N is the note sum in predefined note storehouse; Df (w) is for comprising the note number of w in the predefined note storehouse.
Preferably, described step 12 specifically comprises: according to the words that has nothing to do with short message content in the note behind the part of speech deletion participle.
Preferably, described step 30 specifically comprises: safeguard the spatial cache of a fixed size as the comparison window in internal memory, all notes all appear in this comparison window; Divide the spatial cache of a fixed size as the note buffering area, preserve the file fingerprint of all notes that appear at the comparison window in the same time period; The file fingerprint that the file fingerprint of any note in the same time period and buffering area are preserved is compared, if when the number of the file fingerprint of this note surpasses first predetermined threshold value, judge that then this note is rubbish message.
Preferably, the method for calculating file fingerprint is the MD5 algorithm in the described step 20.
(3) beneficial effect
The present invention proposes a kind of method for filtering spam short messages.This method had both been considered similitude on the refuse messages content based on the filter method of file fingerprint, had also considered the circulation way of refuse messages, identified refuse messages based on these two principal characters.In the filtering junk short messages process, the spatial cache by safeguarding a fixed size is as the comparison window, and all short message contents all appear in this comparison window.By calculating a unique file fingerprint for every short message, can find rapidly whether to have refuse messages.The advantage of method for filtering spam short messages shown in the present is:
(1) speed of refuse messages identification and filtration is fast, can be used in the occasion that real-time is had higher requirements.
(2) can not be subjected to the influence of non-intrinsically safe content change in the refuse messages, can successfully manage the situation that rubbish message constantly changes.
Description of drawings
Fig. 1 is the flow chart of the method for filtering spam short messages of the embodiment of the invention;
Fig. 2 is the method flow diagram that in the method for the invention note is carried out Chinese word segmenting.
Embodiment
Below in conjunction with drawings and Examples, the specific embodiment of the present invention is described in further detail.Following examples are used for explanation the present invention, but are not used for limiting the scope of the invention.
Core concept of the present invention is: Chinese word segmenting is carried out in input or short messages stored, and deletion expresses irrelevant words with general short message content, the note of reservation is calculated its file fingerprint.If the number of times that the file fingerprint of this note occurs in buffer memory surpasses predetermined threshold value, can judge that then this note is refuse messages, otherwise be normal note.
Fig. 1 is the flow chart of the basic method for filtering spam short messages of the embodiment of the invention; As described in Figure 1, described method comprises:
The words that has nothing to do with short message content in steps A, the deletion note;
In this step, the irrelevant words of direct deletion and input or short messages stored content.The irrelevant words of described and input or short messages stored content comprises: control character (as carriage return character, newline, tab, full-shape half-width space etc.), outline symbol (as ●, ◎, ◇, ◆,
Etc.), invisible character (character of a lot of positions is invisible in the character code, perhaps not coding).No matter be control character, or outline symbol or invisible character, all there be not too big the contact with short message content generally speaking, can consider the meaning of these characters fully.
Step B, note is carried out Chinese word segmenting.
Chinese word segmenting refers to the word segmentation in the note is come out.The important indicator of Chinese word segmenting is accuracy rate and the speed of participle.In the environment of filtering short message, cutting speed is extremely important, and the Chinese word segmenting instrument that adopts in this step is based on maximum coupling and word one's own department or unit maximum entropy model, participle rate of accuracy reached to 95.4%, and cutting speed reaches per second 2MB, satisfies the requirement of real-time fully.Fig. 2 is the method flow diagram that in the method for the invention note is carried out Chinese word segmenting; Referring to Fig. 2, this process comprises following two steps:
In this step, only use the maximum matching strategy participle of forward.In order to alleviate the false segmentation in the maximum matching process, in participle, be aided with magnanimity participle resource, for example ambiguity storehouse, biographical dictionary, dictionary of place name and organization's name dictionary.Utilize the participle strategy of the auxiliary maximum coupling of magnanimity participle resource, both can improve the participle accuracy rate, can not reduce participle speed again.
Use ambiguity and the unregistered word recognition strategy of the maximum entropy at word one's own department or unit in this step, namely taked the strategy of rollback one word, utilized the slit mode of the maximum entropy model judgement ambiguity at word one's own department or unit then.This can remedy the deficiency in ambiguity storehouse to a certain extent.Simultaneously in the cutting process, run into surname, then judge whether name or other unregistered word according to surname and context thereof.
The words that has nothing to do with short message content in the note behind step C, the deletion participle.
In this step, the words that has nothing to do with short message content in the note behind the described deletion participle specifically comprises: weight is lower than the words of predetermined threshold value in the note behind the deletion participle.
The weight of described words can be according to IDF(inverse document frequency, contrary document frequencies) formula calculates.IDF is term weighing computational methods commonly used in information retrieval and the data mining, and its basis of calculating the words weight is: the note that comprises words is more many, and then its separating capacity is more limited, and namely the weight of words is more low.Use the weight of the IDF value representation words of words in this step, the computing formula of IDF is:
Wherein w is the words in the note; N: the note sum that refers to predefined note storehouse; Df (w): the note number that refers to comprise in the predefined note storehouse w.
IDF gives higher weight for words exclusive in the field, is fit to carry out text relevant and calculates, and for those words that all occurs, then give lower weight in all or most of text, and these words are lower to the importance of correlation calculations.
Utilize the IDF formula, about 300 everyday words have been extracted, the weight of these words in all texts is minimum, for example: I be once and do not have you in artificially will going up good to also this to all for many years the sky says with wait the earth understanding will be own with also go new we want from be good at two must again economical and down the back not but this newspaper be carried out at most by inferior true time to allow plant can by electricity more make important heart etc.
In this step, the words that has nothing to do with short message content in the note behind the described deletion participle specifically can also comprise: according to the words that has nothing to do with short message content in the note behind the part of speech deletion participle.
Different contents can be expressed in the words of different parts of speech, has different weights.Noun and verb be the emphasis of text representation often, and attribute and the state of noun and verb often only represented in adverbial word and adjective.These attributes and state do not influence the expression of text substance.For example:
1, Hong Se apple is on desk.Green apple is on desk.
2, I like you very much.I like you very much.
3, I Love You.I hate you.
In the 1st example, the content of front and back sentence is the same substantially, because the noun of front and back sentence " apple " is identical.In the 2nd sentence, the front and back sentence is also the same substantially, because the verb of front and back sentence " love " is identical.In the 3rd example, the substance of front and back sentence is very different, because the verb of front and back sentence " love " and " hatred " are inequality.
From automatic syntactic analysis, the word that can become sentence center composition often can influence the expression of substance, otherwise the word that can not become sentence center composition can't influence substance.From this judgement, this method is divided into groups to the weight of part of speech:
The word of following part of speech can influence substance:
Adnoun is called for short abbreviation term name morpheme
Group of noun name place name mechanism
Other proper name place word tense morpheme time words
The secondary verb name of moving morpheme verb verb
The word of following part of speech can not influence substance (mainly being function word):
Number punctuate modal particle interjection
Adjective adverbial word preposition measure word
Auxiliary word Chinese idiom idiom conjunction
Distinction word noun of locality pronoun descriptive word
The file fingerprint of the note behind the words that step D, calculating deletion and short message content have nothing to do.
Because the length of note differs, this is to storing and search and make troubles to note.In order to store and search note more easily, this method is given a unique encoding with every note, i.e. file fingerprint sign, and the length of file fingerprint sign is significantly smaller than text size, and length fixes, and is easy to storage and calculates.Every note behind the words that employing MD5 algorithm (Message Digest Algorithm5, Message Digest Algorithm 5) comes to have nothing to do to deletion and short message content in this step is calculated unique file fingerprint and is identified.
If the number of the file fingerprint of the note behind the words that step e deletion and short message content have nothing to do has surpassed predetermined threshold value, judge that then this note is refuse messages.
In the filtering junk short messages process, calculate a unique file fingerprint by giving every short message, can find rapidly whether to have same or analogous short message.Generally speaking, refuse messages is often mass-sended in same period, and identical or similar in terms of content.Can judge that namely these message are rubbish messages as long as in a period of time, find many identical fingerprints.
Because the scale of note is very big, therefore stores all notes and carry out the scheme of note comparison and infeasible.In view of this characteristics that refuse messages sends, this step is safeguarded the spatial cache of a fixed size as the comparison window in internal memory, and all short message contents all can appear in this comparison window; Divide the spatial cache of a fixed size as the note buffering area, preserve the file fingerprint of all notes that appear at the comparison window in the same time period, all inputs or short messages stored are after having calculated file fingerprint, all the existing file fingerprint of file fingerprint and buffering area is compared, if when having found that number surpasses predetermined threshold value, then this message is judged as rubbish message.
In the file fingerprint buffering area, the speed of file fingerprint comparison is very big to the influence of system.In view of this, this step adopts tree structure to represent the file fingerprint buffering area of note.A given new file fingerprint is at most as long as can determine whether for one time to occur in buffering area to this document finger scan.
Above execution mode only is used for explanation the present invention; and be not limitation of the present invention; the those of ordinary skill in relevant technologies field; under the situation that does not break away from the spirit and scope of the present invention; can also make a variety of changes and modification; therefore all technical schemes that are equal to also belong to category of the present invention, and scope of patent protection of the present invention should be defined by the claims.
Claims (6)
1. a method for filtering spam short messages is characterized in that, this method comprises:
The words that has nothing to do with short message content in step 10, the deletion note;
The file fingerprint of the note behind the words that step 20, calculating deletion and short message content have nothing to do;
If the number of the file fingerprint of the note behind the words that step 30 deletion and short message content have nothing to do has surpassed first predetermined threshold value, judge that then this note is refuse messages;
Between described step 10 and the step 20, this method further comprises:
Step 11, note is carried out Chinese word segmenting;
The words that has nothing to do with short message content in the note behind step 12, the deletion participle; Comprise with the irrelevant words of short message content in the note: one or more in control character, outline symbol and the invisible character;
Described step 11 specifically comprises:
Step 101, note is carried out maximum coupling rough lumber branch, if run into ambiguity, then carry out step 102;
Step 102, note is carried out the identification of ambiguity and unregistered word;
Step 102 specifically comprises:
Take the strategy of rollback one word, utilize the maximum entropy model at word one's own department or unit to judge ambiguity then; If in the cutting process, run into surname, judge whether to be name or other unregistered word according to surname and context thereof.
2. the method for claim 1 is characterized in that, described step 12 specifically comprises: weight is lower than the words of second predetermined threshold value in the note behind the deletion participle.
3. method as claimed in claim 2 is characterized in that, the weight of described words is the contrary document frequencies IDF of words; The computing formula of described IDF is:
Wherein, w is the words in the note; N is the note sum in predefined note storehouse; Df (w) is for comprising the note number of w in the predefined note storehouse.
4. the method for claim 1 is characterized in that, described step 12 specifically comprises: according to the words that has nothing to do with short message content in the note behind the part of speech deletion participle.
5. the method for claim 1 is characterized in that, described step 30 specifically comprises: safeguard the spatial cache of a fixed size as the comparison window in internal memory, all notes all appear in this comparison window; Divide the spatial cache of a fixed size as the note buffering area, preserve the file fingerprint of all notes that appear at the comparison window in the same time period; The file fingerprint that the file fingerprint of any note in the same time period and buffering area are preserved is compared, if when the number of the file fingerprint of this note surpasses first predetermined threshold value, judge that then this note is rubbish message.
6. as each described method among the claim 1-5, it is characterized in that the method for calculating file fingerprint in the described step 20 is the MD5 algorithm.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010504001.1A CN101977360B (en) | 2010-09-30 | 2010-09-30 | Junk short message filter method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201010504001.1A CN101977360B (en) | 2010-09-30 | 2010-09-30 | Junk short message filter method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101977360A CN101977360A (en) | 2011-02-16 |
| CN101977360B true CN101977360B (en) | 2013-07-03 |
Family
ID=43577198
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201010504001.1A Active CN101977360B (en) | 2010-09-30 | 2010-09-30 | Junk short message filter method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101977360B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102905236B (en) * | 2011-07-27 | 2016-08-17 | 华为技术有限公司 | A kind of junk short message monitoring method, Apparatus and system |
| CN103812826A (en) * | 2012-11-08 | 2014-05-21 | 中国电信股份有限公司 | Identification method, identification system, and filter system of spam mail |
| CN103646029B (en) * | 2013-11-04 | 2017-03-15 | 北京中搜网络技术股份有限公司 | A kind of similarity calculating method for blog article |
| CN104615585B (en) * | 2014-01-06 | 2017-07-21 | 腾讯科技（深圳）有限公司 | Handle the method and device of text message |
| CN104794125B (en) * | 2014-01-20 | 2018-09-11 | 中国科学院深圳先进技术研究院 | A kind of recognition methods of refuse messages and device |
| CN105786792A (en) * | 2014-12-26 | 2016-07-20 | ***通信集团公司 | Information processing method and device |
| CN109547319A (en) * | 2017-09-22 | 2019-03-29 | 中移（杭州）信息技术有限公司 | A kind of message treatment method and device |
| CN108230037B (en) * | 2018-01-12 | 2022-10-11 | 北京字节跳动网络技术有限公司 | Advertisement library establishing method, advertisement data identification method and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1520214A (en) * | 2003-09-02 | 2004-08-11 | �ź㴫 | Firewall system for short message and method for building up firewall |
| CN101137087A (en) * | 2007-08-01 | 2008-03-05 | 浙江大学 | Short message monitoring center and monitoring method |
-
2010
- 2010-09-30 CN CN201010504001.1A patent/CN101977360B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1520214A (en) * | 2003-09-02 | 2004-08-11 | �ź㴫 | Firewall system for short message and method for building up firewall |
| CN101137087A (en) * | 2007-08-01 | 2008-03-05 | 浙江大学 | Short message monitoring center and monitoring method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN101977360A (en) | 2011-02-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101977360B (en) | Junk short message filter method | |
| US7461056B2 (en) | Text mining apparatus and associated methods | |
| CN102096703B (en) | Filtering method and equipment of short messages | |
| KR101716905B1 (en) | Method for calculating entity similarities | |
| Faguo et al. | Research on short text classification algorithm based on statistics and rules | |
| CN110019792A (en) | File classification method and device and sorter model training method | |
| Rădulescu et al. | Identification of spam comments using natural language processing techniques | |
| CN103116637A (en) | Text sentiment classification method facing Chinese Web comments | |
| CN110458296B (en) | Method and device for marking target event, storage medium and electronic device | |
| CN102184256A (en) | Clustering method and system aiming at massive similar short texts | |
| US20030167245A1 (en) | Summary evaluation apparatus and method, and computer-readable recording medium in which summary evaluation program is recorded | |
| CN104967558A (en) | Method and device for detecting junk mail | |
| CN109086355A (en) | Hot spot association relationship analysis method and system based on theme of news word | |
| CN115186654B (en) | Method for generating document abstract | |
| CN111310467B (en) | Topic extraction method and system combining semantic inference in long text | |
| CN108875050B (en) | Text-oriented digital evidence-obtaining analysis method and device and computer readable medium | |
| CN114202443A (en) | Policy classification method, device, equipment and storage medium | |
| CN113191145A (en) | Keyword processing method and device, electronic equipment and medium | |
| Pak et al. | The impact of text representation and preprocessing on author identification | |
| Giri et al. | A survey of automatic text summarization system for different regional language in India | |
| Özel et al. | Effects of feature extraction and classification methods on cyberbully detection | |
| CN108427769B (en) | Character interest tag extraction method based on social network | |
| CN102799666B (en) | Method for automatically categorizing texts of network news based on frequent term set | |
| Li-Juan et al. | A classification method of Vietnamese news events based on maximum entropy model | |
| CN112529627B (en) | Method and device for extracting implicit attribute of commodity, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CP02 | Change in the address of a patent holder | ||
| CP02 | Change in the address of a patent holder |
Address after: Room 810, 8 / F, 34 Dian street, Haidian District, Beijing 100080 Patentee after: BEIJING D-MEDIA COMMUNICATION TECHNOLOGY Co.,Ltd. Address before: 100089 Beijing city Haidian District wanquanzhuang Road No. 28 Wanliu new building A block 5 layer Patentee before: BEIJING D-MEDIA COMMUNICATION TECHNOLOGY Co.,Ltd. |