CN102760151B - Implementation method of open source software acquisition and searching system - Google Patents
Implementation method of open source software acquisition and searching system Download PDFInfo
- Publication number
- CN102760151B CN102760151B CN201210097169.4A CN201210097169A CN102760151B CN 102760151 B CN102760151 B CN 102760151B CN 201210097169 A CN201210097169 A CN 201210097169A CN 102760151 B CN102760151 B CN 102760151B
- Authority
- CN
- China
- Prior art keywords
- open source
- community
- project
- attribute
- projects
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
An implementation method of an open source software acquisition and searching system comprises the steps as follows: relational database management software is configured on an enterprise server, so that business-layer codes can access and refresh information in database in the form of objects when an upper application program calls a relational database; an input definition of a user is received, a certain number of open source community sets with known URL (Uniform Resource Locator) address are determined, and open source software home pages in open source communities are captured by crawlers; information, namely open source attributes in the open source software field are extracted from the captured home pages; cross community project information is integrated, an attribute union set is adopted when projects originating from different communities have different attributes, and the dependency relationships among the different projects are saved at the same time; an inverted index is built for the obtained open source attributes; in order to response the user search input or a predefined ranking mechanism, the open source projects that need to be shown are ordered and displayed, so that the acquisition and the searching of the open source software projects can be achieved.
Description
Technical field
The present invention relates to a kind of open source software and obtain the implementation method with search system, particularly under the environment across multiple community that increases income, a kind of building method of the open source software search system based on oriented network reptile, information extraction.
Background technology
In recent years, the motion of domestic and international open source software achieves flourish, has been formed numerously to have the software development of a large amount of developer, open source projects and the community that increases income and the open source software ecosystem of application.Current open source software industry has following development trend:
1. open source projects and the also steady growth year by year of developer's substantial amounts;
2. the open source projects that minority is outstanding develops rapidly and is used widely;
3. most open source projects is in lower development level and is in slow state of development;
These features make to find from magnanimity open source software project and select high-quality and the open source software meeting particular demands becomes urgent need deals with problems.Traditional search engines lacks specific aim due to it, and in Search Results, effective information content is not high, is therefore difficult to tackle such demand.Open source software community is as the hosted platform of open source software, and its page has abundant open source projects descriptor, for open source software share, evaluate and selection there is important value.The community that increases income mostly is integrated with and selects service for the search engine of open source software in community and condition, but will select in the world and search open source software, and comprehensively across the software information of community, the open source software search engine built across community is imperative.
In recent years, a large amount of research and practices has been carried out for the open source software page data collection across community both at home and abroad.The FLOSSmole project of Syracuse university of the U.S. has made outstanding achievement in this field.By using 275 web crawlers, this project has crawled nearly 500 from 20 Duo Ge communities, the page data of 000 open source projects.But because the collection of data has only been carried out in this work, externally only provide the download of sq1 query interface and sq1 packet, and do not provide data comprehensive, represent, the function such as search, people therefore can not be helped to search for and obtain software.
The research of the domestic search engine towards open source software is also in the starting stage, wherein more influentially comprises the Chinese community that increases income, CSDN increases income community, Chinese Linux community etc.The open source software information spinner of increasing income in Chinese community will from user human-edited, number and have obvious limitation qualitatively; Open source resources in CSDN community mainly user shares the kit or code library uploaded voluntarily; Source mixes, form disunity, describes incomplete; And the open source software primary limitation of Linux community is at the relevant bag of linux system and instrument.Generally speaking, these communities current lack one and contain wide, and data are comprehensive, the open source software information bank of uniform format, and are the search engine technique that user inquires about and selection open source software provides recommendation and helps based on a kind of of this information bank.
Summary of the invention
The problem to be solved in the present invention is: for the deficiency of prior art in representing across community information acquisition and informix, a kind of implementation method of open source software search system is proposed, make it possible to the open source software information extensively collecting multiple community, for prospective users provides software lookup and recommendation service.Technical scheme of the present invention comprises the following steps:
Step 1, build data, services.Configuration relation database management language on enterprise-level server, builds relation and object map framework, and operation layer code can be enable when upper level applications is called relational database to access in the mode of object and upgrade the information of lane database.
Step 2, webpage obtain.Receive the input definition of user, determine community's set of increasing income of the known URL address of some, the open source software homepage of being increased income by crawler capturing in community, is divided into open source projects list page to crawl crawling two stages with open source projects homepage by crawling of open source projects.
Step 3, attribute extraction of increasing income.Namely the information extracting open source software field from the page crawled increase income attribute, as project programming language, certificate, running environment, development status, hour of log-on, up-to-date issuing time, download time, homepage link etc.
Step 4, to integrate across community projects information, when the project from different community has different attributes, get attribute union; Preserve the dependence between disparity items simultaneously;
Step 5, set up inverted index to the attribute of increasing income obtained, wherein each all comprises a property value and has the address of each record of this property value, is determined the position of recording, but not determine property value by recording by property value.
Step 6, in response to user search input or predefined rank mechanism, sort to needing the open source projects that represents and show.
Further, step 2 can realize in the following manner:
Step 21, to mate according to the project home page URL template of community, extract project home page URL and be stored in database, situation is formed to community's open source projects homepage of constantly change and carries out snapshot.
Step 22, be crawl open source projects homepage.Spiders constantly takes out project home page URL from database to carry out orientation and crawls, and the complete html page text crawled is saved in database.
More specifically, step 3 is carried out through the following steps.
Step 31, template detect.Find that each community that increases income is for showing the web page template of attribute of increasing income, can be the tag path in HTML tree, also can be the regular expression in html text, as the rule extracting attribute of increasing income.There are two kinds of obtain manners specifically.
Step 311, method based on manual observation.By observing a small amount of webpage sample from the community that increases income, find page location or its regular expression pattern occurred at each attribute place of increasing income.
Step 312, method based on seed set.What obtain attribute of increasing income can Column Properties value, use similarity of character string matching algorithm, automatically in collections of web pages, find to increase income property value place tag path, adopt max support method (namely have in certain position the highest matching times then think have max support at this) to filter out unique web page template.Wherein, the similarity mode of described character string can by Jakarta (Jaccard) coefficient of the corresponding q-gram set of two character strings as two string matching number of degrees values, if when this numerical value is higher than certain threshold value predefined, think coupling.
Step 32, tag location.By the Template Location that the obtained html tag to property value of increasing income.
If the masterplate determined is by manual observation and the regular expression determined, then carry out the localization method of step 321 based on regular expression.Use regular expression locating rule in target web, find property value label node; If the masterplate determined is the tag path determined by seed collection approach, then carry out the localization method of step 322 based on tag path, according to DOM Document Object Model (DOM) tree structure of the html page, each label position is expressed as the tag path from page root node (<html>) to property value place node, by supporting that the web analysis instrument (as DOM4J) in this path carries out tag location and acquisition, thus obtain text label corresponding to property value.
Further, in described step 6, if receive the search input of user, then by calculating the string match condition between the key word of user's input and inverted index data acquisition, the degree of correlation calculating the result of software project in key word and data acquisition sorts; If do not receive user search input, then according to the project rank mechanism of increase income degree of participation and the definition of information complete degree of project, project is carried out sorting and is shown.
Further, community's number of degree of participation trustship used for a project of increasing income of described project, other open source projects numbers associated by it quantize; Described information complete degree is quantized by the attribute number of increasing income of project.
The corresponding User Interface of final exploitation, is responsible for receiving from user's input, is responsible for receiving the ranking results from search engine simultaneously, organizes and show the software project of increasing income in community.
Accompanying drawing explanation
Fig. 1 is the process flow diagram of the present invention across community's open source software search system implementation method;
Fig. 2 is the process flow diagram obtaining different communities open source software project according to an embodiment of the invention.
Fig. 3 is according to an embodiment of the invention to the process flow diagram that the open source software project after acquisition manages and organizes.
Embodiment
As shown in Figure 1, for open source software obtains the process flow diagram realized with search system and method, specifically following steps are implemented:
Step 1, build data, services.Open source software information bank has data volume large (> 100GB), and upgrade frequent, concurrent query processing ability is strong, such as, provide the requirement of the aspect such as access services towards mass users on the internet.Therefore need to adopt the efficient database management language of enterprise-level to build data, services, in the present invention data, services is deployed on enterprise-level blade server, there are 8 core CPU, internal memory 32GB, hard disk 2TB.Relational database or object-oriented database can be used, when using relational database, also need the relation of building and object map framework, to facilitate upper level applications to call relational database, enable operation layer code access in the mode of object and upgrade the information of lane database.
Step 2, webpage obtain.The open source software homepage of being increased income by crawler capturing in community, because target pages URL set is known, therefore can modify to the mode that crawls of traditional common reptile, remove wherein as URL validity judgement, inapplicable conventional components such as new URL discovery, and use by manually prior to each community's predefine of increasing income, and then determine the generation method of the URL needing to crawl.Based on the ever-increasing feature of project in the community that increases income, crawling of open source projects is strictly divided into open source projects list page and crawls and crawl two stages with open source projects homepage.
The target of step 21, first stage analyzes all items list page in community, for same community, its bulleted list page is have certain rule and relatively-stationary in name, such as sourceforge community, the form of its bulleted list page is www.sourceforge.net/search/ ~ 1.html, and what may change may be only the numeral number of html text.Project home page URL template according to community is mated, and extracts project home page URL and is stored in database, plays and forms to community's open source projects homepage of constantly change the effect that situation carries out snapshot.
The target of step 22, subordinate phase crawls open source projects homepage.Spiders constantly takes out project home page URL from database to carry out orientation and crawls, and the complete html page text crawled is saved in database.Adopt multithreading to crawl technology when crawling, crawl efficiency to improve.
Step 3, attribute extraction of increasing income.Namely the information extracting open source software field from the page crawled increase income attribute, as project programming language, certificate, interface, running environment, localization, database, development status, hour of log-on, up-to-date issuing time, download time, homepage link etc.
More specifically, carry out through the following steps.
Step 31, template detect.Find that each community that increases income is for showing the web page template of attribute of increasing income, can be the tag path in HTML tree, also can be the regular expression in html text, as the rule extracting attribute of increasing income.There are two kinds of obtain manners specifically.
Step 311, method based on manual observation.By observing a small amount of webpage sample from the community that increases income, find page location or its regular expression pattern occurred at each attribute place of increasing income.The method is comparatively simple and easy, but reliability is not high, is applicable to structure of web page simple, community's template detection that template is distinct.
Step 312, method based on seed set.What obtain attribute of increasing income can Column Properties value, use similarity of character string matching algorithm, automatically in collections of web pages, find to increase income property value place tag path, adopt max support method (namely have in certain position the highest matching times then think have max support at this) to filter out unique web page template.The similarity mode of character string can by Jakarta (Jaccard) coefficient of the corresponding q-gram set of two character strings as two string matching number of degrees values, if when this numerical value is higher than certain threshold value predefined, think coupling.
Such as, if seed attribute-value is " GPL License ", the party's rule be responsible for find " GPLLicense " in target community institute occur and each appearance tag path, select the path that occurrence number is maximum, as "/html/body/table/tr/td/text () ", as decimation rule and the extraction position of this attribute.
Step 32, tag location.By the Template Location that the obtained html tag to property value of increasing income.
If the masterplate determined is by manual observation and the regular expression determined, then carry out the localization method of step 321 based on regular expression.Use regular expression locating rule in target web, find property value label node.If any html fragment " <li>Li cens e:<a hr ef="/categories/license/gpl " >GPL License</a> ", " License " property value regular expression locating rule of its correspondence may be " <Li>License:<a href="/categories/license/ [a-z | A-Z | 0-9]+" > [a-z | A-Z | 0-9]+</a> ", and this expression formula will navigate to all " License " attribute place labels.
If the masterplate determined is the tag path determined by seed collection approach, then carry out the localization method of step 322 based on tag path.According to DOM Document Object Model (DOM) tree structure of the html page, each label position is expressed as the tag path from page root node (<html>) to property value place node, by supporting that the web analysis instrument (as DOM4J) in this path carries out tag location and acquisition, thus obtain text label corresponding to property value.
Step 4, informix and excavation.
Step 401, across community projects information integration.Namely integration process is carried out to extracting the information obtained.
In embodiment, if what open source projects xwiki extracted in community sourceforge.net themes as " Database; Dynamic Content; Site Management ", in ow2.org, it themes as " Dynamic Content ", the theme set of project xwiki across this Liang Ge community is obtained for (" Database, Dynamic Content, Site Management ") by getting union.
Step 402, excavation Project cooperation relation.Find that item description mentions the situation of sundry item across community, find the conspiracy relation between each open source projects.In an embodiment, if project xwiki mantis has statement " Xwiki Mantis plugin integrates mantis into xwiki " in it describes, by preserving the dependence of xwiki mantis to xwiki, present in user search result.
Step 5, set up inverted index.Set up inverted index to the attribute of increasing income that step 3 obtains, each in this concordance list all comprises a property value and has the address of each record of this property value.Determined the position of recording by property value, instead of determine property value by recording.Concerning project xwiki, it is at sourceforge.net, ow2.org, the attribute that freshmeat.net obtains has (project-real-namein sourceforge, " xwiki "), (user-interface insourceforge, " web-based "), (license in ow2, " gpl "), (programming-language-in-freshmeat, " java "), (description-in-sourceforge, " XWiki is the next generationenterprise wiki ... ").At this, inverted index is set up to these attributes.As search system user input query keyword " enterprise wiki ", system returns by certain sort algorithm in order by according to the project information containing this keyword in index.In the present embodiment xwiki descriptor index in containing " enterprise wiki " keyword, therefore xwiki one of as a result will be returned to user.
Step 6, realization sequence.
Step 601, search result ordering method based on keyword match.The method is namely by calculating the string match condition between the key word of user's input and inverted index data acquisition, and the degree of correlation calculating the result of software project in key word and data acquisition sorts.
Step 602, project rank algorithm without key word.Increase income degree of participation and the information complete degree definition project ranking machine system of a project can be considered, the relevant information different according to project distributes corresponding weight, weighted calculation score, for when user does not input key word, carries out sorting to project and shows.Wherein, community's number of degree of participation trustship used for a project of increasing income of project, other open source projects numbers associated by it quantize; And information complete degree is quantized by the attribute number of increasing income of project.
According to embodiments of the invention, can realize specially for search and the acquisition of open source software, thus make developer for open source software search and utilize more convenient.
It should be noted last that, above embodiment is only in order to illustrate technical scheme of the present invention and unrestricted, although with reference to preferred embodiment to invention has been detailed description, those of ordinary skill in the art is to be understood that, can modify to technical scheme of the present invention or equivalent replacement, and not depart from the spirit and scope of technical solution of the present invention.
Claims (10)
1. open source software obtains the implementation method with search system, comprises the following steps:
Step 1, build data, services, configuration relation database management language on enterprise-level server, build relation and object map framework, when upper level applications is called relational database, operation layer code can be accessed in the mode of object and upgrade the information of lane database;
Step 2, webpage obtain, receive the input definition of user, determine community's set of increasing income of the known URL address of some, the open source software homepage of being increased income by crawler capturing in community, is divided into open source projects list page to crawl crawling two stages with open source projects homepage by crawling of open source projects;
Step 3, attribute extraction of increasing income, namely the information extracting open source software field from the page crawled increase income attribute;
Step 4, to integrate across community projects information, when the project from different community has different attributes, get attribute union; Preserve the dependence between disparity items simultaneously;
Step 5, set up inverted index to the attribute of increasing income obtained, wherein each all comprises a property value and has the address of each record of this property value, is determined the position of recording, but not determine property value by recording by property value;
Step 6, in response to user search input or predefined rank mechanism, sort to needing the open source projects that represents and show.
2. the method for claim 1, wherein said attribute of increasing income comprises project programming language, certificate, running environment, development status, hour of log-on, up-to-date issuing time, download time, homepage link.
3. the method for claim 1, wherein step 2 comprises further:
Crawl open source projects list page, the project home page URL template according to community is mated, and Extraction Projects homepage URL is stored in database, forms situation carry out snapshot to community's open source projects homepage of constantly change; Crawl open source projects homepage, from database, take out project home page URL carry out orientation and crawl, the complete html page text crawled is saved in database.
4. the method for claim 1, wherein said step 3 comprises further:
Step 31, template detect, and find that each community that increases income is for showing the web page template of attribute of increasing income, and can be the tag path in HTML tree, also can be the regular expression in html text, as the rule extracting attribute of increasing income;
Step 32, by the Template Location that obtained to the html tag of property value of increasing income to carry out tag location.
5. method as claimed in claim 4, wherein said regular expression is determined by manual observation.
6. method as claimed in claim 4, wherein said tag path is determined by seed set, what obtain attribute of increasing income can Column Properties value, use similarity of character string matching algorithm, automatically in collections of web pages, find to increase income property value place tag path, when have in certain position the highest matching times time determine this position thus filter out unique web page template.
7. method as claimed in claim 6, the similarity mode of wherein said character string as two string matching number of degrees values, if this numerical value is higher than certain threshold value predefined, thinks coupling by Jakarta (Jaccard) coefficient of the corresponding q-gram set of two character strings.
8. method as claimed in claim 4, wherein said tag location finds property value label node by regular expression locating rule in target web; Or each label position, according to the document object model tree shape structure of the html page, is expressed as the tag path from page root node to property value place node by described tag location, by supporting that the web analysis instrument in this path carries out tag location and acquisition.
9. the method for claim 1, wherein said step 6 comprises further, if receive the search input of user, then the degree of correlation determined by the string matching calculated between the key word of user's input and inverted index data acquisition is sorted; If do not receive user search input, then according to the project rank mechanism of increase income degree of participation and the definition of information complete degree of project, carry out sorting and show.
10. method as claimed in claim 9, increase income community's number of degree of participation trustship used for a project and other open source projects numbers associated by project of wherein said project quantize; Described information complete degree is quantized by the attribute number of increasing income of project.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201210097169.4A CN102760151B (en) | 2012-04-05 | 2012-04-05 | Implementation method of open source software acquisition and searching system |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201210097169.4A CN102760151B (en) | 2012-04-05 | 2012-04-05 | Implementation method of open source software acquisition and searching system |
Publications (2)
Publication Number | Publication Date |
---|---|
CN102760151A CN102760151A (en) | 2012-10-31 |
CN102760151B true CN102760151B (en) | 2015-02-25 |
Family
ID=47054609
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201210097169.4A Active CN102760151B (en) | 2012-04-05 | 2012-04-05 | Implementation method of open source software acquisition and searching system |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN102760151B (en) |
Families Citing this family (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104978379B (en) * | 2014-10-14 | 2019-03-26 | 腾讯科技(深圳)有限公司 | The building method and device at a kind of application program data station |
CN106776636A (en) * | 2015-11-24 | 2017-05-31 | 北京国双科技有限公司 | Data processing method and device |
CN106294623A (en) * | 2016-08-01 | 2017-01-04 | 北京金和网络股份有限公司 | Web data formatting method based on html tag |
CN107391651A (en) * | 2017-07-17 | 2017-11-24 | 河海大学 | Water conservancy information retrieval system and method based on web crawlers |
CN108182217A (en) * | 2017-12-23 | 2018-06-19 | 合肥弹刚信息科技有限公司 | A kind of APP development systems and dissemination method based on mobile Internet |
CN108090223B (en) * | 2018-01-05 | 2020-05-12 | 牛海波 | Openers portrait method based on internet information |
CN110110044B (en) * | 2019-04-11 | 2020-05-05 | 广州探迹科技有限公司 | Method for enterprise information combination screening |
CN110457561A (en) * | 2019-08-08 | 2019-11-15 | 北京航空航天大学 | A kind of construction method and system of open source community project relationship network |
CN110442772A (en) * | 2019-08-13 | 2019-11-12 | 深圳司南数据服务有限公司 | A kind of intelligence grinds report generation method and terminal |
CN111241497A (en) * | 2020-02-13 | 2020-06-05 | 北京高质***科技有限公司 | Open source code tracing detection method based on software multiplexing feature learning |
CN111813378B (en) * | 2020-07-08 | 2023-03-21 | 北京迪力科技有限责任公司 | Code base construction system, method and related device |
CN112860304B (en) * | 2021-01-29 | 2022-04-01 | 中国科学院计算机网络信息中心 | Open source software management system and method for managing software |
CN113362024B (en) * | 2021-06-03 | 2022-04-26 | 深圳市赞悦科技有限公司 | Application program development multi-module collaboration system based on block chain |
CN116185379B (en) * | 2022-11-17 | 2023-09-22 | 北京东方通科技股份有限公司 | Method for optimizing code hosting platform |
Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102004777A (en) * | 2010-11-19 | 2011-04-06 | 中国科学院软件研究所 | Customizable Web information integration method and system |
CN102063488A (en) * | 2010-12-29 | 2011-05-18 | 南京航空航天大学 | Code searching method based on semantics |
Family Cites Families (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8620849B2 (en) * | 2010-03-10 | 2013-12-31 | Lockheed Martin Corporation | Systems and methods for facilitating open source intelligence gathering |
-
2012
- 2012-04-05 CN CN201210097169.4A patent/CN102760151B/en active Active
Patent Citations (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102004777A (en) * | 2010-11-19 | 2011-04-06 | 中国科学院软件研究所 | Customizable Web information integration method and system |
CN102063488A (en) * | 2010-12-29 | 2011-05-18 | 南京航空航天大学 | Code searching method based on semantics |
Non-Patent Citations (1)
Title |
---|
A Research Collaboratory for Open Source Software Research;Yongqin Gao等;《First International Workshop on Emerging Trends in FLOSS Research and Development》;20071231;全文 * |
Also Published As
Publication number | Publication date |
---|---|
CN102760151A (en) | 2012-10-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN102760151B (en) | Implementation method of open source software acquisition and searching system | |
US8312034B2 (en) | Concept bridge and method of operating the same | |
CN100405371C (en) | Method and system for abstracting new word | |
CN100440224C (en) | Automatization processing method of rating of merit of search engine | |
CN102622445B (en) | User interest perception based webpage push system and webpage push method | |
CN102073726B (en) | Structured data import method and device for search engine system | |
CN100498790C (en) | Retrieving method and system | |
CA2610208C (en) | Learning facts from semi-structured text | |
CN101908071A (en) | Method and device thereof for improving search efficiency of search engine | |
CN106126648B (en) | It is a kind of based on the distributed merchandise news crawler method redo log | |
US20120102015A1 (en) | Method and System for Performing a Comparison | |
CN108052632B (en) | Network information acquisition method and system and enterprise information search system | |
CN102930059A (en) | Method for designing focused crawler | |
CN101344881A (en) | Index generation method and device and search system for mass file type data | |
CN103049542A (en) | Domain-oriented network information search method | |
CN113254630B (en) | Domain knowledge map recommendation method for global comprehensive observation results | |
CN101404666A (en) | Infinite layer collection method based on Web page | |
CN102163234A (en) | Equipment and method for error correction of query sequence based on degree of error correction association | |
CN106980651B (en) | Crawling seed list updating method and device based on knowledge graph | |
CN101393565A (en) | Facing virtual museum searching method based on noumenon | |
Prajapati | A survey paper on hyperlink-induced topic search (HITS) algorithms for web mining | |
CN101546308A (en) | Web page search method and web page search system based on overdue retrieval | |
CN103226609A (en) | Searching method for WEB focus searching system | |
CN103778156A (en) | Method and device for searching for data and server for data search | |
CN101894109A (en) | Database building method and device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
C14 | Grant of patent or utility model | ||
GR01 | Patent grant |