CN107590236A - A kind of big data acquisition method and system towards enterprise in charge of construction - Google Patents
A kind of big data acquisition method and system towards enterprise in charge of construction Download PDFInfo
- Publication number
- CN107590236A CN107590236A CN201710809082.8A CN201710809082A CN107590236A CN 107590236 A CN107590236 A CN 107590236A CN 201710809082 A CN201710809082 A CN 201710809082A CN 107590236 A CN107590236 A CN 107590236A
- Authority
- CN
- China
- Prior art keywords
- construction
- enterprise
- charge
- data item
- information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000010276 construction Methods 0.000 title claims abstract description 305
- 238000000034 method Methods 0.000 title claims abstract description 31
- 238000012797 qualification Methods 0.000 claims abstract description 78
- 241000270322 Lepidosauria Species 0.000 claims description 44
- 238000012545 processing Methods 0.000 claims description 35
- 239000000284 extract Substances 0.000 claims description 7
- 230000009193 crawling Effects 0.000 abstract description 12
- 238000013480 data collection Methods 0.000 abstract description 5
- 230000007246 mechanism Effects 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000009412 basement excavation Methods 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Landscapes
- Information Transfer Between Computers (AREA)
Abstract
Description
Claims (9)
- A kind of 1. big data acquisition method towards enterprise in charge of construction, it is characterised in that including:Step 1: multiple enterprises in charge of construction are obtained successively each according to the hierarchical regions position relationship where enterprise in charge of construction Uniform resource position mark URL address and corresponding enterprise name;Step 2: the respective URL addresses of the multiple enterprise in charge of construction are saved in URL texts, will be the multiple The respective enterprise name of enterprise in charge of construction is saved in name text file;Step 3: read the URL addresses of the first enterprise in charge of construction from the URL texts, and from the title The enterprise name of first enterprise in charge of construction is read in text, wherein, first enterprise in charge of construction is institute State any one enterprise in charge of construction in multiple enterprises in charge of construction;After the completion of step 3 execution, step 4 and step 9 are performed respectively;Step 4: first enterprise in charge of construction is got according to the URL addresses of first enterprise in charge of construction respectively The sub- URL addresses of company introduction data item, the sub- URL addresses of honor information data item, the sub- URL for information data item of getting the bid Location, the sub- URL addresses of operation information data item;After the completion of step 4 execution, following steps five, step are performed respectively 6th, Step 7: step 8;It is Step 5: simple to the company according to the sub- URL address acquisitions of the company introduction data item using the first crawlers Content page corresponding to Jie's data item, and content page corresponding to parsing the company introduction data item obtains first building The company introduction information of construction enterprises, by company introduction information storage into enterprise in charge of construction's information database;AndStep 6: believed using the second crawlers according to the sub- URL address acquisitions of the honor information data item to the honor Content page corresponding to data item is ceased, and content page corresponding to parsing the honor information data item obtains first building The honor information of construction enterprises, by honor information storage into enterprise in charge of construction's information database;AndStep 7: believed using the 3rd crawlers according to the sub- URL address acquisitions of the acceptance of the bid information data item to the acceptance of the bid Content page corresponding to data item is ceased, and parses the content page corresponding to information data item of getting the bid and obtains first building The acceptance of the bid information of construction enterprises, by the acceptance of the bid information storage into enterprise in charge of construction's information database;AndStep 8: believed using the 4th crawlers according to the sub- URL address acquisitions of the operation information data item to described manage Content page corresponding to data item is ceased, and content page corresponding to parsing the operation information data item obtains first building The operation information of construction enterprises, by operation information storage into enterprise in charge of construction's information database;Step 9: according to the enterprise name of first enterprise in charge of construction from construction market supervision with being obtained on sincere distribution platform Get the sub- URL addresses of the qualification grade data item of first enterprise in charge of construction and the sub- URL of structure art information data item Address;After the completion of step 9 execution, following steps ten, step 11 are performed respectively;Step 10: using the 5th crawlers according to sub- URL address acquisitions of the qualification grade data item to the qualification etc. Content page corresponding to DBMS item, and content page corresponding to parsing the qualification grade data item obtains first building The qualification grade information of construction enterprises, by qualification grade information storage into enterprise in charge of construction's information database; AndStep 11: built using the 6th crawlers according to the sub- URL address acquisitions of the structure art information data item described in Content page corresponding to teacher of the make'ing information data item, and content page corresponding to parsing the structure art information data item obtain it is described The structure art information of first enterprise in charge of construction, enterprise in charge of construction's information database is arrived into structure art information storage In.
- 2. a kind of big data acquisition method towards enterprise in charge of construction according to claim 1, it is characterised in that described Step 4, specifically comprise the following steps:According to the homepage hypertext mark of the URL address acquisitions of first enterprise in charge of construction to first enterprise in charge of construction Remember language html web page;Homepage html web page is parsed, all hyperlink of the homepage html web page are scanned for, complete time of first level pages The search of the two level page is proceeded by after going through again, by this endless form untill underlying pages search is completed, after the completion of search Export the sub- URL addresses of the company introduction data item of first enterprise in charge of construction, the sub- URL of honor information data item Location, sub- URL addresses, the sub- URL addresses of operation information data item of information data item of getting the bid.
- 3. a kind of big data acquisition method towards enterprise in charge of construction according to claim 2, it is characterised in that described According to the homepage hypertext markup language of the URL address acquisitions of first enterprise in charge of construction to first enterprise in charge of construction Html web page is sayed, is comprised the following steps:When the URL addresses of first enterprise in charge of construction are specially login page URL addresses, looked into by enterprise in charge of construction Ask client input username and password;After logining successfully, judge whether identifying code occur on the homepage html web page of first enterprise in charge of construction;If not occurring identifying code on the homepage html web page, triggering performs the step of parsing homepage html web page;OrIf there is identifying code on the homepage html web page, the identifying code picture on the homepage html web page is identified, is obtained Pictorial information on the identifying code picture;The pictorial information is submitted into server to be verified;When identifying code identification is logical Later, triggering performs the step of parsing homepage html web page;When identifying code identification not over when, re-recognize the homepage Identifying code picture on html web page.
- 4. a kind of big data acquisition method towards enterprise in charge of construction according to claim 1, it is characterised in that described Step 5, specifically comprise the following steps:Using first crawlers, circulation obtains each single item company under the sub- URL addresses of the company introduction data item The URL addresses of profile data content, and parse html web page corresponding to the URL addresses of each single item company introduction data content The company introduction information of first enterprise in charge of construction is obtained, the company introduction information is stored to enterprise in charge of construction and believed Cease in database.
- 5. a kind of big data acquisition method towards enterprise in charge of construction according to claim 1, it is characterised in that described Step 10, specifically comprise the following steps:Judge first enterprise in charge of construction on the construction market supervision and sincere distribution platform with the presence or absence of described the Qualification grade data item corresponding to the enterprise name of one enterprise in charge of construction;If the construction market supervision and the qualification grade number that first enterprise in charge of construction is not present on sincere distribution platform According to item, then continue to read the enterprise name of next enterprise in charge of construction from the name text file;If the construction market supervision and the qualification grade data that first enterprise in charge of construction on sincere distribution platform be present , the qualification grade content page of first enterprise in charge of construction is got using the 5th crawlers, described in parsing Qualification grade content page, extract the CompanyName of first enterprise in charge of construction, qualification classification, qualification certificates number, money Matter title, data of issue, validity period of certificate, licence issuing authority;By the CompanyName of first enterprise in charge of construction, qualification classification, qualification certificates number, qualification title, data of issue, card The book term of validity, licence issuing authority successively sequential storage into enterprise in charge of construction's information database.
- 6. a kind of big data acquisition method towards enterprise in charge of construction according to claim 1, it is characterised in that described Step 11, specifically comprise the following steps:Judge first enterprise in charge of construction on the construction market supervision and sincere distribution platform with the presence or absence of described the Structure art information data item corresponding to the enterprise name of one enterprise in charge of construction;If the construction market supervision and the structure art information that first enterprise in charge of construction is not present on sincere distribution platform Data item, then continue to read the enterprise name of next enterprise in charge of construction from the name text file;If the construction market supervision and the structure art Information Number that first enterprise in charge of construction on sincere distribution platform be present According to item, the structure art content page of first enterprise in charge of construction is got using the 6th crawlers, described in parsing Structure art content page, extract the structure art name of first enterprise in charge of construction, identification card number, registration classification, registration Number, registration specialty;By the structure art name of first enterprise in charge of construction, identification card number, registration classification, number of registration, registration specialty successively Sequential storage is into enterprise in charge of construction's information database.
- 7. a kind of big data acquisition method towards enterprise in charge of construction according to claim 1, it is characterised in that described First crawlers, second crawlers, the 3rd crawlers, the 4th crawlers, the 5th reptile Program and the 6th crawlers crawl corresponding data to be crawled by enterprise in charge of construction's inquiring client terminal simultaneously respectively .
- 8. a kind of big data acquisition method towards enterprise in charge of construction according to claim 1, it is characterised in that described First crawlers, second crawlers, the 3rd crawlers, the 4th crawlers, the 5th reptile Program and the 6th crawlers all use separate scrapy frameworks.
- A kind of 9. big data acquisition system towards enterprise in charge of construction, it is characterised in that including:Company information acquisition module, for obtaining multiple build successively according to the hierarchical regions position relationship where enterprise in charge of construction Build the respective uniform resource position mark URL address of construction enterprises and corresponding enterprise name;File storage module, for the respective URL addresses of the multiple enterprise in charge of construction to be saved in URL texts, The respective enterprise name of the multiple enterprise in charge of construction is saved in name text file;File read module, for reading the URL addresses of the first enterprise in charge of construction from the URL texts, and The enterprise name of first enterprise in charge of construction is read from the name text file, wherein, first building is applied Work enterprise is any one enterprise in charge of construction in the multiple enterprise in charge of construction;After the completion of file read module execution, the first sub- URL acquisition modules are performed respectively and the second sub- URL obtains mould Block, wherein,The first sub- URL acquisition modules, described in being got respectively according to the URL addresses of first enterprise in charge of construction The sub- URL addresses of the company introduction data item of first enterprise in charge of construction, the sub- URL addresses of honor information data item, acceptance of the bid letter Cease sub- URL addresses, the sub- URL addresses of operation information data item of data item;Completion is performed in the described first sub- URL acquisition modules Afterwards, perform respectively following first reptile processing module, the second reptile processing module, the 3rd reptile processing module, at the 4th reptile Manage module;First reptile processing module, for being obtained using the first crawlers according to the sub- URL addresses of the company introduction data item Content page corresponding to the company introduction data item is got, and content page corresponding to parsing the company introduction data item obtains To the company introduction information of first enterprise in charge of construction, enterprise in charge of construction's information is arrived into company introduction information storage In database;AndSecond reptile processing module, for being obtained using the second crawlers according to the sub- URL addresses of the honor information data item Content page corresponding to the honor information data item is got, and content page corresponding to parsing the honor information data item obtains To the honor information of first enterprise in charge of construction, enterprise in charge of construction's information data is arrived into honor information storage In storehouse;And3rd reptile processing module, for being obtained using the 3rd crawlers according to the sub- URL addresses of the acceptance of the bid information data item Content page corresponding to the acceptance of the bid information data item is got, and parses the content page corresponding to information data item of getting the bid and obtains To the acceptance of the bid information of first enterprise in charge of construction, enterprise in charge of construction's information data is arrived into the acceptance of the bid information storage In storehouse;And4th reptile processing module, for being obtained using the 4th crawlers according to the sub- URL addresses of the operation information data item Content page corresponding to the operation information data item is got, and content page corresponding to parsing the operation information data item obtains To the operation information of first enterprise in charge of construction, enterprise in charge of construction's information data is arrived into operation information storage In storehouse;The second sub- URL acquisition modules, supervised for the enterprise name according to first enterprise in charge of construction from construction market The sub- URL addresses of qualification grade data item of the pipe with getting first enterprise in charge of construction on sincere distribution platform and construction The sub- URL addresses of teacher's information data item;After the completion of the described second sub- URL acquisition modules perform, following 5th reptile processing module is performed respectively, at the 6th reptile Manage module;5th reptile processing module, for being obtained using the 5th crawlers according to the sub- URL addresses of the qualification grade data item Content page corresponding to the qualification grade data item is got, and content page corresponding to parsing the qualification grade data item obtains To the qualification grade information of first enterprise in charge of construction, the enterprise in charge of construction is arrived into qualification grade information storage In information database;And6th reptile processing module, for the sub- URL addresses using the 6th crawlers according to the structure art information data item Content page corresponding to the structure art information data item is got, and parses content corresponding to the structure art information data item Webpage obtains the structure art information of first enterprise in charge of construction, and structure art information storage is looked forward to the construction In industry information database.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710809082.8A CN107590236B (en) | 2017-09-09 | 2017-09-09 | Big data acquisition method and system for building construction enterprises |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201710809082.8A CN107590236B (en) | 2017-09-09 | 2017-09-09 | Big data acquisition method and system for building construction enterprises |
Publications (2)
Publication Number | Publication Date |
---|---|
CN107590236A true CN107590236A (en) | 2018-01-16 |
CN107590236B CN107590236B (en) | 2020-08-28 |
Family
ID=61051124
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201710809082.8A Active CN107590236B (en) | 2017-09-09 | 2017-09-09 | Big data acquisition method and system for building construction enterprises |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN107590236B (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110020092A (en) * | 2018-11-20 | 2019-07-16 | 皮商云集(厦门)科技有限公司 | Leather industry data center systems based on web crawlers |
CN110134893A (en) * | 2019-04-03 | 2019-08-16 | 广州朗国电子科技有限公司 | A kind of multimachine structure retrieval display method and device based on cloud information issuing system |
CN110502680A (en) * | 2019-08-27 | 2019-11-26 | 重庆大司空信息科技有限公司 | A kind of abstracting method and device of acceptance of the bid bulletin relevant field |
CN110728418A (en) * | 2019-08-26 | 2020-01-24 | 成都市互联互通大数据科技有限公司 | Method for counting waste standard rate |
CN112801820A (en) * | 2021-02-05 | 2021-05-14 | 郝大伟 | Big data acquisition method for building construction enterprises |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102937989A (en) * | 2012-10-29 | 2013-02-20 | 北京腾逸科技发展有限公司 | Parallel distributed internet data capture method and system |
CN103226568A (en) * | 2013-03-12 | 2013-07-31 | 北京百度网讯科技有限公司 | Method and equipment for crawling page |
US20140032540A1 (en) * | 1999-07-20 | 2014-01-30 | Google Inc. | Internet based system and apparatus for paying users to view content and receiving micropayments |
CN103605764A (en) * | 2013-11-26 | 2014-02-26 | Tcl集团股份有限公司 | Web crawler system and web crawler multitask executing and scheduling method |
CN104376063A (en) * | 2014-11-11 | 2015-02-25 | 南京邮电大学 | Multithreading web crawler method based on sort management and real-time information updating system |
CN105279272A (en) * | 2015-10-30 | 2016-01-27 | 南京未来网络产业创新有限公司 | Content aggregation method based on distributed web crawlers |
CN105389310A (en) * | 2014-09-03 | 2016-03-09 | 上海尧博信息科技有限公司 | Method of applying web crawlers to household registration management |
-
2017
- 2017-09-09 CN CN201710809082.8A patent/CN107590236B/en active Active
Patent Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140032540A1 (en) * | 1999-07-20 | 2014-01-30 | Google Inc. | Internet based system and apparatus for paying users to view content and receiving micropayments |
CN102937989A (en) * | 2012-10-29 | 2013-02-20 | 北京腾逸科技发展有限公司 | Parallel distributed internet data capture method and system |
CN103226568A (en) * | 2013-03-12 | 2013-07-31 | 北京百度网讯科技有限公司 | Method and equipment for crawling page |
CN103605764A (en) * | 2013-11-26 | 2014-02-26 | Tcl集团股份有限公司 | Web crawler system and web crawler multitask executing and scheduling method |
CN105389310A (en) * | 2014-09-03 | 2016-03-09 | 上海尧博信息科技有限公司 | Method of applying web crawlers to household registration management |
CN104376063A (en) * | 2014-11-11 | 2015-02-25 | 南京邮电大学 | Multithreading web crawler method based on sort management and real-time information updating system |
CN105279272A (en) * | 2015-10-30 | 2016-01-27 | 南京未来网络产业创新有限公司 | Content aggregation method based on distributed web crawlers |
Non-Patent Citations (1)
Title |
---|
尤建新 等: "《基于Web数据挖掘的网站知识获取及应用——以大众点评网为例》", 《上海大学学报(自然科学版)》 * |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN110020092A (en) * | 2018-11-20 | 2019-07-16 | 皮商云集(厦门)科技有限公司 | Leather industry data center systems based on web crawlers |
CN110134893A (en) * | 2019-04-03 | 2019-08-16 | 广州朗国电子科技有限公司 | A kind of multimachine structure retrieval display method and device based on cloud information issuing system |
CN110134893B (en) * | 2019-04-03 | 2022-05-31 | 广州朗国电子科技股份有限公司 | Multi-mechanism retrieval display method and device based on cloud information publishing system |
CN110728418A (en) * | 2019-08-26 | 2020-01-24 | 成都市互联互通大数据科技有限公司 | Method for counting waste standard rate |
CN110502680A (en) * | 2019-08-27 | 2019-11-26 | 重庆大司空信息科技有限公司 | A kind of abstracting method and device of acceptance of the bid bulletin relevant field |
CN112801820A (en) * | 2021-02-05 | 2021-05-14 | 郝大伟 | Big data acquisition method for building construction enterprises |
Also Published As
Publication number | Publication date |
---|---|
CN107590236B (en) | 2020-08-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN107590236A (en) | A kind of big data acquisition method and system towards enterprise in charge of construction | |
US8219549B2 (en) | Forum mining for suspicious link spam sites detection | |
DE112010002445T9 (en) | Identification of bots | |
CN103559235B (en) | A kind of online social networks malicious web pages detection recognition methods | |
CN105763543B (en) | A kind of method and device identifying fishing website | |
CN104601573B (en) | A kind of Android platform URL accesses result verification method and device | |
CN107943838B (en) | Method and system for automatically acquiring xpath generated crawler script | |
CN107437026B (en) | Malicious webpage advertisement detection method based on advertisement network topology | |
CN103118035B (en) | Method and the device of analyzing web site access request parameters legal range | |
CN101938466A (en) | Open knowledge bases method and equipment for user authentication | |
CN1949780B (en) | Network message leaving system and leaving message filtering method | |
CN108566399A (en) | Fishing website recognition methods and system | |
CN106126747A (en) | Data capture method based on reptile and device | |
CN110909229A (en) | Webpage data acquisition and storage system based on simulated browser access | |
CN103294732A (en) | Web page crawling method and spider | |
CN107807937A (en) | A kind of website SEO processing methods, apparatus and system | |
CN109104421A (en) | A kind of web site contents altering detecting method, device, equipment and readable storage medium storing program for executing | |
Sanglerdsinlapachai et al. | Web phishing detection using classifier ensemble | |
CN106547803A (en) | The method and apparatus for crawling website incremental resource | |
CN110502680A (en) | A kind of abstracting method and device of acceptance of the bid bulletin relevant field | |
CN109948339A (en) | A kind of malicious script detection method based on machine learning | |
CN104899320A (en) | Webpage repair method, terminal, server and system | |
CN114169432B (en) | Cross-site scripting attack recognition method based on deep learning | |
CN109101657A (en) | Multiple level marketing referrer website identification method, device and equipment | |
CN115114676A (en) | Remote webpage tampering monitoring method, system, equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
CB02 | Change of applicant information | ||
CB02 | Change of applicant information |
Address after: No.01, 9 / F, building 2, Guigu building, ningwei street, Xiaoshan District, Hangzhou City, Zhejiang Province Applicant after: Digital cube (Hangzhou) Information Technology Co.,Ltd. Address before: Yuhang District, Hangzhou City, Zhejiang Province, 311100 West Sea Park No. 998 Building No. 4 802 Applicant before: HANGZHOU DATA CUBE CREDIT REFERENCE Co.,Ltd. |
|
GR01 | Patent grant | ||
GR01 | Patent grant | ||
PE01 | Entry into force of the registration of the contract for pledge of patent right | ||
PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A Big data acquisition method and system for construction enterprises Effective date of registration: 20230713 Granted publication date: 20200828 Pledgee: Hangzhou High-tech Financing Guarantee Co.,Ltd. Pledgor: Digital cube (Hangzhou) Information Technology Co.,Ltd. Registration number: Y2023330001481 |
|
PC01 | Cancellation of the registration of the contract for pledge of patent right |
Granted publication date: 20200828 Pledgee: Hangzhou High-tech Financing Guarantee Co.,Ltd. Pledgor: Digital cube (Hangzhou) Information Technology Co.,Ltd. Registration number: Y2023330001481 |
|
PE01 | Entry into force of the registration of the contract for pledge of patent right |
Denomination of invention: A big data collection method and system for construction enterprises Granted publication date: 20200828 Pledgee: Hangzhou High-tech Financing Guarantee Co.,Ltd. Pledgor: Digital cube (Hangzhou) Information Technology Co.,Ltd. Registration number: Y2024330001201 |