Study and optimization on system architectures of Larbin

Xuan WANG; Yi-xia HUO; Yun-fei CI; Guo-zhen SHI; Li LI

doi:10.11959/j.issn.2096-109x.2016.00076

您当前的位置：

首页 >

文章列表页 >

Study and optimization on system architectures of Larbin

Academic paper | 更新时间：2024-06-03

- Study and optimization on system architectures of Larbin
- Journal of network and information security Vol. 2, Issue 8, Pages: 74-83(2016)
- 作者机构：
  
  1. 北京电子科技学院信息安全系，北京 100070
  2. 西安电子科技大学计算机学院，陕西西安 710000
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-109x.2016.00076
  CLC： TP393
- Online First：2016-08，
  
  Published：15 August 2016
- 稿件说明：
移动端阅览
Xuan WANG, Yi-xia HUO, Yun-fei CI, et al. Study and optimization on system architectures of Larbin[J]. Journal of network and information security, 2016, 2(8): 74-83.
DOI：

Xuan WANG, Yi-xia HUO, Yun-fei CI, et al. Study and optimization on system architectures of Larbin[J]. Journal of network and information security, 2016, 2(8): 74-83. DOI： 10.11959/j.issn.2096-109x.2016.00076.

摘要

网络爬虫是搜索引擎的重要组成部分，其性能直接影响搜索引擎的准确性和及时性。Larbin是一个高效、简单、功能比较完善的开源爬虫框架，基于此，介绍了几种典型的开源爬虫框架，并对其进行多维度比较；对Larbin体系结构进行详细的介绍；然后指出Larbin在程序结构和流程方面存在的不足，提出对应的优化方案；测试结果表明，改进后的方案在速度和性能方面都有所提高。

Abstract

Web crawler is an important part of the search engine

its performance will directly affect the accuracy and timeliness of the search engine.Larbin is an efficient and simple open source crawler with relatively perfect in functions.Several typical open-source crawler were firstly introduced and a multi-dimensional comparison was made among them.Then

the system architecture and working mechanism of Larbin were given in detail.Its short-comings in the program structure and process were pointed out

and improved programs were proposed.Experimen-tal results show that improved program is better in speed and performance.

关键词

Keywords

references

BRIN S , PAGE L . Reprint of:the anatomy of a large-scale hyper-textual Web search engine [J ] . Computer Networks , 2012 , 56 ( 18 ): 3825 - 3833 .

孙骏雄 . 基于网络爬虫的网站信息采集技术研究 [D ] . 大连 : 大连海事大学， 2014 .

SUN J X . The study on technology of website information collec-tion based on Web crawler [D ] . Dalian : Dalian Maritime University , 2014 .

吕阳 . 分布式网络爬虫系统的设计与实现 [D ] . 成都 : 电子科技大学， 2013 .

LV Y . Distributed Web crawler system design and implementation [D ] . Chengdu : University of Electronic Science and Technology of China , 2013 .

单月光 . 基于微博的网络舆情关键技术的研究与实现 [D ] . 成都 : 电子科技大学， 2013 .

SHAN Y G . The research and implementation of network public opinion's key techniques based on microblog [D ] . Chengdu : University of Electronic Science and Technology of China , 2013 .

李跃健，朱程荣 . 基于 Larbin 的网络爬虫体系结构的研究与改进 [J ] . 计算机技术与发展， 2012 , 22 ( 7 ): 147 - 150 .

LI Y J , ZHU C R . Study and improvement on system architectures of Larbin Web cawler [J ] . Computer Technology and Development , 2012 , 22 ( 7 ): 147 - 150 .

罗浩 . 基于CLucene和Larbin的企业搜索引擎的研究与实现 [D ] . 成都 : 电子科技大学， 2010 .

LUO H . The research and implementation of enterprise search en-gine based on CLucene and Larbin [D ] . Chengdu : University of Electronic Science and Technology of China , 2010 .

杜一平 . 主题搜索网络爬虫的设计与研究 [D ] . 合肥 : 中国科学技术大学， 2009 .

DU Y P . Design and implementation of topical search engine Web crawler [D ] . Hefei : University of Electronic Science and Technol-ogy of China , 2009 .

CAFARELLA M , CUTTING D . Building nutch:open source search. [J ] . Queue , 2004 , 2 ( 2 ): 54 - 61 .

敖东阳 , 刘好杰 . Larbin分析与Windows平台下移植 [J ] . 智能计算机与应用， 2009 ( 4 ): 23 - 24 .

AO D Y , LIU H J . Analysis of Larbin and transplantation to Win-dows system [J ] . Computer Study , 2009 ( 4 ): 23 - 24 .

张敏，孙敏 . 基于 Heritrix 限定爬虫的设计与实现 [J ] . 计算机应用与软件， 2013 , 30 ( 4 ): 33 - 35 .

ZHANG M , SUN M . Design and implementation of qualified spi-der based on Heritrix [J ] . Computer Applications and Software , 2013 , 30 ( 4 ): 33 - 35 .

赵本本，殷旭东，王伟 . 基于 Scrapy 的 GitHub 数据爬虫 [J ] . 电子技术与软件工程， 2016 ( 6 ): 199 - 202 .

ZHAO B B , YIN X D , WANG W . GitHub Web crawler based on Scrapy [J ] . Journal of Electronic Technology And Software Engi-neering , 2016 ( 6 ): 199 - 202 .

KARKI R , GENNERT M A . Fresh analysis of streaming media stored on the Web [J ] . 2011 .

TONG W , XIE X Y . A research on a defending policy against the WebCrawler's attack [C ] // International Conference on Anti- Coun-terfeiting,Security,and Identification in Communication . c2009 : 363 - 366 .

Views

1489

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Public key cryptographic algorithm SM2 optimized implementation on low power embedded platform

Related Author

Ganqin LIU

Hui LI

Hui ZHU

Yukun HUANG

Xingdong LIU

Related Institution

No data

⁰