Adaptive Web Crawling through Structure-Based Link Classification - Archive ouverte HAL Access content directly
Conference Papers Year : 2015

Adaptive Web Crawling through Structure-Based Link Classification

(1, 2) , (1, 2)
1
2

Abstract

Generic web crawling approaches cannot distinguish among various page types and cannot target content-rich areas of a website. We study the problem of efficient unsupervised web crawling of content-rich webpages. We propose ACEBot (Adaptive Crawler Bot for data Extraction), a structure-driven crawler that uses the inner structure of the pages and guides the crawling process based on the importance of their content. ACEBot works in two phases: in the learning phase, it constructs a dynamic site map (limiting the number of URLs retrieved) and learns a traversal strategy based on the importance of navigation patterns (selecting those leading to valuable content); in the intensive crawling phase, ACEBot performs massive downloading following the chosen navigation patterns. Experiments over a large dataset illustrate the effectiveness of our system.
Not file

Dates and versions

hal-01261960 , version 1 (26-01-2016)

Identifiers

  • HAL Id : hal-01261960 , version 1

Cite

Muhammad Faheem, Pierre Senellart. Adaptive Web Crawling through Structure-Based Link Classification. ICADL (International Conference on Asian Digital Libraries), Dec 2015, Seoul, South Korea. pp.39-51. ⟨hal-01261960⟩
74 View
0 Download

Share

Gmail Facebook Twitter LinkedIn More