An effective and efficient web content extractor for optimizing the crawling process

Basit Öğe Kaydı
dc.authorscopusid54783608800
dc.authorscopusid24481000500
dc.authorscopusid56538500900
dc.authorscopusid16232085100
dc.authorscopusid55293388500
dc.contributor.authorUzun, Erdinç
dc.contributor.authorGüner, Edip Serdar
dc.contributor.authorKılıçaslan, Yılmaz
dc.contributor.authorYerlikaya, Tarık
dc.contributor.authorAgun, Hayri Volkan
dc.date.accessioned2022-05-11T14:15:47Z
dc.date.available2022-05-11T14:15:47Z
dc.date.issued2014
dc.departmentFakülteler, Çorlu Mühendislik Fakültesi, Bilgisayar Mühendisliği Bölümü
dc.description.abstractClassical Web crawlers make use of only hyperlink information in the crawling process. However, focused crawlers are intended to download only Web pages that are relevant to a given topic by utilizing word information before downloading the Web page. But, Web pages contain additional information that can be useful for the crawling process. We have developed a crawler, iCrawler (intelligent crawler), the backbone of which is a Web content extractor that automatically pulls content out of seven different blocks: menus, links, main texts, headlines, summaries, additional necessaries, and unnecessary texts from Web pages. The extraction process consists of two steps, which invoke each other to obtain information from the blocks. The first step learns which HTML tags refer to which blocks using the decision tree learning algorithm. Being guided by numerous sources of information, the crawler becomes considerably effective. It achieved a relatively high accuracy of 96.37% in our experiments of block extraction. In the second step, the crawler extracts content from the blocks using string matching functions. These functions along with the mapping between tags and blocks learned in the first step provide iCrawler with considerable time and storage efficiency. More specifically, iCrawler performs 14 times faster in the second step than in the first step. Furthermore, iCrawler significantly decreases storage costs by 57.10% when compared with the texts obtained through classical HTML stripping. Copyright © 2013 John Wiley & Sons, Ltd.
dc.identifier.doi10.1002/spe.2195
dc.identifier.endpage1199
dc.identifier.issn0038-0644
dc.identifier.issue10en_US
dc.identifier.scopus2-s2.0-84908473420
dc.identifier.scopusqualityQ2
dc.identifier.startpage1181
dc.identifier.urihttps://doi.org/10.1002/spe.2195
dc.identifier.urihttps://hdl.handle.net/20.500.11776/6074
dc.identifier.volume44
dc.indekslendigikaynakScopus
dc.institutionauthorUzun, Erdinç
dc.language.isoen
dc.publisherJohn Wiley and Sons Ltd
dc.relation.ispartofSoftware - Practice and Experience
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/closedAccess
dc.subjectClassification
dc.subjectIntelligent systems
dc.subjectWeb content extraction
dc.subjectWeb crawling
dc.subjectClassification (of information)
dc.subjectDecision trees
dc.subjectExtraction
dc.subjectHTML
dc.subjectHypertext systems
dc.subjectIntelligent systems
dc.subjectRandom forests
dc.subjectTrees (mathematics)
dc.subjectWebsites
dc.subjectBlock extraction
dc.subjectDecision tree learning algorithm
dc.subjectExtraction process
dc.subjectIntelligent crawlers
dc.subjectSources of informations
dc.subjectStorage efficiency
dc.subjectWeb content extractions
dc.subjectWeb Crawling
dc.subjectWeb crawler
dc.titleAn effective and efficient web content extractor for optimizing the crawling process
dc.typeArticle

Dosyalar

Orijinal paket
Listeleniyor 1 - 1 / 1
Küçük Resim Yok
İsim:
6074.pdf
Boyut:
710.36 KB
Biçim:
Adobe Portable Document Format
Açıklama:
Tam Metin / Full Text
İndir

Koleksiyon

Scopus İndeksli Yayınlar Koleksiyonu
Çorlu Mühendislik Fakültesi Koleksiyonu