A Novel Web Scraping Approach Using the Additional Information Obtained from Web Pages
Yükleniyor...
Dosyalar
Tarih
2020
Yazarlar
Dergi Başlığı
Dergi ISSN
Cilt Başlığı
Yayıncı
Institute of Electrical and Electronics Engineers Inc.
Erişim Hakkı
info:eu-repo/semantics/openAccess
Özet
Web scraping is a process of extracting valuable and interesting text information from web pages. Most of the current studies targeting this task are mostly about automated web data extraction. In the extraction process, these studies first create a DOM tree and then access the necessary data through this tree. The construction process of this tree increases the time cost depending on the data structure of the DOM Tree. In the current web scraping literature, it is observed that time efficiency is ignored. This study proposes a novel approach, namely UzunExt, which extracts content quickly using the string methods and additional information without creating a DOM Tree. The string methods consist of the following consecutive steps: searching for a given pattern, then calculating the number of closing HTML elements for this pattern, and finally extracting content for the pattern. In the crawling process, our approach collects the additional information, including the starting position for enhancing the searching process, the number of inner tag for improving the extraction process, and tag repetition for terminating the extraction process. The string methods of this novel approach are about 60 times faster than extracting with the DOM-based method. Moreover, using these additional information improves extraction time by 2.35 times compared to using only the string methods. Furthermore, this approach can easily be adapted to other DOM-based studies/parsers in this task to enhance their time efficiencies. © 2013 IEEE.
Açıklama
Anahtar Kelimeler
algorithm design and analysis, Computational efficiency, document object model, web crawling and scraping, Efficiency, Extraction, Information use, Trees (mathematics), Websites, Construction process, Extraction process, Extraction time, String methods, Text information, Time efficiencies, Web data extraction, Web scrapings, Data mining
Kaynak
IEEE Access
WoS Q Değeri
Q2
Scopus Q Değeri
Q1
Cilt
8
