1 of 5

CSxxx

Information Retrieval

Self-study 1

Web Crawling

Krishnendu Ghosh

Department of Computer Science & Engineering

Indian Institute of Information Technology Dharwad

2 of 5

Web Crawling

Web crawling is the task of finding and indexing web pages. Web scraping uses bots to extract data found on web pages, often without permission. Web scrapers often use AI to find specific data on a page, copying it for use in analytics software.

3 of 5

Tools: Apache Nutch

​

Apache Nutch

Nutch is a highly extensible, highly scalable, matured, production-ready Web crawler which enables fine grained configuration and accommodates a wide variety of data acquisition tasks.

Link: https://cwiki.apache.org/confluence/display/NUTCH/NutchTutorial

4 of 5

Task: Crawling any Site

​

Crawl all pages from any website with due politeness

5 of 5

Thank You