1 of 5

CSxxx Fundamentals of Information Retrieval

Lecture 2 Web Crawling Fundamentals

Krishnendu Ghosh

Department of Computer Science & Engineering

Indian Institute of Information Technology Dharwad

2 of 5

Web Crawling

Web crawling is the process by which we gather pages from the Web, in

order to index them and support a search engine. The objective of crawling

is to quickly and efficiently gather as many useful web pages as possible,

together with the link structure that interconnects them.

3 of 5

4 of 5

5 of 5

Thank You