1 of 19

Document clustering using Machine learning.

TANUJ JOSHI

1

ASHWIN GUNADEEP

UDIT SINGH GULERIA

Under the guidance of

Dr. Rajeev Tiwari

Assistant Professor (SS)

School of Computer Science & Engineering

2 of 19

Contents

  • Abstract
  • Introduction
  • Problem Statement
  • Objectives
  • Timeline Chart
  • Literature Review
  • Proposed Work
  • Implementations
    • Project Algorithms
    • Workflow
    • Flowcharts
  • Results/Outcomes
    • Code Snippets/Screenshots
  • Conclusion & Limitations
  • References

2

3 of 19

Abstract

  • People are always in dearth of time, they need everything quick and filtered.
  • Hence clustering was introduced to gather the relevant information in a cluster.
  • Two implemented of the various algorithms for vector representation are Word2Vec & Doc2Vec algorithm.
  • Cluster summarization is performed based on weights of the words.
  • The clustering algorithm is based on k-means principle of clustering.

3

4 of 19

Introduction

  • Document clustering provides better efficiency by generating abstract entity in retrieval of relevant information.
  • Its goal is to discover the natural grouping(s) of a set of patterns, points, objects or documents.
  • The challenging problems of document clustering are big volume, high dimensionality and complex semantics.
  • Motive - to extract particular domain of information through a classifier from a huge collection of documents using clustering algorithm and to obtain best clusters which later on can be used for document summarizations.

4

5 of 19

Problem statement

  • To implement document clustering using Doc2Vec algorithm.
  • Cluster summarization by using the highest weighted words to reveal the gist of the whole document.

5

6 of 19

Objectives

  • Document Preprocessing.
  • Document to vector conversion using Doc2Vec algorithm. 
  • Document Clustering using K-means algorithm.

​

​

​

6

7 of 19

7

Pert Chart :-

[1] Start.

[2] Algorithm Study.

[3] Algorithm Complexity Analysis

[4] Algorithm Coding.

[5]Data Set Collection.

[6] Data Preprocessing.

[7] Testing.

[8] Look for more algorithm optimization.

[9] Look for code optimization.

[10] Conquering issues arising from familiar and unfamiliar situations.

[11] End

​

TimeLine:

2

4

7

3

9

11

8

10

1

5

6

8 of 19

Literature Review

​

​

​

8

  • Latent Dirichlet Allocation (LDA) is also a common technique for topic modeling (extracting topics/keywords out of texts) but it’s very hard to tune, and results are hard to evaluate.
  • Information Retrieval in general relies on a vector space model, first proposed by Salton [4](1971) to model text documents as vectors in the feature space.
  • In 2014 the concept of Word2Vec was presented by Mikilov and Le[5][6][7][8][9][10].
  • In 2016 the extension of Word2vec- Doc2vec.

​

9 of 19

Proposed Work

  • Gathering a textual dataset to work upon.
  • Pre-processing the documents in the dataset by performing stemming and stop-word removal on the documents.
  • The algorithms to be implemented are –Word2Vec/ Doc2Vec (In progress).
  • Neural network implementation (Later work).

9

10 of 19

WORKFLOW :

10

11 of 19

Project Algorithm

​

Step1: Start

Step2: Input: Dataset , Query

Step3:Q1=Stop_Word_Removal(Query) // Q1 is the processed Query

Step4:D1=Stop_Word_Removal(dataset) // D1 is the processed dataset

Step5:Q2= gen_term_freq(Q1) //Q2 is term frequency of the Query

Step6:D2= gen_term_freq(D1) //D2 is term frequency of all documents

Step7: Setup database

11

12 of 19

Project Algorithm

Step8: Upload D2 into database

Step9: D3=Doc2Vec(D1) //D3 is a document vector

Step10: Clustering(Query,D3)

Step11: Words with the highest Document frequency will be used for the summary generation.

Step12: End.

12

13 of 19

13

Flow Chart:-

End

Start

Input: Dataset, Query.

​

Q1= Stop_Word_Removal(Query)

Q2=Gen_Term_Freq(Q1)

 

D1= Stop_Word_Remooval(Dataset)

D2= Gen_Term_Feq(D1)

Upload D2 into Database.

Cluster=(Query,D3)

D3= Doc2Vec(D1)

Void summarizer()- Summary of cluster information

14 of 19

Code Snippets & Screenshots

14

15 of 19

Code Snippets & Screenshots

15

16 of 19

Code Snippets & Screenshots

16

Generate the relevant Docs and non-relevant Docs from a dataset of 500 documents.

17 of 19

Generate the Cluster of relevant and non-relevant Docs :

17

18 of 19

Conclusion & Limitations

  • Time Complexity α Dataset Size α Each Document Size
  • Small document Size
  • Fixed user independent dataset.

18

19 of 19

References

​

19