AMITY UNIVERSITY KOLKATA · CSIT406 · 3 CREDITS · UG

Social Media

Analytics

From network foundations and data pipelines to machine learning, tools and a hands-on capstone.

Dr. Indraneel Mukhopadhyay

Amity Institute of Information Technology, Amity University Kolkata

Complete classroom & self-study deck · 300 slides · Fully aligned to the CSIT406 syllabus

II

MODULE

WEIGHTAGE · 15%

Data Collection & Preprocessing

IN THIS MODULE

• Social-media data sources (APIs, scraping, datasets)
• Ethical considerations & privacy
• Preprocessing: cleaning & feature extraction
• Sentiment analysis & text mining
• Handling missing & noisy data

69

CORE SYLLABUS

MODULE II · LEARNING OUTCOMES

What You Will Be Able to Do

Collect social-media data

Gather data responsibly through APIs, scraping and public datasets.

Apply ethical & legal standards

Respect consent, privacy and laws such as the DPDP Act 2023 and GDPR.

Preprocess raw text

Clean, tokenise and normalise messy social-media text.

Extract features & sentiment

Turn text into features and measure sentiment for analysis.

Handle missing & noisy data

Detect and treat noise, duplicates and gaps without distorting the signal.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

70

CORE SYLLABUS

THE DATA PIPELINE

From Raw Platform to Analysis-Ready Data

1

Collect

Pull data via APIs, scraping or datasets.

→

2

Clean

Remove noise, fix encoding, dedupe.

→

3

Transform

Tokenise, normalise, extract features.

→

4

Enrich

Add sentiment, labels, metadata.

→

5

Store

Save tidy data ready for modelling.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

71

CORE SYLLABUS

DATA SOURCES

Where Social-Media Data Comes From

Official APIs

Structured, permitted access provided by the platform itself.

Web scraping

Extracting data from rendered public web pages programmatically.

Public datasets

Pre-collected, often anonymised research datasets.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

72

CORE SYLLABUS

DATA SOURCES

Application Programming Interfaces (APIs)

WHAT IS AN API?

An API is a defined interface that lets your program request data from a platform in a structured, sanctioned way — typically returning JSON over authenticated HTTP requests.

Authenticated

Access keys / OAuth tokens identify and authorise your app.

Rate-limited

Platforms cap requests per window to protect their systems.

Reliable & legal

The cleanest, most compliant way to collect data — when available.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

73

DEEPER DIVE

DATA SOURCES

Major Platform APIs

API terms change often — always read the current developer agreement before collecting or storing data.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

74

PRACTICAL

PRACTICAL

Collecting Tweets with Tweepy

collect_tweets.py

import tweepy

​

client = tweepy.Client(bearer_token=TOKEN)

​

# recent tweets on a topic

resp = client.search_recent_tweets(

query="#AI -is:retweet lang:en",

max_results=100,

tweet_fields=["created_at","lang"])

​

for t in resp.data:

print(t.created_at, t.text)

Authenticate first

A bearer token from the X developer portal authorises the client.

Query operators

Filter by hashtag, language and exclude retweets right in the query.

Mind the limits

Paginate and respect rate limits; store results incrementally.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

75

CORE SYLLABUS

DATA SOURCES

Web Scraping

WHAT IS WEB SCRAPING?

Web scraping programmatically fetches web pages and extracts structured data from their HTML — used when no suitable API exists. It requires care to stay legal and ethical.

How

Request the page, parse the DOM, select elements, extract fields.

Fragile

Breaks when a site changes its layout; needs maintenance.

Boundaries

Respect robots.txt, terms of service and rate limits.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

76

PRACTICAL

PRACTICAL

Extracting Data with BeautifulSoup

scrape.py

import requests

from bs4 import BeautifulSoup

​

r = requests.get(URL, headers=HEADERS)

soup = BeautifulSoup(r.text, "html.parser")

​

# extract all post titles

for post in soup.select(".post"):

title = post.select_one("h2").text

likes = post.select_one(".likes").text

print(title.strip(), likes.strip())

CSS selectors

Target elements by class or tag to pull exactly the fields you need.

Scale with Scrapy

For large crawls, Scrapy adds concurrency, pipelines and retries.

Be polite

Set headers, throttle requests and honour robots.txt.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

77

DEEPER DIVE

DATA SOURCES

Public Datasets for Network Analysis

SNAP (Stanford)

Large network datasets — social, web, citation and collaboration graphs.

Kaggle

Community datasets: tweets, reviews, sentiment corpora and more.

UCI / KONECT

Curated benchmark and network repositories for research.

Twitter research sets

Election, disaster and topical tweet collections (subject to terms).

Data.gov / open data

Government and civic open datasets for context and enrichment.

Academic repositories

Paper-linked datasets on Zenodo, figshare and Harvard Dataverse.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

78

DEEPER DIVE

DATA SOURCES

Choosing a Collection Method

Rule of thumb: prefer an API; fall back to scraping only when necessary and permitted; use datasets for reproducible study.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

79

DEEPER DIVE

PRACTICAL REALITIES

Rate Limits, Pagination & Data Volume

Rate limits

Requests are capped per time window — design collection to back off and resume.

Pagination

Results arrive in pages; follow cursors/tokens to gather a full result set.

Volume & storage

Social data grows fast — plan storage, deduplication and incremental saves.

Sampling

Free tiers return samples, not the full firehose — document what you actually captured.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

80

ETHICS & PRIVACY

Just because data is accessible does not mean it is ethical — or legal — to collect, store and analyse it.

Responsible data collection is a core competency, not an afterthought. It protects users, your institution and your research.

81

CORE SYLLABUS

ETHICS & PRIVACY

Key Ethical Principles

Consent

Respect users’ reasonable expectations about how their data is used.

Anonymisation

Remove or mask identifiers; aggregate wherever possible.

Data minimisation

Collect only what your question truly requires.

Security

Store data safely; limit access and retention.

Fairness

Avoid harm, bias and discriminatory outcomes.

Transparency

Be clear about purpose, method and limitations.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

82

DEEPER DIVE

ETHICS & PRIVACY

Legal Frameworks You Must Know

DPDP Act 2023 (India)

The Digital Personal Data Protection Act governs processing of personal data in India — consent, purpose limitation and rights of data principals.

GDPR (EU)

Strict rules on personal data of EU residents: lawful basis, minimisation, and the right to erasure.

IT Act 2000 (India)

Governs cyber activity, data security and offences — relevant to how data is accessed and stored.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

83

CASE STUDY

ETHICS & PRIVACY

Case Study — Cambridge Analytica

THE SITUATION

In 2018 it emerged that data from tens of millions of Facebook profiles had been harvested — largely without meaningful consent — and used to build psychographic models for political ad targeting.

How it happened

A quiz app collected data on users and their friends via a permissive API, then repurposed it far beyond its stated use.

What went wrong

Consent was illusory, purpose was violated, and friend-of-friend data was taken without any consent at all.

The lessons

It accelerated GDPR-style regulation, tightened platform APIs, and made consent and purpose limitation non-negotiable.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

84

PRACTICAL

ETHICS & PRIVACY

A Responsible Data-Collection Checklist

Define purpose & scope

State exactly why you need the data and collect no more than that.

Check terms & law

Confirm platform ToS and applicable law (DPDP, GDPR, IT Act) allow your use.

Anonymise early

Strip or hash identifiers as soon as possible; store aggregates.

Secure & retain minimally

Encrypt storage, restrict access and delete data when no longer needed.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

85

CORE SYLLABUS

PREPROCESSING

Why Preprocessing Matters

GARBAGE IN, GARBAGE OUT

Social-media text is exceptionally messy — slang, emojis, hashtags, misspellings, URLs and mixed languages. Preprocessing converts this raw text into clean, consistent input that models can actually learn from.

Consistency

Normalise case, spelling and encoding so tokens match.

Signal over noise

Remove elements that add no meaning for the task.

Better models

Clean input directly improves downstream accuracy.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

86

CORE SYLLABUS

PREPROCESSING

The Text Preprocessing Pipeline

1

Normalise

Lowercase, fix encoding, expand contractions.

2

Clean

Strip URLs, mentions, HTML; handle emojis and hashtags.

3

Tokenise

Split text into words / tokens.

4

Remove stopwords

Drop high-frequency, low-information words.

5

Stem / lemmatise

Reduce words to their base form.

6

Vectorise

Turn tokens into numeric features (next section).

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

87

CORE SYLLABUS

PREPROCESSING

Cleaning Social-Media Text

Social text carries platform-specific noise that must be handled deliberately.

URLs & mentions

Usually removed or replaced with a placeholder token like <URL> or <USER>.

Hashtags

Often kept (they carry topic signal) but split from the # symbol; camelCase may be segmented.

Emojis & emoticons

Can be removed, or mapped to sentiment tokens — they carry real emotional signal.

Punctuation & repeats

Normalise elongated words (“soooo”→“so”) and stray punctuation.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

88

CORE SYLLABUS

PREPROCESSING

Tokenisation, Stopwords, Stemming & Lemmatisation

Tokenisation

Split text into tokens (words, subwords or characters) — the atomic units of analysis.

Stopword removal

Drop common words (the, is, and) that add little meaning — though sometimes they matter for sentiment.

Stemming

Chop words to a crude root (running→run, studies→studi). Fast but imprecise.

Lemmatisation

Map words to their dictionary form using grammar (better→good). Slower but accurate.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

89

PRACTICAL

PRACTICAL

Cleaning Tweets in Python

clean_text.py

import re

from nltk.corpus import stopwords

STOP = set(stopwords.words("english"))

​

def clean(text):

text = text.lower()

text = re.sub(r"http\S+", "", text)

text = re.sub(r"@\w+", "", text)

text = re.sub(r"[^a-z#\s]", "", text)

toks = text.split()

return [t for t in toks if t not in STOP]

Regex does the work

A few substitutions strip URLs, mentions and stray symbols.

Keep the #

We retain hashtags because they carry topic signal.

Drop stopwords

A final comprehension removes low-information words.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

90

CORE SYLLABUS

FEATURE EXTRACTION

From Text to Numbers

FEATURE EXTRACTION

Machine-learning models need numeric input. Feature extraction converts cleaned text into vectors that capture which words appear, how often, and how important they are.

Bag of Words

Count word occurrences, ignoring order.

TF-IDF

Weight words by how distinctive they are.

Embeddings

Dense vectors that capture meaning and similarity.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

91

CORE SYLLABUS

FEATURE EXTRACTION

Bag of Words & N-Grams

BAG OF WORDS (BoW)

Represent a document as a vector of word counts over the vocabulary, discarding word order. N-grams extend this to short sequences of n words to recover a little local context.

Simple & strong

A surprisingly effective baseline for classification.

N-grams

Bigrams/trigrams capture phrases like “not good”.

Weakness

High-dimensional, sparse, and blind to meaning/synonyms.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

92

CORE SYLLABUS

FEATURE EXTRACTION

TF-IDF

TERM FREQUENCY – INVERSE DOCUMENT FREQUENCY

TF-IDF weights each word by how often it appears in a document (TF) times how rare it is across all documents (IDF). Common words are down-weighted; distinctive words that characterise a document score high.

TF

How frequent the term is within this document.

IDF

log(N / documents containing the term) — rarity boosts weight.

Result

A vector that highlights each document’s signature words.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

93

DEEPER DIVE

FEATURE EXTRACTION

Word Embeddings — a Preview

WORD2VEC & GloVe

Embeddings map each word to a dense, low-dimensional vector so that words used in similar contexts sit close together. This captures meaning and analogy far better than counts — explored in depth in Module IV.

Distributional idea

“You shall know a word by the company it keeps.”

Geometry of meaning

king − man + woman ≈ queen.

Why it matters

Powers modern sentiment, search and recommendation.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

94

PRACTICAL

PRACTICAL

TF-IDF with scikit-learn

tfidf.py

from sklearn.feature_extraction.text \

import TfidfVectorizer

​

docs = ["great product love it",

"terrible service never again",

"product okay but slow"]

​

vec = TfidfVectorizer(ngram_range=(1,2))

X = vec.fit_transform(docs)

print(X.shape) # (3, vocab)

print(vec.get_feature_names_out()[:6])

One line to vectors

TfidfVectorizer cleans, tokenises and weights in a single call.

Add n-grams

ngram_range=(1,2) captures unigrams and bigrams together.

Feed a model

X plugs straight into any scikit-learn classifier.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

95

CORE SYLLABUS

TEXT MINING

Text Mining for Social Media

Beyond features, text mining extracts higher-level meaning from social content.

Sentiment analysis

Is the text positive, negative or neutral?

Topic modelling

What themes run through a corpus (LDA)?

Named-entity recognition

Find people, places, brands and products.

Keyword & trend extraction

Surface rising terms and hashtags.

Language detection

Route multilingual content correctly.

Intent & stance

Detect what a user wants or supports.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

96

CORE SYLLABUS

SENTIMENT ANALYSIS

What Is Sentiment Analysis?

SENTIMENT ANALYSIS

Sentiment analysis (opinion mining) automatically determines the emotional polarity of text — typically positive, negative or neutral — and sometimes finer-grained emotions or aspect-level opinions.

Polarity

The basic positive / negative / neutral judgement.

Aspect-based

“Great camera, poor battery” — opinions per aspect.

Why it matters

Brand health, customer support, market research.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

97

CORE SYLLABUS

SENTIMENT ANALYSIS

Two Approaches to Sentiment

Lexicon-based

• Uses dictionaries of sentiment-scored words
• No training data required — works out of the box
• Transparent and easy to explain
• Tools: VADER, TextBlob, SentiWordNet
• Struggles with context, sarcasm and slang

Machine-learning

• Learns sentiment from labelled examples
• Adapts to domain, slang and context
• Higher accuracy with enough data
• Models: Naive Bayes, SVM, LSTMs, BERT
• Needs labelled data and more effort

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

98

DEEPER DIVE

SENTIMENT ANALYSIS

Ready-Made Lexicon Tools

VADER

Tuned specifically for social media — understands emojis, slang, capitalisation and punctuation emphasis. Returns a compound polarity score.

TextBlob

Simple API returning polarity and subjectivity; great for teaching and quick baselines.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

99

PRACTICAL

PRACTICAL

Sentiment with VADER

vader.py

from vaderSentiment.vaderSentiment \

import SentimentIntensityAnalyzer

​

sia = SentimentIntensityAnalyzer()

​

texts = ["Love this! 😍", "Worst ever...",

"It is okay I guess"]

for t in texts:

s = sia.polarity_scores(t)

print(t, "->", s["compound"])

# +0.84, -0.62, +0.20

Compound score

A single number in [−1, +1] summarises overall polarity.

Emoji-aware

VADER reads emojis and emphasis that generic tools miss.

Threshold it

Common cut-offs: ≥ 0.05 positive, ≤ −0.05 negative.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

100

DEEPER DIVE

SENTIMENT ANALYSIS

Why Sentiment Is Hard

Sarcasm & irony

“Oh great, another update” — positive words, negative meaning.

Multilingual & code-mixing

Hindi-English “Hinglish” and other blends confuse single-language models.

Context & domain

“Sick” and “wicked” can be praise; slang shifts meaning by community.

Beyond polarity

Real emotion (anger, joy, fear) is richer than positive/negative.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

101

CASE STUDY

SENTIMENT ANALYSIS

Case Study — Tracking Brand Sentiment on X

THE SITUATION

A consumer brand wants to know how a product launch is being received. It collects tweets mentioning the product for two weeks and runs sentiment analysis to track perception over time.

Collect

Tweepy pulls mentions with the product hashtag, excluding retweets, into a timestamped dataset.

Analyse

VADER scores each tweet; results are aggregated per day and split by feature mentioned.

Act

A dip tied to “battery” complaints triggers a support FAQ and messaging fix — sentiment recovers.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

102

CORE SYLLABUS

MISSING & NOISY DATA

Messy Data in Social Networks

THE REALITY OF SOCIAL DATA

Social-network data is incomplete and noisy by nature: private accounts, deleted posts, API sampling, spam, bots and duplicates all distort the picture. Handling this well is essential to trustworthy analysis.

Missing

Gaps from privacy, deletion and sampling.

Noisy

Spam, bots, typos and duplicates.

Biased

Who is present in the data is rarely representative.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

103

CORE SYLLABUS

MISSING & NOISY DATA

Types of Noise & Missing Data

Missing values

Absent fields — no location, no timestamp, private profile.

Duplicates

Reposts and re-collected records inflate counts.

Bot / spam noise

Automated accounts distort volume and sentiment.

Outliers

Viral spikes and anomalies skew averages.

Encoding errors

Broken characters, mojibake and mixed encodings.

Sampling gaps

API returns a slice, not the full population.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

104

CORE SYLLABUS

MISSING & NOISY DATA

Handling Missing Data

Deletion

Drop rows/columns with too many gaps — simple, but loses information.

Imputation

Fill gaps with mean, median, mode or model-based estimates.

Structural inference

Use network structure to infer likely missing ties or attributes.

Flag & model

Keep a “missing” indicator so the model can learn from absence.

Domain rules

Apply sensible defaults grounded in how the platform works.

Report it

Always document how much was missing and how you handled it.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

105

CORE SYLLABUS

MISSING & NOISY DATA

Handling Noisy, Duplicate & Bot Data

Deduplicate

Hash text and IDs to remove exact and near-duplicate posts.

Filter bots & spam

Use behavioural signals or tools (e.g. Botometer) to remove inauthentic accounts.

Treat outliers

Cap, winsorise or separately analyse viral spikes so they don’t dominate.

Fix encoding & language

Normalise Unicode and route by detected language before analysis.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

106

A WORD OF CAUTION

Noise is relative and task-dependent. Remove too little and signal is buried; remove too much and the data becomes sparse and biased.

The “noise-removal fallacy”: aggressive cleaning can delete exactly the rare, informative cases you were looking for. Clean with the question in mind.

107

PRACTICAL

MISSING & NOISY DATA

A Reusable Cleaning Workflow

1

Profile

Inspect the data: missingness, duplicates, distributions.

2

Deduplicate

Remove exact and near-duplicate records.

3

Filter

Drop bots, spam and out-of-scope records.

4

Impute / flag

Fill or mark missing values deliberately.

5

Validate

Re-profile and sanity-check before modelling.

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

108

DEEPER DIVE

DATA QUALITY

Data-Collection Pitfalls — Do & Don’t

Do

• Document exactly what you collected and when
• Keep a raw copy before any cleaning
• Check representativeness and bias
• Anonymise and secure personal data

Don’t

• Assume an API sample equals the whole population
• Over-clean and delete rare, informative cases
• Ignore duplicates and bot inflation
• Collect personal data “just in case”

Module II · Data Collection & Preprocessing

CSIT406 · Social Media Analytics

109

MODULE II SUMMARY

Data & Preprocessing — Key Takeaways

Choose the right source

APIs, scraping and datasets each trade off legality, freshness and effort.

Ethics is non-negotiable

Consent, minimisation and law (DPDP, GDPR, IT Act) come first.

Clean deliberately

Normalise, tokenise and reduce — with the task in mind.

Features unlock models

BoW, TF-IDF and embeddings turn text into numbers.

Sentiment adds meaning

Lexicon and ML approaches each have their place.

Respect the noise

Handle missing and noisy data without erasing the signal.

110

THANK YOU

Where graph theory meets

business intelligence.

From the structure of networks to the discipline of analytics — you now have the full toolkit to mine social media responsibly and well.

Dr. Indraneel Mukhopadhyay · Amity University Kolkata · CSIT406 Social Media Analytics

300