<aside> <img src="/icons/hexagon-two-sixths_blue.svg" alt="/icons/hexagon-two-sixths_blue.svg" width="40px" /> Dataset Used: https://huggingface.co/datasets/carblacac/twitter-sentiment-analysis

</aside>

The dataset used is tweets from Twitter which only consists the tweet and the label.

<aside> <img src="/icons/book-closed_blue.svg" alt="/icons/book-closed_blue.svg" width="40px" /> Colab link: https://colab.research.google.com/drive/1SvLH-F_XoZalT7YN3v4trx06uai4IJab?usp=sharing

</aside>

Before heading into the training, we need to do some EDA and Pre-processing on the data to get better insights.

label_count = df['Sentiment'].value_counts()
px.bar(label_count, x = label_count.index, y = label_count.values, title = 'Distribution of Sentiment')

tweet_len_word = [len(tweet.split()) for tweet in df['SentimentText']]
tweet_len_char = [len(list(tweet)) for tweet in df['SentimentText']]

fig = make_subplots(rows = 1, cols = 2, subplot_titles = ['Word Count', 'Character Count'])

fig.add_trace(go.Box(y = tweet_len_word, name = 'Tweet Word Count'), row = 1, col = 1)
fig.add_trace(go.Box(y = tweet_len_char, name = 'Tweet Character Count'), row = 1, col = 2)

fig.update_layout(title_text = 'Distribution of Tweet Length', showlegend = False)
fig.show()

Next we see the distribution of the text length group by each label

df['length'] = df['SentimentText'].str.split().apply(len)

fig = make_subplots(rows = 1, cols = 2, specs = [[{'type': 'domain'}, {'type': 'xy'}]], subplot_titles =
                    ('Table of Positive Sentiment Statistics', 'Distribution of Text Lengths for Positive Tweets'))

desc_positive = df['length'][df['Sentiment'] == 1].describe().to_frame().round(3)
table_data_positive = go.Table(
    header = dict(values = list(desc_positive.columns), fill_color = 'paleturquoise', align = 'left'),
    cells = dict(values = [desc_positive.index] + [desc_positive[col] for col in desc_positive.columns],
                 fill_color = 'lavender', align = 'left'))

fig.add_trace(table_data_positive, row = 1, col = 1)

graph_data_positive = df[df['Sentiment'] == 1]['length']
fig.add_trace(go.Histogram(x = graph_data_positive, marker_color = 'pink',
                           marker_line_color = 'white',
                           marker_line_width = 1.5), row = 1, col = 2)

fig.update_xaxes(title_text = "Text Length", row=1, col=2)
fig.update_yaxes(title_text = "Frequency", row=1, col=2)
fig.update_layout(title_text = 'Distribution of Text Lengths for Positive Tweets', showlegend = False)
fig.show()

Preprocessing

Since its a raw tweet, there are lots of un-needed words inside the data itself. Preprocessing allows us to get rid of those ‘disturbances’ and allows the maximization of the Machine Learning process.

We need to remove the stop words, lemmatize them and separate the sentences within a single row of data. But since the dataset contains 1.6 mil rows, we need to downsample it first for the sake of speed.

nltk.download('punkt', quiet=True)
nltk.download('stopwords', quiet=True)
nltk.download('wordnet', quiet=True)

def preprocess_text(text):
    text = text.lower()
    text = re.sub(r'https?://\S+|www\.\S+', '', text) #Remove urls
    text = re.sub(r'<.*?>+', '', text) #Remove HTML tags
    text = re.sub(r'[^\w\s]', '', text) #Remove Punctuation and Symbols
    text = re.sub(r'\w*\d\w*', '', text) #Remove numbers
    tokens = word_tokenize(text)
    stop_words = set(stopwords.words('english'))
    lemmatizer = WordNetLemmatizer()

    processed_tokens = [lemmatizer.lemmatize(word) for word in tokens if word not in stop_words]
    return ' '.join(processed_tokens)

import pandas as pd

def downsampling(df, target_size):
    balanced_df = pd.DataFrame()

    for label in df['Sentiment'].unique():
        class_subset = df[df['Sentiment'] == label]

        downsample_size = min(target_size, len(class_subset))

        class_subset_downsampled = class_subset.sample(n = downsample_size, random_state = 0)

        balanced_df = pd.concat([balanced_df, class_subset_downsampled], axis = 0)

    balanced_df = balanced_df.sample(frac=1, random_state = 1).reset_index(drop = True)

    return balanced_df

target_size = 100000
balanced_df = downsampling(df, target_size)

balanced_df['SentimentText'] = balanced_df['SentimentText'].apply(preprocess_text)