ML: RoBERTa model

Run the pre-trained model predictions on sample text.

Relevant material:

1. Original author’s github repo
2. Original paper

Explanation:

• Based on previous experiences, it’s best to subset to a (very) small subset and work with that
• We loop through each subreddit and year to get consolidated data for further analysis
• CAN be customized for authors!
• Saves a consolidated folder

Anyone using: * Please double check all folders and links as they might be hardcoded!

!pip install simpletransformers
!pip install torch

Requirement already satisfied: simpletransformers in /opt/conda/lib/python3.10/site-packages (0.70.0)
Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (1.26.4)
Requirement already satisfied: requests in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (2.31.0)
Requirement already satisfied: tqdm>=4.47.0 in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (4.66.1)
Requirement already satisfied: regex in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (2023.12.25)
Requirement already satisfied: transformers>=4.31.0 in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (4.40.1)
Requirement already satisfied: datasets in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (2.19.0)
Requirement already satisfied: scipy in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (1.13.0)
Requirement already satisfied: scikit-learn in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (1.4.2)
Requirement already satisfied: seqeval in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (1.2.2)
Requirement already satisfied: tensorboard in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (2.16.2)
Requirement already satisfied: tensorboardx in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (2.6.2.2)
Requirement already satisfied: pandas in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (2.2.2)
Requirement already satisfied: tokenizers in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (0.19.1)
Requirement already satisfied: wandb>=0.10.32 in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (0.16.6)
Requirement already satisfied: streamlit in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (1.33.0)
Requirement already satisfied: sentencepiece in /opt/conda/lib/python3.10/site-packages (from simpletransformers) (0.2.0)
Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from transformers>=4.31.0->simpletransformers) (3.13.4)
Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/conda/lib/python3.10/site-packages (from transformers>=4.31.0->simpletransformers) (0.22.2)
Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from transformers>=4.31.0->simpletransformers) (23.2)
Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages (from transformers>=4.31.0->simpletransformers) (6.0.1)
Requirement already satisfied: safetensors>=0.4.1 in /opt/conda/lib/python3.10/site-packages (from transformers>=4.31.0->simpletransformers) (0.4.3)
Requirement already satisfied: Click!=8.0.0,>=7.1 in /opt/conda/lib/python3.10/site-packages (from wandb>=0.10.32->simpletransformers) (8.1.7)
Requirement already satisfied: GitPython!=3.1.29,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from wandb>=0.10.32->simpletransformers) (3.1.43)
Requirement already satisfied: psutil>=5.0.0 in /opt/conda/lib/python3.10/site-packages (from wandb>=0.10.32->simpletransformers) (5.9.8)
Requirement already satisfied: sentry-sdk>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from wandb>=0.10.32->simpletransformers) (2.0.1)
Requirement already satisfied: docker-pycreds>=0.4.0 in /opt/conda/lib/python3.10/site-packages (from wandb>=0.10.32->simpletransformers) (0.4.0)
Requirement already satisfied: setproctitle in /opt/conda/lib/python3.10/site-packages (from wandb>=0.10.32->simpletransformers) (1.3.3)
Requirement already satisfied: setuptools in /opt/conda/lib/python3.10/site-packages (from wandb>=0.10.32->simpletransformers) (69.5.1)
Requirement already satisfied: appdirs>=1.4.3 in /opt/conda/lib/python3.10/site-packages (from wandb>=0.10.32->simpletransformers) (1.4.4)
Requirement already satisfied: protobuf!=4.21.0,<5,>=3.19.0 in /opt/conda/lib/python3.10/site-packages (from wandb>=0.10.32->simpletransformers) (4.25.3)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests->simpletransformers) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests->simpletransformers) (3.6)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests->simpletransformers) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests->simpletransformers) (2024.2.2)
Requirement already satisfied: pyarrow>=12.0.0 in /opt/conda/lib/python3.10/site-packages (from datasets->simpletransformers) (15.0.2)
Requirement already satisfied: pyarrow-hotfix in /opt/conda/lib/python3.10/site-packages (from datasets->simpletransformers) (0.6)
Requirement already satisfied: dill<0.3.9,>=0.3.0 in /opt/conda/lib/python3.10/site-packages (from datasets->simpletransformers) (0.3.8)
Requirement already satisfied: xxhash in /opt/conda/lib/python3.10/site-packages (from datasets->simpletransformers) (3.4.1)
Requirement already satisfied: multiprocess in /opt/conda/lib/python3.10/site-packages (from datasets->simpletransformers) (0.70.16)
Requirement already satisfied: fsspec<=2024.3.1,>=2023.1.0 in /opt/conda/lib/python3.10/site-packages (from fsspec[http]<=2024.3.1,>=2023.1.0->datasets->simpletransformers) (2024.3.1)
Requirement already satisfied: aiohttp in /opt/conda/lib/python3.10/site-packages (from datasets->simpletransformers) (3.9.5)
Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/lib/python3.10/site-packages (from pandas->simpletransformers) (2.9.0)
Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.10/site-packages (from pandas->simpletransformers) (2024.1)
Requirement already satisfied: tzdata>=2022.7 in /opt/conda/lib/python3.10/site-packages (from pandas->simpletransformers) (2024.1)
Requirement already satisfied: joblib>=1.2.0 in /opt/conda/lib/python3.10/site-packages (from scikit-learn->simpletransformers) (1.4.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/lib/python3.10/site-packages (from scikit-learn->simpletransformers) (3.4.0)
Requirement already satisfied: altair<6,>=4.0 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (5.3.0)
Requirement already satisfied: blinker<2,>=1.0.0 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (1.7.0)
Requirement already satisfied: cachetools<6,>=4.0 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (5.3.3)
Requirement already satisfied: pillow<11,>=7.1.0 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (10.3.0)
Requirement already satisfied: rich<14,>=10.14.0 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (13.7.1)
Requirement already satisfied: tenacity<9,>=8.1.0 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (8.2.3)
Requirement already satisfied: toml<2,>=0.10.1 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (0.10.2)
Requirement already satisfied: typing-extensions<5,>=4.3.0 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (4.11.0)
Requirement already satisfied: pydeck<1,>=0.8.0b4 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (0.9.0b1)
Requirement already satisfied: tornado<7,>=6.0.3 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (6.4)
Requirement already satisfied: watchdog>=2.1.5 in /opt/conda/lib/python3.10/site-packages (from streamlit->simpletransformers) (4.0.0)
Requirement already satisfied: absl-py>=0.4 in /opt/conda/lib/python3.10/site-packages (from tensorboard->simpletransformers) (2.1.0)
Requirement already satisfied: grpcio>=1.48.2 in /opt/conda/lib/python3.10/site-packages (from tensorboard->simpletransformers) (1.62.2)
Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.10/site-packages (from tensorboard->simpletransformers) (3.6)
Requirement already satisfied: six>1.9 in /opt/conda/lib/python3.10/site-packages (from tensorboard->simpletransformers) (1.16.0)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /opt/conda/lib/python3.10/site-packages (from tensorboard->simpletransformers) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /opt/conda/lib/python3.10/site-packages (from tensorboard->simpletransformers) (2.3.8)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from altair<6,>=4.0->streamlit->simpletransformers) (3.1.3)
Requirement already satisfied: jsonschema>=3.0 in /opt/conda/lib/python3.10/site-packages (from altair<6,>=4.0->streamlit->simpletransformers) (4.21.1)
Requirement already satisfied: toolz in /opt/conda/lib/python3.10/site-packages (from altair<6,>=4.0->streamlit->simpletransformers) (0.12.1)
Requirement already satisfied: aiosignal>=1.1.2 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->simpletransformers) (1.3.1)
Requirement already satisfied: attrs>=17.3.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->simpletransformers) (23.2.0)
Requirement already satisfied: frozenlist>=1.1.1 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->simpletransformers) (1.4.1)
Requirement already satisfied: multidict<7.0,>=4.5 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->simpletransformers) (6.0.5)
Requirement already satisfied: yarl<2.0,>=1.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->simpletransformers) (1.9.4)
Requirement already satisfied: async-timeout<5.0,>=4.0 in /opt/conda/lib/python3.10/site-packages (from aiohttp->datasets->simpletransformers) (4.0.3)
Requirement already satisfied: gitdb<5,>=4.0.1 in /opt/conda/lib/python3.10/site-packages (from GitPython!=3.1.29,>=1.0.0->wandb>=0.10.32->simpletransformers) (4.0.11)
Requirement already satisfied: markdown-it-py>=2.2.0 in /opt/conda/lib/python3.10/site-packages (from rich<14,>=10.14.0->streamlit->simpletransformers) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /opt/conda/lib/python3.10/site-packages (from rich<14,>=10.14.0->streamlit->simpletransformers) (2.17.2)
Requirement already satisfied: MarkupSafe>=2.1.1 in /opt/conda/lib/python3.10/site-packages (from werkzeug>=1.0.1->tensorboard->simpletransformers) (2.1.5)
Requirement already satisfied: smmap<6,>=3.0.1 in /opt/conda/lib/python3.10/site-packages (from gitdb<5,>=4.0.1->GitPython!=3.1.29,>=1.0.0->wandb>=0.10.32->simpletransformers) (5.0.1)
Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit->simpletransformers) (2023.12.1)
Requirement already satisfied: referencing>=0.28.4 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit->simpletransformers) (0.34.0)
Requirement already satisfied: rpds-py>=0.7.1 in /opt/conda/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6,>=4.0->streamlit->simpletransformers) (0.18.0)
Requirement already satisfied: mdurl~=0.1 in /opt/conda/lib/python3.10/site-packages (from markdown-it-py>=2.2.0->rich<14,>=10.14.0->streamlit->simpletransformers) (0.1.2)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Requirement already satisfied: torch in /opt/conda/lib/python3.10/site-packages (2.3.0)
Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from torch) (3.13.4)
Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/lib/python3.10/site-packages (from torch) (4.11.0)
Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch) (1.12)
Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch) (3.3)
Requirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from torch) (3.1.3)
Requirement already satisfied: fsspec in /opt/conda/lib/python3.10/site-packages (from torch) (2024.3.1)
Requirement already satisfied: nvidia-cuda-nvrtc-cu12==12.1.105 in /opt/conda/lib/python3.10/site-packages (from torch) (12.1.105)
Requirement already satisfied: nvidia-cuda-runtime-cu12==12.1.105 in /opt/conda/lib/python3.10/site-packages (from torch) (12.1.105)
Requirement already satisfied: nvidia-cuda-cupti-cu12==12.1.105 in /opt/conda/lib/python3.10/site-packages (from torch) (12.1.105)
Requirement already satisfied: nvidia-cudnn-cu12==8.9.2.26 in /opt/conda/lib/python3.10/site-packages (from torch) (8.9.2.26)
Requirement already satisfied: nvidia-cublas-cu12==12.1.3.1 in /opt/conda/lib/python3.10/site-packages (from torch) (12.1.3.1)
Requirement already satisfied: nvidia-cufft-cu12==11.0.2.54 in /opt/conda/lib/python3.10/site-packages (from torch) (11.0.2.54)
Requirement already satisfied: nvidia-curand-cu12==10.3.2.106 in /opt/conda/lib/python3.10/site-packages (from torch) (10.3.2.106)
Requirement already satisfied: nvidia-cusolver-cu12==11.4.5.107 in /opt/conda/lib/python3.10/site-packages (from torch) (11.4.5.107)
Requirement already satisfied: nvidia-cusparse-cu12==12.1.0.106 in /opt/conda/lib/python3.10/site-packages (from torch) (12.1.0.106)
Requirement already satisfied: nvidia-nccl-cu12==2.20.5 in /opt/conda/lib/python3.10/site-packages (from torch) (2.20.5)
Requirement already satisfied: nvidia-nvtx-cu12==12.1.105 in /opt/conda/lib/python3.10/site-packages (from torch) (12.1.105)
Requirement already satisfied: triton==2.3.0 in /opt/conda/lib/python3.10/site-packages (from torch) (2.3.0)
Requirement already satisfied: nvidia-nvjitlink-cu12 in /opt/conda/lib/python3.10/site-packages (from nvidia-cusolver-cu12==11.4.5.107->torch) (12.4.127)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2->torch) (2.1.5)
Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch) (1.3.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

# Setup - Run only once per Kernel App
%conda install https://anaconda.org/conda-forge/openjdk/11.0.1/download/linux-64/openjdk-11.0.1-hacce0ff_1021.tar.bz2

# install PySpark
!pip install sagemaker_pyspark

# restart kernel
from IPython.core.display import HTML
HTML("<script>Jupyter.notebook.kernel.restart()</script>")

Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Note: you may need to restart the kernel to use updated packages.
Requirement already satisfied: sagemaker_pyspark in /opt/conda/lib/python3.10/site-packages (1.4.5)
Requirement already satisfied: pyspark==3.3.0 in /opt/conda/lib/python3.10/site-packages (from sagemaker_pyspark) (3.3.0)
Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from sagemaker_pyspark) (1.26.4)
Requirement already satisfied: py4j==0.10.9.5 in /opt/conda/lib/python3.10/site-packages (from pyspark==3.3.0->sagemaker_pyspark) (0.10.9.5)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

# Load the required packages

# Dataframes
import pandas as pd

# Regular expressions
import re

# Unidecoder
import unicodedata

# Timestamp / time measurment
import time

# Simpletransformers classifier
from simpletransformers.classification import ClassificationModel
from transformers import RobertaForSequenceClassification, RobertaTokenizer
import torch

# Softmax function for predicted probabiliy calculation
from scipy.special import softmax

import sagemaker
sess = sagemaker.Session()
bucket = sess.default_bucket() 
print(f"the default SageMaker region specific bucket for this account is {bucket}")

sagemaker.config INFO - Not applying SDK defaults from location: /etc/xdg/sagemaker/config.yaml
sagemaker.config INFO - Not applying SDK defaults from location: /root/.config/sagemaker/config.yaml
the default SageMaker region specific bucket for this account is sagemaker-us-east-1-658225707433

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder.appName("PySparkApp-Preds")
    .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.2")
    .config(
        "fs.s3a.aws.credentials.provider",
        "com.amazonaws.auth.ContainerCredentialsProvider",
    )
    .getOrCreate()
)

print(spark.version)

Warning: Ignoring non-Spark config property: fs.s3a.aws.credentials.provider
Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
org.apache.hadoop#hadoop-aws added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-46759415-705c-4213-b7d2-0a98c6ba88fd;1.0
    confs: [default]
    found org.apache.hadoop#hadoop-aws;3.2.2 in central
    found com.amazonaws#aws-java-sdk-bundle;1.11.563 in central
:: resolution report :: resolve 520ms :: artifacts dl 22ms
    :: modules in use:
    com.amazonaws#aws-java-sdk-bundle;1.11.563 from central in [default]
    org.apache.hadoop#hadoop-aws;3.2.2 from central in [default]
    ---------------------------------------------------------------------
    |                  |            modules            ||   artifacts   |
    |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
    ---------------------------------------------------------------------
    |      default     |   2   |   0   |   0   |   0   ||   2   |   0   |
    ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent-46759415-705c-4213-b7d2-0a98c6ba88fd
    confs: [default]
    0 artifacts copied, 2 already retrieved (0kB/24ms)
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).

:: loading settings :: url = jar:file:/opt/conda/lib/python3.10/site-packages/pyspark/jars/ivy-2.5.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
24/04/28 05:19:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
3.3.0

# from pyspark.sql import SparkSession

# spark = (
#     SparkSession.builder.appName("Roberta_Predictions")
#     .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.2.2")
#     .config(
#         "fs.s3a.aws.credentials.provider",
#         "com.amazonaws.auth.ContainerCredentialsProvider",
#     )
#     .getOrCreate()
# )

# print(spark.version)

sc = spark.sparkContext

sc._jsc.hadoopConfiguration().set("mapred.output.committer.class", "org.apache.hadoop.mapred.FileOutputCommitter")

# label_mapping_df.show()

Loading model:

Only needed for very first run!

# !aws s3 cp s3://sagemaker-us-east-1-658225707433/project/Roberta_Model/CARDS_RoBERTa_Classifier.tar.gz CARDS_RoBERTa_Classifier.tar.gz
# !tar -xvf CARDS_RoBERTa_Classifier.tar.gz -C extracted_model/

Processing data:

# !pip install sparknlp

# Text pre-processing functions: Taken from original author 
def remove_between_square_brackets(text):
    return re.sub('\[[^]]*\]', '', str(text))
def remove_non_ascii(text):
    """Remove non-ASCII characters from list of tokenized words"""
    return unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf-8', 'ignore')
def strip_underscores(text):
    return re.sub(r'_+', ' ', text)
def remove_multiple_spaces(text):
    return re.sub(r'\s{2,}', ' ', text)

# Merge text pre-processing functions for texts 
def denoise_text(text):
    text = remove_between_square_brackets(text)
    text = remove_non_ascii(text)
    text = strip_underscores(text)
    text = remove_multiple_spaces(text)
    return text.strip()

from pyspark.sql.functions import regexp_replace, col

def denoise_col(text_col):
    text_col = regexp_replace(text_col, '\[[^\]]*\]', '') # Remove text between square brackets
    text_col = regexp_replace(text_col, '[^\w\s]', '')   # Remove non-alphanumeric characters
    text_col = regexp_replace(text_col, '_', ' ')        # Replace underscores with spaces
    # Add more text cleaning steps if needed
    return text_col

# Load the tokenizer
tokenizer = RobertaTokenizer.from_pretrained("../extracted_model/CARDS_RoBERTa_Classifier/")

# RobertaTokenizer.from_pretrained("s3a://sagemaker-us-east-1-658225707433/project/Roberta_Model/roberta_extracted/CARDS_RoBERTa_Classifier/CARDS_RoBERTa_Classifier/")

# file_path = "s3://sagemaker-us-east-1-658225707433/project/Roberta_Model/roberta_extracted/CARDS_RoBERTa_Classifier/CARDS_RoBERTa_Classifier/"
# RobertaTokenizer.from_pretrained(file_path) 
# Load the model
model = RobertaForSequenceClassification.from_pretrained("../extracted_model/CARDS_RoBERTa_Classifier/")
# model = RobertaForSequenceClassification.from_pretrained("s3://sagemaker-us-east-1-658225707433/project/Roberta_Model/roberta_extracted/CARDS_RoBERTa_Classifier/CARDS_RoBERTa_Classifier/")

# Setting model to eval mode: 
model.eval()

Some weights of the model checkpoint at ../extracted_model/CARDS_RoBERTa_Classifier/ were not used when initializing RobertaForSequenceClassification: ['roberta.pooler.dense.bias', 'roberta.pooler.dense.weight']
- This IS expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

RobertaForSequenceClassification(
  (roberta): RobertaModel(
    (embeddings): RobertaEmbeddings(
      (word_embeddings): Embedding(50265, 1024, padding_idx=1)
      (position_embeddings): Embedding(514, 1024, padding_idx=1)
      (token_type_embeddings): Embedding(1, 1024)
      (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): RobertaEncoder(
      (layer): ModuleList(
        (0-23): 24 x RobertaLayer(
          (attention): RobertaAttention(
            (self): RobertaSelfAttention(
              (query): Linear(in_features=1024, out_features=1024, bias=True)
              (key): Linear(in_features=1024, out_features=1024, bias=True)
              (value): Linear(in_features=1024, out_features=1024, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (output): RobertaSelfOutput(
              (dense): Linear(in_features=1024, out_features=1024, bias=True)
              (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
          )
          (intermediate): RobertaIntermediate(
            (dense): Linear(in_features=1024, out_features=4096, bias=True)
            (intermediate_act_fn): GELUActivation()
          )
          (output): RobertaOutput(
            (dense): Linear(in_features=4096, out_features=1024, bias=True)
            (LayerNorm): LayerNorm((1024,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
    )
  )
  (classifier): RobertaClassificationHead(
    (dense): Linear(in_features=1024, out_features=1024, bias=True)
    (dropout): Dropout(p=0.1, inplace=False)
    (out_proj): Linear(in_features=1024, out_features=18, bias=True)
  )
)

def predict_label(sample_text, model = model, tokenizer = tokenizer):
    
    # Tokenize the input text
    input_ids = tokenizer.encode(sample_text, 
                                 add_special_tokens=True, 
                                 truncation=True, 
                                 padding=True, 
                                 max_length=128, 
                                 return_tensors="pt")
    
    # Make the prediction
    with torch.no_grad():
        outputs = model(input_ids)
        logits = outputs.logits
    
    # Get the predicted label
    predicted_label_id = logits.argmax().item()
    
    # Convert the label id back to the original label using the model's config
    label_map = model.config.id2label
    predicted_label = label_map[predicted_label_id]
    
    return predicted_label

df_path = "s3a://sagemaker-us-east-1-658225707433/project/cleaned_comments"
df = spark.read.parquet(df_path, header = True)

24/04/28 05:20:02 WARN MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties

from pyspark.sql.functions import monotonically_increasing_id

yyyy_values = ['2021', '2022', '2023']
subreddit_values = ['conspiracy']

for yyyy_value in yyyy_values:
    for subreddit_value in subreddit_values:
        print("Filtering...........................")
        filtered_df = df.filter((df['year'] == yyyy_value) & (df['subreddit'] == subreddit_value))
        fraction_x = 2000/filtered_df.count()
        print(f"Sampling {fraction_x}.........................")
        filtered_df = filtered_df.sample(False, fraction= fraction_x)
        print("Collecting..........................")
        body_list = filtered_df.select('body').rdd.flatMap(lambda x: x).collect()
        print("Running.....................................")
        result_list = []
        for k in body_list:
            result_list.append(predict_label(k))
        print("Collating.................................")
        list_df = spark.createDataFrame([(idx, item) for idx, item in enumerate(result_list)], ["index", "labels"])
        print("Joining.....................................")
        df_with_list = filtered_df.crossJoin(list_df.select("labels")).orderBy(monotonically_increasing_id())
        print("Saving................................................")
        s3_path_parquet = f"s3a://sagemaker-us-east-1-658225707433/project_roberta/roberta_tagged_{yyyy_value}_{subreddit_value}"
        df_with_list.write.mode("append").parquet(s3_path_parquet)
        print(f"Done for {yyyy_value} and {subreddit_value}")

Filtering...........................
Sampling 0.0004173137638216927.........................
Collecting..........................
Running.....................................
Collating.................................
Joining.....................................
Saving................................................
24/04/28 05:58:26 WARN package: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.
Done for 2021 and conspiracy
Filtering...........................
Sampling 0.00046137602168836404.........................
Collecting..........................
Running.....................................
Collating.................................
Joining.....................................
Saving................................................
Done for 2022 and conspiracy
Filtering...........................
Sampling 0.0028669395993165215.........................
Collecting..........................
Running.....................................
Collating.................................
Joining.....................................
Saving................................................
Done for 2023 and conspiracy

Consolidated and saving in project:

s3_parent_path = "s3a://sagemaker-us-east-1-658225707433/project_roberta/_by_year_by_subreddit/"
subfolders = spark.sparkContext.wholeTextFiles(s3_parent_path + "*").keys().collect()

# Looping and reading in data:
combined_df = None

for subfolder in subfolders:
    df = spark.read.parquet(subfolder)
    if combined_df is None:
        combined_df = df
    else:
        combined_df = combined_df.union(df)

combined_df.count()

deduplicated_df = combined_df.dropDuplicates(["ID"])

from pyspark.sql.functions import col
deduplicated_df.groupBy("subreddit").count().show()

from pyspark.sql.functions import col
deduplicated_df.groupBy("year").count().show()

s3_path_parquet = f"s3a://sagemaker-us-east-1-658225707433/project/roberta_tagged/"
deduplicated_df.write.mode("append").parquet(s3_path_parquet)