Denialism

We leveraged a climate misinformation NLP model, developed by Coen et al. (2021). It is based on RoBERTa, an optimized version of the BERT LLM model. It was originally trained on paragraph-length online publications, identifying five distinct climate denial claims and a number of subclaims, that are summarized in Table 1. The classifier we’ve leveraged classified each comment as a claim and subclaim combination, or marked it as no claim.

Prevalent climate denial claims and subclaims

Table 1

Technical Difficulties:

Our initial proposal was to utilize this model for our entire dataset. However, compute constraints and technical difficulties prevented us from doing so. In order to bypass this hurdle, we sampled comments from each subreddit and we illustrate the final distribution of our sample data in Figure 1 below. For r/conspiracy, this meant a very small sample. For r/climateskeptic, about ~3% of comments were sampled. For r/climatechange, around ~6% of claims were sampled.

Figure 1

While this sampling prevented us from making inferences on the general Reddit landscape, we can still gleam insightful information about individual actors and general claims on these subreddits. We did so by leveraging the simpletransformers library to apply this model on this sampled dataset, and discuss important results below.

Goal VIII.

Identify the presence and prevalence of specific climate denial claims, such as ‘climate change isn’t real’ or ‘climate policies don’t work’.

Results:

As a result of the application of this pretrained model, each comment was classified as either “No Claim”, or as containing one of the five main claims. The latter were then further classifies as one of the 17 sub-claims, containing climate misinformation. Figure 2 illustrates the overall prevalence of the 5 claims across the three subreddits.

Figure 2

As could be expected, most comments did not contain any misinformation claims. In particular, we did not find climate denial comments in the r/conspiracy comments. Given our earlier research, this is most likely due to the larger number of comments and relatively small sample size we’re working with. With more than 15 million comments, our sample of ~8,000 was probably not enough to pick up climate denial claims.

However, for the other two subreddits, there are comments which show signs of climate denialism. Since these two spaces are exclusively focused on climate, and overall contain much fewer number of comments, we can assume that our ~3-6% random sample is at least somewhat representative of the overall subreddit space. Here, we’ve found more denialism in the r/climateskeptic space, which aligns with our expectations, but claims were found in r/climatechange as well. Denial about humans causing climate change seems to be the most prominent message.

To investigate these spaces further, we looked at the distribution of claims that are being made in these subreddits, in Figures 3.1 and 3.2.

Figure 3.1

The most dominant claim in this subreddit seems to be that the phenomenon global warming is not occurring at all, followed closely by the claim that human greenhouse gases are not causing the climate change.

The claim distribution for climateskeptics shows a strikingly different pattern.

Figure 3.2

It shows here that these participants primarily claim that human greenhouse gases are not causing climate change at all. The second most frequent claim, which is noticeably less frequent than the first claim, is that climate solutions will not work.

We can understand these claims in even more depth, by diving into the distribution of sub-claims, shown in figure 4 below.

Figure 4

Once again, there are dominating trends that stand out in these diagrams. The most common explanations given by these authors is either that the weather hasn’t changed or that this change is part of the natural cycle variations. While there are nuances to be seen in the less frequent claims - and their underlining subclaims - this analysis revealed that there is a general denial about the phenomenon of climate change itself, and there is less discussion about the possible solutions or scientific research around this phenomenon.

This is strikingly different from the research academics conducted on climate skeptic think thank, blog, and YouTube outputs, where the general consensus seems to be that instead of denying it, the focus shifted to specific policies, solutions, science, or people being unreliable. One reason might be a few voices being louder than others who are not educated about the phenomenon on reddit. Even when talking about misinformation, relatively respectable outputs such as well established blogs or conservative think tanks can probably not deny anymore that climate change is indeed happening. This denial is easier for anonym online commenters. This might also reflect a time lag: for a long time, the predominant misinformation was that global warming isn’t real; our results might reflect people perceptive to this kind of misinformation relying on information received at earlier times.

Goal IX.

Understand what attributes of posts influence climate misinformation: see if the presence of climate denial claims can be predicted based on a number of characteristics of posts

Results:

While we used a pre-trained BERT model for the labeling of the claims, we also wanted to develop our own algorithms that are trained specifically on the comments of the three subreddits we have highlighted thus far. We did not consider the sub-claims at this time.

We selected two machine learning models for this purpose.

1. Logistic Regression
A fundamental machine learning algorithm, Logistic Regression is primarily used for binary classification tasks, where the goal is to predict the probability that an observation belongs to a particular class. It models the relationship between the independent variables and the probability of the outcome using the logistic function, also known as the sigmoid function. Logistic regression is popular due to its simplicity, interpretability, and efficiency, making it a common choice for classification tasks.

2. Gradient-Boosted Trees (GBT) Classifier
This algorithm works by combining multiple decision trees in a sequential manner, where each tree corrects the errors of its predecessors. GBTs are known for their high predictive accuracy and robustness against overfitting.

We leveraged these methods hoping that the differences in the logical flow of these algorithms would help shed light on the relationship between the label classification, and our selected features.

Our dependent feature was whether or not a comment contains claims denying climate crisis. The true labels have been acquired via the RoBERTa model and were converted to binary variables.

An important decision we made was to oversample the minority class i.e. oversample observations that are identified as claims, in order to rectify the label imbalance in the dataset.

The overall flow of the model, including all transformations, is shown in Table 2.

Model Workflow

Table 2

The results and evaluation metrics for both these algorithms is shown below, in Figure 5.

Evaluation Metrics for Model Comparison

Figure 5

Following correction for class imbalance, both models perform well and almost similarly. Generally, we’ve seen an accuracy around ~0.8, with both models more likely to misclassify claims as non-claims. Given the small number of features we’ve used and the size of our dataset, this is still a relatively impressive result that suggests certain characteristics are shared between identifiable misinformation posts.

We also experimented with different hyperparameters, and selected regularization for the Logistic Regression to see if results could be improved, with no significant positive impact. Results for this are shown below, in Figure 6:

Hyperparameter Tuning led to no improvement in performance

Figure 6

The lack of insight in this illustration shows that regularization does not have a significant affect on our model. This may be because we have already been selective in the independent features, and are using a relatively simple model. However, explicitly setting the regularization parameters does improve overall accuracy of the model.

We used this best model to then create a profile, of what a comment that makes these denial claims looks like and we illustrate this in figure 7 below. Based on our findings, a post is likely to contain misinformation in the r/climateskeptic space if it’s controversial, long, and positive in its tone, and received a relatively low score. It might not contain any terms we’ve identified as environmental.