Has default status negatively influenced AskScience?

(A two-day project)

Abstract

Over 90% of participants in the AskScience survey indicated that their primary activity on this subreddit is lurking rather than commenting or asking questions (AskScience survey as of 11/22/2011). Just because we have more readers doesn’t mean that we necessarily have more problems in AskScience - these problems may be different than what we observe in small chunks of our interactions with r/AS. The ever-increasing number of subscriptions is easy to blame for the increases in deleted comments, spam, and policy issues but would unchecking our default status actually solve these issues? This project is a mixed-methods (qual-QUANT) study (Creswell, 2005) into the issues at play related to default subreddit status and will help start a discussion with the AskScience community about modding practices and default status.

Research Questions:

1. Has the increase in subscriptions caused by default status had a negative overall impact upon AskScience in terms of signal to noise?

2. What problems have been caused in AskScience by becoming a default subreddit?

3. What advantages have been enabled in AskScience by becoming a default subreddit?

4. Should AskScience become a non-default subreddit, the mission of AskScience (broaden science education access) aside?

Introduction

What is default status, you ask? On October 14th or thereabouts, AskScience became a default subreddit. This means that all new sign-ups to Reddit.com are automatically subscribed to AskScience with that username. People can unsubscribe if they would like, but AskScience along with other subreddits like AskReddit, r/pics, r/todayilearned, and r/atheism. This means a large increase in subreddit traffic, subscriptions, comments, and even questions. But how does this affect the overall functioning of the subreddit? Is this a problem which needs undone or a growing pain which when finished creates a better, more solid product?

This study is a relatively short investigation into the default subreddit status and its impacts upon AskScience. Using Reddit Enhancement Suite, moderator tools, and top threads from the past a quantitative and qualitative portrait of the evidenced effects of default status on AskScience can start a conversation about this change and future directions.

Data Sources:

Mod tools such as pageviews/traffic stats

Specific search engines to look for AskScience questions

Top posts, sorted and matched according to time period

A few pieces of data from the AskScience survey

Hypotheses:

1. Subscriptions have increased in AskScience, but this has allowed for community upvoting/downvoting to increase in stead of moderator intervention for top posts. Single top-level comments needing deletion, however, has increased but may be due to increased awareness and ‘hyper-moderation’.

2. The overall number page views have increased, but the ratio of page views to subscribers has not.

3. Threads which hit the front page of r/all after becoming a default subreddit do not have a significant increase in the number of deleted comments when compared to similarly phrased/voted questions prior to default status.

4. The decrease in satisfaction with AskScience is a function of a perceived effect caused by the increasing subscribers rather than actual differences. The only exception to this is the ratio of submitted questions compared to available panelists which may be diluting the quality.

Methods:

Some statistics, such as those from the AskScience survey are very easy to obtain using moderator tools or Reddit Enhancement Suite (RES). Other more qualitative methods such as looking at questions with similar upvoting/downvoting patterns on similar topics can provide miniature case-studies to compare non-default and default time periods on measures such as  comment voting behavior and number of deleted comments (expressed as a ratio to total comments to control for volume). Nine comparable threads were located, two sets of two, one set of three, and two ‘controversial’ threads.

The two categories of default status for this study contain:

1. Pre-default (prior to 10/14/2011)

2. Post-default (after 10/15/2011 but not during the Great Unchecking)

The qualifications for a thread to be included in this study are as follows:

1. Must be a popular thread with at least 100 comments in total and a total of 150 upvotes

2. Must be a thread likely to cause off-topic behavior (relating to bodily functions [POOP], drug/alcohol use [TREES], controversial topics [EVOLUTION], etc.)

3. Must have a comparable set of threads/questions in the two time periods.

4. Must not be a meta-thread or an AskScienceAMA/Education thread.

5. Threads that were considered exceptional/highly unique (i.e. response to a specific event) were not included

6. Threads which included a link in the top 5 top-level comments to previous AskScience responses to the questions were not included

RESULTS

The top 350 threads of all time in AskScience were printed and categorized. This number was deemed sufficient because it contained threads as far back as 10 months ago and encompassed all types of threads seen on AskScience prior to and after default status. The threads were categorized by title alone, allowing for minimum interpretation/wiggle room. Those questions which made the ‘short list’ for further investigation were read in entirety but comments were not viewed in order to minimize researcher bias.

Categories for threads and their definitions

Meta (21): These threads were moderator or community started and dealt with ‘housekeeping items’ such as panelists, readership, or community reminders

Educational (29): These threads were mostly community/panelist started, either as scientist AMAs, questions from a 6th grade class, or regarding new educational resources/ideas.

Interesting (245): These were interesting questions which did not fit into any other category and were not ‘exceptional’ in the true sense of the word.

Controversial/Gross (18 & 49): These threads were either regarding a topic of controversy (climate change, evolution, homeopathy, religion) or gross in nature (defined as likely to elicit off-topic responses including anecdotes [POOP] [TREES]).

(Note that the numbers do not match up exactly because I didn’t count them more than once. Before we officially present this as research (instead of science fair) I’ll count again but for now, it’ll do.)

Choice of in-depth threads for analysis

Considering the ability to ‘cherry pick’ threads which show exceptions rather than typical threads (but still gain useful information about problematic threads like controversial/gross questions), questions answered via solely social science means (i.e. anthropology, sociology, economics) where there is not enough evidence for some scientific consensus were excluded. Threads which were questions asked by a panelist or moderator were also eliminated from consideration because of possible question-asking bias. All questions under consideration received at least 150 upvotes and had a minimum of 100 comments in total (except for the ‘Why do we kiss?’ series which has less comments but three instances of data)

List of chosen/matched threads:

Controversial (two threads to compare, pre- and post-default)

1a. 97% of scientists agree that climate change is occuring. How many of them agree that we are accelerating the phenomenon and by how much? (November 4th, 1188 signal/609 noise, 628 comments)

1b. Does circumcision do anything? (August 23rd, 546 signal/266 noise, 496 comments)

Gross/Off-topic potential on same topic (two threads to compare, one pre- and two post-default)

2a. Is it a cultural idea that women have to be moody during their period that has translated into reality? (August 28th, 430 signal/204 noise, 178 comments)

2b. Why are women (or why are women perceived to be) generally more emotionally reactive to things? (November 4th, 466 signal/308 noise, 335 comments)

Gross/Off-topic potential on same topic (two threads to compare, one pre- and two post-default)

3a. Is it possible for someone who is morbidly obese to go without eating for a very long period of time (months) and survive off their own fat as an energy source? (August 3rd, 645 signal/329 noise, 135 comments)

3b. An average sized man, a body builder and a morbidly obese man are all deprived of food. Who would starve to death first and why? (May 8th, 593 signal/221 noise, 340 comments)

Gross/Off-topic potential on same topic (three threads to compare, one pre- and two post-default)

4a. Why do humans kiss? Are there any other animals that do this to display love, or is there any animal equivalent in some species? (February 13th, 96 signal/24 noise, 39 comments)

4b. Why do humans kiss? (October 28th, 67 signal/27 noise, 75 comments)

4c. Why do we kiss? (November 12th, 871 signal/412 noise, 164 comments)

Once the threads were chosen, each thread has been quantified in the following ways:

1. Total number of up/downvotes for the question

2. Total number of up/downvotes for the top comment

3. Total number of comments posted

4. Number of scientific(panelist in area)/scientific(panelist out/community) comments

5. Ratio of discussion/off-topic/abusive/deleted comments to the total number of comments

Table 1: Comparison of threads

Q: Question number; Default status: Pre- or post- default; Vote: Upvote/downvote ratio; TC: Top comment upvote/downvote ratio; TC type: Top comment type; PIA: # Panelist commenting in-area; PCO: # Panelist/community commenting out-area; Disc: % discussion-continuing responses; Off-T: % off-topic (in context) responses; Abuse: % abusive comments; Del: % deleted comments; Total: total # of comments  (*no up/downvote info available)

Table 2: Traffic Stats

The comment summary shown in Table 1 summarizes each comparable thread. The total vote count for the thread is presented, along with data on the top comment. Top comments were either from the community/scientific, from a panelist in their area of expertise, or from a panelist outside of their area of expertise. The number of comments where panelists were speaking within their area of expertise (in the content of their comment, not necessarily related to the topic of the post) were compared to the number of times a panelist’s comment was outside of their area of expertise. The relative percentages of comments which furthered discussion (Disc) can be compared to the percentage of off-topic (anti-discussion) comments and the percentages of abusive (Abuse) and deleted (Del) comments. Deleted comments were generally removed by moderators, but comments where the poster had removed them were also counted in this category. Reported comments were not counted because of the inability to distinguish between reported/removed and non-reported/removed comments. Comments could be in more than one of the four percentage-calculated categories, for example, a comment could be both discussive and deleted, or both off-topic and abusive.

Discussion:

Panelists are very helpful when commenting within their area of expertise, however in controversial threads there tends to be a higher number of out-of-expertise commenting by panelists.

Controversial threads do not seem to have been affected by the change to default status, with a majority of comments remaining discussive. There was a higher rate of off-topic and abusive threads, but relative to the number of discussive comments it does not seem to be an issue. There is, however, more participation in these threads.

Threads which are easy for the community to be off-topic/gross show a different pattern, however. The total number of discussive comments stays relatively constant (within the 60-70% range) but the percentage of abusive comments is much higher. Additionally, the percentage of off-topic responses is sometimes higher, but not drastically so. Compared to the marked increase in deleted comments post-default, it seems as though more comments are being deleted in threads but this has not increased the discussion in AskScience posts.

Recommendations:

- Remind panelists to refrain from posting comments outside of their areas of expertise, especially on controversial threads.

- Watch controversial threads for abusive comments; watch gross/off topic threads for off-topic comments by both panelists and community members alike

- Making sure the top comment is a productive, scientific discussion-starting thread seems to be integral.

- Moderators should discuss whether deleting off-topic (but non-abusive) comments nested within productive discussion top-level comments should be deleted, as reddit allows users to hide comment trees (or set thresholds for hiding comments automatically). This would result in a less-stringent approach to most humor and/or side-bar conversations found in threads and allow downvoting to ‘take care’ of these types of comments.

- Moderators should continue to delete off-topic top-level comment trees, especially in gross/off-topic producing threads.

- Moderators should make sure to delete all abusive comments (non-polite discourse) in all comment threads and trees.

Conclusions:

Default status for AskScience has not decreased the quality of answers in popular threads but has shifted the types of troublesome comments in specific types of threads. These changes, however, are not proportional to the increase in unique users and number of subscriptions (i.e. there are not 100 times more off-topic/abusive comments). In response to this, directed moderation policies and more specific moderation-in-action seems warranted rather than ceasing default status.