The skill of the chase 2:

A 2020 replication

Gabrielle Goldstein, Director of Research

Mallory Roman, Associate Director of Research

Abstract: This study tested the efficacy of postcards that follow up on a successful voter contact made by a campaign in increasing voter turnout. Six state legislative campaigns in 4 states provided SDAN with a list of people that they had successfully contacted and identified as supporters through phonebanking or canvassing. A randomly chosen half of these voters received a GOTV postcard shortly before the election providing them with information about how to vote and, for contacts in states besides Florida^[1], reminding them of the contact they had with the campaign. In the analysis, the raw numbers trended in the expected direction, with more voters in the postcard chaser condition voting than voters who did not receive a postcard, though this difference was not statistically significant (p = 0.923). Interestingly, the postcards had a similar effect as in the 2019 study, with some campaigns completely unaffected, some experiencing backlash, and some experiencing a small boost in turnout. Small sample size, high turnout tendencies, and messaging requirements in Florida were all limitations of this study.

Takeaways:

• People in the postcard chaser condition voted at a slightly higher rate than control participants, but this difference was not statistically significant.

• The turnout for voters who received a chaser postcard was 89.17%, while voters who did not receive a chaser postcard turned out at a rate of 89.46%. This difference of 0.29% was not statistically significant (p = 0.923)

• Effects did vary by candidate, with three candidates seeing a boost in voting rate from postcards, one candidate seeing almost no difference in voting rate, and two candidates seeing a slight decrease in voting rate.

• However, none of the interactions were statistically significant (ps = 0.232 or more).

• We are underpowered to detect a small effect of the size observed, and would need a larger sample to determine if this difference between voters in the two conditions is practically meaningful.
• This data suggests that, while not statistically significant in this study, people who received chaser postcards were slightly more likely to vote than people who did not receive chaser postcards, and this effect was generally in line with our previous chaser postcard results (0.2-0.3%). Though again, it was not statistically significant in this sample.
• Voters included in this study were people who were contacted by and who supported the campaigns in 6 specific state legislative districts in Florida, Georgia, North Carolina, and Wisconsin during the 2020 election. This means they are unlikely to be representative of the larger population. More research is needed to clarify the efficacy of chaser postcards.
• Further, this sample has a high voter turnout baseline (the control group turned out at 89.17%), indicating that this sample did not have much to benefit from any intervention. This study should be replicated in a low-mid turnout propensity sample where voters are less likely to vote.  

1. Background and Research Questions:

         Handwritten postcarding has emerged as a popular voter outreach tactic, but there is a lack of experimental evidence on the best practices around handwritten postcarding. SDAN has obtained some past results, in the context of voter registration experiments we partnered with the Voter Participation Center to run (find them here and here), that indicate that using postcards as a chaser tactic that follows up on a previous successful voter contact may be an especially effective use of postcards.

In order to test this chaser approach in a mobilization context, SDAN previously ran a 2019 pilot with a significant interaction effect that suggested a small increase in turnout for one of the candidates involved (but a small decrease in turnout for another candidate involved). In order to expand on the pilot findings, SDAN ran a larger study in 2020 with twice the candidates and states represented. The larger pilot study investigated the same question: Do voters who have a successful phone or canvass contact with a campaign, as defined by the voter identifying as a supporter, vote at higher rates when they received a GOTV chaser postcard reminding them of the earlier voter contact?

H1: Voters identified as supporters in a contact with participating campaigns who receive a chaser GOTV postcard reminding them of the contact and encouraging them to vote will vote at higher rates than voters identified as supporters via phone or canvass contact who do not receive an additional chaser postcard.

2.  Methods:

1. Study Design.

This study was a randomized controlled trial (RCT) designed by SDAN. The study targeted voters who had been successfully contacted by a campaign via canvassing or phonebanking by one of six state legislative campaigns in Florida, Georgia, North Carolina, and Wisconsin. These candidates sent SDAN 1-2 rounds of lists of contacted voters who had identified themselves to campaigns as 1s (definite support) and 2s (leaning towards support) during the campaign contact.

SDAN randomized these voters into two conditions: control (no further communication) or postcard (received a chaser postcard during GOTV). Volunteers completed the postcards and sent them in bulk to the relevant states for in-state mailing on October 20, 2020. Postcards were mailed locally by SDAN volunteers and staff who lived in the target states mailed the postcards locally by October 24th, with estimated delivery to homes from October 27-Nov 2 due to USPS delays. Overall, 3,538 voters in the analysis received postcards and 3,535 voters were enrolled in the control condition, for a total sample of 7,073. An additional 22 subjects were enrolled in the study but could not be matched back to the voter file. These voters were excluded from the analysis as their outcome data (whether they voted) was unavailable.

2. Statistical Analysis

Main model. The central question posed in this analysis is whether there is an association between being receiving a GOTV chaser postcard after a successful phonebank or canvass contact (vs not receiving a chaser postcard) and voting in the general election. To test the model, a logistic regression model was used. Response to voted (voted) was regressed onto age, partisanship, turnout, the dummy variables for race, the dummy variables for gender, and the dummy variables for candidate.

Moderation model. The moderation model focused on whether the effect of condition was moderated by candidate, replicating the exploratory model in the 2019 chaser postcard pilot. Again, multiple logistic regression was used, but this regression also included interaction terms to represent the effect of condition for each of the 6 candidates. To test the moderation model, response to voted (voted) was regressed onto age, partisanship, turnout, the dummy variables for race, the dummy variables for gender, and the dummy variables for candidate, as well as the dummy variables for the interaction between candidate and condition.

3.  Results:

Descriptive.  The sample voted at a very high rate across the board, with more than 89% of participants in both conditions voting in the 2020 presidential general election. This means that this sample is not a representative GOTV sample, but rather a high turnout sample.

Regression analysis. The regression results indicate that people in the chaser postcard condition did vote more, but this effect did not even approach statistical significance (p = 0.923). Interestingly, the GA candidate voters voted less than the voters of almost all of the other candidates, indicating that these populations of voters are somewhat different from one another (which makes sense, as they come from different districts in different states). The only other variable of note is that, as expected, people predicted to have a high likelihood of turning out to vote in presidential elections were more likely to vote (p < 0.001).

        Main Model

Table 1: Main Model voting outcome- Estimated odds ratios (robust standard errors), 95% confidence intervals, and p-values for the relationship between positive response, randomized condition, and other predictor variables.

N = 6,765, 𝝌2 (17) = 722.14, p <0.001, pseudo R2 = 0.2087

        Moderation Model

When looking at the moderation model, none of the interaction effects are significant compared to the reference interaction term (the GA candidate x postcard). Interestingly, with the interaction terms in the model, the condition effect gets closer to significance (p = 0.444) and the odds ratio suggests a larger effect. However, it is still not even approaching marginal significance, meaning this effect cannot really be interpreted. This indicates that postcards may have complex effects that may be closely tied to the campaign for which they advocate, but those effects may be too small to detect in even a sizable sample with this many covariates.

Table 2: Moderation Model with voting outcome and interactions- Estimated odds ratios (robust standard errors), 95% confidence intervals, and p-values for the relationship between positive response, randomized condition and other predictor variables.

N = 6,765, 𝝌2 (22) = 724.20, p <0.001, pseudo R2 = 0.2093

The graph below illustrates the interaction found in the data.

Costs. The chaser postcard condition netted approximately 10 votes total. Since postcards for this study cost approximately (3,537 postcards at $0.50 each) $1,768.50, each of the votes cost approximately $177. This is slightly higher than the 2019 pilot study. Cost per vote for GOTV tactics varies by tactic and election type, among other things. Green and Gerber report an average range of $31-$91 dollars per vote for GOTV tactics like canvassing and phonebanking in mostly academic research (2015), though we know these tactics are more effective than postcarding. Industry estimates SDAN is aware of range from about $15 per vote to over $250 per vote.

4.  Conclusions, Limitations and Future Research

The hypothesis that the people who received contact chaser postcards would vote at higher rates than people who did not was not borne out at a statistically significant level in this study. But, as observed in the 2019 pilot, the tactic worked better for some campaigns than others. This suggests that chaser postcards may be effective in some cases, but that may depend quite a bit on the campaign. Campaign factors that affect the efficacy of this tactic should be explored in future research.

Further, Florida election finance laws prevented the postcards from explicitly mentioning the candidates, meaning the Florida postcards were meaningfully different from the postcards in the other 3 states that explicitly mentioned the candidate and campaign contact. To make sure that those differences did not impact the results we saw, we ran a test of an interaction between receiving a postcard and being one of the Florida campaigns or not, and this test was also not significant - p = 0.747. Further, we also ran the main model on a restricted sample composed just of the GA, NC, and WI candidates. It displayed a similar pattern of results, and condition and interactions between condition and candidate were similarly non-significant. 

It also appears that the sample was highly likely to vote regardless of the intervention, having a median turnout score of 96.4. Given this skewed sample, it’s not surprising that we did not see much movement in the treatment condition -- the members of the treated group were overwhelmingly likely to vote anyway. This study should be replicated in a wider range of turnout scores to determine if it can mobilize low-mid turnout voters.

Since this was a presidential election year compared to an odd-year election in the pilot, it is notable that the effect remained in the expected range (0.2-0.3% at 0.29%), even if not significant, with the tactic mobilizing approximately 10 net voters. This indicates that this tactic may have promise but more work needs to be done to determine best practices.

Appendix

Postcard messages:

Florida candidates -

Hi <<voter firstname>>,

The last day to vote is Nov 3. You can vote…

-By mail

-Early in person until Nov 1

-On Election Day (Nov 3) from 7am-7pm

Details: vote.org/florida

Thanks for being a voter!

<<Sign volunteer first name>>

GA candidate -

Hi <<voter firstname>>,

Thanks for chatting with [GA candidate}’s campaign earlier this year on the phone[at your house]! The last day to vote is Nov 3. You can vote…

-By mail

-Early in person until Oct 30

-On Election Day (Nov 3) from 7am-7pm

Details: vote.org/georgia

Thanks for being a voter!

<<Sign volunteer first name>>

WI candidate -

Hi <<voter firstname>>,

Thanks for chatting with [WI candidate]’s campaign earlier this year on the phone[at your house]! The last day to vote is Nov 3. You can vote…

-By mail

-Early in person until Oct 30-Nov 1 (varies by municipality)

-On Election Day (Nov 3) from 7am-8pm

Details: vote.org/wisconsin

Thanks for being a voter!

<<Sign volunteer first name>>

NC candidate -

Hi <<voter firstname>>,

Thanks for chatting with [NC candidate]’s campaign on the phone[at your house]! The last day to vote is Nov 3. You can vote…

-By mail

-Early in person until Oct 31

-On Election Day (Nov 3) from 6:30am-7:30pm

Details: vote.org/north-carolina

Thanks for being a voter!

<<Sign volunteer first name>>