paper_title

replicator_team

computational_reproduction

perfect_reproduction

not_perfect_reproduction

why_not_perfect_repro

Antinormative Messaging, Group Cues, and the Nuclear Ban Treaty

"We do not find any coding errors that undermine the authors' analysis or conclusions"; Replicators noted missing observations were a combination of "NA" and 95.

Ascriptive Characteristics and Perceptions of Impropriety in the Rule of Law: Race, Gender, and Public Assessments of Whether Judges Can Be Impartial

Perfect Reproduction

Assortative Matching at the Top of the Distribution: Evidence from the World's Most Exclusive Marriage Market

Perfect Reproduction

Black Workers in White Places: Daytime Racial Diversity and White Public Opinion

Perfect Reproduction

Brahmin Left Versus Merchant Right: Changing Political Cleavages in 21 Western Democracies, 1948–2020

Minor coding error: one of the do-files (results-all-1-region.do) did not run correctly but stopped for Belgium (BE) with the error code “convergence not achieved”. When excluding Belgium from that do-file, we were able to run all codes from beginning to end. This did not change the results. Hence, the results were reproducible.

Bubbles, Crashes, and Economic Growth: Theory and Evidence

Minor coding error when generating Figure 1.

Campaign Contributions and Roll-Call Voting in the U.S. House of Representatives: The Case of the Sugar Industry

Perfect Reproduction

Can Information Reduce Ethnic Discrimination? Evidence from Airbnb

Third-decimal place discrepancies for Tables 1, 2, and 5. Replicators translate Stata code to R.

Can Technology Solve the Principal-Agent Problem? Evidence from China's War on Air Pollution

Succeeded in computational reproducibility and direct replication of key results. However, they fail to replicate the weather variables used in the original study (i.e., rebuilding the data set).

Can't We All Just Get Along? How Women MPs Can Ameliorate Affective Polarization in Western Publics

One file pathway needed to be adjusted, but otherwise everything ran.

Changing Hearts and Minds? Why Media Messages Designed to Foster Empathy Often Fail

The replication packages difficult to understand. Many variables were not correctly labelled making it difficult to reproduce the analysis.

Changing Tides: Public Attitudes on Climate Migration

Perfect Reproduction

Checking and Sharing Alt-Facts

No coding error: not perfect using stata 17 but perfect with stata 16

Child Marriage Bans and Female Schooling and Labor Market Outcomes: Evidence from Natural Experiments in 17 Low- and Middle-Income Countries

Perfect Reproduction

Concentration Bias in Intertemporal Choice

Perfect Reproduction

Cooperative Property Rights and Development: Evidence from Land Reform in El Salvador

This paper published as a comment in the JPE due to a coding error. A subset of the replicators on this report found errors where the original authors did not delete duplicates.

Decentralization Can Increase Cooperation Among Public Officials

Perfect Reproduction

Declining Worker Turnover: The Role of Short-Duration Employment Spells

Perfect Reproduction

Digital Addiction

For a small number of observations, the aggregated dependent variable does not equal the sum of the disaggregated categories

Do Thank-You Calls Increase Charitable Giving? Expert Forecasts and Field Experimental Evidence

Small Coding Errors. An average was taken over an ordinal categorical variable. Replicators chose to take the minimum value of the categorical variable. The discrepancy changes very few observations and none of the results.

Do Transitional Justice Museums Persuade Visitors? Evidence from a Field Experiment

Our reproduction of the main findings was largely successful, though we did encounter one coding error on one line of code. When attempting to reproduce Figure 2: Transitional Justice Treatment Effects, the line of the R code “for (i in 22:34){results.df.tj$Variable[i] <- "pardoned"}” produced an error. The error stated, “replacement has 25 rows, data has 24.” However, the remaining code for that figure ran without any problems and created Figure 2. Aside from the one error, the code replicated the graphs perfectly. There were no differences between the replicated graphs and the graphs shown in the original article.

Does Competence Make Citizens Tolerate Undemocratic Behavior?

Perfect Reproduction

Does Patient Demand Contribute to the Overuse of Prescription Drugs?

Perfect Reproduction

Does Public Opinion Affect the Preferences of Foreign Policy Leaders? Experimental Evidence from the UK Parliament

Effective for Whom? Ethnic Identity and Nonviolent Resistance

Needed to change code to include correct controls to reproduce original images/tables

Enabling or Limiting Cognitive Flexibility? Evidence of Demand for Moral Commitment

Although a perfect reproduction, the replicators found their main variable of interest was incorrectly coded.

Entertaining Beliefs in Economic Mobility

Did not include raw data

Evaluating Deliberative Competence: A Simple Method with an Application to Financial Choice

The replicators suggest they were able to computationally reproduce parts of the paper. However, they note that one measure was incorrectly computer.

Exposure and Preferences: Evidence from Indian Slums

They wanted to conduct robustness checks but author did not answer any email

Finance and Green Growth

Multiple coding errors that change the magnitude and precision of the coefficients of interest, primarily surroudning which standard errors are used

Flight to Safety: COVID-Induced Changes in the Intensity of Status Quo Preference and Voting Behavior

Perfect Reproduction

Gender Differences in Cooperative Environments? Evidence from the U.S. Congress

A perfect reproduction with original code was successful. Control variables on population density and household income seem to be incorrect. Replicators replaced these variables with equivalents. Median income goes from 29265 (original) to 37124 (replicator); population density from 1746 (original) to 2827 (replicator).

Good Reverberations? Teacher Influence in Music Composition since 1450

Three bigger issues. No documentation describing datasets, making variable names hard to understand. Second, variables in the analysis dataset are not found in the raw data. Third, cleaning files are omitted from the pacakges. Replicators were unable to reproduce tempo and key signatures found in the analysis dataset. The standardization of the outcome variable is not explained and the replicators could not exactly reproduce it. There was no code for tables in Appendix E.6 nor the variables in these tables on class and wealth.

Hate Crimes and Gender Imbalances: Fears over Mate Competition and Violence against Refugees

Perfect Reproduction

Historical Lynchings and the Contemporary Voting Behavior of Blacks

Perfect Reproduction

Hobo Economicus

Table 4 incorrectly labels 'within metro stations' when it should be 'across' metro stations, and vice versa. Robustness checks in footenotes 16 and 17 are not created using the replication package.

How Do Beliefs about the Gender Wage Gap Affect the Demand for Public Policy?

Perfect Reproduction

How Effective Are Monetary Incentives to Vote? Evidence from a Nationwide Policy

Perfect Reproduction

How Much Should We Trust the Dictator’s GDP Growth Estimates?

There was no attempt for computational reproduction. I4R website does not suggest it was computationally reproduced elsewhere.

Ideological Asymmetries and the Determinants of Politically Motivated Reasoning

Perfect Reproduction

Immigration and Redistribution

Perfect Reproduction

Indecent Disclosures: Anticorruption Reforms and Political Selection

Inflammatory Political Campaigns and Racial Bias in Policing

Perfect Reproduction

Interaction, Stereotypes, and Performance: Evidence from South Africa

Perfect Reproduction

Interventions and Cognitive Spillovers

A minor difference in the 0.001 rounding of decimals for coefficients

Jumping the Gun: How Dictators Got Ahead of Their Subjects

"[S]ome minor inconsistency in relative file paths."

Liquidity Constraints in the US Housing Market

Unable to reproduce Figure 1. The replicators had no code for Figure 2. The replicators tried to perform robustness checks on the data. They recalibrated the moments of the distribution conforming to different parts of the wealth distribution. The model converged to the same parameters as the original authors model.

Local Elites as State Capacity: How City Chiefs Use Local Information to Increase Tax Compliance in the Democratic Republic of the Congo

Additional Stata packages needed to be downloaded (cibar, binscatter, ranktest). Applying Stata labels produced minor errors. Opposite sign on one coefficient estimates (Table 5, dependent variables Incorrect Exemption).

Major Reforms in Electricity Pricing: Evidence from a Quasi-Experiment

There are minor changes at the third decimal point in tables 4 and 5. There is also a minor discrepancy between the text and the code, though the values created are all the same as the original paper's estimates.

Market Access and Quality Upgrading: Evidence from Four Field Experiments

Perfect Reproduction

Market-Based Monetary Policy Uncertainty

There were multiple version issues with Matlab. Impossible to reproduce Figure 3, Table 4, and figures and tables for the C-D appendices using Matlab R2022a (other versions did not produce these errors).

Market-Based Monetary Policy Uncertainty

hac command in MATLAB has been replaced in 2022. After changing command to modern name, the code ran perfectly.

Measuring the Welfare Effects of Pride and Shame

Two minor coding errors in the replication package

Mental Health Costs of Lockdowns: Evidence from Age-Specific Curfews in Turkey

Very minor: one minor error in the survey questionnaire , where one question in the SRQ-20 instrument was not included.after contacting authors, this was an error in the appendix.the question was in fact used.

Mortality, Temperature, and Public Health Provision: Evidence from Mexico

The code ran perfectly. There was a minor coding error from the paper, however, where the original authors dropped February 29th during leap years.

Motivated Beliefs and Anticipation of Uncertainty Resolution

Multinationals' Sales and Profit Shifting in Tax Havens

Perfect Reproduction

Multiracial Identity and Political Preferences

replication challenges arising from changes in the statistical software R. We were unable to reproduce the authors’ results using either the current version of R, or the version that the authors indicate they used. The lack of reproducibility arose due to a change in the defaults used by base R when generating random numbers starting in version 3.6.0. They say they use version R 3.6.2, but they use a version before 3.6

News Shocks under Financial Frictions

The DSGE model was not reproduced even when following the original authors' README instructions, including trying the correct version of MATLAB and Dynare.

Non-Linearities, State-Dependent Prices and the Transmission Mechanism of Monetary Policy

Able to perfom reproduction of Figures 1 and 4 from original text. Interdependancy between routines makes it difficult for a replication exercise.

Not All Elections Are Created Equal: Election Quality and Civil Conflict

Perfect Reproduction

Parties as Disciplinarians: Charisma and Commitment Problems in Programmatic Campaigning

Replicators re-do analysis in a different programming language but found no substantial coding errors.

Patience, Risk-Taking, and Human Capital Investment Across Countries

Discrepencies between programming and writing in paper

Peer Effects in Academic Research: Senders and Receivers

Perfect Reproduction

Playing Politics with Environmental Protection: The Political Economy of Designating Protected Areas

Policy Deliberation and Voter Persuasion: Experimental Evidence from an Election in the Philippines

Had to download archived versions of software which had been removed from CRAN

Political Turnover, Bureaucratic Turnover, and the Quality of Public Services

The replicator wrote their own codes but used the raw data. Replicator was able to exactly reproduce some estimates from the original paper. There were, however, some discrepancies with the numbers of observations and methods stated.

Pre-Colonial Warfare and Long-Run Development in India

Perfect Reproduction

Public Infrastructure and Economic Development: Evidence from Postal Systems

Of note, RGMC state for this robustness test that “We re-estimated our benchmark models focusing only on counties in states in which no county borders underwent changes between 1896 and 2000, which includes 13 states” (Rogowski et al. 2022, p12 f.). Our understanding of this sentence is that they re-run the analysis including only states where no county border has changed. Yet, from their code it seems they include states in which at least one county has a stable border over time. This analysis has obviously very different implications. We therefore follow the description in the paper and include only states with stable borders for all counties, which applies to seven states in the data

Re-Assessing Elite-Public Gaps in Political Behavior

Computationally reproduced by Odum Institute for Research in Social Science at the University of North Carolina at Chapel Hill. Note by Replicators: Kertzer differs from other replications for two reasons. First, it is a meta-analysis, and so all the meta estimates are precisely estimated and statistically significant. Therefore the magnitude and the percent significant are the quantities of interest. Second, the main claim in Kertzer was a small meta-effect size and a low percent significant, so finding a larger effect size and higher percent significant would be consistent with a failure to replicate.

Rebel on the Canal: Disrupted Trade Access and Social Conflict in China, 1650–1911

One coefficient is different at the 3rd digit, and Figure A.5 could not be reproduced due to technical difficulty

Recessions, Mortality, and Migration Bias: Evidence from the Lancashire Cotton Famine

One very minor error: The only error I could spot is in Table 1 (p. 245): the minimum mortality rate in 1851 is 0.569, and not 0.513, as printed in the paper.

Reshaping Adolescents' Gender Attitudes: Evidence from a School-Based Experiment in India

Minor coding error: some discrepancies in the code compared to what was specified in the article, but these discrepancies did not materially affect the results. One incorrect coeffcient estimates in original authors' table 13, column 1 and one incorrect P-value in table 13, column 4.

Run-off Elections in the Laboratory

Original authors did not provide the script to clean the raw data. Moreover, the original authors had to be contacted for the complete replication package (replicators could not find the data).

School Spending and Student Outcomes: Evidence from Revenue Limit Elections in Wisconsin

Social Class and (Un)Ethical Behaviour: Causal and Correlational Evidence

Major coding error: In reproducing the summary tables, some minor differences that are believed to be attributable to rounding errors, some examples of mislabeling and some changes in coefficients. Nevertheless, the results stay qualitatively the same and the error appears to be unintentional. A larger issue appeared with colums 2 and 4 of the replication of Table 5 and Table 8 in the paper (here Table 3 and Table 5 respectively). This is due to the way the authors do not differentiate between continuous and categorical variables in their regressions. Specifically, they treat categorical variables as continuous variables (see Section 4 for details). However, this changes the coefficients of all coefficients in these estimation models. Some of the coefficients even change (in some cases decrease) their level of significance. Nevertheless, the results stay qualitatively the same and the error appears to be unintentional.

Sorting or Steering: The Effects of Housing Discrimination on Neighborhood Choice

Two coding errors: one which does not clean city names leading to too many fixed effects in their main specification, another which incorrectly codes their discrimination variable.

Spillover Effects of Intellectual Property Protection in the Interwar Aircraft Industry

Use R instead of Stata to recode same data

State Action to Prevent Violence Against Women: The Effect of Women's Police Stations on Men's Attitudes toward Gender-based Violence

One minor coding error and three estimates that are reported with opposite signs; neither is of consequence for the study’s main results.

Student Performance, Peer Effects, and Friend Networks: Evidence from a Randomized Peer Intervention

Miscoding of 6 out of 120 variables for the 'Big Five' personality traits. Once coded to missing, the general results are robust, but summary statistics show difference in means on personality traits

Talking Shops: The Effects of Caucus Discussion on Policy Coalitions

Perfect Reproduction

Targeting High Ability Entrepreneurs Using Community Information: Mechanism Design in the Field

One minor coding error which affects the estimates, but not conclusions

Teaching Norms: Direct Evidence of Parental Transmission

Perfect Reproduction

Technological Change and the Consequences of Job Loss

Perfect Reproduction

The Common-Probability Auction Puzzle

Perfect Reproduction

The Curious Case of Theresa May and the Public That Did Not Rally: Gendered Reactions to Terrorist Attacks Can Cause Slumps Not Bumps

Multiple coding errors: First, the dependent variable in column (2) of HMZ’s Table 2 is coded as one (preferring May as best PM) for observations in which participants did not select May: 35,060 of the 97,155 observations come from survey Waves 7 and 8 in which the survey question referred to then-Prime Minister David Cameron. Second, HMZ state that regression to be a logistic regression – however, the implemented Stata command is xtreg, which conducts generalized least squares regressions on panel datasets. We correct both errors, first independently and then jointly, before re-estimating the affected regression. Results remain statistically robust (p < 0.001 in all cases), i.e., our robustness exercises confirm that May’s approval ratings indeed suffered a slump after the Manchester attack. Full results are referred to Table B1. Also other coding errors in election quarter variables: For example, no elections had been coded for the US, while Japanese elections were only coded for one house of the National Diet (Japan’s bicameral parliament). In addition, run-off elections and multiple-round elections that spanned several quarters were sometimes coded for only one quarter, although HMZ’s coding does not systematically always capture the first or second election. Finally, we identified coding inaccuracies that did not seem to follow a systematic pattern with Austria 2016q4, Ecuador 1997q4, Iceland 2008q2 and 2012q2, Mexico 1991q3, Paraguay 2000q3, and Turkey 2014q3. Beyond the coding errors pertaining to election quarters, we identified three sampling errors that turned out to be inconsequential to our conclusions. First, the woman executive variable had 376 missing observations. We reconstruct that variable from publicly available data and further identify coding errors in observations that did not have missing values. Due to the use of time period t + 1 in the approval variable, and to stay as consistent with HMZ as possible, we define the variable by any quarter in which a woman held the office of head of government or state for at least 15 days in the corresponding quarter. In total, we adjust 612 observations. Second, HMZ’s database features duplicate observations for Macedonia from 2006q2 to 2016q2. Third, HMZ access the GTD for data on international terrorist attacks throughout their sample period. However, the GTD does not feature any data for 1993, as that information was lost in an office move (LaFree and Dugan, 2007; START, 2022). HMZ mechanically code all terrorist attack observations in 1993 with a zero.

The Dynamics and Spillovers of Management Interventions: Evidence from the Training within Industry Program

Stata version needed to be changed; Packages needed to be downloaded; Old packages needed to be downloaded to perfectly reproduce values; Missing Code; Missing Data; Transcription errors (rounding); Different Results found in Table A9, C.

The Economic Effects of Long-Term Climate Change: Evidence from the Little Ice Age

Perfect Reproduction

The Effects of Banking Competition on Growth and Financial Stability: Evidence from the National Banking Era

Output relies on the execution order of a script. The replicators noticed a difference in the magnitudes of Table 9.

The Geography of Repression and Opposition to Autocracy

Perfect Reproduction

The Labor Market Impacts of Universal and Permanent Cash Transfers: Evidence from the Alaska Permanent Fund

Two input datasets required to recreate the main datasets were missing from the replication package. The data was provided by the authors upon request. With these changes, we were able to reproduce the results of the paper. The stata code reproduced the coefficients exactly as in the original paper, in this row there are differences but there were some incorrect values in the tables of the paper.

The Long-Run Effects of Sports Club Vouchers for Primary School Children

Perfect Reproduction

The Long-Term Effects of Measles Vaccination on Earnings and Employment

We could successfully reproduce the results of all tables and figures based on the provided raw data and syntax files. There were only two minor issues. First, the Stata syntax files did not include the version command. This resulted in an error message for one of the syntax files. However, the README file provided the version used, so it was easy to fix. Second, there were some minor rounding/copying errors in the construction of the tables. The provided code reveals that individuals aged 25 and individuals aged 60 are not included.

The Macroeconomics of Sticky Prices with Generalized Hazard Functions

Programming edits and additional setups were needed before they could execute the programs provided. Bootstrapped standard errors were different from what was displayed in the paper online for select statistics. Could not reproduce figure 3 in original authors' paper.

The Morning After: Cabinet Instability and the Purging of Ministers after Failed Coup Attempts in Autocracies

Perfect Reproduction

The Origin of the State: Land Productivity or Appropriability?

These discrepancies are small and do not change the qualitative interpretation of the results. For one specification, one variable is included in the regression but not reported in the published table. For one specification, the F-statistic is different than the one reported. For one specification, one standard error was incorrectly reported.

The Power of Hydroelectric Dams: Historical Evidence from the United States over the Twentieth Century

The reproduction appears perfect, up to the third decimal place, which the replicators attribute to differences in versions.

The Relative Efficiency of Skilled Labor across Countries: Measurement and Interpretation

data from IPUMS International was omitted from the replication package and had to be downloaded. There was no indication of which variables to be downloaded. Additionally, had to install grc1leg2 package where originally the package necessary was grc1leg.

The Side Effects of Immunity: Malaria and African Slavery in the United States

Minor coding error. Author manually inputted the 90th percentile.