Friends don’t let friends copy-paste

### Friends don't let friends copy-paste

![:pull right, 35%](

)

# Conceptual introduction

#### Frederik Aust & Marius Barth

---
exclude: true

---
layout: true
name: footer

<div class="my-footer">
  <div style="float: left;"><span>Frederik Aust & Marius Barth</span></div>
  <div style="text-align: right;"><span>Friends don't let friends copy-paste</span></div>
  <div style="float: center;"><span>21.04.2023</span></div>
</div>

---

const zoomDefault = mediumZoom('#zoom-default')
const zoomMargin = mediumZoom('#zoom-margin', { margin: 45 })
</script>

---
# Agenda

- Computational reproducibility
    - Definition
    - The Problem of non-reproducibility
- Dynamic documents

---

---

---
layout: false
class: inverse, middle, center

# Computational reproducibility

---
layout: true
template: footer
name: comp-rep

# Computational reproducibility

---
exclude: true
.pull-right-40[<img src="data:image/png;base64,#img/scientist.png" width="524" style="display: block; margin: auto;" />]

What does it mean for a scientific finding to be reproducible?

How do we determine if a scientific finding is reproducible?

???
This work is licensed under a Creative Commons Attribution-NonCommercial 2.5 License. https://xkcd.com/242/

---

---
exclude: true

1. Generalizability .grey[(e.g., Baribault et al., 2017; Brandt et al., 2014)]
2. Replicability .grey[(e.g., Open Science Collaboration, 2015; Simons, 2014)]
3. Analytical robustness .grey[(e.g., Silberzahn et al., 2017; Simonsohn, Simmons, & Nelson, 2015; Steegen, Tuerlinckx, Gelman, & Vanpaemel, 2016)]
4. Computational reproducibility .grey[(e.g., Peng, 2011; Rokem, Marwick, & Staneva, 2017; Stodden, 2015)]

<small>But see, for example, Plesser (2018)</small>

???
Computational reproducibility is discussed somewhat less prominently but, I'd argue, is fundamental to this discussion

---
exclude: true

1. Generalizability (e.g., Baribault et al., 2017; Brandt et al., 2014)
2. Replicability (e.g., Open Science Collaboration, 2015; Simons, 2014)
3. Analytical robustness (e.g., Silberzahn et al., 2017; Simonsohn, Simmons, & Nelson, 2015; Steegen, Tuerlinckx, Gelman, & Vanpaemel, 2016)
4. .highlight[Computational reproducibility] (e.g., Peng, 2011; Rokem, Marwick, & Staneva, 2017; Stodden, 2015)

<small>But see, for example, Plesser (2018)</small>

---

.pull-right-40[<img src="data:image/png;base64,#img/scientific-method.png" width="1763" style="display: block; margin: auto;" />]

NSF subcommittee on<br />replicability in science:

<small>

> [Computational] Reproducibility is a minimum necessary condition for a finding to be believable and informative. .grey[(p. 4, Cacioppo, Kaplan, Krosnick, Olds, & Dean, 2015; also see Peng, 2011)]

</small>

]

---

### Is it just me?

1. Full reproduction
  - Access to data
  - Complete analysis plan
  - Lot's of (detective) work

2. Consistency of reported results
  - Reported statistics sufficient
  - In some cases, automatable

---

### Full reproduction

<small>

| Journal | Source | Failure rate |
|:----------------|:-------|-------------:|
| *Journal of Cognition* | .grey[Hardwicke et al. (2018)] | 37% |
| *Quarterly Journal of Political Science* | .grey[Eubank (2016)] | 58% |
| *Strategic Management Journal* | .grey[Bergh et al. (2017)] | 30% |
| *Science* | .grey[Stodden, Seiler, & Ma (2018)] | 41% |

</small>

---

### Full reproduction

<small>

| Field | Source | Failure rate |
|:----------------|:-------|-------------:|
| Psychology | .grey[Artner et al. (2020)] | 30% |
| Psychology (RR) | .grey[Obels et al. (2020)] | 42% |
| Economics | .grey[Vilhuber (2020)] | 39-51% |
| Organismal biology | .grey[Andrew et al. (2015)] | 35% |
| Genetics | .grey[Gilbert et al. (2012)] | 30% |
| Geosciences | .grey[Konkol, Kray & Pfeiffer (2019)] | 56% |
| RCT (primary outcome) | .grey[Naudet et al. (2018)] | 12% |

</small>

---

### Consistency of reported results

<small>

| Field | Source | Failure rate |
|:----------------|:-------|-------------:|
| Psychology | .grey[Nuijten et al. (2020)] | 30% |
| Psychology | .grey[Heathers & Brown (2017)] | 51% |
| Personality and social psychology | .grey[Petrocelli et al. (2013)] | 31% |

</small>

???
- Test statistics and `\(p\)` values .grey[(Bakker & Wicherts, 2011)]
- Descriptive statistics of integer data .grey[(Brown & Heathers, 2017)]
- Equivalence of paths in mediation analysis .grey[(Petrocelli et al., 2013)]

---

.center[
<img src="data:image/png;base64,#img/reporting_errors.png" width="80%" id="zoom-margin" style="display: block; margin: auto;" />
]
.footnote[(Nuijten et al., 2016)]

---

"Non-gross" inconsistency be consequential

> there may have been an .highlight[error in the statistics reported] in the original article […] the .highlight[standard deviations] reported in a similar study [...] were .highlight[approximately 6 times as large and the effect size was substantially smaller] .grey[(p. 323, McCarthy et al., 2018)]
]

???
"exposing participants to hostility-related stimuli caused them subsequently to interpret ambiguous behaviors as more hostile."

---
exclude: true

`statcheck` .grey[(Epskamp & Nuijten, 2016)]

Estimates may be somewhat inaccurate .grey[(e.g., Schmidt, 2016)]

- Corrections for multiple comparisons
- Sphericity corrections
- Misinterpretation of reported statistics<br />(e.g., bootstrap `\(p\)`-values)
- One-tailed test in "wrong" direction
- Misses some reported results (e.g., tables)
- Many undetectable sources of errors

*The problem may even be underestimated!*

---

### Are we just sloppy?

_Quarterly Journal of Political Science_ check data & code for each submission (24 papers)

--
- .highlight[58% produced discrepancies]
- 54% would not run without errors
- 33% did not produce all results

---
exclude: true

### Summary

- Computational non-reproducibility
    - undermines credibility of the literature
    - wastes resources

> _Psychological Science_ uses StatCheck […] on manuscripts that are sent out for extended review

- Computational reproducibility is hard

---
layout: true
template: comp-rep

---

### Interim summary

Computational non-reproducibility
- is wide-spread
--

- undermines credibility of the literature
- wastes resources
- is unethical
--

- is not caused by simple sloppiness

---
exclude: true
layout: false
template: exercise

---
layout: true
template: comp-rep

---

- Typos
- Copy-paste errors
- Incorrect rounding
- "Outdated" results

__Friends don't<br />let friends__

<kbd style = "font-size: 28pt">Ctrl</kbd> + <kbd style = "font-size: 28pt">V</kbd>
]
]

???
Rounding: In reporting and calculations

Automation: Adopt seasoned tools from the computer sciences

---

- File paths
- Incomplete data
- Incomplete scripts
- Outdated libraries

- Automation
- Code review
- Archiving
]

---

# Dynamic documents

---
layout: true
template: footer

# Dynamic documents

---
exclude: true

Computational reproducibility

- requires precise knowledge of the analytic procedure
- is facilitated by access to data, analysis scripts, and _documentation_
    - Extra work

--
exclude: true

> Your closest collaborator is you six months ago, but you don’t reply to emails.

> &mdash; Paul Wilson, University of Wisconsin Madison

---
exclude: true

Computational reproducibility

- requires precise knowledge of the analytic procedure
- is facilitated by access to data, analysis scripts, and _documentation_
    - Extra work
    - Make it a by-product

---
exclude: true

A partial solution to the challenge of computational reproducibility

- Fuse computer code and documentation (Knuth, 1992)
    - Fully documented analysis protocols
    - Results are embedded directly into the document

---

<br />

.center[<small>
  [Allison Horst](https://github.com/allisonhorst/stats-illustrations)
  [(CC-BY-4.0)](https://creativecommons.org/licenses/by/4.0/)
</small>]
]

---
class: middle, center

# A quick demonstration!

---

---

<blockquote class="twitter-tweet" data-width="550" data-lang="en" data-dnt="true" data-theme="light"><p lang="en" dir="ltr">oh my god, now the journal no longer accepts .PDF files, so I have to find a way to blow this out to a .docx.</p>— Joe Hilgard, data guy (@JoeHilgard) <a href="https://twitter.com/JoeHilgard/status/616266797331972096?ref_src=twsrc%5Etfw">July 1, 2015</a></blockquote>

]

---

.center[
<blockquote class="twitter-tweet" data-width="550" data-lang="en" data-dnt="true" data-theme="light"><p lang="en" dir="ltr">Dreaming of a day when resubmitting an article to a different journal doesn't mean hair-pulling reformatting, especially when tables and figures go to the end with separate captions ... <a href="https://twitter.com/hashtag/academictimewasting?src=hash&ref_src=twsrc%5Etfw">#academictimewasting</a> 😱</p>— Alexa Morcom (@alexa_morcom) <a href="https://twitter.com/alexa_morcom/status/1162363829734510592?ref_src=twsrc%5Etfw">August 16, 2019</a></blockquote>

]

---
exclude: true

- There is a learning curve

- Same data, same code, same results
    - Prevents common causes of reproduction failures
- Facilitates sharing "replication packages" .grey[(Eubank, 2016)]

- _Cannot guarantee that reported results are correct!_

---
exclude: true

### [Error in one line of code sinks cancer study](http://retractionwatch.com/2016/09/26/error-in-one-line-of-code-sinks-2016-seer-cancer-study/)

> the authors discovered an error in one line of code in the computer program used to calculate the results […]. Specifically, the error assigned the code “unknown” to the additional variables […] the breast cancer death rate ratio […] was given in the original Table 3 as 1.32 (95% CI, 1.28 to 1.36), but in the corrected version it is now 0.89 (95% CI, 0.86 to 0.93) […].

---

### No guarantee that results are "correct"

[Data error prompts U-turn on study of sex differences in school](http://retractionwatch.com/2017/10/17/boys-will-boys-data-error-prompts-u-turn-study-sex-differences-school/)

> the 0/1 coding for "boy" and "girl" that we had on the paper questionnaires was opposite to the one in the SPSS labels

---

Various packages for different languages and outputs

- [`rmarkdown`](https://rmarkdown.rstudio.com/) / [`knitr`](https://yihui.name/knitr/) / [Quarto](https://quarto.org/)
- [StatTag](http://sites.northwestern.edu/stattag/)
- [`tidystats`](https://github.com/WillemSleegers/tidystats)
- [Jupyter](http://jupyter.org/)
- [`org-mode`](http://orgmode.org/)
- ...

---

.pull-right-45[<img src="data:image/png;base64,#img/rmarkdown_formats.png" width="564" style="display: block; margin: auto;" />]

`rmarkdown` / `knitr` / Quarto

- Widely used and actively developed
- Supports many output formats<br />(e.g., HTML, PDF, DOCX, ODT)
- Customizable and extendable

---

# [Let's get some exercise](http://frederikaust.com/papaja-workshop/exercises/1_introduction_question.html)!