Does it replicate?

class: inverse, title-slide

# Does it replicate?
## Using Bayesian model comparisons<br />to evaluate replication attempts of<br />specific 2×2-interaction patterns
### Frederik Aust, Tobias Heycke, & Christoph Stahl
### DGPs 2018, Frankfurt, Germany

---

layout: true

<div class="my-footer">
  <div style="float: left;"><span>Frederik Aust, Tobias Heycke, & Christoph Stahl</span></div>
  <div style="float: right;"><span>DGPs 2018, Frankfurt, Germany</span></div>
  <div style="text-align: center;"><span>Using Bayesian model comparisons to evaluate replication attempts of specific 2&times;2-interaction patterns</span></div>
</div>

---
# Attitude acquisition

<br />

.pull-left[
<center><img src="figures/bob.png" style="padding-bottom: 30px;"/></center>
]

.pull-right[
<br />

<center><h3>Laughed when a child fell out of a tree
</h3></center>
]

.footnote[Rydell, McConnell, Mackie, & Strain (2006)]

---
exclude: true

# Attitude acquisition

.pull-left[
<center><img src="figures/bob.png" style="padding-bottom: 30px;"/></center>
]

.pull-right[
<br /><br />

<br />

<center>Unconditioned stimulus (US)</center>
]

???
Change in liking following pairings with valent stimuli

---
# Attitude acquisition

<br />

.pull-left[
<center><img src="figures/bob.png" style="padding-bottom: 30px;"/></center>
]

.pull-right[
<br />

<center><h3>Laughed when a child fell out of a tree
</h3></center>

<center><h1 style="opacity:0.3;">Kiss</h1></center>
]

.footnote[Rydell, McConnell, Mackie, & Strain (2006)]

???
Automatic attitude without contingency awareness

---
# Attitude acquisition

### Two minds

<div id="htmlwidget-38553f18b358d26cb8b2" style="width:600px;height:150px;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-38553f18b358d26cb8b2">{"x":{"diagram":"\ndigraph dual_process {\n  graph [rankdir = LR, fontname = Helvetica, ranksep = 0.8, bgcolor = \"#EAE0C3\"]\n\n  node [shape = box, fontname = Helvetica, fontsize = 18, width = 2.2]\n  pairing2 [label = \"Briefly presented\nprimes\n😁\", margin = 0.2]\n\n  # Latent states\n  node [shape = oval, width = 2.5]\n  associative [label = \"Associative\nlearning\", style = filled, fillcolor = \"#385E49\", fontcolor = \"#EAE0C3\"]\n\n  # Outcomes\n  node [shape = \"box\", fontname = Helvetica, margin = \"0.25,0.125\", width = 2.4, height = 0.9]\n  liking2 [label = \"IAT score\n😁\"]\n\n  # Transitions\n  edge [arrowhead = \"normal\", fontname = Helvetica, fontsize = 16]\n\n  ## To latent states\n  pairing2 -> associative\n  \n  ## To observable outcomes\n  edge [label = \"\", xlabel = \"\"]\n  associative -> liking2\n\n  # Rank definition\n  {rank = same; pairing2}\n  {rank = same; associative}\n  {rank = same; liking2}\n}\n","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>

<div id="htmlwidget-5c0c40956da8fcdfd133" style="width:600px;height:150px;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-5c0c40956da8fcdfd133">{"x":{"diagram":"\ndigraph dual_process {\n  graph [rankdir = LR, fontname = Helvetica, ranksep = 0.8, bgcolor = \"#EAE0C3\"]\n\n  node [shape = box, fontname = Helvetica, fontsize = 18, width = 2.2]\n  pairing [label = \"Descriptions\nof behavior\n😢\", margin = 0.2]\n  awareness [label = \"US memory\"]\n\n  # Latent states\n  node [shape = oval, width = 2.5]\n  propositional [label = \"Propositional\nlearning\", style = filled, fillcolor = \"#2EA65A\"]\n\n  # Outcomes\n  node [shape = \"box\", fontname = Helvetica, margin = \"0.25,0.125\", width = 2.4, height = 0.9]\n  awareness [label = \"US memory\", width = 2.4]\n  liking [label = \"Attitude rating\n😢\"]\n\n  # Transitions\n  edge [arrowhead = \"normal\", fontname = Helvetica, fontsize = 16]\n\n  ## To latent states\n  pairing -> propositional\n  \n  ## To observable outcomes\n  edge [label = \"\", xlabel = \"\"]\n  propositional -> awareness\n  propositional -> liking\n\n  # Rank definition\n  {rank = same; pairing}\n  {rank = same; propositional}\n  {rank = same; liking; awareness}\n}\n","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>

</center>

.footnote[Rydell & McConnell (2006) and Rydell, McConnell, Mackie, & Strain (2006)]

---
# Attitude acquisition

### Two minds

.pull-left[

]

.pull-right[

]

.footnote[Rydell, McConnell, Mackie, & Strain (2006)]

---
# Attitude acquisition

### Two minds

.pull-left[

]

.pull-right[

]

.footnote[Rydell, McConnell, Mackie, & Strain (2006)]

---
# Attitude acquisition

### Two minds

.pull-left[

]

.pull-right[

]

.footnote[Rydell, McConnell, Mackie, & Strain (2006)]

---
# Attitude acquisition

### Two minds

.pull-left[

]

.pull-right[

]

.footnote[Rydell, McConnell, Mackie, & Strain (2006)]

---
exclude: true

# Attitude acquisition

### Two minds

.pull-left[

]

.pull-right[

]

.footnote[Notable example of subliminal attitude acquisition (e.g., Sweldens, Corneille, & Yzerbyt, 2014)]

---
exclude: true

# Attitude acquisition

### Two minds

.pull-left[

]

.pull-right[

]

.footnote[Notable example of subliminal attitude acquisition (e.g., Sweldens, Corneille, & Yzerbyt, 2014)]

---
exclude: true

# Attitude acquisition

### Two minds

.pull-left[

]

.pull-right[

]

.footnote[Notable example of subliminal attitude acquisition (e.g., Sweldens, Corneille, & Yzerbyt, 2014)]

???
Participants could not identify primes in memory test!

---
# Attitude acquisition

### Two minds?

.pull-left[

]

.pull-right[

]

.footnote[Heycke, Gehrmann, Haaf, & Stahl (2018)]

---
# Current study

- Adversarial collaboration
- As-close-as-possible re-replication
- Multiple labs
    - Cologne
    - Ghent
    - Harvard

.footnote[Heycke, Aust, Banaji, Conway, Van Dessel, Hu, Jiang, Kurdi, Rydell, Spitzer, Stahl, Vitiello, & De Houwer (in prep)]

---
# Evaluations of replication

- ANOVA and follow-up tests
--

- Replication Bayes factors
--

- Bayesian model comparison with order-constraints
    - Deviations from original protocol
    - Specific interaction patterns

.footnote[Rouder, Haaf, & Aust (2018) also see Hoijtink (2011); Harms (in press) and Verhagen & Wagenmakers (2014)]

???
ANOVA: Sig. vs. Non-Sig. interactions -> Difference significant?

Is the new Cohen’sdcloser to that of the original study or to 0

Also: prior predictive p-value (Hoijtink)

---
# Competing hypotheses

.pull-left[

<center>Hypotheses</center>

]

---
# Competing hypotheses

.pull-left[

<center>Hypotheses</center>

]

---
# Competing hypotheses

.pull-left[

<center>Hypotheses</center>

]

---
# Competing hypotheses

.pull-left[

<center>Hypotheses</center>

]

.pull-right[

<center>Predictions</center>

]

---
# Results

.pull-left[

<center>Observed effects</center>

]

.pull-right[

<center>Predictions</center>

]

???
Black-rimmed points represent means of observed attitude differences between blocks in which Bob was presented with positive descriptions and those in which he was paired with negative descriptions. Ellipses represent 95% Bayesian credible intervals based on the unconstrained model.

---
# Results

.pull-left[

<center>Model comparisons</center><br />

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Model </th>
   <th style="text-align:center;"> BF </th>
   <th style="text-align:center;"> Naive PP </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> No effect </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> .00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> One mind </td>
   <td style="text-align:center;"> 4.00 </td>
   <td style="text-align:center;"> .80 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Two minds </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> .00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Any effect </td>
   <td style="text-align:center;">  </td>
   <td style="text-align:center;"> .20 </td>
  </tr>
</tbody>
</table>

<br />
<center>BF<sub>One mind/Two minds</sub> >> 10.000</center>

]

.pull-right[

<center>Predictions</center>

]

---
# Results

.pull-left[

<center>Model comparisons</center><br />

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Model </th>
   <th style="text-align:center;"> BF </th>
   <th style="text-align:center;"> Naive PP </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> No effect </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> .00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> One mind </td>
   <td style="text-align:center;"> 4.00 </td>
   <td style="text-align:center;"> .25 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> &nbsp;&nbsp;... everywhere </td>
   <td style="text-align:center;"> 11.05 </td>
   <td style="text-align:center;"> .69 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Two minds </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> .00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> &nbsp;&nbsp;... everywhere </td>
   <td style="text-align:center;"> 0.00 </td>
   <td style="text-align:center;"> .00 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Any effect </td>
   <td style="text-align:center;">  </td>
   <td style="text-align:center;"> .06 </td>
  </tr>
</tbody>
</table>

<br />
<center>BF<sub>One mind everywhere/Two minds</sub> >>> 10.000</center>

]

.pull-right[

<center>Predictions</center>

]

---
# Conclusions

- We replicated Heycke, Gehrmann, Haaf, & Stahl (2018)
    - No evidence for second learning process
    - Further studies with lower prime visibility
--
- Bayesian model comparison with order-constraints enable targeted
    - evaluation of replication success
    - informative hypothesis testing more generally

---
# Acknowledgments

<small>
- Code used for visualization of predictions by [Julia Haaf](https://psychology.missouri.edu/people/haaf) ([@JuliaHaaf](https://twitter.com/juliahaaf))
- Title slide illustration [*Replicate This*](https://magoz.is/replicate-this/) &copy; 2013 [Magoz](https://magoz.is).
- Example images of Bob courtesy of Michael J. Tarr, Center for the Neural Basis of Cognition and Department of Psychology, Carnegie Mellon University, http://www.tarrlab.org/. Funding provided by NSF award 0339122 ([CC-BY-NC-SA 3.0](https://creativecommons.org/licenses/by-nc-sa/3.0/))
</small>

---
exclude: true

# Statistical model

.pull-left[

<center>Encompassing unconstrained linear model</center>

$$
`\begin{aligned}
\hat y_{ijk} = & \overbrace{\mu + \nu_i}^{\text{Random participant intercept}} + \overbrace{\eta_l}^{\text{Lab effects}} x_{1il}~+ \\
          & \dots \\
          & \phantom{(\overbrace{\alpha}^{\text{Rating effect}} + \overbrace{\tau_l}^{\text{Lab rating effects}} x_{1il}) x_{2j} x_{3k}~+}
\end{aligned}`
$$

]

---
exclude: true

# Statistical model

.pull-left[

<center>Encompassing unconstrained linear model</center>

$$
`\begin{aligned}
\hat y_{ijk} = & \overbrace{\mu + \nu_i}^{\text{Random participant intercept}} + \overbrace{\eta_l}^{\text{Lab effects}} x_{1il}~+ \\
          & (\overbrace{\alpha}^{\text{Rating effect}} + \overbrace{\tau_l}^{\text{Lab rating deviation}} x_{1il}) x_{2j} x_{3k}~+ \\
          & \dots
\end{aligned}`
$$

]

---
exclude: true

# Statistical model

.pull-left[

<center>Encompassing unconstrained linear model</center>

.footnote[

`$x_{2j}$` was dummy coded (1 = Rating, 0 = IAT score)<br />
`$x_{1il}$` and `$x_{3k}$` were effect coded

]

.pull-right[

<center>Predictions</center>

]

???
The model predicts the `$i$`th participant's response to measure `$j$` in the experimental block `$k$`.
Responses are predicted as a combination of a grand mean `$\mu$`, random participant intercepts `$\nu_i$` (i.e., habitually higher or lower attitudes), a main effect of the labs `$\eta_l$`, and simple effects of experimental block for attitude ratings ($\alpha$) and IAT score ($\beta$).
Additionally, we allowed the simple effects to be moderated by the labs ($\tau_l$ and `$\upsilon_l$` represent the lab-specific deviations from the overall simple effects).
The model does not include a main effect of attitude measure because any mean differences between attitude measures were leveled by the `$z$` standardization.
`$x_{1il}$` represents `$l$` effect coded variables that indicate which lab participant `$i$` belongs to; `$x_{2j}$` indicates the attitude measure (1 for explicit attitude rating and 0 for IAT score), such that `$\alpha + \tau_l$` is only relevant for attitude ratings and `$\beta + \upsilon_l$` is only relevant for IAT scores; `$x_{3k}$` is an effect coded variable that is set to 0.5 for block 1 and -0.5 for block 2.

---
exclude: true

# Priors and order-constraints

.pull-left[

$$
`\begin{aligned}
\mathcal{M}_\text{Any effect}:~ & \alpha \sim \mathcal{N}(0, g_\alpha \sigma^2) \\
  & \beta \sim \mathcal{N}(0, g_\beta \sigma^2) \\
\mathcal{M}_\text{One mind}:~ & \alpha \sim \mathcal{N}^+(0, g_\alpha \sigma^2) \\
  & \beta \sim \mathcal{N}^+(0, g_\beta \sigma^2) \\
\mathcal{M}_\text{Two minds}:~ & \alpha \sim \mathcal{N}^+(0, g_\alpha \sigma^2) \\
  & \beta \sim \mathcal{N}^-(0, g_\beta \sigma^2) \\
\mathcal{M}_\text{No effect}:~ & \alpha = 0 \\ & \beta = 0 \\
  & g_{\alpha}, g_{\beta} \sim \text{Inv-}\chi^2(1, r^2 = (\sqrt{2}/2)^2) \\
\implies & \delta_{\alpha}, \delta_{\beta} \sim \textrm{Cauchy}(0, r = \sqrt2/2) \\
\end{aligned}`
$$

]

.pull-rigth[

]

---
exclude: true

# Consistency across labs

Order constraint for each lab-specific effect

---
exclude: true

# Consistency across labs

Order constraint for each lab-specific effect

$$
`\begin{aligned}
\hat y_{ijk} = & \mu + \nu_i + \eta_l x_{1il}~+ \\
          & (\overbrace{\alpha}^{\text{Rating effects}} + \tau_l x_{1il}) x_{2j} x_{3k}~+ \\
          & (\overbrace{\beta}^{\text{Lab IAT effects}} + \upsilon_l x_{1il}) (1 -  x_{2j}) x_{3k} \\
 & \\
\hat y_{ijk} = & \mu + \nu_i + \eta_l x_{1il}~+ \\
          & (\overbrace{\alpha + \tau_l}^{\text{Lab rating effects}} x_{1il}) x_{2j} x_{3k}~+ \\
          & (\overbrace{\beta + \upsilon_l}^{\text{Lab IAT effects}} x_{1il}) (1 -  x_{2j}) x_{3k}
\end{aligned}`
$$

---
exclude: true

# Results

???
Black-rimmed points represent condition means, error bars represent 95% bootstrap confidence intervals based on 10,000 samples, small points represent individual participants’ responses, and violins represent kernel density estiamtes of smaple distributions.

---
exclude: true

# Results

---
exclude: true

# Results

---
exclude: true

# Theoretical accounts of evaluative conditioning

.pull-left[

<h3>Single learning process</h3>

<div id="htmlwidget-20e7315c18aad6b4f689" style="width:450px;height:175px;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-20e7315c18aad6b4f689">{"x":{"diagram":"\ndigraph dual_process {\n  graph [rankdir = LR, fontname = Helvetica, ranksep = 0.8, bgcolor = \"#EAE0C3\"]\n\n  node [shape = box, fontname = Helvetica, fontsize = 18]\n  pairing [label = \"CS-US\npairings\", margin = 0.2]\n\n  # Latent states\n  node [shape = oval, width = 2.5, margin = \"0.11,0.15\"]\n  propositional [label = \"Propositional\nlearning\", style = filled, fillcolor = \"#2EA65A\"]\n\n  # Outcomes\n  node [shape = \"box\", fontname = Helvetica, margin = \"0.25,0.125\", width = 2, height = 0.9]\n  liking [label = \"Attitude\"]\n  awareness [label = \"Contingency\nawareness\"]\n\n  # Transitions\n  edge [arrowhead = \"normal\", fontname = Helvetica, fontsize = 16]\n\n  ## To latent states\n  pairing -> propositional\n  \n  ## To observable outcomes\n  edge [label = \"\", xlabel = \"\"]\n  propositional -> awareness\n  propositional -> liking\n\n  # Rank definition\n  {rank = same; pairing}\n  {rank = same; propositional}\n  {rank = same; liking; awareness;}\n}\n","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>

</center>

]

.pull-right[

<h3>Dual learning process</h3>

<div id="htmlwidget-3a2f3f18469c53461288" style="width:450px;height:175px;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-3a2f3f18469c53461288">{"x":{"diagram":"\ndigraph dual_process {\n  graph [rankdir = LR, fontname = Helvetica, ranksep = 0.8, bgcolor = \"#EAE0C3\"]\n\n  node [shape = box, fontname = Helvetica, fontsize = 18]\n  pairing2 [label = \"CS-US\npairings\", margin = 0.2]\n\n  # Latent states\n  node [shape = oval, width = 2.5, margin = \"0.11,0.15\"]\n  associative2 [label = \"Associative\nlearning\", style = filled, fillcolor = \"#385E49\", fontcolor = \"#EAE0C3\"]\n  propositional2 [label = \"Propositional\nlearning\", style = filled, fillcolor = \"#2EA65A\"]\n\n  # Outcomes\n  node [shape = \"box\", fontname = Helvetica, margin = \"0.25,0.125\", width = 2, height = 0.9]\n  liking2 [label = \"Attitude\"]\n  awareness2 [label = \"Contingency\nawareness\"]\n\n  # Transitions\n  edge [arrowhead = \"normal\", fontname = Helvetica, fontsize = 16]\n\n  ## To latent states\n  pairing2 -> associative2\n  pairing2 -> propositional2\n  \n  ## To observable outcomes\n  edge [label = \"\", xlabel = \"\"]\n  associative2 -> liking2\n  propositional2 -> awareness2\n  propositional2 -> liking2\n\n  # Rank definition\n  {rank = same; pairing2}\n  {rank = same; associative2; propositional2}\n  {rank = same; liking2; awareness2}\n}\n","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>

<h3>"Two minds"</h3>

</center>

]

.footnote[(Rydell & McConnell, 2006)]

---
exclude: true

# Theoretical accounts of evaluative conditioning

.pull-left[

<h3>Single learning process</h3>

<div id="htmlwidget-8611e2804fcce11f3460" style="width:450px;height:175px;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-8611e2804fcce11f3460">{"x":{"diagram":"\ndigraph dual_process {\n  graph [rankdir = LR, fontname = Helvetica, ranksep = 0.8, bgcolor = \"#EAE0C3\"]\n\n  node [shape = box, fontname = Helvetica, fontsize = 18]\n  pairing [label = \"CS-US\npairings\", margin = 0.2]\n\n  # Latent states\n  node [shape = oval, width = 2.5, margin = \"0.11,0.15\"]\n  propositional [label = \"Propositional\nlearning\", style = filled, fillcolor = \"#2EA65A\"]\n\n  # Outcomes\n  node [shape = \"box\", fontname = Helvetica, margin = \"0.25,0.125\", width = 2, height = 0.9]\n  liking [label = \"Attitude\"]\n  awareness [label = \"Contingency\nawareness\"]\n\n  # Transitions\n  edge [arrowhead = \"normal\", fontname = Helvetica, fontsize = 16]\n\n  ## To latent states\n  pairing -> propositional\n  \n  ## To observable outcomes\n  edge [label = \"\", xlabel = \"\"]\n  propositional -> awareness\n  propositional -> liking\n\n  # Rank definition\n  {rank = same; pairing}\n  {rank = same; propositional}\n  {rank = same; liking; awareness;}\n}\n","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>

</center>

]

.pull-right[

<h3>Dual learning process</h3>

<div id="htmlwidget-1467e7dbf326b3c640a2" style="width:450px;height:175px;" class="grViz html-widget"></div>
<script type="application/json" data-for="htmlwidget-1467e7dbf326b3c640a2">{"x":{"diagram":"\ndigraph dual_process {\n  graph [rankdir = LR, fontname = Helvetica, ranksep = 0.8, bgcolor = \"#EAE0C3\"]\n\n  node [shape = box, fontname = Helvetica, fontsize = 18]\n  pairing2 [label = \"CS-US\npairings\", margin = 0.2]\n\n  # Latent states\n  node [shape = oval, width = 2.5, margin = \"0.11,0.15\"]\n  associative2 [label = \"Associative\nlearning\", style = filled, fillcolor = \"#385E49\", fontcolor = \"#EAE0C3\"]\n  propositional2 [label = \"Propositional\nlearning\", color = \"#7c7c7c\", fontcolor = \"#7c7c7c\", style = filled, fillcolor = \"#2EA65A50\"]\n\n  # Outcomes\n  node [shape = \"box\", fontname = Helvetica, margin = \"0.25,0.125\", width = 2, height = 0.9]\n  liking2 [label = \"Attitude\"]\n  awareness2 [label = \"Contingency\nawareness\"]\n\n  # Transitions\n  edge [arrowhead = \"normal\", fontname = Helvetica, fontsize = 16]\n\n  ## To latent states\n  pairing2 -> associative2\n  pairing2 -> propositional2 [color = \"#9b9b9b\", style = \"dashed\"]\n  \n  ## To observable outcomes\n  edge [label = \"\", xlabel = \"\"]\n  associative2 -> liking2\n  propositional2 -> awareness2 [color = \"#9b9b9b\", style = \"dashed\"]\n  propositional2 -> liking2 [color = \"#9b9b9b\", style = \"dashed\"]\n\n  # Rank definition\n  {rank = same; pairing2}\n  {rank = same; associative2; propositional2}\n  {rank = same; liking2; awareness2}\n}\n","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script>

<h3>"Two minds"</h3>

</center>

]

.footnote[(Lovibond & Shanks, 2002)]

???
Independent representations