ggplot2 – Editing or adding stat-summary lines to individual panels in facet-wrap r-studio

I would like some help please with a question about facet_wrap in R

Here is some example data

library(tidyverse)

score <- c(6,5,3,2,2,1,4,5,3,4,5,4,3,2,2,2,4,6)
time <- c(1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2,1,2)
set <- c(1,1,1,1,1,1,1,1,2,2,3,3,3,3,3,3,3,3)
ID <- c(1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9)

df <- data.frame(score, time, set, ID)

plot <- ggplot(df, aes(x = time, y = score, group = ID)) +
  geom_point(size = .5) +
  geom_line(linetype="dashed") +
  stat_summary(aes(group = 1, linetype = "median"), geom="line", fun = "median", size = .5) + labs(linetype = "") +
  facet_wrap("set", ncol = 1) 
plot

And the resulting plot is:

I want to remove the median line from Set 2 and just retain a dashed line.

Can anyone help?

I have tried adding a subset to the stat_summary code but get the error:
mapping must be created by aes()
I have tried adding the stat_summary after the facet wrap and specifying the group but I get the error:
Aesthetics must be either length 1 or the same as the data
Thank you in advance 🙂