ggplot2 – Plotting linear regression line with two numeric predictors in R

So my data consists of one numeric response variable and two numeric predictor variables.
It could be something like the sim4 dataset included in the “modelr” package (included in R Studio)

Now my lm is:
lm (y ~ x1 * x2)
To show the interaction I could use a 3D plot, however, I don’t like 3D plots. So I thought about creating one 2D scatterplot for each x and adding the regression line for my lm.

I would like to use something like the following, but with my lm instead of the y~x that is used here. Is that possible?

ggplot(sim4, aes(x1, y, colour = x2, group = x2)) + 
  geom_point() +
  geom_smooth(method = "lm",se=FALSE)

ggplot(sim4, aes(x2, y, colour = x1, group = x1)) +
  geom_point() +
  geom_smooth(method = "lm",se=FALSE)