How to Change Backgrounds with Stable Diffusion (Inpaint Anything)

Changing backgrounds has never been easier or more exciting, especially if you lack advanced Photoshop skills. In this tutorial, we'll walk you through the process of seamlessly turning one image into another using Stable Diffusion. We'll explore two powerful techniques: in-painting and the awesome inpaint-anything extension. Additionally, we'll delve into fine-tuning to address small background details. Let's dive into the magical world of Automatic1111.

To begin, select a Stable Diffusion checkpoint. For this tutorial, we recommend choosing an in-painting version. I'll opt for "ReV Animated inpainting v1.2.1."

Selecting the In-Painting Tab

Navigate to the "Image to Image" tab and select the "In-Painting" tab.

Drag and drop your starting image

Drag and drop your image into the tab. I'll be using this image as my starting image.

Creating the Mask

Now, the crucial step is creating a mask for what you want to change or retain. In my case, I'll only mask the person in this image, because that is the only thing I want to retain and it's easier to just mask out the subject rather than masking the whole background.

Writing the Prompt

Compose a prompt for the image. For this example, let's use "explosions" to make the picture more engaging. You can add a generic negative prompt to balance it.

Setting Parameters

Adjust the parameters as follows:

• Mask blur: I recommend 4 or higher, see what is best for you.
• Mask mode: Inpaint not masked
• Sampling method: DPM++ 2M SDE Karras
• Sampling steps: Use a minimum of 25, but higher is better.
• Width & Height: Use the appropriate dimensions (e.g., 768x512 for landscape).
• Denoising strength: 1

Keep other parameters as default.

Generating a New Background

Generate a new background, and voila! You've transformed your image into something captivating. Clearly, this approach has its imperfections; in my situation, it results in a minor yet noticeable border around my character. This is where the second technique becomes essential.

Now, let's explore another way to change backgrounds using inpaint-anything, an incredible extension for Stable Diffusion.

Install Inpaint-Anything

If you don't have the Inpaint-Anything extension, you can install it from extensions tab.

• Click on "Extensions"
• Click on "Available"
• Click on "Load from"
• Search for "Inpaint Anything"
• Click on "Install"

Following the installation, make sure to restart the UI.

Select and download a Model

After installing the extension and restarting the UI head to the “Inpaint Anything” tab and select a segment Model. I'll use “sam_vit_l_0b3195.pth” but feel free to try out any model. Press “Download model” after selecting your Segment Anything Model, when download is complete we can move on to the next step.

Drag and Drop the Image

Drag and drop your image. I’ll be using the same image as before to have a good comparison.

Inpaint Anything makes masking a breeze. After you’ve dropped your image click on “Run Segment Anything”, this should take a few moments.

Mark each colored segment by simply clicking on the color to create a mask. Be precise, and avoid selecting areas that shouldn't be masked, I recommend zooming in a bit to avoid mistakes. Regularly check the mask's progress to ensure accuracy, every few clicks press on “Create Mask” to check your progress. If you masked a wrong area by accident you can revert this by clicking on the back button in the segmented image.

Now you should have a very detailed mask. However I do recommend clicking on “Expand mask region” once or twice to expand the mask by 1 or 2 pixels.

Using the Mask

On the left side select the “Mask only” tab and click on “Get mask”. Then right under the mask hit “Send to img2img inpaint”.

Now your highly detailed mask is in the img2img Inpaint upload tab. From here we can follow the same steps as before and compare our results.

Write your prompts, select the correct mask mode and dimensions. Adjust the sampling method and sampling steps and set the denoising strength to 1. I will use the same prompt as before to compare our results.

Inpaint-Anything offers more precision by allowing detailed segmentation of the image, reducing errors and delivering better results, as demonstrated in the comparison below.

You've just unlocked the power of Stable Diffusion for background transformations. Whether you're turning a simple picture into an action movie scene or fixing imperfections in your images, Stable Diffusion empowers you to create stunning visuals. Experiment, refine, and elevate your image editing game.