How To Use IP-Adapter Models for Image Prompting (A1111)

Updated
How To Use IP-Adapter Models for Image Prompting (A1111)
Elevate your visuals! Dive into the world of image prompts with Stable Diffusion and ControlNet. Master composition, style, faces and colors in our detailed tutorial.

1. Introduction

Image prompting enables you to incorporate an image alongside a prompt, shaping the resulting image's composition, style, color palette or even faces. This guide will walk you through the process of employing image prompts within the Stable Diffusion interface alongside ControlNet and its Image Prompt Adapter (IP-Adapter model).

2. Requirements For Image Prompts

To start off let's make sure we have all the required extensions and models to begin. Here’s what you need.

Stable Diffusion

This tutorial employs the widely-used and free Stable Diffusion WebUI. Compatible with Windows, Mac, and Google Colab, it offers versatile usage. Check out our Stable Diffusion Installation Guide for windows if you haven’t already.

ControlNet

You will also need ControlNet installed, and updated to its latest version. Check out our ControlNet Installation Guide for a detailed explanation on how to do that.

How to Install ControlNet Extension in Stable Diffusion (A1111)

IP-Adapter Models

Lastly you will need the IP-adapter models for ControlNet which are available on Huggingface.co There are a few different models you can choose from.

For this tutorial we will be using the SD15 models. I recommend downloading these 4 models:

  • ip-adapter_sd15.safetensors - Standard image prompt adapter
  • ip-adapter-plus_sd15.safetensors - Plus image prompt adapter
  • ip-adapter-full-face_sd15 - Standard face image prompt adapter
  • ip-adapter-plus-face_sd15.safetensors - Plus face image prompt adapter

After downloading the models, move them to your ControlNet models folder. use-images-as-prompts-in-stable-diffusion-models-fodler.webp

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds
20% bonus on first deposit

3. What is Image Prompting in Stable Diffusion?

Relying solely on a text prompt for generating desired images can be challenging due to the complexity of prompt engineering. An alternative approach involves using an image prompt, supporting the notion that "an image is worth a thousand words."

You have the option to integrate image prompting into stable diffusion by employing ControlNet and choosing the recently downloaded IP-adapter models. The image prompt can be applied across various techniques, including txt2img, img2img, inpainting, and more.

Lets Introducing the IP-Adapter, an efficient and lightweight adapter designed to enable image prompt capability for pretrained text-to-image diffusion models. For more information check out the comparison for yourself on the IP-adapter GitHub page. For now, let's initiate the use of image prompting with the IP-Adapter models.

4. Using IP-adapter (txt2img)

Initially, I will produce an image without utilizing an image prompt within the txt2img tab in Stable Diffusion. Subsequently, I will introduce the image prompt to observe the contrast. Let's examine the initial generated image using only a positive prompt:

use-images-as-prompts-in-stable-diffusion-example-no-ip-adapter.webp

  • Checkpoint: Photon_v1
  • Prompt: a medium shot of a 25yo european woman standing in a bust street, dawn, bokeh background, <lora:add_detail:1>
  • Sampling Method: Euler a
  • Sampling Steps: 30
  • Width & Height: 768x768
  • CFG Scale: 7

IP-Adapter (ip-adapter_sd15)

Now, let's begin incorporating the first IP-Adapter model (ip-adapter_sd15) and explore how it can be utilized to implement image prompting. Open ControlNet, import an image of your choice (woman sitting on motorcycle), and activate ControlNet by checking the enable checkbox.

use-images-as-prompts-in-stable-diffusion-controlnet-settings.webp

  • Control Type: IP-Adapter
  • Model: ip-adapter_sd15

Take a look at a comparison with different Control Weight values using the standard IP-Adapter model (ip-adapter_sd15). Notice how the original image undergoes a more pronounced transformation into the image prompt as the control weight is increased.

use-images-as-prompts-in-stable-diffusion-example-ip-adapter.webp

IP-Adapter plus (ip-adapter-plus_sd15)

The plus model is quite powerful, typically reproducing the image prompt very accurately, but it struggles on faces with a higher Control Weight in my testing. I recommend using a low Control Weight value when using the IP-adapter plus model.

Take a look at the results below, using the same settings as before.

use-images-as-prompts-in-stable-diffusion-example-ip-adapter-plus.webp

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

5. Face Swap with IP-Adapter (txt2img)

Just as before we will generate an image of a face without using an image prompt, then later we will use ip-adapter-full-face & ip-adapter-plus-face models to see the difference. Below you can find the initial image we generated without employing image prompting:

ip-adapter-face-no-adapter.webp

  • Checkpoint: Realistic Vision V5.1
  • Prompt: medium shot of a woman with blonde hair, 8k uhd, dslr, soft lighting, high quality,
  • Sampling Method: Euler a
  • Sampling Steps: 30
  • Width & Height: 768x768
  • CFG Scale: 7

IP-Adapter Full Face (ip-adapter-full-face)

Next, we'll delve into the IP-Adapter designed for faces, operating similarly to previous models but necessitating an image containing a face. Let's incorporate a face image (Jutta Leerdam) into ControlNet to observe the distinctions.

  • open up ControlNet and drag in any image of your choice.
  • Enable ControlNet by clicking the "enable" checkbox.
  • Enable Pixel Perfect
  • Control Type select IP-Adapter
  • Model: ip-adapter-full-face

use-images-as-prompts-in-stable-diffusion-controlnet-settings-ip-adapter-face.webp

Examine a comparison at different Control Weight values for the IP-Adapter full face model. Notice how the original image undergoes a more pronounced transformation into the image just uploaded in ControlNet as the control weight is increased.

use-images-as-prompts-in-stable-diffusion-example-ip-adapter-full-face.webp

IP-Adapter Plus Face (ip-adapter-plus-face)

For even more facial resemblance you can select the ip-adapter-plus-face model, as it is even more powerful. I recommend this one as the outcome looks a little better. Just see the results for yourself, I used the same generation settings as before.

use-images-as-prompts-in-stable-diffusion-example-ip-adapter-plus-face.webp

6. Using IP-Adapter for Color Palette (txt2img)

We can use the IP-adapted to set a color palette in our generated images. To do this and show the difference with and without the IP-Adapter model I will first generate an image with ControlNet disabled. I used the following settings to generate an image.

  • Checkpoint: Photon_v1
  • Prompt: photograph of a woman face, <lora:add_detail:0.7>
  • Sampling Method: Euler a
  • Sampling Steps: 20
  • Width & Height: 768x768
  • CFG Scale: 7

ip-adapter-stable-diffusion-automatic1111-color-palette-off.webp

Now enable ControlNet with the standard IP-Adapter model and upload a colorful image of your choice and adjust the following settings.

  • Control Type: IP-Adapter
  • Preprocessor: ip-adapter_clip_sd15
  • Model: ip-adapter_sd15
  • Control Weight: 0,75 (Adjust to your liking)

use-images-as-prompts-in-stable-diffusion-controlnet-settings-ip-adapter-color-palette.webp

Now press generate and watch how your image comes to life with these vibrant colors! Just look at the examples below. use-images-as-prompts-in-stable-diffusion-example-ip-adapter-color-palette.webp

For the last example I also set the Ending Control Step to 0,7. I recommend experimenting with these settings to get the best result possible.

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

7. Conclusion

In conclusion, you've successfully navigated the diverse capabilities of Stable Diffusion, ControlNet, and the powerful IP-Adapter models. This tutorial has equipped you to master composition, style, and even facial features in your generated images. Importantly, it's essential to note that the IP-Adapter models offer even more possibilities beyond what we covered here.

For a deeper exploration of the IP-Adapter's potential, including advanced functionalities not covered in this tutorial, I encourage you to visit the IP-Adapter GitHub page. There, you'll find comprehensive information and resources to enhance your understanding and creativity.

Congratulations on completing the tutorial, and best of luck on your continued exploration of image generation possibilities with the IP-Adapter models!

Frequently Asked Questions

IP-Adapter models, integrated with ControlNet, allow users to incorporate image prompts seamlessly. They provide additional control over composition, style, and even facial features, enhancing the creative possibilities of image generation.

Yes, Stable Diffusion and its associated tools, including ControlNet and IP-Adapter, are compatible with various operating systems, including Windows, Mac, and Google Colab.

Certainly! The techniques demonstrated in the tutorial are designed for practical application. Feel free to apply these methods to your own projects, experiment with different settings, and unleash your creativity in image generation.