How To Style Your Face Portrait with Stable Diffusion (ControlNet & IP-Adapter)

Updated
How To Style Your Face Portrait with Stable Diffusion (ControlNet & IP-Adapter)
Discover the art of face portrait styling with this step-by-step guide on using Stable Diffusion, ControlNet, and IP-Adapter. Learn how to transform your face portraits and express your unique style effortlessly.

1. Introduction Face Portrait Styling in Stable Diffusion

Welcome to the world of Face Portrait Styling! This guide is your passport to mastering the nuances of Stable Diffusion, ControlNet, and IP-Adapter technologies, transforming your face portraits into personalized works of art. Whether you want your photos to look really polished or just want to make your creative selfies stand out, this guide will make the whole process clear and easy. It combines innovation and style effortlessly, promising a straightforward journey into the world of face portrait styling. Let's dive in.

2. Requirements & Downloads (ControlNet & IP-Adapter Models)

Before diving into the exciting realm of face portrait styling, let's ensure you have the essentials ready for this creative journey. Here are the key requirements to make the most of ControlNet and IP-Adapter technology:

ControlNet

To get started, it's essential to have the ControlNet Extension installed, which will serve as our backbone. Haven't installed the ControlNet Extension?

How to Install ControlNet Extension in Stable Diffusion (A1111)

IP-Adapter Face Model

To transfer and manipulate your facial features effectively, you'll need a dedicated IP-Adapter model specifically designed for faces. This model will be instrumental in accurately reflecting your facial expressions and features. For this you can visit the Hugging Face IP-Adapter Models page. Here you should download the following safetensors file:

how_to_style_your_face_portrait_with_stable_diffusion_controlnet_ip_adapter_face_ip_adapter_nextdiffusion.webp

Note: Please be aware that the safetensors file requires the latest ControlNet Version to function correctly. Ensure you "Check for updates" and restart the UI to update the ControlNet Extension to the most recent version.

File Location IP-Adapter

Once you downloaded the "ip-adapter-plus-face_sd15.safetensors" you need to do the following to make it ready for use:

  • Place the file inside the following folder: "stable-diffusion-webui\extensions\sd-webui-controlnet\models"

how_to_style_your_face_portrait_with_stable_diffusion_controlnet_ip_adapter_face_ip_adapter_file_location_nextdiffusion.webp

Having these essentials in place sets the stage for a seamless and creative face portrait styling experience, unlocking the full potential of ControlNet and IP-Adapter technologies.

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds
20% bonus on first deposit

3. Verifying the ControlNet Extension & IP Face Adapter Model

Now that the ControlNet extension and the IP Face Adapter Model are successfully installed, let's proceed to verify their visibility within the Stable Diffusion web UI. Navigate to either the "txt2img" or "img2img" tab, and as you scroll down, you should observe the expansion panel for the ControlNet extension.

ControlNet Extension & IP Face Adapter Model

Upon opening this panel and selecting the "IP-Adapter" Control Type, ensure that the "ip-adapter-plus-face_sd15" model is available for selection.

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-controlnet-face-model-nextdiffusion.webp

Troubleshooting

If, for any reason, you do not find this model in the options you can try two things:

  • Click the blue refresh icon on the right side to update the models.
  • If the model is still not visible, double-check the placement of the "ip-adapter-plus-face_sd15" file to ensure it is in the correct folder mentioned in section 2.

When everything looks right on the user interface, we can start creating our first styled face portrait. Let's give it a try!

4. Creating a Styled Face Portrait (ControlNet Settings & Examples)

In this section, we delve into the exciting process of creating a styled face portrait using the ControlNet canvas and the IP-Adapter technology. Let's do the following:

ControlNet Settings

  • Begin by placing a Face Portrait of your preference onto the ControlNet Unit 0 canvas.
  • Enable ControlNet by checking the "Enable" checkbox
  • Enable "Pixel Perfect"
  • Choose for the "IP-Adapter" as the "Control Type"
  • Select the "ip-adapter-plus-face_sd15" model.
  • Leave the rest of the ControlNet settings as the default

In our initial example, we'll feature the beloved Angelina Jolie as the stunning face model.

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-controlnet-settings-nextdiffusion.webp

To enhance outcomes, create a prompt mirroring a traditional studio face portrait. In this initial attempt, we'll employ the ToonYou checkpoint for a charming cartoon-style effect.

Styled Face Portrait Examples

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-angelina-jolie-example-nextdiffusion.webp

Face Portrait Prompt & Settings

  • Positive Prompt: best quality, beautiful, 30 year old woman, (full face portrait:1.2), face portrait, detailed face, freckles, (facing camera), (looking into the camera), studio portrait, natural lightning, perfect shadows
  • Negative Prompt: (worst quality:1.2), (low quality:1.2), (lowres:1.1), (monochrome:1.1), (greyscale), multiple views, comic, sketch, (((bad anatomy))), (((deformed))), (((disfigured))), mutation hands, mutation fingers, extra fingers, missing fingers, watermark
  • Sampling Steps: 30
  • Sampler: DPM++ 2M Karras
  • CFG scale: 7
  • Size: 512x512
  • Model: ToonYou

Adjusting Prompt for Styling

Feel free to customize the prompt according to your preferences and make minor tweaks to the resulting face portrait. For instance, if I wish for Angelina Jolie to have red hair and blue eyes, I can include relevant keywords in the prompt to achieve the desired outcome, as demonstrated below:

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-angelina-joli-redhair-nextdiffusion.webp

  • Adjusted Positive Prompt: best quality, beautiful, 30 year old woman, (red hair:1.2), (ginger hair:1.2), (blue eyes:1.2), (full face portrait:1.2), face portrait, detailed face, freckles, (facing camera), (looking into the camera), studio portrait, natural lightning, perfect shadows

Upscaling Styled Face Portrait (Hires. fix Settings)

If the outcome doesn't meet your expectations, you have the option to enhance your image using the "Hires. fix" feature. Typically, your selected checkpoint provides specific recommendations for upscaling settings. For the ToonYou Checkpoint, it suggests the following settings when upscaling your image:

  • Upscaler: R-ESRGAN 4x+ Anime6B
  • Hires Step: 14
  • Denoising Strength: 0.35
  • Upscale factor: 1.5+

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-controlnet-upscale-settings-hire-fix-nextdiffusion.webp

Utilizing the upscaler enhances the image quality, providing superior results. Now that our initial styled face portrait with the ToonYou checkpoint is complete, feel free to experiment with various checkpoints for unique styles! Let's delve in and explore some of our favorite checkpoints using an XYZ plot.

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

5. Comparing Face Portrait Styles (XYZ Plot)

In this section, we embark on a visual exploration by comparing various face portrait styles through the use of an XYZ plot. Haven't heard about the XYZ plotting technique within Stable Diffusion? Here's a simple guide:

XYZ Plotting to Compare Stable Diffusion Checkpoints

XYZ Plotting Script

This XYZ plotting method serves as a swift and efficient tool for assessing different checkpoints and their transformative impact on our initial face portrait. So let's do some plotting now. First click the X/Y/Z plot script under the "Script" expansion panel to open the options:

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-controlnet-xyz-plotting-technique-nextdiffusion.webp

Once you have selected the X/Y/Z plot, a panel will appear where you can customize and fill in specific settings according to your desired comparison parameters. Now it is time for our comparison.

XYZ Settings & Comparison (Checkpoints vs Sampling Method)

Our comparison will be different checkpoints/models compared to different Sampling methods. To achieve this we will need the following settings:

  • set X type to "Checkpoint name"
  • at the X values select the checkpoints you want to compare
  • set Y type to "Sampler"
  • at the Y values select the sampling methods you want to compare

Below you can find our filled in settings for our XYZ plot.

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-controlnet-xyz-plotting-settings-nextdiffusion.webp

You have the flexibility to include additional checkpoints and sampling methods. Moreover, there's the option to introduce another "Z type" to compare additional parameters. However, for the purpose of this tutorial, we'll keep it as outlined above.

Comparing Face Portrait Styles

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-controlnet-xyz-plot-example-nextdiffusion.webp

Now, when crafting a new face portrait, you can quickly identify your preferred style at a glance. Personally, I'll stick to the ToonYou Checkpoint as I appreciate the cartoonish style it reflects.

6. Styled Face Portrait Examples

Here are a few instances of styled face portraits created with the ToonYou Checkpoint, employing the settings and prompts detailed at the beginning of this article.

Jennifer Lawrence

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-jennifer-lawrence-fix-nextdiffusion.webp

Ana de Armas

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-ana-de-armas-nextdiffusion.webp

Sofia Vergara

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-sofia-vergara-nextdiffusion.webp

Emma Stone

how-to-style-your-face-portrait-with-stable-diffusion-controlnet-ip-adapter-emma-stone-nextdiffusion.webp

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

7. Conclusion

In summary, we've delved into the realm of face portrait styling using the specialized IP-Adapter for faces within the ControlNet Extension. From setup to experimentation with various checkpoints, we've explored the creative possibilities at the intersection of technology and personal expression. The XYZ plot simplified comparisons, providing a quick overview of diverse styles. With the flexibility to adjust prompts, this tutorial empowers artistic self-expression, transforming face portraits into personalized visual statements.

Frequently Asked Questions

The IP Face Adapter distinguishes itself by focusing on the intricate details of facial features, ensuring a nuanced and realistic transformation in face portrait styling. Its compatibility with Stable Diffusion and ControlNet guarantees stability and control throughout the process. Choosing the IP-Adapter means opting for a sophisticated tool that empowers users to infuse creativity into their face portraits.

The fundamental requirements include the installation of the ControlNet Extension and the IP-Adapter model, specifically the face model for transferring facial features. It's crucial to ensure the visibility of these components within the Stable Diffusion web UI.