How to Create Consistent AI Influencers in Stable Diffusion (A1111)

The realm of AI influencers is experiencing a surge in popularity, with a growing demand for authenticity and uniqueness. In this article, we delve into the fascinating world of crafting realistic AI influencers. We will delve into the cutting-edge developments, specifically focusing on the groundbreaking Multi-inputs feature from ControlNet. This technology enables us to effortlessly combine multiple faces, facilitated by the assistance of IP-Adapters. Join us on this journey as we uncover the techniques behind creating unique and authentic-looking AI influencers that captivate audiences with their lifelike appearances.

Before diving into the creation of breathtakingly realistic & consistent AI influencers, it's essential to install the ControlNet Extension along with its corresponding models, including various IP-Adapter models to test our creations. Let's break down the necessary requirements, shall we?

Stable Diffusion (A1111)

In this tutorial, we utilize the popular and free Stable Diffusion WebUI. It is compatible with Windows, Mac, and Google Colab, providing versatility in usage. Make sure to explore our Stable Diffusion Installation Guide for Windows if you haven't done so already.

ControlNet Extension

Also ensure that the ControlNet Extension is installed and updated to its latest version so that the Multi-inputs option appears in the ControlNet Extension's UI. Refer to our ControlNet Installation Guide for a comprehensive explanation of the installation process.

How to Install ControlNet Extension in Stable Diffusion (A1111)

IP-Adapter Models

To complete the setup, you'll need the IP-Adapter models for ControlNet. There are several models available, and for this tutorial, we'll specifically utilize the SD15 models. To ensure diverse options for your AI influencer creations, I recommend downloading the following four models:

For this tutorial we will be using the SD15 models. I recommend downloading these 4 models so we can create some comparisons and make sure you have different options: when creating your AI Influencer Creations.

• ip-adapter_sd15.safetensors - Standard image prompt adapter
• ip-adapter-plus_sd15.safetensors - Plus image prompt adapter
• ip-adapter-plus-face_sd15.safetensors - Plus face image prompt adapter
• ip-adapter-full-face_sd15 - Standard face image prompt adapter

After downloading the models, move them to your ControlNet models folder:

OpenPose Model (Optional)

Do you wish to place your consistent AI influencer in various poses? Yes, you heard it right—any pose! You might also consider downloading the OpenPose Model, a tool we'll use later in this tutorial to achieve just that, altering the pose of our AI influencer. Follow these straightforward steps to get prepared:

• Head over to the ControlNet Models Page
• Install the "control_v11p_sd15_openpose_fp16.safetensors" file
• Make sure to place the file inside the "\stable-diffusion-webui\extensions\sd-webui-controlnet\models" folder

All set! With the necessary downloads and preparations complete, let's dive into the exciting part. Now, open up Stable Diffusion and let the creative journey begin!

In this section, we'll use Stable Diffusion to merge faces and create a unique and one of a kind AI Influencer. Before we begin, make sure you have some amazing face portraits in mind that you want to merge into a unique AI influencer. If you don't have any ready, feel free to grab some from the internet. They could be pictures of your favorite actresses, singers, or anyone you like. Here's what we'll do:

ControlNet (Multi-Inputs) Settings

• Open the ControlNet Expansion Panel.
• Go to the "Multi-Inputs" section within ControlNet Unit 0.
• Click on "Upload Images" to upload multiple images from a specific folder.
• Enable the ControlNet Extension by checking the checkbox.
• Turn on "Pixel Perfect" for accurate results.
• Select the "IP-Adapter" as the Control Type
• For the preprocessor make sure you select the "ip-adapter_clip_sd15"
• As for this example we will select "ip-adapter-plus_sd15" as our model

You can keep the remaining settings in their default state for now. In the following section, we'll explore the various IP-Adapters we downloaded earlier. Utilizing different IP-Adapters to merge your face portraits into a unique AI influencer can significantly influence the style of your merge. However, for the time being, let's proceed to craft our initial AI influencer appearance using the face portraits we've uploaded.

Checkpoint & Prompt

Now, let's navigate to the top of the "txt2img" tab and select our checkpoint. Since we're aiming for a realistic outcome for our AI influencer, let's select the "Realistic Vision" checkpoint. Write a prompt, then click the Generate button. In this example, we'll also utilize the LCM LoRA to significantly enhance our image generation speed. Unsure how to use the LCM LoRA & adjust the settings accordingly? Check out our guide for detailed instructions:

How to Speed Up Image Generation with LCM LoRA's (A1111)

After clicking the generate button and producing several AI influencers, here are some of the outcomes utilizing the three face portraits from our example. Feel free to experiment with various face portraits to craft a truly unique AI influencer that remarkably maintains consistency across multiple generations.

• Positive Prompt: 30 year old woman, white silk dress, red lips, (cleavage:1.3), (upper body:1.3), (freckles:1.1), looking in camera, front view, natural lighting, perfect shadows, tropical plants in background, <lora:LCM_SD1.5:1>
• Negative Prompt: (worst quality:1.2), (low quality:1.2), (lowres:1.1), (monochrome:1.1), (greyscale), multiple views, comic, sketch, (((bad anatomy))), (((deformed))), (((disfigured))), mutation hands, mutation fingers, extra fingers, missing fingers, watermark, multiple
• Sampling Steps: 8
• Sampler: LCM Method
• CFG scale: 2
• Size: 400x712
• Model: Realistic Vision

Wow look at that! Observe the consistency across the generations of AI influencers achieved through the utilization of the Multi-Inputs feature alongside the chosen IP-Adapter model. This remarkable outcome resembles an "instant LoRA" achieved simply by uploading different face portraits. Now, let's proceed by conducting quick tests of the IP-Adapters we've downloaded, shall we?

In the requirements section, we got four different IP-Adapter Models, each working as image prompt adapters, but with different methods. We can sort them into either "Image Prompt Adapters" or "Face Image Prompt Adapters":

Image Prompt Adapters

• ip-adapter_sd15.safetensors - Standard image prompt adapter
• ip-adapter-plus_sd15.safetensors - Plus image prompt adapter

Face Image Prompt Adapters

• ip-adapter-full-face_sd15 - Standard face image prompt adapter
• ip-adapter-plus-face_sd15.safetensors - Plus face image prompt adapter

Testing Different IP-Adapter Models

Now, it's time to conduct some tests to determine which IP-Adapter is most suitable for AI influencer creation. Let's scroll down to the "Script" dropdown menu and select the "X/Y/Z Plotting" script. Here, we will test the different IP-Adapter Models against various Control Weights. Let's make sure to test agains those different parameters by filling in the specific X & Y Fields:

• For the X Type let's select: [ControlNet] Weight
• For the X Values let's input: 0.5,1,1.5
• For the Y Type let's select: [ControlNet] Model
• For the Y Values let's input: ip-adapter_sd15 & **ip-adapter-plus_sd15 **

These settings will test the two "Image Prompt Adapters" described above! If you want to test all the IP-Adapter Models at once, make sure to include all four IP-Adapter Models in the Y Values input field.

Image Prompt Adapters: Results XY Plot

Face Image Prompt Adapters: Results XY Plot

Happy we could conduct some tests! As it turns out, most of the "Face Image Prompt Adapters" look rather terrifying, so we'll definitely stick with the "Image Prompt Adapters" as we move on to the next section. In the upcoming segment, we'll employ another ControlNet Unit to implement some Pose Changes. For this, we'll use the OpenPose model to gracefully place our AI influencer in a new pose—let's strike a pose and keep going!

Welcome to the next stage of our exploration, where we venture into the intriguing domain of altering poses for our AI influencer. In this section, we unveil another ControlNet Unit. Here, we'll leverage the capabilities of the OpenPose Control Type to smoothly maneuver and position our AI influencer into different poses. If you've previously engaged in XYZ Plotting tests, ensure to deactivate the script by selecting "None" from the Script dropdown menu. This will disable the XYZ Plotting Script.

AI Influencer Poses Package

To help you change your AI influencer's poses, we've put together an AI Influencer Poses Package for you to download on CivitAI.

This will simplify the process of adjusting your influencer's pose quickly using the premade pose images. Let's ensure you download the AI Influencer Poses Package and extract the files from that folder.

ControlNet Unit 1 Settings (OpenPose)

Now, let's open up a new ControlNet Unit and proceed with the following steps:

• Drag & Drop the Stick Figure Poses (for example 01-pose.png) into ControlNet Unit 1 and do the following:
• To enable ControlNet, simply check the checkboxes for "Enable" along with "Pixel Perfect".
• Select "OpenPose" as the Control Type
• Select "None" as the Preprocessor (Since the stick figure poses are already processed)
• Select "Control_v11p_sd15_openpose" as the Model

Now, let's click the "Generate" Button and witness how effortlessly you can transform your AI influencer into one of the poses from the package we just downloaded. For your convenience, I've created a brief demo video showcasing how the poses from the package might appear.

Our journey through crafting AI influencers has been both enlightening and rewarding. From exploring hyper-realistic techniques to experimenting with various IP-Adapters and ControlNet Units, we've uncovered the intricacies of creating unique and authentic virtual personalities. By merging faces, testing different adapters, and mastering pose adjustments, we've gained valuable insights into the art of AI influencer creation. Armed with Stable Diffusion and ControlNet tools, along with the invaluable resources like the AI Influencer Poses Package, we're equipped to unleash our creativity and shape AI influencers with unparalleled realism. Cheers to the exciting journey ahead!