How to Use Open Pose & ControlNet in Stable Diffusion

In the realm of character animation, the ability to transfer poses seamlessly between different characters is a valuable asset. Traditionally, achieving this involved complex 3D software and time-consuming processes. However, with the advent of OpenPose and its integration with Stable Diffusion, a revolutionary solution has emerged. In this article, we delve into the remarkable capabilities of OpenPose and how it synergizes with Stable Diffusion, opening up new possibilities for character animation.

To transfer the pose of a character using ControlNet and the Open Pose Editor extension in Stable Diffusion, follow these steps:

Step 1: Install ControlNet Extension

• Launch Stable Diffusion
• Click on "Extensions" in the menu.
• Select "Install from URL.
• Paste Controlnet Github URL: https://github.com/Mikubill/sd-webui-controlnet
• Click on "Install" to add the extension to your workspace.

• To ensure successful installation, go to the "Installed" section and check for updates.
• After confirming the installation, click on "Apply and restart UI" to apply the changes and restart.

Step 2: Install OpenPose Editor Extension

• Launch Stable Diffusion
• Click on "Extensions" in the menu.
• Select "Install from URL.
• Paste OpenPose Github URL: https://github.com/fkunn1326/openpose-editor
• Click on "Install" to add the extension to your Stable Diffusion UI.

• To ensure successful installation, go to the "Installed" section and check for updates.
• After confirming the installation, click on "Apply and restart UI" to apply the changes and restart.

Prior to utilizing the blend of OpenPose and ControlNet, it is necessary to set up the ControlNet Models, specifically focusing on the OpenPose model installation. For this you can follow the steps below:

• Go to ControlNet Models
• Download all ControlNet model files (filenames ending with .pth)

Or just download the OpenPose model "control_v11p_sd15_openpose.pth"

• Put the model file(s) in the ControlNet extension’s models directory:

Now that you have the OpenPose Model & Open Pose Editor Extension installed, you can easily edit and pose stick figures. Within the OpenPose Editor tab, you'll find a dedicated box where all the manual editing takes place. The extension mimics the shapes and colors of the Open Pose model, ensuring consistency and ease of use. Adjust the stick figure's pose as desired within the designated box.

Step 1: Create Pose & Send Pose to ControlNet

With the newfound capability, you can now click on the button "Send to txt2img" below. By clicking this button the stick figure pose will be seamlessly transmitted to ControlNet.

Step 2: Enable ControlNet Settings

• To enable ControlNet, simply check the checkboxes for "Enable" and "Pixel Perfect" (If you have 4GB of VRAM you can also check the "Low VRAM" checkbox).
• Select "OpenPose" as the Control Type
• Select "None" as the Preprocessor (This is because the image has already been processed by the OpenPose Editor).
• Select "Control_v11p_sd15_openpose" as the Model

Step 3: Create Prompt and Generate Image

To create stunning images, you must select the checkpoint and carefully craft a prompt for the character. This allows you to bring your artistic vision to life. With everything set, a single click on the "Generate" button unleashes the transformative power of ControlNet and the OpenPose Editor, turning your scene into a dynamic masterpiece.

• Positive Prompt: ((detailed)), ((masterpiece)), ((best quality)), ((Astounding)), ((Enchanting)), (Striking), woman, (lara croft:1.2), (tomb raider:1.2), (tomb:1.1), natural light inside tomb, beach, beach background, sunny, jungle, plants in background, beach background, beach, tropical beach, water, (detailed face:1.2), clear skin, (cleavage:1.1), perfect light, perfect shadows
• Negative Prompt: (worst quality:1.2), (low quality:1.2), (lowres:1.1), (monochrome:1.1), (greyscale), multiple views, comic, sketch, (((bad anatomy))), (((deformed))), (((disfigured))), watermark, multiple_views, mutation hands, mutation fingers, extra fingers, missing fingers, watermark
• Sampling Steps: 30
• Sampler: DDIM
• CFG scale: 7
• Size: 912x512 (16:9 ratio)
• Model: reV Animated

Presenting the Dynamic Poses Package, a collection of poses meticulously crafted for seamless integration with both ControlNet and the OpenPose Editor. This package offers an array of expressive poses that can effortlessly be employed, enhancing the visual appeal and interactivity of your projects.

Included in Downloads

The following is included when you download the poses:

• Preview of Pose
• JSON file of Pose
• Stick figure of Pose

Additionally there is a "presets.json" file with all poses in JSON format.

How to use?

To use with ControlNet & OpenPose:

• Drag & Drop the Stick Figure Poses into a ControlNet Unit and do the following:
• To enable ControlNet, simply check the checkboxes for "Enable" along with "Pixel Perfect".
• Select "OpenPose" as the Control Type
• Select "None" as the Preprocessor (Since the stick figure poses are already processed)
• Select "Control_v11p_sd15_openpose" as the Model

To use with OpenPose Editor:

For this purpose I created the "presets.json" file, which can be found in the downloaded zip file. You can place this file in the root directory of the "openpose-editor" folder within the extensions directory:

The OpenPose Editor Extension will load all of the Dynamic Pose Presets from the "presets.json" file. You can then choose a pose from the Presets dropdown menu, and by clicking "Load Preset", you'll be able to preview the selected pose. From there you can send it to either "txt2img" or "img2img".

If you are still unsure how all of this works, please checkout the video above.

If you wish to have multiple characters in the same scene, the Open Pose Editor extension has you covered. Simply click on the "Add" button to add a new skeleton. Once added, you can pose and adjust it to your liking. Repeat this process to introduce additional characters into the scene, each with their unique poses. If you are happy with the poses you can again send it to ControlNet via the "Send to txt2img" button below the stick figure canvas.

Before clicking on "Generate" in the txt2img tool, be sure to steps below:

• To enable ControlNet, simply check the checkboxes for "Enable" and "Pixel Perfect" (If you have 4GB of VRAM you can also check the "Low VRAM" checkbox).
• Select "OpenPose" as the Control Type
• Select "None" as the Preprocessor (This is because the image has already been processed by the OpenPose Editor).
• Select "Control_v11p_sd15_openpose" as the Model

And of course write a good prompt! Once done, simply wait for the magic to unfold as the generated image appears before your eyes.

Examples of Multiple Characters in a Scene

• Positive Prompt: ((detailed)), ((masterpiece)), ((best quality)), ((Astounding)), ((Enchanting)), (Striking), woman, lara croft twin, twin women, twins, (lara croft:1.2), (tomb raider:1.2), (tomb:1.1), natural light inside tomb, beach, beach background, sunny, jungle, plants in background, beach background, beach, tropical beach, water, (detailed face:1.2), clear skin, (cleavage:1.1), perfect light, perfect shadows
• Negative Prompt: (worst quality:1.2), (low quality:1.2), (lowres:1.1), (monochrome:1.1), (greyscale), multiple views, comic, sketch, (((bad anatomy))), (((deformed))), (((disfigured))), watermark, multiple_views, mutation hands, mutation fingers, extra fingers, missing fingers, watermark
• Sampling Steps: 30
• Sampler: DDIM
• CFG scale: 7
• Size: 912x512 (16:9 ratio)
• Model: reV Animated

With the Open Pose Editor extension in Stable Diffusion, transferring poses between characters has become a breeze. You can achieve this without the need for complex 3D software. By following the steps outlined in this blog post, you can easily edit and pose stick figures, generate multiple characters in a scene, and unleash your creativity. Have fun exploring the possibilities and enjoy your newfound ability to transfer poses seamlessly. Happy animating!