Transform Videos into any Style with Deforum & Stable Diffusion

Updated
Transform Videos into any Style with Deforum & Stable Diffusion
Ready to transform your videos into various styles? Discover the power of Stable Diffusion and Deforum.

1. Introduction

Embark on a captivating journey with our comprehensive step-by-step tutorial, as we expertly guide you through the effortlessly smooth process of installing Deforum, skillfully configuring prompts and settings, and achieving a truly remarkable transformation of your videos, all awaiting your exploration. The moment has arrived for you to take the plunge into this exciting endeavor!

2. Installations

Before we get started, make sure you have Deforum and ControlNet installed. If you haven't installed the Stable Diffusion and ControlNet yet, you can follow our comprehensive guides.

How to make Amazing AI Videos with Deforum (Stable Diffusion)

How to Install ControlNet Extension in Stable Diffusion (A1111)

Make sure you also download the tile and the openpose models for controlnet, this can be found on the huggingface controlnet models website.

how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-tile.webp

how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-openpose.webp

Place the files in following directory: stable-diffusion-webui\extensions\sd-webui-controlnet\models For this tutorial I will be using the Rev Animated checkpoint.

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds
20% bonus on first deposit

3. Adjust Initial Noise Multiplier

The initial noise multiplier parameter increases the amount of noise resulting in more detailed images. But in the case of Deforum we don't want a lot of detail to avoid flickering. By default the minimum value is 0.5. We are going to change it so we are able to set this to 0.

  • Go to stable-diffusion-webui\modules and search for shared.py
  • Right click shared.py and click edit
  • Hit CTRL + F and look for initial_noise make sure “Wrap around” is selected. If you cant find it you can go to line 420 manually.
  • Now change the minimum from 0.5 to 0.0
  • Save the file and close it

how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-initial-noise-multiplier.webp

To access the initial noise multiplier easily in Stable Diffusion you want to enable it in the Quicksettings list in the user interface. To do this, head to the Settings tab and look for “User interface” and scroll down a little bit until you find the Quicksettings list. Here you want to type “initial_noise_multiplier” and press enter. Apply the settings and reload the UI.

how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-quicksetting-initial-noise-multiplier.webp

4. Image to Image Settings

Get started with creating prompts and settings for your videos. First we need an image or frame from the video we want to transform. There are multiple ways of doing this but I suggest using an editing software like Premiere Pro or DaVinci resolve. Alternatively you can make a screenshot of the video with the snipping tool built in to windows. Once you have the Image we can start with our prompt.

Open the Image to Image tab. Set your prompt, negative prompt, and load your image. It’s crucial that you use short prompts, I suggest staying under 75 characters for the Positive prompt. For the negative prompt I advise to use the negative embedding; EasyNegative.

  • Sampler: Euler a
  • Steps: 20
  • Width & Height: Set this to the same dimensions as your input image (or select resize by 1) If you have a large input image it’s advisable to lower the output image. Keep the max dimensions to 1024 for example 576 by 1024 for a 9:16 image
  • CFG Scale: 3 to 5
  • Denoising Strength: 1

how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-img2img-settings.webp

Make sure the initial noise multiplier is set to 0. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-noise-multiplier.webp

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

5. ControlNet Settings

We will be utilizing 2 ControlNet units. For the first ControlNet unit use these settings:

  • Pixel Perfect: Yes
  • Control Type: Tile
  • Preprocessor: None
  • Model: control_v11f1e_sd15_tile
  • Control Weight: 1.5 to 1.75
  • Control Mode: ControlNet is more important

how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-controlnet-tile.webp

For the second ControlNet Unit follow these settings:

  • Pixel Perfect: No
  • Control Type: OpenPose
  • Preprocessor: openpose
  • Model: control_v11p_sd15_openpose
  • Control Weight: 1
  • Control Mode: ControlNet is more important

how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-controlnet-openpose.webp

Make sure both ControlNet units are enabled and hit generate!

You now have the option to modify certain settings until you achieve an output that meets your preferences. Feel free to make changes such as adjusting the prompts, modifying the CFG Scale, and altering the Control Weight of the Tile ControlNet unit. Once you're content with the output, it's time to transfer all these settings to Deforum!

how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-img2img-before-after.webp

6. Enhance Styles with LoRA

To achieve your desired style, utilizing the power of LoRA's is key. If you're unfamiliar with LoRA's, I highly recommend checking out our guide.

How to Install & Use LoRA Models in Stable Diffusion

With LoRA's, you can easily transform your original video into different styles, like this One Piece style I found on CivitAI. Just take a look at the before and after! how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-img2img-before-after-lora.webp

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

7. Deforum Settings

At first glance the Deforum tab looks a bit overwhelming, but we only need to make a few adjustments, after that we can save these settings and load them easily with a few clicks! I will run you trough all the tabs that we will be using starting from left to right. If you don't want to do all of this manually you can load in a settings file that I have created for you, you can download this on mega.nz.

Run

  • Sampler: Euler A
  • Steps: 20
  • Width & Height: Same as video input (576 by 1024 in my case)
  • Seed: -1
  • Batch Name: Name your folder so you can easily find it back in the img2img folder

how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-run.webp

Keyframes

  • Animation Mode: 3D
  • Border Mode: Wrap
  • Cadence: 2 or 1 (for videos with a lot of movement I suggest 1)

how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-keyframes-01.webp

Within the Keyframes tab, we encounter several additional sub-tabs where we need to adjust some settings.

Strength

Set the strength schedule from 0.65 to 0. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-strength-schedule.webp

CFG

Set the CFG scale schedule from 7 to about 3 to 5. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-cfg.webp

Seed

Set the seed behavior to “fixed”, this is to keep consistency in the video. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-seed.webp

Now we move on to the tabs below.

Motion

Set translation Z to 0 how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-motion.webp

Noise

For the noise schedule I advise staying between 0 and 0.03, usually 0 works the best. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-noise.webp

Coherence

Here we change the Color Coherence to None, if you want to keep the original colors from the video, you can change this to Video input. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-coherence.webp

Anti Blur

Change the Amount Schedule to 0.05 how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-anti-blur.webp

Depth Warping & FOV

If you’re using an NVIDIA graphics card you don’t need to change anything here, but if you are on an AMD you want to disable “Use depth warping”.

That was it for the Keyframes tab, now we move back to the main tabs.

Prompts

Paste your prompt here that you used from the img2img tab, make sure to use the right format: { "0": "Your prompt here" } Don’t forget to paste your negative prompts in the Negative Prompt section. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-prompts.webp

Init

We will change 2 sub-tabs in the Init tab, here is where we select our video.

Image Init

Set the Strength to 0. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-image-init.webp

Video Init

Here you want to paste the path to your original video, you can copy the path of your video by right clicking and press copy as path (On windows 10 use Shift+right click). Make sure to remove the double quotes at the start and end of the path. Then, enable “overwrite extracted frames”. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-video-init.webp

ControlNet

Here we will copy over all of our settings that we used in the img2img ControlNet units. The layout is a little bit different but there’s nothing to worry about.

Enable the first ControlNet unit and put the same settings we used before.

  • Pixel Perfect: Yes
  • Preprocessor: None
  • Model: control_v11f1e_sd15_tile
  • Control Weight: 1.5 to 1.75
  • Control Mode: ControlNet is more important how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-deforum-controlnet-tile.webp

Now enable the second ControlNet Unit and copy over those settings as well.

  • Pixel Perfect: No
  • Preprocessor: openpose
  • Model: control_v11p_sd15_openpose
  • Control Weight: 1
  • Control Mode: ControlNet is more important how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-deforum-controlnet-openpose.webp

Now for both ControlNet units it’s important to copy the original video path into the “ControlNet Input Video/Image path”, simply copy the path from the “init” tab and paste it in both ControlNet units.

Hybrid Video

  • Hybrid Settings: Before Motion
  • Generate inputframes: Yes
  • Hybrid Motion: Optical Flow
  • Flow Method: DIS Medium

Then open Hybrid Schedules beneath the settings we just adjusted and set “Comp alpha schedule” to 1. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-hybrid-video.webp

Output

Set the frames per second to match the frames per second from your original video. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-output.webp

8. Save the Deforum Settings

To make your life a whole lot easier we can save these settings to a text file and load them in next time we start up stable diffusion. Simply give your settings file a name and click on save settings. This will save a text file to your stable diffusion webui directory. To load these settings next time you start up Stable Diffusion you can copy the path of the settings file and paste it under Settings File and press “Load All Settings”. how-to-transform-videos-into-any-style-with-deforum-and-stable-diffusion-save-settings.webp

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

9. Generate Deforum Video

Now, let's generate the magic! Click the Generate Button and witness Deforum in action, transforming your original video into an incredible masterpiece!

Now it’s time to upscale your video, I highly recommend TopazLab’s AI video upscaler for this task. Topaz also has a built in feature that allows you to interpolate frames for an even smoother video!

If you're getting some unexpected errors please head to the FAQ section down below.

10. Conclusion

In conclusion, this step-by-step guide has equipped you with the tools and knowledge to unleash your creative potential through the transformative capabilities of Stable Diffusion and Deforum. From installations to advanced settings, you've learned how to seamlessly merge technology and creativity to craft mesmerizing videos that captivate and inspire. As you embark on your video editing journey, remember that the fusion of Deforum's cinematic magic and ControlNet's innovative technique opens up new dimensions of storytelling. With your newfound expertise, you're ready to dive into the world of creating stunning videos that leave a lasting impression on your audience. So go ahead, hit that Generate Button, and watch as Deforum works its magic, turning your original videos into captivating masterpieces that showcase your unique vision and creativity.

Frequently Asked Questions

Absolutely! This guide is designed to be user-friendly, making it accessible to both budding creators and seasoned professionals. The step-by-step instructions break down complex concepts, ensuring that even beginners can create stunning videos with Deforum and Stable Diffusion.

Some extensions cause issues between Deforum and ControlNet, I advise you to disable other extensions that you don't need during this process. You can do this by going to the extensions tab, unchecking the extensions you don't need and then press Apply and restart UI.

ControlNet is a groundbreaking technique that enhances video transformations by providing precise control over various aspects of the process. This guide shows you how to leverage ControlNet units, such as Tile and OpenPose, to achieve desired effects and seamlessly merge technology and creativity.

Verify the accuracy of the path / URL leading to the video within the "Init" > "Video Init" > "Video init path/ URL" tab. Furthermore, ensure that both ControlNets, CN Model 1 and CN Model 2, have the accurate video path entered in the "ControlNet Input Video/ Image Path" field.

Example of video path: "C:\Users\Desktop\video\video1.mp4"