Video to Video Style Transfer With Diffex Desktop UI

Updated
Video to Video Style Transfer With Diffex Desktop UI
In this tutorial, we'll harness DiffEx for mesmerizing video-to-video animations. Get ready to elevate your content creation game!

1. Introduction

In the previous tutorial, I demonstrated how to install DiffEx and leverage its capabilities to effortlessly transform text into captivating animations. Building upon that foundation, our focus now shifts to exploring the exciting realm of video-to-video transformations with DiffEx. Prepare to delve into a world where your videos undergo stunning metamorphosis, unlocking a new dimension of creativity and expression. Join me as we uncover the power of DiffEx in transforming videos with ease and precision.

2. Requirements

To embark on this video-to-video journey with DiffEx, all you need is a starting video that serves as the foundation for your transformation. With this essential requirement met, we're prepared to unleash the full potential of DiffEx in bringing your vision to life.

For optimal results, ensure that if there's a face in your video, it remains visible and not too far from the camera throughout. This helps prevent glitches and artifacts in the final video.

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds
20% bonus on first deposit

3. Stylize Videos in DiffEx

Let’s boot up DiffEx and adjust our settings for the first method, Stylize.

Setup

  • In the Setup menu go ahead and name your project.
  • Select your preferred model, I will opt for Mistoon Anime, giving my video an anime styled look.
  • Next, select the motion module and VAE if needed.
  • For the scheduler I suggest euler_a or k_dpmpp_2m (DPM++ 2M Karras).
  • Set the steps to a value between 15 and 25.
  • Set CFG scale to 7.
  • And set the FPS to 12.

video-stylize-diffex-vid2vid-setup.webp

Stylize

Go to the Stylize tab and press “Select Video”, a pop-up will appear telling you a new project will be started, press “OK”.

Now select your video. Make sure the FPS matches the FPS you set in the Setup menu and press “Process”. DiffEx will now automatically create prompts for each frame of the video using the WD14tagger, add the video to 2 ControlNet units and adjust the Execute settings.

video-stylize-diffex-vid2vid-stylize.webp

Prompt

In the prompt tab you can review the prompts the WD14tagger has automatically created for you. It’s also advised to add your own "Head Prompt" and "Negative Prompt". video-stylize-diffex-vid2vid-prompt.webp

Advanced

In the advanced tab there are a few settings you might want to take a look at.

  • Clip Skip: Set the Clip Skip to 2 for anime style videos and 1 for realistic videos.
  • Save Input Images: You can turn this off to save space on your hard drive if you need to.
  • Show Console: I recommend turning this on to view the progress in your console, this might require a restart.

video-stylize-diffex-vid2vid-advanced.webp

Execute

In the execute tab everything should be set correctly already thanks to the Stylize processing. But if you want you can adjust the Width & Height. I lowered mine a little bit for faster generation, since I will upscale the video with A1111 later anyway. Now press “Build & Run” and sit back, relax while DiffEx does the rest. video-stylize-diffex-vid2vid-execute.webp

My video had a total of 69 frames, using this method at 20 Steps and a 720x1280 resolution the generation only took 8 minutes to generate.

This feature is still in beta, the animation does not completely match our original video as you will see in the examples at the end. Let’s explore a more consistent way of video style transfer within DiffEx.

4. Consistent Stylize Video with DiffEx

To more consistently stylize your video with DiffEx we are going to set up the ControlNet units ourselfs. This sounds tricky but I assure you it’s just as easy.

I suggest heading to the "Stylize" tab to process your video. This automatically sets most of the settings correctly for you.

Prompt

Go to the Prompt tab and disable the IP-Adapter on the right. Adjust the prompts if needed. video-stylize-diffex-vid2vid-tile-prompt.webp

ControlNet

Now head to the ControlNet tab and disable the IP2P model. Scroll down in the list to find the “Tile” model and enable it, make sure you don’t select the “Use Preprocessor” option. Set the scale to somewhere between 0.8 & 1.25. Now you should only have the OpenPose and the Tile model selected. video-stylize-diffex-vid2vid-controlnet-tile.webp

Execute

Go to the execute tab and press on “Build & Run” to start generating your video! You can watch the progression in your console if you have it enabled in the advanced settings.

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

5. Locate DiffEx videos

When your videos are done rendering they can be found in the “animatediff-cli-prompt-travel” folder. Open the folder, if you used the stylize function you can find your video in the “stylize” folder, even if you adjusted the ControlNet afterwards. If you haven’t used the stylize function, your videos can be found in the “output” folder. video-stylize-diffex-vid2vid-a1111-output-location.webp

6. Upscale your DiffEx animation

To further enhance our video we’re going to upscale it by using Automatic1111’s Stable Diffusion interface. I have a whole dedicated tutorial on this if you’re interested, but for now I’ll show you quickly how it’s done.

How to Upscale any Video with AI for Free (Stable Diffusion)

  • Open your A1111 and head to the “extras” tab.
  • Go to Batch from Directory.
  • Put the image sequence location of your video in the Input directory.
  • Create an empty folder for the output directory and paste the location here.
  • Set “Scale by” to 2.
  • Choose your favorite upscaler and hit “Generate”.

video-stylize-diffex-vid2vid-a1111-upscale-animation.webp This process will only take a minute.

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

7. Interpolate with FlowFrames

Now that we have successfully upscaled our animation we are left with individual frames. You can stitch them together with any video software of your choice. Or even better, use FlowFrames to interpolate the frames and create a video automatically! Download FlowFrames and select all the default options when installing.

  • For Interpolation AI select RIFE, if you have an Nvidia GPU select the Nvidia only version.
  • For your Input Video, click "Browse (Folder)" and choose your upscaled Folder.
  • Select a folder for the Output Directory, then paste the path here or leave it at the default path.
  • When configuring Output FPS and Speed make sure you input the correct FPS, I recommend going with x2 or x3 and select Normal Speed.
  • Leave the rest of the settings at their default values and press “Interpolate!” At the bottom left corner.

video-stylize-diffex-vid2vid-a1111-upscale-interpolate-flowframes.webp

That only took 22 seconds to interpolate and it magically doubled our frames per second!

8. Examples DiffEx Videos

Now let's look at the video we just created. I’ll include both methods in the video so you can see what works best for yourself. At the end there’s a bonus clip using SoftEdge & Tile instead of OpenPose!

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

9. Conclusion DiffEx vid2vid

In summary, DiffEx is a versatile tool for transforming videos into captivating creations. With intuitive features and limitless potential, it empowers users to unleash their creativity effortlessly. DiffEx is the go-to solution for content creators seeking to push the boundaries of visual storytelling.

Frequently Asked Questions

DiffEx is a user-friendly Desktop UI designed for AnimateDiff, streamlining the animation creation process with its intuitive features.

DiffEx simplifies animation creation by automating tasks like importing videos and images, offering multi ControlNet support, and integrating models and motion modules seamlessly.

Yes, DiffEx excels at video-to-video transformations, offering features like Stylize mode and ControlNet setup for consistent stylization and enhancement of videos.

Absolutely! DiffEx's user-friendly interface and intuitive features make it accessible for beginners while offering advanced capabilities for seasoned professionals.

Yes, DiffEx is versatile and supports both text-based and video-based animations. Whether you want to transform text prompts into animations or stylize existing videos, DiffEx has you covered.

With DiffEx, the possibilities are endless. You can create various types of videos, including animated shorts, promotional videos, educational content, and more. Let your creativity soar!