Video to Video Style Transfer With Diffex Desktop UI

In the previous tutorial, I demonstrated how to install DiffEx and leverage its capabilities to effortlessly transform text into captivating animations. Building upon that foundation, our focus now shifts to exploring the exciting realm of video-to-video transformations with DiffEx. Prepare to delve into a world where your videos undergo stunning metamorphosis, unlocking a new dimension of creativity and expression. Join me as we uncover the power of DiffEx in transforming videos with ease and precision.

To embark on this video-to-video journey with DiffEx, all you need is a starting video that serves as the foundation for your transformation. With this essential requirement met, we're prepared to unleash the full potential of DiffEx in bringing your vision to life.

For optimal results, ensure that if there's a face in your video, it remains visible and not too far from the camera throughout. This helps prevent glitches and artifacts in the final video.

Let’s boot up DiffEx and adjust our settings for the first method, Stylize.

Setup

• In the Setup menu go ahead and name your project.
• Select your preferred model, I will opt for Mistoon Anime, giving my video an anime styled look.
• Next, select the motion module and VAE if needed.
• For the scheduler I suggest euler_a or k_dpmpp_2m (DPM++ 2M Karras).
• Set the steps to a value between 15 and 25.
• Set CFG scale to 7.
• And set the FPS to 12.

Stylize

Go to the Stylize tab and press “Select Video”, a pop-up will appear telling you a new project will be started, press “OK”.

Now select your video. Make sure the FPS matches the FPS you set in the Setup menu and press “Process”. DiffEx will now automatically create prompts for each frame of the video using the WD14tagger, add the video to 2 ControlNet units and adjust the Execute settings.

Prompt

In the prompt tab you can review the prompts the WD14tagger has automatically created for you. It’s also advised to add your own "Head Prompt" and "Negative Prompt".

Advanced

In the advanced tab there are a few settings you might want to take a look at.

• Clip Skip: Set the Clip Skip to 2 for anime style videos and 1 for realistic videos.
• Save Input Images: You can turn this off to save space on your hard drive if you need to.
• Show Console: I recommend turning this on to view the progress in your console, this might require a restart.

Execute

In the execute tab everything should be set correctly already thanks to the Stylize processing. But if you want you can adjust the Width & Height. I lowered mine a little bit for faster generation, since I will upscale the video with A1111 later anyway. Now press “Build & Run” and sit back, relax while DiffEx does the rest.

My video had a total of 69 frames, using this method at 20 Steps and a 720x1280 resolution the generation only took 8 minutes to generate.

This feature is still in beta, the animation does not completely match our original video as you will see in the examples at the end. Let’s explore a more consistent way of video style transfer within DiffEx.

To more consistently stylize your video with DiffEx we are going to set up the ControlNet units ourselfs. This sounds tricky but I assure you it’s just as easy.

I suggest heading to the "Stylize" tab to process your video. This automatically sets most of the settings correctly for you.

Prompt

Go to the Prompt tab and disable the IP-Adapter on the right. Adjust the prompts if needed.

ControlNet

Now head to the ControlNet tab and disable the IP2P model. Scroll down in the list to find the “Tile” model and enable it, make sure you don’t select the “Use Preprocessor” option. Set the scale to somewhere between 0.8 & 1.25. Now you should only have the OpenPose and the Tile model selected.

Execute

Go to the execute tab and press on “Build & Run” to start generating your video! You can watch the progression in your console if you have it enabled in the advanced settings.

When your videos are done rendering they can be found in the “animatediff-cli-prompt-travel” folder. Open the folder, if you used the stylize function you can find your video in the “stylize” folder, even if you adjusted the ControlNet afterwards. If you haven’t used the stylize function, your videos can be found in the “output” folder.

To further enhance our video we’re going to upscale it by using Automatic1111’s Stable Diffusion interface. I have a whole dedicated tutorial on this if you’re interested, but for now I’ll show you quickly how it’s done.

How to Upscale any Video with AI for Free (Stable Diffusion)

• Open your A1111 and head to the “extras” tab.
• Go to Batch from Directory.
• Put the image sequence location of your video in the Input directory.
• Create an empty folder for the output directory and paste the location here.
• Set “Scale by” to 2.
• Choose your favorite upscaler and hit “Generate”.

This process will only take a minute.

Now that we have successfully upscaled our animation we are left with individual frames. You can stitch them together with any video software of your choice. Or even better, use FlowFrames to interpolate the frames and create a video automatically! Download FlowFrames and select all the default options when installing.

• For Interpolation AI select RIFE, if you have an Nvidia GPU select the Nvidia only version.
• For your Input Video, click "Browse (Folder)" and choose your upscaled Folder.
• Select a folder for the Output Directory, then paste the path here or leave it at the default path.
• When configuring Output FPS and Speed make sure you input the correct FPS, I recommend going with x2 or x3 and select Normal Speed.
• Leave the rest of the settings at their default values and press “Interpolate!” At the bottom left corner.

That only took 22 seconds to interpolate and it magically doubled our frames per second!

Now let's look at the video we just created. I’ll include both methods in the video so you can see what works best for yourself. At the end there’s a bonus clip using SoftEdge & Tile instead of OpenPose!

In summary, DiffEx is a versatile tool for transforming videos into captivating creations. With intuitive features and limitless potential, it empowers users to unleash their creativity effortlessly. DiffEx is the go-to solution for content creators seeking to push the boundaries of visual storytelling.