Transform Videos into AI Animations with Stable Diffusion (mov2mov)

Updated
Transform Videos into AI Animations with Stable Diffusion (mov2mov)
Mov2mov Extension, an automated solution for video to video tasks. Follow the step-by-step guide to install the extension, adjust settings, and generate impressive animated videos with ease.

1. Introduction

Video-to-video tasks can often be time-consuming and require manual effort. However, with the Mov2mov extension, you can automate many of these steps and simplify the process. This blog post will guide you trough the Mov2mov extension to create impressive videos effortlessly. Whether you're a beginner or an experienced video enthusiast, this tutorial will help you achieve remarkable results.

2. Install Mov2mov extension in Stable Diffusion

The first step in using the Mov2mov extension is to install it. Within the AUTOMATIC1111 Web-UI:

  • Navigate to the Extension page and select the "Install from URL" tab.
  • Enter the following URL for the extension's git repository: https://github.com/Scholar01/sd-webui-mov2mov
  • Click "Install" and wait a few seconds until the extension is finished installing. video-to-video-with-mov2mov-extension-in-stable-diffusion-install-github-link.PNG
  • Now go to the "Installed" tab, click on check for updates and then on "Apply and restart UI" how-to-create-infinite-zoom-in-stable-diffusion-restart-ui.PNG
  • Once Installed, completely close and restart the Web-UI to ensure the extension is properly installed.
Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds
20% bonus on first deposit

3. Enter mov2mov settings (Video to Video)

After installing the extension, you'll find a new page called "mov2mov." On this page, select a Stable Diffusion checkpoint from the dropdown menu.

For example, you can choose the "revAnimated" model. Checkout this tutorial on how to install & use checkpoints/models:

How to Install Stable Diffusion Checkpoints & Models

Step1 - Enter a Prompt

Next, enter the desired prompt and negative prompt for your video. You can use a detailed description or specific keywords to guide the video generation process.

Step2 - Upload Video

Upload the video you wish to work with by dropping it onto the video canvas. Set the Resize mode to: "Crop and resize".

Step3 - Mov2mov Settings

When using the Mov2mov extension, here are some key settings to consider:

  • Sampling method: Keep in mind that deterministic samplers like Euler, LMS, and DPM++2M Karras might not work well with this extension, as they may not effectively reduce flickering.
  • Noise Multiplier: Utilize the slider to adjust the noise multiplier. For smoother results and reduced flickering, keep it at 0.
  • CFG Scale:: Control the extent to which the prompt is followed by adjusting the CFG scale. In the provided video, a scale of 7 was used.
  • Denoising Strength: Fine-tune the amount of change applied to the video by adjusting the denoising strength. A value of 0.6 was used in the example video.
  • Movie Frames: The frames per second of your output. The higher this value, the smoother your video, but this will have to render more images.
  • Max Frame: Determine the total number of frames to be generated. For initial testing, set it to a low number such as 10. To generate a full-length video, set it to -1.
  • Seed: The seed determines the value for the first frame. All frames will use the same seed value, even if you set it to -1 for a random seed. video-to-video-with-mov2mov-extension-in-stable-diffusion-settings.PNG By understanding and adjusting these settings, you can have greater control over the output and achieve desired results when utilizing the Mov2mov extension.

4. ControlNet settings (Video to Video)

If you haven't installed ControlNet yet, you can refer to our ControlNet installation guide.

"Enable" ControlNet and set the following options:

  • Pixel Perfect: Yes
  • Control Type: Lineart
  • Preprocessor: Lineart_realistic
  • Model: control_xxxx_lineart
  • Control weight: (0.6). video-to-video-with-mov2mov-extension-in-stable-diffusion-controlnet-settings.PNG

It's important to note that Mov2mov will use the current frame as the reference image, so you don't need to upload a separate reference image.

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

5. Generate video with mov2mov extension

With all the settings in place, it's time to generate the video. Click the "Generate" button to start the process. Be patient as it may take some time. Once the generation is complete, your new video will appear on the right side of the page.

Click "Save" to download and save the video to your device. If you can't locate the video, check the output/mov2mov-videos folder.

Example

6. Troubleshooting and Tips

In case of any errors during video generation, try different Video Modes to see if that resolves the issue. If the generation fails altogether, you can manually create a video from the image series found in the output/mov2mov-images folder.

Stable Diffusion in the Cloud⚡️Run Automatic1111 in your browser in under 90 seconds

7. Conclusion

In conclusion, the Mov2mov extension in Stable Diffusion is a game changer for video-to-video transformations. With its user-friendly interface and powerful capabilities, it enables effortless and stunning video creation. Embrace this innovative tool and take your video editing skills to the next level.