Motion Controlled Animations with AnimateDiff & Motion LoRA's (A1111)

Welcome to the realm of cutting-edge animation control! In this guide, we delve into the dynamic fusion of Motion LoRA's and the AnimateDiff extension within Stable Diffusion (A1111). Get ready to elevate your animation game with unparalleled control, all while ensuring a watermark-free final masterpiece. Let's embark on a journey to unlock the full potential of these tools and redefine the possibilities in motion design.

Before we embark on the journey of crafting watermark-free, stunning animations, certain prerequisites must be in order. Presented below is a list of the necessary requirements to get started.

Requirement 1: AnimateDiff Extension & LCM LoRA's

The AnimateDiff extension takes the spotlight as our favored tool, streamlining the process of creating videos or GIFs. If you haven't yet installed the AnimateDiff extension and the accelerating LCM LoRA's for rendering, please consult a dedicated article linked below. It provides step-by-step instructions on downloading and installation:

Fast Video Generation with AnimateDiff & LCM LoRA's (A1111)

Requirement 2: Motion LoRA's & Domain Adapter LoRA

In essence, acquiring a Motion LoRA is essential for maintaining control over the motion in your final animation, be it a video or GIF. Additionally, a Domain Adapter LoRA is required to create an animation without any watermark.

Motion LoRA's Download

Acquire these Motion LoRA's directly from the Hugging Face website. All files come in Safetensor format, be sure to choose the Motion LoRA that aligns with your preferred motion style. Similar to standard LoRA's, relocate a downloaded file to the LoRa models folder located at: "stable-diffusion-webui > models > Lora."

Domain Adapter LoRA Download

Subsequently, download the Domain Adapter file identified as "mm_sd15_v3_adapter.safetensors" and be sure to place this LoRA also in the lora models directory: "stable-diffusion-webui > models > Lora".

Let's launch Stable Diffusion and proceed by navigating to the dropdown settings menu in the AnimateDiff extension, where we'll make the required adjustments. The specific settings are detailed below:

• Motion module: mm_sd_v15_v2.ckpt (Please be aware that the Motion LoRA's are compatible only with the v2 motion module.)
• Save format: MP4 (Moreover, you have the option to save it as a GIF as well)
• "Enable" AnimateDiff
• Number of frames: 16
• FPS (Frame Per Second): 8

You can leave the remaining settings in their default state. If you seek additional information about the AnimateDiff extension, feel free to visit the AnimateDiff GitHub page.

If all configurations are correctly adjusted in the AnimateDiff Menu, let's move on by scrolling to the top. To start, we'll choose our checkpoint. In our case, aiming for a realistic video animation, we'll opt for the "Realistic Vision" checkpoint. We also employ the LCM LoRA within the prompt to enhance and expedite the rendering process.

Txt2img Prompt Settings

• Checkpoint: Realistic Vision
• Prompt: 30 year old latina woman, cleavage, red lips, upper body, close up, golden necklace, purple dress, (tropical plants in background:1.1), elegant pose, front view, looking in the camera, <lora:LCM_SD1.5:1>, <lora:mm_sd15_v3_adapter:1>, <lora:mm_sd15_v2_lora_TiltDown:1>
• Negative Prompt: (worst quality:1.2), (low quality:1.2), (lowres:1.1), (monochrome:1.1), (greyscale), multiple views, comic, sketch, (((bad anatomy))), (((deformed))), (((disfigured))), watermark, multiple_views, mutation hands, mutation fingers, extra fingers, missing fingers, watermark, nude, nsfw

Three LoRA's Explained

• Orange color: represents the LCM LoRA for accelerated rendering.
• Pink color: signifies the domain adaptor LoRA, ensuring a watermark-free animation.
• Green color: denotes our Motion LoRA (specifically, the tilt-up Motion LoRA in our scenario).

Txt2Img Settings

Moving forward, let's navigate to the global settings in the "txt2img" tab, a crucial step in enhancing rendering speed when using the LCM LoRA. Here are the settings to accelerate your video generation:

• Sampling Method: LCM
• Sampling Steps: Set to 8 (Adjust between 2-8; a Sampling Steps of 8 yields the highest quality results).
• CFG Scale: Set to 2 (Select between 1-2; CFG Scale of 2 includes the negative prompt, while CFG Scale of 1 excludes it).
• Width & Height: Choose your preferred ratio; in our case, we'll generate a video and GIF with a 9:16 ratio.

After preparing your video, click "Generate," and see the Motion LoRA create a motion-controlled animation. When it's done, find your video in the "stable-diffusion-webui > outputs > txt2img-images > AnimateDiff" folder, complete with the date it was made.

For the purpose of this tutorial, we've utilized the "TiltUp" Motion LoRA. However, feel free to experiment with various Motion LoRAs to achieve different results. Particularly, trying out the "ZoomIn", ZoomOut", "PanLeft" and "PanRight"motion modules can yield incredible outcomes. So, please feel free to experiment with them. Below you will find our final result with the use different Motion LoRA's:

Congratulations on exploring the powerful combination of Motion LoRA's and the AnimateDiff extension in Stable Diffusion (A1111)! By following the steps outlined in this guide, you've learned how to achieve watermark-free, unparalleled animation control. Whether you're creating videos or GIFs, this fusion of tools opens up a realm of possibilities for elevating your motion design.