CartoonSegmentation

Implementations of the paper Instance-guided Cartoon Editing with a Large-scale Dataset, including an instance segmentation for cartoon/anime characters and some visual techniques built around it.

Preperation

Install Dependencies

Install Python 3.10 and pytorch:

conda create -n anime-seg python=3.10
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
conda activate anime-seg

Install mmdet:

pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0"
mim install mmdet
pip install -r requirements.txt

Download models

huggingface-cli lfs-enable-largefiles .
mkdir models
git clone https://huggingface.co/dreMaz/AnimeInstanceSegmentation models/AnimeInstanceSegmentation

Run Segmentation

See `run_segmentation.ipynb``.

Besides, we have prepared a simple Huggingface Space for you to test with the segmentation on the browser.

Copyright BANDAI NAMCO Entertainment Inc., We believe this is a fair use for research and educational purpose only.

Run 3d Kenburns

https://github.com/dmMaze/CartoonSegmentation/assets/51270320/503c87c3-39d7-40f8-88f9-3ead20e1e5c5

Install cupy following https://docs.cupy.dev/en/stable/install.html

Run

python run_kenburns.py --cfg configs/3dkenburns.yaml --input-img examples/kenburns_lion.png

or with the interactive interface:

python naive_interface.py --cfg configs/3dkenburns.yaml

and open http://localhost:8080 in your browser.

Please read configs/3dkenburns.yaml for more advanced settings.

To use Marigold as depth estimator, run

git submodule update --init --recursive

and set depth_est to marigold in configs/3dkenburns.yaml

Better Inpainting using Stable-diffusion

To get better inpainting results with Stable-diffusion, you need to install stable-diffusion-webui first, and download the tagger:

git clone https://huggingface.co/SmilingWolf/wd-v1-4-swinv2-tagger-v2 models/wd-v1-4-swinv2-tagger-v2

If you're on Windows, download compiled libs from https://github.com/AnimeIns/PyPatchMatch/releases/tag/v1.0 and save them to data/libs, otherwise, you need to compile patchmatch in order to run 3dkenburns or style editing:

Compile Patchmatch

mkdir -P data/libs
apt install build-essential libopencv-dev -y
git clone https://github.com/AnimeIns/PyPatchMatch && cd PyPatchMatch

mkdir release && cd release
cmake -DCMAKE_BUILD_TYPE=Release ..
make

cd ../..
mv PyPatchMatch/release/libpatchmatch_inpaint.so ./data/libs
rm -rf PyPatchMatch

If you have activated conda and encountered `GLIBCXX_3.4.30' not found or libpatchmatch_inpaint.so: cannot open shared object file: No such file or directory, follow the solution here https://askubuntu.com/a/1445330

Launch the stable-diffusion-webui with argument --api and set the base model to sd-v1-5-inpainting, modify inpaint_type: default to inpaint_type: ldm in configs/3dkenburns.yaml.

Finally, run 3dkenburns with pre-mentioned commands.

Run Style Editing

It also requires stable-diffusion-webui, patchmatch, and the danbooru tagger, so please follow the Run 3d Kenburns and download/install these first.
Download sd_xl_base_1.0_0.9vae, style lora and diffusers_xl_canny_mid and save them to corresponding directory in stable-diffusion-webui, launch stable-diffusion-webui with argument --argment and set sd_xl_base_1.0_0.9vae as base model, then run

python run_style.py --img_path examples/kenburns_lion.png --cfg configs/3d_pixar.yaml

set onebyone to False in configs/3d_pixar.yaml to disable instance-aware style editing.

Run Web UI (Including both 3D Ken Burns and Style Editing), based on Gradio

All required libraries and configurations have been included, now we just need to execute the Web UI from its Launcher:

python Web_UI/Launcher.py

In default configurations, you can find the Web UI here:

• http://localhost:1234 in local
• A random temporary public URL generated by Gradio, such like this: https://1ec9f82dc15633683e.gradio.live