Stable-Diffusion-Unity-Integration
A basic but functional Unity Editor Component for image generation using Stable Diffusion Automatic 1111 webui from within the Unity Editor.
If you like it, feel free to Buy me a Coffee (https://www.buymeacoffee.com/dobrado76)
Cheers!
Simple server configuration with StableDiffusionConfiguration component and SDSettings ScriptableObject assets.
Easy Text-to-Image generation for 3D models:
Straightforward Text-to-Image generation for UI elements:
This Unity Editor tool requires access to a Stable Diffusion Automatic 1111 server. Installation or usage of such server is outside the scope of this documentation. Please refer to the SD A111 WebUI repo: https://github.com/AUTOMATIC1111/stable-diffusion-webui
Launch Stable DIffusion WebUI Automatic 1111 with the '--api' argument in the command line define in the "webui-user.bat" command file, in the root folder of your SD server. Without Authentication: set COMMANDLINE_ARGS= --api ; Or with Authentication: set COMMANDLINE_ARGS=--listen --api --api-auth YOUR_USERNAME:YOUR_PASSWORD --cors-allow-origins=*
Either, Download the stable-diffusion-unity-integration repository, for example by running git clone https://github.com/dobrado76/Stable-Diffusion-Unity-Integration.git
. Then, Open the stable-diffusion-unity-integration project in Unity.then open the Demo Scene provided in the package found in StableDiffusionIntegration/Scenes/DemoScene.unity
.
Or;
Create and open a new Unity project for Built-in Pipeline, download and import the stable-diffusion-unity-integration.unitypackage file into the project (https://github.com/dobrado76/Stable-Diffusion-Unity-Integration/blob/main/Stable-Diffusion-Unity-Integration.unitypackage). By starting from a new project and importing the package, none of the demo pre-generated images will be available in the Asset folder but the project should work as-is, since the textures and images are embedded in the Unity project.
Open the Demo Scene provided in the package found in StableDiffusionIntegration/Scenes/DemoScene.unity
. There is a version with Post-Processing (you need to install the Post-Processing package, you will need to re-open the scene after installing the package) and a version without Post-Processing.
Select the StableDiffusionConfiguration component and make sure the selected setting point to the correct URL and Port (default: http://127.0.0.1:7860/).
Click on the List Model
button to get the list of all available models from your local Stable Diffusion server.
Be aware of a few limitations:
To contribute, clone the repository, make your changes, commit and push to your clone, and submit a pull request (PR).
Before submitting a PR, make sure that:
If you think you found a bug, please first make sure that you are using the latest version of both this repo (as well as the Stable Diffusion repo) and that the bug has not already been fixed. If you are submitting a bug fix, there must be a way for me to reproduce the bug. Please, provide step-by-step instructions on how to reproduce the bug.
Thanks to UnityCoder for a straightforward and functional algorithm to automatically generate a normal map from a texture. - https://github.com/unitycoder/NormalMapFromTexture. Also, thanks to ALBRRT, FOXYTOCIN, PeixuanL for their contributions here.
This repository is under Lesser General Public License (LGPL) v2.1, that you can find in the LICENSE
file (https://github.com/dobrado76/Stable-Diffusion-Unity-Integration/blob/72560920786b909e829b567567699f332256ea19/LICENSE).
Mainly, it allows you to:
Requires that the source code for the derative work be made available to others. This does not introduce any limitations on your own Unity projects, since this tool is a Design time, Editor only tool (your game or application is obviously not required to be open source). Requires that any partial, full or modified distribution of this repo mentions the original source (this repo/author).
2023-03-15: v1.2.6 - Added StableDiffusionImage2Material, thanks to PeixuanL.
2023-03-10: v1.2.5 - Merged Pull Request from FOXYTOCIN to add support for Authentication API keys and URP materials.
2023-02-02: v1.2.4 - Merged Pull Request from ALBRRT to add support for Image-2-Image for UI Image and RawImage. To that effect, renamed StableDiffusionImage to StableDiffusionText2Image and StableDiffusionMaterial to StableDiffusionText2Material, and added a new StableDiffusionImage2Image component.
2023-01-31: v1.2.3 - Merged Pull Request from ALBRRT to fix an issue with progress bar when generating an image with StableDiffusionImage. Not perfect but somewhat/sometimes working ok.
2023-01-31: v1.2.2 - Changed the call to Stable Diffusion server to be asynchronous and non-blocking. Also added a Progress Bar for generation progress feedback.
2023-01-30: v1.2.1 - Added some comments in the code and updated the package with new codebase, to include last merge and post-processing settings for the demo scene with post-processing.
2023-01-25: v1.2.0 - Merged Pull Request from ALBRRT to enable Editor assets refresh only when not in Play mode, or returning from Play mode. Also added the display of the GUID for easier reference between the Component and the Image file.
2023-01-24: v1.1.0 - Added handling of the samplers as generation parameter. Manual list in StableDiffusionConfiguration that can be culled to remove samplers that you don't use.
2023-01-23: v1.0.0 - Initial public distribution version 1.0.0 with brief documentation.