DiffusionCraft AI (An InvokeAI Fork)

This is a fork that uses InvokeAI to process Minecraft images at realtime

DiffusionCraft AI Guide

Installation

• Follow the installation guide below for regular InvokeAI following the Windows instructions:
  Installing anaconda
  Cloning this repo
  cd to repo
  conda env create
  Instead of conda activate invokeai, run conda activate diffusioncraft
  python scripts/preload_models.py
  Download the latest sd checkpoint into models/ldm/stable-diffusion-v1\model.ckpt (Ensure it's named model.ckpt)
• Proceed to How to Use.

How to Use

> Stage A. Launching the Capture & Display

1. Open a command prompt window
2. cd to Project Directory
3. Run the command conda activate diffusioncraft
4. Then run the command: python scripts\dc_capture.py
5. Give it a Session ID (any string that it will use to uniquely identify this session)
6. With the command prompt in focus, move your cursor to the top left corner of the window to capture and press enter.
7. Repeat the same for the bottom right corner. This will define the boundaries of the region to capture. Note: If you are running this on seperate screens, this might result in a black screen. You can check the screen capture results in the outputs/img2img folder.

> Stage B. Launching Stable Diffusion

1. Open another command prompt window
2. cd to Project Directory
3. Run the command conda activate diffusioncraft
4. Then run the command: python scripts\dc_invoke.py
5. Enter the same Session ID you used for Stage B.
6. Enter a prompt key from the list.

Minecraft-Specific Instructions

To capture a timelapse-style video in Minecraft you will need to:

1. Run two copies of minecraft and have them playing in the same world (either through server/LAN party).
2. Position one player as the spectator looking at the creation. This is the screen you will target the dc_capture on.
3. The other player/s will build the creation. The builder can observe and gain inspiration from what details Stable Diffusion adds in on the display window.

How it Works

• There are two main scripts in this repo: scripts/dc_invoke.py and scripts/dc_capture.py
• dc_capture is used to take a screen capture of the game window
• dc_invoke searches for the newest screen capture in the outputs/img2img/{SESSION_ID}_input folder
• When dc_invoke finds a new image, it runs it through Stable Diffusion via Invoke AI, and outputs it into the outputs/img2img/{SESSION_ID}_output folder
• dc_capture also searches for the newest Stable Diffusion processed image in the outputs/img2img/{SESSION_ID}_output folder
• When dc_capture finds a new image, it displays it via the tk library

Additional Notes

• The settings to modify the prompts you choose between are in the settings.json file
• Session Identifier = Unique ID for Session (can be text / numbers)
• Prompt Key = The key for a prompt. Stored in the settings.json file. Saves having to type the entire prompt out each time (just reference the prompt by a single word).
• For best results, ensure the window you are capturing from has an aspect ratio similar to a square, the dc_capture will automatically resize the input to a square.
• This has only been tested on Windows so far.

InvokeAI: A Stable Diffusion Toolkit

Formerly known as lstein/stable-diffusion

This is a fork of CompVis/stable-diffusion, the open source text-to-image generator. It provides a streamlined process with various new features and options to aid the image generation process. It runs on Windows, Mac and Linux machines, with GPU cards with as little as 4 GB of RAM. It provides both a polished Web interface (see below), and an easy-to-use command-line interface.

Quick links: [Discord Server] [Documentation and Tutorials] [Code and Downloads] [Bug Reports] [Discussion, Ideas & Q&A]

Note: This fork is rapidly evolving. Please use the Issues tab to report bugs and make feature requests. Be sure to use the provided templates. They will help aid diagnose issues faster.

Table of Contents

1. Installation
2. Hardware Requirements
3. Features
4. Latest Changes
5. Troubleshooting
6. Contributing
7. Contributors
8. Support
9. Further Reading

Installation

This fork is supported across multiple platforms. You can find individual installation instructions below.

• Linux

• Windows

• Macintosh

Hardware Requirements

System

You wil need one of the following:

• An NVIDIA-based graphics card with 4 GB or more VRAM memory.
• An Apple computer with an M1 chip.

Memory

• At least 12 GB Main Memory RAM.

Disk

• At least 12 GB of free disk space for the machine learning model, Python, and all its dependencies.

Note

If you have a Nvidia 10xx series card (e.g. the 1080ti), please run the dream script in full-precision mode as shown below.

Similarly, specify full-precision mode on Apple M1 hardware.

Precision is auto configured based on the device. If however you encounter errors like 'expected type Float but found Half' or 'not implemented for Half' you can try starting invoke.py with the --precision=float32 flag:

(invokeai) ~/InvokeAI$ python scripts/invoke.py --precision=float32

Features

Major Features

• Web Server
• Interactive Command Line Interface
• Image To Image
• Inpainting Support
• Outpainting Support
• Upscaling, face-restoration and outpainting
• Reading Prompts From File
• Prompt Blending
• Thresholding and Perlin Noise Initialization Options
• Negative/Unconditioned Prompts
• Variations
• Personalizing Text-to-Image Generation
• Simplified API for text to image generation

Other Features

• Google Colab
• Seamless Tiling
• Shortcut: Reusing Seeds
• Preload Models

Latest Changes

• v2.0.1 (13 October 2022)

  • fix noisy images at high step count when using k* samplers
  • dream.py script now calls invoke.py module directly rather than via a new python process (which could break the environment)

• v2.0.0 (9 October 2022)

  • dream.py script renamed invoke.py. A dream.py script wrapper remains for backward compatibility.
  • Completely new WebGUI - launch with python3 scripts/invoke.py --web
  • Support for inpainting and outpainting
  • img2img runs on all k* samplers
  • Support for negative prompts
  • Support for CodeFormer face reconstruction
  • Support for Textual Inversion on Macintoshes
  • Support in both WebGUI and CLI for post-processing of previously-generated images using facial reconstruction, ESRGAN upscaling, outcropping (similar to DALL-E infinite canvas), and "embiggen" upscaling. See the !fix command.
  • New --hires option on invoke> line allows larger images to be created without duplicating elements, at the cost of some performance.
  • New --perlin and --threshold options allow you to add and control variation during image generation (see Thresholding and Perlin Noise Initialization
  • Extensive metadata now written into PNG files, allowing reliable regeneration of images and tweaking of previous settings.
  • Command-line completion in invoke.py now works on Windows, Linux and Mac platforms.
  • Improved command-line completion behavior. New commands added:
    • List command-line history with !history
    • Search command-line history with !search
    • Clear history with !clear
  • Deprecated --full_precision / -F. Simply omit it and invoke.py will auto configure. To switch away from auto use the new flag like --precision=float32.

For older changelogs, please visit the CHANGELOG.

Troubleshooting

Please check out our Q&A to get solutions for common installation problems and other issues.

Contributing

Anyone who wishes to contribute to this project, whether documentation, features, bug fixes, code cleanup, testing, or code reviews, is very much encouraged to do so. If you are unfamiliar with how to contribute to GitHub projects, here is a Getting Started Guide.

A full set of contribution guidelines, along with templates, are in progress, but for now the most important thing is to make your pull request against the "development" branch, and not against "main". This will help keep public breakage to a minimum and will allow you to propose more radical changes.

Contributors

This fork is a combined effort of various people from across the world. Check out the list of all these amazing people. We thank them for their time, hard work and effort.

Support

For support, please use this repository's GitHub Issues tracking service. Feel free to send me an email if you use and like the script.

Original portions of the software are Copyright (c) 2020 Lincoln D. Stein

Further Reading

Please see the original README for more information on this software and underlying algorithm, located in the file README-CompViz.md.