inference code added
Browse files
README.md
CHANGED
|
@@ -12,6 +12,7 @@ tags:
|
|
| 12 |
---
|
| 13 |
|
| 14 |
# Introduction
|
|
|
|
| 15 |
|
| 16 |
The λ-ECLIPSE model is a light weight support for multi-concept personalization. λ-ECLIPSE is tiny T2I prior model designed for Kandinsky v2.2 diffusion image generator.
|
| 17 |
|
|
@@ -32,14 +33,61 @@ More examples at: [Gallery](https://eclipse-t2i.github.io/Lambda-ECLIPSE/gallery
|
|
| 32 |
|
| 33 |
## Installation
|
| 34 |
```bash
|
| 35 |
-
git clone
|
| 36 |
conda create -p ./venv python=3.9
|
| 37 |
pip install -r requirements.txt
|
| 38 |
```
|
| 39 |
|
| 40 |
## Run Inference
|
|
|
|
| 41 |
|
| 42 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 43 |
|
| 44 |
## Important Notes (and limitations):
|
| 45 |
|
|
|
|
| 12 |
---
|
| 13 |
|
| 14 |
# Introduction
|
| 15 |
+
<a href="https://colab.research.google.com/drive/1VcqzXZmilntec3AsIyzCqlstEhX4Pa1o?usp=sharing" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
|
| 16 |
|
| 17 |
The λ-ECLIPSE model is a light weight support for multi-concept personalization. λ-ECLIPSE is tiny T2I prior model designed for Kandinsky v2.2 diffusion image generator.
|
| 18 |
|
|
|
|
| 33 |
|
| 34 |
## Installation
|
| 35 |
```bash
|
| 36 |
+
git clone https://github.com/eclipse-t2i/lambda-eclipse-inference.git
|
| 37 |
conda create -p ./venv python=3.9
|
| 38 |
pip install -r requirements.txt
|
| 39 |
```
|
| 40 |
|
| 41 |
## Run Inference
|
| 42 |
+
<a href="https://colab.research.google.com/drive/1VcqzXZmilntec3AsIyzCqlstEhX4Pa1o?usp=sharing" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
|
| 43 |
|
| 44 |
+
```bash
|
| 45 |
+
import os
|
| 46 |
+
import torch
|
| 47 |
+
from transformers import (
|
| 48 |
+
CLIPTextModelWithProjection,
|
| 49 |
+
CLIPTokenizer,
|
| 50 |
+
)
|
| 51 |
+
from src.pipelines.pipeline_kandinsky_subject_prior import KandinskyPriorPipeline
|
| 52 |
+
from src.priors.lambda_prior_transformer import PriorTransformer
|
| 53 |
+
from diffusers import DiffusionPipeline
|
| 54 |
+
|
| 55 |
+
text_encoder = CLIPTextModelWithProjection.from_pretrained(
|
| 56 |
+
"laion/CLIP-ViT-bigG-14-laion2B-39B-b160k",
|
| 57 |
+
projection_dim=1280,
|
| 58 |
+
torch_dtype=torch.float32,
|
| 59 |
+
)
|
| 60 |
+
tokenizer = CLIPTokenizer.from_pretrained("laion/CLIP-ViT-bigG-14-laion2B-39B-b160k")
|
| 61 |
+
|
| 62 |
+
prior = PriorTransformer.from_pretrained("ECLIPSE-Community/Lambda-ECLIPSE-Prior-v1.0")
|
| 63 |
+
pipe_prior = KandinskyPriorPipeline.from_pretrained(
|
| 64 |
+
"kandinsky-community/kandinsky-2-2-prior",
|
| 65 |
+
prior=prior,
|
| 66 |
+
text_encoder=text_encoder,
|
| 67 |
+
tokenizer=tokenizer,
|
| 68 |
+
).to("cuda")
|
| 69 |
+
|
| 70 |
+
pipe = DiffusionPipeline.from_pretrained(
|
| 71 |
+
"kandinsky-community/kandinsky-2-2-decoder"
|
| 72 |
+
).to("cuda")
|
| 73 |
+
|
| 74 |
+
raw_data = {
|
| 75 |
+
"prompt": args.prompt,
|
| 76 |
+
"subject_images": [args.subject1_path, args.subject2_path],
|
| 77 |
+
"subject_keywords": [args.subject1_name, args.subject2_name]
|
| 78 |
+
}
|
| 79 |
+
image_emb, negative_image_emb = pipe_prior(
|
| 80 |
+
raw_data=raw_data,
|
| 81 |
+
).to_tuple()
|
| 82 |
+
image = pipe(
|
| 83 |
+
image_embeds=image_emb,
|
| 84 |
+
negative_image_embeds=negative_image_emb,
|
| 85 |
+
num_inference_steps=50,
|
| 86 |
+
guidance_scale=7.5,
|
| 87 |
+
).images
|
| 88 |
+
|
| 89 |
+
image[0]
|
| 90 |
+
```
|
| 91 |
|
| 92 |
## Important Notes (and limitations):
|
| 93 |
|