ScribbleTextToImage¶

class agentlego.tools.ScribbleTextToImage(model='sd', device='cuda', toolmeta=None)[source]

A tool to generate image according to a scribble sketch.

Parameters:

model (str) – The model name used to inference. Which can be found in the diffusers repository. Defaults to ‘lllyasviel/sd-controlnet_scribble’.
model – The scribble controlnet model to use. You can only choose “sd” by now. Defaults to “sd”.
device (str) – The device to load the model. Defaults to ‘cuda’.
toolmeta (None | dict | ToolMeta) – The additional info of the tool. Defaults to None.

Default Tool Meta¶

name: ScribbleTextToImage
description: This tool can generate an image from a sketch scribble image and a text.
inputs:
- image (ImageIO)
- keywords (str): A series of English keywords separated by comma.
outputs:
- ImageIO

Examples¶

Download the demo resource

wget http://download.openmmlab.com/agentlego/scribble.png

Use the tool directly (without agent)

from agentlego.apis import load_tool

# load tool
tool = load_tool('ScribbleTextToImage', device='cuda')

# apply tool
image = tool('scribble.png', 'a pair of cartoon style pets')
print(image)

With Lagent

from lagent import ReAct, GPTAPI, ActionExecutor
from agentlego.apis import load_tool

# load tools and build agent
# please set `OPENAI_API_KEY` in your environment variable.
tool = load_tool('ScribbleTextToImage', device='cuda').to_lagent()
agent = ReAct(GPTAPI(temperature=0.), action_executor=ActionExecutor([tool]))

# agent running with the tool.
img_path = 'scribble.png'
ret = agent.chat(f'According to the scribble sketch `{img_path}`, draw a pair of cartoon style cat and dog.')
for step in ret.inner_steps[1:]:
    print('------')
    print(step['content'])

Set up¶

Before using the tool, please confirm you have installed the related dependencies by the below commands.

pip install -U diffusers

Reference¶

This tool uses a Control Net model in default settings. See the following paper for details.

@misc{zhang2023adding,
      title={Adding Conditional Control to Text-to-Image Diffusion Models},
      author={Lvmin Zhang and Maneesh Agrawala},
      year={2023},
      eprint={2302.05543},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}